release: update go.mod for v1.14.0

* fix(eks): GPU node AMI declares uefi boot mode

The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but
the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under
legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at
firmware with zero serial console output.

* fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs

Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
initialize under the closed/proprietary kernel module: dmesg shows
RmInitAdapter failing with "requires use of the NVIDIA open kernel
modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to
nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia
recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no
separate open variant (userspace tools only), so it is unchanged.

* fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons

A cluster with a GPU-only nodegroup has no untainted node: the CP
carries CriticalAddonsOnly (EKS parity keeps user workloads off it)
and every worker carries nvidia.com/gpu. Without tolerations for both,
ebs-csi-controller and aws-load-balancer-controller have nowhere to
schedule and stay Pending forever. Real EKS never needs this since its
control plane isn't a schedulable node, so the stock upstream
manifests carry no such toleration.

* fix(images): give the libguestfs appliance a working resolver

The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
(127.0.0.53), which isn't running inside the appliance chroot, so DNS
is dead until package installs need it. Point it at a real resolver
before package installs, then restore the stub symlink afterward so
the shipped AMI still gets its DNS from systemd-resolved at boot like
a stock cloud image.

* fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind

A GPU's IOMMU group can include an upstream PCIe bridge when the
platform lacks ACS isolation between them. vfio-pci refuses to bind
non-endpoint bridge devices, so Claim/ReclaimByAddress must filter
bridge-class (0x06) members out of the group before touching driver
bindings, rather than failing the whole claim or leaving the bridge
rebound incorrectly.

* fix(ecs): GPU node AMI declares uefi boot mode

The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the
Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS
no OVMF/pflash firmware is attached, so the guest hangs at firmware with
zero serial console output.

Same defect and same fix as the eks-node-gpu entry: both images are built
by the same script from the same stock Ubuntu cloudimg, so both were
EFI-only and only one was corrected.

* fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch

ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary
kernel module. NVIDIA ships no proprietary module for Blackwell (GB202)
and later, so ECS GPU container-instances could not bind those cards on
any driver branch. It is the third verbatim copy of the routine and was
missed when the other two were fixed.

Also pins the driver branch across all three copies. The branch was
resolved at build time via apt-cache search | sort -rn | head -1, which
sorts on the number in the package name — but that number is a branch
alias, not a driver version (nvidia-dkms-535-server installs 580.126.20),
so the ordering key need not match what installs. Resolving also let
identical source produce different drivers on different days.

The pin lives in setup.sh rather than manifest.conf because manifest
variables are not passed into the guest: the setup script is uploaded and
executed by virt-customize --run, with no environment carried across.

* fix(eks): tolerate CP and GPU taints in argocd addon

All six argocd Deployments and its StatefulSet carried no tolerations. A
cluster with a GPU-only nodegroup has no untainted node to place them on:
the control plane carries CriticalAddonsOnly and every worker carries
nvidia.com/gpu, so the pods stay Pending forever.

Same defect the ebs-csi-controller and load-balancer-controller addons
already had. Real EKS never hits it because its control plane is not a
schedulable node, so the stock upstream manifests carry no such
toleration and the divergence is ours to maintain.

---------

Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>

.githooks/pre-commit

@@ -1,17 +0,0 @@
-#!/usr/bin/env bash
-set -e
-
-branch=$(git rev-parse --abbrev-ref HEAD)
-
-# Only enforce preflight on dev and main
-if [ "$branch" != "dev" ] && [ "$branch" != "main" ]; then
-  exit 0
-fi
-
-# Only run preflight if Go files were staged
-if git diff --cached --name-only | grep -q '\.go$'; then
-  echo "Go files staged on $branch — running make preflight..."
-  make preflight
-else
-  echo "No Go files staged, skipping preflight."
-fi

.github/actions/otel-job-id/action.yml

@@ -0,0 +1,52 @@
+name: Resolve OTEL job id
+description: >-
+  Append ci.job_id to SPINIFEX_OTEL_ATTRS so telemetry emitted by this job can
+  be pivoted straight back to its GitHub job log. Must run before the step that
+  bakes SPINIFEX_OTEL_ATTRS into a VM's telemetry.env, and requires
+  `actions: read` on the calling job.
+
+inputs:
+  job-name:
+    description: >-
+      Rendered name of the calling job, exactly as it appears in the Actions UI.
+      github.job is only the YAML key and no context exposes the numeric id, so
+      the id has to be looked up by name.
+    required: true
+  token:
+    description: Token used for the Actions jobs API lookup.
+    required: false
+    default: ${{ github.token }}
+
+runs:
+  using: composite
+  steps:
+    - name: Resolve CI job id (best-effort)
+      shell: bash
+      env:
+        GH_TOKEN: ${{ inputs.token }}
+        JOB_NAME: ${{ inputs.job-name }}
+        REPO: ${{ github.repository }}
+        RUN_ID: ${{ github.run_id }}
+      run: |
+        # Never fail the caller: on any lookup error or name mismatch ci.job_id
+        # is simply omitted and the run proceeds with the labels it already has.
+        set +e
+
+        if [ -z "${SPINIFEX_OTEL_ATTRS:-}" ]; then
+          echo "SPINIFEX_OTEL_ATTRS is unset or empty — telemetry is disabled, nothing to label"
+          exit 0
+        fi
+
+        # env.JOB_NAME rather than shell interpolation into the jq filter: job
+        # names carry quotes, slashes and non-ASCII separators that would
+        # otherwise break the expression.
+        JOB_ID=$(gh api "repos/${REPO}/actions/runs/${RUN_ID}/jobs" --paginate \
+          --jq '.jobs[] | select(.name == env.JOB_NAME) | .id' 2>/dev/null | head -n1)
+
+        if [ -n "$JOB_ID" ]; then
+          echo "SPINIFEX_OTEL_ATTRS=${SPINIFEX_OTEL_ATTRS},ci.job_id=${JOB_ID}" >> "$GITHUB_ENV"
+          echo "resolved ci.job_id=${JOB_ID} for job '${JOB_NAME}'"
+        else
+          echo "::warning::could not resolve numeric job id for '${JOB_NAME}' — continuing without ci.job_id"
+        fi
+        exit 0

.github/assets/architecture.svg

@@ -1,225 +1,114 @@
-<svg xmlns="http://www.w3.org/2000/svg" width="900" height="620" viewBox="0 0 900 620">
-  <defs>
-    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
-      <stop offset="0%" style="stop-color:#0a0a0a"/>
-      <stop offset="100%" style="stop-color:#0d1117"/>
-    </linearGradient>
-    <linearGradient id="bar-grad" x1="0%" y1="0%" x2="100%" y2="0%">
-      <stop offset="0%" style="stop-color:#161b22"/>
-      <stop offset="45%" style="stop-color:#1f2028"/>
-      <stop offset="100%" style="stop-color:#161b22"/>
-    </linearGradient>
-    <linearGradient id="client-grad" x1="0%" y1="0%" x2="100%" y2="0%">
-      <stop offset="0%" style="stop-color:#3fb95022"/>
-      <stop offset="100%" style="stop-color:#3fb95008"/>
-    </linearGradient>
-    <linearGradient id="gw-grad" x1="0%" y1="0%" x2="100%" y2="0%">
-      <stop offset="0%" style="stop-color:#DC262633"/>
-      <stop offset="100%" style="stop-color:#7F1D1D14"/>
-    </linearGradient>
-    <linearGradient id="nats-grad" x1="0%" y1="0%" x2="100%" y2="0%">
-      <stop offset="0%" style="stop-color:#F87171"/>
-      <stop offset="50%" style="stop-color:#DC2626"/>
-      <stop offset="100%" style="stop-color:#7F1D1D"/>
-    </linearGradient>
-  </defs>
-
-  <rect width="900" height="620" fill="url(#bg)" rx="12"/>
-
-  <!-- Title bar -->
-  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">SPINIFEX ARCHITECTURE</text>
-  <text x="24" y="52" font-family="'Segoe UI', system-ui, sans-serif" font-size="16" fill="#f5f5f5" font-weight="500">AWS SDK → Gateway → NATS → Service daemons</text>
-  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">stateless · message-driven · horizontally scalable</text>
-
-  <!-- ═══ CLIENT LAYER ════════════════════════════════════ -->
-  <rect x="16" y="72" width="868" height="78" rx="8" fill="url(#client-grad)" stroke="#3fb950" stroke-width="1" stroke-opacity="0.5"/>
-  <text x="32" y="93" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" font-weight="700" letter-spacing="1">CLIENTS</text>
-  <text x="32" y="108" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">Standard AWS tooling — unmodified, no shims, no custom SDK</text>
-
-  <g transform="translate(32, 120)" font-family="'SF Mono', ui-monospace, monospace" font-size="10">
-    <rect x="0" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="43" y="15" fill="#7ee787" text-anchor="middle">aws-cli</text>
-
-    <rect x="94" y="0" width="96" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="142" y="15" fill="#7ee787" text-anchor="middle">Terraform</text>
-
-    <rect x="198" y="0" width="76" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="236" y="15" fill="#7ee787" text-anchor="middle">boto3</text>
-
-    <rect x="282" y="0" width="98" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="331" y="15" fill="#7ee787" text-anchor="middle">aws-sdk-go</text>
-
-    <rect x="388" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="431" y="15" fill="#7ee787" text-anchor="middle">kubectl</text>
-
-    <rect x="482" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="525" y="15" fill="#7ee787" text-anchor="middle">Packer</text>
-
-    <rect x="576" y="0" width="120" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="636" y="15" fill="#7ee787" text-anchor="middle">your application</text>
-  </g>
-
-  <!-- ═══ HTTPS + SigV4 connector ═══════════════════════════ -->
-  <line x1="450" y1="150" x2="450" y2="178" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
-  <rect x="390" y="154" width="120" height="18" rx="9" fill="#0a0a0a" stroke="#DC2626" stroke-width="0.8" stroke-opacity="0.5"/>
-  <text x="450" y="166" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#F87171" text-anchor="middle">HTTPS · SigV4 · :9999</text>
-
-  <!-- ═══ AWS GATEWAY ════════════════════════════════════ -->
-  <rect x="16" y="178" width="868" height="68" rx="8" fill="url(#gw-grad)" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.75"/>
-  <text x="32" y="200" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" font-weight="700" letter-spacing="1">AWS GATEWAY</text>
-  <text x="32" y="215" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">spinifex/services/awsgw · TLS termination · SigV4 verification · IAM policy eval · NATS routing</text>
-
-  <g transform="translate(32, 224)" font-family="'SF Mono', ui-monospace, monospace" font-size="9">
-    <rect x="0" y="0" width="78" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="39" y="11" fill="#FCA5A5" text-anchor="middle">TLS 1.3</text>
-
-    <rect x="86" y="0" width="78" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="125" y="11" fill="#FCA5A5" text-anchor="middle">SigV4 auth</text>
-
-    <rect x="172" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="221" y="11" fill="#FCA5A5" text-anchor="middle">policy eval</text>
-
-    <rect x="278" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="327" y="11" fill="#FCA5A5" text-anchor="middle">req routing</text>
-
-    <rect x="384" y="0" width="88" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="428" y="11" fill="#FCA5A5" text-anchor="middle">rate-limit</text>
-
-    <rect x="480" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="529" y="11" fill="#FCA5A5" text-anchor="middle">audit log</text>
-  </g>
-
-  <!-- ═══ NATS BUS ═══════════════════════════════════════ -->
-  <line x1="450" y1="246" x2="450" y2="272" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
-  <rect x="16" y="272" width="868" height="34" rx="6" fill="url(#nats-grad)" opacity="0.92"/>
-  <text x="32" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#f5f5f5" font-weight="700" letter-spacing="1.5">NATS BUS</text>
-  <text x="140" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#fee2e2" opacity="0.95">ec2.*   ebs.*   s3.*   vpc.*   iam.*</text>
-  <text x="876" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#fee2e2" text-anchor="end" opacity="0.9">queue groups · fan-out · request/reply</text>
-
-  <!-- ═══ DAEMON FANOUT CONNECTORS ═══════════════════════ -->
-  <g stroke="#DC2626" stroke-width="0.9" stroke-dasharray="3,3" opacity="0.55" fill="none">
-    <line x1="100" y1="306" x2="100" y2="328"/>
-    <line x1="275" y1="306" x2="275" y2="328"/>
-    <line x1="450" y1="306" x2="450" y2="328"/>
-    <line x1="625" y1="306" x2="625" y2="328"/>
-    <line x1="800" y1="306" x2="800" y2="328"/>
-  </g>
-
-  <!-- ═══ SERVICE DAEMONS ROW ═══════════════════════════ -->
-  <!-- EC2 -->
-  <g transform="translate(28, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">ec2.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">RunInstances</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">TerminateInstances</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">QMP · cloud-init</text>
-  </g>
-
-  <!-- EBS -->
-  <g transform="translate(204, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">ebs.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">CreateVolume</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">AttachVolume</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">snapshots · WAL</text>
-  </g>
-
-  <!-- S3 -->
-  <g transform="translate(380, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">s3.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">PutObject · GetObject</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Multipart · presigned</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">erasure-coded</text>
-  </g>
-
-  <!-- VPC -->
-  <g transform="translate(556, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">vpc.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Subnets · SGs · RTs</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">ALB · NAT · IGW</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">OVN · OVS</text>
-  </g>
-
-  <!-- IAM -->
-  <g transform="translate(732, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">iam.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Users · Roles · Keys</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Policy evaluation</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">audit trail</text>
-  </g>
-
-  <!-- ═══ DAEMON → BACKEND CONNECTORS ═══════════════════ -->
-  <g stroke="#71717a" stroke-width="0.8" stroke-dasharray="2,3" opacity="0.55" fill="none">
-    <line x1="100" y1="410" x2="100" y2="440"/>
-    <line x1="276" y1="410" x2="276" y2="440"/>
-    <line x1="452" y1="410" x2="452" y2="440"/>
-    <line x1="628" y1="410" x2="628" y2="440"/>
-    <line x1="804" y1="410" x2="804" y2="440"/>
-  </g>
-
-  <!-- ═══ BACKEND RESOURCES BAR ═════════════════════════ -->
-  <rect x="16" y="440" width="868" height="26" rx="5" fill="url(#bar-grad)"/>
-  <text x="450" y="458" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle" letter-spacing="1.5">BACKEND RESOURCES · local hardware · no cloud dependency</text>
-
-  <!-- ═══ BACKEND ROW ═══════════════════════════════════ -->
-  <!-- QEMU/KVM -->
-  <g transform="translate(28, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">hypervisor</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">host kernel · virtio</text>
-  </g>
-
-  <!-- Viperblock -->
-  <g transform="translate(204, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Viperblock</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">block storage</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">NVMe WAL · NBD</text>
-  </g>
-
-  <!-- Predastore -->
-  <g transform="translate(380, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Predastore</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">object storage</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">Reed–Solomon</text>
-  </g>
-
-  <!-- OVN/OVS -->
-  <g transform="translate(556, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">OVN / OVS</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">SDN fabric</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">overlays · flow rules</text>
-  </g>
-
-  <!-- IAM store -->
-  <g transform="translate(732, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Identity store</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">users · keys · rules</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">local · audit-logged</text>
-  </g>
-
-  <!-- ═══ FOOTER NOTES ══════════════════════════════════ -->
-  <g transform="translate(16, 558)">
-    <rect width="868" height="48" rx="6" fill="#0a0a0a" stroke="#27272a" stroke-width="0.8"/>
-    <text x="24" y="20" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">
-      <tspan fill="#F87171" font-weight="600">Queue groups</tspan> load-balance ops across daemon replicas.
-      <tspan fill="#F87171" font-weight="600"> Fan-out</tspan> (no queue group) dispatches describe calls to every node.
-    </text>
-    <text x="24" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">
-      Daemons are <tspan fill="#F87171" font-weight="600">stateless</tspan> — scale horizontally by starting more.
-      No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.
-    </text>
-  </g>
-</svg>
+<svg xmlns="http://www.w3.org/2000/svg" width="1020" height="770" viewBox="0 0 1020 770" role="img" aria-label="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus; stateless ec2/ebs/s3/vpc/iam daemons answer and drive local backends (QEMU/KVM, Viperblock, Predastore, OVN/OVS, identity store).">
+<defs><linearGradient id="nats" x1="0" y1="0" x2="1" y2="0"><stop offset="0" stop-color="#F5652E"/><stop offset="1" stop-color="#C63C0E"/></linearGradient></defs>
+<rect x="0.0" y="0.0" width="1020.0" height="770.0" rx="20" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="28.0" y="42.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#EE4B12" font-weight="600" text-anchor="start" letter-spacing="2.5">SPINIFEX ARCHITECTURE</text>
+<text x="992.0" y="42.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#9AA1A8" font-weight="400" text-anchor="end">stateless · message-driven · horizontally scalable</text>
+<text x="28.0" y="72.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="23" fill="#F5F3EE" font-weight="800" text-anchor="start">AWS SDK → Gateway → NATS → service daemons</text>
+<rect x="28.0" y="92.0" width="964.0" height="92.0" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="48.0" y="119.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">CLIENTS</text>
+<text x="48.0" y="140.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#5B636C" font-weight="400" text-anchor="start">Standard AWS tooling, unmodified. No shims, no custom SDK.</text>
+<rect x="46.0" y="153.0" width="68.2" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="80.1" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">aws-cli</text>
+<rect x="123.2" y="153.0" width="81.4" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="163.9" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">Terraform</text>
+<rect x="213.6" y="153.0" width="55.0" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="241.1" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">boto3</text>
+<rect x="277.6" y="153.0" width="88.0" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="321.6" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">aws-sdk-go</text>
+<rect x="374.6" y="153.0" width="68.2" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="408.7" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">kubectl</text>
+<rect x="451.8" y="153.0" width="61.6" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="482.6" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">Packer</text>
+<rect x="522.4" y="153.0" width="74.8" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="559.8" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">your app</text>
+<line x1="510.0" y1="186.0" x2="510.0" y2="192.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
+<rect x="429.9" y="191.0" width="160.3" height="22.0" rx="11" fill="#1E242B" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.4"/>
+<text x="510.0" y="205.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#EE4B12" font-weight="400" text-anchor="middle" fill-opacity="0.9">HTTPS · SigV4 · :9999</text>
+<line x1="510.0" y1="212.0" x2="510.0" y2="220.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
+<rect x="28.0" y="224.0" width="964.0" height="94.0" rx="13" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
+<text x="48.0" y="251.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#EE4B12" font-weight="700" text-anchor="start" letter-spacing="1.5">AWS GATEWAY</text>
+<text x="48.0" y="272.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">spinifex/services/awsgw · TLS termination · SigV4 verification · IAM policy eval · NATS routing</text>
+<rect x="46.0" y="287.0" width="68.2" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="80.1" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">TLS 1.3</text>
+<rect x="123.2" y="287.0" width="88.0" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="167.2" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">SigV4 auth</text>
+<rect x="220.2" y="287.0" width="94.6" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="267.5" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">policy eval</text>
+<rect x="323.8" y="287.0" width="94.6" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="371.1" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">req routing</text>
+<rect x="427.4" y="287.0" width="88.0" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="471.4" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">rate-limit</text>
+<rect x="524.4" y="287.0" width="81.4" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="565.1" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">audit log</text>
+<line x1="510.0" y1="320.0" x2="510.0" y2="332.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
+<rect x="28.0" y="334.0" width="964.0" height="52.0" rx="12" fill="url(#nats)" fill-opacity="1.0" stroke="none" stroke-width="1" stroke-opacity="1.0"/>
+<rect x="28.0" y="334.0" width="964.0" height="52.0" rx="12" fill="none" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55"/>
+<text x="50.0" y="365.6" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="16" fill="#F5F3EE" font-weight="800" text-anchor="start">NATS BUS</text>
+<text x="178.0" y="364.6" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#F5F3EE" font-weight="400" text-anchor="start" fill-opacity="0.92">ec2.*  ebs.*  s3.*  vpc.*  iam.*</text>
+<text x="972.0" y="363.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F5F3EE" font-weight="400" text-anchor="end" fill-opacity="0.9">queue groups · fan-out · request/reply</text>
+<line x1="119.6" y1="388.0" x2="119.6" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<line x1="314.8" y1="388.0" x2="314.8" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<line x1="510.0" y1="388.0" x2="510.0" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<line x1="705.2" y1="388.0" x2="705.2" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<line x1="900.4" y1="388.0" x2="900.4" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<rect x="28.0" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="44.0" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">ec2.* daemon</text>
+<line x1="44.0" y1="448.0" x2="195.2" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="44.0" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">RunInstances</text>
+<text x="44.0" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">TerminateInstances</text>
+<text x="44.0" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">QMP · cloud-init</text>
+<rect x="223.2" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="239.2" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">ebs.* daemon</text>
+<line x1="239.2" y1="448.0" x2="390.4" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="239.2" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">CreateVolume</text>
+<text x="239.2" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">AttachVolume</text>
+<text x="239.2" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">snapshots · WAL</text>
+<rect x="418.4" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="434.4" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">s3.* daemon</text>
+<line x1="434.4" y1="448.0" x2="585.6" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="434.4" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">PutObject · GetObject</text>
+<text x="434.4" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">Multipart · presigned</text>
+<text x="434.4" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">erasure-coded</text>
+<rect x="613.6" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="629.6" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">vpc.* daemon</text>
+<line x1="629.6" y1="448.0" x2="780.8" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="629.6" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">Subnets · SGs · RTs</text>
+<text x="629.6" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">ALB · NAT · IGW</text>
+<text x="629.6" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">OVN · OVS</text>
+<rect x="808.8" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="824.8" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">iam.* daemon</text>
+<line x1="824.8" y1="448.0" x2="976.0" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="824.8" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">Users · Roles · Keys</text>
+<text x="824.8" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">Policy evaluation</text>
+<text x="824.8" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">audit trail</text>
+<line x1="119.6" y1="516.0" x2="119.6" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<line x1="314.8" y1="516.0" x2="314.8" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<line x1="510.0" y1="516.0" x2="510.0" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<line x1="705.2" y1="516.0" x2="705.2" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<line x1="900.4" y1="516.0" x2="900.4" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<rect x="28.0" y="530.0" width="964.0" height="30.0" rx="9" fill="#1E242B" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="510.0" y="548.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle" letter-spacing="1">BACKEND RESOURCES  ·  local hardware  ·  no cloud dependency</text>
+<rect x="28.0" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="119.6" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">QEMU / KVM</text>
+<text x="119.6" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">hypervisor</text>
+<text x="119.6" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">host kernel · virtio</text>
+<rect x="223.2" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="314.8" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">Viperblock</text>
+<text x="314.8" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">block storage</text>
+<text x="314.8" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">NVMe WAL · NBD</text>
+<rect x="418.4" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="510.0" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">Predastore</text>
+<text x="510.0" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">object storage</text>
+<text x="510.0" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Reed-Solomon</text>
+<rect x="613.6" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="705.2" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">OVN / OVS</text>
+<text x="705.2" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">SDN fabric</text>
+<text x="705.2" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">overlays · flow rules</text>
+<rect x="808.8" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="900.4" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">Identity store</text>
+<text x="900.4" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">users · keys · rules</text>
+<text x="900.4" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">local · audit-logged</text>
+<rect x="28.0" y="670.0" width="964.0" height="72.0" rx="11" fill="#181D23" fill-opacity="0.6" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="48.0" y="697.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400"><tspan fill="#EE4B12" font-weight="700">Queue groups</tspan><tspan fill="#5B636C"> load-balance ops across daemon replicas. </tspan><tspan fill="#EE4B12" font-weight="700">Fan-out</tspan><tspan fill="#5B636C"> (no queue group) dispatches describe calls to every node.</tspan></text>
+<text x="48.0" y="720.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400"><tspan fill="#5B636C">Daemons are </tspan><tspan fill="#EE4B12" font-weight="700">stateless</tspan><tspan fill="#5B636C">: scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.</tspan></text>
+</svg>

.github/assets/platform.svg

@@ -1,144 +1,84 @@
-<svg xmlns="http://www.w3.org/2000/svg" width="900" height="560" viewBox="0 0 900 560">
-  <defs>
-    <linearGradient id="pf-bg" x1="0%" y1="0%" x2="100%" y2="100%">
-      <stop offset="0%" style="stop-color:#0a0a0a"/>
-      <stop offset="100%" style="stop-color:#0d1117"/>
-    </linearGradient>
-    <linearGradient id="hw-grad" x1="0%" y1="0%" x2="0%" y2="100%">
-      <stop offset="0%" style="stop-color:#52525b;stop-opacity:0.18"/>
-      <stop offset="100%" style="stop-color:#52525b;stop-opacity:0.06"/>
-    </linearGradient>
-    <linearGradient id="os-grad" x1="0%" y1="0%" x2="0%" y2="100%">
-      <stop offset="0%" style="stop-color:#a855f7;stop-opacity:0.18"/>
-      <stop offset="100%" style="stop-color:#a855f7;stop-opacity:0.05"/>
-    </linearGradient>
-    <linearGradient id="spx-grad" x1="0%" y1="0%" x2="0%" y2="100%">
-      <stop offset="0%" style="stop-color:#DC2626;stop-opacity:0.28"/>
-      <stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0.1"/>
-    </linearGradient>
-    <linearGradient id="app-grad" x1="0%" y1="0%" x2="0%" y2="100%">
-      <stop offset="0%" style="stop-color:#3fb950;stop-opacity:0.18"/>
-      <stop offset="100%" style="stop-color:#3fb950;stop-opacity:0.05"/>
-    </linearGradient>
-  </defs>
-
-  <rect width="900" height="560" fill="url(#pf-bg)" rx="12"/>
-
-  <!-- Subtle mesh dots background -->
-  <g opacity="0.12" fill="#7F1D1D">
-    <circle cx="40" cy="40" r="1"/><circle cx="120" cy="30" r="1"/><circle cx="220" cy="50" r="1"/>
-    <circle cx="320" cy="25" r="1"/><circle cx="440" cy="45" r="1"/><circle cx="560" cy="30" r="1"/>
-    <circle cx="680" cy="50" r="1"/><circle cx="780" cy="30" r="1"/><circle cx="860" cy="45" r="1"/>
-    <circle cx="60" cy="530" r="1"/><circle cx="160" cy="520" r="1"/><circle cx="260" cy="540" r="1"/>
-    <circle cx="380" cy="525" r="1"/><circle cx="500" cy="540" r="1"/><circle cx="620" cy="520" r="1"/>
-    <circle cx="740" cy="540" r="1"/><circle cx="840" cy="525" r="1"/>
-  </g>
-
-  <!-- Title -->
-  <text x="450" y="32" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" text-anchor="middle" letter-spacing="2.8">THE MULGA PLATFORM</text>
-  <text x="450" y="58" font-family="'Segoe UI', system-ui, sans-serif" font-size="20" font-weight="600" fill="#f5f5f5" text-anchor="middle">AWS-style workflows, on your own stack</text>
-
-  <!-- Layer 4: Applications (top) -->
-  <g transform="translate(60, 95)">
-    <rect width="780" height="88" rx="10" fill="url(#app-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
-    <text x="20" y="26" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" letter-spacing="1.5" font-weight="600">LAYER 4 · APPLICATIONS &amp; TOOLING</text>
-    <text x="20" y="48" font-family="'Segoe UI', sans-serif" font-size="15" fill="#f5f5f5" font-weight="500">Your workloads — unmodified AWS tooling</text>
-
-    <g transform="translate(20, 60)" font-family="'Segoe UI', sans-serif" font-size="10.5">
-      <rect x="0" y="0" width="70" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="35" y="14" fill="#7ee787" text-anchor="middle">aws-cli</text>
-
-      <rect x="78" y="0" width="78" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="117" y="14" fill="#7ee787" text-anchor="middle">Terraform</text>
-
-      <rect x="164" y="0" width="88" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="208" y="14" fill="#7ee787" text-anchor="middle">AWS SDKs</text>
-
-      <rect x="260" y="0" width="88" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="304" y="14" fill="#7ee787" text-anchor="middle">Kubernetes</text>
-
-      <rect x="356" y="0" width="70" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="391" y="14" fill="#7ee787" text-anchor="middle">Packer</text>
-
-      <rect x="434" y="0" width="98" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="483" y="14" fill="#7ee787" text-anchor="middle">Your Services</text>
-    </g>
-  </g>
-
-  <!-- Down arrow -->
-  <g transform="translate(450, 188)" opacity="0.45">
-    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
-    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
-  </g>
-
-  <!-- Layer 3: Spinifex (highlighted) -->
-  <g transform="translate(60, 215)">
-    <rect width="780" height="140" rx="10" fill="url(#spx-grad)" stroke="#DC2626" stroke-width="1.5" stroke-opacity="0.75"/>
-    <text x="20" y="26" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" letter-spacing="1.5" font-weight="600">LAYER 3 · SPINIFEX · AWS ORCHESTRATION</text>
-    <text x="20" y="50" font-family="'Segoe UI', sans-serif" font-size="15" fill="#f5f5f5" font-weight="500">AWS Gateway → NATS → Service Daemons · SigV4, IAM, VPC-aware</text>
-
-    <g transform="translate(20, 66)" font-family="'Segoe UI', sans-serif">
-      <!-- EC2 -->
-      <rect x="0" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
-      <text x="59" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">EC2</text>
-      <text x="59" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">QEMU/KVM VMs</text>
-      <text x="59" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">cloud-init · AMIs</text>
-
-      <!-- EBS -->
-      <rect x="126" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
-      <text x="185" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">EBS</text>
-      <text x="185" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">Viperblock</text>
-      <text x="185" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">WAL · NBD · snapshots</text>
-
-      <!-- S3 -->
-      <rect x="252" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
-      <text x="311" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">S3</text>
-      <text x="311" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">Predastore</text>
-      <text x="311" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Reed–Solomon EC</text>
-
-      <!-- VPC -->
-      <rect x="378" y="0" width="118" height="58" rx="6" fill="#7F1D1D" fill-opacity="0.22" stroke="#F87171" stroke-opacity="0.55" stroke-width="1"/>
-      <text x="437" y="18" font-size="13" fill="#FCA5A5" text-anchor="middle" font-weight="600">VPC</text>
-      <text x="437" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">OVN / OVS</text>
-      <text x="437" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">subnets · SG · ALB</text>
-
-      <!-- IAM -->
-      <rect x="504" y="0" width="118" height="58" rx="6" fill="#7F1D1D" fill-opacity="0.22" stroke="#F87171" stroke-opacity="0.55" stroke-width="1"/>
-      <text x="563" y="18" font-size="13" fill="#FCA5A5" text-anchor="middle" font-weight="600">IAM</text>
-      <text x="563" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">SigV4 · policies</text>
-      <text x="563" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">users · roles · keys</text>
-
-      <!-- Control -->
-      <rect x="630" y="0" width="110" height="58" rx="6" fill="#27272a" fill-opacity="0.8" stroke="#71717a" stroke-opacity="0.6" stroke-width="1"/>
-      <text x="685" y="18" font-size="13" fill="#e4e4e7" text-anchor="middle" font-weight="600">Control</text>
-      <text x="685" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">NATS bus</text>
-      <text x="685" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">spx · daemons</text>
-    </g>
-  </g>
-
-  <!-- Down arrow -->
-  <g transform="translate(450, 360)" opacity="0.45">
-    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
-    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
-  </g>
-
-  <!-- Layer 2: OS -->
-  <g transform="translate(60, 387)">
-    <rect width="780" height="62" rx="10" fill="url(#os-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.5"/>
-    <text x="20" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" letter-spacing="1.5" font-weight="600">LAYER 2 · OPERATING SYSTEM</text>
-    <text x="20" y="44" font-family="'Segoe UI', sans-serif" font-size="14" fill="#f5f5f5">Debian / Ubuntu LTS · systemd · KVM · Linux bridge · headless autoinstall</text>
-  </g>
-
-  <!-- Down arrow -->
-  <g transform="translate(450, 454)" opacity="0.45">
-    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
-    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
-  </g>
-
-  <!-- Layer 1: Hardware -->
-  <g transform="translate(60, 481)">
-    <rect width="780" height="62" rx="10" fill="url(#hw-grad)" stroke="#71717a" stroke-width="1.2" stroke-opacity="0.6"/>
-    <text x="20" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#a1a1aa" letter-spacing="1.5" font-weight="600">LAYER 1 · HARDWARE</text>
-    <text x="20" y="44" font-family="'Segoe UI', sans-serif" font-size="14" fill="#f5f5f5">Single node · multi-node cluster · edge appliance · air-gapped rack · commodity x86 / ARM</text>
-  </g>
-</svg>
+<svg xmlns="http://www.w3.org/2000/svg" width="1000" height="664" viewBox="0 0 1000 664" role="img" aria-label="The Spinifex stack: applications and AWS tooling on top, the Spinifex AWS-compatible service layer, running on standard Linux on CPU/GPU hardware, deployable to Neocloud, on-prem or edge.">
+<rect x="0.0" y="0.0" width="1000.0" height="664.0" rx="20" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<rect x="28.0" y="204.0" width="950.0" height="438.0" rx="16" fill="none" fill-opacity="1.0" stroke="#9AA1A8" stroke-width="1" stroke-opacity="0.12" stroke-dasharray="2 4"/>
+<rect x="25.0" y="201.0" width="950.0" height="438.0" rx="16" fill="none" fill-opacity="1.0" stroke="#9AA1A8" stroke-width="1" stroke-opacity="0.2" stroke-dasharray="2 4"/>
+<rect x="22.0" y="198.0" width="950.0" height="438.0" rx="16" fill="#181D23" fill-opacity="0.25" stroke="#9AA1A8" stroke-width="1.3" stroke-opacity="0.45" stroke-dasharray="2 4"/>
+<text x="36.0" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="500" text-anchor="start">Apps</text>
+<rect x="136.0" y="34.0" width="836.0" height="54.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="156.0" y="66.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="15" fill="#F5F3EE" font-weight="700" text-anchor="start">Your infrastructure &amp; apps</text>
+<rect x="516.7" y="51.0" width="89.3" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.16" stroke="none" stroke-width="1" stroke-opacity="1.0"/>
+<text x="561.4" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#EE4B12" font-weight="500" text-anchor="middle">AI training</text>
+<rect x="613.0" y="51.0" width="76.7" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.16" stroke="none" stroke-width="1" stroke-opacity="1.0"/>
+<text x="651.4" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#EE4B12" font-weight="500" text-anchor="middle">Inference</text>
+<rect x="696.7" y="51.0" width="79.0" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="736.2" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Web &amp; APIs</text>
+<rect x="782.7" y="51.0" width="72.7" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="819.1" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Databases</text>
+<rect x="862.4" y="51.0" width="91.6" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="908.2" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Batch &amp; data</text>
+<text x="36.0" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="500" text-anchor="start">Tooling</text>
+<rect x="136.0" y="102.0" width="836.0" height="54.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="156.0" y="134.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="15" fill="#F5F3EE" font-weight="700" text-anchor="start">AWS tooling</text>
+<rect x="600.9" y="119.0" width="64.1" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.45"/>
+<text x="633.0" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.85">aws-cli</text>
+<rect x="672.0" y="119.0" width="76.7" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.45"/>
+<text x="710.4" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.85">Terraform</text>
+<rect x="755.7" y="119.0" width="64.1" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.45"/>
+<text x="787.8" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.85">kubectl</text>
+<rect x="826.8" y="119.0" width="66.4" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="860.0" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">AWS SDKs</text>
+<rect x="900.2" y="119.0" width="53.8" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="927.1" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Packer</text>
+<text x="484.0" y="184.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="end">↓  AWS API · SigV4</text>
+<text x="624.0" y="184.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">RESPONSE  ↑</text>
+<text x="136.0" y="231.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="15" fill="#F5F3EE" font-weight="700" text-anchor="start">One portable stack.</text>
+<text x="136.0" y="249.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="start">Runs the same on a single node, cluster, edge, or air-gapped rack.</text>
+<rect x="752.3" y="222.0" width="72.4" height="22.0" rx="6" fill="none" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="788.5" y="236.5" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="500" text-anchor="middle">NEOCLOUD</text>
+<rect x="832.7" y="222.0" width="66.1" height="22.0" rx="6" fill="none" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="865.7" y="236.5" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="500" text-anchor="middle">ON-PREM</text>
+<rect x="906.8" y="222.0" width="47.2" height="22.0" rx="6" fill="none" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="930.4" y="236.5" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="500" text-anchor="middle">EDGE</text>
+<text x="152.0" y="295.6" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="16" fill="#F5F3EE" font-weight="800" text-anchor="start">AWS orchestration</text>
+<text x="956.0" y="293.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#EE4B12" font-weight="400" text-anchor="end" fill-opacity="0.9">Gateway → NATS → daemons</text>
+<rect x="150.0" y="312.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="164.0" y="333.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">EC2</text>
+<text x="164.0" y="349.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Compute</text>
+<text x="164.0" y="363.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">QEMU / KVM</text>
+<rect x="422.7" y="312.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="436.7" y="333.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">EBS</text>
+<text x="436.7" y="349.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Block storage</text>
+<text x="436.7" y="363.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">Viperblock</text>
+<rect x="695.3" y="312.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="709.3" y="333.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">S3</text>
+<text x="709.3" y="349.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Object storage</text>
+<text x="709.3" y="363.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">Predastore</text>
+<rect x="150.0" y="382.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="164.0" y="403.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">VPC</text>
+<text x="164.0" y="419.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Networking</text>
+<text x="164.0" y="433.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">OVN / OVS</text>
+<rect x="422.7" y="382.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="436.7" y="403.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">IAM</text>
+<text x="436.7" y="419.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Identity</text>
+<text x="436.7" y="433.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">SigV4 · policies</text>
+<rect x="695.3" y="382.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="709.3" y="403.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">EKS</text>
+<text x="709.3" y="419.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Kubernetes</text>
+<text x="709.3" y="433.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">K8s-conformant</text>
+<rect x="150.0" y="452.0" width="808.0" height="32.0" rx="8" fill="#14181D" fill-opacity="0.6" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0" stroke-dasharray="3 3"/>
+<text x="164.0" y="472.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#9AA1A8" font-weight="400" text-anchor="start">+ ALB · ECS · ECR · RDS and more of the core AWS services</text>
+<text x="944.0" y="471.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#EE4B12" font-weight="400" text-anchor="end">See all →</text>
+<text x="36.0" y="386.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#EE4B12" font-weight="500" text-anchor="start">Spinifex</text>
+<rect x="136.0" y="268.0" width="836.0" height="230.0" rx="12" fill="#EE4B12" fill-opacity="0.07" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
+<text x="36.0" y="536.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="500" text-anchor="start">OS</text>
+<rect x="136.0" y="510.0" width="836.0" height="46.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="156.0" y="537.9" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="700" text-anchor="start">Operating system</text>
+<text x="952.0" y="536.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="end">Debian / Ubuntu LTS · systemd · KVM · Linux bridge · Open vSwitch</text>
+<text x="36.0" y="596.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="500" text-anchor="start">Hardware</text>
+<rect x="136.0" y="568.0" width="413.0" height="50.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="156.0" y="597.9" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="700" text-anchor="start">CPU</text>
+<text x="533.0" y="596.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="end">Intel · AMD · ARM</text>
+<rect x="559.0" y="568.0" width="413.0" height="50.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="579.0" y="597.9" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="700" text-anchor="start">GPU</text>
+<text x="956.0" y="596.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#EE4B12" font-weight="400" text-anchor="end">NVIDIA · AMD</text>
+</svg>

.github/resolve-sibling-refs.sh

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+# Resolve which branch of each sibling repo (viperblock, predastore) the CI
+# workspace should build against. A cross-repo change is developed on a branch
+# of the SAME name in each repo; build against that branch when it exists so the
+# set can be tested together, and fall back to dev otherwise. This mirrors the
+# resolution the e2e path already does in
+# mulga/scripts/tofu-cluster/build-install-artifacts.sh, so unit CI and e2e CI
+# agree on how sibling branches are selected.
+#
+# Reads $BRANCH (the triggering head/ref name) and writes viperblock=<ref> and
+# predastore=<ref> to $GITHUB_OUTPUT for the checkout steps to consume.
+set -euo pipefail
+
+: "${BRANCH:?BRANCH must be set to the triggering branch name}"
+: "${GITHUB_OUTPUT:?GITHUB_OUTPUT must be set}"
+
+for repo in viperblock predastore; do
+  if git ls-remote --exit-code --heads \
+      "https://github.com/mulgadc/${repo}.git" "$BRANCH" >/dev/null 2>&1; then
+    ref="$BRANCH"
+  else
+    ref="dev"
+  fi
+  echo "${repo}: building against ${ref}"
+  echo "${repo}=${ref}" >>"$GITHUB_OUTPUT"
+done

.github/scripts/suite-sets.sh

@@ -0,0 +1,58 @@
+# shellcheck shell=bash
+# shellcheck disable=SC2034  # every variable here is consumed by the sourcing workflow
+# Source from a workflow `run:` block:
+#   . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+#
+# Which e2e suites each topology can run. Sourced by every workflow that drives
+# suites (e2e.yml, e2e-suites.yml, e2e-nightly.yml) so the lists cannot drift
+# apart — they previously disagreed three ways.
+#
+# Eligibility is a property of the suite, not of the workflow: a suite belongs
+# to a topology when all of its assertions are reachable there. lb is
+# multi-only for exactly that reason — on a single node every internet-facing
+# subtest skips for want of a peer, so a single-node lb run reports green
+# having exercised a fraction of the suite.
+
+# Suites runnable against a one-node environment.
+#
+# partialblock gates merges because it earned it, not because it is new: on one
+# node, same binary and workload, it fails against the pre-fix engine (1758 of
+# 8192 halves reverted a generation) and passes against the fixed one (0 of
+# 8192). A suite that has never been shown to go red for the reason it claims
+# does not belong here.
+E2E_SUITES_SINGLE="single iam cert eks ecs storagegrowth partialblock"
+
+# Suites runnable against a multi-node environment.
+E2E_SUITES_MULTI="multinode lb cert"
+
+# grep -xE alternation form, for narrowing a requested set to the eligible one.
+E2E_SUITES_SINGLE_RE="$(printf '%s' "${E2E_SUITES_SINGLE}" | tr ' ' '|')"
+E2E_SUITES_MULTI_RE="$(printf '%s' "${E2E_SUITES_MULTI}" | tr ' ' '|')"
+
+# e2e-nightly runs a narrower set than the two sets above, deliberately: its
+# ~19 cells each have a 35-minute budget and exist to prove that every
+# install / network / host-OS permutation boots and serves. Suite breadth is
+# the PR gate's job, and it already runs on every push. These are subsets, and
+# e2e_suites_assert_subset below enforces that.
+E2E_SUITES_NIGHTLY_SINGLE="single cert"
+E2E_SUITES_NIGHTLY_MULTI="multinode cert lb"
+
+# Fail loudly when a narrowed list names a suite its topology cannot run. This
+# is the guard that stops the lists drifting again: a suite added to a nightly
+# subset without also being declared eligible above stops the job here rather
+# than silently skipping half its assertions in CI.
+#   e2e_suites_assert_subset <label> <subset> <eligible-set>
+e2e_suites_assert_subset() {
+  local label="$1" subset="$2" eligible="$3" suite found eligible_suite
+  for suite in ${subset}; do
+    found=0
+    for eligible_suite in ${eligible}; do
+      if [ "${suite}" = "${eligible_suite}" ]; then found=1; break; fi
+    done
+    if [ "${found}" -ne 1 ]; then
+      echo "::error::${label} names suite '${suite}', which is not eligible for that topology (eligible: ${eligible}). Fix the list in .github/scripts/suite-sets.sh." >&2
+      return 1
+    fi
+  done
+  return 0
+}

.github/workflows/e2e-baremetal.yml

@@ -9,6 +9,11 @@ on:
         required: false
         type: string
         default: ""
+      otel-attrs:
+        description: "OTEL_RESOURCE_ATTRIBUTES value for this run, e.g. ci.run_id=123,ci.cell=20 (caller-specific, so it's an input rather than a workflow-level default)"
+        required: false
+        type: string
+        default: ""
 
 permissions:
   contents: read
@@ -16,6 +21,10 @@ permissions:
 
 env:
   GIT_BRANCH: ${{ github.ref_name }}
+  # Same central ES/APM sink as the other four e2e workflows. Bare-metal had
+  # no default here before — this cell's telemetry never reached the sink at
+  # all (not just missing ci.cell), see bm-bootstrap.sh's telemetry.env step.
+  SPINIFEX_OTLP_ENDPOINT: ${{ vars.SPINIFEX_OTLP_ENDPOINT || 'http://192.168.1.13:8200' }}
 
 jobs:
   e2e_baremetal:
@@ -28,6 +37,9 @@ jobs:
     env:
       BM_ISO_CACHE: /var/cache/spinifex-iso-builder
       ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-20-${{ github.run_id }}
+      # Caller-supplied (workflow_call input); empty for a manual
+      # workflow_dispatch run, which bm-bootstrap.sh treats as "no attrs".
+      SPINIFEX_OTEL_ATTRS: ${{ inputs.otel-attrs }}
     steps:
       - name: Clean Workspace
         run: |

.github/workflows/e2e-ddil.yml

@@ -13,6 +13,9 @@ on:
 
 permissions:
   contents: read
+  # actions: read lets the otel-job-id action look up this job's numeric id via
+  # the Actions API — github.job is only the YAML key, no context exposes it.
+  actions: read
 
 env:
   GIT_BRANCH: ${{ github.ref_name }}
@@ -28,6 +31,10 @@ jobs:
     timeout-minutes: 60
     env:
       TOFU_LOG: /tmp/e2e-tofu-ddil-${{ github.run_id }}.log
+      # Job-level env wins over the workflow-level default. ci.cell=ddil marks
+      # this workflow's telemetry so a DDIL run's partition-induced errors are
+      # never mistaken for a healthy multi-node run's in the sink.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=ddil
     steps:
       - name: Clean Workspace
         run: sudo rm -rf "$GITHUB_WORKSPACE"/*
@@ -72,6 +79,13 @@ jobs:
           echo "${BIN_DIR}" >> "${GITHUB_PATH}"
           "${BIN_DIR}/gotestsum" --version
 
+      # Adds ci.job_id alongside ci.cell, before Bootstrap bakes the attrs into
+      # the VMs, so a telemetry doc points back at this job's log directly.
+      - name: Resolve CI job id
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: DDIL Real Multi-Node
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -208,3 +222,12 @@ jobs:
           quiet_run "tofu destroy"          "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
           quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
           ok "teardown complete"
+
+      - name: Upload tofu log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: tofu-log-ddil-${{ github.run_id }}
+          path: ${{ env.TOFU_LOG }}
+          if-no-files-found: ignore
+          retention-days: 30

.github/workflows/e2e-nightly.yml

@@ -11,6 +11,10 @@ on:
 
 permissions:
   contents: read
+  # actions: read lets the "Resolve CI job id" step below look up this job's
+  # numeric id via the Actions API — github.job is only the YAML key, no
+  # context exposes the id itself.
+  actions: read
 
 # Scheduled runs are loaded from the default branch (main), so the schedule
 # trigger must stay in sync on main for cron to fire.
@@ -46,6 +50,11 @@ jobs:
     env:
       TOFU_LOG: /tmp/e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}.log
       ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}
+      # Job-level env wins over the workflow-level default above, so telemetry
+      # from this cell carries ci.cell/ci.matrix. Nine cells run under one
+      # run id and every cell names its hosts node1/node2/node3, so without
+      # this every cell's telemetry collapses into one unattributable bucket.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=${{ matrix.cell }},ci.matrix=${{ matrix.topo }}/${{ matrix.net }}/${{ matrix.host }}
 
     steps:
       # blank or scheduled run = full matrix.
@@ -92,6 +101,16 @@ jobs:
         with:
           path: spinifex
 
+      # Nine cells share one run id and every cell names its hosts
+      # node1/node2/node3, so ci.cell above separates them and ci.job_id here
+      # links each cell's telemetry to the job log that produced it. Runs
+      # before Bootstrap bakes SPINIFEX_OTEL_ATTRS into the VM's telemetry.env.
+      - name: Resolve CI job id
+        if: steps.gate.outputs.selected == 'true'
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: "cell-${{ matrix.cell }} (${{ matrix.topo }}/${{ matrix.net }}/${{ matrix.host }}${{ matrix.extra && format('/{0}', matrix.extra) || '' }})"
+
       - name: Build Install Artifacts
         if: steps.gate.outputs.selected == 'true'
         working-directory: mulga/scripts/tofu-cluster
@@ -147,13 +166,20 @@ jobs:
         if: steps.gate.outputs.selected == 'true' && matrix.topo == 'single' && matrix.extra == ''
         working-directory: mulga/scripts/tofu-cluster
         run: |
+          set -euo pipefail
+          # Nightly runs a documented subset of the single-node set; the assert
+          # fails the cell if that subset ever names a suite this topology
+          # cannot fully exercise.
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          e2e_suites_assert_subset "e2e-nightly single" "${E2E_SUITES_NIGHTLY_SINGLE}" "${E2E_SUITES_SINGLE}"
+
           INVENTORY=$(tofu output -json inventory)
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
           ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
             "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
-             for SUITE in single cert lb; do \
+             for SUITE in ${E2E_SUITES_NIGHTLY_SINGLE}; do \
                echo \"::group::suite \$SUITE\"; \
                date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
                GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
@@ -190,6 +216,11 @@ jobs:
         if: steps.gate.outputs.selected == 'true' && matrix.topo == 'real-multi'
         working-directory: mulga/scripts/tofu-cluster
         run: |
+          set -euo pipefail
+          # See the single-node step: same subset guard, multi-node set.
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          e2e_suites_assert_subset "e2e-nightly multi" "${E2E_SUITES_NIGHTLY_MULTI}" "${E2E_SUITES_MULTI}"
+
           INVENTORY=$(tofu output -json inventory)
           NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
@@ -204,7 +235,7 @@ jobs:
 
           ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
             "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
-             for SUITE in multinode cert lb; do \
+             for SUITE in ${E2E_SUITES_NIGHTLY_MULTI}; do \
                echo \"::group::suite \$SUITE\"; \
                date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
                GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
@@ -348,6 +379,15 @@ jobs:
           tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1
           ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
 
+      - name: Upload tofu log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: tofu-log-cell-${{ matrix.cell }}-${{ github.run_id }}
+          path: ${{ env.TOFU_LOG }}
+          if-no-files-found: ignore
+          retention-days: 14
+
   # Bare-metal single-node suite. Doesn't fit the tofu matrix shape, so it
   # runs as a sibling job via the reusable e2e-baremetal workflow. Gated like a
   # matrix cell ("cell 20"): every weeknight cron, or dispatch with 20 selected
@@ -364,4 +404,8 @@ jobs:
     uses: ./.github/workflows/e2e-baremetal.yml
     with:
       ref: ${{ github.event_name == 'schedule' && 'dev' || github.ref_name }}
+      # A reusable-workflow call gets none of this workflow's context (env:
+      # above, matrix.cell) for free — cell-20 needs its own attrs string.
+      # There's no matrix leg here, just the one fixed baremetal cell.
+      otel-attrs: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=20,ci.matrix=baremetal/pxe/real-hw
     secrets: inherit

.github/workflows/e2e-suites.yml

@@ -3,11 +3,14 @@
 # Restructures the looped e2e.yml into env-grouped job blocks so each suite
 # surfaces as its own node in the Actions graph (mulga-siv-283):
 #
-#   single_provision ─┬─ single_suite (matrix: single, cert, lb, eks, iam)
+#   single_provision ─┬─ single_suite (matrix: the single-node-eligible suites)
 #                     └─ single_teardown (if: always)
-#   multi_provision  ─┬─ multi_suite  (matrix: multinode)
+#   multi_provision  ─┬─ multi_suite  (matrix: the multi-node-eligible suites)
 #                     └─ multi_teardown (if: always)
 #
+# Both matrices come from .github/scripts/suite-sets.sh, the one definition of
+# which suites a topology can run, shared with e2e.yml and e2e-nightly.yml.
+#
 # Constraint: tofu state is runner-filesystem-local, so *_provision and
 # *_teardown must execute on the same self-hosted runner (one runner per label
 # today). Teardown reuses the persisted mulga/ tofu-cluster workspace and does
@@ -25,9 +28,13 @@ on:
   workflow_dispatch:
     inputs:
       single_suites:
-        description: "Single-node suites to run as matrix legs"
+        description: "Single-node suites to run as matrix legs (empty = every single-node-eligible suite, per .github/scripts/suite-sets.sh)"
         required: false
-        default: "single cert lb eks iam"
+        default: ""
+      multi_suites:
+        description: "Multi-node suites to run as matrix legs (empty = every multi-node-eligible suite, per .github/scripts/suite-sets.sh)"
+        required: false
+        default: ""
       run_multinode:
         description: "Also run the multinode env block"
         type: boolean
@@ -40,6 +47,10 @@ on:
 
 permissions:
   contents: read
+  # actions: read lets the otel-job-id action look up a provision job's numeric
+  # id via the Actions API — github.job is only the YAML key, no context
+  # exposes it.
+  actions: read
 
 env:
   GIT_BRANCH: ${{ github.ref_name }}
@@ -64,6 +75,13 @@ jobs:
       matrix: ${{ steps.suites.outputs.matrix }}
     env:
       TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
+      # This job (not single_suite) is where "Bootstrap via Binary Install"
+      # bakes SPINIFEX_OTEL_ATTRS into the VM's telemetry.env, before the
+      # suite matrix even exists — single_suite's legs share this one
+      # already-booted VM, so daemon telemetry can't be scoped any finer
+      # than "single env" per suite leg. ci.cell=single distinguishes it
+      # from multi_provision's VMs within the same run id.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=single
     steps:
       - name: Reap Orphan Procs
         run: |
@@ -107,7 +125,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -115,12 +133,22 @@ jobs:
       - name: Re-enable Go toolchain auto-download
         run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
 
+      # Default and allow-list both come from the shared definition, so this
+      # workflow cannot drift from e2e.yml about what a single node can run.
       - name: Compute suite matrix
         id: suites
+        env:
+          REQUESTED: ${{ inputs.single_suites }}
         run: |
-          LIST="${{ inputs.single_suites || 'single cert lb eks iam' }}"
+          set -euo pipefail
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          LIST="${REQUESTED:-${E2E_SUITES_SINGLE}}"
+          e2e_suites_assert_subset "single_suites" "${LIST}" "${E2E_SUITES_SINGLE}"
           echo "list=${LIST}" >> "$GITHUB_OUTPUT"
-          JSON=$(printf '%s\n' $LIST | jq -R . | jq -cs .)
+          # Unquoted on purpose: the space-separated list must word-split into
+          # one JSON element per suite.
+          # shellcheck disable=SC2086
+          JSON=$(printf '%s\n' ${LIST} | jq -R . | jq -cs .)
           echo "matrix=${JSON}" >> "$GITHUB_OUTPUT"
 
       - name: Reclaim docker build cache
@@ -130,6 +158,14 @@ jobs:
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
+      # ci.job_id names the provision job, not the suite job that later drives
+      # the env: the suite legs share this env and its baked telemetry.env, so
+      # the provisioner is the only job the daemon telemetry can belong to.
+      - name: Resolve CI job id
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: Single env · provision
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -267,6 +303,15 @@ jobs:
             "$VM":/tmp/start-net-capture.sh
           ssh "${SSH_OPTS[@]}" "$VM" "bash /tmp/start-net-capture.sh" || true
 
+      - name: Upload tofu log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: tofu-log-single-${{ github.run_id }}
+          path: ${{ env.TOFU_LOG }}
+          if-no-files-found: ignore
+          retention-days: 14
+
   single_suite:
     name: Single env · suite
     needs: single_provision
@@ -285,7 +330,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -467,6 +512,15 @@ jobs:
           quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
           ok "teardown complete"
 
+      - name: Upload tofu log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: tofu-log-single-td-${{ github.run_id }}
+          path: ${{ env.TOFU_LOG }}
+          if-no-files-found: ignore
+          retention-days: 14
+
   # ---------------------------------------------------------------------------
   # Multinode env
   # ---------------------------------------------------------------------------
@@ -479,8 +533,13 @@ jobs:
       node1_ip: ${{ steps.cluster.outputs.node1_ip }}
       ssh_key: ${{ steps.cluster.outputs.ssh_key }}
       nodes_csv: ${{ steps.cluster.outputs.nodes_csv }}
+      suites: ${{ steps.suites.outputs.list }}
+      matrix: ${{ steps.suites.outputs.matrix }}
     env:
       TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
+      # See single_provision above — daemon telemetry is baked in here, at
+      # provision time, before the multi_suite matrix exists.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=multi
     steps:
       - name: Reap Orphan Procs
         run: |
@@ -521,7 +580,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -536,6 +595,12 @@ jobs:
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
+      # See the matching note on single_provision.
+      - name: Resolve CI job id
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: Multi env · provision
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -583,14 +648,36 @@ jobs:
           echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
           echo "nodes_csv=${NODES_CSV}" >> "$GITHUB_OUTPUT"
 
-      - name: Build multinode binary on runner
+      # Mirrors single_provision's step: the multi leg previously ran multinode
+      # alone, silently dropping the lb and cert coverage that only a
+      # multi-node env can complete.
+      - name: Compute suite matrix
+        id: suites
+        env:
+          REQUESTED: ${{ inputs.multi_suites }}
+        run: |
+          set -euo pipefail
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          LIST="${REQUESTED:-${E2E_SUITES_MULTI}}"
+          e2e_suites_assert_subset "multi_suites" "${LIST}" "${E2E_SUITES_MULTI}"
+          echo "list=${LIST}" >> "$GITHUB_OUTPUT"
+          # Unquoted on purpose: the space-separated list must word-split into
+          # one JSON element per suite.
+          # shellcheck disable=SC2086
+          JSON=$(printf '%s\n' ${LIST} | jq -R . | jq -cs .)
+          echo "matrix=${JSON}" >> "$GITHUB_OUTPUT"
+
+      - name: Build multinode binaries on runner
         working-directory: spinifex/tests/e2e
         run: |
           set -euo pipefail
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          banner "build multinode, teardown"
+          # teardown always builds: the run-scoped sweep binary runs in the
+          # teardown job, it is never a selectable suite leg.
+          SUITES_BUILD="${{ steps.suites.outputs.list }} teardown"
+          banner "build ${SUITES_BUILD// /, }"
           group_open "go test -c"
-          make e2e-build SUITES_BUILD="multinode teardown"
+          make e2e-build SUITES_BUILD="${SUITES_BUILD}"
           group_close
           ok "$(basename _bin/multinode.test) $(stat -c '%s' _bin/multinode.test | numfmt --to=iec)"
 
@@ -617,6 +704,15 @@ jobs:
             ssh "${SSH_OPTS[@]}" tf-user@"$ip" "bash /tmp/start-net-capture.sh" || true
           done
 
+      - name: Upload tofu log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: tofu-log-multi-${{ github.run_id }}
+          path: ${{ env.TOFU_LOG }}
+          if-no-files-found: ignore
+          retention-days: 14
+
   multi_suite:
     name: Multi env · suite
     needs: multi_provision
@@ -625,7 +721,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        suite: [multinode]
+        suite: ${{ fromJSON(needs.multi_provision.outputs.matrix) }}
     env:
       NODE1_IP: ${{ needs.multi_provision.outputs.node1_ip }}
       SSH_KEY: ${{ needs.multi_provision.outputs.ssh_key }}
@@ -636,7 +732,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -789,6 +885,15 @@ jobs:
           quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
           ok "teardown complete"
 
+      - name: Upload tofu log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: tofu-log-multi-td-${{ github.run_id }}
+          path: ${{ env.TOFU_LOG }}
+          if-no-files-found: ignore
+          retention-days: 14
+
   e2e_suites_status:
     name: E2E suite blocks · summary
     if: always()

.github/workflows/e2e.yml

@@ -26,10 +26,13 @@ on:
 permissions:
   contents: read
   statuses: write
+  # actions: read lets the otel-job-id action look up a job's numeric id via
+  # the Actions API — github.job is only the YAML key, no context exposes it.
+  actions: read
 
 env:
   GIT_BRANCH: ${{ github.ref_name }}
-  SUITES: ${{ inputs.suites || 'e2e-single e2e-multinode e2e-cert e2e-lb e2e-iam' }}
+  SUITES: ${{ inputs.suites || 'e2e-single e2e-multinode e2e-cert e2e-lb e2e-iam e2e-partialblock' }}
   TEST_RUN: ${{ inputs.test_run || '' }}
   # OTLP export to the central Elastic sink (wattle). Repo variable overrides;
   # empty string disables telemetry entirely.
@@ -37,54 +40,28 @@ env:
   SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }}
 
 jobs:
-  # Shadow-mode suite selector. Computes which suites the diff *would* run and logs it to the job summary.
-  # Does NOT gate or override the real SUITES env yet — it runs alongside the full matrix so we
-  # can compare predicted vs actual before flipping it on.
-  select_suites:
-    name: Select Suites (shadow)
-    runs-on: ubuntu-26.04
-    timeout-minutes: 5
-    steps:
-      - uses: actions/checkout@v7
-        with:
-          fetch-depth: 0
-      - uses: actions/setup-go@v6
-        with:
-          go-version-file: go.mod
-          cache: false
-      - name: Compute targeted suite selection
-        run: |
-          set -euo pipefail
-          BASE="${{ github.event.before }}"
-          HEAD="${{ github.sha }}"
-          # First push to a branch reports an all-zero before-SHA; fall back to
-          # the first parent so the diff is still meaningful.
-          if [ -z "$BASE" ] || [ "$BASE" = "0000000000000000000000000000000000000000" ]; then
-            BASE="${HEAD}~1"
-          fi
-          echo "## Targeted suite selection (shadow)" >> "$GITHUB_STEP_SUMMARY"
-          echo '```' >> "$GITHUB_STEP_SUMMARY"
-          echo "base=$BASE head=$HEAD" >> "$GITHUB_STEP_SUMMARY"
-          go run ./tests/e2e/select-suites/cmd/select-suites \
-            --manifest docs/service-interfaces.yaml \
-            --base-sha "$BASE" --head-sha "$HEAD" --repo-root . \
-            2>>"$GITHUB_STEP_SUMMARY" \
-            | tee -a "$GITHUB_STEP_SUMMARY"
-          echo '```' >> "$GITHUB_STEP_SUMMARY"
-          echo "Shadow only — actual run uses SUITES='${SUITES}'." >> "$GITHUB_STEP_SUMMARY"
-
   e2e_single:
     name: Single-Node E2E
-    # Skip when the dispatch input requests no in-VM suites (e.g.
-    # suites='e2e-multinode' only). Push triggers and the default SUITES list
-    # both include single/cert/lb so the inputs.suites null-check keeps the
-    # job running on plain PR pushes.
-    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-lb') || contains(github.event.inputs.suites, 'e2e-eks') || contains(github.event.inputs.suites, 'e2e-iam')
+    # The dispatch check gates on exactly the suites this job's Run step
+    # allow-lists, i.e. E2E_SUITES_SINGLE in .github/scripts/suite-sets.sh —
+    # an `if:` expression cannot source that file, so this one list is
+    # hand-mirrored and must be updated with it. Gating on a suite the job
+    # won't run — lb, which only executes in the multinode job — would
+    # provision a tofu VM and pay the Spinifex install only to run nothing and
+    # tear down. Push triggers keep running via the null-check on
+    # inputs.suites.
+    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-iam') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-eks') || contains(github.event.inputs.suites, 'e2e-ecs') || contains(github.event.inputs.suites, 'e2e-storagegrowth') || contains(github.event.inputs.suites, 'e2e-partialblock')
     runs-on: [self-hosted, ci-single]
     timeout-minutes: 50
     env:
       TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
       ARTIFACT_DIR: /tmp/spinifex-e2e-artifacts-${{ github.run_id }}
+      # Job-level env wins over the workflow-level default. This job and
+      # e2e_multi share one run id and both name their hosts node1/node2/node3,
+      # so without ci.cell their telemetry collapses into one bucket. This job
+      # is where "Bootstrap via Binary Install" bakes SPINIFEX_OTEL_ATTRS into
+      # the VM's telemetry.env, so the label has to be set here.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=single
     steps:
       - name: Reap Orphan Procs
         run: |
@@ -134,7 +111,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -149,6 +126,13 @@ jobs:
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
+      # Adds ci.job_id alongside ci.cell, before Bootstrap bakes the attrs into
+      # the VM, so a telemetry doc points back at this job's log directly.
+      - name: Resolve CI job id
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: Single-Node E2E
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -312,14 +296,23 @@ jobs:
         id: e2e
         run: |
           set +e
-          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(multinode)$' | tr '\n' ' ')
+          # Allow-list, not a deny-list: a suite runs here only if named. Deny-lists
+          # silently fan a newly added suite out to every job, which is how lb ended up
+          # running twice per e2e run. The allow-list itself is shared with the other
+          # e2e workflows so the three cannot disagree about what a topology can run;
+          # suite-sets.sh also carries the per-suite rationale.
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -xE "${E2E_SUITES_SINGLE_RE}" | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
             echo "no in-VM suites requested (SUITES=${SUITES}); skipping"
             echo "rc=0" >> "$GITHUB_OUTPUT"
             exit 0
           fi
           if [ -n "${TEST_RUN}" ]; then
-            TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+            # The value is expanded into a command that a remote shell re-parses over
+            # ssh, so single-quote it — otherwise regex metacharacters like ( | ) $ are
+            # taken as shell syntax on the VM and bash fails before the test binary runs.
+            TEST_RUN_FLAG="-test.run='${TEST_RUN}'"
           else
             TEST_RUN_FLAG=""
           fi
@@ -466,6 +459,15 @@ jobs:
           quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
           ok "teardown complete"
 
+      - name: Upload tofu log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: tofu-log-single-${{ github.run_id }}
+          path: ${{ env.TOFU_LOG }}
+          if-no-files-found: ignore
+          retention-days: 14
+
       - name: Fail job if any suite failed
         if: always()
         run: |
@@ -477,12 +479,19 @@ jobs:
 
   e2e_multi:
     name: Multi-Node E2E
-    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-multinode')
+    # The dispatch check gates on exactly the suites this job's Run step
+    # allow-lists (multinode|lb|cert) so lb — which only runs here — still
+    # provisions the 3-node env when requested on its own, and the env stays
+    # unprovisioned when none of them are asked for.
+    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-multinode') || contains(github.event.inputs.suites, 'e2e-lb') || contains(github.event.inputs.suites, 'e2e-cert')
     runs-on: [self-hosted, ci-multi]
     timeout-minutes: 60
     env:
       TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
       ARTIFACT_DIR: /tmp/spinifex-e2e-multi-artifacts-${{ github.run_id }}
+      # See the matching note on e2e_single: ci.cell is what separates this
+      # job's 3-node telemetry from the single-node job's under one run id.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=multi
     steps:
       - name: Reap Orphan Procs
         run: |
@@ -532,7 +541,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -547,6 +556,12 @@ jobs:
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
+      # See the matching note on e2e_single.
+      - name: Resolve CI job id
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: Multi-Node E2E
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -688,14 +703,18 @@ jobs:
         id: e2e
         run: |
           set +e
-          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(single|eks|iam)$' | tr '\n' ' ')
+          # Allow-list, not a deny-list — see the single-node job for why.
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -xE "${E2E_SUITES_MULTI_RE}" | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
             echo "no multinode-eligible suites requested (SUITES=${SUITES}); skipping"
             echo "rc=0" >> "$GITHUB_OUTPUT"
             exit 0
           fi
           if [ -n "${TEST_RUN}" ]; then
-            TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+            # Single-quote for the remote shell — see the single-node job for the
+            # metacharacter-injection reason.
+            TEST_RUN_FLAG="-test.run='${TEST_RUN}'"
           else
             TEST_RUN_FLAG=""
           fi
@@ -881,6 +900,15 @@ jobs:
           quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
           ok "teardown complete"
 
+      - name: Upload tofu log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: tofu-log-multi-${{ github.run_id }}
+          path: ${{ env.TOFU_LOG }}
+          if-no-files-found: ignore
+          retention-days: 14
+
       - name: Fail job if any suite failed
         if: always()
         run: |
@@ -917,3 +945,136 @@ jobs:
             -f context=e2e \
             -f description="$DESC" \
             -f target_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
+
+  # Emits one summary doc per run to the central Elastic sink, keyed by
+  # labels.ci_run_id, so Kibana can list "did run X pass, and where are its
+  # artifacts" without hand-aggregating traces-apm*/logs-apm.error*. This is
+  # a separate self-hosted job (not a step on e2e_status) because e2e_status
+  # runs on GitHub-hosted ubuntu-26.04, which has no route to the private
+  # 192.168.1.13 sink, and moving e2e_status off GH-hosted would risk the PR
+  # status-report check.
+  e2e_es_summary:
+    name: ES CI Run Summary
+    needs: [e2e_single, e2e_multi]
+    if: always()
+    runs-on: [self-hosted, ci-single]
+    timeout-minutes: 5
+    permissions:
+      actions: read
+    env:
+      # Repo var overrides; defaults to the LAN Elastic sink reachable from
+      # ci-single/ci-multi runners. Anonymous ES requests are superuser on
+      # this host, so no credentials are needed for the POST.
+      MULGA_ES_SINK: ${{ vars.MULGA_ES_SINK || '192.168.1.13:9200' }}
+    steps:
+      # Either leg may not have run (e.g. a workflow_dispatch that only
+      # requested single-node suites), so a missing artifact is tolerated —
+      # the aggregation script below treats an absent suite-status.txt as
+      # "no failures from that leg".
+      - name: Download single-node artifacts
+        uses: actions/download-artifact@v8
+        continue-on-error: true
+        with:
+          name: e2e-artifacts-${{ github.run_id }}
+          path: ${{ github.workspace }}/es-summary-single
+
+      - name: Download multi-node artifacts
+        uses: actions/download-artifact@v8
+        continue-on-error: true
+        with:
+          name: e2e-multi-artifacts-${{ github.run_id }}
+          path: ${{ github.workspace }}/es-summary-multi
+
+      - name: Build and post CI run summary
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+          RUN_ID: ${{ github.run_id }}
+          RUN_ATTEMPT: ${{ github.run_attempt }}
+          WORKFLOW: ${{ github.workflow }}
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+          SHA: ${{ github.sha }}
+          SERVER_URL: ${{ github.server_url }}
+          SINGLE_RESULT: ${{ needs.e2e_single.result }}
+          MULTI_RESULT: ${{ needs.e2e_multi.result }}
+          SINGLE_DIR: ${{ github.workspace }}/es-summary-single
+          MULTI_DIR: ${{ github.workspace }}/es-summary-multi
+        run: |
+          set -euo pipefail
+
+          SINGLE_STATUS="${SINGLE_DIR}/suite-status.txt"
+          MULTI_STATUS="${MULTI_DIR}/suite-status.txt"
+
+          # Collect "<suite>: FAILED" lines from both legs' suite-status.txt
+          # (either may be absent — a leg that didn't run, or a passing leg
+          # that never wrote the file). Accumulate as newline-separated text
+          # rather than a bash array so this stays correct under `set -u`
+          # even when nothing failed.
+          FAILED_SUITES=""
+          for f in "$SINGLE_STATUS" "$MULTI_STATUS"; do
+            [ -f "$f" ] || continue
+            while IFS= read -r line; do
+              case "$line" in
+                *": FAILED") FAILED_SUITES="${FAILED_SUITES}${line%: FAILED}"$'\n' ;;
+              esac
+            done < "$f"
+          done
+
+          FAILED_JSON=$(printf '%s' "$FAILED_SUITES" | jq -R -s 'split("\n") | map(select(length > 0))')
+          FAILED_COUNT=$(printf '%s' "$FAILED_JSON" | jq 'length')
+
+          # Mirror e2e_status's single_ok/multi_ok logic exactly so the PR
+          # status and this ES doc can never disagree on overall pass/fail.
+          single_ok=true
+          case "$SINGLE_RESULT" in success | skipped) ;; *) single_ok=false ;; esac
+          multi_ok=true
+          case "$MULTI_RESULT" in success | skipped) ;; *) multi_ok=false ;; esac
+
+          STATE=success
+          if [ "$single_ok" != true ] || [ "$multi_ok" != true ] || [ "$FAILED_COUNT" -gt 0 ]; then
+            STATE=failure
+          fi
+
+          RUN_STARTED_AT=$(gh api "repos/${REPO}/actions/runs/${RUN_ID}" --jq .run_started_at)
+          START_EPOCH=$(date -u -d "${RUN_STARTED_AT}" +%s)
+          NOW_EPOCH=$(date -u +%s)
+          DURATION=$((NOW_EPOCH - START_EPOCH))
+
+          DOC=$(jq -n \
+            --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
+            --arg run_id "${RUN_ID}" \
+            --arg workflow "${WORKFLOW}" \
+            --arg branch "${BRANCH}" \
+            --arg sha "${SHA}" \
+            --arg run_attempt "${RUN_ATTEMPT}" \
+            --arg state "${STATE}" \
+            --arg single_result "${SINGLE_RESULT}" \
+            --arg multi_result "${MULTI_RESULT}" \
+            --argjson duration "${DURATION}" \
+            --arg artifact_url "${SERVER_URL}/${REPO}/actions/runs/${RUN_ID}" \
+            --argjson failed_suites "${FAILED_JSON}" \
+            '{
+              "@timestamp": $ts,
+              labels: { ci_run_id: $run_id },
+              ci: { workflow: $workflow, branch: $branch, sha: $sha, run_attempt: $run_attempt },
+              state: $state,
+              single_result: $single_result,
+              multi_result: $multi_result,
+              failed_suites: $failed_suites,
+              duration_seconds: $duration,
+              artifact_url: $artifact_url
+            }')
+
+          echo "$DOC" | jq .
+
+          HTTP_CODE=$(curl -sS -o /tmp/es-summary-resp.json -w '%{http_code}' \
+            -X POST "http://${MULGA_ES_SINK}/mulga-ci-runs/_doc" \
+            -H 'Content-Type: application/json' \
+            -d "${DOC}")
+
+          if [ "$HTTP_CODE" != "200" ] && [ "$HTTP_CODE" != "201" ]; then
+            echo "::error::ES POST to ${MULGA_ES_SINK}/mulga-ci-runs failed (HTTP ${HTTP_CODE})"
+            cat /tmp/es-summary-resp.json
+            exit 1
+          fi
+          echo "CI run summary doc posted to ${MULGA_ES_SINK}/mulga-ci-runs (HTTP ${HTTP_CODE})"

.github/workflows/gpu-e2e.yml

@@ -0,0 +1,347 @@
+# GPU E2E — drives wattle's persistent, shared-host Spinifex stack over SSH to
+# exercise GPU passthrough (raw EC2, ECS task GPU exposure, EKS pod GPU
+# exposure). See docs/development/feature/wattle-deploy-and-ci.md (Part B).
+#
+# ---------------------------------------------------------------------------
+# ONE-TIME HOST-SIDE SETUP — NOT part of this repo, a human must do this:
+#
+#   1. Register ONE self-hosted GitHub Actions runner on a banksia host with
+#      the dedicated label `ci-gpu` (in addition to `self-hosted`). Do
+#      NOT add it to the `ci-single` / `ci-multi` pool and do NOT give any
+#      other runner the `ci-gpu` label — a second labelled runner would
+#      let two GPU jobs race the single RTX A1000. With exactly one runner
+#      carrying the label, `runs-on: [self-hosted, ci-gpu]` alone forces
+#      serialization; the `concurrency:` block below queues same-group runs
+#      when that runner is already busy with a live job.
+#   2. Drop the wattle SSH private key on that runner's filesystem and wire
+#      it into this repo as `secrets.WATTLE_SSH_KEY` (raw private key
+#      contents). Optionally override `vars.WATTLE_SSH_HOST` /
+#      `vars.WATTLE_SSH_USER` if they differ from the defaults below.
+#   3. Confirm the runner user can drive wattle's control plane over SSH. The
+#      job calls `spx` (read-only checks) and the `aws` CLI with the already-
+#      configured `spinifex` profile directly — no wrapper scripts, no install
+#      on wattle (tf-user is already in the `spinifex` group there):
+#        ssh tf-user@192.168.1.13 'AWS_PROFILE=spinifex aws ec2 describe-regions'
+#   4. Manual-work mutex: an agent doing hands-on GPU testing on wattle can
+#      block CI by touching a sentinel file in the RUNNER's home directory
+#      (NOT on wattle):
+#        touch ~/.ci-gpu-paused     # blocks CI
+#        rm ~/.ci-gpu-paused        # resumes CI
+#      or by setting the repo/environment variable `CI_GPU_PAUSED=true`
+#      (no runner filesystem access needed). Either one causes this workflow
+#      to skip cleanly (job succeeds, all work steps skipped) instead of
+#      queuing behind manual work indefinitely.
+#
+#   Runbook: docs/development/feature/wattle-deploy-and-ci.md (Part B).
+# ---------------------------------------------------------------------------
+
+name: GPU E2E
+
+on:
+  workflow_dispatch:
+    inputs:
+      test_run:
+        description: "Optional -test.run regex scoped to tests/e2e/gpu (empty = run all GPU tests)"
+        required: false
+        default: ""
+
+permissions:
+  contents: read
+
+# Single dedicated runner (see setup note 1 above) + this group together
+# enforce single-flight on the one RTX A1000: a second dispatch has nowhere
+# else to land (unique label) and, if it somehow overlaps in scheduling,
+# queues here rather than cancelling/stomping the in-flight run.
+concurrency:
+  group: ci-gpu
+  cancel-in-progress: false
+
+env:
+  # OTLP export to the central Elastic sink (wattle is the sink itself — see
+  # Part A.2). Repo variable overrides; carried for correlation even though
+  # the GPU suite's own daemon-side spans are emitted by wattle's already-
+  # configured obs-agent, not by this job.
+  SPINIFEX_OTLP_ENDPOINT: ${{ vars.SPINIFEX_OTLP_ENDPOINT || 'http://192.168.1.13:8200' }}
+  # ci.cell=gpu keeps this workflow's telemetry separable from the tofu-env
+  # workflows'. Nothing in this job bakes these attrs into a daemon (see the
+  # note above), so the label only takes effect once wattle's obs-agent is
+  # given them; there is deliberately no ci.job_id lookup for the same reason.
+  SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=gpu
+  WATTLE_SSH_HOST: ${{ vars.WATTLE_SSH_HOST || '192.168.1.13' }}
+  WATTLE_SSH_USER: ${{ vars.WATTLE_SSH_USER || 'tf-user' }}
+  TEST_RUN: ${{ inputs.test_run || '' }}
+
+jobs:
+  gpu_e2e:
+    name: GPU E2E (wattle)
+    runs-on: [self-hosted, ci-gpu]
+    timeout-minutes: 75
+    env:
+      ARTIFACT_DIR: /tmp/spinifex-gpu-e2e-artifacts-${{ github.run_id }}
+    steps:
+      # Manual-work mutex gate — see setup note 4 above. Checked BEFORE any
+      # SSH to wattle so a paused run touches the box zero times.
+      - name: Manual-work mutex gate
+        id: gate
+        env:
+          REPO_VAR_PAUSED: ${{ vars.CI_GPU_PAUSED }}
+        run: |
+          set -euo pipefail
+          PAUSED=false
+          if [ -f "$HOME/.ci-gpu-paused" ]; then
+            PAUSED=true
+            echo "::warning::ci-gpu runner paused via ~/.ci-gpu-paused sentinel on the runner — skipping this run (manual GPU work in progress on wattle). Remove the file to resume CI."
+          fi
+          if [ "$(echo "${REPO_VAR_PAUSED:-}" | tr '[:upper:]' '[:lower:]')" = "true" ]; then
+            PAUSED=true
+            echo "::warning::ci-gpu paused via repo/environment variable CI_GPU_PAUSED=true — skipping this run."
+          fi
+          echo "paused=${PAUSED}" >> "$GITHUB_OUTPUT"
+
+      - name: Paused — nothing to do
+        if: steps.gate.outputs.paused == 'true'
+        run: echo "ci-gpu is paused; job exits cleanly without touching wattle. See gate step above for the reason."
+
+      - name: Clean Workspace
+        if: steps.gate.outputs.paused != 'true'
+        run: sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/checkout@v7
+        if: steps.gate.outputs.paused != 'true'
+
+      - uses: actions/setup-go@v7
+        if: steps.gate.outputs.paused != 'true'
+        with:
+          go-version-file: go.mod
+          cache: false
+
+      - name: Install wattle SSH key
+        if: steps.gate.outputs.paused != 'true'
+        run: |
+          set -euo pipefail
+          install -m 0600 /dev/null "${RUNNER_TEMP}/wattle-ci"
+          echo "${{ secrets.WATTLE_SSH_KEY }}" > "${RUNNER_TEMP}/wattle-ci"
+          echo "ssh_opts=-o StrictHostKeyChecking=no -o ConnectTimeout=10 -i ${RUNNER_TEMP}/wattle-ci" >> "$GITHUB_ENV"
+
+      # Healthcheck: confirm wattle's persistent AWS-compat control plane is
+      # actually up before spending time building/shipping the test binary.
+      - name: Healthcheck — describe-regions
+        if: steps.gate.outputs.paused != 'true'
+        run: |
+          set -euo pipefail
+          ssh ${{ env.ssh_opts }} "${WATTLE_SSH_USER}@${WATTLE_SSH_HOST}" \
+            'AWS_PROFILE=spinifex aws ec2 describe-regions --output text'
+
+      # Ensure the base EC2 GPU AMI is present — the raw-EC2 GPU passthrough
+      # test is the always-on baseline, so CI self-provisions it rather than
+      # depending on a manual import. `--name` selects the catalog entry (pulls
+      # the prebuilt image from iso.mulgadc.com); `--ami-name` is REQUIRED to
+      # name the AMI ubuntu-26.04-nvidia-gpu-x86_64 — without it the import
+      # falls back to the generic ami-<distro>-<version>-<arch>, which the
+      # tests/e2e/gpu filter would never match. Idempotent — skipped once
+      # wattle's persistent loop-fs already has the named AMI (the 4.4G source
+      # qcow2 is cached in the images dir, so a re-import wouldn't re-download).
+      # `spx admin images import` writes local spinifex storage, so it needs
+      # the `spinifex` group active (sg); aws-CLI calls hit the gateway over
+      # HTTPS and don't.
+      - name: Ensure base GPU AMI present (import-if-absent)
+        if: steps.gate.outputs.paused != 'true'
+        run: |
+          set -euo pipefail
+          REMOTE='
+            set -e
+            BASE=ubuntu-26.04-nvidia-gpu-x86_64
+            present() {
+              AWS_PROFILE=spinifex aws ec2 describe-images \
+                --filters "Name=name,Values=$BASE" "Name=state,Values=available" \
+                --query "Images[0].ImageId" --output text 2>/dev/null
+            }
+            ID=$(present)
+            if [ -n "$ID" ] && [ "$ID" != "None" ]; then
+              echo "base GPU AMI already present: $BASE = $ID"; exit 0
+            fi
+            echo "base GPU AMI $BASE absent — importing prebuilt from catalog..."
+            sg spinifex -c "AWS_PROFILE=spinifex spx admin images import --name $BASE --ami-name $BASE"
+            for i in 1 2 3 4 5 6; do
+              ID=$(present)
+              [ -n "$ID" ] && [ "$ID" != "None" ] && break
+              sleep 5
+            done
+            [ -n "$ID" ] && [ "$ID" != "None" ] || { echo "import ran but $BASE still not available" >&2; exit 1; }
+            echo "imported base GPU AMI: $BASE = $ID"
+          '
+          ssh ${{ env.ssh_opts }} "${WATTLE_SSH_USER}@${WATTLE_SSH_HOST}" "$REMOTE"
+
+      # Preflight — the base GPU AMI is REQUIRED: the setup step above just
+      # ensured it, so a miss here means the import silently failed. Hard-fail
+      # rather than run a false-green all-skipped suite. ECS/EKS node AMIs are
+      # opt-in tiers (heavier — node AMIs + cluster spin-up); their tests
+      # self-skip when absent, so only warn.
+      - name: Preflight — verify GPU AMIs
+        if: steps.gate.outputs.paused != 'true'
+        run: |
+          set -euo pipefail
+          REMOTE='
+            set -e
+            present() {
+              AWS_PROFILE=spinifex aws ec2 describe-images \
+                --filters "Name=name,Values=$1" "Name=state,Values=available" \
+                --query "Images[0].ImageId" --output text 2>/dev/null
+            }
+            BASE=ubuntu-26.04-nvidia-gpu-x86_64
+            ID=$(present "$BASE")
+            if [ -z "$ID" ] || [ "$ID" = "None" ]; then
+              echo "required base GPU AMI $BASE missing — cannot run the baseline EC2 GPU test" >&2
+              exit 1
+            fi
+            echo "required: $BASE = $ID"
+            for OPT in spinifex-ecs-node-gpu spinifex-eks-node spinifex-eks-node-gpu; do
+              OID=$(present "$OPT")
+              if [ -z "$OID" ] || [ "$OID" = "None" ]; then
+                echo "::warning::$OPT not imported — its GPU test(s) will skip (opt-in tier)"
+              else
+                echo "opt-in: $OPT = $OID"
+              fi
+            done
+          '
+          ssh ${{ env.ssh_opts }} "${WATTLE_SSH_USER}@${WATTLE_SSH_HOST}" "$REMOTE"
+
+      # Same build command CI's other e2e workflows use to cross-compile a
+      # suite into a shippable binary (tests/e2e/Makefile e2e-build target,
+      # SUITES_BUILD scoped to just gpu). Building on the runner keeps go
+      # toolchain / module cache off wattle entirely.
+      - name: Build GPU e2e test binary on runner
+        if: steps.gate.outputs.paused != 'true'
+        working-directory: tests/e2e
+        run: |
+          set -euo pipefail
+          make e2e-build SUITES_BUILD=gpu
+          ls -la _bin/gpu.test
+
+      - name: Install go-junit-report on runner
+        if: steps.gate.outputs.paused != 'true'
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Ship GPU e2e test binary to wattle
+        if: steps.gate.outputs.paused != 'true'
+        run: |
+          set -euo pipefail
+          VM="${WATTLE_SSH_USER}@${WATTLE_SSH_HOST}"
+          ssh ${{ env.ssh_opts }} "$VM" "mkdir -p \$HOME/spinifex-gpu-e2e-bin"
+          scp ${{ env.ssh_opts }} tests/e2e/_bin/gpu.test "$VM":spinifex-gpu-e2e-bin/gpu.test
+
+      # Same invocation shape e2e.yml uses to run a pre-built suite binary
+      # (SPINIFEX_E2E=1 SPINIFEX_MODE=single, ./<suite>.test -test.v
+      # -test.timeout=NNm), plus SPINIFEX_WAN_IP: wattle's ~24 bridges make
+      # net.InterfaceAddrs() enumeration pick a co-tenant bridge instead of
+      # the advertised gateway (mulga-7suds), so the override is required
+      # here (proper fix tracked separately: tests/e2e/harness/env.go should
+      # prefer spinifex.toml's advertised bridge).
+      - name: Run GPU e2e suite on wattle
+        id: e2e
+        if: steps.gate.outputs.paused != 'true'
+        run: |
+          set +e
+          VM="${WATTLE_SSH_USER}@${WATTLE_SSH_HOST}"
+          if [ -n "${TEST_RUN}" ]; then
+            TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+          else
+            TEST_RUN_FLAG=""
+          fi
+          ssh -tt ${{ env.ssh_opts }} "$VM" \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/spinifex-gpu-e2e-bin; \
+             GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+               SPINIFEX_WAN_IP='${WATTLE_SSH_HOST}' \
+               ARTIFACT_DIR='${ARTIFACT_DIR}' \
+               ./gpu.test -test.v -test.timeout=60m ${TEST_RUN_FLAG} \
+               2>&1 | tee '${ARTIFACT_DIR}/test-gpu.log'; \
+             exit \${PIPESTATUS[0]}"
+          E2E_RC=$?
+          echo "rc=${E2E_RC}" >> "$GITHUB_OUTPUT"
+          exit 0
+
+      - name: Collect artifacts from wattle
+        if: always() && steps.gate.outputs.paused != 'true'
+        run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
+          VM="${WATTLE_SSH_USER}@${WATTLE_SSH_HOST}"
+          ssh ${{ env.ssh_opts }} "$VM" "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+          echo "collected: $(ls "${ARTIFACT_DIR}" 2>/dev/null | wc -l) entries"
+
+      - name: Convert test log to JUnit XML
+        if: always() && steps.gate.outputs.paused != 'true'
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log)
+            SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Publish test summary
+        if: always() && steps.gate.outputs.paused != 'true'
+        uses: ./.github/actions/junit-report
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          title: "GPU E2E results"
+
+      - name: Analyze e2e failures
+        if: always() && steps.gate.outputs.paused != 'true'
+        uses: ./.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "GPU E2E failure analysis"
+
+      - uses: actions/upload-artifact@v7
+        if: always() && steps.gate.outputs.paused != 'true'
+        with:
+          name: gpu-e2e-artifacts-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
+      # Cleanup is NEVER a host teardown: it never runs setup-ovn.sh,
+      # dev-reset, or cluster-reset. It only reconciles EC2-API-level state —
+      # terminate any GPU-typed instance still pending/running (the Go
+      # suite's own t.Cleanup already does this on normal pass/fail exit;
+      # this is the backstop for a hard `-test.timeout` kill or a workflow
+      # cancellation, which skip deferred Go cleanups). Termination triggers
+      # gpuManager.Release() on wattle, returning the A1000 to the pool for
+      # the next queued run or manual test.
+      - name: Cleanup — terminate leftover GPU instances, release GPU
+        if: always() && steps.gate.outputs.paused != 'true'
+        run: |
+          set +e
+          VM="${WATTLE_SSH_USER}@${WATTLE_SSH_HOST}"
+          REMOTE='
+            set +e
+            GPU_TYPES=$(AWS_PROFILE=spinifex aws ec2 describe-instance-types \
+              --query "InstanceTypes[?GpuInfo!=null].InstanceType" --output text | tr "\t" ",")
+            if [ -z "$GPU_TYPES" ]; then
+              echo "no GPU instance types advertised; nothing to reconcile"
+              exit 0
+            fi
+            IDS=$(AWS_PROFILE=spinifex aws ec2 describe-instances \
+              --filters "Name=instance-type,Values=${GPU_TYPES}" "Name=instance-state-name,Values=pending,running" \
+              --query "Reservations[].Instances[].InstanceId" --output text)
+            if [ -z "$IDS" ] || [ "$IDS" = "None" ]; then
+              echo "no leftover GPU instances — GPU already released"
+              exit 0
+            fi
+            echo "terminating leftover GPU instance(s): $IDS"
+            AWS_PROFILE=spinifex aws ec2 terminate-instances --instance-ids $IDS
+          '
+          ssh ${{ env.ssh_opts }} "$VM" "$REMOTE"
+
+      - name: Fail job if GPU suite failed
+        if: always() && steps.gate.outputs.paused != 'true'
+        run: |
+          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
+            echo "::error::GPU e2e suite failed (rc=${{ steps.e2e.outputs.rc }}) — see analysis above; raw ES/APM traces at ${SPINIFEX_OTLP_ENDPOINT} (ci.run_id=${{ github.run_id }})"
+            exit 1
+          fi

.github/workflows/publish-eks-ami.yml

@@ -1,18 +1,34 @@
 name: Publish EKS + ECS node AMIs
 
-# Builds the eks-node and ecs-agent system AMIs (libguestfs, on the runner) and
-# publishes each compressed qcow2 + .sha256 sidecar to the Cloudflare R2 bucket
-# that backs iso.mulgadc.com/system-ami/. Operators then install them on any
-# cluster with:
+# Builds the system AMIs (libguestfs, on the runner) and publishes each
+# compressed qcow2 + .sha256 sidecar to the Cloudflare R2 bucket that backs
+# iso.mulgadc.com/system-ami/. Four images ship:
+#   spinifex-eks-node       Alpine EKS K3s node (server|agent, role at boot)
+#   spinifex-ecs-node       Alpine ECS container instance
+#   spinifex-eks-node-gpu   Ubuntu EKS K3s GPU worker (NVIDIA driver + toolkit)
+#   spinifex-ecs-node-gpu   Ubuntu ECS GPU container instance
+# Operators then install them on any cluster with:
 #   spx admin images import --name spinifex-eks-node
 #   spx admin images import --name spinifex-ecs-node
+#   spx admin images import --name spinifex-eks-node-gpu
+#   spx admin images import --name spinifex-ecs-node-gpu
 #
 # Manual-only: publishing overwrites the mirror artifact, so it is a deliberate
-# action, not something every push should trigger.
+# action, not something every push should trigger. The `variant` input scopes
+# which images build — the Ubuntu GPU legs are 16G and much slower (NVIDIA
+# driver download + DKMS build), so publishing only the CPU AMIs stays cheap.
 
 on:
   workflow_dispatch:
     inputs:
+      variant:
+        description: "Which images to build"
+        type: choice
+        default: all
+        options:
+          - all
+          - cpu
+          - gpu
       dry_run:
         description: "Build + convert only; skip the R2 upload"
         type: boolean
@@ -26,9 +42,37 @@ concurrency:
   cancel-in-progress: false
 
 jobs:
-  publish:
-    name: Build + publish eks-node + ecs-agent AMIs
+  matrix:
+    name: Resolve image matrix
     runs-on: [ self-hosted, ci-single ]
+    outputs:
+      images: ${{ steps.gen.outputs.images }}
+    steps:
+      - name: Select images for variant
+        id: gen
+        env:
+          VARIANT: ${{ inputs.variant }}
+        run: |
+          set -euo pipefail
+          cpu='{"name":"spinifex-eks-node","manifest":"scripts/images/eks-node/manifest.conf"},{"name":"spinifex-ecs-node","manifest":"scripts/images/ecs-agent/manifest.conf"}'
+          gpu='{"name":"spinifex-eks-node-gpu","manifest":"scripts/images/eks-node-gpu/manifest.conf"},{"name":"spinifex-ecs-node-gpu","manifest":"scripts/images/ecs-agent-gpu/manifest.conf"}'
+          case "${VARIANT}" in
+            cpu) items="${cpu}" ;;
+            gpu) items="${gpu}" ;;
+            *)   items="${cpu},${gpu}" ;;
+          esac
+          echo "images={\"include\":[${items}]}" >> "$GITHUB_OUTPUT"
+
+  publish:
+    name: Build + publish ${{ matrix.name }}
+    needs: matrix
+    runs-on: [ self-hosted, ci-single ]
+    strategy:
+      fail-fast: false
+      # ci-single is one runner and each GPU build needs ~16G of scratch disk;
+      # serialise the legs so they never contend for /_temp or the mirror.
+      max-parallel: 1
+      matrix: ${{ fromJSON(needs.matrix.outputs.images) }}
 
     steps:
       - name: Checkout Spinifex
@@ -51,7 +95,7 @@ jobs:
           path: predastore
 
       - name: Setup Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           go-version-file: "spinifex/go.mod"
           # Self-hosted runner: the Go module + build cache already persists on
@@ -61,7 +105,13 @@ jobs:
           cache: false
 
       - name: Setup Go Workspace
-        run: go work init ./spinifex ./viperblock ./predastore
+        # Idempotent: the ci-single runner reuses one workspace across matrix
+        # legs and runs, and checkout never scrubs the root go.work. Remove any
+        # stale file so each leg re-inits cleanly instead of failing on
+        # "go.work already exists".
+        run: |
+          rm -f go.work go.work.sum
+          go work init ./spinifex ./viperblock ./predastore
 
       - name: Install build + upload tooling
         run: |
@@ -75,7 +125,7 @@ jobs:
             rm -rf /tmp/awscliv2.zip /tmp/aws-install
           fi
 
-      - name: Build + publish eks-node to R2
+      - name: Build + publish ${{ matrix.name }} to R2
         working-directory: spinifex
         env:
           # Map the repo's R2 secrets onto the names publish-system-image.sh reads.
@@ -84,34 +134,20 @@ jobs:
           R2_ENDPOINT: ${{ secrets.CF_R2_ENDPOINT_URL }}
           R2_BUCKET: spinifex-iso
           DRY_RUN: ${{ inputs.dry_run }}
+          MANIFEST: ${{ matrix.manifest }}
+          # Ubuntu GPU images are 16G — a RAM-backed /tmp cannot hold the qcow2 +
+          # expanded raw, so point the build scratch at the runner's on-disk
+          # _temp. Harmless for the small Alpine legs too.
+          SYSTEM_IMAGE_BUILD_DIR: ${{ runner.temp }}/system-image-build
         run: |
           set -euo pipefail
           # libguestfs needs to read a host kernel; Ubuntu ships it mode 0600.
           # direct backend = no libvirt on the runner.
           export LIBGUESTFS_BACKEND=direct
           sudo chmod 0644 /boot/vmlinuz-* || true
+          mkdir -p "${SYSTEM_IMAGE_BUILD_DIR}"
           FLAGS=(--build)
           if [[ "${DRY_RUN}" == "true" ]]; then
             FLAGS+=(--dry-run)
           fi
-          ./scripts/publish-system-image.sh \
-            scripts/images/eks-node/manifest.conf "${FLAGS[@]}"
-
-      - name: Build + publish ecs-agent to R2
-        working-directory: spinifex
-        env:
-          AWS_ACCESS_KEY_ID: ${{ secrets.CF_R2_ACCESS_KEY_ID }}
-          AWS_SECRET_ACCESS_KEY: ${{ secrets.CF_R2_SECRET_ACCESS_KEY }}
-          R2_ENDPOINT: ${{ secrets.CF_R2_ENDPOINT_URL }}
-          R2_BUCKET: spinifex-iso
-          DRY_RUN: ${{ inputs.dry_run }}
-        run: |
-          set -euo pipefail
-          export LIBGUESTFS_BACKEND=direct
-          sudo chmod 0644 /boot/vmlinuz-* || true
-          FLAGS=(--build)
-          if [[ "${DRY_RUN}" == "true" ]]; then
-            FLAGS+=(--dry-run)
-          fi
-          ./scripts/publish-system-image.sh \
-            scripts/images/ecs-agent/manifest.conf "${FLAGS[@]}"
+          ./scripts/publish-system-image.sh "${MANIFEST}" "${FLAGS[@]}"

.github/workflows/release.yml

@@ -32,7 +32,7 @@ jobs:
         uses: docker/setup-buildx-action@v4
 
       - name: Set up Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           go-version: "1.26.5"
 
@@ -280,7 +280,7 @@ jobs:
         run: sudo make -C mulga/scripts/iso-builder clean
 
       - name: Set up Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false

.github/workflows/spinifex.yml

@@ -32,22 +32,29 @@ jobs:
           path: spinifex
           fetch-depth: 2
 
+      - name: Resolve sibling refs
+        id: siblings
+        env:
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+        run: bash .github/resolve-sibling-refs.sh
+        working-directory: spinifex
+
       - name: Checkout Viperblock
         uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
-          ref: dev
+          ref: ${{ steps.siblings.outputs.viperblock }}
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
-          ref: dev
+          ref: ${{ steps.siblings.outputs.predastore }}
           path: predastore
 
       - name: Setup Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           cache-dependency-path: "spinifex/go.sum"
           go-version-file: "spinifex/go.mod"
@@ -70,6 +77,55 @@ jobs:
       - name: Check Diff Coverage
         run: cd spinifex && scripts/diff-coverage.sh coverage.out --base HEAD~1
 
+  # In-process integration tier (gateway.SetupRoutes over embedded NATS
+  # JetStream). Its own job so it runs in parallel with the unit tiers rather
+  # than adding to their wall time, and it still blocks a PR — unlike e2e.yml's
+  # "Integration (front gate)" job, which is push-triggered against main/dev on
+  # a self-hosted runner and so never gates a PR before merge.
+  integration_tests:
+    name: Integration Tests
+    runs-on: ubuntu-26.04
+
+    steps:
+      - name: Checkout Spinifex
+        uses: actions/checkout@v7
+        with:
+          path: spinifex
+          fetch-depth: 2
+
+      - name: Resolve sibling refs
+        id: siblings
+        env:
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+        run: bash .github/resolve-sibling-refs.sh
+        working-directory: spinifex
+
+      - name: Checkout Viperblock
+        uses: actions/checkout@v7
+        with:
+          repository: mulgadc/viperblock
+          ref: ${{ steps.siblings.outputs.viperblock }}
+          path: viperblock
+
+      - name: Checkout Predastore
+        uses: actions/checkout@v7
+        with:
+          repository: mulgadc/predastore
+          ref: ${{ steps.siblings.outputs.predastore }}
+          path: predastore
+
+      - name: Setup Go
+        uses: actions/setup-go@v7
+        with:
+          cache-dependency-path: "spinifex/go.sum"
+          go-version-file: "spinifex/go.mod"
+
+      - name: Setup Go Workspace
+        run: go work init ./spinifex ./viperblock ./predastore
+
+      - name: Run Integration Tests
+        run: make -C spinifex test-integration
+
   race_detection:
     name: Race Detection
     runs-on: ubuntu-26.04
@@ -80,22 +136,29 @@ jobs:
         with:
           path: spinifex
 
+      - name: Resolve sibling refs
+        id: siblings
+        env:
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+        run: bash .github/resolve-sibling-refs.sh
+        working-directory: spinifex
+
       - name: Checkout Viperblock
         uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
-          ref: dev
+          ref: ${{ steps.siblings.outputs.viperblock }}
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
-          ref: dev
+          ref: ${{ steps.siblings.outputs.predastore }}
           path: predastore
 
       - name: Setup Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           cache-dependency-path: "spinifex/go.sum"
           go-version-file: "spinifex/go.mod"
@@ -116,22 +179,29 @@ jobs:
         with:
           path: spinifex
 
+      - name: Resolve sibling refs
+        id: siblings
+        env:
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+        run: bash .github/resolve-sibling-refs.sh
+        working-directory: spinifex
+
       - name: Checkout Viperblock
         uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
-          ref: dev
+          ref: ${{ steps.siblings.outputs.viperblock }}
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
-          ref: dev
+          ref: ${{ steps.siblings.outputs.predastore }}
           path: predastore
 
       - name: Setup Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           cache-dependency-path: "spinifex/go.sum"
           go-version-file: "spinifex/go.mod"

.github/workflows/ui.yml

@@ -40,7 +40,7 @@ jobs:
           package_json_file: spinifex/services/spinifexui/frontend/package.json
 
       - name: Setup Node.js
-        uses: actions/setup-node@v6
+        uses: actions/setup-node@v7
         with:
           cache: "pnpm"
           cache-dependency-path: spinifex/services/spinifexui/frontend/pnpm-lock.yaml

.golangci.yml

@@ -1,5 +1,8 @@
 version: "2"
 
+issues:
+  max-same-issues: 10
+
 formatters:
   enable:
     - gofmt
@@ -14,24 +17,29 @@ linters:
     - bidichk # checks for dangerous unicode bidi character sequences
     - bodyclose # checks whether HTTP response body is closed successfully
     - canonicalheader # checks whether net/http.Header uses canonical header keys
-    - decorder # checks declaration order of types, consts, vars, and funcs
+    - depguard # blocks imports of packages with a preferred replacement
     - durationcheck # checks for two durations multiplied together
     - embeddedstructfieldcheck # checks embedded sync.Mutex/RWMutex in structs
     - errcheck # checks for unchecked errors in type assertions and I/O
     - errchkjson # checks types passed to json encoding functions, flags omittable error checks
     - errname # checks that sentinel errors are prefixed with Err, error types suffixed with Error
+    - errorlint # catches ==/!= on errors, type assertions on errors, and %v instead of %w — all break on wrapped errors
     - exptostd # detects golang.org/x/exp/ functions replaceable by stdlib
     - fatcontext # detects nested contexts in loops
-    - forcetypeassert # flags unchecked type assertions that can panic
+    - forbidigo # blocks the legacy JetStream/KV symbols
     - gocheckcompilerdirectives # validates go compiler directive comments (//go:)
+    - godot # checks that declaration comments end in a period
     - goprintffuncname # checks that printf-like functions are named with f at the end
     - gosec # inspects source code for security problems
     - govet # reports suspicious constructs (Printf alignment, shadow, etc.)
+    - iface # detects interface pollution — duplicate interface declarations
     - ineffassign # detects assignments to existing variables that are not used
+    - iotamixing # flags const blocks that mix iota with non-iota declarations
     - makezero # finds slice declarations with non-zero initial length followed by append
     - mirror # catches wrong mirror patterns in bytes/strings usage
-    - musttag # checks structs passed to marshal/unmarshal have proper tags
+    - misspell # finds commonly misspelled English words
     - modernize # modernizes code to use newer Go idioms (go fix analyzers)
+    - musttag # checks structs passed to marshal/unmarshal have proper tags
     - nakedret # finds naked returns in functions
     - nilerr # catches code that returns nil even after checking err != nil
     - nilnesserr # catches returning a different nil error variable after err != nil check
@@ -43,6 +51,8 @@ linters:
     - recvcheck # checks for receiver type consistency (pointer vs value)
     - sloglint # enforces consistent log/slog usage (key-value pairs, no globals)
     - staticcheck # extensive static analysis checks
+    - testifylint # checks common testify mistakes (assert.Len over assert.Equal on len, etc.)
+    - tparallel # detects inappropriate usage of t.Parallel()
     - unconvert # removes unnecessary type conversions
     - usestdlibvars # detects hardcoded values that should use stdlib constants
     - usetesting # catches os.TempDir() etc. in tests — should use t.TempDir()
@@ -50,14 +60,41 @@ linters:
     - whitespace # detects leading and trailing whitespace in functions
 
   settings:
+    depguard:
+      rules:
+        main:
+          deny:
+            - pkg: log$
+              desc: Use log/slog instead, see https://go.dev/blog/slog
+            - pkg: math/rand$
+              desc: Use math/rand/v2 instead, see https://go.dev/blog/randv2
+
     embeddedstructfieldcheck:
       forbid-mutex: true
 
     errcheck:
-      check-type-assertions: true
+      check-type-assertions: true # replaces forcetypeassert
       exclude-functions:
         - (*github.com/nats-io/nats.go.Subscription).Unsubscribe # cleanup-only, errors not actionable
 
+    forbidigo:
+      analyze-types: true
+      forbid:
+        - pattern: '\.JetStream$'
+          pkg: '^github\.com/nats-io/nats\.go$'
+          msg: legacy nc.JetStream() is superseded; construct with jetstream.New(nc).
+        - pattern: '^nats\.(JetStreamContext|KeyValue.*|KeyWatcher|Err(KeyNotFound|KeyExists|BucketNotFound|NoKeysFound|StreamNotFound))$'
+          pkg: '^github\.com/nats-io/nats\.go$'
+          msg: legacy nats KV types/sentinels are superseded; use github.com/nats-io/nats.go/jetstream.
+
+    gosec:
+      excludes:
+        - G104 # unchecked errors — errcheck is stricter
+        - G116 # bidi chars — bidichk is more thorough
+        - G117 # use of net/http serve functions (we handle TLS separately)
+        - G703 # path traversal taint analysis — false positives on config file loading
+        - G705 # XSS taint analysis — false positives on internal error responses
+
     govet:
       enable-all: true
       disable:
@@ -65,6 +102,28 @@ linters:
         - shadow # too noisy to retrofit — all 50 findings are err re-declarations
         - unusedwrite # false positives on struct field assignments before marshal/send
 
+    modernize:
+      disable:
+        - newexpr # don't replace aws.String() with new() — AWS SDK idiom
+        - stringsbuilder # don't replace small string concatenations with strings.Builder
+
+    nakedret:
+      max-func-lines: 0
+
+    nolintlint:
+      require-explanation: true
+      require-specific: true
+
+    perfsprint:
+      concat-loop: false
+      err-error: false # don't flag err.Error()
+      errorf: false # don't convert fmt.Errorf to errors.New
+      strconcat: false
+
+    reassign:
+      patterns:
+        - ".*"
+
     staticcheck:
       checks:
         - all
@@ -76,40 +135,14 @@ linters:
         - -ST1020 # godoc on exported types — not enforcing yet
         - -ST1021 # godoc on exported type groups — not enforcing yet
         - -ST1022 # godoc on exported vars/consts — not enforcing yet
-        - -SA1019 # deprecated usage (handle separately)
         - -SA4006 # value overwritten — ineffassign + wastedassign cover this better
-        - -SA9005 # ineffective break (staticcheck false positives)
+        - -SA9005 # marshaling structs with no exported fields — intended {} for empty API-response types
 
-    gosec:
-      excludes:
-        - G104 # unchecked errors — errcheck is stricter
-        - G116 # bidi chars — bidichk is more thorough
-        - G117 # use of net/http serve functions (we handle TLS separately)
-        - G703 # path traversal taint analysis — false positives on config file loading
-        - G705 # XSS taint analysis — false positives on internal error responses
-        - G706 # log injection taint analysis — false positives on structured slog usage
-
-    modernize:
+    testifylint:
       disable:
-        - newexpr # don't replace aws.String() with new() — AWS SDK idiom
-        - stringsbuilder # don't replace small string concatenations with strings.Builder
-
-    perfsprint:
-      concat-loop: false
-      err-error: false # don't flag err.Error()
-      errorf: false # don't convert fmt.Errorf to errors.New
-      strconcat: false
-
-    nakedret:
-      max-func-lines: 0
-
-    nolintlint:
-      require-explanation: true
-      require-specific: true
-
-    reassign:
-      patterns:
-        - ".*"
+        - encoded-compare # misidentifies non-JSON strings when identifiers contain "JSON"
+        - float-compare # false positives
+        - require-error # assert-on-error is deliberate where a test continues to collect further failures
 
   exclusions:
     presets:
@@ -133,3 +166,8 @@ linters:
         linters: [gosec] # subprocess execution — legitimate in VM orchestration
       - text: "G304"
         linters: [gosec] # file path from variable — necessary for config loading
+
+      # aws-sdk-go v1 is deprecated wholesale (EOL 2025-07-31); the v1->v2
+      # migration is tracked separately, so suppress its SA1019 noise for now.
+      - text: "aws-sdk-go is deprecated"
+        linters: [staticcheck]

Makefile

@@ -120,8 +120,13 @@ install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
 
-# Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
-# Use this before committing to catch CI failures locally.
+# Preflight — the pre-commit gate: manifest checks, lint, vuln, and the unit,
+# race and e2e-harness tiers.
+#
+# The integration tier is deliberately NOT here. CI runs it as its own parallel
+# job, so it is still gated before merge, and keeping it out leaves preflight
+# the fast common-path check — run `make test-integration` directly when a
+# change touches the gateway router or the NATS subjects it drives.
 preflight:
 	@$(MAKE) --no-print-directory QUIET=1 manifest-check manifest-lint lint govulncheck test-cover diff-coverage test-race test-harness
 	@echo -e "\n ✅ Preflight passed — safe to commit."
@@ -133,6 +138,18 @@ test-harness:
 	@echo -e "\n....Running e2e harness unit tests...."
 	$(_Q)LOG_IGNORE=1 go test -tags=e2e -timeout 60s ./tests/e2e/harness/... $(_RACEQ)
 
+# In-process integration tier: the real gateway router against embedded NATS
+# JetStream, with only the daemon-side NATS subjects stubbed. Build-tagged
+# `integration` so it's skipped by the default `go test ./spinifex/...` and by
+# `test-cover`/`test-race`. Nothing provisioned — no tofu, no docker, no
+# Spinifex daemons, so the whole package runs in well under a minute. It gets
+# its own PR-blocking CI job rather than the self-hosted, push-triggered live
+# e2e tiers, so a regression here is caught before it can be merged, not just
+# after.
+test-integration:
+	@echo -e "\n....Running in-process integration tests...."
+	$(_Q)LOG_IGNORE=1 go test -tags=integration -timeout 60s ./tests/integration/... $(_RACEQ)
+
 # Validate docs/service-interfaces.yaml. Schema check + cross-reference
 # of services/suites/fixtures + on-disk path existence. Subject content
 # vs source is enforced separately in Bead 5 drift lint.
@@ -254,6 +271,13 @@ govulncheck:
 	$(_Q)go tool govulncheck ./...
 	@echo "  govulncheck ok"
 
+# NilAway — advisory nil-panic analysis. Not in preflight: it has a known
+# false-positive rate, so findings are triaged by hand rather than gating commits.
+nilaway:
+	@echo "Running nilaway..."
+	$(_Q)go tool nilaway -include-pkgs=github.com/mulgadc/spinifex -exclude-test-files ./...
+	@echo "  nilaway ok"
+
 # Build release tarballs — use distro-ARCH for single arch, distro for both
 distro: distro-amd64 distro-arm64
 	@echo ""
@@ -297,8 +321,8 @@ distro-arm64:
 distro-clean:
 	rm -rf dist/
 
-.PHONY: build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-harness manifest-check manifest-lint manifest-lint-update \
+.PHONY: build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-harness test-integration manifest-check manifest-lint manifest-lint-update \
 	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
-	lint fix govulncheck \
+	lint fix govulncheck nilaway \
 	distro distro-amd64 distro-arm64 distro-clean

README.md

@@ -1,5 +1,5 @@
 <p align="center">
-  <img src=".github/assets/banner.svg" alt="Spinifex — AWS-Compatible Infrastructure Stack" width="900">
+  <img src=".github/assets/banner.svg" alt="Spinifex: the open, AWS-compatible cloud you run yourself. EC2, EBS, S3, VPC, and IAM on your infrastructure, at the edge, or on a Neocloud." width="900">
 </p>
 
 <p align="center">
@@ -11,6 +11,8 @@
 <p align="center">
   <a href="#what-is-spinifex">What is Spinifex?</a> ·
   <a href="#the-platform">Platform</a> ·
+  <a href="#three-ways-to-deploy">Deploy</a> ·
+  <a href="#aws-compatibility">AWS services</a> ·
   <a href="#core-components">Components</a> ·
   <a href="#architecture-at-a-glance">Architecture</a> ·
   <a href="#installation">Installation</a> ·
@@ -19,31 +21,90 @@
 
 ---
 
-# Spinifex: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
+# Spinifex: the open, AWS-compatible cloud you run yourself
 
-Spinifex developed by [Mulga Defense Corporation](https://mulgadc.com) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
+Spinifex, built by [Mulga](https://mulgadc.com), recreates the AWS services your
+software already uses (EC2, EBS, S3, VPC, IAM) on infrastructure you control.
+Move workloads off the hyperscalers, run AI at a fraction of the cost, or take
+the cloud to the edge, without rewriting a line. The exact build you ship to AWS,
+Spinifex serves on your own hardware: same APIs, same Terraform, same SDKs.
 
 ## What is Spinifex?
 
-Spinifex replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
+Spinifex is an open-source infrastructure platform that recreates the AWS service
+surface (EC2, EBS, S3, VPC, IAM) on bare-metal, edge, and on-premises
+environments. Run cloud-native software without a hyperscaler.
 
-It's designed for developers and operators who need:
+Most AWS alternatives wrap an API and call it a cloud. We rebuilt the engineering
+underneath: distributed object storage with erasure coding, block storage built
+to survive failure, and compute on bare metal. That depth is the reason your
+software runs unchanged, instead of rewritten.
 
-- Cloud-like infrastructure without the cloud
-- Minimal dependencies, full control
-- Drop-in compatibility with tools like the AWS CLI, SDKs, and Terraform
-- A secure cloud environment you control and own the entire hardware, network and software stack
+It's built for teams that need:
 
-You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren't feasible.
+- The AWS tooling they already use (AWS CLI, SDKs, Terraform), with nothing to rewrite
+- Full control of the stack: your hardware, your network, your data, your keys
+- A cloud that keeps running offline, through disconnection, with no external control plane
+- An open core (AGPL-3.0), auditable and yours to keep, with no lock-in
+
+You change *where* your software runs, not *what* it is.
 
 ## The Platform
 
 <p align="center">
-  <img src=".github/assets/platform.svg" alt="The Mulga Platform — Hardware to Applications" width="900">
+  <img src=".github/assets/platform.svg" alt="The Spinifex stack: your apps and AWS tooling on top, the Spinifex AWS-compatible service layer, running on standard Linux on CPU and GPU hardware, deployable to Neocloud, on-premise, or the edge." width="900">
 </p>
 
 From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
 
+## Three ways to deploy
+
+One AWS-compatible surface, three deployment shapes. Pick the one that matches
+your reality; the platform on top is the same.
+
+### Neocloud
+
+Lift and shift onto a partner Neocloud. Move workloads off the hyperscalers and
+onto our Neocloud partner ecosystem without rewriting them. GPU capacity
+(H100 / H200 / B200), cheaper, available now. Change an endpoint, keep the
+software.
+
+### On-premise
+
+Bring your own hardware and host Spinifex in your own data centre. A real
+multi-node HA cluster integrated with your storage and networking, with full
+control of stack, data, and jurisdiction. A predictable bill, no egress
+surprises, an auditable open-source core.
+
+### Edge
+
+Cloud where the cloud can't reach. Air-gapped sites, vehicles, vessels,
+factories, and clinics. Compute next to the data, running through disconnection,
+on hardware you own. The same AWS APIs your software already uses.
+
+## AWS compatibility
+
+Speak the AWS API surface, natively. The AWS SDKs, AWS CLI, and Terraform:
+everything you deploy on AWS deploys on Spinifex unchanged. At the edge,
+on-premise, or on a partner Neocloud.
+
+| Service | What it is | Status |
+| --- | --- | --- |
+| **EC2** | Compute | Available |
+| **EBS** | Block storage | Available |
+| **S3** | Object storage | Available |
+| **VPC** | Networking | Available |
+| **IAM** | Identity & auth | Available |
+| **ALB / NLB** | Load balancers | Available |
+| **EKS** | Kubernetes | Available |
+| **ECR** | Container registry | Available |
+| **ECS** | Container service | Available |
+| **RDS** | Databases | Q3 2026 |
+
+Roadmap items (Q3 2026) ship under the same AWS API surface. Code written for AWS
+today keeps working the moment they land.
+[Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).
+
 ## Core Components
 
 ### Spinifex (Compute Service – EC2 Alternative)
@@ -75,19 +136,23 @@ Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs
 ## Architecture at a Glance
 
 <p align="center">
-  <img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture — AWS Gateway, NATS bus, service daemons" width="900">
+  <img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus answered by stateless ec2/ebs/s3/vpc/iam daemons over local backends." width="900">
 </p>
 
-Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless — scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
+Every AWS API call is authenticated at the gateway, published to a NATS subject,
+and answered by whichever daemon claims it. Daemons are stateless: scale
+horizontally by starting more. No etcd, no Kubernetes, no external control plane.
+Just systemd units, a NATS cluster, and your hardware. Deep dive:
+**[`docs/DESIGN.md`](docs/DESIGN.md)**.
 
 ## Key Features
 
-AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, or SDKs you already use.
-
-- Designed for Control – Run on your own terms, whether that's in a datacenter, remote site, or sensitive environment.
-- Minimal Dependencies – Each service is standalone and avoids complex orchestration layers.
-- Works Offline – No reliance on centralized cloud services or external networks.
-- Open Source – Licensed under AGPL 3.0. Fork it, extend it, or deploy it as-is.
+- **AWS-compatible APIs.** Use the AWS CLI, SDKs, and Terraform you already know. Repoint your endpoint and ship; nothing to rewrite.
+- **Zero cloud dependency.** Runs entirely on your hardware. No phone-home, no control plane, no external authority. Works fully offline.
+- **Bare-metal compute.** QEMU-based with direct hardware access. No hypervisor tax, no abstraction overhead.
+- **Built-in storage.** Block and object storage included: NVMe caching, Reed-Solomon erasure coding, and replication out of the box.
+- **Edge-first architecture.** Designed for disconnected, contested, and resource-constrained environments from day one.
+- **Open core, no lock-in.** AGPL-3.0 with a commercial option. Inspect it, modify it, deploy it. The platform is yours to keep.
 
 ## Installation
 
@@ -192,6 +257,10 @@ While Spinifex is architected and implemented by experienced engineers, we lever
 
 This hybrid approach ensures Spinifex benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
 
+## Trademarks
+
+"AWS", "Amazon Web Services", and all related service names (EC2, EBS, S3, VPC, IAM, and others) are trademarks of Amazon.com, Inc. or its affiliates. Mulga Defense Corporation and Spinifex are independent and not affiliated with, endorsed by, or sponsored by Amazon.com, Inc. or Amazon Web Services, Inc. References to AWS services describe interoperability and compatibility only.
+
 ## License
 
 Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE). You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.

build/systemd/spinifex-northstar.service

@@ -0,0 +1,54 @@
+[Unit]
+Description=Spinifex Northstar (Authoritative DNS)
+PartOf=spinifex.target
+After=spinifex-nats.service spinifex-predastore.service
+Wants=spinifex-predastore.service
+
+[Service]
+EnvironmentFile=-/etc/spinifex/telemetry.env
+Type=simple
+Slice=spinifex-system.slice
+User=spinifex-northstar
+Group=spinifex
+ExecStart=/usr/local/bin/spx service northstar start
+Restart=on-failure
+RestartSec=5
+OOMScoreAdjust=-500
+Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
+Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/northstar/
+
+# Filesystem access
+NoNewPrivileges=yes
+ProtectSystem=strict
+PrivateDevices=yes
+ReadOnlyPaths=/etc/spinifex/northstar /etc/spinifex/spinifex.toml /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/northstar /var/log/spinifex
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+# RG-9: exactly CAP_NET_BIND_SERVICE so the unprivileged user binds :53 (no root).
+AmbientCapabilities=CAP_NET_BIND_SERVICE
+CapabilityBoundingSet=CAP_NET_BIND_SERVICE
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
+
+[Install]
+WantedBy=spinifex.target

build/systemd/spinifex.target

@@ -2,7 +2,7 @@
 Description=Spinifex Infrastructure Platform
 After=network-online.target
 Wants=network-online.target
-Wants=spinifex-nats.service spinifex-predastore.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-qmp-collector.service spinifex-shutdown.service spinifex-nats-watchdog.timer openvswitch-switch.service ovn-controller.service
+Wants=spinifex-nats.service spinifex-predastore.service spinifex-northstar.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-qmp-collector.service spinifex-shutdown.service spinifex-nats-watchdog.timer openvswitch-switch.service ovn-controller.service
 
 [Install]
 WantedBy=multi-user.target

cmd/dhcptest/main.go

@@ -6,7 +6,7 @@
 // Usage (must run as root — needs AF_PACKET):
 //
 //	sudo dhcptest --iface=br-wan
-//	sudo dhcptest --iface=br-wan --mac=02:00:00:7c:b7:b9 --promisc
+//	sudo dhcptest --iface=br-wan --mac=02:54:df:71:22:5c --promisc
 //	sudo dhcptest --iface=br-wan --spinifex-id=gateway-wan1
 package main
 
@@ -14,15 +14,15 @@ import (
 	"context"
 	"flag"
 	"fmt"
-	"log"
+	"log/slog"
 	"net"
 	"os"
 	"os/exec"
 	"strings"
 	"time"
 
-	"github.com/insomniacslk/dhcp/dhcpv4"
 	"github.com/insomniacslk/dhcp/dhcpv4/nclient4"
+	"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
 )
 
 func main() {
@@ -44,20 +44,25 @@ func main() {
 
 	switch {
 	case *spinifexID != "":
-		hwAddr, err = net.ParseMAC(generateMAC(*spinifexID))
+		// vpcd's own derivation, so the probe leases under the MAC the daemon
+		// would have used for this id.
+		hwAddr, err = dhcp.DeriveMAC(*spinifexID)
 		if err != nil {
-			log.Fatalf("generateMAC(%q): %v", *spinifexID, err)
+			slog.Error("Could not derive MAC", "spinifex-id", *spinifexID, "error", err)
+			os.Exit(1)
 		}
 		fmt.Printf("spinifex virtual MAC for %q: %s\n", *spinifexID, hwAddr)
 	case *macStr != "":
 		hwAddr, err = net.ParseMAC(*macStr)
 		if err != nil {
-			log.Fatalf("parse --mac %q: %v", *macStr, err)
+			slog.Error("Could not parse --mac", "mac", *macStr, "error", err)
+			os.Exit(1)
 		}
 	default:
 		hwAddr, err = ifaceMAC(*iface)
 		if err != nil {
-			log.Fatalf("get MAC for %s: %v", *iface, err)
+			slog.Error("Could not get MAC for interface", "iface", *iface, "error", err)
+			os.Exit(1)
 		}
 		fmt.Printf("using real interface MAC: %s\n", hwAddr)
 	}
@@ -83,7 +88,7 @@ func main() {
 
 	if *promisc {
 		if err := setPromisc(*iface, true); err != nil {
-			log.Printf("WARNING: failed to set promisc on %s: %v", *iface, err)
+			slog.Warn("Failed to set promisc on interface", "iface", *iface, "error", err)
 		} else {
 			fmt.Printf("set %s promisc ON\n\n", *iface)
 		}
@@ -96,26 +101,19 @@ func main() {
 		nclient4.WithRetry(*retries),
 	}
 	if *verbose {
-		opts = append(opts, nclient4.WithLogger(nclient4.ShortSummaryLogger{Printfer: log.New(os.Stderr, "[nclient4] ", 0)}))
+		opts = append(opts, nclient4.WithLogger(nclient4.ShortSummaryLogger{Printfer: slogPrintfer{}}))
 	}
 
 	client, err := nclient4.New(*iface, opts...)
 	if err != nil {
-		log.Fatalf("nclient4.New(%s): %v", *iface, err)
+		slog.Error("Could not create nclient4 client", "iface", *iface, "error", err)
+		os.Exit(1)
 	}
 	defer client.Close()
 
-	// Identity modifiers — client-id / hostname / vendor-class for upstream lease tagging.
-	var mods []dhcpv4.Modifier
-	if cid != "" {
-		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClientIdentifier([]byte(cid))))
-	}
-	if *hostname != "" {
-		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptHostName(*hostname)))
-	}
-	if *vendorClass != "" {
-		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClassIdentifier(*vendorClass)))
-	}
+	// Identity modifiers — client-id / hostname / vendor-class for upstream lease
+	// tagging. Built by the vpcd code path so the probe cannot drift from it.
+	mods := dhcp.IdentityModifiers(cid, *hostname, *vendorClass, hwAddr)
 
 	fmt.Println("Sending DHCP Discover...")
 	start := time.Now()
@@ -141,14 +139,12 @@ func main() {
 	}
 }
 
-// generateMAC is the same deterministic hash used by vpcd's topology.go.
-// It produces a locally-administered MAC (02:00:00:xx:xx:xx) from a resource ID.
-func generateMAC(resourceID string) string {
-	h := uint32(0)
-	for _, c := range resourceID {
-		h = h*31 + uint32(c)
-	}
-	return fmt.Sprintf("02:00:00:%02x:%02x:%02x", (h>>16)&0xff, (h>>8)&0xff, h&0xff)
+// slogPrintfer adapts nclient4's Printfer to slog. nclient4 only emits through
+// it under --verbose, so the packet summaries log at info level to stay visible.
+type slogPrintfer struct{}
+
+func (slogPrintfer) Printf(format string, v ...any) {
+	slog.Info(fmt.Sprintf(format, v...), "component", "nclient4")
 }
 
 func ifaceMAC(name string) (net.HardwareAddr, error) {

cmd/ecs-agent/agent.go

@@ -43,6 +43,11 @@ type Agent struct {
 	// cred serves task IAM role credentials at 169.254.170.2 (nil in unit tests).
 	cred *credEndpoint
 
+	// gpu is the local pinning ledger over the host's discovered GPU UUIDs.
+	// Always non-nil (newAgent seeds an empty one); New() reseeds it from
+	// nvidia-smi discovery.
+	gpu *gpuLedger
+
 	closers []func() error
 }
 
@@ -61,6 +66,7 @@ func newAgent(cfg config, id identity, cp controlPlane, puller ctrruntime.ImageP
 		reg:      newRegistrar(cp, id),
 		hb:       newHeartbeater(cp, id, cfg.Heartbeat),
 		netns:    newTaskNetns(execNetRunner{}),
+		gpu:      newGPULedger(id.Capacity.GPUIDs),
 	}
 }
 
@@ -84,7 +90,7 @@ func New(cfg config) (*Agent, error) {
 		InstanceID:   meta.InstanceID,
 		AZ:           meta.AZ,
 		Hostname:     host,
-		Capacity:     detectCapacity(),
+		Capacity:     detectCapacity(discoverNvidiaGPUs(execCommandRunner)),
 		AgentVersion: version,
 	}
 
@@ -153,11 +159,15 @@ func (a *Agent) Stop() error {
 	return firstErr
 }
 
-// detectCapacity reports the host's total CPU units (1 vCPU = 1024) and memory.
-func detectCapacity() bus.InstanceCapacity {
+// detectCapacity reports the host's total CPU units (1 vCPU = 1024), memory,
+// and GPU count/UUIDs from the caller's nvidia-smi discovery (nil on a
+// non-GPU host).
+func detectCapacity(gpuUUIDs []string) bus.InstanceCapacity {
 	return bus.InstanceCapacity{
 		CPU:       goruntime.NumCPU() * 1024,
 		MemoryMiB: memTotalMiB(),
+		GPU:       len(gpuUUIDs),
+		GPUIDs:    gpuUUIDs,
 	}
 }
 

cmd/ecs-agent/agent_test.go

@@ -42,7 +42,7 @@ func TestAgent_RunRegistersThenStopsOnContext(t *testing.T) {
 }
 
 func TestDetectCapacity_Positive(t *testing.T) {
-	c := detectCapacity()
+	c := detectCapacity(nil)
 	if c.CPU <= 0 {
 		t.Errorf("CPU = %d, want > 0", c.CPU)
 	}
@@ -50,4 +50,17 @@ func TestDetectCapacity_Positive(t *testing.T) {
 	if c.MemoryMiB < 0 {
 		t.Errorf("MemoryMiB = %d, want >= 0", c.MemoryMiB)
 	}
+	if c.GPU != 0 || c.GPUIDs != nil {
+		t.Errorf("no GPU UUIDs given: want GPU=0/GPUIDs=nil, got %d/%v", c.GPU, c.GPUIDs)
+	}
+}
+
+func TestDetectCapacity_WithGPUUUIDs(t *testing.T) {
+	c := detectCapacity([]string{"GPU-aaa", "GPU-bbb"})
+	if c.GPU != 2 {
+		t.Errorf("GPU = %d, want 2", c.GPU)
+	}
+	if len(c.GPUIDs) != 2 {
+		t.Errorf("GPUIDs = %v, want 2 entries", c.GPUIDs)
+	}
 }

cmd/ecs-agent/controlplane.go

@@ -29,6 +29,9 @@ type controlPlane interface {
 	// assign taskIDs and stop taskIDs accepted on the previous poll. Returns the
 	// still-pending assignments and stop directives.
 	PollAssignments(cluster, instance string, ackAssigns, ackStops []string) ([]bus.Assign, []bus.StopDirective, error)
+	// ReportTaskGPU reports the local ledger's pinned device UUIDs for a task's
+	// containers, so the control plane can populate DescribeTasks gpuIds.
+	ReportTaskGPU(cluster, task string, containers []handlers_ecs.ContainerGPUReport) error
 }
 
 // gatewayControlPlane implements controlPlane over the SigV4 ECS gateway client.
@@ -74,6 +77,11 @@ func (g *gatewayControlPlane) Register(id identity) error {
 		},
 		VersionInfo: &ecs.VersionInfo{AgentVersion: aws.String(id.AgentVersion)},
 	}
+	if len(id.Capacity.GPUIDs) > 0 {
+		in.TotalResources = append(in.TotalResources, &ecs.Resource{
+			Name: aws.String("GPU"), Type: aws.String("STRINGSET"), StringSetValue: aws.StringSlice(id.Capacity.GPUIDs),
+		})
+	}
 	body, err := json.Marshal(in)
 	if err != nil {
 		return fmt.Errorf("marshal register: %w", err)
@@ -133,3 +141,15 @@ func (g *gatewayControlPlane) PollAssignments(cluster, instance string, ackAssig
 	}
 	return out.Assignments, out.Stops, nil
 }
+
+func (g *gatewayControlPlane) ReportTaskGPU(cluster, task string, containers []handlers_ecs.ContainerGPUReport) error {
+	in := &handlers_ecs.ReportTaskGPUInput{Cluster: cluster, Task: task, Containers: containers}
+	body, err := json.Marshal(in)
+	if err != nil {
+		return fmt.Errorf("marshal report-task-gpu: %w", err)
+	}
+	if _, err := g.client.Call("ReportTaskGPU", body); err != nil {
+		return fmt.Errorf("report task gpu: %w", err)
+	}
+	return nil
+}

cmd/ecs-agent/controlplane_test.go

@@ -0,0 +1,136 @@
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/aws/aws-sdk-go/service/ecs"
+	"github.com/mulgadc/spinifex/cmd/ecs-agent/credentials"
+	handlers_ecs "github.com/mulgadc/spinifex/spinifex/handlers/ecs"
+)
+
+// fixedCredsProvider is a static credentials.CredentialsProvider for tests
+// that only need a valid SigV4 signature, not real IMDS rotation.
+type fixedCredsProvider struct{}
+
+func (fixedCredsProvider) Retrieve(context.Context) (credentials.Credentials, error) {
+	return credentials.Credentials{AccessKeyID: "AKIATEST", SecretAccessKey: "secret"}, nil
+}
+
+func newTestGatewayControlPlane(t *testing.T, srv *httptest.Server) *gatewayControlPlane {
+	t.Helper()
+	cp, err := newGatewayControlPlane(config{GatewayURL: srv.URL, Region: "us-east-1"}, fixedCredsProvider{})
+	if err != nil {
+		t.Fatalf("newGatewayControlPlane: %v", err)
+	}
+	return cp
+}
+
+// Register carries the discovered GPU UUIDs as a STRINGSET resource, matching
+// the control plane's RegisterContainerInstance GPU parsing (Epic C3).
+func TestGatewayControlPlane_Register_SendsGPUStringSet(t *testing.T) {
+	var body []byte
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		body, _ = io.ReadAll(r.Body)
+		_, _ = w.Write([]byte(`{}`))
+	}))
+	defer srv.Close()
+
+	cp := newTestGatewayControlPlane(t, srv)
+	id := identity{
+		ClusterName: "default", InstanceID: "i-1",
+		Capacity: detectCapacity([]string{"GPU-aaa", "GPU-bbb"}),
+	}
+	if err := cp.Register(id); err != nil {
+		t.Fatalf("Register: %v", err)
+	}
+
+	var in ecs.RegisterContainerInstanceInput
+	if err := json.Unmarshal(body, &in); err != nil {
+		t.Fatalf("decode body: %v", err)
+	}
+	var gpu *ecs.Resource
+	for _, r := range in.TotalResources {
+		if r.Name != nil && *r.Name == "GPU" {
+			gpu = r
+		}
+	}
+	if gpu == nil {
+		t.Fatal("no GPU resource in TotalResources")
+	}
+	if *gpu.Type != "STRINGSET" {
+		t.Errorf("GPU resource type = %q, want STRINGSET", *gpu.Type)
+	}
+	got := make([]string, len(gpu.StringSetValue))
+	for i, v := range gpu.StringSetValue {
+		got[i] = *v
+	}
+	if len(got) != 2 || got[0] != "GPU-aaa" || got[1] != "GPU-bbb" {
+		t.Errorf("GPU UUIDs = %v, want [GPU-aaa GPU-bbb]", got)
+	}
+}
+
+// A non-GPU host (no discovered UUIDs) registers with no GPU resource at all.
+func TestGatewayControlPlane_Register_NoGPUOmitsResource(t *testing.T) {
+	var body []byte
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		body, _ = io.ReadAll(r.Body)
+		_, _ = w.Write([]byte(`{}`))
+	}))
+	defer srv.Close()
+
+	cp := newTestGatewayControlPlane(t, srv)
+	id := identity{ClusterName: "default", InstanceID: "i-1", Capacity: detectCapacity(nil)}
+	if err := cp.Register(id); err != nil {
+		t.Fatalf("Register: %v", err)
+	}
+
+	var in ecs.RegisterContainerInstanceInput
+	if err := json.Unmarshal(body, &in); err != nil {
+		t.Fatalf("decode body: %v", err)
+	}
+	for _, r := range in.TotalResources {
+		if r.Name != nil && *r.Name == "GPU" {
+			t.Fatalf("unexpected GPU resource on a non-GPU host: %+v", r)
+		}
+	}
+}
+
+// ReportTaskGPU posts the per-container device UUIDs under the ReportTaskGPU
+// action, decodable as the handlers_ecs internal shape.
+func TestGatewayControlPlane_ReportTaskGPU(t *testing.T) {
+	var gotTarget string
+	var body []byte
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		gotTarget = r.Header.Get("X-Amz-Target")
+		body, _ = io.ReadAll(r.Body)
+		_, _ = w.Write([]byte(`{}`))
+	}))
+	defer srv.Close()
+
+	cp := newTestGatewayControlPlane(t, srv)
+	err := cp.ReportTaskGPU("web", "t-1", []handlers_ecs.ContainerGPUReport{
+		{Name: "trainer", GPUIDs: []string{"GPU-aaa"}},
+	})
+	if err != nil {
+		t.Fatalf("ReportTaskGPU: %v", err)
+	}
+	if gotTarget == "" {
+		t.Fatal("no X-Amz-Target header sent")
+	}
+
+	var in handlers_ecs.ReportTaskGPUInput
+	if err := json.Unmarshal(body, &in); err != nil {
+		t.Fatalf("decode body: %v", err)
+	}
+	if in.Cluster != "web" || in.Task != "t-1" {
+		t.Errorf("cluster/task = %q/%q, want web/t-1", in.Cluster, in.Task)
+	}
+	if len(in.Containers) != 1 || in.Containers[0].Name != "trainer" || len(in.Containers[0].GPUIDs) != 1 {
+		t.Errorf("containers = %+v, want one trainer entry with one UUID", in.Containers)
+	}
+}

cmd/ecs-agent/credendpoint.go

@@ -3,6 +3,7 @@ package main
 import (
 	"context"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"log/slog"
 	"net"
@@ -143,7 +144,7 @@ func (c *credEndpoint) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	}
 	creds, roleARN, err := c.fetch(r.Context(), credID)
 	if err != nil {
-		if err == errUnknownCredID {
+		if errors.Is(err, errUnknownCredID) {
 			http.Error(w, "not found", http.StatusNotFound)
 			return
 		}

cmd/ecs-agent/gpu_discovery.go

@@ -0,0 +1,42 @@
+package main
+
+import (
+	"os/exec"
+	"strings"
+)
+
+// commandRunner abstracts exec.Command so discovery is unit-testable without
+// shelling out. Production uses execCommandRunner; tests inject a fake.
+type commandRunner func(name string, args ...string) ([]byte, error)
+
+// execCommandRunner runs a real host command and returns its stdout.
+func execCommandRunner(name string, args ...string) ([]byte, error) {
+	return exec.Command(name, args...).Output()
+}
+
+// discoverNvidiaGPUs enumerates local NVIDIA GPU device UUIDs via nvidia-smi.
+// A missing binary or any run failure (no GPU host) yields an empty list, not
+// an error — the agent still registers with zero GPU capacity.
+func discoverNvidiaGPUs(run commandRunner) []string {
+	out, err := run("nvidia-smi", "--query-gpu=uuid", "--format=csv,noheader")
+	if err != nil {
+		return nil
+	}
+	return parseNvidiaSMIUUIDs(string(out))
+}
+
+// parseNvidiaSMIUUIDs splits nvidia-smi's newline-delimited UUID output,
+// dropping blank lines. Exported as a pure function so fixture strings can be
+// tested without exec.
+func parseNvidiaSMIUUIDs(out string) []string {
+	lines := strings.Split(strings.TrimSpace(out), "\n")
+	uuids := make([]string, 0, len(lines))
+	for _, line := range lines {
+		line = strings.TrimSpace(line)
+		if line == "" {
+			continue
+		}
+		uuids = append(uuids, line)
+	}
+	return uuids
+}

cmd/ecs-agent/gpu_discovery_test.go

@@ -0,0 +1,71 @@
+package main
+
+import (
+	"errors"
+	"reflect"
+	"testing"
+)
+
+func TestParseNvidiaSMIUUIDs(t *testing.T) {
+	cases := []struct {
+		name string
+		in   string
+		want []string
+	}{
+		{"empty", "", nil},
+		{"whitespace only", "   \n\n  ", nil},
+		{"single", "GPU-11111111-1111-1111-1111-111111111111\n", []string{"GPU-11111111-1111-1111-1111-111111111111"}},
+		{"multi", "GPU-aaa\nGPU-bbb\nGPU-ccc\n", []string{"GPU-aaa", "GPU-bbb", "GPU-ccc"}},
+		{"blank lines between", "GPU-aaa\n\nGPU-bbb\n", []string{"GPU-aaa", "GPU-bbb"}},
+		{"trailing whitespace per line", "GPU-aaa \r\nGPU-bbb\r\n", []string{"GPU-aaa", "GPU-bbb"}},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			got := parseNvidiaSMIUUIDs(c.in)
+			if !reflect.DeepEqual(got, c.want) && (len(got) != 0 || len(c.want) != 0) {
+				t.Errorf("parseNvidiaSMIUUIDs(%q) = %v, want %v", c.in, got, c.want)
+			}
+		})
+	}
+}
+
+func TestDiscoverNvidiaGPUs_AbsentBinaryReturnsEmptyNoError(t *testing.T) {
+	run := func(name string, args ...string) ([]byte, error) {
+		return nil, errors.New(`exec: "nvidia-smi": executable file not found in $PATH`)
+	}
+	got := discoverNvidiaGPUs(run)
+	if len(got) != 0 {
+		t.Errorf("want empty list on absent nvidia-smi, got %v", got)
+	}
+}
+
+func TestDiscoverNvidiaGPUs_ParsesFixtureOutput(t *testing.T) {
+	var gotName string
+	var gotArgs []string
+	run := func(name string, args ...string) ([]byte, error) {
+		gotName = name
+		gotArgs = args
+		return []byte("GPU-aaa\nGPU-bbb\n"), nil
+	}
+	got := discoverNvidiaGPUs(run)
+	want := []string{"GPU-aaa", "GPU-bbb"}
+	if !reflect.DeepEqual(got, want) {
+		t.Errorf("got %v, want %v", got, want)
+	}
+	if gotName != "nvidia-smi" {
+		t.Errorf("command = %q, want nvidia-smi", gotName)
+	}
+	if len(gotArgs) != 2 || gotArgs[0] != "--query-gpu=uuid" || gotArgs[1] != "--format=csv,noheader" {
+		t.Errorf("args = %v, want the uuid/csv,noheader query", gotArgs)
+	}
+}
+
+func TestDiscoverNvidiaGPUs_NoDevicesReturnsEmpty(t *testing.T) {
+	run := func(name string, args ...string) ([]byte, error) {
+		return []byte(""), nil
+	}
+	got := discoverNvidiaGPUs(run)
+	if len(got) != 0 {
+		t.Errorf("want empty list, got %v", got)
+	}
+}

cmd/ecs-agent/gpu_ledger.go

@@ -0,0 +1,62 @@
+package main
+
+import (
+	"fmt"
+	"strings"
+	"sync"
+)
+
+// gpuLedger is the agent's local source of truth for which of the host's
+// discovered GPU UUIDs are free vs pinned to a running task's container. C4
+// reads a task's pinned UUIDs to build the CDI device list for containerd.
+type gpuLedger struct {
+	mu     sync.Mutex
+	free   []string
+	pinned map[string][]string // key ("taskID/containerName") -> uuids
+}
+
+// newGPULedger seeds the ledger with the host's discovered device UUIDs. A nil
+// or empty slice is a valid non-GPU host: every Pin then fails, which callers
+// treat as best-effort (no device, no CDI injection later).
+func newGPULedger(uuids []string) *gpuLedger {
+	free := append([]string(nil), uuids...)
+	return &gpuLedger{free: free, pinned: make(map[string][]string)}
+}
+
+// gpuKey builds the ledger key for one container's pin/release.
+func gpuKey(taskID, containerName string) string {
+	return taskID + "/" + containerName
+}
+
+// Pin reserves n free UUIDs under key, returning them. An error means fewer
+// than n devices were free; the key is left unpinned so a retry (or a
+// best-effort caller) can proceed without a partial reservation.
+func (l *gpuLedger) Pin(key string, n int) ([]string, error) {
+	if n <= 0 {
+		return nil, nil
+	}
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	if len(l.free) < n {
+		return nil, fmt.Errorf("gpu ledger: requested %d GPUs, %d free", n, len(l.free))
+	}
+	uuids := append([]string(nil), l.free[:n]...)
+	l.free = l.free[n:]
+	l.pinned[key] = uuids
+	return uuids, nil
+}
+
+// ReleaseTask frees every UUID pinned under any container key belonging to
+// taskID, called once the task's terminal STOPPED state is reported.
+func (l *gpuLedger) ReleaseTask(taskID string) {
+	prefix := taskID + "/"
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	for key, uuids := range l.pinned {
+		if !strings.HasPrefix(key, prefix) {
+			continue
+		}
+		l.free = append(l.free, uuids...)
+		delete(l.pinned, key)
+	}
+}

cmd/ecs-agent/gpu_ledger_test.go

@@ -0,0 +1,155 @@
+package main
+
+import (
+	"sync"
+	"testing"
+)
+
+func TestGPULedger_PinReturnsDistinctUUIDs(t *testing.T) {
+	l := newGPULedger([]string{"GPU-a", "GPU-b", "GPU-c"})
+	got, err := l.Pin("t-1/web", 2)
+	if err != nil {
+		t.Fatalf("Pin: %v", err)
+	}
+	if len(got) != 2 {
+		t.Fatalf("want 2 UUIDs, got %v", got)
+	}
+	if got[0] == got[1] {
+		t.Errorf("pinned UUIDs must be distinct, got %v", got)
+	}
+	if len(l.free) != 1 {
+		t.Errorf("free = %d, want 1 remaining", len(l.free))
+	}
+}
+
+func TestGPULedger_PinInsufficientFreeErrors(t *testing.T) {
+	l := newGPULedger([]string{"GPU-a"})
+	_, err := l.Pin("t-1/web", 2)
+	if err == nil {
+		t.Fatal("want error when requesting more GPUs than free")
+	}
+	if len(l.free) != 1 {
+		t.Errorf("failed pin must not consume any device, free = %d", len(l.free))
+	}
+}
+
+func TestGPULedger_PinZeroIsNoop(t *testing.T) {
+	l := newGPULedger([]string{"GPU-a"})
+	got, err := l.Pin("t-1/web", 0)
+	if err != nil || got != nil {
+		t.Errorf("Pin(0) = %v, %v; want nil, nil", got, err)
+	}
+}
+
+func TestGPULedger_ReleaseTaskFreesOnlyThatTasksDevices(t *testing.T) {
+	l := newGPULedger([]string{"GPU-a", "GPU-b", "GPU-c", "GPU-d"})
+	if _, err := l.Pin(gpuKey("t-1", "web"), 2); err != nil {
+		t.Fatalf("pin t-1: %v", err)
+	}
+	if _, err := l.Pin(gpuKey("t-2", "worker"), 1); err != nil {
+		t.Fatalf("pin t-2: %v", err)
+	}
+	if len(l.free) != 1 {
+		t.Fatalf("free = %d, want 1 before release", len(l.free))
+	}
+
+	l.ReleaseTask("t-1")
+	if len(l.free) != 3 {
+		t.Errorf("free = %d, want 3 after releasing t-1's two devices", len(l.free))
+	}
+	if _, ok := l.pinned[gpuKey("t-2", "worker")]; !ok {
+		t.Error("t-2's pin should survive releasing t-1")
+	}
+	if _, ok := l.pinned[gpuKey("t-1", "web")]; ok {
+		t.Error("t-1's pin should be gone after release")
+	}
+}
+
+// Multi-device pinning across concurrent task assigns must hand out disjoint
+// UUID sets — no two tasks may ever share a device.
+func TestGPULedger_ConcurrentPinsHandOutDistinctUUIDs(t *testing.T) {
+	const nDevices = 20
+	uuids := make([]string, nDevices)
+	for i := range uuids {
+		uuids[i] = "GPU-" + string(rune('a'+i))
+	}
+	l := newGPULedger(uuids)
+
+	var wg sync.WaitGroup
+	var mu sync.Mutex
+	seen := make(map[string]bool)
+	errs := 0
+	for i := range nDevices {
+		wg.Add(1)
+		go func(n int) {
+			defer wg.Done()
+			key := gpuKey("t-concurrent", string(rune('A'+n)))
+			got, err := l.Pin(key, 1)
+			mu.Lock()
+			defer mu.Unlock()
+			if err != nil {
+				errs++
+				return
+			}
+			for _, u := range got {
+				if seen[u] {
+					t.Errorf("UUID %s handed out to more than one concurrent pin", u)
+				}
+				seen[u] = true
+			}
+		}(i)
+	}
+	wg.Wait()
+	if errs != 0 {
+		t.Errorf("want all %d pins to succeed (exactly enough devices), got %d errors", nDevices, errs)
+	}
+	if len(seen) != nDevices {
+		t.Errorf("distinct UUIDs handed out = %d, want %d", len(seen), nDevices)
+	}
+	if len(l.free) != 0 {
+		t.Errorf("free = %d, want 0 after all devices pinned", len(l.free))
+	}
+}
+
+// Releasing a task's devices concurrently with new pins never double-hands a
+// UUID: a device is only ever free or pinned to exactly one key at a time.
+func TestGPULedger_ConcurrentPinAndReleaseNoDoubleAssignment(t *testing.T) {
+	l := newGPULedger([]string{"GPU-a", "GPU-b"})
+	var wg sync.WaitGroup
+	for round := range 50 {
+		wg.Add(2)
+		taskID := "t-" + string(rune('a'+round%26))
+		go func(id string) {
+			defer wg.Done()
+			_, _ = l.Pin(gpuKey(id, "web"), 1)
+		}(taskID)
+		go func(id string) {
+			defer wg.Done()
+			l.ReleaseTask(id)
+		}(taskID)
+	}
+	wg.Wait()
+	// Invariant: every currently-pinned UUID appears in exactly one pinned entry
+	// and never simultaneously in free.
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	freeSet := make(map[string]bool, len(l.free))
+	for _, u := range l.free {
+		if freeSet[u] {
+			t.Errorf("duplicate UUID %s in free list", u)
+		}
+		freeSet[u] = true
+	}
+	pinnedSet := make(map[string]bool)
+	for _, uuids := range l.pinned {
+		for _, u := range uuids {
+			if pinnedSet[u] {
+				t.Errorf("UUID %s pinned under more than one key", u)
+			}
+			if freeSet[u] {
+				t.Errorf("UUID %s is both free and pinned", u)
+			}
+			pinnedSet[u] = true
+		}
+	}
+}

cmd/ecs-agent/registration_test.go

@@ -7,16 +7,19 @@ import (
 	"testing"
 	"time"
 
+	handlers_ecs "github.com/mulgadc/spinifex/spinifex/handlers/ecs"
 	"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
 )
 
-// fakeCP is a test double for the gateway control-plane. It records registers and
-// task-state reports, and replays a scripted queue of poll responses.
+// fakeCP is a test double for the gateway control-plane. It records registers,
+// task-state reports and GPU reports, and replays a scripted queue of poll
+// responses.
 type fakeCP struct {
 	mu sync.Mutex
 
 	registers    int
 	states       []bus.TaskState
+	gpuReports   []gpuReport
 	pollAcks     [][]string
 	pollStopAcks [][]string
 	pollReplies  [][]bus.Assign
@@ -27,6 +30,13 @@ type fakeCP struct {
 	submitErr   bool
 }
 
+// gpuReport is one recorded ReportTaskGPU call.
+type gpuReport struct {
+	cluster    string
+	task       string
+	containers []handlers_ecs.ContainerGPUReport
+}
+
 var _ controlPlane = (*fakeCP)(nil)
 
 func (f *fakeCP) Register(_ identity) error {
@@ -66,6 +76,25 @@ func (f *fakeCP) PollAssignments(_, _ string, ackAssigns, ackStops []string) ([]
 	return out, stops, nil
 }
 
+func (f *fakeCP) ReportTaskGPU(cluster, task string, containers []handlers_ecs.ContainerGPUReport) error {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.gpuReports = append(f.gpuReports, gpuReport{cluster: cluster, task: task, containers: containers})
+	return nil
+}
+
+func (f *fakeCP) gpuReportsFor(taskID string) []gpuReport {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	var out []gpuReport
+	for _, r := range f.gpuReports {
+		if r.task == taskID {
+			out = append(out, r)
+		}
+	}
+	return out
+}
+
 func (f *fakeCP) registerCount() int {
 	f.mu.Lock()
 	defer f.mu.Unlock()
@@ -102,6 +131,15 @@ func testIdentity() identity {
 	}
 }
 
+// testIdentityWithGPUs is testIdentity seeded with discovered GPU UUIDs, so
+// newAgent's ledger has real devices to pin from.
+func testIdentityWithGPUs(uuids ...string) identity {
+	id := testIdentity()
+	id.Capacity.GPU = len(uuids)
+	id.Capacity.GPUIDs = uuids
+	return id
+}
+
 func TestRegistrar_Register(t *testing.T) {
 	cp := &fakeCP{}
 	if err := newRegistrar(cp, testIdentity()).Register(); err != nil {

cmd/ecs-agent/runtime/containerd_run.go

@@ -6,12 +6,41 @@ import (
 	"sort"
 
 	containerd "github.com/containerd/containerd/v2/client"
+	"github.com/containerd/containerd/v2/pkg/cdi"
 	"github.com/containerd/containerd/v2/pkg/cio"
 	"github.com/containerd/containerd/v2/pkg/namespaces"
 	"github.com/containerd/containerd/v2/pkg/oci"
 	specs "github.com/opencontainers/runtime-spec/specs-go"
 )
 
+// cdiVendorGPU is the NVIDIA CDI vendor/class prefix for whole-GPU devices, per
+// the NVIDIA CDI naming convention (nvidia.com/gpu=<uuid>).
+const cdiVendorGPU = "nvidia.com/gpu"
+
+// cdiDeviceNames maps a container's pinned GPU device UUIDs to CDI device
+// names. Empty input (non-GPU container) returns nil so no CDI opt is added.
+func cdiDeviceNames(uuids []string) []string {
+	if len(uuids) == 0 {
+		return nil
+	}
+	names := make([]string, len(uuids))
+	for i, id := range uuids {
+		names[i] = cdiVendorGPU + "=" + id
+	}
+	return names
+}
+
+// cdiSpecOpts returns the oci.SpecOpts needed to inject gpuIDs (via
+// containerd's CDI machinery) as devices into a container's OCI spec. It
+// returns nil for a non-GPU container so Run adds nothing to the opts list.
+func cdiSpecOpts(gpuIDs []string) []oci.SpecOpts {
+	devices := cdiDeviceNames(gpuIDs)
+	if len(devices) == 0 {
+		return nil
+	}
+	return []oci.SpecOpts{cdi.WithCDIDevices(devices...)}
+}
+
 var _ Runner = (*containerdPuller)(nil)
 
 // Run creates a container from an already-pulled image and starts its task. The
@@ -51,6 +80,7 @@ func (p *containerdPuller) Run(ctx context.Context, id string, spec RunSpec) (st
 	if len(spec.Env) > 0 {
 		specOpts = append(specOpts, oci.WithEnv(envSlice(spec.Env)))
 	}
+	specOpts = append(specOpts, cdiSpecOpts(spec.GPUIDs)...)
 
 	container, err := p.client.NewContainer(ctx, id,
 		containerd.WithNewSnapshot(id+"-snapshot", image),

cmd/ecs-agent/runtime/containerd_run_test.go

@@ -0,0 +1,47 @@
+package runtime
+
+import (
+	"reflect"
+	"testing"
+)
+
+// cdiDeviceNames maps a container's pinned GPU UUIDs to CDI device names
+// (nvidia.com/gpu=<uuid>); Run wires the result into oci.WithCDIDevices when
+// non-empty, so this is the injection logic's unit-testable core.
+func TestCDIDeviceNames(t *testing.T) {
+	cases := []struct {
+		name  string
+		uuids []string
+		want  []string
+	}{
+		{name: "empty", uuids: nil, want: nil},
+		{name: "single", uuids: []string{"GPU-aaaaaaaa-1111-2222-3333-444444444444"},
+			want: []string{"nvidia.com/gpu=GPU-aaaaaaaa-1111-2222-3333-444444444444"}},
+		{name: "multiple preserves order", uuids: []string{"GPU-aaa", "GPU-bbb"},
+			want: []string{"nvidia.com/gpu=GPU-aaa", "nvidia.com/gpu=GPU-bbb"}},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			got := cdiDeviceNames(tc.uuids)
+			if !reflect.DeepEqual(got, tc.want) {
+				t.Errorf("cdiDeviceNames(%v) = %v, want %v", tc.uuids, got, tc.want)
+			}
+		})
+	}
+}
+
+// TestCDISpecOpts_Wiring confirms Run's specOpts list picks up exactly one CDI
+// opt when GPUIDs is non-empty, and none for a non-GPU container — the wiring
+// Run relies on (a live oci.Spec is needed to unit test the opt's effect).
+func TestCDISpecOpts_Wiring(t *testing.T) {
+	if opts := cdiSpecOpts(nil); len(opts) != 0 {
+		t.Errorf("non-GPU container: want no spec opts, got %d", len(opts))
+	}
+	opts := cdiSpecOpts([]string{"GPU-aaa", "GPU-bbb"})
+	if len(opts) != 1 {
+		t.Fatalf("GPU container: want exactly one CDI spec opt, got %d", len(opts))
+	}
+	if opts[0] == nil {
+		t.Error("GPU container: spec opt is nil")
+	}
+}

cmd/ecs-agent/runtime/runtime.go

@@ -48,6 +48,12 @@ type RunSpec struct {
 	// (awsvpc task ENI). Empty keeps the container in the host (VM) netns —
 	// bridge/host mode behaviour.
 	NetnsPath string
+	// GPU is the whole-GPU count requested for this container.
+	GPU int
+	// GPUIDs are the ledger-pinned device UUIDs assigned to this container, one
+	// per GPU requested. Empty for non-GPU containers or if pinning fell short.
+	// The runner injects these as CDI devices into the OCI spec.
+	GPUIDs []string
 }
 
 // RunStatus is a finished container's outcome.

cmd/ecs-agent/taskrun.go

@@ -8,6 +8,7 @@ import (
 	"time"
 
 	ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
+	handlers_ecs "github.com/mulgadc/spinifex/spinifex/handlers/ecs"
 	"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
 )
 
@@ -124,12 +125,15 @@ func (a *Agent) runTask(ctx context.Context, as *bus.Assign) {
 		}
 
 		cid := containerID(as.TaskID, c.Name)
+		gpuIDs := a.pinContainerGPUs(as.TaskID, c.Name, c.GPU)
 		spec := ctrruntime.RunSpec{
 			Image:     c.Image,
 			Command:   c.Command,
 			Env:       withCredEnv(c.Environment, credID),
 			Labels:    taskLabels(as, c.Name),
 			NetnsPath: netnsPath,
+			GPU:       c.GPU,
+			GPUIDs:    gpuIDs,
 		}
 		id, err := a.runner.Run(ctx, cid, spec)
 		if err != nil {
@@ -138,7 +142,9 @@ func (a *Agent) runTask(ctx context.Context, as *bus.Assign) {
 			a.reportTaskState(as, bus.TaskStatusStopped, "container start failed: "+err.Error(), statuses)
 			return
 		}
-		statuses = append(statuses, bus.ContainerStatus{Name: c.Name, Status: bus.TaskStatusRunning, ContainerID: id})
+		statuses = append(statuses, bus.ContainerStatus{
+			Name: c.Name, Status: bus.TaskStatusRunning, ContainerID: id, GPUIDs: gpuIDs,
+		})
 		go a.waitContainer(ctx, as, c.Name, id)
 	}
 
@@ -234,8 +240,26 @@ func withCredEnv(env map[string]string, credID string) map[string]string {
 	return out
 }
 
+// pinContainerGPUs reserves n device UUIDs for a container from the local
+// ledger, before the container is started so the runner can inject them as
+// CDI devices. A short ledger (fewer free devices than requested, e.g.
+// discovery found none) is logged and treated as best-effort: the container
+// still runs without pinned UUIDs (and without GPU access) rather than
+// failing the task.
+func (a *Agent) pinContainerGPUs(taskID, containerName string, n int) []string {
+	if n <= 0 || a.gpu == nil {
+		return nil
+	}
+	uuids, err := a.gpu.Pin(gpuKey(taskID, containerName), n)
+	if err != nil {
+		slog.Warn("ecs-agent: gpu pin short", "task", taskID, "container", containerName, "err", err)
+	}
+	return uuids
+}
+
 // reportTaskState reports a task transition through the gateway's
-// SubmitTaskStateChange action.
+// SubmitTaskStateChange action, then (RUNNING with pinned GPUs) reports the
+// per-container device UUIDs, and (STOPPED) releases them back to the ledger.
 func (a *Agent) reportTaskState(as *bus.Assign, status, reason string, containers []bus.ContainerStatus) {
 	if a.cp == nil {
 		return
@@ -253,6 +277,28 @@ func (a *Agent) reportTaskState(as *bus.Assign, status, reason string, container
 	if err := a.cp.SubmitTaskState(msg); err != nil {
 		slog.Error("ecs-agent: report task-state failed", "task", as.TaskID, "err", err)
 	}
+	a.reportTaskGPUs(as.TaskID, containers)
+	if status == bus.TaskStatusStopped && a.gpu != nil {
+		a.gpu.ReleaseTask(as.TaskID)
+	}
+}
+
+// reportTaskGPUs sends the control plane the pinned device UUIDs for any
+// container in the report that has them, so DescribeTasks can surface gpuIds.
+// A no-op when nothing was pinned (the common non-GPU-task case).
+func (a *Agent) reportTaskGPUs(taskID string, containers []bus.ContainerStatus) {
+	reports := make([]handlers_ecs.ContainerGPUReport, 0, len(containers))
+	for _, c := range containers {
+		if len(c.GPUIDs) > 0 {
+			reports = append(reports, handlers_ecs.ContainerGPUReport{Name: c.Name, GPUIDs: c.GPUIDs})
+		}
+	}
+	if len(reports) == 0 {
+		return
+	}
+	if err := a.cp.ReportTaskGPU(a.id.ClusterName, taskID, reports); err != nil {
+		slog.Error("ecs-agent: report task-gpu failed", "task", taskID, "err", err)
+	}
 }
 
 // containerID composes a containerd-valid container ID from a task + container.

cmd/ecs-agent/taskrun_test.go

@@ -70,6 +70,160 @@ func TestRunTask_PullRunReportsRunning(t *testing.T) {
 	}
 }
 
+// TestRunTask_GPUCarriedToRunSpec covers mulga-11opz (Epic C task C1): the
+// AssignContainer's GPU count reaches the runtime RunSpec unchanged; a
+// container with no GPU request keeps RunSpec.GPU at zero (regression).
+func TestRunTask_GPUCarriedToRunSpec(t *testing.T) {
+	cp := &fakeCP{}
+	rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
+	a := newRunAgent(cp, rt)
+	as := testAssign()
+	as.Containers = append(as.Containers, bus.AssignContainer{
+		Name: "trainer", Image: "registry/trainer:1", GPU: 2,
+	})
+	a.runTask(context.Background(), as)
+
+	if len(rt.Runs) != 2 {
+		t.Fatalf("expected two runs, got %+v", rt.Runs)
+	}
+	if rt.Runs[0].GPU != 0 {
+		t.Errorf("web container: want GPU=0, got %d", rt.Runs[0].GPU)
+	}
+	if rt.Runs[1].GPU != 2 {
+		t.Errorf("trainer container: want GPU=2, got %d", rt.Runs[1].GPU)
+	}
+}
+
+// TestRunTask_GPUPinsAndReportsUUIDs covers C3: a container's GPU request is
+// pinned from the agent's local ledger, carried on the RUNNING report's
+// ContainerStatus, and forwarded to the control plane via ReportTaskGPU.
+func TestRunTask_GPUPinsAndReportsUUIDs(t *testing.T) {
+	cp := &fakeCP{}
+	rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
+	a := newAgent(config{}, testIdentityWithGPUs("GPU-aaa", "GPU-bbb"), cp, rt, rt, nil)
+	as := testAssign()
+	as.Containers = append(as.Containers, bus.AssignContainer{
+		Name: "trainer", Image: "registry/trainer:1", GPU: 1,
+	})
+	a.runTask(context.Background(), as)
+
+	st := cp.taskStates()
+	if len(st) == 0 || st[len(st)-1].LastStatus != bus.TaskStatusRunning {
+		t.Fatalf("want RUNNING final state, got %+v", st)
+	}
+	last := st[len(st)-1]
+	var trainer *bus.ContainerStatus
+	for i := range last.Containers {
+		if last.Containers[i].Name == "trainer" {
+			trainer = &last.Containers[i]
+		}
+	}
+	if trainer == nil || len(trainer.GPUIDs) != 1 {
+		t.Fatalf("trainer container missing pinned GPU UUID: %+v", last.Containers)
+	}
+	if trainer.GPUIDs[0] != "GPU-aaa" && trainer.GPUIDs[0] != "GPU-bbb" {
+		t.Errorf("unexpected pinned UUID %q", trainer.GPUIDs[0])
+	}
+
+	reports := cp.gpuReportsFor(as.TaskID)
+	if len(reports) != 1 {
+		t.Fatalf("want one ReportTaskGPU call, got %d", len(reports))
+	}
+	if reports[0].cluster != as.ClusterName {
+		t.Errorf("report cluster = %q, want %q", reports[0].cluster, as.ClusterName)
+	}
+	if len(reports[0].containers) != 1 || reports[0].containers[0].Name != "trainer" {
+		t.Errorf("report containers = %+v, want one trainer entry", reports[0].containers)
+	}
+}
+
+// TestRunTask_GPUIDsReachRunSpec covers C4: the ledger's pinned UUIDs for a
+// container reach RunSpec.GPUIDs (not just the RunSpec.GPU count), so the
+// runner can inject them as CDI devices; a non-GPU container's RunSpec
+// carries no GPUIDs.
+func TestRunTask_GPUIDsReachRunSpec(t *testing.T) {
+	cp := &fakeCP{}
+	rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
+	a := newAgent(config{}, testIdentityWithGPUs("GPU-aaa", "GPU-bbb"), cp, rt, rt, nil)
+	as := testAssign()
+	as.Containers = append(as.Containers, bus.AssignContainer{
+		Name: "trainer", Image: "registry/trainer:1", GPU: 1,
+	})
+	a.runTask(context.Background(), as)
+
+	if len(rt.Runs) != 2 {
+		t.Fatalf("expected two runs, got %+v", rt.Runs)
+	}
+	if len(rt.Runs[0].GPUIDs) != 0 {
+		t.Errorf("web container: want no GPUIDs, got %+v", rt.Runs[0].GPUIDs)
+	}
+	trainer := rt.Runs[1]
+	if len(trainer.GPUIDs) != 1 {
+		t.Fatalf("trainer container: want one pinned GPUID, got %+v", trainer.GPUIDs)
+	}
+	if trainer.GPUIDs[0] != "GPU-aaa" && trainer.GPUIDs[0] != "GPU-bbb" {
+		t.Errorf("unexpected pinned UUID %q", trainer.GPUIDs[0])
+	}
+}
+
+// A container with no GPU request never triggers a ReportTaskGPU call.
+func TestRunTask_NoGPUSkipsReport(t *testing.T) {
+	cp := &fakeCP{}
+	rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
+	a := newRunAgent(cp, rt)
+	as := testAssign()
+	a.runTask(context.Background(), as)
+
+	if reports := cp.gpuReportsFor(as.TaskID); len(reports) != 0 {
+		t.Errorf("want no GPU report for a non-GPU task, got %+v", reports)
+	}
+}
+
+// A GPU request exceeding the ledger's free devices (discovery found none, or
+// fewer than requested) is best-effort: the container still runs, it just
+// carries no pinned UUIDs and so no CDI devices are injected.
+func TestRunTask_GPUShortLedgerStillRuns(t *testing.T) {
+	cp := &fakeCP{}
+	rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
+	a := newRunAgent(cp, rt) // testIdentity() has no discovered GPU UUIDs
+	as := testAssign()
+	as.Containers = append(as.Containers, bus.AssignContainer{
+		Name: "trainer", Image: "registry/trainer:1", GPU: 2,
+	})
+	a.runTask(context.Background(), as)
+
+	if len(rt.Runs) != 2 {
+		t.Fatalf("expected both containers to run despite the short ledger, got %+v", rt.Runs)
+	}
+	if reports := cp.gpuReportsFor(as.TaskID); len(reports) != 0 {
+		t.Errorf("want no GPU report when nothing was pinned, got %+v", reports)
+	}
+}
+
+// stopTask releases the task's pinned GPU UUIDs back to the ledger so a later
+// task on the same instance can reuse them.
+func TestStopTask_ReleasesGPUsBackToLedger(t *testing.T) {
+	cp := &fakeCP{}
+	rt := &ctrruntime.FakePuller{Containers: []ctrruntime.Container{
+		{ID: "t-001-trainer", Running: true, Labels: map[string]string{
+			"mulga.ecs.taskID": "t-001", "mulga.ecs.containerName": "trainer",
+		}},
+	}}
+	a := newAgent(config{}, testIdentityWithGPUs("GPU-aaa"), cp, rt, rt, nil)
+	if _, err := a.gpu.Pin(gpuKey("t-001", "trainer"), 1); err != nil {
+		t.Fatalf("seed pin: %v", err)
+	}
+	if len(a.gpu.free) != 0 {
+		t.Fatalf("setup: want ledger fully pinned, free = %v", a.gpu.free)
+	}
+
+	a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "bye"})
+
+	if len(a.gpu.free) != 1 {
+		t.Errorf("want the GPU released back to the ledger, free = %v", a.gpu.free)
+	}
+}
+
 // A pull failure reports STOPPED and never starts the container.
 func TestRunTask_PullFailureReportsStopped(t *testing.T) {
 	cp := &fakeCP{}

cmd/eks-gateway-fetch/main.go

@@ -156,9 +156,10 @@ func emitAddonsTSV(out io.Writer, body []byte) error {
 // recoveryDirective mirrors the gateway's RecoveryDirective wire shape; duplicated
 // here to keep this tiny VM binary free of the handler package's dependency graph.
 type recoveryDirective struct {
-	Epoch    int64  `json:"epoch"`
-	Action   string `json:"action"`
-	Snapshot string `json:"snapshot"`
+	Epoch            int64  `json:"epoch"`
+	Action           string `json:"action"`
+	Snapshot         string `json:"snapshot"`
+	SnapshotRequired bool   `json:"snapshotRequired"`
 }
 
 type internalRecoveryResponse struct {
@@ -166,8 +167,10 @@ type internalRecoveryResponse struct {
 }
 
 // emitRecoveryTSV decodes the recovery directive and writes a single
-// epoch\taction\tsnapshot line the on-VM k3s-recovery agent consumes without a
-// JSON parser. An unset action defaults to "none" (steady state).
+// epoch\taction\tsnapshot\tsnapshotRequired line the on-VM k3s-recovery agent
+// consumes without a JSON parser. An unset action defaults to "none" (steady
+// state); snapshotRequired is 1 when the snapshot MUST restore (fresh DR seed
+// aborts boot on fetch failure) else 0.
 func emitRecoveryTSV(out io.Writer, body []byte) error {
 	var resp internalRecoveryResponse
 	if err := json.Unmarshal(body, &resp); err != nil {
@@ -177,7 +180,11 @@ func emitRecoveryTSV(out io.Writer, body []byte) error {
 	if d.Action == "" {
 		d.Action = "none"
 	}
-	if _, err := fmt.Fprintf(out, "%d\t%s\t%s\n", d.Epoch, d.Action, d.Snapshot); err != nil {
+	required := "0"
+	if d.SnapshotRequired {
+		required = "1"
+	}
+	if _, err := fmt.Fprintf(out, "%d\t%s\t%s\t%s\n", d.Epoch, d.Action, d.Snapshot, required); err != nil {
 		return err
 	}
 	return nil

cmd/installer/install/install.go

@@ -539,7 +539,7 @@ func installBootloader(disk string) error {
 	); biosErr != nil {
 		if efiErr != nil {
 			// Both targets failed — the system will not boot.
-			return fmt.Errorf("both bootloader targets failed (EFI: %v; BIOS: %w)", efiErr, biosErr)
+			return fmt.Errorf("both bootloader targets failed (EFI: %w; BIOS: %w)", efiErr, biosErr)
 		}
 		return biosErr
 	}

cmd/spinifex/cmd/admin_eks.go

@@ -0,0 +1,77 @@
+package cmd
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/admin"
+	handlers_eks "github.com/mulgadc/spinifex/spinifex/handlers/eks"
+	"github.com/spf13/cobra"
+)
+
+var adminEksCmd = &cobra.Command{
+	Use:   "eks",
+	Short: "Manage EKS control-plane disaster recovery",
+}
+
+var adminEksRestoreSnapshotCmd = &cobra.Command{
+	Use:   "restore-snapshot",
+	Short: "Rebuild a single-CP cluster's control plane from an etcd snapshot",
+	Long: `restore-snapshot drives the single-CP total-loss DR path: it launches a fresh
+control-plane VM as a cluster-init seed, directs its boot-time recovery agent to
+run k3s server --cluster-reset (restoring the given etcd snapshot, or the latest
+one found in predastore when --snapshot is omitted), and re-points the cluster
+NLB target groups at the new CP.
+
+Only single-CP clusters are supported. An HA cluster has a potentially
+surviving etcd quorum and should be recovered via quorum reformation instead
+of this destructive single-node rebuild.`,
+	Run: runAdminEksRestoreSnapshot,
+}
+
+func init() {
+	adminCmd.AddCommand(adminEksCmd)
+	adminEksCmd.AddCommand(adminEksRestoreSnapshotCmd)
+
+	adminEksRestoreSnapshotCmd.Flags().String("cluster", "", "Cluster name to restore (required)")
+	adminEksRestoreSnapshotCmd.Flags().String("snapshot", "", "Etcd snapshot key to restore; defaults to the latest snapshot found in predastore")
+	adminEksRestoreSnapshotCmd.Flags().String("account", "", "Account ID that owns the cluster; defaults to the bootstrap account")
+	_ = adminEksRestoreSnapshotCmd.MarkFlagRequired("cluster")
+}
+
+func runAdminEksRestoreSnapshot(cmd *cobra.Command, _ []string) {
+	clusterName, _ := cmd.Flags().GetString("cluster")
+	snapshot, _ := cmd.Flags().GetString("snapshot")
+	accountID, _ := cmd.Flags().GetString("account")
+	if accountID == "" {
+		accountID = admin.DefaultAccountID()
+	}
+
+	_, nc, err := loadConfigAndConnect()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		os.Exit(1)
+	}
+	defer nc.Close()
+
+	svc := handlers_eks.NewNATSEKSService(nc)
+	// The server side launches a VM (30-60s) then fences the old CP with retries,
+	// so the RPC needs generous headroom over a bare launch.
+	ctx, cancel := context.WithTimeout(context.Background(), 300*time.Second)
+	defer cancel()
+
+	fmt.Printf("Restoring control plane for cluster %q (account %s)...\n", clusterName, accountID)
+	out, err := svc.RestoreSnapshot(ctx, &handlers_eks.RestoreSnapshotInput{
+		ClusterName: clusterName,
+		Snapshot:    snapshot,
+	}, accountID)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		os.Exit(1)
+	}
+
+	fmt.Printf("New control plane %s launched (snapshot %s)\n", out.NewInstanceID, out.Snapshot)
+	fmt.Printf("Status: %s\n", out.Status)
+}

cmd/spinifex/cmd/admin_nat_test.go

@@ -46,6 +46,55 @@ func TestSpinifexTomlTemplate_NATMode(t *testing.T) {
 	assert.NotContains(t, content, "range_start", "nat mode has no public IP range")
 }
 
+func TestSpinifexTomlTemplate_GwLrpRange(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node: "node1", Az: "ap-southeast-2a", Port: "4432",
+		Region: "ap-southeast-2", BindIP: "10.11.12.1",
+		AccessKey: "AKIATEST", SecretKey: "SECRET", AccountID: "123456789012",
+		NatsToken: "token", ConfigDir: dir,
+		OVNNBAddr: "tcp:127.0.0.1:6641", OVNSBAddr: "tcp:127.0.0.1:6642",
+
+		ExternalMode: "nat", BridgeMode: "nat",
+		Pools: []admin.PoolData{{
+			Name: "nat-transit", Gateway: "100.127.0.1", PrefixLen: 24,
+			GwLrpRangeStart: "100.127.0.16", GwLrpRangeEnd: "100.127.0.254",
+		}},
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	content := string(data)
+	assert.Contains(t, content, `gw_lrp_range_start = "100.127.0.16"`)
+	assert.Contains(t, content, `gw_lrp_range_end   = "100.127.0.254"`)
+}
+
+func TestSpinifexTomlTemplate_GwLrpRangeOmittedWhenUnset(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node: "node1", Az: "ap-southeast-2a", Port: "4432",
+		Region: "ap-southeast-2", BindIP: "10.11.12.1",
+		AccessKey: "AKIATEST", SecretKey: "SECRET", AccountID: "123456789012",
+		NatsToken: "token", ConfigDir: dir,
+		OVNNBAddr: "tcp:127.0.0.1:6641", OVNSBAddr: "tcp:127.0.0.1:6642",
+
+		ExternalMode: "pool", ExternalIface: "enp0s3",
+		Pools: []admin.PoolData{{
+			Name: "wan", Source: "static",
+			Start: "192.168.1.150", End: "192.168.1.250",
+			Gateway: "192.168.1.1", PrefixLen: 24,
+		}},
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	assert.NotContains(t, string(data), "gw_lrp_range", "unset range must not render")
+}
+
 func TestSpinifexTomlTemplate_PoolModeOmitsBridgeMode(t *testing.T) {
 	dir := t.TempDir()
 	path := filepath.Join(dir, "spinifex.toml")
@@ -89,8 +138,8 @@ func TestIsNonBridgeableUplink(t *testing.T) {
 
 func TestBridgeModeFor(t *testing.T) {
 	assert.Equal(t, "nat", bridgeModeFor("nat"))
-	assert.Equal(t, "", bridgeModeFor("pool"))
-	assert.Equal(t, "", bridgeModeFor(""))
+	assert.Empty(t, bridgeModeFor("pool"))
+	assert.Empty(t, bridgeModeFor(""))
 }
 
 func TestSpinifexTomlTemplate_NATModeWithPublicPool(t *testing.T) {

cmd/spinifex/cmd/admin_test.go

@@ -10,7 +10,11 @@ import (
 	"testing"
 
 	"github.com/mulgadc/spinifex/spinifex/admin"
+	"github.com/mulgadc/spinifex/spinifex/config"
 	"github.com/mulgadc/spinifex/spinifex/formation"
+	handlers_dns "github.com/mulgadc/spinifex/spinifex/handlers/dns"
+	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
+	"github.com/mulgadc/spinifex/spinifex/hostdns"
 	"github.com/mulgadc/spinifex/spinifex/utils"
 	toml "github.com/pelletier/go-toml/v2"
 	"github.com/spf13/cobra"
@@ -50,7 +54,7 @@ func TestBuildRemoteNodes_AdvertiseFallback(t *testing.T) {
 		"node2": {Name: "node2", BindIP: "10.0.0.2"}, // legacy joiner
 		"node3": {Name: "node3", BindIP: "10.0.0.3", AdvertiseIP: "203.0.113.3"},
 	}
-	got := buildRemoteNodes(nodes, "node3")
+	got := buildRemoteNodes(nodes, "node3", "")
 	if assert.Len(t, got, 2) {
 		assert.Equal(t, "node1", got[0].Name)
 		assert.Equal(t, "203.0.113.1", got[0].Host, "advertise wins when set")
@@ -59,6 +63,29 @@ func TestBuildRemoteNodes_AdvertiseFallback(t *testing.T) {
 	}
 }
 
+// Every peer must carry the northstar stanza, since a node that describes only
+// its own northstar computes a single-entry seed set and pins the base zone's NS
+// records to whichever node won the create-if-absent race.
+func TestBuildRemoteNodes_NorthstarConfigPath(t *testing.T) {
+	nodes := map[string]formation.NodeInfo{
+		"node1": {Name: "node1", BindIP: "10.0.0.1"},
+		"node2": {Name: "node2", BindIP: "10.0.0.2"},
+	}
+
+	got := buildRemoteNodes(nodes, "node2", "/etc/spinifex/northstar/northstar.toml")
+	if assert.Len(t, got, 1) {
+		assert.Equal(t, "/etc/spinifex/northstar/northstar.toml", got[0].NorthstarConfigPath)
+	}
+
+	// No distributed credentials means no node runs northstar. Publishing peers
+	// as resolvers anyway would point guests at 169.254.169.253 for a service
+	// that never starts.
+	got = buildRemoteNodes(nodes, "node2", "")
+	if assert.Len(t, got, 1) {
+		assert.Empty(t, got[0].NorthstarConfigPath)
+	}
+}
+
 func TestResolveAdvertiseIP(t *testing.T) {
 	wan := &admin.DetectedNetwork{WAN: &admin.DetectedInterface{IP: "192.168.1.21"}}
 	noWAN := &admin.DetectedNetwork{}
@@ -145,6 +172,44 @@ func TestSpinifexTomlTemplate_AdvertiseOmittedWhenEmpty(t *testing.T) {
 	assert.NotContains(t, string(data), "advertise =")
 }
 
+// The northstar stanza carries the non-secret domains so confined services
+// (vpcd) resolve DNS names without reading the 0600 northstar.toml.
+func TestSpinifexTomlTemplate_NorthstarDomains(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node:                    "node1",
+		Az:                      "ap-southeast-2a",
+		Port:                    "4432",
+		Region:                  "ap-southeast-2",
+		BindIP:                  "10.11.12.1",
+		AccessKey:               "AKIATEST",
+		SecretKey:               "SECRET",
+		AccountID:               "123456789012",
+		NatsToken:               "token",
+		ConfigDir:               dir,
+		OVNNBAddr:               "tcp:127.0.0.1:6641",
+		OVNSBAddr:               "tcp:127.0.0.1:6642",
+		NorthstarConfigPath:     "/etc/spinifex/northstar/northstar.toml",
+		NorthstarDefaultDomain:  "spx3.net",
+		NorthstarInternalDomain: "compute.internal",
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	content := string(data)
+	assert.Contains(t, content, `config_path = "/etc/spinifex/northstar/northstar.toml"`)
+	assert.Contains(t, content, `default_domain = "spx3.net"`)
+	assert.Contains(t, content, `internal_domain = "compute.internal"`)
+
+	// The stanza must round-trip into NorthstarConfig so the resolvers read it.
+	cfg, err := config.LoadConfig(path)
+	require.NoError(t, err)
+	assert.Equal(t, "spx3.net", cfg.Nodes["node1"].Northstar.DefaultDomain)
+	assert.Equal(t, "compute.internal", cfg.Nodes["node1"].Northstar.InternalDomain)
+}
+
 // Legacy `wan_bridge` TOML key must fail-start vpcd with guidance, not silently
 // alias (per mulga-998 D3). Prevents the footgun where operators inherited the
 // old key pointing at 'br-ext' and got broken DHCP on veth-mode hosts.
@@ -209,7 +274,7 @@ func TestPredastoreMultinodeTemplate_NatsURLLoopbackShim(t *testing.T) {
 		{ID: 3, Host: "10.0.0.3"},
 	}
 	content, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "0.0.0.0", 0,
+		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "0.0.0.0", 0, admin.NorthstarCredentials{},
 	)
 	require.NoError(t, err)
 	assert.Contains(t, content, `nats_url = "nats://localhost:4222"`)
@@ -217,7 +282,7 @@ func TestPredastoreMultinodeTemplate_NatsURLLoopbackShim(t *testing.T) {
 
 	// Specific bind IP → stays as-is.
 	content2, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "10.11.12.1", 0,
+		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "10.11.12.1", 0, admin.NorthstarCredentials{},
 	)
 	require.NoError(t, err)
 	assert.Contains(t, content2, `nats_url = "nats://10.11.12.1:4222"`)
@@ -255,23 +320,6 @@ func TestWritePredastoreEncryptionKey(t *testing.T) {
 	assert.NotEqual(t, key1, key2, "per-node keys must differ")
 }
 
-// The viperblock key gates at-rest encryption for every volume on a fresh
-// install. masterkey.LoadShared rejects anything that isn't exactly 32 bytes at
-// 0640-or-stricter, so a regression in size/mode would silently fall back to
-// cleartext (empty path) or fail service startup.
-func TestWriteViperblockEncryptionKey(t *testing.T) {
-	configDir := t.TempDir()
-
-	keyPath, err := writeViperblockEncryptionKey(configDir)
-	require.NoError(t, err)
-	assert.Equal(t, filepath.Join(configDir, "viperblock", "encryption.key"), keyPath)
-
-	info, err := os.Stat(keyPath)
-	require.NoError(t, err)
-	assert.Equal(t, int64(32), info.Size(), "viperblock master key must be exactly 32 bytes")
-	assert.Equal(t, os.FileMode(0600), info.Mode().Perm(), "viperblock master key must be mode 0600")
-}
-
 // Joiners persist the leader's shared key via saveViperblockEncryptionKey
 // against a configDir with no viperblock/ subdir yet. The bytes round-trip
 // verbatim — the whole cluster shares one key, so any mutation would orphan
@@ -292,6 +340,181 @@ func TestSaveViperblockEncryptionKey_RoundTrip(t *testing.T) {
 	assert.Equal(t, key, got, "shared key must be written verbatim")
 }
 
+// A re-init must not rotate the per-node predastore key: rotating it would orphan
+// every fragment already sealed under the old key. The helper preserves an
+// existing key byte-for-byte.
+func TestWritePredastoreEncryptionKey_PreservesExisting(t *testing.T) {
+	configDir := t.TempDir()
+
+	keyPath, err := writePredastoreEncryptionKey(configDir)
+	require.NoError(t, err)
+	orig, err := os.ReadFile(keyPath)
+	require.NoError(t, err)
+
+	keyPath2, err := writePredastoreEncryptionKey(configDir)
+	require.NoError(t, err)
+	assert.Equal(t, keyPath, keyPath2)
+	after, err := os.ReadFile(keyPath2)
+	require.NoError(t, err)
+	assert.Equal(t, orig, after, "existing predastore key must be preserved on re-init")
+}
+
+// ensureViperblockEncryptionKey generates a shared 32-byte 0600 key on a fresh
+// dir and returns bytes that match the on-disk file; a second call preserves it
+// so a re-init keeps every encrypted volume readable.
+func TestEnsureViperblockEncryptionKey(t *testing.T) {
+	configDir := t.TempDir()
+
+	key, keyPath, err := ensureViperblockEncryptionKey(configDir)
+	require.NoError(t, err)
+	assert.Equal(t, filepath.Join(configDir, "viperblock", "encryption.key"), keyPath)
+	assert.Len(t, key, 32, "viperblock key must be 32 bytes")
+
+	info, err := os.Stat(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, int64(32), info.Size(), "viperblock master key must be exactly 32 bytes")
+	assert.Equal(t, os.FileMode(0600), info.Mode().Perm(), "viperblock master key must be mode 0600")
+
+	onDisk, err := os.ReadFile(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, key, onDisk, "returned bytes must match the file")
+
+	key2, _, err := ensureViperblockEncryptionKey(configDir)
+	require.NoError(t, err)
+	assert.Equal(t, key, key2, "existing viperblock key must be preserved on re-init")
+}
+
+// A present-but-corrupt viperblock key must fail loud rather than regenerate,
+// which would orphan every already-encrypted volume in the cluster.
+func TestEnsureViperblockEncryptionKey_CorruptFailsLoud(t *testing.T) {
+	configDir := t.TempDir()
+	keyDir := filepath.Join(configDir, "viperblock")
+	require.NoError(t, os.MkdirAll(keyDir, 0750))
+	keyPath := filepath.Join(keyDir, "encryption.key")
+	require.NoError(t, os.WriteFile(keyPath, []byte("bad"), 0600))
+
+	_, _, err := ensureViperblockEncryptionKey(configDir)
+	require.Error(t, err, "a truncated viperblock key must error, not regenerate")
+
+	after, err := os.ReadFile(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, []byte("bad"), after, "corrupt viperblock key must not be overwritten")
+}
+
+// ensureMasterKey generates a fresh in-memory key without touching disk (the
+// bootstrap writer persists it), and preserves an existing on-disk key so a
+// re-init never rotates the key that encrypts IAM secrets in NATS KV.
+func TestEnsureMasterKey(t *testing.T) {
+	t.Run("FreshGeneratesWithoutWriting", func(t *testing.T) {
+		configDir := t.TempDir()
+		key, existed, err := ensureMasterKey(configDir)
+		require.NoError(t, err)
+		assert.False(t, existed, "no master.key present yet")
+		assert.Len(t, key, 32)
+		assert.NoFileExists(t, filepath.Join(configDir, "master.key"),
+			"ensureMasterKey must not persist on the fresh path")
+	})
+
+	t.Run("PreservesExisting", func(t *testing.T) {
+		configDir := t.TempDir()
+		seed := make([]byte, 32)
+		for i := range seed {
+			seed[i] = byte(200 - i)
+		}
+		require.NoError(t, handlers_iam.SaveMasterKey(filepath.Join(configDir, "master.key"), seed))
+
+		key, existed, err := ensureMasterKey(configDir)
+		require.NoError(t, err)
+		assert.True(t, existed, "existing master.key must be detected")
+		assert.Equal(t, seed, key, "existing master key must be loaded verbatim")
+	})
+
+	// A present-but-corrupt master.key must fail loud, never be silently
+	// regenerated: rotating it would orphan every IAM secret in NATS KV.
+	t.Run("CorruptFailsLoud", func(t *testing.T) {
+		configDir := t.TempDir()
+		keyPath := filepath.Join(configDir, "master.key")
+		require.NoError(t, os.WriteFile(keyPath, []byte("too-short"), 0600))
+
+		_, _, err := ensureMasterKey(configDir)
+		require.Error(t, err, "a truncated master.key must error, not regenerate")
+
+		after, err := os.ReadFile(keyPath)
+		require.NoError(t, err)
+		assert.Equal(t, []byte("too-short"), after, "corrupt master key must not be overwritten")
+	})
+}
+
+// The identity bundle is preserved holistically on re-init: the load helpers
+// recover the exact system credentials and never rewrite bootstrap.json. Admin
+// credentials are not recovered from disk — awsgw consumes and deletes
+// bootstrap.json after first boot, so the preserve path must not read it back.
+func TestPreservedIdentityBundle(t *testing.T) {
+	configDir := t.TempDir()
+	bootstrapDir := filepath.Join(configDir, "awsgw")
+
+	masterKey, err := handlers_iam.GenerateMasterKey()
+	require.NoError(t, err)
+	const (
+		sysAccess   = "AKIASYSTEM0000000000"
+		sysSecret   = "system-secret-value"
+		adminAccess = "AKIAADMIN00000000000"
+		adminSecret = "admin-secret-value"
+		accountID   = "123456789012"
+	)
+	require.NoError(t, writeBootstrapFilesWithAdmin(configDir, bootstrapDir, masterKey,
+		sysAccess, sysSecret, accountID, adminAccess, adminSecret))
+	require.NoError(t, writeSystemCredentials(configDir, sysAccess, sysSecret))
+
+	bootstrapPath := filepath.Join(bootstrapDir, "bootstrap.json")
+	bootstrapBefore, err := os.ReadFile(bootstrapPath)
+	require.NoError(t, err)
+
+	// master.key round-trips and is flagged as pre-existing.
+	loadedKey, existed, err := ensureMasterKey(configDir)
+	require.NoError(t, err)
+	assert.True(t, existed)
+	assert.Equal(t, masterKey, loadedKey)
+
+	// System credentials come back verbatim (they must match the NATS KV seed).
+	gotSysAccess, gotSysSecret, err := loadSystemCredentials(configDir)
+	require.NoError(t, err)
+	assert.Equal(t, sysAccess, gotSysAccess)
+	assert.Equal(t, sysSecret, gotSysSecret)
+
+	// The preserve helpers must not rewrite the seed file.
+	bootstrapAfter, err := os.ReadFile(bootstrapPath)
+	require.NoError(t, err)
+	assert.Equal(t, bootstrapBefore, bootstrapAfter, "bootstrap.json must not be rewritten on re-init")
+}
+
+// loadSystemCredentials surfaces a clear error when the file is absent rather
+// than returning empty credentials that would silently break SigV4 auth.
+func TestLoadSystemCredentials_Missing(t *testing.T) {
+	_, _, err := loadSystemCredentials(t.TempDir())
+	require.Error(t, err)
+}
+
+// A present-but-empty or malformed system-credentials.json must error rather
+// than yield blank credentials that would silently break SigV4 between services.
+func TestLoadSystemCredentials_Invalid(t *testing.T) {
+	t.Run("EmptyValues", func(t *testing.T) {
+		configDir := t.TempDir()
+		require.NoError(t, os.WriteFile(filepath.Join(configDir, "system-credentials.json"),
+			[]byte(`{"access_key_id":"","secret_access_key":""}`), 0600))
+		_, _, err := loadSystemCredentials(configDir)
+		require.Error(t, err)
+	})
+
+	t.Run("MalformedJSON", func(t *testing.T) {
+		configDir := t.TempDir()
+		require.NoError(t, os.WriteFile(filepath.Join(configDir, "system-credentials.json"),
+			[]byte("not json"), 0600))
+		_, _, err := loadSystemCredentials(configDir)
+		require.Error(t, err)
+	})
+}
+
 // A fresh install must render encryption_key_file so viperblockd enables
 // at-rest encryption; an empty field (upgrade / legacy) must omit the line
 // entirely so the volume stays in cleartext legacy mode.
@@ -323,6 +546,32 @@ func TestSpinifexTomlTemplate_EncryptionKeyFile(t *testing.T) {
 	assert.NotContains(t, string(data2), "encryption_key_file", "blank key must omit the field")
 }
 
+// A rendered node config must turn chunk GC on. Asserting the template text
+// alone is not enough: the value has to survive parsing into a non-nil true
+// pointer, because every consumer treats nil as off. Rendering and loading
+// together is the only check that covers the whole path.
+func TestSpinifexTomlTemplate_GCEnabled(t *testing.T) {
+	dir := t.TempDir()
+	settings := admin.ConfigSettings{
+		Node: "node1", Az: "ap-southeast-2a", Port: "4432", Region: "ap-southeast-2",
+		BindIP: "0.0.0.0", AccessKey: "AKIATEST", SecretKey: "SECRET",
+		AccountID: "123456789012", NatsToken: "token",
+		OVNNBAddr: "tcp:127.0.0.1:6641", OVNSBAddr: "tcp:127.0.0.1:6642",
+		ConfigDir: dir,
+	}
+
+	path := filepath.Join(dir, "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	cfg, err := config.LoadConfig(path)
+	require.NoError(t, err)
+
+	node, ok := cfg.Nodes["node1"]
+	require.True(t, ok, "rendered config must define the node")
+	require.NotNil(t, node.Viperblock.GCEnabled, "gc_enabled must be rendered; nil leaves chunk GC off on every deployed node")
+	assert.True(t, *node.Viperblock.GCEnabled, "chunk GC must be enabled in a freshly rendered node config")
+}
+
 // Joiners run `spx admin join` against a configDir that doesn't yet contain
 // a predastore/ subdir. The helper must create it (predastore subdir is
 // owned by spinifex-storage in prod, but the helper just needs the dir to
@@ -365,15 +614,15 @@ const cobraRequiredAnnotation = "cobra_annotation_bash_completion_one_required_f
 func TestAdminInitCmd_ExternalDHCPFlagSchema(t *testing.T) {
 	sourceFlag := adminInitCmd.Flags().Lookup("external-source")
 	require.NotNil(t, sourceFlag, "--external-source must be defined")
-	assert.Equal(t, "", sourceFlag.DefValue)
+	assert.Empty(t, sourceFlag.DefValue)
 
 	bindFlag := adminInitCmd.Flags().Lookup("external-bind-bridge")
 	require.NotNil(t, bindFlag, "--external-bind-bridge must be defined")
-	assert.Equal(t, "", bindFlag.DefValue)
+	assert.Empty(t, bindFlag.DefValue)
 
 	poolFlag := adminInitCmd.Flags().Lookup("external-pool")
 	require.NotNil(t, poolFlag)
-	assert.Equal(t, "", poolFlag.DefValue)
+	assert.Empty(t, poolFlag.DefValue)
 }
 
 // DHCP-sourced pool must emit source + bind_bridge into spinifex.toml so the
@@ -472,6 +721,297 @@ func predastoreMultinodeNodes() []admin.PredastoreNodeConfig {
 	}
 }
 
+// predastoreBucketAuth parses the buckets and auth entries a northstar-enabled
+// multi-node predastore must render.
+type predastoreBucketAuth struct {
+	Buckets []struct {
+		Name string `toml:"name"`
+		Type string `toml:"type"`
+	} `toml:"buckets"`
+	Auth []struct {
+		AccessKeyID     string `toml:"access_key_id"`
+		SecretAccessKey string `toml:"secret_access_key"`
+		Policy          []struct {
+			Bucket  string   `toml:"bucket"`
+			Actions []string `toml:"actions"`
+		} `toml:"policy"`
+	} `toml:"auth"`
+}
+
+// policyFor returns the actions the given key is granted on the given bucket,
+// and whether the key holds any policy for it at all.
+func (c predastoreBucketAuth) policyFor(accessKey, bucket string) ([]string, bool) {
+	for _, auth := range c.Auth {
+		if auth.AccessKeyID != accessKey {
+			continue
+		}
+		for _, p := range auth.Policy {
+			if p.Bucket == bucket {
+				return p.Actions, true
+			}
+		}
+	}
+	return nil, false
+}
+
+func (c predastoreBucketAuth) hasBucket(name string) bool {
+	for _, b := range c.Buckets {
+		if b.Name == name {
+			return true
+		}
+	}
+	return false
+}
+
+// Northstar on a multi-node cluster needs three things from predastore, all
+// gated on the same credential: the zone bucket, write access for the system
+// key (which is what BootstrapBaseZone seeds zones with), and the resolver's
+// own read-only key. The template rendered all three correctly, but the caller
+// never populated the credential, so multi-node clusters silently got none of
+// them.
+func TestPredastoreMultinodeTemplate_NorthstarProvisioned(t *testing.T) {
+	content, err := admin.GenerateMultiNodePredastoreConfig(
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0,
+		admin.NorthstarCredentials{AccessKey: "NSAK", SecretKey: "NSSK", Bucket: admin.NorthstarBucketName},
+	)
+	require.NoError(t, err)
+
+	var cfg predastoreBucketAuth
+	require.NoError(t, toml.Unmarshal([]byte(content), &cfg))
+
+	assert.True(t, cfg.hasBucket(admin.NorthstarBucketName), "northstar zone bucket not rendered")
+
+	// The zone bucket must be readable from every node's local predastore, so
+	// a resolver on any node can serve it.
+	for _, b := range cfg.Buckets {
+		if b.Name == admin.NorthstarBucketName {
+			assert.Equal(t, "distributed", b.Type)
+		}
+	}
+
+	// BootstrapBaseZone writes zone files with the system key, not the
+	// resolver's. Without this grant, seeding fails on every node.
+	systemActions, ok := cfg.policyFor("AK", admin.NorthstarBucketName)
+	require.True(t, ok, "system key has no policy for the northstar bucket")
+	assert.Contains(t, systemActions, "s3:GetObject", "bootstrap must check whether the zone already exists")
+	assert.Contains(t, systemActions, "s3:PutObject", "bootstrap must create a missing zone")
+
+	// The resolver's key stays read-only and scoped to the zone bucket alone.
+	nsActions, ok := cfg.policyFor("NSAK", admin.NorthstarBucketName)
+	require.True(t, ok, "northstar key has no policy for the northstar bucket")
+	assert.ElementsMatch(t, []string{"s3:ListBucket", "s3:GetObject"}, nsActions)
+	for _, auth := range cfg.Auth {
+		if auth.AccessKeyID == "NSAK" {
+			assert.Equal(t, "NSSK", auth.SecretAccessKey)
+		}
+	}
+
+	_, ok = cfg.policyFor("NSAK", "predastore")
+	assert.False(t, ok, "northstar key must not reach the predastore bucket")
+}
+
+// Without a credential the northstar stanzas must be omitted wholesale rather
+// than half-rendered: an empty-keyed [[auth]] entry would let anyone read the
+// zone bucket.
+func TestPredastoreMultinodeTemplate_NorthstarOmittedWithoutCredentials(t *testing.T) {
+	content, err := admin.GenerateMultiNodePredastoreConfig(
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0,
+		admin.NorthstarCredentials{},
+	)
+	require.NoError(t, err)
+
+	var cfg predastoreBucketAuth
+	require.NoError(t, toml.Unmarshal([]byte(content), &cfg))
+
+	assert.False(t, cfg.hasBucket(admin.NorthstarBucketName))
+	_, ok := cfg.policyFor("AK", admin.NorthstarBucketName)
+	assert.False(t, ok, "system key granted northstar access without a northstar credential")
+	assert.Len(t, cfg.Auth, 1, "only the system key should be rendered")
+}
+
+// A joiner must provision its predastore with the leader's pair verbatim, since
+// each node's predastore only honours the keys in its own config. Per-node
+// generation would have node A's predastore reject node B's resolver.
+func TestNorthstarFromFormation_UsesDistributedPair(t *testing.T) {
+	dirs := configDirs{Northstar: "/etc/spinifex/northstar"}
+	creds := &formation.SharedCredentials{NorthstarAccessKey: "NSAK", NorthstarSecretKey: "NSSK"}
+
+	got, path := northstarFromFormation(creds, dirs)
+
+	assert.Equal(t, "NSAK", got.AccessKey)
+	assert.Equal(t, "NSSK", got.SecretKey)
+	assert.Equal(t, admin.NorthstarBucketName, got.Bucket, "bucket is a constant, derived node-side")
+	assert.Equal(t, "/etc/spinifex/northstar/northstar.toml", path)
+}
+
+// A leader predating credential distribution, or returning only half a pair,
+// must yield no Northstar config. Advertising a resolver without usable keys
+// would send guests to a service that cannot read its zones.
+func TestNorthstarFromFormation_IncompletePairYieldsNoConfig(t *testing.T) {
+	tests := []struct {
+		name   string
+		access string
+		secret string
+	}{
+		{name: "legacy leader"},
+		{name: "access key only", access: "NSAK"},
+		{name: "secret key only", secret: "NSSK"},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got, path := northstarFromFormation(&formation.SharedCredentials{
+				NorthstarAccessKey: tt.access,
+				NorthstarSecretKey: tt.secret,
+			}, configDirs{Northstar: "/etc/spinifex/northstar"})
+
+			assert.Equal(t, admin.NorthstarCredentials{}, got)
+			assert.Empty(t, path, "config path without credentials advertises a resolver that cannot read its zones")
+		})
+	}
+}
+
+// multinodeNorthstarSettings mirrors what both formation paths hand
+// generateAndWriteConfigs once the distributed pair is wired through.
+func multinodeNorthstarSettings(configPath string) admin.ConfigSettings {
+	s := northstarSettings()
+	s.Node = "node1"
+	s.Az = "ap-southeast-2a"
+	s.Port = "4432"
+	s.AccountID = "123456789012"
+	s.OVNNBAddr = "tcp:127.0.0.1:6641"
+	s.OVNSBAddr = "tcp:127.0.0.1:6642"
+	s.NorthstarConfigPath = configPath
+	return s
+}
+
+// The multi-node formation paths must produce both halves of a working
+// resolver: the northstar.toml the service reads, and the spinifex.toml stanza
+// that makes configuredNorthstarNodeNames count this node. Rendering was
+// previously reachable only from the single-node path.
+func TestGenerateAndWriteConfigs_MultinodeRendersNorthstar(t *testing.T) {
+	dirs, err := createConfigSubdirs(t.TempDir())
+	require.NoError(t, err)
+	nsPath := filepath.Join(dirs.Northstar, "northstar.toml")
+	spinifexTomlPath := filepath.Join(t.TempDir(), "spinifex.toml")
+
+	require.NoError(t, generateAndWriteConfigs(dirs, spinifexTomlPath, multinodeNorthstarSettings(nsPath), true))
+
+	nsContent, err := os.ReadFile(nsPath)
+	require.NoError(t, err)
+	assert.Contains(t, string(nsContent), `access_key = "AKIANORTHSTAR"`)
+
+	// config_path is the sole predicate for "this node runs northstar", so its
+	// presence is what puts the node in the nameserver set and the resolver list.
+	cfg, err := config.LoadConfig(spinifexTomlPath)
+	require.NoError(t, err)
+	assert.Equal(t, nsPath, cfg.Nodes["node1"].Northstar.ConfigPath)
+}
+
+// Incomplete credentials must yield neither file nor stanza. A stanza without
+// a usable northstar.toml would point guests at a resolver that never starts.
+func TestGenerateAndWriteConfigs_NoNorthstarWithoutCompleteCredentials(t *testing.T) {
+	tests := []struct {
+		name   string
+		access string
+		secret string
+	}{
+		{name: "no credentials"},
+		{name: "access key only", access: "AKIANORTHSTAR"},
+		{name: "secret key only", secret: "northstar-secret"},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			dirs, err := createConfigSubdirs(t.TempDir())
+			require.NoError(t, err)
+			spinifexTomlPath := filepath.Join(t.TempDir(), "spinifex.toml")
+
+			settings := multinodeNorthstarSettings(filepath.Join(dirs.Northstar, "northstar.toml"))
+			settings.NorthstarAccessKey = tt.access
+			settings.NorthstarSecretKey = tt.secret
+			require.NoError(t, generateAndWriteConfigs(dirs, spinifexTomlPath, settings, false))
+
+			_, err = os.Stat(filepath.Join(dirs.Northstar, "northstar.toml"))
+			assert.True(t, os.IsNotExist(err), "northstar.toml rendered without complete credentials")
+
+			cfg, err := config.LoadConfig(spinifexTomlPath)
+			require.NoError(t, err)
+			assert.Empty(t, cfg.Nodes["node1"].Northstar.ConfigPath)
+
+			predastoreContent, err := os.ReadFile(filepath.Join(dirs.Predastore, "predastore.toml"))
+			require.NoError(t, err)
+			var predastoreCfg predastoreBucketAuth
+			require.NoError(t, toml.Unmarshal(predastoreContent, &predastoreCfg))
+			assert.False(t, predastoreCfg.hasBucket(admin.NorthstarBucketName))
+			assert.Len(t, predastoreCfg.Auth, 1, "incomplete pair rendered a Northstar auth entry")
+		})
+	}
+}
+
+// renderClusterNode renders one node's spinifex.toml as its own formation path
+// would: its local northstar stanza plus a stanza for every peer.
+func renderClusterNode(t *testing.T, node string, allNodes map[string]formation.NodeInfo) *config.ClusterConfig {
+	t.Helper()
+	const nsPath = "/etc/spinifex/northstar/northstar.toml"
+
+	settings := multinodeNorthstarSettings(nsPath)
+	settings.Node = node
+	settings.BindIP = allNodes[node].BindIP
+	settings.AdvertiseIP = allNodes[node].AdvertiseIP
+	settings.RemoteNodes = buildRemoteNodes(allNodes, node, nsPath)
+
+	path := filepath.Join(t.TempDir(), "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	cfg, err := config.LoadConfig(path)
+	require.NoError(t, err)
+	return cfg
+}
+
+// Every node must derive the same nameserver set from its own rendered config.
+// EnsureBaseZone is create-if-absent and never overwrites, so if each node saw
+// only itself the zone's NS records would be pinned to whichever node won the
+// startup race — permanently, for the life of the zone object. Identical sets
+// make that race benign: whoever writes, writes the same bytes.
+func TestSpinifexTomlTemplate_PeerNorthstarSeedsAreUniform(t *testing.T) {
+	allNodes := map[string]formation.NodeInfo{
+		"node1": {Name: "node1", BindIP: "10.0.0.1", AdvertiseIP: "192.168.1.31"},
+		"node2": {Name: "node2", BindIP: "10.0.0.2", AdvertiseIP: "192.168.1.32"},
+		"node3": {Name: "node3", BindIP: "10.0.0.3", AdvertiseIP: "192.168.1.33"},
+	}
+
+	wantIPs := []string{"192.168.1.31", "192.168.1.32", "192.168.1.33"}
+	wantHosts := []string{"ns1", "ns2", "ns3"}
+	for _, node := range []string{"node1", "node2", "node3"} {
+		cfg := renderClusterNode(t, node, allNodes)
+		seeds := handlers_dns.NameserverSeeds(cfg)
+		require.Len(t, seeds, 3)
+		for i, seed := range seeds {
+			assert.Equal(t, wantHosts[i], seed.Host)
+			assert.Equal(t, wantIPs[i], seed.IP)
+		}
+		assert.Equal(t, wantIPs, handlers_dns.ResolverNameserverIPs(cfg),
+			"%s derived a different nameserver set from its own config", node)
+	}
+}
+
+// A peer's stanza carries config_path alone: it is the only field ever read back
+// from a peer, and the domains would be fiction — the local node cannot know
+// what a peer configured.
+func TestSpinifexTomlTemplate_PeerNorthstarStanzaIsPathOnly(t *testing.T) {
+	allNodes := map[string]formation.NodeInfo{
+		"node1": {Name: "node1", BindIP: "10.0.0.1", AdvertiseIP: "192.168.1.31"},
+		"node2": {Name: "node2", BindIP: "10.0.0.2", AdvertiseIP: "192.168.1.32"},
+	}
+
+	cfg := renderClusterNode(t, "node1", allNodes)
+	peer := cfg.Nodes["node2"].Northstar
+	assert.Equal(t, "/etc/spinifex/northstar/northstar.toml", peer.ConfigPath)
+	assert.Empty(t, peer.DefaultDomain, "peer domains are unknowable from here")
+	assert.Empty(t, peer.InternalDomain)
+}
+
 // Unset knob must leave production config byte-identical: no [compaction] block
 // and the original trailing bytes unchanged (single-node has no trailing
 // newline, multinode keeps one).
@@ -485,7 +1025,7 @@ func TestPredastoreTemplates_UnsetCompactionEmitsNoBlock(t *testing.T) {
 		"unset single-node tail changed: %q", single[len(single)-40:])
 
 	multi, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0,
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0, admin.NorthstarCredentials{},
 	)
 	require.NoError(t, err)
 	assert.NotContains(t, multi, "[compaction]")
@@ -506,7 +1046,7 @@ func TestPredastoreTemplates_SetCompactionEmitsParseableBlock(t *testing.T) {
 	assert.Equal(t, interval, singleCfg.Compaction.IntervalSeconds)
 
 	multi, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", interval,
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", interval, admin.NorthstarCredentials{},
 	)
 	require.NoError(t, err)
 	var multiCfg compactionConfig
@@ -543,3 +1083,169 @@ func TestAdminInitFlag_CompactionIntervalReachesRenderedConfig(t *testing.T) {
 	})
 	assert.Contains(t, out, "interval_seconds = 45")
 }
+
+func TestFilterDNSServersExcludesLocalNorthstar(t *testing.T) {
+	servers := []string{
+		"10.0.0.5",
+		"127.0.0.53",
+		"1.1.1.1",
+		"1.1.1.1",
+		"not-an-ip",
+		"2001:0db8::1",
+		"8.8.8.8",
+		"9.9.9.9",
+		"4.4.4.4",
+	}
+
+	got := filterDNSServers(servers, "10.0.0.5", "2001:db8::1")
+	assert.Equal(t, []string{"1.1.1.1", "8.8.8.8", "9.9.9.9"}, got)
+}
+
+func TestDetectDNSServersContinuesAfterExcludedSource(t *testing.T) {
+	var queries []string
+	sources := dnsDetectionSources{
+		queryResolvectl: func(args ...string) (string, error) {
+			query := strings.Join(args, " ")
+			queries = append(queries, query)
+			switch query {
+			case "dns br-wan":
+				return "Link 2 (br-wan): 10.0.0.5 127.0.0.53\n", nil
+			case "dns":
+				return "Global: 10.0.0.5 10.0.0.5 127.0.0.53 10.20.0.53 8.8.8.8\n", nil
+			default:
+				return "", fmt.Errorf("unexpected query %q", query)
+			}
+		},
+		readResolvConf: func() (string, error) {
+			t.Fatal("resolv.conf read despite usable global resolvers")
+			return "", nil
+		},
+	}
+
+	got := detectDNSServersWithSources("br-wan", []string{"10.0.0.5"}, sources)
+	assert.Equal(t, []string{"10.20.0.53", "8.8.8.8"}, got)
+	assert.Equal(t, []string{"dns br-wan", "dns"}, queries)
+}
+
+func TestDetectDNSServersFallsBackToResolvConf(t *testing.T) {
+	sources := dnsDetectionSources{
+		queryResolvectl: func(_ ...string) (string, error) {
+			return "", errors.New("resolver manager unavailable")
+		},
+		readResolvConf: func() (string, error) {
+			return "nameserver 10.0.0.5\nnameserver 10.20.0.53\n", nil
+		},
+	}
+
+	got := detectDNSServersWithSources("br-wan", []string{"10.0.0.5"}, sources)
+	assert.Equal(t, []string{"10.20.0.53"}, got)
+}
+
+func TestHostDNSParams(t *testing.T) {
+	// northstarEnabled requires both credentials; the zones and IPs mirror what
+	// admin init/join hold in memory after rendering northstar.toml.
+	enabled := admin.ConfigSettings{
+		NorthstarAccessKey:      "AK",
+		NorthstarSecretKey:      "SK",
+		NorthstarDefaultDomain:  "spx3.net",
+		NorthstarInternalDomain: "compute.internal",
+		AdvertiseIP:             "10.0.0.5",
+		BindIP:                  "0.0.0.0",
+	}
+
+	tests := []struct {
+		name       string
+		settings   admin.ConfigSettings
+		skip       bool
+		wantOK     bool
+		wantParams hostdns.Params
+	}{
+		{
+			name:     "enabled selects AdvertiseIP over BindIP",
+			settings: enabled,
+			wantOK:   true,
+			wantParams: hostdns.Params{
+				ResolverIP:     "10.0.0.5",
+				BaseDomain:     "spx3.net",
+				InternalDomain: "compute.internal",
+			},
+		},
+		{
+			name: "falls back to BindIP when AdvertiseIP is unset",
+			settings: func() admin.ConfigSettings {
+				s := enabled
+				s.AdvertiseIP = ""
+				s.BindIP = "10.0.0.9"
+				return s
+			}(),
+			wantOK: true,
+			wantParams: hostdns.Params{
+				ResolverIP:     "10.0.0.9",
+				BaseDomain:     "spx3.net",
+				InternalDomain: "compute.internal",
+			},
+		},
+		{
+			name:     "opt-out skips even when enabled",
+			settings: enabled,
+			skip:     true,
+			wantOK:   false,
+		},
+		{
+			name: "northstar disabled skips",
+			settings: func() admin.ConfigSettings {
+				s := enabled
+				s.NorthstarSecretKey = ""
+				return s
+			}(),
+			wantOK: false,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got, ok := hostDNSParams(tt.settings, tt.skip)
+			assert.Equal(t, tt.wantOK, ok)
+			if tt.wantOK {
+				assert.Equal(t, tt.wantParams, got)
+			}
+		})
+	}
+}
+
+// The image is written into a root volume of exactly the size this returns, so
+// anything less than the image size truncates the image and the guest never
+// finds its root. The old flooring conversion undersized every non-round image.
+func TestAMIVolumeSizeGiB(t *testing.T) {
+	const giB int64 = 1024 * 1024 * 1024
+
+	tests := []struct {
+		name  string
+		bytes int64
+		want  uint64
+	}{
+		// The regression: a real 4.94 GiB mkosi image floored to 4 GiB and the
+		// guest hung with no root partition.
+		{name: "non-round image rounds up", bytes: 5306470400, want: 5},
+		// Exact multiples must not gain a spare GiB — the round sizes are the
+		// common case and were the only ones the old code got right.
+		{name: "exact multiple is unchanged", bytes: 16 * giB, want: 16},
+		{name: "one byte over a multiple rounds up", bytes: 16*giB + 1, want: 17},
+		{name: "one byte under a multiple rounds up", bytes: 16*giB - 1, want: 16},
+		// Sub-GiB images still need a whole GiB to live in.
+		{name: "sub-GiB image gets a full GiB", bytes: 1, want: 1},
+		{name: "empty image", bytes: 0, want: 0},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := amiVolumeSizeGiB(tt.bytes)
+			assert.Equal(t, tt.want, got)
+			// The invariant the guest depends on, stated directly.
+			if tt.bytes > 0 {
+				assert.GreaterOrEqual(t, int64(got)*giB, tt.bytes,
+					"volume must be large enough to hold the image")
+			}
+		})
+	}
+}

cmd/spinifex/cmd/cluster.go

@@ -123,7 +123,7 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
 		phaseStart := time.Now()
 		fmt.Printf("[%s] Sending to %d node(s)...\n", strings.ToUpper(phase), nodeCount)
 
-		acks, err := collectShutdownACKs(nc, topic, reqData, nodeCount, timeout)
+		acks, err := collectShutdownACKs(nc, topic, reqData, nodeCount, "", timeout)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "[%s] Error: %v\n", strings.ToUpper(phase), err)
 			if !force {
@@ -216,36 +216,14 @@ func runNodeDrainLocal(cmd *cobra.Command, args []string) {
 // from the local node, ignoring ACKs from any other node. Returns an error if
 // the local node does not respond within timeout.
 func collectLocalShutdownACK(nc *nats.Conn, topic string, reqData []byte, node string, timeout time.Duration) (daemon.ShutdownACK, error) {
-	inbox := nats.NewInbox()
-	sub, err := nc.SubscribeSync(inbox)
+	acks, err := collectShutdownACKs(nc, topic, reqData, 1, node, timeout)
 	if err != nil {
-		return daemon.ShutdownACK{}, fmt.Errorf("failed to subscribe to inbox: %w", err)
+		return daemon.ShutdownACK{}, err
 	}
-	defer sub.Unsubscribe()
-
-	if err := nc.PublishRequest(topic, inbox, reqData); err != nil {
-		return daemon.ShutdownACK{}, fmt.Errorf("failed to publish request: %w", err)
-	}
-	nc.Flush()
-
-	deadline := time.Now().Add(timeout)
-	for {
-		remaining := time.Until(deadline)
-		if remaining <= 0 {
-			return daemon.ShutdownACK{}, fmt.Errorf("timeout waiting for local node %q ACK", node)
-		}
-		msg, err := sub.NextMsg(remaining)
-		if err != nil {
-			return daemon.ShutdownACK{}, fmt.Errorf("timeout waiting for local node %q ACK", node)
-		}
-		var ack daemon.ShutdownACK
-		if err := json.Unmarshal(msg.Data, &ack); err != nil {
-			continue
-		}
-		if ack.Node == node {
-			return ack, nil
-		}
+	if len(acks) == 0 {
+		return daemon.ShutdownACK{}, fmt.Errorf("timeout waiting for local node %q ACK", node)
 	}
+	return acks[0], nil
 }
 
 // runClusterDrainDHCP asks every vpcd to DHCPRELEASE all external-pool leases
@@ -322,7 +300,9 @@ func drainDHCPLeases(nc *nats.Conn, timeout time.Duration) (released, responders
 }
 
 // collectShutdownACKs publishes a shutdown request and collects ACKs from nodes.
-func collectShutdownACKs(nc *nats.Conn, topic string, reqData []byte, nodeCount int, timeout time.Duration) ([]daemon.ShutdownACK, error) {
+// When nodeFilter is non-empty, only ACKs from that node count toward nodeCount
+// and all others are ignored (used by the single-node local-drain path).
+func collectShutdownACKs(nc *nats.Conn, topic string, reqData []byte, nodeCount int, nodeFilter string, timeout time.Duration) ([]daemon.ShutdownACK, error) {
 	inbox := nats.NewInbox()
 	sub, err := nc.SubscribeSync(inbox)
 	if err != nil {
@@ -350,6 +330,9 @@ func collectShutdownACKs(nc *nats.Conn, topic string, reqData []byte, nodeCount
 		if err := json.Unmarshal(msg.Data, &ack); err != nil {
 			continue
 		}
+		if nodeFilter != "" && ack.Node != nodeFilter {
+			continue
+		}
 		acks = append(acks, ack)
 	}
 	return acks, nil

cmd/spinifex/cmd/get.go

@@ -138,6 +138,12 @@ func formatRoles(resp types.NodeStatusResponse) string {
 	if resp.NATSRole != "" {
 		roles = append(roles, "nats:"+resp.NATSRole)
 	}
+	if resp.OVNNBRole != "" {
+		roles = append(roles, "ovn-nb:"+resp.OVNNBRole)
+	}
+	if resp.OVNSBRole != "" {
+		roles = append(roles, "ovn-sb:"+resp.OVNSBRole)
+	}
 	if resp.PredastoreRole != "" {
 		roles = append(roles, "predastore:"+resp.PredastoreRole)
 	}

cmd/spinifex/cmd/get_test.go

@@ -38,6 +38,26 @@ func TestFormatRoles(t *testing.T) {
 			resp: types.NodeStatusResponse{NATSRole: "leader", PredastoreRole: "leader"},
 			want: "nats:leader,predastore:leader",
 		},
+		{
+			name: "ovn roles only",
+			resp: types.NodeStatusResponse{OVNNBRole: "leader", OVNSBRole: "follower"},
+			want: "ovn-nb:leader,ovn-sb:follower",
+		},
+		{
+			name: "all four roles ordered",
+			resp: types.NodeStatusResponse{
+				NATSRole:       "leader",
+				OVNNBRole:      "leader",
+				OVNSBRole:      "follower",
+				PredastoreRole: "follower",
+			},
+			want: "nats:leader,ovn-nb:leader,ovn-sb:follower,predastore:follower",
+		},
+		{
+			name: "ovn-nb leader with nats, sb absent",
+			resp: types.NodeStatusResponse{NATSRole: "follower", OVNNBRole: "leader"},
+			want: "nats:follower,ovn-nb:leader",
+		},
 	}
 
 	for _, tt := range tests {

cmd/spinifex/cmd/northstar_template_test.go

@@ -0,0 +1,89 @@
+package cmd
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/admin"
+	toml "github.com/pelletier/go-toml/v2"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func renderTemplate(t *testing.T, tmpl string, settings admin.ConfigSettings) string {
+	t.Helper()
+	path := filepath.Join(t.TempDir(), "out.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, tmpl, settings))
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	var parsed map[string]any
+	require.NoError(t, toml.Unmarshal(data, &parsed), "rendered template must be valid TOML")
+	return string(data)
+}
+
+func northstarSettings() admin.ConfigSettings {
+	return admin.ConfigSettings{
+		Region:                  "ap-southeast-2",
+		BindIP:                  "0.0.0.0",
+		AdvertiseIP:             "192.168.1.31",
+		AccessKey:               "AKIASYSTEM",
+		SecretKey:               "SYSTEMSECRET",
+		NatsToken:               "NATSTOKEN",
+		PoolDNSServers:          []string{"1.1.1.1", "8.8.8.8"},
+		NorthstarAccessKey:      "AKIANORTHSTAR",
+		NorthstarSecretKey:      "NORTHSTARSECRET",
+		NorthstarBucket:         admin.NorthstarBucketName,
+		NorthstarDefaultDomain:  admin.NorthstarDefaultDomain,
+		NorthstarInternalDomain: admin.NorthstarInternalDomain,
+	}
+}
+
+func TestNorthstarTomlTemplate(t *testing.T) {
+	content := renderTemplate(t, northstarTomlTemplate, northstarSettings())
+
+	// Northstar binds :5300 on the wildcard and the privileged :53 on the node
+	// WAN IP (avoids the systemd-resolved stub on 127.0.0.53:53).
+	assert.Contains(t, content, `listen     = "0.0.0.0:5300,192.168.1.31:53"`)
+	assert.Contains(t, content, `default_domain = "spx3.net"`)
+	assert.Contains(t, content, `bucket     = "northstar"`)
+	assert.Contains(t, content, `access_key = "AKIANORTHSTAR"`)
+	assert.Contains(t, content, `secret_key = "NORTHSTARSECRET"`)
+	// 0.0.0.0 bind must resolve to a loopback S3 endpoint, not a dial to 0.0.0.0.
+	assert.Contains(t, content, `endpoint   = "https://127.0.0.1:8443"`)
+	assert.Contains(t, content, `nameservers = ["1.1.1.1:53", "8.8.8.8:53"]`)
+	assert.Contains(t, content, `internal_domain = "compute.internal"`)
+	var parsed struct {
+		Quotas struct {
+			Enabled              bool `toml:"enabled"`
+			RecordsPerHostedZone int  `toml:"records_per_hosted_zone"`
+		} `toml:"quotas"`
+	}
+	require.NoError(t, toml.Unmarshal([]byte(content), &parsed))
+	assert.False(t, parsed.Quotas.Enabled)
+	assert.Equal(t, 10_000, parsed.Quotas.RecordsPerHostedZone)
+	assert.NotContains(t, content, "nats_url")
+	assert.NotContains(t, content, "NATSTOKEN")
+}
+
+// When provisioned, predastore.toml gains the northstar bucket plus a scoped
+// read-only [[auth]] entry — the isolation guarantee.
+func TestPredastoreTemplateWithNorthstar(t *testing.T) {
+	content := renderTemplate(t, predastoreTomlTemplate, northstarSettings())
+
+	assert.Contains(t, content, `name = "northstar"`)
+	assert.Contains(t, content, `access_key_id = "AKIANORTHSTAR"`)
+	assert.Contains(t, content, `{ bucket = "northstar", actions = ["s3:ListBucket", "s3:GetObject"] }`)
+}
+
+// Without northstar creds (e.g. cluster-join paths), predastore.toml renders
+// exactly as before — no empty bucket/auth stanzas.
+func TestPredastoreTemplateWithoutNorthstar(t *testing.T) {
+	s := northstarSettings()
+	s.NorthstarAccessKey = ""
+	s.NorthstarSecretKey = ""
+	content := renderTemplate(t, predastoreTomlTemplate, s)
+
+	assert.NotContains(t, content, `name = "northstar"`)
+	assert.NotContains(t, content, "AKIANORTHSTAR")
+}

cmd/spinifex/cmd/paths.go

@@ -8,7 +8,7 @@ import (
 
 // DefaultConfigDir returns the default configuration directory.
 // Production: /etc/spinifex (when /etc/spinifex exists)
-// Development: ~/spinifex/config
+// Development: ~/spinifex/config.
 func DefaultConfigDir() string {
 	if isProductionLayout() {
 		return "/etc/spinifex"
@@ -18,7 +18,7 @@ func DefaultConfigDir() string {
 
 // DefaultDataDir returns the default data directory.
 // Production: /var/lib/spinifex
-// Development: ~/spinifex
+// Development: ~/spinifex.
 func DefaultDataDir() string {
 	if isProductionLayout() {
 		return "/var/lib/spinifex"
@@ -28,7 +28,7 @@ func DefaultDataDir() string {
 
 // LogDirFor returns the log directory for a given data directory.
 // Production: /var/log/spinifex (matches systemd ReadWritePaths)
-// Development: <dataDir>/logs (supports custom per-node data dirs)
+// Development: <dataDir>/logs (supports custom per-node data dirs).
 func LogDirFor(dataDir string) string {
 	if isProductionLayout() {
 		return "/var/log/spinifex"

cmd/spinifex/cmd/root.go

@@ -15,7 +15,7 @@ var (
 	appConfig *config.ClusterConfig
 )
 
-// rootCmd represents the base command when called without any subcommands
+// rootCmd represents the base command when called without any subcommands.
 var rootCmd = &cobra.Command{
 	Use:   "spx",
 	Short: "Spinifex - Open source AWS-compatible platform for secure edge deployments",

cmd/spinifex/cmd/service.go

@@ -8,11 +8,13 @@ import (
 	"strings"
 
 	"github.com/mulgadc/spinifex/spinifex/config"
+	handlers_dns "github.com/mulgadc/spinifex/spinifex/handlers/dns"
 	"github.com/mulgadc/spinifex/spinifex/network/external"
 	"github.com/mulgadc/spinifex/spinifex/otelsetup"
 	"github.com/mulgadc/spinifex/spinifex/service"
 	"github.com/mulgadc/spinifex/spinifex/services/awsgw"
 	"github.com/mulgadc/spinifex/spinifex/services/nats"
+	"github.com/mulgadc/spinifex/spinifex/services/northstar"
 	"github.com/mulgadc/spinifex/spinifex/services/predastore"
 	"github.com/mulgadc/spinifex/spinifex/services/qmpcollector"
 	"github.com/mulgadc/spinifex/spinifex/services/spinifexui"
@@ -201,7 +203,7 @@ var predastoreStatusCmd = &cobra.Command{
 	},
 }
 
-// Repeat for viperblock
+// Repeat for viperblock.
 var viperblockStartCmd = &cobra.Command{
 	Use:   "start",
 	Short: "Start the viperblock service",
@@ -295,6 +297,14 @@ var viperblockStartCmd = &cobra.Command{
 			shardWAL = *nodeConfig.Viperblock.ShardWAL
 		}
 
+		// Resolve chunk GC setting: default false unless explicitly set to true.
+		// GC is not a per-volume toggle — flipping it on applies to every VB this
+		// node constructs (nbdkit plugin + volume service).
+		gcEnabled := false
+		if nodeConfig.Viperblock.GCEnabled != nil {
+			gcEnabled = *nodeConfig.Viperblock.GCEnabled
+		}
+
 		encryptionKeyFile := nodeConfig.Viperblock.EncryptionKeyFile
 		if envKey := viper.GetString("viperblock-encryption-key-file"); envKey != "" {
 			encryptionKeyFile = envKey
@@ -315,6 +325,7 @@ var viperblockStartCmd = &cobra.Command{
 			BaseDir:           nodeConfig.Predastore.BaseDir,
 			NodeName:          clusterConfig.Node,
 			ShardWAL:          shardWAL,
+			GCEnabled:         gcEnabled,
 			EncryptionKeyFile: encryptionKeyFile,
 		})
 
@@ -365,7 +376,7 @@ var viperblockStatusCmd = &cobra.Command{
 	},
 }
 
-// Repeat for nats
+// Repeat for nats.
 var natsStartCmd = &cobra.Command{
 	Use:   "start",
 	Short: "Start the nats service",
@@ -434,7 +445,7 @@ var natsStatusCmd = &cobra.Command{
 	},
 }
 
-// Repeat for spinifex
+// Repeat for spinifex.
 var spinifexStartCmd = &cobra.Command{
 	Use:   "start",
 	Short: "Start the spinifex service",
@@ -793,20 +804,23 @@ var vpcdStartCmd = &cobra.Command{
 		defer initTelemetry("vpcd", false)()
 
 		svc, err := service.New("vpcd", &vpcd.Config{
-			NatsHost:          nodeConfig.NATS.Host,
-			NatsToken:         nodeConfig.NATS.ACL.Token,
-			NatsCACert:        nodeConfig.NATS.CACert,
-			OVNNBAddr:         nodeConfig.VPCD.OVNNBAddr,
-			OVNSBAddr:         nodeConfig.VPCD.OVNSBAddr,
-			BaseDir:           nodeConfig.BaseDir,
-			Debug:             false,
-			ExternalMode:      clusterConfig.Network.ExternalMode,
-			ExternalPools:     extPools,
-			Bootstrap:         bootstrap,
-			ExternalInterface: nodeConfig.VPCD.ExternalInterface,
-			BridgeMode:        nodeConfig.VPCD.BridgeMode,
-			AZ:                nodeConfig.AZ,
-			NATExemptCIDRs:    clusterConfig.Network.NATExemptCIDRs,
+			NatsHost:                nodeConfig.NATS.Host,
+			NatsToken:               nodeConfig.NATS.ACL.Token,
+			NatsCACert:              nodeConfig.NATS.CACert,
+			OVNNBAddr:               nodeConfig.VPCD.OVNNBAddr,
+			OVNSBAddr:               nodeConfig.VPCD.OVNSBAddr,
+			BaseDir:                 nodeConfig.BaseDir,
+			Debug:                   false,
+			ExternalMode:            clusterConfig.Network.ExternalMode,
+			ExternalPools:           extPools,
+			Bootstrap:               bootstrap,
+			ExternalInterface:       nodeConfig.VPCD.ExternalInterface,
+			BridgeMode:              nodeConfig.VPCD.BridgeMode,
+			AZ:                      nodeConfig.AZ,
+			NorthstarBaseDomain:     handlers_dns.ResolveBaseDomain(&nodeConfig),
+			NorthstarInternalDomain: handlers_dns.ResolveInternalDomain(&nodeConfig),
+			ResolverNameservers:     handlers_dns.ResolverNameserverIPs(clusterConfig),
+			NATExemptCIDRs:          clusterConfig.Network.NATExemptCIDRs,
 		})
 		if err != nil {
 			fmt.Println("Error starting vpcd service:", err)
@@ -849,6 +863,157 @@ var vpcdStatusCmd = &cobra.Command{
 	},
 }
 
+var northstarCmd = &cobra.Command{
+	Use:   "northstar",
+	Short: "Manage the northstar (DNS) service",
+}
+
+// northstarStartOptions contains the command inputs that select Northstar configuration.
+type northstarStartOptions struct {
+	configFile      string
+	configOverride  string
+	baseDirOverride string
+}
+
+// northstarStarter is the service operation required by Northstar activation.
+type northstarStarter interface {
+	Start() (int, error)
+}
+
+// northstarStartDependencies isolates startup side effects for behavioral tests.
+type northstarStartDependencies struct {
+	loadConfig        func(string) (*config.ClusterConfig, error)
+	bootstrapBaseZone func(string, *config.ClusterConfig) error
+	newService        func(*northstar.Config) (northstarStarter, error)
+}
+
+var northstarStartCmd = &cobra.Command{
+	Use:           "start",
+	Short:         "Start the northstar service",
+	SilenceErrors: true,
+	SilenceUsage:  true,
+	RunE: func(cmd *cobra.Command, args []string) error {
+		fmt.Println("Starting northstar service...")
+
+		defer initTelemetry("northstar", false)()
+
+		return runNorthstarStart(northstarStartOptions{
+			configFile:      viper.GetString("config"),
+			configOverride:  viper.GetString("northstar-config"),
+			baseDirOverride: viper.GetString("base-dir"),
+		}, northstarStartDependencies{
+			loadConfig:        loadRequiredClusterConfig,
+			bootstrapBaseZone: northstar.BootstrapBaseZone,
+			newService: func(cfg *northstar.Config) (northstarStarter, error) {
+				return service.New("northstar", cfg)
+			},
+		})
+	},
+}
+
+// loadRequiredClusterConfig loads the service's explicit cluster configuration.
+func loadRequiredClusterConfig(path string) (*config.ClusterConfig, error) {
+	if path == "" {
+		return nil, fmt.Errorf("config file is not set")
+	}
+	if _, err := os.Stat(path); err != nil {
+		return nil, fmt.Errorf("stat cluster config %s: %w", path, err)
+	}
+	clusterConfig, err := config.LoadConfig(path)
+	if err != nil {
+		return nil, fmt.Errorf("load cluster config %s: %w", path, err)
+	}
+	return clusterConfig, nil
+}
+
+// runNorthstarStart starts configured Northstar nodes and cleanly skips nodes where it is optional.
+func runNorthstarStart(options northstarStartOptions, deps northstarStartDependencies) error {
+	clusterConfig, err := deps.loadConfig(options.configFile)
+	if err != nil {
+		return err
+	}
+
+	nodeConfig, ok := clusterConfig.Nodes[clusterConfig.Node]
+	if !ok {
+		return fmt.Errorf("node %q not found in cluster config", clusterConfig.Node)
+	}
+
+	configPath := resolveNorthstarConfigPath(nodeConfig.Northstar.ConfigPath, options.configOverride)
+	if options.configOverride != "" {
+		fmt.Println("Overwriting northstar config path to:", options.configOverride)
+	}
+	if configPath == "" {
+		// An absent path disables this optional service and must not trigger systemd restarts.
+		slog.Info("northstar is not configured on this node; skipping service start")
+		return nil
+	}
+
+	baseDir := nodeConfig.BaseDir
+	if options.baseDirOverride != "" {
+		baseDir = options.baseDirOverride
+	}
+
+	// Seed the default_domain base zone before the read-only daemon starts.
+	// The S3 polling path remains the backstop when this best-effort seed fails.
+	if err := deps.bootstrapBaseZone(configPath, clusterConfig); err != nil {
+		slog.Warn("northstar base zone bootstrap failed (continuing)", "error", err)
+	}
+
+	svc, err := deps.newService(&northstar.Config{
+		ConfigPath: configPath,
+		BasePath:   baseDir,
+		NodeID:     nodeConfig.Predastore.NodeID,
+		NatsHost:   nodeConfig.NATS.Host,
+		NatsToken:  nodeConfig.NATS.ACL.Token,
+		NatsCACert: nodeConfig.NATS.CACert,
+	})
+	if err != nil {
+		return fmt.Errorf("create northstar service: %w", err)
+	}
+
+	if _, err := svc.Start(); err != nil {
+		return fmt.Errorf("start northstar service: %w", err)
+	}
+	fmt.Println("northstar service started")
+	return nil
+}
+
+// resolveNorthstarConfigPath applies the explicit override before the node config.
+func resolveNorthstarConfigPath(nodePath, override string) string {
+	if override != "" {
+		return override
+	}
+	return nodePath
+}
+
+var northstarStopCmd = &cobra.Command{
+	Use:   "stop",
+	Short: "Stop the northstar service",
+	Run: func(cmd *cobra.Command, args []string) {
+		fmt.Println("Stopping northstar service...")
+
+		svc, err := service.New("northstar", &northstar.Config{})
+		if err != nil {
+			fmt.Println("Error stopping northstar service:", err)
+			return
+		}
+
+		if err = svc.Stop(); err != nil {
+			fmt.Println("Error stopping northstar service:", err)
+			os.Exit(1)
+		}
+		fmt.Println("northstar service stopped")
+	},
+}
+
+var northstarStatusCmd = &cobra.Command{
+	Use:   "status",
+	Short: "Get status of the northstar service",
+	Run: func(cmd *cobra.Command, args []string) {
+		fmt.Println("northstar service status: ...")
+	},
+}
+
 var qmpCollectorCmd = &cobra.Command{
 	Use:   "qmp-collector",
 	Short: "Manage the qmp-collector (guest metrics) service",
@@ -970,11 +1135,6 @@ func init() {
 	viper.BindEnv("encryption-key-file", "SPINIFEX_PREDASTORE_ENCRYPTION_KEY_FILE")
 	viper.BindPFlag("encryption-key-file", predastoreCmd.PersistentFlags().Lookup("encryption-key-file"))
 
-	// Predastore Backend
-	predastoreCmd.PersistentFlags().String("backend", "distributed", "Predastore (S3) backend")
-	viper.BindEnv("backend", "SPINIFEX_PREDASTORE_BACKEND")
-	viper.BindPFlag("backend", predastoreCmd.PersistentFlags().Lookup("backend"))
-
 	// Predastore Node ID. Default -1 is dev mode (launch every configured
 	// QUIC node in-process). Production deployments set this to the node's
 	// real ID (>= 1) via SPINIFEX_PREDASTORE_NODE_ID or --node-id.
@@ -1120,6 +1280,17 @@ func init() {
 	vpcdCmd.AddCommand(vpcdStopCmd)
 	vpcdCmd.AddCommand(vpcdStatusCmd)
 
+	// northstar
+	serviceCmd.AddCommand(northstarCmd)
+
+	northstarCmd.PersistentFlags().String("northstar-config", "", "Path to northstar.toml (overrides config file)")
+	viper.BindEnv("northstar-config", "SPINIFEX_NORTHSTAR_CONFIG")
+	viper.BindPFlag("northstar-config", northstarCmd.PersistentFlags().Lookup("northstar-config"))
+
+	northstarCmd.AddCommand(northstarStartCmd)
+	northstarCmd.AddCommand(northstarStopCmd)
+	northstarCmd.AddCommand(northstarStatusCmd)
+
 	// qmp-collector
 	serviceCmd.AddCommand(qmpCollectorCmd)
 

cmd/spinifex/cmd/service_test.go

@@ -0,0 +1,207 @@
+package cmd
+
+import (
+	"errors"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/config"
+	"github.com/mulgadc/spinifex/spinifex/services/northstar"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+type stubNorthstarService struct {
+	starts   int
+	startErr error
+}
+
+func (s *stubNorthstarService) Start() (int, error) {
+	s.starts++
+	return 0, s.startErr
+}
+
+func TestRunNorthstarStartActivation(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name           string
+		nodePath       string
+		override       string
+		wantPath       string
+		wantConfigured bool
+	}{
+		{
+			name: "unconfigured node is skipped",
+		},
+		{
+			name:           "node config starts service",
+			nodePath:       "/etc/spinifex/northstar/northstar.toml",
+			wantPath:       "/etc/spinifex/northstar/northstar.toml",
+			wantConfigured: true,
+		},
+		{
+			name:           "override starts service",
+			override:       "/run/spinifex/northstar.toml",
+			wantPath:       "/run/spinifex/northstar.toml",
+			wantConfigured: true,
+		},
+		{
+			name:           "override takes precedence",
+			nodePath:       "/etc/spinifex/northstar/northstar.toml",
+			override:       "/run/spinifex/northstar.toml",
+			wantPath:       "/run/spinifex/northstar.toml",
+			wantConfigured: true,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			clusterConfig := &config.ClusterConfig{
+				Node: "node1",
+				Nodes: map[string]config.Config{
+					"node1": {
+						BaseDir:   "/var/lib/spinifex/northstar",
+						Northstar: config.NorthstarConfig{ConfigPath: tt.nodePath},
+					},
+				},
+			}
+			stub := &stubNorthstarService{}
+			bootstrapCalls := 0
+			factoryCalls := 0
+			var startedConfig *northstar.Config
+			deps := northstarStartDependencies{
+				loadConfig: func(string) (*config.ClusterConfig, error) {
+					return clusterConfig, nil
+				},
+				bootstrapBaseZone: func(path string, cfg *config.ClusterConfig) error {
+					bootstrapCalls++
+					assert.Equal(t, tt.wantPath, path)
+					assert.Same(t, clusterConfig, cfg)
+					return nil
+				},
+				newService: func(cfg *northstar.Config) (northstarStarter, error) {
+					factoryCalls++
+					startedConfig = cfg
+					return stub, nil
+				},
+			}
+
+			err := runNorthstarStart(northstarStartOptions{
+				configFile:     "/etc/spinifex/spinifex.toml",
+				configOverride: tt.override,
+			}, deps)
+			require.NoError(t, err)
+
+			if !tt.wantConfigured {
+				assert.Zero(t, bootstrapCalls)
+				assert.Zero(t, factoryCalls)
+				assert.Zero(t, stub.starts)
+				assert.Nil(t, startedConfig)
+				return
+			}
+			assert.Equal(t, 1, bootstrapCalls)
+			assert.Equal(t, 1, factoryCalls)
+			assert.Equal(t, 1, stub.starts)
+			require.NotNil(t, startedConfig)
+			assert.Equal(t, tt.wantPath, startedConfig.ConfigPath)
+		})
+	}
+}
+
+func TestLoadRequiredClusterConfigRejectsMissingFile(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name string
+		path string
+	}{
+		{name: "empty path"},
+		{name: "missing path", path: t.TempDir() + "/missing.toml"},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+			_, err := loadRequiredClusterConfig(tt.path)
+			require.Error(t, err)
+		})
+	}
+}
+
+func TestRunNorthstarStartReturnsLoadFailure(t *testing.T) {
+	t.Parallel()
+
+	loadErr := errors.New("cluster config unavailable")
+	deps := northstarStartDependencies{
+		loadConfig: func(string) (*config.ClusterConfig, error) {
+			return nil, loadErr
+		},
+	}
+
+	err := runNorthstarStart(northstarStartOptions{configFile: "/etc/spinifex/spinifex.toml"}, deps)
+	require.ErrorIs(t, err, loadErr)
+}
+
+func TestRunNorthstarStartContinuesAfterBootstrapFailure(t *testing.T) {
+	t.Parallel()
+
+	bootstrapErr := errors.New("predastore unavailable")
+	stub := &stubNorthstarService{}
+	deps := northstarStartDependencies{
+		loadConfig: func(string) (*config.ClusterConfig, error) {
+			return &config.ClusterConfig{
+				Node: "node1",
+				Nodes: map[string]config.Config{
+					"node1": {Northstar: config.NorthstarConfig{ConfigPath: "/etc/spinifex/northstar/northstar.toml"}},
+				},
+			}, nil
+		},
+		bootstrapBaseZone: func(string, *config.ClusterConfig) error { return bootstrapErr },
+		newService: func(*northstar.Config) (northstarStarter, error) {
+			return stub, nil
+		},
+	}
+
+	err := runNorthstarStart(northstarStartOptions{configFile: "/etc/spinifex/spinifex.toml"}, deps)
+	require.NoError(t, err)
+	assert.Equal(t, 1, stub.starts)
+}
+
+func TestRunNorthstarStartRejectsMissingNode(t *testing.T) {
+	t.Parallel()
+
+	deps := northstarStartDependencies{
+		loadConfig: func(string) (*config.ClusterConfig, error) {
+			return &config.ClusterConfig{Node: "missing", Nodes: map[string]config.Config{}}, nil
+		},
+	}
+
+	err := runNorthstarStart(northstarStartOptions{configFile: "/etc/spinifex/spinifex.toml"}, deps)
+	require.EqualError(t, err, `node "missing" not found in cluster config`)
+}
+
+func TestRunNorthstarStartReturnsConfiguredFailure(t *testing.T) {
+	t.Parallel()
+
+	startErr := errors.New("configured startup failed")
+	stub := &stubNorthstarService{startErr: startErr}
+	deps := northstarStartDependencies{
+		loadConfig: func(string) (*config.ClusterConfig, error) {
+			return &config.ClusterConfig{
+				Node: "node1",
+				Nodes: map[string]config.Config{
+					"node1": {Northstar: config.NorthstarConfig{ConfigPath: "/etc/spinifex/northstar/northstar.toml"}},
+				},
+			}, nil
+		},
+		bootstrapBaseZone: func(string, *config.ClusterConfig) error { return nil },
+		newService: func(*northstar.Config) (northstarStarter, error) {
+			return stub, nil
+		},
+	}
+
+	err := runNorthstarStart(northstarStartOptions{configFile: "/etc/spinifex/spinifex.toml"}, deps)
+	require.ErrorIs(t, err, startErr)
+	assert.Equal(t, 1, stub.starts)
+}

cmd/spinifex/cmd/templates/northstar.toml

@@ -0,0 +1,40 @@
+# Northstar DNS service configuration (TOML).
+# Written by `spx admin init`; read by the spinifex northstar service.
+# Contains bucket-scoped S3 credentials — keep mode 0600.
+
+# UDP+TCP bind addresses (comma separated). :5300 binds the wildcard for local
+# tooling; :53 binds the node WAN IP specifically (via CAP_NET_BIND_SERVICE) so
+# instances resolve against the node directly. Binding the WAN IP rather than
+# 0.0.0.0 avoids the systemd-resolved stub on 127.0.0.53:53. A future per-VPC
+# resolver VIP (OVN) will front :53.
+listen     = "0.0.0.0:5300,{{if .AdvertiseIP}}{{.AdvertiseIP}}{{else}}{{.BindIP}}{{end}}:53"
+dot_listen = ""
+doh_listen = ""
+tls_cert   = ""
+tls_key    = ""
+
+# Authoritative base domain for internal resource names.
+default_domain = "{{.NorthstarDefaultDomain}}"
+
+# AWS-parity private DNS zone for instance ip-<addr> records (IMDS hostname).
+internal_domain = "{{.NorthstarInternalDomain}}"
+
+# Seconds between S3 zone re-scans (anti-entropy; live changes arrive via NATS).
+sync_interval = 30
+
+# DNS (Route 53) service quotas (default off, mirrors [quota] in awsgw.toml).
+# Applied at daemon start; restart to change.
+[quotas]
+enabled = false
+records_per_hosted_zone = 10000   # AWS default; enforced only when enabled
+
+[s3]
+endpoint   = "https://{{if eq .BindIP "0.0.0.0"}}127.0.0.1{{else}}{{.BindIP}}{{end}}:8443"
+bucket     = "{{.NorthstarBucket}}"
+region     = "{{.Region}}"
+access_key = "{{.NorthstarAccessKey}}"
+secret_key = "{{.NorthstarSecretKey}}"
+
+# Forwarders for non-authoritative (recursive) queries. Empty => REFUSED.
+[upstream]
+nameservers = [{{range $i, $s := .PoolDNSServers}}{{if $i}}, {{end}}"{{$s}}:53"{{end}}]

cmd/spinifex/cmd/templates/predastore-multinode.toml

@@ -46,13 +46,40 @@ public = false
 encryption = ""
 account_id = "000000000000"
 
+{{- if .NorthstarAccessKey}}
+# Northstar DNS zone storage. Read-only for the northstar service; the control
+# plane writes zone files with the system credentials.
+[[buckets]]
+name = "{{.NorthstarBucket}}"
+region = "{{.Region}}"
+type = "distributed"
+public = false
+encryption = ""
+account_id = "000000000000"
+{{- end}}
+
 [[auth]]
 access_key_id = "{{.AccessKey}}"
 secret_access_key = "{{.SecretKey}}"
 account_id = "000000000000"
 policy = [
   { bucket = "predastore", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
+{{- if .NorthstarAccessKey}}
+  { bucket = "{{.NorthstarBucket}}", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"] },
+{{- end}}
 ]
+{{- if .NorthstarAccessKey}}
+
+# Bucket-scoped, read-only credentials for the northstar service (isolation
+# guarantee: ListBucket + GetObject on the northstar bucket only).
+[[auth]]
+access_key_id = "{{.NorthstarAccessKey}}"
+secret_access_key = "{{.NorthstarSecretKey}}"
+account_id = "000000000000"
+policy = [
+  { bucket = "{{.NorthstarBucket}}", actions = ["s3:ListBucket", "s3:GetObject"] },
+]
+{{- end}}
 
 # API request throttling (per-account, per-action token bucket)
 [ratelimit]

cmd/spinifex/cmd/templates/predastore.toml

@@ -75,13 +75,40 @@ public = false
 encryption = ""
 account_id = "000000000000"
 
+{{- if .NorthstarAccessKey}}
+# Northstar DNS zone storage. Read-only for the northstar service; the control
+# plane writes zone files with the system credentials.
+[[buckets]]
+name = "{{.NorthstarBucket}}"
+region = "{{.Region}}"
+type = "distributed"
+public = false
+encryption = ""
+account_id = "000000000000"
+{{- end}}
+
 [[auth]]
 access_key_id = "{{.AccessKey}}"
 secret_access_key = "{{.SecretKey}}"
 account_id = "000000000000"
 policy = [
   { bucket = "predastore", actions = ["s3:ListBucket",  "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
+{{- if .NorthstarAccessKey}}
+  { bucket = "{{.NorthstarBucket}}", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"] },
+{{- end}}
 ]
+{{- if .NorthstarAccessKey}}
+
+# Bucket-scoped, read-only credentials for the northstar service (isolation
+# guarantee: ListBucket + GetObject on the northstar bucket only).
+[[auth]]
+access_key_id = "{{.NorthstarAccessKey}}"
+secret_access_key = "{{.NorthstarSecretKey}}"
+account_id = "000000000000"
+policy = [
+  { bucket = "{{.NorthstarBucket}}", actions = ["s3:ListBucket", "s3:GetObject"] },
+]
+{{- end}}
 
 # API request throttling (per-account, per-action token bucket)
 [ratelimit]

cmd/spinifex/cmd/templates/spinifex.toml

@@ -39,6 +39,10 @@ gateway     = "{{.Gateway}}"
 {{- if .GatewayIP}}
 gateway_ip  = "{{.GatewayIP}}"
 {{- end}}
+{{- if .GwLrpRangeStart}}
+gw_lrp_range_start = "{{.GwLrpRangeStart}}"
+gw_lrp_range_end   = "{{.GwLrpRangeEnd}}"
+{{- end}}
 prefix_len  = {{if .PrefixLen}}{{.PrefixLen}}{{else}}24{{end}}
 {{- if .DNSServers}}
 dns_servers = [{{range $i, $s := .DNSServers}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
@@ -88,7 +92,7 @@ gpu_passthrough = true
 host = "{{.BindIP}}:9999"
 tlskey = "config/server.key"
 tlscert = "config/server.pem"
-debug = true
+debug = false
 config = "awsgw/awsgw.toml"
 
 [nodes.{{.Node}}.nats]
@@ -102,6 +106,7 @@ token = "{{.NatsToken}}"
 
 [nodes.{{.Node}}.viperblock]
 shardwal = false
+gc_enabled = true
 {{- if .EncryptionKeyFile}}
 encryption_key_file = "{{.EncryptionKeyFile}}"
 {{- end}}
@@ -125,6 +130,14 @@ secretkey = "{{.SecretKey}}"
 {{- if gt .PredastoreNodeID 0}}
 node_id = {{.PredastoreNodeID}}
 {{- end}}
+
+{{- if .NorthstarConfigPath}}
+
+[nodes.{{.Node}}.northstar]
+config_path = "{{.NorthstarConfigPath}}"
+default_domain = "{{.NorthstarDefaultDomain}}"
+internal_domain = "{{.NorthstarInternalDomain}}"
+{{- end}}
 {{- range .RemoteNodes}}
 
 [nodes.{{.Name}}]
@@ -135,4 +148,9 @@ az = "{{.AZ}}"
 {{- if .Services}}
 services = [{{range $i, $s := .Services}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
 {{- end}}
+{{- if .NorthstarConfigPath}}
+
+[nodes.{{.Name}}.northstar]
+config_path = "{{.NorthstarConfigPath}}"
+{{- end}}
 {{- end}}

cmd/spinifex/cmd/version.go

@@ -8,7 +8,7 @@ import (
 )
 
 // Version variables set via ldflags at build time.
-// Example: go build -ldflags "-X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=v1.0.0"
+// Example: go build -ldflags "-X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=v1.0.0".
 var (
 	Version = "dev"
 	Commit  = "unknown"

docs/COMMANDS.md

@@ -31,6 +31,9 @@ Service lifecycle commands for starting, stopping, and checking status of all Sp
 | `spx service vpcd start` | — | Loads cluster config → starts VPC daemon (subscribes to `vpc.*` NATS events, translates to OVN logical switches/ports/routers) |
 | `spx service vpcd stop` | — | Stops the vpcd service |
 | `spx service vpcd status` | — | Reports vpcd service status |
+| `spx service northstar start` | `--northstar-config` (overrides `nodes.<node>.northstar.config_path`) | Loads cluster config → reads `northstar.toml` → starts the northstar DNS server (authoritative for internal `*.spx3.net`, recursive via upstream forwarders), syncing zones from its S3 bucket. Guests resolve via `169.254.169.253`, served by vpcd's per-tap DNS shim which forwards to northstar |
+| `spx service northstar stop` | — | Stops the northstar service |
+| `spx service northstar status` | — | Reports northstar service status |
 | `spx service qmp-collector start` | — | Starts the guest-metrics collector (polls per-VM telemetry QMP sockets + tap counters, publishes CloudWatch-shaped series to NATS `metrics.ec2.*`) |
 | `spx service qmp-collector stop` | — | Stops the qmp-collector service |
 | `spx service qmp-collector status` | — | Reports qmp-collector service status |
@@ -54,8 +57,8 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Flags | Description |
 |---------|-------|-------------|
-| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; add `--external-pool start-end --external-gateway <ip>` or `--external-source=dhcp [--external-bind-bridge <iface>]` for a public pool with full EIP support), `--no-external` | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` |
-| `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services` | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations |
+| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--skip-host-dns`, `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; add `--external-pool start-end --external-gateway <ip>` or `--external-source=dhcp [--external-bind-bridge <iface>]` for a public pool with full EIP support; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` → points the host resolver at this node's local northstar so the Spinifex zones (ELB/EKS names) resolve from the node itself (opt out with `--skip-host-dns`). `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
+| `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services`, `--skip-host-dns` | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations → points the host resolver at this node's local northstar so the Spinifex zones resolve from the node itself (opt out with `--skip-host-dns`) |
 
 ### Version
 
@@ -171,6 +174,11 @@ read.
 | `spx admin gpu mig enable` | `--profile <name>` (required, e.g. `1g.10gb`), `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → enables MIG mode on each target (`gpu.EnableMIGMode`) → lists available profiles (`gpu.ListProfiles`) and validates requested profile name → destroys any existing instances (`gpu.DestroyAllInstances`) → creates new instances filling GPU capacity (`gpu.CreateInstances`) → writes `mig_profile` to `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. |
 | `spx admin gpu mig disable` | `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → destroys all GPU instances (`gpu.DestroyAllInstances`) → disables MIG mode (`gpu.DisableMIGMode`) → clears `mig_profile` in `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. |
 
+### EKS Control-Plane Disaster Recovery
+
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin eks restore-snapshot` | `--cluster` (required), `--snapshot` (optional, defaults to the latest snapshot in predastore), `--account` (optional, defaults to the bootstrap account) | Single-CP total-loss DR path (fail-safe): validates the snapshot exists in predastore BEFORE any mutation (a typo'd/missing key hard-fails, never resets into an empty datastore) → launches a fresh control-plane VM as a cluster-init seed (replaying the persisted create-time launch template) → sets a required-snapshot `RecoveryDirective` (`cluster-reset`) so the boot-time recovery agent aborts rather than resets-into-empty if it cannot fetch the snapshot → persists the replacement in cluster meta BEFORE re-pointing the NLB (so an NLB failure is convergeable by the reconciler, returned as a provisional status, not a hard error) → re-points the cluster NLB's apiserver and konnectivity target groups from the old CP's ENI to the new one → fences the old CP with retries, failing loudly if it cannot be confirmed terminated (split-brain guard). Any failure before the meta commit unwinds the fresh CP (terminate + clear directive) so a re-run does not stack a second resetting control plane. The returned status is provisional — success means the sequence completed, not that etcd is restored and serving; verify cluster health. HA clusters (a spread with a potentially surviving quorum) are rejected — recover those via quorum reformation instead. |
 
 ## AWS-Compatible API
 
@@ -178,7 +186,7 @@ read.
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
-| `run-instances` | `--image-id`, `--instance-type`, `--count`, `--key-name`, `--user-data`, `--subnet-id`, `--security-group-ids`, `--tag-specifications` (instance-scoped), `--block-device-mappings` (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination), `--placement` (GroupName), `--iam-instance-profile` (Name/Arn), `--capacity-reservation-specification` (CapacityReservationTarget.CapacityReservationId, targeted-by-id only), `--metadata-options` (HttpPutResponseHopLimit; IMDSv2-only enforced — rejects `http-tokens=optional`) | `--dry-run`, `--client-token`, `--disable-api-termination`, `--ebs-optimized`, `--network-interfaces`, `--private-ip-address`, `--monitoring`, `--credit-specification`, `--cpu-options`, `--launch-template`, `--hibernate-options` | **DONE** |
+| `run-instances` | `--image-id`, `--instance-type`, `--count`, `--key-name`, `--user-data`, `--subnet-id`, `--security-group-ids`, `--tag-specifications` (instance-scoped), `--block-device-mappings` (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination), `--placement` (GroupName), `--iam-instance-profile` (Name/Arn), `--capacity-reservation-specification` (CapacityReservationTarget.CapacityReservationId, targeted-by-id only), `--metadata-options` (HttpPutResponseHopLimit; IMDSv2-only enforced — rejects `http-tokens=optional`), `--launch-template` (LaunchTemplateId/LaunchTemplateName, Version — resolves `$Default`/`$Latest`; direct params override the template) | `--dry-run`, `--client-token`, `--disable-api-termination`, `--ebs-optimized`, `--network-interfaces`, `--private-ip-address`, `--monitoring`, `--credit-specification`, `--cpu-options`, `--hibernate-options` | **DONE** |
 | `describe-instances` | `--instance-ids`, `--filters` (instance-state-name, instance-id, instance-type, vpc-id, subnet-id, tag:*, tag-key, tag-value) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
 | `start-instances` | `--instance-ids` | `--dry-run`, `--force` | **DONE** |
 | `stop-instances` | `--instance-ids` | `--force`, `--hibernate`, `--dry-run` | **DONE** |
@@ -480,11 +488,13 @@ EIP handlers are always registered. Without a public IPAM pool (external mode di
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
-| `create-launch-template` | — | `--launch-template-name`, `--launch-template-data`, `--tag-specifications` | **NOT STARTED** |
-| `create-launch-template-version` | — | `--launch-template-id`/`--launch-template-name`, `--launch-template-data`, `--source-version` | **NOT STARTED** |
-| `delete-launch-template` | — | `--launch-template-id`/`--launch-template-name`, `--dry-run` | **NOT STARTED** |
-| `describe-launch-templates` | — | `--launch-template-ids`, `--launch-template-names`, `--filters` | **NOT STARTED** |
-| `describe-launch-template-versions` | — | `--launch-template-id`/`--launch-template-name`, `--versions`, `--min-version`, `--max-version` | **NOT STARTED** |
+| `create-launch-template` | `--launch-template-name`, `--launch-template-data` (full nested RequestLaunchTemplateData), `--version-description`, `--tag-specifications` (launch-template-scoped), `--dry-run` (no-op) | `--client-token` (idempotency) | **DONE** |
+| `create-launch-template-version` | `--launch-template-id`/`--launch-template-name`, `--launch-template-data`, `--source-version` (clone-and-override), `--version-description`, `--dry-run` (no-op) | `--client-token`, `--resolve-alias` | **DONE** |
+| `delete-launch-template` | `--launch-template-id`/`--launch-template-name`, `--dry-run` (no-op) | — | **DONE** |
+| `delete-launch-template-versions` | `--launch-template-id`/`--launch-template-name`, `--versions` (rejects the current default version), `--dry-run` (no-op) | — | **DONE** |
+| `modify-launch-template` | `--launch-template-id`/`--launch-template-name`, `--default-version`, `--dry-run` (no-op) | — | **DONE** |
+| `describe-launch-templates` | `--launch-template-ids`, `--launch-template-names`, `--filters` (launch-template-id, launch-template-name, create-time, tag:*, tag-key) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
+| `describe-launch-template-versions` | `--launch-template-id`/`--launch-template-name`, `--versions` (`$Default`/`$Latest`/numeric), `--min-version`, `--max-version`, `--filters` (is-default-version, image-id, instance-type, kernel-id, ram-disk-id, ebs-optimized) | `--max-results`, `--next-token`, `--dry-run`, `--resolve-alias` | **DONE** |
 
 ### EC2 — Dedicated Hosts, IPv4 Pools, DHCP, Capacity Reservations
 
@@ -662,6 +672,12 @@ All IAM operations are account-scoped. Root user (account `000000000000`) bypass
 | `delete-group-policy` | `--group-name`, `--policy-name` | — | **DONE** |
 | `list-group-policies` | `--group-name` | `--max-items`, `--marker` | **DONE** |
 
+### IAM — Account
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `get-account-summary` | — | — | **DONE** |
+
 ---
 
 ## STS

docs/compute/vpc-networking/README.md

@@ -198,6 +198,12 @@ public IPs, no Elastic IPs, and no inbound from WAN — all subnets behave as
 private subnets with internet access. On routed-NAT nodes, adding a public pool
 restores full public IP parity (see below).
 
+> **Limitation (routed-NAT v1):** System instances (ECS/EKS/load-balancer
+> agents) source egress from ExternalIPAM pool IPs, which do not exist in
+> `external_mode=nat`. Those features require `external_mode=pool`. A v2 will
+> either allocate transit IPs for system instances or reject the feature at the
+> API level in nat mode.
+
 The `gateway_ip` is the IP that OVN uses for SNAT. You can set it statically or
 use `setup-ovn.sh --dhcp` to obtain one from the router. This is the router's
 DHCP — not Spinifex's internal OVN DHCP for VMs.
@@ -549,6 +555,7 @@ dns_servers = ["8.8.8.8"]           # DNS for VMs (optional)
 | `region`      | No          | Scopes pool to a region. Instances in this region prefer this pool.                                                                                                    |
 | `az`          | No          | Scopes pool to an AZ. More specific than region.                                                                                                                       |
 | `dns_servers` | No          | DNS servers propagated to VMs via OVN DHCP.                                                                                                                            |
+| `gw_lrp_range_start` / `gw_lrp_range_end` | No | Reserve gateway-LRP IPs for per-VPC OVN routers. When unset, the allocator auto-derives the top 16 host IPs of the pool subnet (~15 concurrent VPCs). Widen to raise the concurrent-VPC ceiling — the `nat-transit` pool defaults to `100.127.0.16`-`100.127.0.254` (239 VPCs). Must not overlap `range_start`/`range_end`. |
 
 ### Why range_start/range_end Instead of CIDR?
 

docs/install/install-airgapped/README.md

@@ -75,7 +75,7 @@ Mirror the cloud image you intend to run as guest VMs:
 
 ```bash
 mkdir -p images
-curl -fsSL "https://cloud.debian.org/images/cloud/trixie/20260518-2482/debian-13-genericcloud-amd64-20260518-2482.qcow2" \
+curl -fsSL "https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2" \
   -o images/debian-13-amd64.qcow2
 ```
 

docs/reference-architectures/RTX-Pro-6000-EKS/README.md

@@ -0,0 +1,453 @@
+---
+title: "EKS AI Platform on Bare Metal: llama.cpp + YOLO on Spinifex"
+description: "Deploy a GPU-accelerated AI inference platform — an OpenAI-compatible LLM API and a real-time CV stream — in Kubernetes on bare-metal hardware, managed entirely with standard AWS tooling."
+category: "Reference Architectures"
+tags:
+  - nvidia
+  - rtx-pro-6000
+  - eks
+  - kubernetes
+  - llama.cpp
+  - yolo
+  - ecr
+  - bare-metal
+  - terraform
+resources:
+  - title: "eks-ai-platform (Terraform workbook, Dockerfiles, scripts)"
+    url: "https://github.com/mulgadc/eks-ai-platform"
+  - title: "GPU Passthrough"
+    url: "/docs/gpu-passthrough"
+  - title: "Launching EKS Clusters"
+    url: "/docs/eks"
+  - title: "llama.cpp"
+    url: "https://github.com/ggml-org/llama.cpp"
+  - title: "Ultralytics YOLO"
+    url: "https://docs.ultralytics.com"
+sections:
+  - overview
+  - prerequisites
+  - instructions
+---
+
+## Overview
+
+Spinifex is an open-source infrastructure platform that brings core AWS services such as EC2, S3, EBS and EKS, to bare-metal, edge, and on-prem deployments. It exposes a fully AWS-compatible API, so any tooling that works against AWS — the `aws` CLI, OpenTofu, `kubectl` — works against a Spinifex node unchanged, with a single profile swap.
+
+This guide walks through the use of Spinifex to deploy a self-contained AI inference platform on Supermicro's bare metal X14 platform using only standard AWS tooling. We use Terraform to create resources (wrapped in simple `make` commands) in the exact same way you would create AWS resources.
+
+
+Specifically, we create an EKS cluster with two worker nodes, each consisting of a g7e.2xlarge EC2 instance with an attached GPU via VFIO passthrough, an ALB to route traffic to each node, ECR for storing and managing our workload images, and all of the associated security and certificate management requirements (IAM, ACM) you would expect from real AWS.
+
+### Platform
+
+| Component | Specification |
+|---|---|
+| **Chassis** | Supermicro X14 2U CloudDC with Intel Xeon 6730P |
+| **GPUs** | 2× NVIDIA RTX Pro 6000 Blackwell Server Edition (96 GiB GDDR7 each, 192 GiB total) |
+| **Storage** | 4× NVMe SSD: 2× 1.5 TB, 2× 880 GB — one 1.5 TB drive carries the OS; the remaining three back Predastore |
+| **Spinifex instance family** | `g7e` — one RTX Pro 6000 per instance via VFIO PCIe passthrough |
+| **Kubernetes** | k3s, managed via Spinifex EKS API |
+| **API endpoint** | `https://<host>:9999` (AWS-compatible) |
+
+### Workloads
+Three workloads run in the `inference` namespace of an EKS cluster backed by two GPU worker nodes:
+
+| Pod | Role | GPU | Model | Image source |
+|---|---|---|---|---|
+| `llm-server` | OpenAI-compatible chat API (llama.cpp) | 1× RTX Pro 6000 | Llama 3.2 3B Instruct Q4_K_M GGUF | ECR `llm-server:latest` |
+| `yolo-stream` | MJPEG object-detection stream (CUDA) | 1× RTX Pro 6000 | YOLO11x | ECR `yolo-stream:latest` |
+| `ai-dashboard` | Web UI: chat + live detection feed | CPU only | — | ECR `ai-dashboard:latest` |
+
+Both GPU workloads bake their model weights into the Docker image at build time — `llm-server`'s GGUF weights (~2 GiB) and `yolo-stream`'s YOLO11x checkpoint (~109 MB) are present in the image when it starts.
+
+`yolo-stream` renders a 1280×720 sample video through YOLO11x once on startup (~25 s), caches the annotated frames in memory, then serves `/stream` from that cache — smooth MJPEG playback decoupled from per-frame inference cost after the initial warm-up.
+
+The two GPU nodes are split one-per-workload. With one RTX Pro 6000 per node and both pods requesting `nvidia.com/gpu: "1"`, the scheduler cannot fit both on the same node, so the split happens naturally without explicit affinity rules.
+
+
+
+## Architecture
+
+<p align="center"><img src="../../../.github/assets/images/rtx6000/ai-platform-request-flow-2.png" alt="AI Platform request flow: HTTPS ingress → ALB routing → EKS ai-platform GPU workers"></p>
+
+### AWS services exercised
+
+| Service | Role |
+|---|---|
+| **IAM** | Cluster role, node role (with ECR, LBC, EBS-CSI permissions inline), viewer access entry |
+| **ECR** | Private registry for all three images; source of truth for builds, though nodes receive images via sideload rather than live OCI pull for this demo |
+| **EC2** | 2× GPU microVM (`g7e.2xlarge`), one RTX Pro 6000 each via VFIO PCIe passthrough |
+| **EKS** | Cluster, GPU nodegroup (`desired_size = 2`), LBC and EBS-CSI managed addons, access entries |
+| **ELBv2** | ALB provisioned by the LBC addon; single shared IngressGroup across all three services |
+| **ACM** | Self-signed cert (`ai-platform.spinifex.local`) imported and attached to the ALB HTTPS listener |
+| **EBS (Viperblock)** | 200 GB root volume per GPU worker node, provisioned by the nodegroup (`disk_size = 200`)|
+
+All permissions for the LBC and EBS-CSI addons are attached directly to the node role. Both addons support IRSA, but fall back to the node's instance profile when no `service_account_role_arn` is supplied — sufficient for a single-cluster deployment.
+
+## Prerequisites
+
+### On the Spinifex host
+
+**1. Configure host-local VPC networking**
+
+Spinifex uses bridged networking via OVN. The X14 exposes a single public IP on one physical NIC — before installing Spinifex, ensure `br-wan` exists with the physical NIC enslaved to it, and attach a host-local address range for EC2 instance communication:
+
+```yaml
+# /etc/netplan/…
+bridges:
+  br-wan:
+    addresses:
+      - 192.168.10.1/24             # VM pool gateway — host-local
+      - <existing-wan-ip>/<prefix>  # existing WAN IP — unchanged
+    routes:
+      - to: default
+        via: <upstream-gateway>
+```
+
+Apply with `sudo netplan apply`. For outbound connectivity from instances through the host's WAN interface:
+
+```bash
+sysctl -w net.ipv4.ip_forward=1
+iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o br-wan -j MASQUERADE
+```
+
+The full process is described in the [VPC Networking](/docs/vpc-networking#host-local-subnet-no-upstream-router) guide.
+
+**2. Install Spinifex**
+
+Follow the [Single Node Install](/docs/install) guide. This installs Spinifex and starts all services, however `spinifex.toml` needs to be edited to wire up the bridge created in the previous step, as described in the following section.
+
+**3. Configure spinifex.toml and restart services**
+
+Edit `/etc/spinifex/spinifex.toml` to point the external pool at the bridge address range created in step 1:
+
+```toml
+[network]
+external_mode = "pool"
+
+[[network.external_pools]]
+name        = "wan"
+source      = "static"
+range_start = "192.168.10.2"
+range_end   = "192.168.10.100"
+gateway     = "192.168.10.1"
+prefix_len  = 24
+dns_servers = ["8.8.8.8"]
+```
+
+Then restart all services:
+
+```bash
+sudo systemctl restart spinifex.target
+sudo systemctl status spinifex.target
+```
+
+**4. Bind the GPUs to VFIO**
+
+```bash
+sudo spx admin gpu setup
+# Reboot, then:
+sudo spx admin gpu enable
+```
+
+Confirm both GPUs are bound:
+
+```bash
+lspci -d 10de: -nn
+# Expect: NVIDIA Corporation Device [10de:2bb5] appearing twice
+```
+
+**5. Attach Predastore storage**
+
+The X14 has four NVMe drives: two 1.5 TB SSDs (one carries the OS) and two ~880 GB SSDs. Predastore is distributed across the three non-OS drives — one storage node per physical drive, with Reed–Solomon redundancy so a single drive failure is recoverable.
+
+Confirm drive assignments with `lsblk` before proceeding, as device names vary between systems.
+
+```bash
+lsblk   # identify the OS drive and the three data drives
+
+# Stop services so Predastore isn't writing while we relocate its data directories
+sudo systemctl stop spinifex.target
+
+# --- Repeat the block below for each data drive (node-1/nvme-1, node-2/nvme-2, node-3/nvme-3) ---
+
+# Drive 1: non-OS 1.5 TB (e.g. /dev/nvme1n1)
+sudo mkdir -p /mnt/nvme-1
+sudo mount /dev/nvme1n1 /mnt/nvme-1
+sudo mkdir -p /mnt/nvme-1/nodes /mnt/nvme-1/db
+
+sudo mv /var/lib/spinifex/predastore/distributed/nodes/node-1 /mnt/nvme-1/nodes/node-1
+sudo mv /var/lib/spinifex/predastore/distributed/db/node-1    /mnt/nvme-1/db/node-1
+
+sudo ln -s /mnt/nvme-1/nodes/node-1 /var/lib/spinifex/predastore/distributed/nodes/node-1
+sudo ln -s /mnt/nvme-1/db/node-1    /var/lib/spinifex/predastore/distributed/db/node-1
+
+echo "/dev/nvme1n1  /mnt/nvme-1  auto  defaults  0  2" | sudo tee -a /etc/fstab
+
+# Drive 2: first ~880 GB drive (e.g. /dev/nvme2n1)
+sudo mkdir -p /mnt/nvme-2
+sudo mount /dev/nvme2n1 /mnt/nvme-2
+sudo mkdir -p /mnt/nvme-2/nodes /mnt/nvme-2/db
+
+sudo mv /var/lib/spinifex/predastore/distributed/nodes/node-2 /mnt/nvme-2/nodes/node-2
+sudo mv /var/lib/spinifex/predastore/distributed/db/node-2    /mnt/nvme-2/db/node-2
+
+sudo ln -s /mnt/nvme-2/nodes/node-2 /var/lib/spinifex/predastore/distributed/nodes/node-2
+sudo ln -s /mnt/nvme-2/db/node-2    /var/lib/spinifex/predastore/distributed/db/node-2
+
+echo "/dev/nvme2n1  /mnt/nvme-2  auto  defaults  0  2" | sudo tee -a /etc/fstab
+
+# Drive 3: second ~880 GB drive (e.g. /dev/nvme3n1)
+sudo mkdir -p /mnt/nvme-3
+sudo mount /dev/nvme3n1 /mnt/nvme-3
+sudo mkdir -p /mnt/nvme-3/nodes /mnt/nvme-3/db
+
+sudo mv /var/lib/spinifex/predastore/distributed/nodes/node-3 /mnt/nvme-3/nodes/node-3
+sudo mv /var/lib/spinifex/predastore/distributed/db/node-3    /mnt/nvme-3/db/node-3
+
+sudo ln -s /mnt/nvme-3/nodes/node-3 /var/lib/spinifex/predastore/distributed/nodes/node-3
+sudo ln -s /mnt/nvme-3/db/node-3    /var/lib/spinifex/predastore/distributed/db/node-3
+
+echo "/dev/nvme3n1  /mnt/nvme-3  auto  defaults  0  2" | sudo tee -a /etc/fstab
+
+# --- End of per-drive block ---
+
+sudo systemctl start spinifex.target
+```
+
+Verify the nodes are healthy before proceeding:
+
+```bash
+export AWS_PROFILE=spinifex
+aws s3 ls
+# Should return without error (empty bucket list is fine)
+```
+
+**6. Verify the GPU instance type**
+
+```bash
+sudo spx admin gpu status
+```
+
+This will print confirmation that GPU passthrough has been configured correctly along with the available GPU instance types. The RTX Pro 6000 Blackwell Server Edition (PCI device `10de:2bb5`) maps to the `g7e` family. The workbook defaults to `g7e.2xlarge` (one GPU per node); override with `GPU_TYPE=g7e.4xlarge` (or the size your host reports) if needed. Do not use `g7e.12xlarge` — that is the 2× GPU size.
+
+### Local tooling
+
+- [OpenTofu](https://opentofu.org/) >= 1.6
+- [kubectl](https://kubernetes.io/docs/tasks/tools/)
+- [Docker](https://docs.docker.com/get-docker/)
+
+
+### Clone the workbook
+
+```bash
+git clone https://github.com/mulgadc/eks-ai-platform
+cd eks-ai-platform
+```
+
+## Instructions
+
+### 1. Import the EKS GPU node AMI
+
+GPU worker nodes require the `ecr-credential-provider` binary so kubelet can call `GetAuthorizationToken` against ECR. This binary is included in the dedicated EKS GPU node AMI in the Spinifex image catalogue. List available images and import it:
+
+```bash
+spx admin images list
+# Look for the EKS GPU node image
+
+spx admin images import --name spinifex-eks-node-gpu
+```
+
+Confirm the AMI is registered:
+
+```bash
+aws ec2 describe-images --query 'Images[*].[Name,ImageId]' --output table
+```
+
+### 2. Provision the cluster
+
+```bash
+make infra ENDPOINT=https://<host>:9999
+```
+
+This provisions the VPC (`10.33.0.0/16`, two public and two private subnets with a NAT gateway), IAM roles, ECR repositories, EKS cluster, GPU nodegroup (2× `g7e.2xlarge`, 200 GB disk each), LBC and EBS-CSI managed addons, a self-signed ACM certificate, and the NodePort security group rules the ALB needs to reach the worker nodes.
+
+Update your kubeconfig once the cluster reports `ACTIVE`:
+
+```bash
+$(tofu -chdir=workbook output -raw update_kubeconfig)
+kubectl get nodes
+# Expect: 2 Ready nodes in the gpu-workers nodegroup
+```
+
+The Makefile wraps `tofu -chdir=workbook apply -var spinifex_endpoint=... -var gpu_instance_type=...` — running Tofu directly is equivalent and lets you pass any additional variables. The workbook provisions `aws_vpc`, `aws_subnet` (two public, two private), `aws_eks_cluster`, `aws_eks_node_group` (two `g7e.2xlarge` nodes, each with a 200 GB Viperblock root volume via `disk_size = 200`), `aws_eks_addon` for LBC and EBS-CSI, three `aws_ecr_repository` resources, three IAM roles, and a self-signed `aws_acm_certificate` — all via Spinifex's AWS-compatible endpoint at `:9999`.
+
+### 3. Build and push container images
+
+```bash
+make images
+```
+
+This authenticates to ECR, then builds and pushes all three images:
+
+- **`llm-server`** — based on `ghcr.io/ggml-org/llama.cpp:server-cuda`; downloads Llama 3.2 3B Instruct Q4_K_M GGUF (~2 GiB) from Hugging Face at build time and bakes it into the image. Exposes an OpenAI-compatible `/v1/chat/completions` API.
+- **`yolo-stream`** — based on `pytorch/pytorch:2.7.1-cuda12.8-cudnn9-runtime`; installs Ultralytics and downloads YOLO11x weights at build time. PyTorch 2.7.1+cu128 is required: the RTX Pro 6000 Blackwell is compute capability sm_120, and earlier PyTorch releases ship no sm_120 kernels.
+- **`ai-dashboard`** — lightweight Flask proxy (`FROM python:3.11-slim`) that aggregates the LLM API and YOLO stream into a single page.
+
+The ECR registry URI always includes `:9999` — for example, `<account>.dkr.ecr.ap-southeast-2.<suffix>:9999`. Use the `ecr_registry` Tofu output directly in `docker login` and image references; do not construct the hostname manually.
+
+Image URIs come from `tofu -chdir=workbook output -raw ecr_registry`. ECR authentication uses the same API as AWS: `aws ecr get-login-password` calls `GetAuthorizationToken` against the Spinifex ECR endpoint and returns a short-lived JWT that Docker accepts as a registry password. The `make images` target is equivalent to running those `docker build` and `docker push` commands directly against `$REGISTRY` from that Tofu output.
+
+### 4. Sideload images onto GPU worker nodes
+
+```bash
+make sideload
+```
+
+ECR is the source of truth for all three images — authentication, push, and registry management all work identically to AWS. In a standard EKS deployment, nodes would pull images directly from ECR at scheduling time. In this demo, live pulls of these image sizes (~2 GiB for `llm-server`, ~4.5 GiB for `yolo-stream`) through the Spinifex ECR gateway proved unreliable — large transfers stalled or failed mid-stream, a combination of network conditions on this single-host setup and a rough edge in early Spinifex ECR support. This step works around that by staging the images directly into each node's containerd store before the pods are scheduled.
+
+It exports each image from the local Docker daemon, serves the tarballs over HTTP from the Spinifex host, and imports them directly into each worker node's containerd store via short-lived privileged pods. All three Deployments use `imagePullPolicy: IfNotPresent` and depend on the images already being present in containerd.
+
+The script also labels the two GPU nodes deterministically (`workload=llm-server` / `workload=yolo-stream`, sorted by node name), and the Deployments use matching `nodeSelector` values so each pod lands on the node that already has its image. `ai-dashboard` has no GPU requirement and is imported on both nodes since it can schedule onto either.
+
+Expect several minutes for `yolo-stream`'s image (CUDA + PyTorch + YOLO11x weights, ~4.5 GiB).
+
+This step has no Tofu equivalent — it operates directly on the running cluster via `kubectl`. The ECR registry URI is read from the Tofu state; node labelling uses `kubectl label` and image import uses short-lived privileged pods that run `ctr images import` into each node's containerd store.
+
+### 5. Deploy workloads
+
+```bash
+make workloads ENDPOINT=https://<host>:9999
+```
+
+This deploys the NVIDIA GPU Operator via Helm, then all three application Deployments, ClusterIP and NodePort services, and the shared ALB Ingresses.
+
+The GPU Operator requires two adjustments for k3s:
+
+- `driver.enabled=false` — the NVIDIA driver is pre-built into the GPU AMI at image creation time; the Operator installs only the toolkit and device plugin.
+- `CONTAINERD_SOCKET=/run/k3s/containerd/containerd.sock` — k3s bundles its own containerd at a different socket and config path than the standalone containerd default. Without this override the toolkit DaemonSet crash-loops with `no such file or directory`.
+
+Watch the GPU Operator complete, then the inference pods come up:
+
+```bash
+kubectl -n gpu-operator get pods -w
+kubectl -n inference get pods -w
+```
+
+Confirm each GPU node reports `nvidia.com/gpu: 1` in allocatables:
+
+```bash
+$(tofu -chdir=workbook output -raw gpu_verify_hint)
+```
+
+All three routes share a single ALB via `alb.ingress.kubernetes.io/group.name`. Explicit `group.order` values (`/v1` = 10, `/stream` = 20, `/` = 100) ensure the dashboard's catch-all path evaluates last — without them, the LBC sorts Ingress resources alphabetically, which places the catch-all first and swallows the other routes.
+
+Retrieve the ALB IP (the DNS name `*.elb.spinifex.local` is a label, not a resolvable entry):
+
+```bash
+$(tofu -chdir=workbook output -raw alb_ip_hint)
+ALB_IP=<output>
+```
+
+Validate the LLM endpoint:
+
+```bash
+curl -sk https://$ALB_IP/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{"model":"llama-3.2-3b-instruct","messages":[{"role":"user","content":"What is Spinifex?"}]}' \
+  | python3 -m json.tool
+```
+
+Open `https://$ALB_IP/` to reach the dashboard. If accessing from a remote machine:
+
+```bash
+ssh -L 8443:$ALB_IP:443 <spinifex-host>
+# Open https://ai-platform.spinifex.local:8443/
+# Add 127.0.0.1 ai-platform.spinifex.local to /etc/hosts if the browser requires hostname match
+```
+
+The workloads module reads cluster coordinates, ECR image URIs, NodePort values, and the ACM cert ARN from the parent module's Tofu state, then creates the NVIDIA GPU Operator `helm_release`, three `kubernetes_deployment_v1` resources, six `kubernetes_service_v1` resources (ClusterIP + NodePort per workload), and three `kubernetes_ingress_v1` resources — all through the Terraform Kubernetes and Helm providers, which authenticate to the cluster via `aws eks get-token`.
+
+### Dashboard
+
+After the successful completion of the `make workloads` step, the dashboard should be available and displaying the outputs of the two worker nodes; YOLO computer vision in the left pane, and a LLM chat in the right pane, as shown in the images below.
+
+<p align="center"><img src="../../../.github/assets/images/rtx6000/yolo.png" alt="YOLO real-time object detection stream"></p>
+
+<p align="center"><img src="../../../.github/assets/images/rtx6000/llm-chat.png" alt="LLM chat interface"></p>
+
+
+
+### 6. Teardown
+
+```bash
+make destroy ENDPOINT=https://<host>:9999
+```
+
+Workloads are destroyed before infra. Both GPU worker instances terminate, immediately returning their RTX Pro 6000s to the Spinifex pool.
+
+The Makefile runs the two-module destroy sequence — `tofu -chdir=workbook/workloads destroy` first, then `tofu -chdir=workbook destroy` — because the parent module's security group rules are referenced by the ALB created in the workloads layer. Running Tofu directly in that order is equivalent.
+
+## Troubleshooting
+
+### `llm-server` or `yolo-stream` pod stuck in `Pending`
+
+The GPU Operator DaemonSet must complete before `nvidia.com/gpu` appears in node allocatables:
+
+```bash
+kubectl -n gpu-operator get daemonset -w
+kubectl get node -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu'
+```
+
+If a pod shows `Insufficient nvidia.com/gpu` on a node that looks healthy, the `workload=` labels from `make sideload` may be stale — for example, after a nodegroup recreation that assigned new node names. Re-run `make sideload` to relabel the nodes and re-import the images.
+
+### GPU worker instances fail to launch (`bind ... to vfio-pci: invalid argument`)
+
+On hosts where the GPU's IOMMU group contains an upstream PCIe root-port bridge (no ACS isolation), Spinifex attempts to bind the bridge to `vfio-pci`. Bridges are non-endpoint devices that `vfio-pci` refuses to bind, causing the instance to crash immediately after launch:
+
+```
+GPU claim failed... bind IOMMU group member 0000:14:02.0: bind 0000:14:02.0 to vfio-pci: invalid argument
+```
+
+Check IOMMU group membership with `lspci -nnk` and `/sys/kernel/iommu_groups/*/devices/`. This is fixed in Spinifex by excluding bridge-class PCI devices from the bind lifecycle — ensure your Spinifex build includes that fix. If the failed bind left a bridge without a driver, restore it:
+
+```bash
+echo | sudo tee /sys/bus/pci/devices/<bridge-addr>/driver_override
+echo <bridge-addr> | sudo tee /sys/bus/pci/drivers/pcieport/bind
+```
+
+If the nodegroup got wedged in `CREATING` after hitting this, delete it and let Terraform recreate it:
+
+```bash
+aws eks delete-nodegroup --cluster-name ai-platform --nodegroup-name gpu-workers
+```
+
+### Nodegroup stuck `CREATING` with healthy nodes, Tofu times out
+
+`tofu apply` hangs for 20 minutes and fails with `workers did not become Ready: timed out`, even though `kubectl get nodes` shows both workers `Ready`. The cause is a missing `eks.amazonaws.com/nodegroup` label on the node — without it, the control plane never tallies the nodegroup as satisfied. Confirm with:
+
+```bash
+kubectl get node <worker> -o jsonpath='{.metadata.labels}'
+```
+
+This is fixed upstream in Spinifex. If stuck, delete both the nodegroup and cluster and let Terraform recreate them cleanly:
+
+```bash
+aws eks delete-nodegroup --cluster-name ai-platform --nodegroup-name gpu-workers
+aws eks delete-cluster --name ai-platform
+```
+
+### ALB returns 502 immediately after rollout
+
+Both `llm-server` and `yolo-stream` require a few seconds after container start before their readiness probes pass — CUDA and GPU Operator initialisation contribute to the delay. The ALB marks targets unhealthy during this window. Watch the pods become ready:
+
+```bash
+kubectl -n inference get pods -w
+kubectl -n inference logs -f deploy/llm-server
+```
+
+## Conclusion
+
+This guide demonstrates how Spinifex turns a single bare-metal chassis into a production-shaped AI serving platform managed entirely with standard AWS tooling. IAM roles, ECR repositories, an EKS cluster, GPU worker nodes, addons, and an ALB are all provisioned with the same Terraform resources and AWS CLI commands that work on real AWS — with a single `AWS_PROFILE` swap.
+
+The RTX Pro 6000 Blackwell Server Edition's 96 GiB GDDR7 fits substantial GPU workloads in a single PCIe slot, and Spinifex's `g7e` instance family exposes each GPU as a standard EC2 instance. Teams already operating AWS infrastructure can point their existing tooling at a Spinifex node and retain the full EKS workflow — from `aws ecr get-login-password` to `kubectl get ingress` — on hardware they own. ECR acts as the canonical registry throughout: image build, push, and authentication are identical to AWS, with direct kubelet pulls being the intended path as Spinifex's ECR gateway matures.

docs/security/network-connections/README.md

@@ -107,8 +107,10 @@ Spinifex nodes initiate a small, fixed set of outbound connections.
 
 | Destination | Purpose | Protocol | Verification |
 |-------------|---------|----------|--------------|
-| `https://cloud.debian.org/images/cloud/trixie/` | Debian 13 cloud image download | HTTPS | TLS + checksum verification |
-| `https://cloud-images.ubuntu.com/resolute/` | Ubuntu 26.04 LTS cloud image download | HTTPS | TLS + checksum verification |
+| `https://cloud.debian.org/images/cloud/trixie/latest/` | Debian 13 cloud image download | HTTPS | TLS + checksum verification |
+| `https://cloud-images.ubuntu.com/releases/resolute/release/` | Ubuntu 26.04 LTS cloud image download | HTTPS | TLS + checksum verification |
+| `https://dl.rockylinux.org/pub/rocky/10/images/` | Rocky Linux 10 cloud image download | HTTPS | TLS + checksum verification |
+| `https://dl-cdn.alpinelinux.org/alpine/` | Alpine Linux cloud image download | HTTPS | TLS + checksum verification |
 | `https://d2yp8ipz5jfqcw.cloudfront.net` | Alpine image for managed HAProxy load-balancer | HTTPS | TLS + checksum verification |
 | `https://install.mulgadc.com/install` | One-shot install telemetry POST on `spx admin init` / `join`. | HTTPS | TLS |
 
@@ -116,7 +118,7 @@ Spinifex nodes initiate a small, fixed set of outbound connections.
 
 **Update checks and metadata.** Spinifex does not check for updates and does not consume a cloud metadata service (`169.254.169.254` is served *by* the cluster to guest VMs). Node software updates come from the operator's OS package channel. The install-telemetry endpoint above is the only vendor-operated destination contacted by a node; closed-egress deployments should disable it and record the opt-out in the security plan.
 
-**Air-gapped deployments.** The three image URLs are the only destinations needed for the standard image catalogue. Mirror them locally and use `spx admin images import --file` with pre-staged files. Telemetry must also be disabled. See [Air-Gapped Install](/docs/install-airgapped).
+**Air-gapped deployments.** The image URLs above are the only destinations needed for the standard image catalogue. Mirror them locally and use `spx admin images import --file` with pre-staged files. Telemetry must also be disabled. See [Air-Gapped Install](/docs/install-airgapped).
 
 ## 3. Cross-Node (Internal) Connections
 

docs/service-interfaces.yaml

@@ -300,6 +300,14 @@ suites:
     # spinifex_daemon. docker/crane/skopeo data-plane subtests skip when the
     # client binary or registry-host resolution is absent.
 
+  e2e-bedrock:
+    path: tests/e2e/bedrock
+    covers: [awsgw]
+    # Ochre (AWS-Bedrock) control plane: static catalog list/get served
+    # straight from awsgw (gateway/bedrock.go, gateway/bedrock/) — no NATS
+    # hop to spinifex_daemon. Converse (Anthropic-direct / self-host vLLM)
+    # subtests are gated on backend availability and skip by default.
+
   e2e-ecs:
     path: tests/e2e/ecs
     covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]

go.mod

@@ -4,45 +4,52 @@ go 1.26.5
 
 require (
 	github.com/aws/aws-sdk-go v1.55.8
-	github.com/aws/aws-sdk-go-v2 v1.42.1
-	github.com/aws/aws-sdk-go-v2/config v1.32.27
-	github.com/aws/aws-sdk-go-v2/credentials v1.19.26
+	github.com/aws/aws-sdk-go-v2 v1.43.0
+	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14
+	github.com/aws/aws-sdk-go-v2/config v1.32.31
+	github.com/aws/aws-sdk-go-v2/credentials v1.19.30
 	github.com/cenkalti/backoff/v4 v4.3.0
 	github.com/charmbracelet/bubbles v1.0.0
 	github.com/charmbracelet/bubbletea v1.3.10
 	github.com/charmbracelet/lipgloss v1.1.0
-	github.com/containerd/containerd/v2 v2.3.2
+	github.com/containerd/containerd/v2 v2.3.3
 	github.com/distribution/reference v0.6.0
-	github.com/go-chi/chi/v5 v5.3.0
+	github.com/go-chi/chi/v5 v5.3.1
 	github.com/golang-jwt/jwt/v5 v5.3.1
 	github.com/google/uuid v1.6.0
 	github.com/insomniacslk/dhcp v0.0.0-20260407060928-11b94ed970f2
 	github.com/klauspost/cpuid/v2 v2.4.0
-	github.com/mulgadc/predastore v1.11.0
-	github.com/mulgadc/viperblock v1.11.0
+	github.com/miekg/dns v1.1.72
+	github.com/mulgadc/northstar v1.14.0
+	github.com/mulgadc/predastore v1.14.0
+	github.com/mulgadc/viperblock v1.14.0
 	github.com/nats-io/nats-server/v2 v2.14.3
 	github.com/nats-io/nats.go v1.52.0
 	github.com/opencontainers/runtime-spec v1.3.0
 	github.com/ovn-kubernetes/libovsdb v0.8.1
-	github.com/pelletier/go-toml/v2 v2.4.2
+	github.com/pelletier/go-toml/v2 v2.4.3
 	github.com/pterm/pterm v0.12.83
 	github.com/spf13/cobra v1.10.2
 	github.com/spf13/viper v1.21.0
 	github.com/stretchr/testify v1.11.1
+	go.opentelemetry.io/contrib/bridges/otelslog v0.19.0
 	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0
 	go.opentelemetry.io/contrib/instrumentation/runtime v0.69.0
 	go.opentelemetry.io/otel v1.44.0
+	go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.20.0
 	go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.44.0
 	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.44.0
+	go.opentelemetry.io/otel/log v0.20.0
 	go.opentelemetry.io/otel/metric v1.44.0
 	go.opentelemetry.io/otel/sdk v1.44.0
+	go.opentelemetry.io/otel/sdk/log v0.20.0
 	go.opentelemetry.io/otel/sdk/metric v1.44.0
 	go.opentelemetry.io/otel/trace v1.44.0
 	go.uber.org/goleak v1.3.0
 	go.yaml.in/yaml/v3 v3.0.4
-	golang.org/x/crypto v0.53.0
-	golang.org/x/sync v0.21.0
-	golang.org/x/sys v0.46.0
+	golang.org/x/crypto v0.54.0
+	golang.org/x/sync v0.22.0
+	golang.org/x/sys v0.47.0
 	gopkg.in/ini.v1 v1.67.3
 )
 
@@ -55,17 +62,20 @@ require (
 	github.com/antithesishq/antithesis-sdk-go v0.7.0 // indirect
 	github.com/armon/go-metrics v0.4.1 // indirect
 	github.com/atotto/clipboard v0.1.4 // indirect
-	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.30 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.30 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.30 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.31 // indirect
+	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.31 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.31 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.31 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.32 // indirect
 	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.30 // indirect
-	github.com/aws/aws-sdk-go-v2/service/signin v1.2.2 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sso v1.31.5 // indirect
-	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.36.8 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sts v1.43.5 // indirect
-	github.com/aws/smithy-go v1.27.3 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.24 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.31 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.32 // indirect
+	github.com/aws/aws-sdk-go-v2/service/s3 v1.106.0 // indirect
+	github.com/aws/aws-sdk-go-v2/service/signin v1.5.0 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sso v1.33.0 // indirect
+	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.0 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sts v1.45.0 // indirect
+	github.com/aws/smithy-go v1.27.4 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/boltdb/bolt v1.3.1 // indirect
@@ -94,13 +104,13 @@ require (
 	github.com/containerd/ttrpc v1.2.8 // indirect
 	github.com/containerd/typeurl/v2 v2.2.3 // indirect
 	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
-	github.com/dgraph-io/badger/v4 v4.9.2 // indirect
+	github.com/dgraph-io/badger/v4 v4.9.5 // indirect
 	github.com/dgraph-io/ristretto/v2 v2.4.0 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
 	github.com/fatih/color v1.19.0 // indirect
 	github.com/felixge/httpsnoop v1.0.4 // indirect
-	github.com/fsnotify/fsnotify v1.9.0 // indirect
+	github.com/fsnotify/fsnotify v1.10.1 // indirect
 	github.com/gabriel-vasile/mimetype v1.4.13 // indirect
 	github.com/go-logr/logr v1.4.3 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
@@ -139,6 +149,7 @@ require (
 	github.com/minio/crc64nvme v1.1.1 // indirect
 	github.com/minio/highwayhash v1.0.4 // indirect
 	github.com/moby/locker v1.0.1 // indirect
+	github.com/moby/sys/capability v0.4.0 // indirect
 	github.com/moby/sys/mountinfo v0.7.2 // indirect
 	github.com/moby/sys/sequential v0.6.0 // indirect
 	github.com/moby/sys/signal v0.7.1 // indirect
@@ -153,6 +164,7 @@ require (
 	github.com/nats-io/nuid v1.0.1 // indirect
 	github.com/opencontainers/go-digest v1.0.0 // indirect
 	github.com/opencontainers/image-spec v1.1.1 // indirect
+	github.com/opencontainers/runtime-tools v0.9.1-0.20251114084447-edf4cb3d2116 // indirect
 	github.com/pierrec/lz4/v4 v4.1.18 // indirect
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
@@ -160,7 +172,7 @@ require (
 	github.com/prometheus/client_model v0.6.2 // indirect
 	github.com/prometheus/common v0.67.5 // indirect
 	github.com/prometheus/procfs v0.20.1 // indirect
-	github.com/quic-go/quic-go v0.60.0 // indirect
+	github.com/quic-go/quic-go v0.61.0 // indirect
 	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/sagikazarmark/locafero v0.12.0 // indirect
 	github.com/sirupsen/logrus v1.9.4 // indirect
@@ -175,20 +187,25 @@ require (
 	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
 	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.44.0 // indirect
 	go.opentelemetry.io/proto/otlp v1.10.0 // indirect
+	go.uber.org/nilaway v0.0.0-20260721205819-4a0f7653488a // indirect
 	go.yaml.in/yaml/v2 v2.4.4 // indirect
-	golang.org/x/mod v0.36.0 // indirect
+	golang.org/x/exp/typeparams v0.0.0-20260611194520-c48552f49976 // indirect
+	golang.org/x/mod v0.37.0 // indirect
 	golang.org/x/net v0.56.0 // indirect
-	golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6 // indirect
-	golang.org/x/term v0.44.0 // indirect
-	golang.org/x/text v0.38.0 // indirect
+	golang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57 // indirect
+	golang.org/x/term v0.45.0 // indirect
+	golang.org/x/text v0.40.0 // indirect
 	golang.org/x/time v0.15.0 // indirect
-	golang.org/x/tools v0.45.0 // indirect
-	golang.org/x/vuln v1.1.4 // indirect
+	golang.org/x/tools v0.47.0 // indirect
+	golang.org/x/vuln v1.4.0 // indirect
 	google.golang.org/genproto/googleapis/api v0.0.0-20260618152121-87f3d3e198d3 // indirect
 	google.golang.org/genproto/googleapis/rpc v0.0.0-20260618152121-87f3d3e198d3 // indirect
-	google.golang.org/grpc v1.81.1 // indirect
+	google.golang.org/grpc v1.82.1 // indirect
 	google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
+	sigs.k8s.io/yaml v1.6.0 // indirect
+	tags.cncf.io/container-device-interface v1.1.0 // indirect
+	tags.cncf.io/container-device-interface/specs-go v1.1.0 // indirect
 )
 
 // go.opencensus.io -> grpc@v1.33.2 drags in the pre-split monolithic genproto,
@@ -200,4 +217,7 @@ require (
 // stop requiring the monolith.
 replace google.golang.org/genproto => google.golang.org/genproto v0.0.0-20260622175928-b703f567277d
 
-tool golang.org/x/vuln/cmd/govulncheck
+tool (
+	go.uber.org/nilaway/cmd/nilaway
+	golang.org/x/vuln/cmd/govulncheck
+)

go.sum

@@ -402,6 +402,8 @@ codeberg.org/go-latex/latex v0.0.1/go.mod h1:AiC91vVG2uURZRd4ZN1j3mAac0XBrLsxK6+
 codeberg.org/go-latex/latex v0.1.0/go.mod h1:LA0q/AyWIYrqVd+A9Upkgsb+IqPcmSTKc9Dny04MHMw=
 codeberg.org/go-pdf/fpdf v0.10.0/go.mod h1:Y0DGRAdZ0OmnZPvjbMp/1bYxmIPxm0ws4tfoPOc4LjU=
 contrib.go.opencensus.io/exporter/stackdriver v0.13.15-0.20230702191903-2de6d2748484/go.mod h1:uxw+4/0SiKbbVSD/F2tk5pJTdVcfIBBcsQ8gwcu4X+E=
+cyphar.com/go-pathrs v0.2.1 h1:9nx1vOgwVvX1mNBWDu93+vaceedpbsDqo+XuBGL40b8=
+cyphar.com/go-pathrs v0.2.1/go.mod h1:y8f1EMG7r+hCuFf/rXsKqMJrJAUoADZGNh5/vZPKcGc=
 dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
 dmitri.shuralyov.com/gpu/mtl v0.0.0-20201218220906-28db891af037/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
 dmitri.shuralyov.com/gpu/mtl v0.0.0-20221208032759-85de2813cf6b/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
@@ -522,78 +524,78 @@ github.com/aws/aws-sdk-go-v2 v0.18.0/go.mod h1:JWVYvqSMppoMJC0x5wdwiImzgXTI9FuZw
 github.com/aws/aws-sdk-go-v2 v1.17.5/go.mod h1:uzbQtefpm44goOPmdKyAlXSNcwlRgF3ePWVW6EtJvvw=
 github.com/aws/aws-sdk-go-v2 v1.25.2/go.mod h1:Evoc5AsmtveRt1komDwIsjHFyrP5tDuF1D1U+6z6pNo=
 github.com/aws/aws-sdk-go-v2 v1.36.3/go.mod h1:LLXuLpgzEbD766Z5ECcRmi8AzSwfZItDtmABVkRLGzg=
-github.com/aws/aws-sdk-go-v2 v1.42.1 h1:9eOTgu1z/dVtYpNZ3/8/XbbaX0x/BqE3HUzAzs6K0ek=
-github.com/aws/aws-sdk-go-v2 v1.42.1/go.mod h1:5pKeft2eJj+gElQ38Jqg4ibCqh+/AK33/0X3hip7IjM=
+github.com/aws/aws-sdk-go-v2 v1.43.0 h1:fharf/WhbRAVZ1du0QL7roNFxZ6T/sWr+4Ni617bwSI=
+github.com/aws/aws-sdk-go-v2 v1.43.0/go.mod h1:5pKeft2eJj+gElQ38Jqg4ibCqh+/AK33/0X3hip7IjM=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 h1:3IZY0XAJquT3aHzbkHfPzy4ACPcEjVG0x87KOwtpqGY=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14/go.mod h1:zwM6veDkhGgQFqkBy+uT28AAYpLu+uFMlPl+rCg/73E=
 github.com/aws/aws-sdk-go-v2/config v1.18.14/go.mod h1:0pI6JQBHKwd0JnwAZS3VCapLKMO++UL2BOkWwyyzTnA=
 github.com/aws/aws-sdk-go-v2/config v1.27.4/go.mod h1:zq2FFXK3A416kiukwpsd+rD4ny6JC7QSkp4QdN1Mp2g=
 github.com/aws/aws-sdk-go-v2/config v1.29.12/go.mod h1:xse1YTjmORlb/6fhkWi8qJh3cvZi4JoVNhc+NbJt4kI=
-github.com/aws/aws-sdk-go-v2/config v1.32.27 h1:SJwJ9Q4kM7v5QVSYYyXj3znRr6lNyZEhSgAXmXXcVbI=
-github.com/aws/aws-sdk-go-v2/config v1.32.27/go.mod h1:uBfrzTRedDmB2u+b6+UlaKJy2O6VSH5un2jP24t/KvQ=
+github.com/aws/aws-sdk-go-v2/config v1.32.31 h1:n4nY9O3QKoHIkL85EX+V8RcMFtOhlpTFhGArg915PXk=
+github.com/aws/aws-sdk-go-v2/config v1.32.31/go.mod h1:PN0NYDCCoOpGGsZ2+elDUidmHfQBPyYzN2GCgl8HEBs=
 github.com/aws/aws-sdk-go-v2/credentials v1.13.14/go.mod h1:85ckagDuzdIOnZRwws1eLKnymJs3ZM1QwVC1XcuNGOY=
 github.com/aws/aws-sdk-go-v2/credentials v1.17.4/go.mod h1:+30tpwrkOgvkJL1rUZuRLoxcJwtI/OkeBLYnHxJtVe0=
 github.com/aws/aws-sdk-go-v2/credentials v1.17.65/go.mod h1:4zyjAuGOdikpNYiSGpsGz8hLGmUzlY8pc8r9QQ/RXYQ=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.26 h1:Si8kk1kyJnuJWCEgiwpBtTdtgSdR7i611596NnC0YIQ=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.26/go.mod h1:lBckz+W9SAdNtSDw3pYgQUJDJFcBBWry0GSzw+bK0TY=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.30 h1:TTCvvzFU6gXa4iJecNG/0F/B0oYTiazoRECr2XyLHrY=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.30/go.mod h1:jKxAp2AEncnliinzpgOSZDFv6+VjvWhjw/AtbfsWT9U=
 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.23/go.mod h1:mOtmAg65GT1HIL/HT/PynwPbS+UG0BgCZ6vhkPqnxWo=
 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.15.2/go.mod h1:iRlGzMix0SExQEviAyptRWRGdYNo3+ufW/lCzvKVTUc=
 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.30/go.mod h1:Jpne2tDnYiFascUEs2AWHJL9Yp7A5ZVy3TNyxaAjD6M=
-github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.30 h1:/hi1JADLEW9YYryEz1w4GQu0EtP23pP553Cf9KgsDV4=
-github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.30/go.mod h1:/3AOgy4K17Dm4ucMZVC/MJkzy5kmfKUcINRHZyo0koQ=
+github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.31 h1:kfVL5wAunCJycL6MOQ6aNh6PlAYEymflcjuKmrWUA0o=
+github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.31/go.mod h1:nWfRNDAppujCQgOUd43lKT4yeLv9z3nJ3bw1G3BgQKo=
 github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.29/go.mod h1:Dip3sIGv485+xerzVv24emnjX5Sg88utCL8fwGmCeWg=
 github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.2/go.mod h1:wRQv0nN6v9wDXuWThpovGQjqF1HFdcgWjporw14lS8k=
 github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.34/go.mod h1:p4VfIceZokChbA9FzMbRGz5OV+lekcVtHlPKEO0gSZY=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.30 h1:xM/Is9cKMHa8Jj8zkvWhvrFkZsXJV9E+BB4g0HW0duQ=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.30/go.mod h1:WueJeNDZvK1fMYEWJIkcivBfEzUkTpBhzlrUKKY8EuA=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.31 h1:Z8F3hfCY33IGpJjFAnv0wvtv1FIKj1GHmRDEYqy64tw=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.31/go.mod h1:aVyUoytEyOViR6jhq6jula0xkc5NfBE2hgeF6BvOrao=
 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.23/go.mod h1:mr6c4cHC+S/MMkrjtSlG4QA36kOznDep+0fga5L/fGQ=
 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.2/go.mod h1:tyF5sKccmDz0Bv4NrstEr+/9YkSPJHrcO7UsUKf7pWM=
 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.34/go.mod h1:dFZsC0BLo346mvKQLWmoJxT+Sjp+qcVR1tRVHQGOH9Q=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.30 h1:jn46zC9LdsVR/ZpMIJqMqb8hHv31BlLx3ulVqNspUOk=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.30/go.mod h1:1hTMsAgbdS/AtUi4bw8+gUuh1pceo+eXRLfpSuSQj3M=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.31 h1:hyOxUyXdh3AyjE93gBgsfziJag9ACwcs+ZpDBLzi8mw=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.31/go.mod h1:OERqI9k0draSLB8O8woxY3q25ZWTELRK4RRoLMuMZFo=
 github.com/aws/aws-sdk-go-v2/internal/ini v1.3.30/go.mod h1:vsbq62AOBwQ1LJ/GWKFxX8beUEYeRp/Agitrxee2/qM=
 github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0/go.mod h1:8tu/lYfQfFe6IGnaOdrpVgEL2IrrDOf6/m9RQum4NkY=
 github.com/aws/aws-sdk-go-v2/internal/ini v1.8.3/go.mod h1:H5O/EsxDWyU+LP/V8i5sm8cxoZgc2fdNR9bxlOFrQTo=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.31 h1:3GUprIsfmGcC5SACIyB0e7E0BM1O1b3Erl5CePYIAeQ=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.31/go.mod h1:7PuV1yl5e2xnUbm+RqvVg5i2iBM8EyijZNoI9wsOoOc=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.32 h1:0MrUL35H/Y4kdFfItoR5jCgtDQ4Z/8LudAoIHRfA4hE=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.32/go.mod h1:2tNZkuWz54arj8mHVf+8Y7cKkcD8Wr/fBpENgEXpjLc=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.1/go.mod h1:JKpmtYhhPs7D97NL/ltqz7yCkERFW5dOlHyVl66ZYF8=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.3/go.mod h1:0yKJC/kb8sAnmlYa6Zs3QVYqaC8ug2AbnNChv5Ox3uA=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13 h1:mbRIur/BiHK6SKPjoBIXSE/hJ6g6JGRLuxQy1jGjlN4=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13/go.mod h1:ITg9em2KbJx1s0y4aqRX5OYWG6HBZ5TVR//OdpEZ2CQ=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23 h1:9Fjh6fi/U5JEStVZijmaMpUwE/gvBJj7x2B/PjbO9To=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23/go.mod h1:iMoT2f1tClxrWAAnKCXjZQ6LOmfLrMG14wmnWpM+F14=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.24 h1:mdPwDQPqxlw9Sc62Nt15yjEcARaDbPXkjRYtXsUripo=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.24/go.mod h1:ls5ytnwLTcQaUu32fMYXFI3MjpKuTwL840PAm9iqyEg=
 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.23/go.mod h1:9uPh+Hrz2Vn6oMnQYiUi/zbh3ovbnQk19YKINkQny44=
 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.2/go.mod h1:Ru7vg1iQ7cR4i7SZ/JTLYN9kaXtbL69UdgG0OQWQxW0=
 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.15/go.mod h1:SwFBy2vjtA0vZbjjaFtfN045boopadnoVPhu4Fv66vY=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.30 h1:/Z5jmNrKsSD7EmDjzAPsm/3L9IuOkzaynklJZ1qX7S4=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.30/go.mod h1:lEzEZnOosE7zi8Z6royW1cFJTD9fpab4Ul1SBrllewk=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31 h1:uao4A3QZ5UmB326V6KF+qRpv9Tjz7IlnlnTbbANntlU=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31/go.mod h1:I/1+z0VwL1GhQyLgkoHDlygpUZ+iTAwOQ/NsftiUL2I=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.104.2 h1:bAY6O/TDv1HQnvylh9E247IyIKsUWUt2G965S7qX110=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.104.2/go.mod h1:zdmCoFO/dSI7GlrwsPqFJI+WlFnSU4Tc8TJnlXrM1Do=
-github.com/aws/aws-sdk-go-v2/service/signin v1.2.2 h1:69JEZSDTQ+UNbTWQJCZMmbpQb5sfc79KUt0O7Pyfjmo=
-github.com/aws/aws-sdk-go-v2/service/signin v1.2.2/go.mod h1:mxC0nT/C8wMMS97DemZPzvUZxvIt+2Iq+eS3JdFZGgg=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.31 h1:w2SIhW92DZPFrSL4ksVCr8IYff5OZwIcxg8+95tzvAI=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.31/go.mod h1:wAhpCQbkov+IcvjozJbd2xRCoZybUEHNkcFunssNACg=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.32 h1:jWXtZdCnhXa9sGFixRaU2AxT4DIVse9HS4E2f+/KwV0=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.32/go.mod h1:9JS1UpfVvyD/ZPX8GsKb/Pq8scEM+7GP5fqh9SwH7po=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.106.0 h1:7QZWVJZWzHivHWIa+5TELLaBBkbuoj0GPwQtMlJ0sqk=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.106.0/go.mod h1:fcvq5L7dK+5cQFicEJwpI6e6Wn8NY2i6yT5wRLYVc7s=
+github.com/aws/aws-sdk-go-v2/service/signin v1.5.0 h1:OHH5iTQvVGmfHjX/5Q+vFuA/Rf2x6/95aJ/75QCQSm4=
+github.com/aws/aws-sdk-go-v2/service/signin v1.5.0/go.mod h1:mCF3AK9PpL49oOrhniUXWAfhVBVQ/XbytoE5eccZUIs=
 github.com/aws/aws-sdk-go-v2/service/sso v1.12.3/go.mod h1:jtLIhd+V+lft6ktxpItycqHqiVXrPIRjWIsFIlzMriw=
 github.com/aws/aws-sdk-go-v2/service/sso v1.20.1/go.mod h1:RsYqzYr2F2oPDdpy+PdhephuZxTfjHQe7SOBcZGoAU8=
 github.com/aws/aws-sdk-go-v2/service/sso v1.25.2/go.mod h1:qs4a9T5EMLl/Cajiw2TcbNt2UNo/Hqlyp+GiuG4CFDI=
-github.com/aws/aws-sdk-go-v2/service/sso v1.31.5 h1:xlK3Tdc8FO7Tq1k0+hL+otF33glj+dE+qeM5iINiDvU=
-github.com/aws/aws-sdk-go-v2/service/sso v1.31.5/go.mod h1:u8af9Nqkmqnr96f7v9nHqzZT9XBwbXEkTiqT4ROuJSE=
+github.com/aws/aws-sdk-go-v2/service/sso v1.33.0 h1:CaJyYhxBE0M/HJX/YvSaSmQlsI91VHB0lKU8LtLxL3A=
+github.com/aws/aws-sdk-go-v2/service/sso v1.33.0/go.mod h1:+e6BMRMPjBQoCw/WovYR9GLy2IU0z4Q77smOB1DraSg=
 github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.3/go.mod h1:zVwRrfdSmbRZWkUkWjOItY7SOalnFnq/Yg2LVPqDjwc=
 github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.1/go.mod h1:YjAPFn4kGFqKC54VsHs5fn5B6d+PCY2tziEa3U/GB5Y=
 github.com/aws/aws-sdk-go-v2/service/ssooidc v1.30.0/go.mod h1:MlYRNmYu/fGPoxBQVvBYr9nyr948aY/WLUvwBMBJubs=
-github.com/aws/aws-sdk-go-v2/service/ssooidc v1.36.8 h1:yX1IbiBfC7SdEgDwIGnRaZyPPDRbQPDOJxl8102PcGk=
-github.com/aws/aws-sdk-go-v2/service/ssooidc v1.36.8/go.mod h1:DMPWJBjYs6+3+f/qhBFEFPPlQ6NlhWjai3dJNvipJ84=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.0 h1:tC323YV77QdafeBr6LUhLDTsboyuyHLNRwAyCP44kGU=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.0/go.mod h1:SfLK1sgviHmbI+MozR9iDwDjL4cdCVZtahsjoR+z7wg=
 github.com/aws/aws-sdk-go-v2/service/sts v1.18.4/go.mod h1:1mKZHLLpDMHTNSYPJ7qrcnCQdHCWsNQaT0xRvq2u80s=
 github.com/aws/aws-sdk-go-v2/service/sts v1.28.1/go.mod h1:uQ7YYKZt3adCRrdCBREm1CD3efFLOUNH77MrUCvx5oA=
 github.com/aws/aws-sdk-go-v2/service/sts v1.33.17/go.mod h1:cQnB8CUnxbMU82JvlqjKR2HBOm3fe9pWorWBza6MBJ4=
-github.com/aws/aws-sdk-go-v2/service/sts v1.43.5 h1:T3ANO8QWDbzQD8f4+UaX+fvJlyGnOFMKLbW+NGBHg04=
-github.com/aws/aws-sdk-go-v2/service/sts v1.43.5/go.mod h1:9gdl4RrflIdpDb2TlXshWgR1F9TeCkvqDx77Vpr4Z/Q=
+github.com/aws/aws-sdk-go-v2/service/sts v1.45.0 h1:Pd6PNlp4t8PTXxqzstICl52Wsy78vpjFZ7PRUj44mJc=
+github.com/aws/aws-sdk-go-v2/service/sts v1.45.0/go.mod h1:rmQ0TnHzuLPmabgjPcsywhsSOmaBDgzR4zvDxSPsGdg=
 github.com/aws/smithy-go v1.13.5/go.mod h1:Tg+OJXh4MB2R/uN61Ko2f6hTZwB/ZYGOtib8J3gBHzA=
 github.com/aws/smithy-go v1.20.1/go.mod h1:krry+ya/rV9RDcV/Q16kpu6ypI4K2czasz0NC3qS14E=
 github.com/aws/smithy-go v1.22.2/go.mod h1:irrKGvNn1InZwb2d7fkIRNucdfwR8R+Ts3wxYa/cJHg=
 github.com/aws/smithy-go v1.22.3/go.mod h1:t1ufH5HMublsJYulve2RKmHDC15xu1f26kHCp/HgceI=
-github.com/aws/smithy-go v1.27.3 h1:F3Zb497UhhskkfpJmfkXswyo+t0sh9OTBnIHjogWbVY=
-github.com/aws/smithy-go v1.27.3/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
+github.com/aws/smithy-go v1.27.4 h1:JQcphmBN4f0q/sPqXqROIItRNV/hy10cgu7CsFy616M=
+github.com/aws/smithy-go v1.27.4/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/bazelbuild/rules_go v0.49.0/go.mod h1:Dhcz716Kqg1RHNWos+N6MlXNkjNP2EwZQ0LukRKJfMs=
@@ -602,6 +604,8 @@ github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+Ce
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs=
+github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM=
+github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ=
 github.com/boltdb/bolt v1.3.1 h1:JQmyP4ZBrce+ZQu0dY660FMfatumYDLun9hBCUVIkF4=
 github.com/boltdb/bolt v1.3.1/go.mod h1:clJnj/oiGkjum5o1McbSZDSLxVThjynRyGBgiAx27Ps=
 github.com/boombuler/barcode v1.0.0/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
@@ -697,8 +701,8 @@ github.com/containerd/console v1.0.5 h1:R0ymNeydRqH2DmakFNdmjR2k0t7UPuiOV/N/27/q
 github.com/containerd/console v1.0.5/go.mod h1:YynlIjWYF8myEu6sdkwKIvGQq+cOckRm6So2avqoYAk=
 github.com/containerd/containerd/api v1.11.1 h1:h8nfoDW9+fNsC/9TwiAHj8B1GzXKtR4eFtkhi/X5RLU=
 github.com/containerd/containerd/api v1.11.1/go.mod h1:CaQFRu+N1MtbgL6JDOJLUB1hCKESU1lD6MuTJhgtdlw=
-github.com/containerd/containerd/v2 v2.3.2 h1:eLven1YxRMkeiKu7IcMrPKE+gn8sGR1DqHbbshMEvWM=
-github.com/containerd/containerd/v2 v2.3.2/go.mod h1:rHKGm3VW6wNrINb3x8mNT+w7qYXFVElTt/8HTuxVhD4=
+github.com/containerd/containerd/v2 v2.3.3 h1:MUNBVVBTBpPll7KPh5GTvkC3cfG03PQLAHVdsUoue9k=
+github.com/containerd/containerd/v2 v2.3.3/go.mod h1:rHKGm3VW6wNrINb3x8mNT+w7qYXFVElTt/8HTuxVhD4=
 github.com/containerd/continuity v0.5.0 h1:7a85HZpCSs+1Zps0Ee3DPSuAWY+0SJM1JNM51nlEVDg=
 github.com/containerd/continuity v0.5.0/go.mod h1:/lNJvtJKUQStBzpVQ1+rasXO1LAWtUQssk28EZvJ3nE=
 github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI=
@@ -725,12 +729,14 @@ github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:ma
 github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
 github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY=
 github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
+github.com/cyphar/filepath-securejoin v0.6.0 h1:BtGB77njd6SVO6VztOHfPxKitJvd/VPT+OFBFMOi1Is=
+github.com/cyphar/filepath-securejoin v0.6.0/go.mod h1:A8hd4EnAeyujCJRrICiOWqjS1AX0a9kM5XL+NwKoYSc=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
 github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/dgraph-io/badger/v4 v4.9.2 h1:Wb5qw8gElqwV1a8msHTeQKova9b1V10heFKMIiPd80E=
-github.com/dgraph-io/badger/v4 v4.9.2/go.mod h1:nJjaJTUOSsQEBhsq209FmwCvMJzEA3e74RjZw6V2pQI=
+github.com/dgraph-io/badger/v4 v4.9.5 h1:zT46OMrF3ntqsfI3ynKp7hUkQrGlcK2CX5psQmH0iW0=
+github.com/dgraph-io/badger/v4 v4.9.5/go.mod h1:Xa9dAupjbwAacupWFCpa6YEn9E1PjBXkfZYr2I/8aWg=
 github.com/dgraph-io/ristretto/v2 v2.4.0 h1:I/w09yLjhdcVD2QV192UJcq8dPBaAJb9pOuMyNy0XlU=
 github.com/dgraph-io/ristretto/v2 v2.4.0/go.mod h1:0KsrXtXvnv0EqnzyowllbVJB8yBonswa2lTCK2gGo9E=
 github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
@@ -808,13 +814,13 @@ github.com/franela/goreq v0.0.0-20171204163338-bcd34c9993f8/go.mod h1:ZhphrRTfi2
 github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
 github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
-github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
-github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
+github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
+github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
 github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9Ufpk2AcUM=
 github.com/gabriel-vasile/mimetype v1.4.13/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
-github.com/go-chi/chi/v5 v5.3.0 h1:halUjDxhshgXHMrao5bB8eNBXo/rnzwr8m5m36glehM=
-github.com/go-chi/chi/v5 v5.3.0/go.mod h1:R+tYY2hNuVUUjxoPtqUdgBqevM9s9njzkTLutVsOCto=
+github.com/go-chi/chi/v5 v5.3.1 h1:3j4HZLGZQ3JpMCrPJF/Jl3mYJfWLKBfNJ6quurUGCf8=
+github.com/go-chi/chi/v5 v5.3.1/go.mod h1:R+tYY2hNuVUUjxoPtqUdgBqevM9s9njzkTLutVsOCto=
 github.com/go-fonts/dejavu v0.1.0/go.mod h1:4Wt4I4OU2Nq9asgDCteaAaWZOV24E+0/Pwo0gppep4g=
 github.com/go-fonts/dejavu v0.3.2/go.mod h1:m+TzKY7ZEl09/a17t1593E4VYW8L1VaBXHzFZOIjGEY=
 github.com/go-fonts/latin-modern v0.2.0/go.mod h1:rQVLdDMK+mK1xscDwsqM5J8U2jrRa3T0ecnM9pNujks=
@@ -1103,6 +1109,8 @@ github.com/hamba/avro/v2 v2.17.2/go.mod h1:Q9YK+qxAhtVrNqOhwlZTATLgLA8qxG2vtvkhK
 github.com/hashicorp/consul/api v1.3.0/go.mod h1:MmDNSzIMUjNpY/mQ398R4bk2FnqQLoPndWW5VkKPlCE=
 github.com/hashicorp/consul/sdk v0.3.0/go.mod h1:VKf9jXwCTEY1QZP2MOLRhb5i/I/ssyNV1vwHyQBF0x8=
 github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
+github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
+github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
 github.com/hashicorp/go-cleanhttp v0.5.0/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80=
 github.com/hashicorp/go-cleanhttp v0.5.1/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80=
 github.com/hashicorp/go-hclog v1.6.3 h1:Qr2kF+eVWjTiYmU7Y31tYlP1h0q/X3Nl3tPGdaB11/k=
@@ -1118,6 +1126,8 @@ github.com/hashicorp/go-msgpack v0.5.5/go.mod h1:ahLV/dePpqEmjfWmKiqvPkv/twdG7iP
 github.com/hashicorp/go-msgpack/v2 v2.1.5 h1:Ue879bPnutj/hXfmUk6s/jtIK90XxgiUIcXRl656T44=
 github.com/hashicorp/go-msgpack/v2 v2.1.5/go.mod h1:bjCsRXpZ7NsJdk45PoCQnzRGDaK8TKm5ZnDI/9y3J4M=
 github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk=
+github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
+github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
 github.com/hashicorp/go-retryablehttp v0.5.3/go.mod h1:9B5zBasrRhHXnJnui7y6sL7es7NDiJgTc6Er0maI1Xs=
 github.com/hashicorp/go-rootcerts v1.0.0/go.mod h1:K6zTfqpRlCUIjkwsN4Z+hiSfzSTQa6eBIzfwKfwNnHU=
 github.com/hashicorp/go-sockaddr v1.0.0/go.mod h1:7Xibr9yA9JjQq1JpNB2Vw7kxv8xerXegt+ozgdvDeDU=
@@ -1266,6 +1276,8 @@ github.com/mdlayher/packet v1.1.2/go.mod h1:GEu1+n9sG5VtiRE4SydOmX5GTwyyYlteZiFU
 github.com/mdlayher/socket v0.5.1 h1:VZaqt6RkGkt2OE9l3GcC6nZkqD3xKeQLyfleW/uBcos=
 github.com/mdlayher/socket v0.5.1/go.mod h1:TjPLHI1UgwEv5J1B5q0zTZq12A/6H7nKmtTanQE37IQ=
 github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
+github.com/miekg/dns v1.1.72 h1:vhmr+TF2A3tuoGNkLDFK9zi36F2LS+hKTRW0Uf8kbzI=
+github.com/miekg/dns v1.1.72/go.mod h1:+EuEPhdHOsfk6Wk5TT2CzssZdqkmFhf8r+aVyDEToIs=
 github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8/go.mod h1:mC1jAcsrzbxHt8iiaC+zU4b1ylILSosueou12R++wfY=
 github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3/go.mod h1:RagcQ7I8IeTMnF8JTXieKnO4Z6JCsikNEzj0DwauVzE=
 github.com/minio/crc64nvme v1.1.1 h1:8dwx/Pz49suywbO+auHCBpCtlW1OfpcLN7wYgVR6wAI=
@@ -1282,6 +1294,8 @@ github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh
 github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
 github.com/moby/locker v1.0.1 h1:fOXqR41zeveg4fFODix+1Ch4mj/gT0NE1XJbp/epuBg=
 github.com/moby/locker v1.0.1/go.mod h1:S7SDdo5zpBK84bzzVlKr2V0hz+7x9hWbYC/kq7oQppc=
+github.com/moby/sys/capability v0.4.0 h1:4D4mI6KlNtWMCM1Z/K0i7RV1FkX+DBDHKVJpCndZoHk=
+github.com/moby/sys/capability v0.4.0/go.mod h1:4g9IK291rVkms3LKCDOoYlnV8xKwoDTpIrNEE35Wq0I=
 github.com/moby/sys/mountinfo v0.7.2 h1:1shs6aH5s4o5H2zQLn796ADW1wMrIwHsyJ2v9KouLrg=
 github.com/moby/sys/mountinfo v0.7.2/go.mod h1:1YOa8w8Ih7uW0wALDUgT1dTTSBrZ+HiBLGws92L2RU4=
 github.com/moby/sys/sequential v0.6.0 h1:qrx7XFUd/5DxtqcoH1h438hF5TmOvzC/lspjy7zgvCU=
@@ -1303,10 +1317,12 @@ github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELU
 github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
 github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
 github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
-github.com/mulgadc/predastore v1.11.0 h1:FD/Gk7cXhMZNcRBXectwBcR7jsMoS77WM3jcFgNVzY0=
-github.com/mulgadc/predastore v1.11.0/go.mod h1:gPrTLe9jY3CQyNtlpFurzUp2D0e8wPtVw/v48BmkhKA=
-github.com/mulgadc/viperblock v1.11.0 h1:/xIDWhwRIKkWiD1pvXH/f3aQe2SBH6MYQAlStNdek9Y=
-github.com/mulgadc/viperblock v1.11.0/go.mod h1:B4upzru+3opexPHNp6jkIqFvr8rdNs9lVVb+FJUd+9M=
+github.com/mulgadc/northstar v1.14.0 h1:8JzDcS4k3SN5PF3jBdB8x9FOWC24eITCClmOdwpoY8Q=
+github.com/mulgadc/northstar v1.14.0/go.mod h1:6EArbWIQUrCFQ+XZ5ZJrGS/ep/u/rF6l+dKZh5vVIRs=
+github.com/mulgadc/predastore v1.14.0 h1:vG+egKIP08SLql4hDUwke31ritValUYaDbyH39Ta6qw=
+github.com/mulgadc/predastore v1.14.0/go.mod h1:C/mlefzQgqRAXNMbKc/rmW5BVSoB3R0OPHsgOi/islI=
+github.com/mulgadc/viperblock v1.14.0 h1:Q6REPTPHh+3fnkjbyBSfLXWSppgiLsEcxwuvaHow0LY=
+github.com/mulgadc/viperblock v1.14.0/go.mod h1:5ENQb0OrjTplIRiLShRW/oNWOzbFvSCFgo+PkNH4858=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
@@ -1340,6 +1356,10 @@ github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJw
 github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M=
 github.com/opencontainers/runtime-spec v1.3.0 h1:YZupQUdctfhpZy3TM39nN9Ika5CBWT5diQ8ibYCRkxg=
 github.com/opencontainers/runtime-spec v1.3.0/go.mod h1:jwyrGlmzljRJv/Fgzds9SsS/C5hL+LL3ko9hs6T5lQ0=
+github.com/opencontainers/runtime-tools v0.9.1-0.20251114084447-edf4cb3d2116 h1:tAKu3NkKWZYpqBSOJKwTxT1wIGueiF7gcmcNgr5pNTY=
+github.com/opencontainers/runtime-tools v0.9.1-0.20251114084447-edf4cb3d2116/go.mod h1:DKDEfzxvRkoQ6n9TGhxQgg2IM1lY4aM0eaQP4e3oElw=
+github.com/opencontainers/selinux v1.13.1 h1:A8nNeceYngH9Ow++M+VVEwJVpdFmrlxsN22F+ISDCJE=
+github.com/opencontainers/selinux v1.13.1/go.mod h1:S10WXZ/osk2kWOYKy1x2f/eXF5ZHJoUs8UU/2caNRbg=
 github.com/opentracing-contrib/go-observer v0.0.0-20170622124052-a52f23424492/go.mod h1:Ngi6UdF0k5OKD5t5wlmGhe/EDKPoUM3BXZSSfIuJbis=
 github.com/opentracing/basictracer-go v1.0.0/go.mod h1:QfBfYuafItcjQuMwinw9GhYKwFXS9KnPs5lxoYwgW74=
 github.com/opentracing/opentracing-go v1.0.2/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
@@ -1355,8 +1375,8 @@ github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FI
 github.com/pascaldekloe/goe v0.1.0 h1:cBOtyMzM9HTpWjXfbbunk26uA6nG3a8n06Wieeh0MwY=
 github.com/pascaldekloe/goe v0.1.0/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
 github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=
-github.com/pelletier/go-toml/v2 v2.4.2 h1:M2fKKbmyvI+hGId/D0W64qDBMVhJnNR10O5gIbMc//Q=
-github.com/pelletier/go-toml/v2 v2.4.2/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
+github.com/pelletier/go-toml/v2 v2.4.3 h1:GTRvJQutkOSftxIFD5xw9aepkYNuPWmVJpffdDPYVpY=
+github.com/pelletier/go-toml/v2 v2.4.3/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
 github.com/performancecopilot/speed v3.0.0+incompatible/go.mod h1:/CLtqpZ5gBg1M9iaPbIdPPGyKcA8hKdoy6hAWba7Yac=
 github.com/phpdave11/gofpdf v1.4.2/go.mod h1:zpO6xFn9yxo3YLyMvW8HcKWVdbNqgIfOOp2dXMnm1mY=
 github.com/phpdave11/gofpdi v1.0.12/go.mod h1:vBmVV0Do6hSBHC8uKUQ71JGW+ZGQq74llk/7bXwjDoI=
@@ -1455,8 +1475,8 @@ github.com/pterm/pterm v0.12.83 h1:ie+YmGmA727VuhxBlyGr74Ks+7McV6kT99IB8EU80aA=
 github.com/pterm/pterm v0.12.83/go.mod h1:xlgc6bFWyJIMtmLJvGim+L7jhSReilOlOnodeIYe4Tk=
 github.com/quic-go/go-ossfuzz-seeds v0.1.0 h1:APacT+iIaNF6fd8AGEiN3bT/Jtkd2jz4v4TzM7MFjy0=
 github.com/quic-go/go-ossfuzz-seeds v0.1.0/go.mod h1:3IOHRbJIc+L6YKMwfDtJAM9Vj9k0YY4muhuyUYk5tbk=
-github.com/quic-go/quic-go v0.60.0 h1:xcQioE8OM66UQLeUMHltK1CCcOu3JbVB4JAQdDQSB+0=
-github.com/quic-go/quic-go v0.60.0/go.mod h1:wpKpjmPpftl30sL6pFh7REVpjbcCVy4zt2vDyK1TuJk=
+github.com/quic-go/quic-go v0.61.0 h1:ui88A53s8MSVYLC56en0KQ17HARk+9986Dn0SBfKNvA=
+github.com/quic-go/quic-go v0.61.0/go.mod h1:9So2anK4Tp22URSQq00k+Vo2PNkle96ycDPDHL4s9vs=
 github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
 github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
 github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
@@ -1526,6 +1546,7 @@ github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+
 github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
 github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
+github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
 github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
@@ -1557,6 +1578,12 @@ github.com/u-root/uio v0.0.0-20230220225925-ffce2a382923 h1:tHNk7XK9GkmKUR6Gh8gV
 github.com/u-root/uio v0.0.0-20230220225925-ffce2a382923/go.mod h1:eLL9Nub3yfAho7qB0MzZizFhTU2QkLeoVsWdHtDW264=
 github.com/urfave/cli v1.20.0/go.mod h1:70zkFmudgCuE/ngEzBv17Jvp/497gISqfk5gWijbERA=
 github.com/urfave/cli v1.22.1/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0=
+github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb h1:zGWFAtiMcyryUHoUjUJX0/lt1H2+i2Ka2n+D3DImSNo=
+github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU=
+github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 h1:EzJWgHovont7NscjpAxXsDA8S8BMYve8Y5+7cuRE7R0=
+github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415/go.mod h1:GwrjFmJcFw6At/Gs6z4yjiIwzuJ1/+UwLxMQDVQXShQ=
+github.com/xeipuuv/gojsonschema v1.2.0 h1:LhYJRs+L4fBtjZUfuSZIKGeVu0QRy8e5Xi7D17UxZ74=
+github.com/xeipuuv/gojsonschema v1.2.0/go.mod h1:anYRn/JVcOK2ZgGU+IjEV4nwlhoK5sQluxsYJ78Id3Y=
 github.com/xhit/go-str2duration v1.2.0/go.mod h1:3cPSlfZlUHVlneIVfePFWcJZsuwf+P1v2SRTV4cUmp4=
 github.com/xhit/go-str2duration/v2 v2.1.0/go.mod h1:ohY8p+0f07DiV6Em5LKB0s2YpLtXVyJfNt1+BlmyAsU=
 github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
@@ -1596,6 +1623,8 @@ go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo=
 go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
 go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
 go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
+go.opentelemetry.io/contrib/bridges/otelslog v0.19.0 h1:5RgvxieNq9tS3ewrV1vnODvbHPfKUIJcYtF9Cvz+6aQ=
+go.opentelemetry.io/contrib/bridges/otelslog v0.19.0/go.mod h1:iTBIdNwx/xmUhfgJs6+84S4dIK059811cO1eUBjKcHY=
 go.opentelemetry.io/contrib/detectors/gcp v1.28.0/go.mod h1:9BIqH22qyHWAiZxQh0whuJygro59z+nbMVuc7ciiGug=
 go.opentelemetry.io/contrib/detectors/gcp v1.29.0/go.mod h1:GW2aWZNwR2ZxDLdv8OyC2G8zkRoQBuURgV7RPQgcPoU=
 go.opentelemetry.io/contrib/detectors/gcp v1.31.0/go.mod h1:tzQL6E1l+iV44YFTkcAeNQqzXUiekSYP9jjJjXwEd00=
@@ -1658,6 +1687,8 @@ go.opentelemetry.io/otel v1.42.0/go.mod h1:lJNsdRMxCUIWuMlVJWzecSMuNjE7dOYyWlqOX
 go.opentelemetry.io/otel v1.43.0/go.mod h1:JuG+u74mvjvcm8vj8pI5XiHy1zDeoCS2LB1spIq7Ay0=
 go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
 go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
+go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.20.0 h1:rydZ9sxbcFdm/oWrVyfLTjHIygMgv0bEeMd+3B/BvoM=
+go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.20.0/go.mod h1:earQ25dooT0Hhspq59DZ8YCC50jWfOlFEeWoxy/P444=
 go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.44.0 h1:SUplec5dp06reu1zaXmOXdvqH398taqrDXqUl99jxSc=
 go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.44.0/go.mod h1:ho2g4N+ane+swq5I/VBkKWnRDY4kUINH3FuqyZqX/Ug=
 go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.44.0 h1:4YsVu3B8+3qtWYYrsUYgn0OG78pN0rnNPRGX4SbokQI=
@@ -1669,6 +1700,8 @@ go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.29.0/go.mod h1:BLbf7zb
 go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.35.0/go.mod h1:U2R3XyVPzn0WX7wOIypPuptulsMcPDPs/oiSVOMVnHY=
 go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.36.0/go.mod h1:dowW6UsM9MKbJq5JTz2AMVp3/5iW5I/TStsk8S+CfHw=
 go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.42.0/go.mod h1:so9ounLcuoRDu033MW/E0AD4hhUjVqswrMF5FoZlBcw=
+go.opentelemetry.io/otel/log v0.20.0 h1:/5i0vuHxCLWUfChWG41K9wkM0jafruPw9NU1/RCJirs=
+go.opentelemetry.io/otel/log v0.20.0/go.mod h1:wOcMcjsZpG8x7Bak7IhSi/lg8wscV2C1VdrKCLPlt0E=
 go.opentelemetry.io/otel/metric v0.20.0/go.mod h1:598I5tYlH1vzBjn+BTuhzTCSb/9debfNp6R3s7Pr1eU=
 go.opentelemetry.io/otel/metric v1.21.0/go.mod h1:o1p3CA8nNHW8j5yuQLdc1eeqEaPfzug24uvsyIEJRWM=
 go.opentelemetry.io/otel/metric v1.22.0/go.mod h1:evJGjVpZv0mQ5QBRJoBF64yMuOf4xCWdXjK8pzFvliY=
@@ -1716,6 +1749,10 @@ go.opentelemetry.io/otel/sdk v1.42.0/go.mod h1:rGHCAxd9DAph0joO4W6OPwxjNTYWghRWm
 go.opentelemetry.io/otel/sdk v1.43.0/go.mod h1:P+IkVU3iWukmiit/Yf9AWvpyRDlUeBaRg6Y+C58QHzg=
 go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58=
 go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0=
+go.opentelemetry.io/otel/sdk/log v0.20.0 h1:vM3xI7TQgKPiSghe6urZtAkyFY7SodrSpC83CffDFuY=
+go.opentelemetry.io/otel/sdk/log v0.20.0/go.mod h1:Knej2nmsTUzN79T2eeXdRsjjPcoxoq2pUyUHz9TFyyU=
+go.opentelemetry.io/otel/sdk/log/logtest v0.20.0 h1:OqdRZ1guyzamK3M6LlRsmGqRrjkHWw6WZOKKli5ELpg=
+go.opentelemetry.io/otel/sdk/log/logtest v0.20.0/go.mod h1:PuMIlm7zAt7c3z8zfOI5ox4iT1Z87We+PF6YoINux/M=
 go.opentelemetry.io/otel/sdk/metric v1.28.0/go.mod h1:cWPjykihLAPvXKi4iZc1dpER3Jdq2Z0YLse3moQUCpg=
 go.opentelemetry.io/otel/sdk/metric v1.29.0/go.mod h1:6zZLdCl2fkauYoZIOn/soQIDSWFmNSRcICarHfuhNJQ=
 go.opentelemetry.io/otel/sdk/metric v1.30.0/go.mod h1:waS6P3YqFNzeP01kuo/MBBYqaoBJl7efRQHOaydhy1Y=
@@ -1776,6 +1813,8 @@ go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/
 go.uber.org/multierr v1.3.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4=
 go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU=
 go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU=
+go.uber.org/nilaway v0.0.0-20260721205819-4a0f7653488a h1:CsRd824LAzMmlpu9nYv2PHOqPQYv4Zjd4kGM+g0ahS0=
+go.uber.org/nilaway v0.0.0-20260721205819-4a0f7653488a/go.mod h1:i9Hi8GeZFLBtUCXP73ZNSHKZ4K2JtxT0pLmo3sbJzlo=
 go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA=
 go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
 go.uber.org/zap v1.13.0/go.mod h1:zwrFLgMcdUuIBviXEYEH1YKNaOBnKXsx2IPda5bBwHM=
@@ -1842,8 +1881,8 @@ golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVo
 golang.org/x/crypto v0.49.0/go.mod h1:ErX4dUh2UM+CFYiXZRTcMpEcN8b/1gxEuv3nODoYtCA=
 golang.org/x/crypto v0.50.0/go.mod h1:3muZ7vA7PBCE6xgPX7nkzzjiUq87kRItoJQM1Yo8S+Q=
 golang.org/x/crypto v0.51.0/go.mod h1:8AdwkbraGNABw2kOX6YFPs3WM22XqI4EXEd8g+x7Oc8=
-golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
-golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
+golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
+golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
 golang.org/x/exp v0.0.0-20180321215751-8460e604b9de/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20180807140117-3d87b88a115f/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
@@ -1875,6 +1914,8 @@ golang.org/x/exp/shiny v0.0.0-20230801115018-d63ba01acd4b/go.mod h1:UH99kUObWAZk
 golang.org/x/exp/shiny v0.0.0-20230817173708-d852ddb80c63/go.mod h1:UH99kUObWAZkDnWqppdQe5ZhPYESUw8I0zVV1uWBR+0=
 golang.org/x/exp/shiny v0.0.0-20240707233637-46b078467d37/go.mod h1:3F+MieQB7dRYLTmnncoFbb1crS5lfQoTfDgQy6K4N0o=
 golang.org/x/exp/shiny v0.0.0-20241009180824-f66d83c29e7c/go.mod h1:3F+MieQB7dRYLTmnncoFbb1crS5lfQoTfDgQy6K4N0o=
+golang.org/x/exp/typeparams v0.0.0-20260611194520-c48552f49976 h1:GTD/WuaexTazIG/SxLOz4rEKZPDVilmVVC2nz4xhwfE=
+golang.org/x/exp/typeparams v0.0.0-20260611194520-c48552f49976/go.mod h1:PqrXSW65cXDZH0k4IeUbhmg/bcAZDbzNz3byBpKCsXo=
 golang.org/x/image v0.0.0-20180708004352-c73c2afc3b81/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs=
 golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
 golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
@@ -1955,8 +1996,8 @@ golang.org/x/mod v0.32.0/go.mod h1:SgipZ/3h2Ci89DlEtEXWUk/HteuRin+HHhN+WbNhguU=
 golang.org/x/mod v0.33.0/go.mod h1:swjeQEj+6r7fODbD2cqrnje9PnziFuw4bmLbBZFrQ5w=
 golang.org/x/mod v0.34.0/go.mod h1:ykgH52iCZe79kzLLMhyCUzhMci+nQj+0XkbXpNYtVjY=
 golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU=
-golang.org/x/mod v0.36.0 h1:JJjpVx6myfUsUdAzZuOSTTmRE0PfZeNWzzvKrP7amb4=
-golang.org/x/mod v0.36.0/go.mod h1:moc6ELqsWcOw5Ef3xVprK5ul/MvtVvkIXLziUOICjUQ=
+golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
+golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -2162,8 +2203,9 @@ golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
 golang.org/x/sync v0.18.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
 golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
 golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
-golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
 golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
+golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
+golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
 golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -2306,8 +2348,8 @@ golang.org/x/sys v0.42.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
 golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
 golang.org/x/sys v0.44.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
 golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
-golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
-golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
+golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
 golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
 golang.org/x/telemetry v0.0.0-20240521205824-bda55230c457/go.mod h1:pRgIJT+bRLFKnoM1ldnzKoxTIn14Yxz928LQRYYgIN0=
 golang.org/x/telemetry v0.0.0-20250710130107-8d8967aff50b/go.mod h1:4ZwOYna0/zsOKwuR5X/m0QFOJpSZvAxFfkQT+Erd9D4=
@@ -2320,8 +2362,8 @@ golang.org/x/telemetry v0.0.0-20260109210033-bd525da824e2/go.mod h1:b7fPSJ0pKZ3c
 golang.org/x/telemetry v0.0.0-20260209163413-e7419c687ee4/go.mod h1:g5NllXBEermZrmR51cJDQxmJUHUOfRAaNyWBM+R+548=
 golang.org/x/telemetry v0.0.0-20260311193753-579e4da9a98c/go.mod h1:TpUTTEp9frx7rTdLpC9gFG9kdI7zVLFTFFlqaH2Cncw=
 golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa/go.mod h1:kHjTxDEnAu6/Nl9lDkzjWpR+bmKfxeiRuSDlsMb70gE=
-golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6 h1:HjU6IWBiAgRIdAJ9/y1rwCn+UELEmwV+VsTLzj/W4sE=
-golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6/go.mod h1:Eqhaxk/wZsWEH8CRxLwj6xzEJbz7k1EFGqx7nyCoabE=
+golang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57 h1:nwGZBCt+FnXUrGsj5vjzAsEmkcaFvd82BbOjECiFYZc=
+golang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57/go.mod h1:3AWMyWHS+caVoiEXpiq6+tzKA40J4vQT3MYr80ZtQpc=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
@@ -2369,8 +2411,8 @@ golang.org/x/term v0.40.0/go.mod h1:w2P8uVp06p2iyKKuvXIm7N/y0UCRt3UfJTfZ7oOpglM=
 golang.org/x/term v0.41.0/go.mod h1:3pfBgksrReYfZ5lvYM0kSO0LIkAl4Yl2bXOkKP7Ec2A=
 golang.org/x/term v0.42.0/go.mod h1:Dq/D+snpsbazcBG5+F9Q1n2rXV8Ma+71xEjTRufARgY=
 golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=
-golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc=
-golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y=
+golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
+golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -2415,8 +2457,8 @@ golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
 golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
 golang.org/x/text v0.36.0/go.mod h1:NIdBknypM8iqVmPiuco0Dh6P5Jcdk8lJL0CUebqK164=
 golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
-golang.org/x/text v0.38.0 h1:sXmwo9DwP3OK9EZ7PqAdaooSGozfl/3a6/xJcbzPRhE=
-golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4=
+golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
+golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
 golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
@@ -2531,14 +2573,14 @@ golang.org/x/tools v0.41.0/go.mod h1:XSY6eDqxVNiYgezAVqqCeihT4j1U2CCsqvH3WhQpnlg
 golang.org/x/tools v0.42.0/go.mod h1:Ma6lCIwGZvHK6XtgbswSoWroEkhugApmsXyrUmBhfr0=
 golang.org/x/tools v0.43.0/go.mod h1:uHkMso649BX2cZK6+RpuIPXS3ho2hZo4FVwfoy1vIk0=
 golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI=
-golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8=
-golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0=
+golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
+golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
 golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM=
 golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY=
 golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM=
 golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated/go.mod h1:RVAQXBGNv1ib0J382/DPCRS/BPnsGebyM1Gj5VSDpG8=
-golang.org/x/vuln v1.1.4 h1:Ju8QsuyhX3Hk8ma3CesTbO8vfJD9EvUBgHvkxHBzj0I=
-golang.org/x/vuln v1.1.4/go.mod h1:F+45wmU18ym/ca5PLTPLsSzr2KppzswxPP603ldA67s=
+golang.org/x/vuln v1.4.0 h1:FpmTZiV4PyqY3lFfuCkz1JftEXb/+8M2NEkjJM5TF4g=
+golang.org/x/vuln v1.4.0/go.mod h1:FJ7XyKs83nAdxQ7PMsia2PoynwZMJ/QajXVMBBIgFe8=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -3057,8 +3099,9 @@ google.golang.org/grpc v1.79.1/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhH
 google.golang.org/grpc v1.79.2/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ=
 google.golang.org/grpc v1.79.3/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ=
 google.golang.org/grpc v1.80.0/go.mod h1:ho/dLnxwi3EDJA4Zghp7k2Ec1+c2jqup0bFkw07bwF4=
-google.golang.org/grpc v1.81.1 h1:VnnIIZ88UzOOKLukQi+ImGz8O1Wdp8nAGGnvOfEIWQQ=
 google.golang.org/grpc v1.81.1/go.mod h1:xGH9GfzOyMTGIOXBJmXt+BX/V0kcdQbdcuwQ/zNw42I=
+google.golang.org/grpc v1.82.1 h1:NnAxzGRA0677vCa4BUkOAnO5+FfQqVl9iUXeD0IqcGE=
+google.golang.org/grpc v1.82.1/go.mod h1:yzTZ1TB1Z3SG+LIYaI+WiE8D5+PZ3ArnrSp8zF3+/ZA=
 google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0/go.mod h1:6Kw0yEErY5E/yWrBtf03jp27GLLJujG4z/JK95pnjjw=
 google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.3.0/go.mod h1:Dk1tviKTvMCz5tvh7t+fh94dhmQVHuCt2OzJB3CTW9Y=
 google.golang.org/grpc/examples v0.0.0-20201112215255-90f1b3ee835b/go.mod h1:IBqQ7wSUJ2Ep09a8rMWFsg4fmI2r38zwsq8a0GgxXpM=
@@ -3182,4 +3225,10 @@ rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=
 rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=
 rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=
 sigs.k8s.io/yaml v1.1.0/go.mod h1:UJmg0vDUVViEyp3mgSv9WPwZCDxu4rQW1olrI1uml+o=
+sigs.k8s.io/yaml v1.6.0 h1:G8fkbMSAFqgEFgh4b1wmtzDnioxFCUgTZhlbj5P9QYs=
+sigs.k8s.io/yaml v1.6.0/go.mod h1:796bPqUfzR/0jLAl6XjHl3Ck7MiyVv8dbTdyT3/pMf4=
 sourcegraph.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0/go.mod h1:hI742Nqp5OhwiqlzhgfbWU4mW4yO10fP+LoT9WOswdU=
+tags.cncf.io/container-device-interface v1.1.0 h1:RnxNhxF1JOu6CJUVpetTYvrXHdxw9j9jFYgZpI+anSY=
+tags.cncf.io/container-device-interface v1.1.0/go.mod h1:76Oj0Yqp9FwTx/pySDc8Bxjpg+VqXfDb50cKAXVJ34Q=
+tags.cncf.io/container-device-interface/specs-go v1.1.0 h1:QRZVeAceQM+zTZe12eyfuJuuzp524EKYwhmvLd+h+yQ=
+tags.cncf.io/container-device-interface/specs-go v1.1.0/go.mod h1:u86hoFWqnh3hWz3esofRFKbI261bUlvUfLKGrDhJkgQ=

internal/gwsign/signer_test.go

@@ -12,7 +12,7 @@ import (
 
 	"github.com/aws/aws-sdk-go-v2/aws"
 	"github.com/aws/aws-sdk-go-v2/credentials"
-	"github.com/mulgadc/predastore/auth"
+	"github.com/mulgadc/predastore/pkg/sigv4"
 )
 
 // failingProvider models a cold IMDS datapath: Retrieve always errors, as the
@@ -40,13 +40,14 @@ func TestSign_StaticVerifies(t *testing.T) {
 
 	var verifyErr error
 	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		req, err := auth.ParseReq(r)
+		// sigv4.Parse reads and hashes the body itself, so no explicit body hash.
+		sr, err := sigv4.Parse(r)
 		if err != nil {
 			verifyErr = err
 			http.Error(w, err.Error(), http.StatusBadRequest)
 			return
 		}
-		verifyErr = req.Verify(testSecretKey, testService, testRegion, auth.WithBodyHash(payloadHash))
+		_, verifyErr = sr.Verify(testSecretKey, testRegion, testService)
 		w.WriteHeader(http.StatusOK)
 	}))
 	defer srv.Close()

scripts/build-system-image.sh

@@ -72,8 +72,10 @@ for field in IMAGE_NAME IMAGE_SIZE; do
     fi
 done
 
-# Derived paths
-BUILD_DIR="/tmp/${IMAGE_NAME}-image-build"
+# Derived paths. BUILD_DIR defaults to /tmp, but Ubuntu GPU images are 16G and a
+# typical /tmp is a RAM-backed tmpfs too small to hold the qcow2 + expanded raw,
+# so allow an override onto a real disk (SYSTEM_IMAGE_BUILD_DIR).
+BUILD_DIR="${SYSTEM_IMAGE_BUILD_DIR:-/tmp}/${IMAGE_NAME}-image-build"
 
 case "$DISTRO" in
     alpine)
@@ -116,7 +118,7 @@ import_ami() {
         args+=(--ami-name "$AMI_NAME")
     fi
     if [[ -n "${SYSTEM_TAG:-}" ]]; then
-        args+=(--tag "$SYSTEM_TAG")
+        for _tag in ${SYSTEM_TAG}; do args+=(--tag "$_tag"); done
     fi
     spx admin images import "${args[@]}"
 }
@@ -263,6 +265,15 @@ else
     CUST+=(--memsize 2048)
 fi
 
+# The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
+# (127.0.0.53), which isn't running inside the appliance chroot, so DNS is
+# dead until package installs need it. Point it at a real resolver first, then
+# restore the stub symlink after package installs so the shipped AMI still
+# gets its DNS from systemd-resolved at boot like a stock cloud image.
+if [[ -n "${APK_PACKAGES:-}${APT_PACKAGES:-}" ]]; then
+    CUST+=(--run-command 'rm -f /etc/resolv.conf; printf "nameserver 1.1.1.1\nnameserver 8.8.8.8\n" > /etc/resolv.conf')
+fi
+
 # Packages
 if [[ "$DISTRO" == "alpine" ]] && [[ -n "${APK_PACKAGES:-}" ]]; then
     echo "Will install packages: ${APK_PACKAGES}"
@@ -336,6 +347,12 @@ else
     CUST+=(--run-command 'export DEBIAN_FRONTEND=noninteractive; apt-get clean; rm -rf /var/lib/apt/lists/* /tmp/* 2>/dev/null || true')
 fi
 
+# Restore the stock systemd-resolved stub symlink so the shipped AMI gets its
+# DNS from the instance's own resolver at boot, not the static appliance one.
+if [[ "$DISTRO" == "ubuntu" ]] && [[ -n "${APK_PACKAGES:-}${APT_PACKAGES:-}${SETUP_SCRIPT:-}" ]]; then
+    CUST+=(--run-command 'rm -f /etc/resolv.conf; ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf')
+fi
+
 echo "Customizing image (libguestfs appliance)..."
 "${CUST[@]}"
 
@@ -367,7 +384,9 @@ else
     fi
     if [[ -n "${SYSTEM_TAG:-}" ]]; then
         echo -e "    --distro ${DISTRO} --version ${DISTRO_VERSION} --arch x86_64 --boot-mode ${BOOT_MODE}${NAME_HINT} \\"
-        echo "    --tag ${SYSTEM_TAG}"
+        _tag_line=""
+        for _tag in ${SYSTEM_TAG}; do _tag_line="${_tag_line} --tag ${_tag}"; done
+        echo "   ${_tag_line}"
     else
         echo -e "    --distro ${DISTRO} --version ${DISTRO_VERSION} --arch x86_64 --boot-mode ${BOOT_MODE}${NAME_HINT}"
     fi

scripts/images/ecs-agent-gpu/ecs-agent.service

@@ -0,0 +1,22 @@
+[Unit]
+Description=Spinifex ECS container-instance agent
+# Resolves host identity from IMDSv2, registers + heartbeats + polls task
+# assignments through the AWS gateway over TLS+SigV4 (no NATS), and pulls
+# task images through containerd.
+Requires=containerd.service
+After=containerd.service ecs-ca-install.service network-online.target mulga-cdi-generate.service
+Wants=network-online.target
+
+[Service]
+Type=simple
+# Static config (gateway URL, CA, region, cluster, seeded IAM creds,
+# containerd socket) comes from /etc/spinifex-ecs/agent.env, dropped by
+# cloud-init at launch. Absent file is tolerated — the agent falls back to
+# its built-in defaults.
+EnvironmentFile=-/etc/spinifex-ecs/agent.env
+ExecStart=/usr/local/bin/ecs-agent
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target

scripts/images/ecs-agent-gpu/ecs-ca-install.service

@@ -0,0 +1,22 @@
+[Unit]
+Description=Install Spinifex gateway CA into the system trust (first boot)
+# The CA is delivered per-instance (not baked) at
+# /etc/spinifex-ecs/gateway-ca.pem by cloud-init user-data, mirroring how EKS
+# delivers EKS_GATEWAY_CA. No-op when the file is absent — containerd then
+# relies on the stock system trust.
+Before=containerd.service ecs-agent.service
+After=local-fs.target
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/bin/sh -c '\
+    if [ -f /etc/spinifex-ecs/gateway-ca.pem ]; then \
+        install -D -m 0644 /etc/spinifex-ecs/gateway-ca.pem /usr/local/share/ca-certificates/spinifex-gateway.crt && \
+        update-ca-certificates >/dev/null 2>&1; \
+    else \
+        echo "no gateway CA at /etc/spinifex-ecs/gateway-ca.pem; using stock system trust"; \
+    fi'
+
+[Install]
+WantedBy=multi-user.target

scripts/images/ecs-agent-gpu/manifest.conf

@@ -0,0 +1,36 @@
+IMAGE_NAME="ecs-agent-gpu"
+IMAGE_DESCRIPTION="Ubuntu ECS GPU container-instance — containerd + runc + ecs-agent + NVIDIA server driver + nvidia-container-toolkit"
+DISTRO="ubuntu"
+UBUNTU_VERSION="26.04"
+UBUNTU_CODENAME="resolute"
+
+# 16G — same headroom rationale as eks-node-gpu: kernel headers, NVIDIA
+# driver + DKMS build artefacts, nvidia-container-toolkit.
+IMAGE_SIZE="16G"
+
+# Ubuntu's containerd apt package ships its own systemd unit (unlike Alpine's
+# containerd-openrc, which needed one written) — nothing to author for it
+# here, just `systemctl enable containerd` below. nvidia-container-toolkit
+# lives in its own NVIDIA-hosted apt repo, not enabled on the base cloud
+# image, so it is added+installed in setup.sh rather than here (see
+# scripts/images/ubuntu-gpu-nvidia-setup.sh).
+APT_PACKAGES="containerd runc containernetworking-plugins ca-certificates curl iproute2 initramfs-tools"
+
+# ecs-agent is the only built binary (static, FIPS boot like the host build) —
+# BUILD_COMMANDS/INSTALL_BINARIES copied verbatim from scripts/images/ecs-agent/.
+BUILD_COMMANDS="CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags \"-s -w\" -o bin/ecs-agent ./cmd/ecs-agent"
+
+INSTALL_BINARIES="bin/ecs-agent:/usr/local/bin/ecs-agent"
+
+INSTALL_FILES="\
+scripts/images/ecs-agent/10-bridge.conflist:/etc/cni/net.d/10-bridge.conflist \
+scripts/images/ecs-agent-gpu/ecs-agent.service:/etc/systemd/system/ecs-agent.service \
+scripts/images/ecs-agent-gpu/ecs-ca-install.service:/etc/systemd/system/ecs-ca-install.service \
+scripts/images/ecs-agent-gpu/mulga-cdi-generate.service:/etc/systemd/system/mulga-cdi-generate.service"
+
+ENABLE_SERVICES="containerd ecs-ca-install ecs-agent mulga-cdi-generate"
+
+SETUP_SCRIPT="scripts/images/ecs-agent-gpu/setup.sh"
+
+AMI_NAME="spinifex-ecs-node-gpu"
+SYSTEM_TAG="spinifex:managed-by=ecs gpu-vendor=nvidia"

scripts/images/ecs-agent-gpu/mulga-cdi-generate.service

@@ -0,0 +1,16 @@
+[Unit]
+Description=Generate NVIDIA CDI spec for the container runtime (GPU-gated, first boot)
+After=systemd-modules-load.service
+Before=containerd.service ecs-agent.service
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+# Silently skip on non-GPU boots: only generate when an NVIDIA GPU is
+# actually present and the driver has bound.
+ExecCondition=/bin/sh -c 'command -v nvidia-smi >/dev/null 2>&1 && nvidia-smi -L >/dev/null 2>&1'
+ExecStartPre=/bin/mkdir -p /etc/cdi
+ExecStart=/usr/bin/nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
+
+[Install]
+WantedBy=multi-user.target

scripts/images/ecs-agent-gpu/setup.sh

@@ -0,0 +1,158 @@
+#!/bin/sh
+set -eu
+
+# setup.sh — guest customisation for the ecs-agent-gpu AMI (Ubuntu ECS GPU
+# container-instance).
+#
+# Runs inside the libguestfs appliance (via virt-customize --run) under
+# build-system-image.sh, after packages, binaries and INSTALL_FILES are
+# placed. Wires the CNI bin path containerd's CRI plugin expects (mirroring
+# scripts/images/ecs-agent/setup.sh, but Ubuntu's containernetworking-plugins
+# package lays out differently to Alpine's — see below), preps the containerd
+# content store + agent config dir, and bakes the headless NVIDIA driver +
+# nvidia-container-toolkit (steps reused from
+# scripts/images/ubuntu-gpu-nvidia-setup.sh — see install_nvidia_gpu_stack()
+# below, duplicated verbatim from scripts/images/eks-node-gpu/setup.sh so
+# both GPU node images stay consistent; each image builds standalone from
+# its own manifest so there is no shared-script mechanism to source from).
+#
+# Unlike scripts/images/ecs-agent/setup.sh there is no chmod dance for OpenRC
+# init scripts (systemd unit files don't need an exec bit) and no SYSLINUX/
+# serial-console tweaks (Ubuntu boots via GRUB/UEFI, not Alpine's extlinux).
+
+# Ubuntu's containernetworking-plugins package installs plugin binaries
+# directly under /usr/lib/cni (verified via `dpkg -L containernetworking-plugins`
+# — no /bin subdir, unlike Alpine's /usr/libexec/cni). containerd's CRI plugin
+# defaults its bin dir to /opt/cni/bin. Symlink so the baked bridge conflist
+# resolves its plugins when task networking is wired up.
+mkdir -p /opt/cni
+ln -sf /usr/lib/cni /opt/cni/bin
+
+# Content-store + agent config dirs.
+mkdir -p /var/lib/containerd /etc/spinifex-ecs
+
+# ── NVIDIA headless driver + container toolkit (node-image subset) ──────────
+# Reused from scripts/images/ubuntu-gpu-nvidia-setup.sh: kernel detection,
+# linux-headers-<kver>, the latest versioned nvidia-dkms-<ver>-server +
+# nvidia-utils-<ver>-server, `dkms install`, nvidia.ko verification,
+# `apt-mark hold` on the kernel/nvidia/libnvidia packages, the nouveau
+# modprobe blacklist, and `update-initramfs -u -k <kver>`.
+#
+# Dropped for a node image: pciutils/python3/git/curl-extras/htop/tmux/ffmpeg/
+# libgl1/libglib dev payload, and Docker CE entirely — GPU nodes run
+# containerd (this image) or k3s (scripts/images/eks-node-gpu), never Docker.
+# `nvidia-ctk runtime configure --runtime=docker` and the docker-group
+# cloud-init drop-in are dropped for the same reason; the runtime is instead
+# wired via CDI at first boot by mulga-cdi-generate.service, gated on an
+# NVIDIA GPU actually being present.
+install_nvidia_gpu_stack() {
+    KVER=$(ls /boot/vmlinuz-* 2>/dev/null | sort -V | tail -1 | sed 's|/boot/vmlinuz-||') || true
+    if [ -z "${KVER}" ]; then
+        # Ubuntu 26.04 minimal cloud image keeps the kernel on the ESP (not
+        # mounted in the appliance chroot), so /boot is empty — fall back to
+        # /lib/modules.
+        KVER=$(ls /lib/modules/ 2>/dev/null | sort -V | tail -1)
+    fi
+    if [ -z "${KVER}" ]; then
+        echo "ERROR: No kernel found in /boot or /lib/modules — base image may be missing a kernel"
+        exit 1
+    fi
+    echo "[nvidia-gpu-stack] target kernel: ${KVER}"
+
+    # Prevent kernel postinst hooks from attempting grub-install against the
+    # host disk — this fails silently inside the appliance and can abort the
+    # postinst, leaving /boot incomplete.
+    cat > /etc/kernel-img.conf <<'EOF'
+do_symlinks = yes
+do_bootloader = no
+do_initrd = yes
+link_in_boot = yes
+EOF
+
+    apt-get install -y --no-install-recommends "linux-headers-${KVER}"
+
+    # Ubuntu 26.04+ no longer ships unversioned meta-packages, so the branch is
+    # named explicitly. Pinned rather than resolved at build time: the number in
+    # the package name is a branch alias, not a driver version (nvidia-dkms-535-server
+    # installs 580.126.20), so picking the highest number sorts on a value that
+    # need not match what gets installed. Resolving also let identical source
+    # produce different drivers on different days. Bumping is a deliberate edit.
+    NVIDIA_VER=595
+    echo "[nvidia-gpu-stack] installing NVIDIA server driver branch: ${NVIDIA_VER}"
+    # The -open (GSP/open-kernel-module) DKMS variant, not the closed/proprietary
+    # one: Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
+    # initialize under the closed kernel module — RmInitAdapter fails with
+    # "requires use of the NVIDIA open kernel modules" (dmesg NVRM error 0x22).
+    # nvidia-utils has no separate open variant (userspace tools only).
+    apt-get install -y --no-install-recommends \
+        "nvidia-dkms-${NVIDIA_VER}-server-open" \
+        "nvidia-utils-${NVIDIA_VER}-server"
+
+    NVIDIA_DKMS_NAME=$(dkms status 2>/dev/null | awk -F'[,/ ]+' '/nvidia/{print $1; exit}')
+    NVIDIA_DKMS_VER=$(dkms status 2>/dev/null  | awk -F'[,/ ]+' '/nvidia/{print $2; exit}')
+    if [ -n "${NVIDIA_DKMS_NAME}" ] && [ -n "${NVIDIA_DKMS_VER}" ]; then
+        echo "[nvidia-gpu-stack] building NVIDIA DKMS module ${NVIDIA_DKMS_NAME}/${NVIDIA_DKMS_VER} for kernel ${KVER}..."
+        dkms install -m "${NVIDIA_DKMS_NAME}" -v "${NVIDIA_DKMS_VER}" --kernelver "${KVER}" --force
+    else
+        echo "WARNING: nvidia DKMS module not found in dkms status — driver install may have failed"
+    fi
+
+    NVIDIA_KO=$(find "/lib/modules/${KVER}/updates" -name "nvidia.ko*" 2>/dev/null | head -1)
+    if [ -z "${NVIDIA_KO}" ]; then
+        echo "ERROR: nvidia.ko not found under /lib/modules/${KVER}/updates after DKMS build"
+        exit 1
+    fi
+    echo "[nvidia-gpu-stack] NVIDIA kernel module: ${NVIDIA_KO}"
+
+    # Pin the exact kernel + headers + nvidia/libnvidia packages so
+    # unattended-upgrades cannot replace what the DKMS module was built for.
+    apt-mark hold "linux-image-${KVER}" "linux-headers-${KVER}"
+
+    NVIDIA_PKGS=$(dpkg --get-selections 'nvidia-*' 2>/dev/null | awk '/install$/{print $1}' || true)
+    LIBNVIDIA_PKGS=$(dpkg --get-selections 'libnvidia-*' 2>/dev/null | awk '/install$/{print $1}' || true)
+    # shellcheck disable=SC2086
+    [ -n "${NVIDIA_PKGS}" ]    && apt-mark hold ${NVIDIA_PKGS}
+    # shellcheck disable=SC2086
+    [ -n "${LIBNVIDIA_PKGS}" ] && apt-mark hold ${LIBNVIDIA_PKGS}
+
+    # nvidia.ko and nouveau conflict; without this the NVIDIA driver will not
+    # bind after boot.
+    cat > /etc/modprobe.d/blacklist-nouveau.conf <<'EOF'
+blacklist nouveau
+options nouveau modeset=0
+EOF
+
+    # Load the NVIDIA modules at boot so /dev/nvidia* exist before CDI
+    # generation and the container runtime start — a headless node never runs
+    # an X server or CUDA app to trigger the on-demand load.
+    cat > /etc/modules-load.d/nvidia.conf <<'EOF'
+nvidia
+nvidia_uvm
+nvidia_modeset
+EOF
+
+    # nvidia-container-toolkit lives in its own NVIDIA-hosted apt repo, which
+    # is not enabled on the base Ubuntu cloud image — add it here (same
+    # gpg key + list as scripts/images/ubuntu-gpu-nvidia-setup.sh) and
+    # install only the headless toolkit (no Docker CE).
+    apt-get install -y --no-install-recommends gnupg
+    mkdir -p /usr/share/keyrings /etc/apt/sources.list.d
+    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
+        | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
+    curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
+        | sed 's|deb https://|deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://|g' \
+        > /etc/apt/sources.list.d/nvidia-container-toolkit.list
+    apt-get update -qq
+    apt-get install -y --no-install-recommends nvidia-container-toolkit
+    apt-mark hold nvidia-container-toolkit nvidia-container-toolkit-base \
+        libnvidia-container1 libnvidia-container-tools 2>/dev/null || true
+
+    # Rebuild initramfs with nouveau blacklisted and the NVIDIA module included.
+    update-initramfs -u -k "${KVER}"
+
+    echo "[nvidia-gpu-stack] done: kernel=${KVER}, NVIDIA headless driver + nvidia-container-toolkit installed"
+}
+
+install_nvidia_gpu_stack
+
+echo "[ecs-agent-gpu-setup] done"

scripts/images/eks-node-gpu/99-mulga-ebs-byid.rules

@@ -0,0 +1,14 @@
+# Mint the EBS-style /dev/disk/by-id symlink the upstream aws-ebs-csi-driver
+# node plugin resolves, for hot-attached Viperblock virtio-blk volumes (vd*,
+# serial = volume-id). This is the udev equivalent of the Alpine mdev rule
+# wired by scripts/images/eks-node/setup.sh into /etc/mdev.conf.
+#
+# The shared helper /usr/local/sbin/mulga-ebs-byid (reused verbatim from
+# scripts/images/eks-node/mulga-ebs-byid.sh) reads the device name from
+# $MDEV or, if unset, $1 — so passing the kernel name via %k as the sole
+# argument works without any udev-specific shim. udev already sets ACTION to
+# add/remove, matching the values the helper's case statement expects.
+# Already-attached devices at boot are coldplugged separately by
+# mulga-ebs-byid.service; this rule only covers hotplug events.
+ACTION=="add", SUBSYSTEM=="block", KERNEL=="vd[a-z]", RUN+="/usr/local/sbin/mulga-ebs-byid %k"
+ACTION=="remove", SUBSYSTEM=="block", KERNEL=="vd[a-z]", RUN+="/usr/local/sbin/mulga-ebs-byid %k"

scripts/images/eks-node-gpu/k3s-agent-prestart.sh

@@ -0,0 +1,48 @@
+#!/bin/sh
+set -eu
+
+# k3s-agent-prestart — reproduces the start_pre logic from
+# scripts/images/eks-node/k3s-agent.initd for the systemd unit: requires
+# K3S_URL + K3S_TOKEN from /etc/spinifex-eks/agent.env (sourced by the unit's
+# own EnvironmentFile=), then resolves the kubelet provider-id from IMDSv2
+# (aws:///<az>/<instance-id>, EKS parity so the AWS Load Balancer Controller
+# can resolve nodes for ip-mode target registration) and writes it as an
+# extra k3s agent argument for the unit's ExecStart to source.
+#
+# Run as ExecStartPre=; a nonzero exit here fails k3s-agent.service before k3s
+# is ever invoked, same as the OpenRC start_pre returning 1.
+
+ENV_FILE=/etc/spinifex-eks/agent.env
+DROPIN=/run/k3s-agent-extra-args.env
+LOGTAG="k3s-agent-prestart"
+
+# EnvironmentFile= already injects agent.env into this process' environment,
+# but source it directly too so this script also works if invoked standalone.
+if [ -f "${ENV_FILE}" ]; then
+    # shellcheck disable=SC1090
+    . "${ENV_FILE}"
+fi
+
+if [ -z "${K3S_URL:-}" ] || [ -z "${K3S_TOKEN:-}" ]; then
+    echo "[${LOGTAG}] K3S_URL and K3S_TOKEN must be set in ${ENV_FILE}" >&2
+    exit 1
+fi
+
+IMDS=http://169.254.169.254/latest
+CURL="curl -fsS --connect-timeout 3 --max-time 5"
+
+EXTRA_ARGS=""
+TOK=$(${CURL} -X PUT -H 'X-aws-ec2-metadata-token-ttl-seconds: 120' "${IMDS}/api/token" 2>/dev/null || true)
+if [ -n "${TOK}" ]; then
+    IID=$(${CURL} -H "X-aws-ec2-metadata-token: ${TOK}" "${IMDS}/meta-data/instance-id" 2>/dev/null || true)
+    AZ=$(${CURL} -H "X-aws-ec2-metadata-token: ${TOK}" "${IMDS}/meta-data/placement/availability-zone" 2>/dev/null || true)
+    if [ -n "${IID}" ] && [ -n "${AZ}" ]; then
+        EXTRA_ARGS="--kubelet-arg=provider-id=aws:///${AZ}/${IID}"
+    else
+        echo "[${LOGTAG}] IMDS instance-id/az empty; keeping default providerID" >&2
+    fi
+else
+    echo "[${LOGTAG}] IMDS token request failed; keeping default providerID" >&2
+fi
+
+echo "K3S_AGENT_EXTRA_ARGS=${EXTRA_ARGS}" > "${DROPIN}"

scripts/images/eks-node-gpu/k3s-agent.service

@@ -0,0 +1,41 @@
+[Unit]
+Description=K3s Kubernetes agent (Mulga EKS GPU worker)
+# K3S_URL points at the cluster's NLB endpoint (https://{cluster}.{account}
+# .eks.{region}.{suffix}:443) so the agent join path stays stable across the
+# v2 HA migration (single-VM v1 -> 3-node etcd quorum v2). No AMI rebuild
+# needed when the control plane expands.
+After=network-online.target mulga-eks-provider-id.service mulga-vpc-mtu.service mulga-ebs-byid.service mulga-cdi-generate.service
+Wants=network-online.target
+
+[Service]
+Type=simple
+# k3s embeds containerd + kubelet in-process, so on a cgroup-v2 host the unit
+# must delegate a cgroup subtree or the kubelet cannot create /kubepods and
+# ContainerManager.Start fails before Node registration (OpenRC ran k3s in the
+# root cgroup, so this was implicit). overlay/br_netfilter feed the embedded
+# containerd snapshotter + kube-proxy.
+Delegate=yes
+KillMode=process
+TasksMax=infinity
+LimitNOFILE=1048576
+LimitNPROC=infinity
+LimitCORE=infinity
+# Cloud-init drops K3S_URL/K3S_TOKEN/K3S_NODE_NAME (plus SPINIFEX_NODEGROUP and,
+# for GPU workers, SPINIFEX_GPU_NODE) here at RunInstances time per the Nodegroup
+# bootstrap flow; mulga-eks-provider-id reads the SPINIFEX_* vars for node-labels.
+EnvironmentFile=-/etc/spinifex-eks/agent.env
+# Tee k3s (and its embedded kubelet's klog) stderr to the serial console as well
+# as journald: Ubuntu logs to journald only, which is off-box-invisible on a
+# worker in an isolated customer VPC, so a kubelet that fails to register the
+# Node leaves no trace. The serial console is captured per-VM by the daemon.
+StandardOutput=journal+console
+StandardError=journal+console
+ExecStartPre=-/sbin/modprobe overlay
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=/usr/local/sbin/k3s-agent-prestart
+ExecStart=/bin/sh -c '. /run/k3s-agent-extra-args.env 2>/dev/null || true; exec /usr/local/bin/k3s agent ${K3S_AGENT_EXTRA_ARGS:-}'
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target

scripts/images/eks-node-gpu/manifest.conf

@@ -0,0 +1,52 @@
+IMAGE_NAME="eks-node-gpu"
+IMAGE_DESCRIPTION="Ubuntu EKS K3s GPU worker node — NVIDIA server driver (DKMS pre-built) + nvidia-container-toolkit; agent-only (GPU nodes never run the EKS control plane)"
+DISTRO="ubuntu"
+UBUNTU_VERSION="26.04"
+UBUNTU_CODENAME="resolute"
+
+# 16G — headroom for kernel headers, NVIDIA driver + DKMS build artefacts, and
+# nvidia-container-toolkit, mirroring scripts/images/ubuntu-gpu-nvidia.conf.
+IMAGE_SIZE="16G"
+
+# K3s bundles its own containerd, so no containerd/runc package here. The
+# NVIDIA server driver + nvidia-utils come from the stock Ubuntu repos and are
+# installed in setup.sh (kernel-version-pinned DKMS build, like the ubuntu-gpu
+# recipe). nvidia-container-toolkit lives in its own NVIDIA-hosted apt repo,
+# which is not enabled on the base cloud image, so it is also added+installed
+# in setup.sh rather than here (see scripts/images/ubuntu-gpu-nvidia-setup.sh).
+APT_PACKAGES="ca-certificates curl jq iptables conntrack cgroup-tools initramfs-tools"
+
+# ecr-credential-provider is the ONLY server binary a GPU worker needs: the
+# nodegroup userdata points kubelet at it (image-credential-provider-config +
+# bin-dir=/usr/local/bin) for ECR pulls, and kubelet treats a referenced-but-
+# missing provider binary as a FATAL init error (RegisterCredentialProviderPlugins),
+# so the node never registers. The Alpine eks-node ships it; the GPU AMI must too.
+# Server-only binaries (token-webhook, gateway-*, konnectivity, k3s server) stay
+# omitted — GPU nodes are agent-only.
+BUILD_COMMANDS="CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -o bin/ecr-credential-provider ./cmd/ecr-credential-provider"
+INSTALL_BINARIES="bin/ecr-credential-provider:/usr/local/bin/ecr-credential-provider"
+
+# Role-agnostic helper scripts are reused verbatim from scripts/images/eks-node/
+# (only the init wiring changes: OpenRC -> systemd). k3s-agent-prestart is new —
+# it reproduces the k3s-agent.initd start_pre logic (env require + IMDS
+# provider-id) for the systemd unit. Server-only units (role selector, k3s
+# server, eks-token-webhook, konnectivity, state-report, k3s recovery,
+# mgmt-net) are intentionally not ported — GPU nodes are single-NIC workers.
+INSTALL_FILES="\
+scripts/images/eks-node/mulga-ebs-byid.sh:/usr/local/sbin/mulga-ebs-byid \
+scripts/images/eks-node/mulga-eks-provider-id.sh:/usr/local/sbin/mulga-eks-provider-id \
+scripts/images/eks-node/mulga-vpc-mtu.sh:/usr/local/sbin/mulga-vpc-mtu \
+scripts/images/eks-node-gpu/k3s-agent-prestart.sh:/usr/local/sbin/k3s-agent-prestart \
+scripts/images/eks-node-gpu/k3s-agent.service:/etc/systemd/system/k3s-agent.service \
+scripts/images/eks-node-gpu/mulga-ebs-byid.service:/etc/systemd/system/mulga-ebs-byid.service \
+scripts/images/eks-node-gpu/mulga-eks-provider-id.service:/etc/systemd/system/mulga-eks-provider-id.service \
+scripts/images/eks-node-gpu/mulga-vpc-mtu.service:/etc/systemd/system/mulga-vpc-mtu.service \
+scripts/images/eks-node-gpu/mulga-cdi-generate.service:/etc/systemd/system/mulga-cdi-generate.service \
+scripts/images/eks-node-gpu/99-mulga-ebs-byid.rules:/etc/udev/rules.d/99-mulga-ebs-byid.rules"
+
+ENABLE_SERVICES="k3s-agent mulga-ebs-byid mulga-eks-provider-id mulga-vpc-mtu mulga-cdi-generate"
+
+SETUP_SCRIPT="scripts/images/eks-node-gpu/setup.sh"
+
+AMI_NAME="spinifex-eks-node-gpu"
+SYSTEM_TAG="spinifex:managed-by=eks gpu-vendor=nvidia"

scripts/images/eks-node-gpu/mulga-cdi-generate.service

@@ -0,0 +1,16 @@
+[Unit]
+Description=Generate NVIDIA CDI spec for the container runtime (GPU-gated, first boot)
+After=systemd-modules-load.service
+Before=k3s-agent.service
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+# Silently skip on non-GPU boots: only generate when an NVIDIA GPU is
+# actually present and the driver has bound.
+ExecCondition=/bin/sh -c 'command -v nvidia-smi >/dev/null 2>&1 && nvidia-smi -L >/dev/null 2>&1'
+ExecStartPre=/bin/mkdir -p /etc/cdi
+ExecStart=/usr/bin/nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
+
+[Install]
+WantedBy=multi-user.target

scripts/images/eks-node-gpu/mulga-ebs-byid.service

@@ -0,0 +1,18 @@
+[Unit]
+# Role-agnostic boot oneshot that coldplugs the EBS by-id symlink bridge for
+# the aws-ebs-csi-driver node plugin — any virtio-blk volume already attached
+# at boot. Hot-attached volumes are handled separately by the udev rule
+# 99-mulga-ebs-byid.rules, which fires the same helper (see
+# scripts/images/eks-node-gpu/99-mulga-ebs-byid.rules).
+Description=Mulga EBS /dev/disk/by-id symlink bridge (coldplug)
+After=local-fs.target
+Before=k3s-agent.service
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+Environment=ACTION=coldplug
+ExecStart=/usr/local/sbin/mulga-ebs-byid
+
+[Install]
+WantedBy=multi-user.target

scripts/images/eks-node-gpu/mulga-eks-provider-id.service

@@ -0,0 +1,20 @@
+[Unit]
+Description=Set K3s provider-id + topology labels from IMDS for EBS CSI
+# After cloud-config.service so cloud-init has written /etc/spinifex-eks/agent.env
+# (write_files completes by the config stage): this script folds the nodegroup +
+# GPU node-labels (read from agent.env) into the same node-label list, so agent.env
+# must exist before it runs. NOT cloud-final.service: that unit is ordered after
+# multi-user.target and k3s-agent.service (ordered After this unit) is
+# WantedBy=multi-user.target, so chaining onto cloud-final forms an ordering cycle
+# that makes systemd skip both this unit and k3s-agent, so the worker never joins.
+After=network-online.target cloud-config.service
+Wants=network-online.target
+Before=k3s-agent.service
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/local/sbin/mulga-eks-provider-id
+
+[Install]
+WantedBy=multi-user.target

scripts/images/eks-node-gpu/mulga-vpc-mtu.service

@@ -0,0 +1,13 @@
+[Unit]
+Description=Pin VPC data NIC MTU so flannel fits the OVN geneve underlay
+After=network-online.target
+Wants=network-online.target
+Before=k3s-agent.service
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/local/sbin/mulga-vpc-mtu
+
+[Install]
+WantedBy=multi-user.target

scripts/images/eks-node-gpu/setup.sh

@@ -0,0 +1,177 @@
+#!/bin/sh
+set -eu
+
+# setup.sh — guest customisation for the eks-node-gpu AMI (Ubuntu, K3s agent
+# worker only — GPU nodes never run the EKS control plane, so there is no
+# server/role-selector path here, unlike scripts/images/eks-node/setup.sh).
+#
+# Runs inside the libguestfs appliance (via virt-customize --run) under
+# build-system-image.sh, after packages, binaries and INSTALL_FILES are
+# placed. Downloads + verifies the pinned K3s binary (same recipe as
+# scripts/images/eks-node/setup.sh), sets exec bits on the uploaded shell
+# helpers, and bakes the headless NVIDIA driver + nvidia-container-toolkit
+# (steps reused from scripts/images/ubuntu-gpu-nvidia-setup.sh — see
+# install_nvidia_gpu_stack() below, duplicated verbatim in
+# scripts/images/ecs-agent-gpu/setup.sh so both images stay consistent).
+#
+# Network access inside the appliance is provided by virt-customize --network.
+
+K3S_VERSION="v1.32.5+k3s1"
+K3S_URL_BASE="https://github.com/k3s-io/k3s/releases/download/${K3S_VERSION}"
+
+echo "[eks-node-gpu-setup] fetching k3s checksums ${K3S_VERSION}"
+curl -fsSL -o /tmp/k3s-checksums.txt "${K3S_URL_BASE}/sha256sum-amd64.txt"
+K3S_SHA256=$(awk '/[ \t]k3s$/{print $1; exit}' /tmp/k3s-checksums.txt)
+if [ -z "${K3S_SHA256}" ]; then
+    echo "[eks-node-gpu-setup] could not parse k3s sha256 from upstream checksums"
+    cat /tmp/k3s-checksums.txt
+    exit 1
+fi
+echo "[eks-node-gpu-setup] downloading k3s ${K3S_VERSION} (sha256=${K3S_SHA256})"
+curl -fsSL -o /usr/local/bin/k3s "${K3S_URL_BASE}/k3s"
+echo "${K3S_SHA256}  /usr/local/bin/k3s" > /tmp/k3s.sha256
+if ! sha256sum -c /tmp/k3s.sha256; then
+    echo "[eks-node-gpu-setup] k3s SHA256 mismatch — refusing to bake AMI"
+    exit 1
+fi
+chmod 0755 /usr/local/bin/k3s
+ln -sf /usr/local/bin/k3s /usr/local/bin/kubectl
+ln -sf /usr/local/bin/k3s /usr/local/bin/crictl
+ln -sf /usr/local/bin/k3s /usr/local/bin/ctr
+
+# Reused shell helpers + the new prestart wrapper land 0644 via INSTALL_FILES;
+# make them executable. The systemd unit files and the udev rules file do not
+# need an exec bit (unlike OpenRC's /etc/init.d/*).
+chmod 0755 /usr/local/sbin/mulga-ebs-byid /usr/local/sbin/mulga-eks-provider-id \
+    /usr/local/sbin/mulga-vpc-mtu /usr/local/sbin/k3s-agent-prestart
+
+# K3s config dir (mulga-eks-provider-id's drop-in + cloud-init/first-boot
+# agent config) and the shared spinifex-eks state dir (agent.env lands here
+# from cloud-init at launch).
+mkdir -p /etc/rancher/k3s/config.yaml.d /etc/spinifex-eks
+
+# ── NVIDIA headless driver + container toolkit (node-image subset) ──────────
+# Reused from scripts/images/ubuntu-gpu-nvidia-setup.sh: kernel detection,
+# linux-headers-<kver>, the latest versioned nvidia-dkms-<ver>-server +
+# nvidia-utils-<ver>-server, `dkms install`, nvidia.ko verification,
+# `apt-mark hold` on the kernel/nvidia/libnvidia packages, the nouveau
+# modprobe blacklist, and `update-initramfs -u -k <kver>`.
+#
+# Dropped for a node image: pciutils/python3/git/curl-extras/htop/tmux/ffmpeg/
+# libgl1/libglib dev payload, and Docker CE entirely — GPU nodes run k3s
+# (this image) or containerd (scripts/images/ecs-agent-gpu), never Docker.
+# `nvidia-ctk runtime configure --runtime=docker` and the docker-group
+# cloud-init drop-in are dropped for the same reason; the runtime is instead
+# wired via CDI at first boot by mulga-cdi-generate.service, gated on an
+# NVIDIA GPU actually being present.
+install_nvidia_gpu_stack() {
+    KVER=$(ls /boot/vmlinuz-* 2>/dev/null | sort -V | tail -1 | sed 's|/boot/vmlinuz-||') || true
+    if [ -z "${KVER}" ]; then
+        # Ubuntu 26.04 minimal cloud image keeps the kernel on the ESP (not
+        # mounted in the appliance chroot), so /boot is empty — fall back to
+        # /lib/modules.
+        KVER=$(ls /lib/modules/ 2>/dev/null | sort -V | tail -1)
+    fi
+    if [ -z "${KVER}" ]; then
+        echo "ERROR: No kernel found in /boot or /lib/modules — base image may be missing a kernel"
+        exit 1
+    fi
+    echo "[nvidia-gpu-stack] target kernel: ${KVER}"
+
+    # Prevent kernel postinst hooks from attempting grub-install against the
+    # host disk — this fails silently inside the appliance and can abort the
+    # postinst, leaving /boot incomplete.
+    cat > /etc/kernel-img.conf <<'EOF'
+do_symlinks = yes
+do_bootloader = no
+do_initrd = yes
+link_in_boot = yes
+EOF
+
+    apt-get install -y --no-install-recommends "linux-headers-${KVER}"
+
+    # Ubuntu 26.04+ no longer ships unversioned meta-packages, so the branch is
+    # named explicitly. Pinned rather than resolved at build time: the number in
+    # the package name is a branch alias, not a driver version (nvidia-dkms-535-server
+    # installs 580.126.20), so picking the highest number sorts on a value that
+    # need not match what gets installed. Resolving also let identical source
+    # produce different drivers on different days. Bumping is a deliberate edit.
+    NVIDIA_VER=595
+    echo "[nvidia-gpu-stack] installing NVIDIA server driver branch: ${NVIDIA_VER}"
+    # The -open (GSP/open-kernel-module) DKMS variant, not the closed/proprietary
+    # one: Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
+    # initialize under the closed kernel module — RmInitAdapter fails with
+    # "requires use of the NVIDIA open kernel modules" (dmesg NVRM error 0x22).
+    # nvidia-utils has no separate open variant (userspace tools only).
+    apt-get install -y --no-install-recommends \
+        "nvidia-dkms-${NVIDIA_VER}-server-open" \
+        "nvidia-utils-${NVIDIA_VER}-server"
+
+    NVIDIA_DKMS_NAME=$(dkms status 2>/dev/null | awk -F'[,/ ]+' '/nvidia/{print $1; exit}')
+    NVIDIA_DKMS_VER=$(dkms status 2>/dev/null  | awk -F'[,/ ]+' '/nvidia/{print $2; exit}')
+    if [ -n "${NVIDIA_DKMS_NAME}" ] && [ -n "${NVIDIA_DKMS_VER}" ]; then
+        echo "[nvidia-gpu-stack] building NVIDIA DKMS module ${NVIDIA_DKMS_NAME}/${NVIDIA_DKMS_VER} for kernel ${KVER}..."
+        dkms install -m "${NVIDIA_DKMS_NAME}" -v "${NVIDIA_DKMS_VER}" --kernelver "${KVER}" --force
+    else
+        echo "WARNING: nvidia DKMS module not found in dkms status — driver install may have failed"
+    fi
+
+    NVIDIA_KO=$(find "/lib/modules/${KVER}/updates" -name "nvidia.ko*" 2>/dev/null | head -1)
+    if [ -z "${NVIDIA_KO}" ]; then
+        echo "ERROR: nvidia.ko not found under /lib/modules/${KVER}/updates after DKMS build"
+        exit 1
+    fi
+    echo "[nvidia-gpu-stack] NVIDIA kernel module: ${NVIDIA_KO}"
+
+    # Pin the exact kernel + headers + nvidia/libnvidia packages so
+    # unattended-upgrades cannot replace what the DKMS module was built for.
+    apt-mark hold "linux-image-${KVER}" "linux-headers-${KVER}"
+
+    NVIDIA_PKGS=$(dpkg --get-selections 'nvidia-*' 2>/dev/null | awk '/install$/{print $1}' || true)
+    LIBNVIDIA_PKGS=$(dpkg --get-selections 'libnvidia-*' 2>/dev/null | awk '/install$/{print $1}' || true)
+    # shellcheck disable=SC2086
+    [ -n "${NVIDIA_PKGS}" ]    && apt-mark hold ${NVIDIA_PKGS}
+    # shellcheck disable=SC2086
+    [ -n "${LIBNVIDIA_PKGS}" ] && apt-mark hold ${LIBNVIDIA_PKGS}
+
+    # nvidia.ko and nouveau conflict; without this the NVIDIA driver will not
+    # bind after boot.
+    cat > /etc/modprobe.d/blacklist-nouveau.conf <<'EOF'
+blacklist nouveau
+options nouveau modeset=0
+EOF
+
+    # Load the NVIDIA modules at boot so /dev/nvidia* exist before CDI
+    # generation and the container runtime start — a headless node never runs
+    # an X server or CUDA app to trigger the on-demand load.
+    cat > /etc/modules-load.d/nvidia.conf <<'EOF'
+nvidia
+nvidia_uvm
+nvidia_modeset
+EOF
+
+    # nvidia-container-toolkit lives in its own NVIDIA-hosted apt repo, which
+    # is not enabled on the base Ubuntu cloud image — add it here (same
+    # gpg key + list as scripts/images/ubuntu-gpu-nvidia-setup.sh) and
+    # install only the headless toolkit (no Docker CE).
+    apt-get install -y --no-install-recommends gnupg
+    mkdir -p /usr/share/keyrings /etc/apt/sources.list.d
+    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
+        | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
+    curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
+        | sed 's|deb https://|deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://|g' \
+        > /etc/apt/sources.list.d/nvidia-container-toolkit.list
+    apt-get update -qq
+    apt-get install -y --no-install-recommends nvidia-container-toolkit
+    apt-mark hold nvidia-container-toolkit nvidia-container-toolkit-base \
+        libnvidia-container1 libnvidia-container-tools 2>/dev/null || true
+
+    # Rebuild initramfs with nouveau blacklisted and the NVIDIA module included.
+    update-initramfs -u -k "${KVER}"
+
+    echo "[nvidia-gpu-stack] done: kernel=${KVER}, NVIDIA headless driver + nvidia-container-toolkit installed"
+}
+
+install_nvidia_gpu_stack
+
+echo "[eks-node-gpu-setup] done"

scripts/images/eks-node/addons/argocd/3.0.23/argocd.yaml

@@ -24939,6 +24939,17 @@ spec:
       labels:
         app.kubernetes.io/name: argocd-applicationset-controller
     spec:
+      # Local delta over the vendored upstream manifest: a cluster with a
+      # GPU-only nodegroup has no untainted node — the CP carries
+      # CriticalAddonsOnly and every worker carries nvidia.com/gpu, so without
+      # these the pod stays Pending forever. Upstream never needs it because
+      # real EKS control planes are not schedulable nodes.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       containers:
       - args:
         - /usr/local/bin/argocd-applicationset-controller
@@ -25180,6 +25191,17 @@ spec:
       labels:
         app.kubernetes.io/name: argocd-dex-server
     spec:
+      # Local delta over the vendored upstream manifest: a cluster with a
+      # GPU-only nodegroup has no untainted node — the CP carries
+      # CriticalAddonsOnly and every worker carries nvidia.com/gpu, so without
+      # these the pod stays Pending forever. Upstream never needs it because
+      # real EKS control planes are not schedulable nodes.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
@@ -25304,6 +25326,17 @@ spec:
       labels:
         app.kubernetes.io/name: argocd-notifications-controller
     spec:
+      # Local delta over the vendored upstream manifest: a cluster with a
+      # GPU-only nodegroup has no untainted node — the CP carries
+      # CriticalAddonsOnly and every worker carries nvidia.com/gpu, so without
+      # these the pod stays Pending forever. Upstream never needs it because
+      # real EKS control planes are not schedulable nodes.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       containers:
       - args:
         - /usr/local/bin/argocd-notifications
@@ -25403,6 +25436,17 @@ spec:
       labels:
         app.kubernetes.io/name: argocd-redis
     spec:
+      # Local delta over the vendored upstream manifest: a cluster with a
+      # GPU-only nodegroup has no untainted node — the CP carries
+      # CriticalAddonsOnly and every worker carries nvidia.com/gpu, so without
+      # these the pod stays Pending forever. Upstream never needs it because
+      # real EKS control planes are not schedulable nodes.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
@@ -25486,6 +25530,17 @@ spec:
       labels:
         app.kubernetes.io/name: argocd-repo-server
     spec:
+      # Local delta over the vendored upstream manifest: a cluster with a
+      # GPU-only nodegroup has no untainted node — the CP carries
+      # CriticalAddonsOnly and every worker carries nvidia.com/gpu, so without
+      # these the pod stays Pending forever. Upstream never needs it because
+      # real EKS control planes are not schedulable nodes.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
@@ -25840,6 +25895,17 @@ spec:
       labels:
         app.kubernetes.io/name: argocd-server
     spec:
+      # Local delta over the vendored upstream manifest: a cluster with a
+      # GPU-only nodegroup has no untainted node — the CP carries
+      # CriticalAddonsOnly and every worker carries nvidia.com/gpu, so without
+      # these the pod stays Pending forever. Upstream never needs it because
+      # real EKS control planes are not schedulable nodes.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
@@ -26251,6 +26317,17 @@ spec:
       labels:
         app.kubernetes.io/name: argocd-application-controller
     spec:
+      # Local delta over the vendored upstream manifest: a cluster with a
+      # GPU-only nodegroup has no untainted node — the CP carries
+      # CriticalAddonsOnly and every worker carries nvidia.com/gpu, so without
+      # these the pod stays Pending forever. Upstream never needs it because
+      # real EKS control planes are not schedulable nodes.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:

scripts/images/eks-node/addons/aws-ebs-csi-driver/1.40.1/10-controller.yaml

@@ -44,6 +44,18 @@ spec:
         kubernetes.io/os: linux
       serviceAccountName: ebs-csi-controller-sa
       priorityClassName: system-cluster-critical
+      # A cluster with a GPU-only nodegroup has no untainted node: the CP carries
+      # CriticalAddonsOnly (EKS parity keeps user workloads off it) and every
+      # worker carries nvidia.com/gpu (nodegroup_userdata.go). Without these
+      # tolerations the controller has nowhere to schedule and stays Pending
+      # forever. Real EKS never needs this: its control plane isn't a schedulable
+      # node, so the stock upstream manifest carries no such toleration.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       affinity:
         nodeAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:

scripts/images/eks-node/addons/aws-ebs-csi-driver/1.40.1/20-node.yaml

@@ -46,6 +46,13 @@ spec:
                 - a1.4xlarge
       nodeSelector:
         kubernetes.io/os: linux
+      # The node plugin must run on every worker, including GPU-only workers
+      # tainted nvidia.com/gpu=present:NoSchedule, or GPU pods there cannot
+      # mount EBS volumes (no node plugin = no CSI NodeStage/NodePublish).
+      tolerations:
+        - key: nvidia.com/gpu
+          operator: Exists
+          effect: NoSchedule
       serviceAccountName: ebs-csi-node-sa
       terminationGracePeriodSeconds: 30
       priorityClassName: system-node-critical

scripts/images/eks-node/addons/aws-load-balancer-controller/2.11.0/10-aws-load-balancer-controller.yaml

@@ -348,6 +348,18 @@ spec:
           readOnly: true
 {{IRSA_VOLUME_MOUNT}}
       priorityClassName: system-cluster-critical
+      # A cluster with a GPU-only nodegroup has no untainted node: the CP carries
+      # CriticalAddonsOnly (EKS parity keeps user workloads off it) and every
+      # worker carries nvidia.com/gpu (nodegroup_userdata.go). Without these
+      # tolerations the controller has nowhere to schedule and stays Pending
+      # forever. Real EKS never needs this: its control plane isn't a schedulable
+      # node, so the stock upstream manifest carries no such toleration.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       securityContext:
         fsGroup: 1337
       serviceAccountName: aws-load-balancer-controller

scripts/images/eks-node/addons/nvidia-device-plugin/0.17.4/10-daemonset.yaml

@@ -0,0 +1,75 @@
+# nvidia-device-plugin 0.17.4 — CDI-mode DaemonSet exposing nvidia.com/gpu
+# allocatable on GPU-tainted nodes. Gated to GPU nodes via nodeSelector on the
+# node-label nvidia.com/gpu.present=true (written by mulga-eks-provider-id from
+# the SPINIFEX_GPU_NODE agent.env signal); inert
+# elsewhere. --device-list-strategy=cdi-cri makes the plugin generate a CDI spec
+# for GPU workloads (containerd 2.x resolves the CRI-passed CDI devices; no
+# enable_cdi drop-in needed on k3s v1.32.5). The plugin's OWN container needs
+# NVML + the driver tree to run nvml.Init and generate that spec, so it runs
+# under the k3s-auto-configured "nvidia" runtimeClass with the host root mounted
+# at /driver-root and the host /var/run/cdi mounted writable so the generated
+# spec is visible to containerd. GPU workloads themselves need only
+# resources.limits nvidia.com/gpu — no runtimeClass.
+---
+kind: DaemonSet
+apiVersion: apps/v1
+metadata:
+  name: nvidia-device-plugin-daemonset
+  namespace: kube-system
+  labels:
+    app.kubernetes.io/name: nvidia-device-plugin
+spec:
+  selector:
+    matchLabels:
+      name: nvidia-device-plugin-ds
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        name: nvidia-device-plugin-ds
+    spec:
+      nodeSelector:
+        nvidia.com/gpu.present: "true"
+      tolerations:
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
+      priorityClassName: "system-node-critical"
+      runtimeClassName: nvidia
+      containers:
+      - image: nvcr.io/nvidia/k8s-device-plugin:v0.17.4
+        name: nvidia-device-plugin-ctr
+        args:
+        - --device-list-strategy=cdi-cri
+        env:
+          - name: FAIL_ON_INIT_ERROR
+            value: "false"
+          - name: NVIDIA_VISIBLE_DEVICES
+            value: all
+          - name: NVIDIA_DRIVER_CAPABILITIES
+            value: all
+        securityContext:
+          allowPrivilegeEscalation: false
+          capabilities:
+            drop: ["ALL"]
+        volumeMounts:
+        - name: kubelet-device-plugins-dir
+          mountPath: /var/lib/kubelet/device-plugins
+        - name: driver-root
+          mountPath: /driver-root
+          readOnly: true
+        - name: cdi-root
+          mountPath: /var/run/cdi
+      volumes:
+      - name: kubelet-device-plugins-dir
+        hostPath:
+          path: /var/lib/kubelet/device-plugins
+          type: Directory
+      - name: driver-root
+        hostPath:
+          path: /
+      - name: cdi-root
+        hostPath:
+          path: /var/run/cdi
+          type: DirectoryOrCreate

scripts/images/eks-node/coredns/coredns-mulga.yaml

@@ -135,6 +135,13 @@ spec:
                     operator: DoesNotExist
       nodeSelector:
         kubernetes.io/os: linux
+      # With internalTrafficPolicy=Local each worker resolves through its OWN
+      # CoreDNS pod, so a GPU-only worker (tainted nvidia.com/gpu=present:
+      # NoSchedule) with no local pod would have no DNS at all for its pods.
+      tolerations:
+        - key: nvidia.com/gpu
+          operator: Exists
+          effect: NoSchedule
       containers:
         - name: coredns
           image: rancher/mirrored-coredns-coredns:1.12.1

scripts/images/eks-node/eks-node-role.sh

@@ -48,14 +48,26 @@ if [ -z "${ROLE}" ] && [ -f "${AGENT_ENVFILE}" ]; then
     ROLE="agent"
 fi
 
+case "${ROLE}" in
+    server | server-join | agent) ;;
+    *)
+        die "SPINIFEX_K3S_ROLE missing or invalid: '${ROLE}'"
+        ;;
+esac
+
+# Written before the rc-service calls below: mulga-eks-k3s-recovery, started
+# inline for server/server-join, reads ROLE_FILE at the top of its own logic
+# and must find it already in place, not empty.
+mkdir -p "$(dirname "${ROLE_FILE}")"
+printf '%s\n' "${ROLE}" > "${ROLE_FILE}"
+
 case "${ROLE}" in
     server)
         log "configuring server role"
         rc-update add eks-token-webhook default
-        # Pre-k3s etcd recovery: added to the runlevel so it runs before k3s on
-        # every subsequent boot, but not started now — a fresh cluster has no
-        # directive and the role file it guards on is written at the end of this
-        # script. It first acts on the reboot the reconciler triggers.
+        # Pre-k3s etcd recovery: enabled for later boots and started inline
+        # now too, so a restore-snapshot DR seed's directive applies before
+        # k3s starts on this first boot. A plain create has no directive, so it's a no-op.
         rc-update add mulga-eks-k3s-recovery default
         rc-update add k3s default
         # konnectivity-server fronts apiserver egress; every server replica runs
@@ -69,6 +81,7 @@ case "${ROLE}" in
         # (mulga-siv-231.7); server-join nodes do not run it.
         rc-update add mulga-eks-addon-sync default
         rc-service eks-token-webhook start
+        rc-service mulga-eks-k3s-recovery start
         rc-service k3s start
         rc-service konnectivity-server start
         rc-service k3s-first-boot start
@@ -79,12 +92,14 @@ case "${ROLE}" in
         log "configuring server-join role"
         rc-update add eks-token-webhook default
         # Pre-k3s etcd recovery (see server role): server-join members are the
-        # wipe-rejoin followers of a cluster-reset, so they need it too.
+        # wipe-rejoin followers of a cluster-reset, so they need it too, and for
+        # the same first-boot reason must be started here, not deferred.
         rc-update add mulga-eks-k3s-recovery default
         rc-update add k3s default
         rc-update add konnectivity-server default
         rc-update add mulga-eks-state-report default
         rc-service eks-token-webhook start
+        rc-service mulga-eks-k3s-recovery start
         rc-service k3s start
         rc-service konnectivity-server start
         rc-service mulga-eks-state-report start
@@ -99,7 +114,5 @@ case "${ROLE}" in
         ;;
 esac
 
-mkdir -p "$(dirname "${ROLE_FILE}")"
-printf '%s\n' "${ROLE}" > "${ROLE_FILE}"
 rc-update del eks-node-role default 2>/dev/null || true
 log "role '${ROLE}' configured"

scripts/images/eks-node/eks-node-role_test.sh

@@ -65,8 +65,11 @@ check "server: enables k3s-first-boot" "yes" "$(grep -q 'rc-update add k3s-first
 check "server: starts k3s" "yes" "$(grep -q 'rc-service k3s start' "${CALLS}" && echo yes || echo no)"
 check "server: enables konnectivity-server" "yes" "$(grep -q 'rc-update add konnectivity-server default' "${CALLS}" && echo yes || echo no)"
 check "server: enables k3s-recovery" "yes" "$(grep -q 'rc-update add mulga-eks-k3s-recovery default' "${CALLS}" && echo yes || echo no)"
-# Recovery is added to the runlevel (pre-k3s on later boots) but not started now.
-check "server: does not start k3s-recovery" "no" "$(grep -q 'rc-service mulga-eks-k3s-recovery' "${CALLS}" && echo yes || echo no)"
+# Recovery is added to the runlevel (so it also runs pre-k3s on later boots)
+# AND started inline now — a restore-snapshot DR seed's directive must apply
+# on this very first boot, before k3s starts.
+check "server: starts k3s-recovery" "yes" "$(grep -q 'rc-service mulga-eks-k3s-recovery start' "${CALLS}" && echo yes || echo no)"
+check "server: starts k3s-recovery before k3s" "yes" "$(awk '/rc-service mulga-eks-k3s-recovery start/{r=NR} /rc-service k3s start/{k=NR} END{print (r>0 && k>0 && r<k) ? "yes" : "no"}' "${CALLS}")"
 check "server: no k3s-agent" "no" "$(grep -q 'k3s-agent' "${CALLS}" && echo yes || echo no)"
 check "server: self-disables" "yes" "$(grep -q 'rc-update del eks-node-role default' "${CALLS}" && echo yes || echo no)"
 
@@ -81,6 +84,7 @@ check "server-join: enables k3s" "yes" "$(grep -q 'rc-update add k3s default' "$
 check "server-join: enables state-report" "yes" "$(grep -q 'rc-update add mulga-eks-state-report default' "${CALLS}" && echo yes || echo no)"
 check "server-join: enables konnectivity-server" "yes" "$(grep -q 'rc-update add konnectivity-server default' "${CALLS}" && echo yes || echo no)"
 check "server-join: enables k3s-recovery" "yes" "$(grep -q 'rc-update add mulga-eks-k3s-recovery default' "${CALLS}" && echo yes || echo no)"
+check "server-join: starts k3s-recovery" "yes" "$(grep -q 'rc-service mulga-eks-k3s-recovery start' "${CALLS}" && echo yes || echo no)"
 # Join servers must NOT re-publish bootstrap; the first server already did.
 check "server-join: no k3s-first-boot" "no" "$(grep -q 'k3s-first-boot' "${CALLS}" && echo yes || echo no)"
 check "server-join: no k3s-agent" "no" "$(grep -q 'k3s-agent' "${CALLS}" && echo yes || echo no)"

scripts/images/eks-node/k3s-agent.initd

@@ -29,7 +29,7 @@ start_pre() {
     if [ -f /etc/spinifex-eks/agent.env ]; then
         # shellcheck disable=SC1091
         . /etc/spinifex-eks/agent.env
-        export K3S_URL K3S_TOKEN K3S_NODE_NAME K3S_NODE_LABEL
+        export K3S_URL K3S_TOKEN K3S_NODE_NAME
     fi
     if [ -z "${K3S_URL:-}" ] || [ -z "${K3S_TOKEN:-}" ]; then
         eerror "K3S_URL and K3S_TOKEN must be set in /etc/spinifex-eks/agent.env"

scripts/images/eks-node/k3s.initd

@@ -28,6 +28,16 @@ depend() {
 }
 
 start_pre() {
+    # mulga-eks-k3s-recovery drops this marker when a required-snapshot restore
+    # could not fetch its etcd snapshot. Refuse to start: a fresh DR seed with no
+    # snapshot would cluster-init an EMPTY datastore (silent total data loss). The
+    # marker is in /run tmpfs, cleared on reboot, so recovery retries the fetch
+    # next boot and k3s starts once the snapshot restores.
+    if [ -f /run/spinifex-eks/k3s-restore-block ]; then
+        eerror "restore-snapshot could not fetch the required etcd snapshot; refusing to start k3s on an empty datastore (see mulga-eks-k3s-recovery log)"
+        return 1
+    fi
+
     # config.yaml is written by k3s-first-boot from the skeleton + cloud-init
     # data. Refuse to start if neither exists.
     if [ ! -f /etc/rancher/k3s/config.yaml ]; then

scripts/images/eks-node/konnectivity/konnectivity-agent.yaml

@@ -9,7 +9,9 @@
 #
 # Runs on workers only: it carries no toleration for the control-plane
 # CriticalAddonsOnly=true:NoExecute taint, so it is never scheduled onto the CP
-# nodes (which run the server, not an agent).
+# nodes (which run the server, not an agent). It DOES tolerate the GPU
+# nvidia.com/gpu=present:NoSchedule taint — a GPU-only worker still needs
+# kubectl logs/exec against pods there, which routes through this agent.
 apiVersion: v1
 kind: ServiceAccount
 metadata:
@@ -39,6 +41,13 @@ spec:
     spec:
       serviceAccountName: konnectivity-agent
       priorityClassName: system-node-critical
+      # Without this, a GPU-only worker (tainted nvidia.com/gpu=present:NoSchedule)
+      # shows DESIRED=0 for this DaemonSet and kubectl logs/exec against pods
+      # scheduled there fails "No agent available".
+      tolerations:
+        - key: nvidia.com/gpu
+          operator: Exists
+          effect: NoSchedule
       # hostNetwork so the agent reaches the NLB over the node network and
       # identifies itself to the server by the node's own IP (the address the
       # apiserver dials back through for kubelet/pod egress).

scripts/images/eks-node/manifest.conf

@@ -61,6 +61,7 @@ scripts/images/eks-node/addons/aws-ebs-csi-driver/1.40.1/00-rbac.yaml:/usr/share
 scripts/images/eks-node/addons/aws-ebs-csi-driver/1.40.1/10-controller.yaml:/usr/share/spinifex-eks/addons/aws-ebs-csi-driver/1.40.1/10-controller.yaml \
 scripts/images/eks-node/addons/aws-ebs-csi-driver/1.40.1/20-node.yaml:/usr/share/spinifex-eks/addons/aws-ebs-csi-driver/1.40.1/20-node.yaml \
 scripts/images/eks-node/addons/aws-ebs-csi-driver/1.40.1/30-storageclass.yaml:/usr/share/spinifex-eks/addons/aws-ebs-csi-driver/1.40.1/30-storageclass.yaml \
+scripts/images/eks-node/addons/nvidia-device-plugin/0.17.4/10-daemonset.yaml:/usr/share/spinifex-eks/addons/nvidia-device-plugin/0.17.4/10-daemonset.yaml \
 scripts/images/eks-node/coredns/coredns-mulga.yaml:/usr/share/spinifex-eks/coredns/coredns-mulga.yaml \
 scripts/images/eks-node/cloud-init-ec2.cfg:/etc/cloud/cloud.cfg.d/90-spinifex-ec2-imds.cfg \
 scripts/images/eks-node/mulga-eks-etcd-snapshot.sh:/etc/periodic/daily/mulga-eks-etcd-snapshot \

scripts/images/eks-node/mulga-eks-etcd-snapshot.sh

@@ -66,14 +66,24 @@ fi
 #
 # Note: --aws-sigv4 requires curl >= 7.75 (Alpine 3.21 ships 8.x).
 URL="${SPINIFEX_PREDASTORE_ENDPOINT%/}/${BUCKET}/${KEY}"
-curl -fsSL \
+# Capture curl's own exit: piping straight into logger masks it behind logger's
+# (near-always 0), so a failed upload would log "uploaded" and delete the only
+# copy — snapshots would silently never land. Keep the local snapshot and exit
+# non-zero on failure so cron surfaces it and the next run still has data on disk.
+if UPLOAD_OUT=$(curl -fsSL \
     --aws-sigv4 "aws:amz:${AWS_REGION:-au-mel-1}:s3" \
     --user "${SPINIFEX_PREDASTORE_AKID}:${SPINIFEX_PREDASTORE_SECRET}" \
     -H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
     --upload-file "${SNAPSHOT_FILE}" \
-    "${URL}" 2>&1 | logger -t mulga-eks-etcd-snapshot
-
-# Remove the local snapshot to bound disk usage; the canonical copy lives in
-# predastore. Restore path (`spx admin eks restore-snapshot`) pulls from there.
-rm -f "${SNAPSHOT_FILE}"
-logger -t mulga-eks-etcd-snapshot "uploaded ${KEY}"
+    "${URL}" 2>&1); then
+    [ -n "${UPLOAD_OUT}" ] && printf '%s\n' "${UPLOAD_OUT}" | logger -t mulga-eks-etcd-snapshot
+    # Remove the local snapshot only after a confirmed upload to bound disk usage;
+    # the canonical copy lives in predastore (`spx admin eks restore-snapshot`).
+    rm -f "${SNAPSHOT_FILE}"
+    logger -t mulga-eks-etcd-snapshot "uploaded ${KEY}"
+else
+    UPLOAD_RC=$?
+    [ -n "${UPLOAD_OUT}" ] && printf '%s\n' "${UPLOAD_OUT}" | logger -t mulga-eks-etcd-snapshot
+    logger -t mulga-eks-etcd-snapshot "ERROR: upload of ${KEY} failed (curl rc=${UPLOAD_RC}); keeping local snapshot ${SNAPSHOT_FILE}"
+    exit 1
+fi

scripts/images/eks-node/mulga-eks-etcd-snapshot_test.sh

@@ -29,10 +29,13 @@ done
 :
 EOF
 
-# curl stub: record the full argument line (carries --upload-file + URL).
+# curl stub: record the full argument line (carries --upload-file + URL). Exits
+# non-zero when CURL_FAIL=1 so the harness can exercise the failed-upload path.
 cat > "${STUBBIN}/curl" <<'EOF'
 #!/bin/sh
 echo "curl $*" >> "${CURL_CALLS}"
+[ "${CURL_FAIL:-0}" = "1" ] && { echo "curl: (7) connection refused" >&2; exit 7; }
+exit 0
 EOF
 
 # logger stub: swallow.
@@ -102,6 +105,24 @@ run server "${WORK}/15min/mulga-eks-etcd-snapshot"
 [ -s "${WORK}/curl.calls" ] && fail "no-env: must not upload" || pass "no-env: no upload without creds"
 ENVFILE="${ENVFILE_SAVE}"
 
+# --- Case 5: upload FAILS -> non-zero exit + local snapshot KEPT (not masked) ---
+# Set up like run() but expect a non-zero exit and assert the snapshot survives.
+K3S_CALLS="${WORK}/k3s.calls"; CURL_CALLS="${WORK}/curl.calls"
+SNAPSHOT_DIR="${WORK}/snapshots"; ROLE_FILE="${WORK}/role"; K3S_BIN=k3s
+: > "${K3S_CALLS}"; : > "${CURL_CALLS}"; rm -rf "${SNAPSHOT_DIR}"
+printf '%s' server > "${ROLE_FILE}"
+export K3S_CALLS CURL_CALLS SNAPSHOT_DIR ROLE_FILE K3S_BIN ENVFILE
+export TIER=frequent CURL_FAIL=1
+if sh "${WORK}/15min/mulga-eks-etcd-snapshot" </dev/null; then
+    fail "upload-fail: must exit non-zero when curl fails"
+else
+    pass "upload-fail: non-zero exit surfaces the failure"
+fi
+ls "${SNAPSHOT_DIR}"/etcd-frequent-* >/dev/null 2>&1 \
+    && pass "upload-fail: local snapshot kept for the next run" \
+    || fail "upload-fail: local snapshot deleted despite failed upload"
+unset TIER CURL_FAIL
+
 if [ "${FAILS}" -eq 0 ]; then
     echo "PASS: all mulga-eks-etcd-snapshot cases"
     exit 0

scripts/images/eks-node/mulga-eks-k3s-recovery.sh

@@ -7,10 +7,10 @@ set -eu
 # but embedded etcd never reformed quorum after a simultaneous restart).
 #
 # The reconciler cannot push into the guest, so recovery is pull-based: this
-# fetches a per-member directive epoch<TAB>action<TAB>snapshot through the AWS
-# gateway (eks-gateway-fetch -resource recovery), epoch-guarded so a directive
-# applies exactly once. Ordered `before k3s` so the etcd datastore is mutated
-# while the apiserver is stopped. Actions:
+# fetches a per-member directive epoch<TAB>action<TAB>snapshot<TAB>required
+# through the AWS gateway (eks-gateway-fetch -resource recovery), epoch-guarded
+# so a directive applies exactly once. Ordered `before k3s` so the etcd datastore
+# is mutated while the apiserver is stopped. Actions:
 #   none          steady state — nothing to do
 #   cluster-reset seed member: `k3s server --cluster-reset` reforms a
 #                 single-member etcd from this node's intact data (optionally
@@ -18,6 +18,13 @@ set -eu
 #   wipe-rejoin   follower: delete the local etcd datastore so k3s re-joins the
 #                 reset seed's quorum fresh on the next k3s start
 #
+# required=1 marks a snapshot that MUST restore (fresh DR seed via
+# `spx admin eks restore-snapshot`): a fresh seed has no intact local etcd, so a
+# cluster-reset without the snapshot would boot an EMPTY cluster. On a fetch
+# failure this drops BLOCK_MARKER and aborts boot; k3s.initd refuses to start on
+# the marker, so k3s never resets into an empty datastore. The fetch is retried
+# on the next boot (BLOCK_MARKER lives in /run tmpfs, cleared on reboot).
+#
 # Env knobs default to production paths; overridable so the logic is unit-testable
 # without root, IMDS, a gateway, or a real k3s.
 
@@ -25,6 +32,7 @@ ROLE_FILE=${ROLE_FILE:-/etc/spinifex-eks/role}
 ENVFILE=${ENVFILE:-/etc/spinifex-eks/first-boot.env}
 SNAPSHOT_ENVFILE=${SNAPSHOT_ENVFILE:-/etc/spinifex-eks/etcd-snapshot.env}
 EPOCH_FILE=${EPOCH_FILE:-/var/lib/spinifex-eks/recovery.epoch}
+BLOCK_MARKER=${BLOCK_MARKER:-/run/spinifex-eks/k3s-restore-block}
 K3S_BIN=${K3S_BIN:-/usr/local/bin/k3s}
 K3S_CONFIG=${K3S_CONFIG:-/etc/rancher/k3s/config.yaml}
 ETCD_DIR=${ETCD_DIR:-/var/lib/rancher/k3s/server/db/etcd}
@@ -72,12 +80,26 @@ fetch_snapshot() {
     [ -n "${SPINIFEX_PREDASTORE_SECRET:-}" ] || return 1
     _url="${SPINIFEX_PREDASTORE_ENDPOINT%/}/eks-backups-system/${EKS_ACCOUNT_ID}/${EKS_CLUSTER_NAME}/${_name}"
     mkdir -p "${SNAPSHOT_DIR}"
-    curl -fsSL \
+    # Capture curl's own exit — piping straight into logger would mask it behind
+    # logger's (near-always 0), so a failed download would look like success and
+    # the caller would wrongly cluster-reset from local etcd.
+    if _out=$(curl -fsSL \
         --aws-sigv4 "aws:amz:${AWS_REGION:-au-mel-1}:s3" \
         --user "${SPINIFEX_PREDASTORE_AKID}:${SPINIFEX_PREDASTORE_SECRET}" \
         -H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
         -o "${SNAPSHOT_DIR}/${_name}" \
-        "${_url}" 2>&1 | logger -t "${LOGTAG}"
+        "${_url}" 2>&1); then
+        [ -n "${_out}" ] && printf '%s\n' "${_out}" | logger -t "${LOGTAG}"
+        return 0
+    else
+        # Capture rc as the first else statement: $? read after `fi` would be the
+        # if's own 0, re-masking the failure we are here to surface.
+        _rc=$?
+        [ -n "${_out}" ] && printf '%s\n' "${_out}" | logger -t "${LOGTAG}"
+        rm -f "${SNAPSHOT_DIR}/${_name}"
+        log "snapshot ${_name} download failed (curl rc=${_rc})"
+        return 1
+    fi
 }
 
 # Instance ID from IMDS keys the per-member directive. IMDSv2 token if enforced,
@@ -109,8 +131,11 @@ fi
 epoch=$(printf '%s' "${line}" | cut -f1)
 action=$(printf '%s' "${line}" | cut -f2)
 snapshot=$(printf '%s' "${line}" | cut -f3)
+snapshot_required=$(printf '%s' "${line}" | cut -f4)
 [ -n "${action}" ] || action=none
 case "${epoch}" in '' | *[!0-9]*) epoch=0 ;; esac
+# Absent 4th field (older gateway) reads as not-required, preserving HA-reform semantics.
+case "${snapshot_required}" in 1) snapshot_required=1 ;; *) snapshot_required=0 ;; esac
 
 if [ "${action}" = "none" ]; then
     log "directive action=none (epoch ${epoch}); nothing to do"
@@ -136,6 +161,13 @@ case "${action}" in
             if fetch_snapshot "${snapshot}"; then
                 restore_arg="--cluster-reset-restore-path=${SNAPSHOT_DIR}/${snapshot}"
                 log "cluster-reset restoring snapshot ${snapshot}"
+            elif [ "${snapshot_required}" = "1" ]; then
+                # Fresh DR seed: no intact local etcd, so a reset without the snapshot
+                # would boot an EMPTY cluster. Block k3s and abort; retry next boot.
+                log "ERROR: required snapshot ${snapshot} could not be fetched; aborting boot to avoid resetting into an EMPTY datastore (retries next boot)"
+                mkdir -p "$(dirname "${BLOCK_MARKER}")"
+                : > "${BLOCK_MARKER}"
+                exit 1
             else
                 log "WARN: snapshot ${snapshot} fetch failed; cluster-reset from local etcd only"
             fi

scripts/images/eks-node/mulga-eks-k3s-recovery_test.sh

@@ -24,7 +24,9 @@ printf '%s\n' "${DIRECTIVE_LINE}"
 EOF
 
 # curl stub: answers the IMDS token PUT + instance-id GET, and materialises a
-# snapshot file for any `-o <path>` download. Records every call.
+# snapshot file for any `-o <path>` download. Records every call. A snapshot
+# download (an `-o` arg) fails with a non-zero exit when CURL_SNAP_FAIL=1, so the
+# harness can exercise the fetch-failure path.
 cat > "${STUBBIN}/curl" <<'EOF'
 #!/bin/sh
 echo "curl $*" >> "${CURL_CALLS}"
@@ -36,7 +38,10 @@ for a in "$@"; do
 done
 prev=""
 for a in "$@"; do
-    [ "$prev" = "-o" ] && echo snapdata > "$a"
+    if [ "$prev" = "-o" ]; then
+        [ "${CURL_SNAP_FAIL:-0}" = "1" ] && { echo "curl: (22) 404" >&2; exit 22; }
+        echo snapdata > "$a"
+    fi
     prev="$a"
 done
 exit 0
@@ -80,7 +85,7 @@ pass() { echo "ok: $*"; }
 
 # run <role> <directive-line>: reset capture files + state, then run the script
 # with every path knob pointed into the temp dir.
-run() {
+_setup_run() {
     _role=$1
     DIRECTIVE_LINE=$2
     K3S_CALLS="${WORK}/k3s.calls"
@@ -89,13 +94,29 @@ run() {
     EPOCH_FILE="${WORK}/recovery.epoch"
     ETCD_DIR="${WORK}/etcd"
     SNAPSHOT_DIR="${WORK}/snapshots"
+    BLOCK_MARKER="${WORK}/run/k3s-restore-block"
     : > "${K3S_CALLS}"
     : > "${CURL_CALLS}"
+    rm -f "${BLOCK_MARKER}"
     printf '%s' "${_role}" > "${ROLE_FILE}"
-    export DIRECTIVE_LINE K3S_CALLS CURL_CALLS ROLE_FILE EPOCH_FILE ETCD_DIR SNAPSHOT_DIR
+    export DIRECTIVE_LINE K3S_CALLS CURL_CALLS ROLE_FILE EPOCH_FILE ETCD_DIR SNAPSHOT_DIR BLOCK_MARKER
     export ENVFILE SNAPSHOT_ENVFILE
     export K3S_BIN=k3s FETCH_BIN=eks-gateway-fetch
-    sh "${SCRIPT}" </dev/null || fail "role=${_role}: non-zero exit"
+}
+
+# run <role> <directive-line>: expects a zero exit (the steady-state path).
+run() {
+    _setup_run "$1" "$2"
+    sh "${SCRIPT}" </dev/null || fail "role=${1}: non-zero exit"
+}
+
+# run_abort <role> <directive-line>: expects a NON-zero exit (the required-snapshot
+# fetch-failure abort). CURL_SNAP_FAIL must be set by the caller.
+run_abort() {
+    _setup_run "$1" "$2"
+    if sh "${SCRIPT}" </dev/null; then
+        fail "role=${1}: expected non-zero exit (required-snapshot abort)"
+    fi
 }
 
 TAB=$(printf '\t')
@@ -149,6 +170,36 @@ grep -q 'aws-sigv4' "${WORK}/curl.calls" \
 [ -f "${WORK}/snapshots/etcd-daily-20260706T000000Z.snap" ] \
     && pass "snapshot: fetched into snapshot dir" || fail "snapshot: file not materialised"
 
+# --- Case 7: REQUIRED snapshot, fetch OK -> restore-path + no block marker ---
+rm -f "${WORK}/recovery.epoch"
+CURL_SNAP_FAIL=0; export CURL_SNAP_FAIL
+run server "5${TAB}cluster-reset${TAB}etcd-frequent-20260709T010000Z.snap${TAB}1"
+grep -q 'cluster-reset-restore-path=.*etcd-frequent-20260709T010000Z.snap' "${WORK}/k3s.calls" \
+    && pass "required-ok: restore-path passed" || fail "required-ok: no restore-path: $(cat "${WORK}/k3s.calls")"
+[ -f "${WORK}/run/k3s-restore-block" ] && fail "required-ok: must not block k3s on success" || pass "required-ok: no block marker"
+
+# --- Case 8: REQUIRED snapshot, fetch FAILS -> abort: block marker, no k3s, no epoch ---
+rm -f "${WORK}/recovery.epoch"
+CURL_SNAP_FAIL=1; export CURL_SNAP_FAIL
+run_abort server "6${TAB}cluster-reset${TAB}etcd-frequent-20260709T010000Z.snap${TAB}1"
+[ -f "${WORK}/run/k3s-restore-block" ] \
+    && pass "required-fail: block marker dropped to stop k3s" || fail "required-fail: no block marker"
+[ -s "${WORK}/k3s.calls" ] && fail "required-fail: must NOT cluster-reset into empty datastore" || pass "required-fail: k3s not reset"
+[ -f "${WORK}/recovery.epoch" ] && fail "required-fail: epoch must not record (retry next boot)" || pass "required-fail: epoch not recorded"
+
+# --- Case 9: NON-required snapshot, fetch FAILS -> local reset, no block, epoch recorded ---
+rm -f "${WORK}/recovery.epoch"
+CURL_SNAP_FAIL=1; export CURL_SNAP_FAIL
+run server "7${TAB}cluster-reset${TAB}etcd-frequent-20260709T010000Z.snap${TAB}0"
+grep -q 'server --config .* --cluster-reset' "${WORK}/k3s.calls" \
+    && pass "optional-fail: falls back to local cluster-reset" || fail "optional-fail: no reset: $(cat "${WORK}/k3s.calls")"
+grep -q 'cluster-reset-restore-path' "${WORK}/k3s.calls" \
+    && fail "optional-fail: must not pass a restore-path for a failed fetch" || pass "optional-fail: local-data reset only"
+[ -f "${WORK}/run/k3s-restore-block" ] && fail "optional-fail: must not block k3s (HA-reform semantics)" || pass "optional-fail: no block marker"
+[ "$(cat "${WORK}/recovery.epoch" 2>/dev/null)" = 7 ] \
+    && pass "optional-fail: epoch recorded" || fail "optional-fail: epoch not recorded"
+CURL_SNAP_FAIL=0; export CURL_SNAP_FAIL
+
 if [ "${FAILS}" -eq 0 ]; then
     echo "PASS: all mulga-eks-k3s-recovery cases"
     exit 0