release: update go.mod for v1.14.0

* fix(eks): GPU node AMI declares uefi boot mode

The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but
the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under
legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at
firmware with zero serial console output.

* fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs

Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
initialize under the closed/proprietary kernel module: dmesg shows
RmInitAdapter failing with "requires use of the NVIDIA open kernel
modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to
nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia
recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no
separate open variant (userspace tools only), so it is unchanged.

* fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons

A cluster with a GPU-only nodegroup has no untainted node: the CP
carries CriticalAddonsOnly (EKS parity keeps user workloads off it)
and every worker carries nvidia.com/gpu. Without tolerations for both,
ebs-csi-controller and aws-load-balancer-controller have nowhere to
schedule and stay Pending forever. Real EKS never needs this since its
control plane isn't a schedulable node, so the stock upstream
manifests carry no such toleration.

* fix(images): give the libguestfs appliance a working resolver

The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
(127.0.0.53), which isn't running inside the appliance chroot, so DNS
is dead until package installs need it. Point it at a real resolver
before package installs, then restore the stub symlink afterward so
the shipped AMI still gets its DNS from systemd-resolved at boot like
a stock cloud image.

* fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind

A GPU's IOMMU group can include an upstream PCIe bridge when the
platform lacks ACS isolation between them. vfio-pci refuses to bind
non-endpoint bridge devices, so Claim/ReclaimByAddress must filter
bridge-class (0x06) members out of the group before touching driver
bindings, rather than failing the whole claim or leaving the bridge
rebound incorrectly.

* fix(ecs): GPU node AMI declares uefi boot mode

The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the
Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS
no OVMF/pflash firmware is attached, so the guest hangs at firmware with
zero serial console output.

Same defect and same fix as the eks-node-gpu entry: both images are built
by the same script from the same stock Ubuntu cloudimg, so both were
EFI-only and only one was corrected.

* fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch

ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary
kernel module. NVIDIA ships no proprietary module for Blackwell (GB202)
and later, so ECS GPU container-instances could not bind those cards on
any driver branch. It is the third verbatim copy of the routine and was
missed when the other two were fixed.

Also pins the driver branch across all three copies. The branch was
resolved at build time via apt-cache search | sort -rn | head -1, which
sorts on the number in the package name — but that number is a branch
alias, not a driver version (nvidia-dkms-535-server installs 580.126.20),
so the ordering key need not match what installs. Resolving also let
identical source produce different drivers on different days.

The pin lives in setup.sh rather than manifest.conf because manifest
variables are not passed into the guest: the setup script is uploaded and
executed by virt-customize --run, with no environment carried across.

* fix(eks): tolerate CP and GPU taints in argocd addon

All six argocd Deployments and its StatefulSet carried no tolerations. A
cluster with a GPU-only nodegroup has no untainted node to place them on:
the control plane carries CriticalAddonsOnly and every worker carries
nvidia.com/gpu, so the pods stay Pending forever.

Same defect the ebs-csi-controller and load-balancer-controller addons
already had. Real EKS never hits it because its control plane is not a
schedulable node, so the stock upstream manifests carry no such
toleration and the divergence is ours to maintain.

---------

Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>

.githooks/pre-commit

@@ -1,17 +0,0 @@
-#!/usr/bin/env bash
-set -e
-
-branch=$(git rev-parse --abbrev-ref HEAD)
-
-# Only enforce preflight on dev and main
-if [ "$branch" != "dev" ] && [ "$branch" != "main" ]; then
-  exit 0
-fi
-
-# Only run preflight if Go files were staged
-if git diff --cached --name-only | grep -q '\.go$'; then
-  echo "Go files staged on $branch — running make preflight..."
-  make preflight
-else
-  echo "No Go files staged, skipping preflight."
-fi

.github/actions/e2e-analyze/analyze.go

@@ -1,7 +1,6 @@
 // Package main implements the e2e-analyze GitHub Action.
 //
-// Stage 1 of docs/development/improvements/e2e-go-failure-analysis.md:
-// cluster failures from go-junit-report's junit-*.xml by error signature,
+// Stage 1: cluster failures from go-junit-report's junit-*.xml by error signature,
 // pick the earliest non-cascade failure per suite as the likely root cause,
 // and render a human-readable report.
 //
@@ -463,8 +462,7 @@ func ParseFile(path string, data []byte) (SuiteReport, error) {
 	return rep, nil
 }
 
-// Render writes the markdown report. Format is documented in
-// docs/development/improvements/e2e-go-failure-analysis.md (Stage 1).
+// Render writes the markdown report (Stage 1).
 func Render(r Report) string {
 	var b strings.Builder
 	fmt.Fprintf(&b, "## %s\n\n", r.Title)

.github/actions/e2e-analyze/bundle.go

@@ -1,7 +1,6 @@
 package main
 
-// Stage 2 of docs/development/improvements/e2e-go-failure-analysis.md:
-// for each failure surfaced by Stage 1, materialise a per-failure bundle
+// Stage 2: for each failure surfaced by Stage 1, materialise a per-failure bundle
 // directory containing the failure summary, a time-window slice of the
 // daemon journal, a testcase-window slice of the suite's stdout log, and
 // any per-test artifacts the harness already dumped at run-time.

.github/actions/e2e-analyze/slice.go

@@ -1,7 +1,6 @@
 package main
 
-// Stage 2 of docs/development/improvements/e2e-go-failure-analysis.md:
-// per-failure log slicers used by bundle.go to materialise time-window /
+// Stage 2: per-failure log slicers used by bundle.go to materialise time-window /
 // testcase-window log excerpts next to each named failure.
 
 import (

.github/actions/otel-job-id/action.yml

@@ -0,0 +1,52 @@
+name: Resolve OTEL job id
+description: >-
+  Append ci.job_id to SPINIFEX_OTEL_ATTRS so telemetry emitted by this job can
+  be pivoted straight back to its GitHub job log. Must run before the step that
+  bakes SPINIFEX_OTEL_ATTRS into a VM's telemetry.env, and requires
+  `actions: read` on the calling job.
+
+inputs:
+  job-name:
+    description: >-
+      Rendered name of the calling job, exactly as it appears in the Actions UI.
+      github.job is only the YAML key and no context exposes the numeric id, so
+      the id has to be looked up by name.
+    required: true
+  token:
+    description: Token used for the Actions jobs API lookup.
+    required: false
+    default: ${{ github.token }}
+
+runs:
+  using: composite
+  steps:
+    - name: Resolve CI job id (best-effort)
+      shell: bash
+      env:
+        GH_TOKEN: ${{ inputs.token }}
+        JOB_NAME: ${{ inputs.job-name }}
+        REPO: ${{ github.repository }}
+        RUN_ID: ${{ github.run_id }}
+      run: |
+        # Never fail the caller: on any lookup error or name mismatch ci.job_id
+        # is simply omitted and the run proceeds with the labels it already has.
+        set +e
+
+        if [ -z "${SPINIFEX_OTEL_ATTRS:-}" ]; then
+          echo "SPINIFEX_OTEL_ATTRS is unset or empty — telemetry is disabled, nothing to label"
+          exit 0
+        fi
+
+        # env.JOB_NAME rather than shell interpolation into the jq filter: job
+        # names carry quotes, slashes and non-ASCII separators that would
+        # otherwise break the expression.
+        JOB_ID=$(gh api "repos/${REPO}/actions/runs/${RUN_ID}/jobs" --paginate \
+          --jq '.jobs[] | select(.name == env.JOB_NAME) | .id' 2>/dev/null | head -n1)
+
+        if [ -n "$JOB_ID" ]; then
+          echo "SPINIFEX_OTEL_ATTRS=${SPINIFEX_OTEL_ATTRS},ci.job_id=${JOB_ID}" >> "$GITHUB_ENV"
+          echo "resolved ci.job_id=${JOB_ID} for job '${JOB_NAME}'"
+        else
+          echo "::warning::could not resolve numeric job id for '${JOB_NAME}' — continuing without ci.job_id"
+        fi
+        exit 0

.github/assets/architecture.svg

@@ -1,225 +1,114 @@
-<svg xmlns="http://www.w3.org/2000/svg" width="900" height="620" viewBox="0 0 900 620">
-  <defs>
-    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
-      <stop offset="0%" style="stop-color:#0a0a0a"/>
-      <stop offset="100%" style="stop-color:#0d1117"/>
-    </linearGradient>
-    <linearGradient id="bar-grad" x1="0%" y1="0%" x2="100%" y2="0%">
-      <stop offset="0%" style="stop-color:#161b22"/>
-      <stop offset="45%" style="stop-color:#1f2028"/>
-      <stop offset="100%" style="stop-color:#161b22"/>
-    </linearGradient>
-    <linearGradient id="client-grad" x1="0%" y1="0%" x2="100%" y2="0%">
-      <stop offset="0%" style="stop-color:#3fb95022"/>
-      <stop offset="100%" style="stop-color:#3fb95008"/>
-    </linearGradient>
-    <linearGradient id="gw-grad" x1="0%" y1="0%" x2="100%" y2="0%">
-      <stop offset="0%" style="stop-color:#DC262633"/>
-      <stop offset="100%" style="stop-color:#7F1D1D14"/>
-    </linearGradient>
-    <linearGradient id="nats-grad" x1="0%" y1="0%" x2="100%" y2="0%">
-      <stop offset="0%" style="stop-color:#F87171"/>
-      <stop offset="50%" style="stop-color:#DC2626"/>
-      <stop offset="100%" style="stop-color:#7F1D1D"/>
-    </linearGradient>
-  </defs>
-
-  <rect width="900" height="620" fill="url(#bg)" rx="12"/>
-
-  <!-- Title bar -->
-  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">SPINIFEX ARCHITECTURE</text>
-  <text x="24" y="52" font-family="'Segoe UI', system-ui, sans-serif" font-size="16" fill="#f5f5f5" font-weight="500">AWS SDK → Gateway → NATS → Service daemons</text>
-  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">stateless · message-driven · horizontally scalable</text>
-
-  <!-- ═══ CLIENT LAYER ════════════════════════════════════ -->
-  <rect x="16" y="72" width="868" height="78" rx="8" fill="url(#client-grad)" stroke="#3fb950" stroke-width="1" stroke-opacity="0.5"/>
-  <text x="32" y="93" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" font-weight="700" letter-spacing="1">CLIENTS</text>
-  <text x="32" y="108" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">Standard AWS tooling — unmodified, no shims, no custom SDK</text>
-
-  <g transform="translate(32, 120)" font-family="'SF Mono', ui-monospace, monospace" font-size="10">
-    <rect x="0" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="43" y="15" fill="#7ee787" text-anchor="middle">aws-cli</text>
-
-    <rect x="94" y="0" width="96" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="142" y="15" fill="#7ee787" text-anchor="middle">Terraform</text>
-
-    <rect x="198" y="0" width="76" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="236" y="15" fill="#7ee787" text-anchor="middle">boto3</text>
-
-    <rect x="282" y="0" width="98" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="331" y="15" fill="#7ee787" text-anchor="middle">aws-sdk-go</text>
-
-    <rect x="388" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="431" y="15" fill="#7ee787" text-anchor="middle">kubectl</text>
-
-    <rect x="482" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="525" y="15" fill="#7ee787" text-anchor="middle">Packer</text>
-
-    <rect x="576" y="0" width="120" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="636" y="15" fill="#7ee787" text-anchor="middle">your application</text>
-  </g>
-
-  <!-- ═══ HTTPS + SigV4 connector ═══════════════════════════ -->
-  <line x1="450" y1="150" x2="450" y2="178" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
-  <rect x="390" y="154" width="120" height="18" rx="9" fill="#0a0a0a" stroke="#DC2626" stroke-width="0.8" stroke-opacity="0.5"/>
-  <text x="450" y="166" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#F87171" text-anchor="middle">HTTPS · SigV4 · :9999</text>
-
-  <!-- ═══ AWS GATEWAY ════════════════════════════════════ -->
-  <rect x="16" y="178" width="868" height="68" rx="8" fill="url(#gw-grad)" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.75"/>
-  <text x="32" y="200" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" font-weight="700" letter-spacing="1">AWS GATEWAY</text>
-  <text x="32" y="215" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">spinifex/services/awsgw · TLS termination · SigV4 verification · IAM policy eval · NATS routing</text>
-
-  <g transform="translate(32, 224)" font-family="'SF Mono', ui-monospace, monospace" font-size="9">
-    <rect x="0" y="0" width="78" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="39" y="11" fill="#FCA5A5" text-anchor="middle">TLS 1.3</text>
-
-    <rect x="86" y="0" width="78" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="125" y="11" fill="#FCA5A5" text-anchor="middle">SigV4 auth</text>
-
-    <rect x="172" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="221" y="11" fill="#FCA5A5" text-anchor="middle">policy eval</text>
-
-    <rect x="278" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="327" y="11" fill="#FCA5A5" text-anchor="middle">req routing</text>
-
-    <rect x="384" y="0" width="88" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="428" y="11" fill="#FCA5A5" text-anchor="middle">rate-limit</text>
-
-    <rect x="480" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="529" y="11" fill="#FCA5A5" text-anchor="middle">audit log</text>
-  </g>
-
-  <!-- ═══ NATS BUS ═══════════════════════════════════════ -->
-  <line x1="450" y1="246" x2="450" y2="272" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
-  <rect x="16" y="272" width="868" height="34" rx="6" fill="url(#nats-grad)" opacity="0.92"/>
-  <text x="32" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#f5f5f5" font-weight="700" letter-spacing="1.5">NATS BUS</text>
-  <text x="140" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#fee2e2" opacity="0.95">ec2.*   ebs.*   s3.*   vpc.*   iam.*</text>
-  <text x="876" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#fee2e2" text-anchor="end" opacity="0.9">queue groups · fan-out · request/reply</text>
-
-  <!-- ═══ DAEMON FANOUT CONNECTORS ═══════════════════════ -->
-  <g stroke="#DC2626" stroke-width="0.9" stroke-dasharray="3,3" opacity="0.55" fill="none">
-    <line x1="100" y1="306" x2="100" y2="328"/>
-    <line x1="275" y1="306" x2="275" y2="328"/>
-    <line x1="450" y1="306" x2="450" y2="328"/>
-    <line x1="625" y1="306" x2="625" y2="328"/>
-    <line x1="800" y1="306" x2="800" y2="328"/>
-  </g>
-
-  <!-- ═══ SERVICE DAEMONS ROW ═══════════════════════════ -->
-  <!-- EC2 -->
-  <g transform="translate(28, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">ec2.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">RunInstances</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">TerminateInstances</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">QMP · cloud-init</text>
-  </g>
-
-  <!-- EBS -->
-  <g transform="translate(204, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">ebs.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">CreateVolume</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">AttachVolume</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">snapshots · WAL</text>
-  </g>
-
-  <!-- S3 -->
-  <g transform="translate(380, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">s3.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">PutObject · GetObject</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Multipart · presigned</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">erasure-coded</text>
-  </g>
-
-  <!-- VPC -->
-  <g transform="translate(556, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">vpc.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Subnets · SGs · RTs</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">ALB · NAT · IGW</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">OVN · OVS</text>
-  </g>
-
-  <!-- IAM -->
-  <g transform="translate(732, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">iam.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Users · Roles · Keys</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Policy evaluation</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">audit trail</text>
-  </g>
-
-  <!-- ═══ DAEMON → BACKEND CONNECTORS ═══════════════════ -->
-  <g stroke="#71717a" stroke-width="0.8" stroke-dasharray="2,3" opacity="0.55" fill="none">
-    <line x1="100" y1="410" x2="100" y2="440"/>
-    <line x1="276" y1="410" x2="276" y2="440"/>
-    <line x1="452" y1="410" x2="452" y2="440"/>
-    <line x1="628" y1="410" x2="628" y2="440"/>
-    <line x1="804" y1="410" x2="804" y2="440"/>
-  </g>
-
-  <!-- ═══ BACKEND RESOURCES BAR ═════════════════════════ -->
-  <rect x="16" y="440" width="868" height="26" rx="5" fill="url(#bar-grad)"/>
-  <text x="450" y="458" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle" letter-spacing="1.5">BACKEND RESOURCES · local hardware · no cloud dependency</text>
-
-  <!-- ═══ BACKEND ROW ═══════════════════════════════════ -->
-  <!-- QEMU/KVM -->
-  <g transform="translate(28, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">hypervisor</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">host kernel · virtio</text>
-  </g>
-
-  <!-- Viperblock -->
-  <g transform="translate(204, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Viperblock</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">block storage</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">NVMe WAL · NBD</text>
-  </g>
-
-  <!-- Predastore -->
-  <g transform="translate(380, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Predastore</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">object storage</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">Reed–Solomon</text>
-  </g>
-
-  <!-- OVN/OVS -->
-  <g transform="translate(556, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">OVN / OVS</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">SDN fabric</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">overlays · flow rules</text>
-  </g>
-
-  <!-- IAM store -->
-  <g transform="translate(732, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Identity store</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">users · keys · rules</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">local · audit-logged</text>
-  </g>
-
-  <!-- ═══ FOOTER NOTES ══════════════════════════════════ -->
-  <g transform="translate(16, 558)">
-    <rect width="868" height="48" rx="6" fill="#0a0a0a" stroke="#27272a" stroke-width="0.8"/>
-    <text x="24" y="20" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">
-      <tspan fill="#F87171" font-weight="600">Queue groups</tspan> load-balance ops across daemon replicas.
-      <tspan fill="#F87171" font-weight="600"> Fan-out</tspan> (no queue group) dispatches describe calls to every node.
-    </text>
-    <text x="24" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">
-      Daemons are <tspan fill="#F87171" font-weight="600">stateless</tspan> — scale horizontally by starting more.
-      No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.
-    </text>
-  </g>
-</svg>
+<svg xmlns="http://www.w3.org/2000/svg" width="1020" height="770" viewBox="0 0 1020 770" role="img" aria-label="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus; stateless ec2/ebs/s3/vpc/iam daemons answer and drive local backends (QEMU/KVM, Viperblock, Predastore, OVN/OVS, identity store).">
+<defs><linearGradient id="nats" x1="0" y1="0" x2="1" y2="0"><stop offset="0" stop-color="#F5652E"/><stop offset="1" stop-color="#C63C0E"/></linearGradient></defs>
+<rect x="0.0" y="0.0" width="1020.0" height="770.0" rx="20" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="28.0" y="42.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#EE4B12" font-weight="600" text-anchor="start" letter-spacing="2.5">SPINIFEX ARCHITECTURE</text>
+<text x="992.0" y="42.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#9AA1A8" font-weight="400" text-anchor="end">stateless · message-driven · horizontally scalable</text>
+<text x="28.0" y="72.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="23" fill="#F5F3EE" font-weight="800" text-anchor="start">AWS SDK → Gateway → NATS → service daemons</text>
+<rect x="28.0" y="92.0" width="964.0" height="92.0" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="48.0" y="119.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">CLIENTS</text>
+<text x="48.0" y="140.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#5B636C" font-weight="400" text-anchor="start">Standard AWS tooling, unmodified. No shims, no custom SDK.</text>
+<rect x="46.0" y="153.0" width="68.2" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="80.1" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">aws-cli</text>
+<rect x="123.2" y="153.0" width="81.4" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="163.9" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">Terraform</text>
+<rect x="213.6" y="153.0" width="55.0" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="241.1" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">boto3</text>
+<rect x="277.6" y="153.0" width="88.0" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="321.6" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">aws-sdk-go</text>
+<rect x="374.6" y="153.0" width="68.2" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="408.7" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">kubectl</text>
+<rect x="451.8" y="153.0" width="61.6" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="482.6" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">Packer</text>
+<rect x="522.4" y="153.0" width="74.8" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="559.8" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">your app</text>
+<line x1="510.0" y1="186.0" x2="510.0" y2="192.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
+<rect x="429.9" y="191.0" width="160.3" height="22.0" rx="11" fill="#1E242B" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.4"/>
+<text x="510.0" y="205.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#EE4B12" font-weight="400" text-anchor="middle" fill-opacity="0.9">HTTPS · SigV4 · :9999</text>
+<line x1="510.0" y1="212.0" x2="510.0" y2="220.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
+<rect x="28.0" y="224.0" width="964.0" height="94.0" rx="13" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
+<text x="48.0" y="251.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#EE4B12" font-weight="700" text-anchor="start" letter-spacing="1.5">AWS GATEWAY</text>
+<text x="48.0" y="272.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">spinifex/services/awsgw · TLS termination · SigV4 verification · IAM policy eval · NATS routing</text>
+<rect x="46.0" y="287.0" width="68.2" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="80.1" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">TLS 1.3</text>
+<rect x="123.2" y="287.0" width="88.0" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="167.2" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">SigV4 auth</text>
+<rect x="220.2" y="287.0" width="94.6" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="267.5" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">policy eval</text>
+<rect x="323.8" y="287.0" width="94.6" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="371.1" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">req routing</text>
+<rect x="427.4" y="287.0" width="88.0" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="471.4" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">rate-limit</text>
+<rect x="524.4" y="287.0" width="81.4" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="565.1" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">audit log</text>
+<line x1="510.0" y1="320.0" x2="510.0" y2="332.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
+<rect x="28.0" y="334.0" width="964.0" height="52.0" rx="12" fill="url(#nats)" fill-opacity="1.0" stroke="none" stroke-width="1" stroke-opacity="1.0"/>
+<rect x="28.0" y="334.0" width="964.0" height="52.0" rx="12" fill="none" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55"/>
+<text x="50.0" y="365.6" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="16" fill="#F5F3EE" font-weight="800" text-anchor="start">NATS BUS</text>
+<text x="178.0" y="364.6" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#F5F3EE" font-weight="400" text-anchor="start" fill-opacity="0.92">ec2.*  ebs.*  s3.*  vpc.*  iam.*</text>
+<text x="972.0" y="363.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F5F3EE" font-weight="400" text-anchor="end" fill-opacity="0.9">queue groups · fan-out · request/reply</text>
+<line x1="119.6" y1="388.0" x2="119.6" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<line x1="314.8" y1="388.0" x2="314.8" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<line x1="510.0" y1="388.0" x2="510.0" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<line x1="705.2" y1="388.0" x2="705.2" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<line x1="900.4" y1="388.0" x2="900.4" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<rect x="28.0" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="44.0" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">ec2.* daemon</text>
+<line x1="44.0" y1="448.0" x2="195.2" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="44.0" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">RunInstances</text>
+<text x="44.0" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">TerminateInstances</text>
+<text x="44.0" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">QMP · cloud-init</text>
+<rect x="223.2" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="239.2" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">ebs.* daemon</text>
+<line x1="239.2" y1="448.0" x2="390.4" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="239.2" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">CreateVolume</text>
+<text x="239.2" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">AttachVolume</text>
+<text x="239.2" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">snapshots · WAL</text>
+<rect x="418.4" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="434.4" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">s3.* daemon</text>
+<line x1="434.4" y1="448.0" x2="585.6" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="434.4" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">PutObject · GetObject</text>
+<text x="434.4" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">Multipart · presigned</text>
+<text x="434.4" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">erasure-coded</text>
+<rect x="613.6" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="629.6" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">vpc.* daemon</text>
+<line x1="629.6" y1="448.0" x2="780.8" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="629.6" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">Subnets · SGs · RTs</text>
+<text x="629.6" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">ALB · NAT · IGW</text>
+<text x="629.6" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">OVN · OVS</text>
+<rect x="808.8" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="824.8" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">iam.* daemon</text>
+<line x1="824.8" y1="448.0" x2="976.0" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="824.8" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">Users · Roles · Keys</text>
+<text x="824.8" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">Policy evaluation</text>
+<text x="824.8" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">audit trail</text>
+<line x1="119.6" y1="516.0" x2="119.6" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<line x1="314.8" y1="516.0" x2="314.8" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<line x1="510.0" y1="516.0" x2="510.0" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<line x1="705.2" y1="516.0" x2="705.2" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<line x1="900.4" y1="516.0" x2="900.4" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<rect x="28.0" y="530.0" width="964.0" height="30.0" rx="9" fill="#1E242B" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="510.0" y="548.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle" letter-spacing="1">BACKEND RESOURCES  ·  local hardware  ·  no cloud dependency</text>
+<rect x="28.0" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="119.6" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">QEMU / KVM</text>
+<text x="119.6" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">hypervisor</text>
+<text x="119.6" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">host kernel · virtio</text>
+<rect x="223.2" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="314.8" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">Viperblock</text>
+<text x="314.8" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">block storage</text>
+<text x="314.8" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">NVMe WAL · NBD</text>
+<rect x="418.4" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="510.0" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">Predastore</text>
+<text x="510.0" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">object storage</text>
+<text x="510.0" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Reed-Solomon</text>
+<rect x="613.6" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="705.2" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">OVN / OVS</text>
+<text x="705.2" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">SDN fabric</text>
+<text x="705.2" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">overlays · flow rules</text>
+<rect x="808.8" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="900.4" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">Identity store</text>
+<text x="900.4" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">users · keys · rules</text>
+<text x="900.4" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">local · audit-logged</text>
+<rect x="28.0" y="670.0" width="964.0" height="72.0" rx="11" fill="#181D23" fill-opacity="0.6" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="48.0" y="697.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400"><tspan fill="#EE4B12" font-weight="700">Queue groups</tspan><tspan fill="#5B636C"> load-balance ops across daemon replicas. </tspan><tspan fill="#EE4B12" font-weight="700">Fan-out</tspan><tspan fill="#5B636C"> (no queue group) dispatches describe calls to every node.</tspan></text>
+<text x="48.0" y="720.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400"><tspan fill="#5B636C">Daemons are </tspan><tspan fill="#EE4B12" font-weight="700">stateless</tspan><tspan fill="#5B636C">: scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.</tspan></text>
+</svg>

.github/assets/platform.svg

@@ -1,144 +1,84 @@
-<svg xmlns="http://www.w3.org/2000/svg" width="900" height="560" viewBox="0 0 900 560">
-  <defs>
-    <linearGradient id="pf-bg" x1="0%" y1="0%" x2="100%" y2="100%">
-      <stop offset="0%" style="stop-color:#0a0a0a"/>
-      <stop offset="100%" style="stop-color:#0d1117"/>
-    </linearGradient>
-    <linearGradient id="hw-grad" x1="0%" y1="0%" x2="0%" y2="100%">
-      <stop offset="0%" style="stop-color:#52525b;stop-opacity:0.18"/>
-      <stop offset="100%" style="stop-color:#52525b;stop-opacity:0.06"/>
-    </linearGradient>
-    <linearGradient id="os-grad" x1="0%" y1="0%" x2="0%" y2="100%">
-      <stop offset="0%" style="stop-color:#a855f7;stop-opacity:0.18"/>
-      <stop offset="100%" style="stop-color:#a855f7;stop-opacity:0.05"/>
-    </linearGradient>
-    <linearGradient id="spx-grad" x1="0%" y1="0%" x2="0%" y2="100%">
-      <stop offset="0%" style="stop-color:#DC2626;stop-opacity:0.28"/>
-      <stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0.1"/>
-    </linearGradient>
-    <linearGradient id="app-grad" x1="0%" y1="0%" x2="0%" y2="100%">
-      <stop offset="0%" style="stop-color:#3fb950;stop-opacity:0.18"/>
-      <stop offset="100%" style="stop-color:#3fb950;stop-opacity:0.05"/>
-    </linearGradient>
-  </defs>
-
-  <rect width="900" height="560" fill="url(#pf-bg)" rx="12"/>
-
-  <!-- Subtle mesh dots background -->
-  <g opacity="0.12" fill="#7F1D1D">
-    <circle cx="40" cy="40" r="1"/><circle cx="120" cy="30" r="1"/><circle cx="220" cy="50" r="1"/>
-    <circle cx="320" cy="25" r="1"/><circle cx="440" cy="45" r="1"/><circle cx="560" cy="30" r="1"/>
-    <circle cx="680" cy="50" r="1"/><circle cx="780" cy="30" r="1"/><circle cx="860" cy="45" r="1"/>
-    <circle cx="60" cy="530" r="1"/><circle cx="160" cy="520" r="1"/><circle cx="260" cy="540" r="1"/>
-    <circle cx="380" cy="525" r="1"/><circle cx="500" cy="540" r="1"/><circle cx="620" cy="520" r="1"/>
-    <circle cx="740" cy="540" r="1"/><circle cx="840" cy="525" r="1"/>
-  </g>
-
-  <!-- Title -->
-  <text x="450" y="32" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" text-anchor="middle" letter-spacing="2.8">THE MULGA PLATFORM</text>
-  <text x="450" y="58" font-family="'Segoe UI', system-ui, sans-serif" font-size="20" font-weight="600" fill="#f5f5f5" text-anchor="middle">AWS-style workflows, on your own stack</text>
-
-  <!-- Layer 4: Applications (top) -->
-  <g transform="translate(60, 95)">
-    <rect width="780" height="88" rx="10" fill="url(#app-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
-    <text x="20" y="26" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" letter-spacing="1.5" font-weight="600">LAYER 4 · APPLICATIONS &amp; TOOLING</text>
-    <text x="20" y="48" font-family="'Segoe UI', sans-serif" font-size="15" fill="#f5f5f5" font-weight="500">Your workloads — unmodified AWS tooling</text>
-
-    <g transform="translate(20, 60)" font-family="'Segoe UI', sans-serif" font-size="10.5">
-      <rect x="0" y="0" width="70" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="35" y="14" fill="#7ee787" text-anchor="middle">aws-cli</text>
-
-      <rect x="78" y="0" width="78" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="117" y="14" fill="#7ee787" text-anchor="middle">Terraform</text>
-
-      <rect x="164" y="0" width="88" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="208" y="14" fill="#7ee787" text-anchor="middle">AWS SDKs</text>
-
-      <rect x="260" y="0" width="88" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="304" y="14" fill="#7ee787" text-anchor="middle">Kubernetes</text>
-
-      <rect x="356" y="0" width="70" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="391" y="14" fill="#7ee787" text-anchor="middle">Packer</text>
-
-      <rect x="434" y="0" width="98" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="483" y="14" fill="#7ee787" text-anchor="middle">Your Services</text>
-    </g>
-  </g>
-
-  <!-- Down arrow -->
-  <g transform="translate(450, 188)" opacity="0.45">
-    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
-    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
-  </g>
-
-  <!-- Layer 3: Spinifex (highlighted) -->
-  <g transform="translate(60, 215)">
-    <rect width="780" height="140" rx="10" fill="url(#spx-grad)" stroke="#DC2626" stroke-width="1.5" stroke-opacity="0.75"/>
-    <text x="20" y="26" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" letter-spacing="1.5" font-weight="600">LAYER 3 · SPINIFEX · AWS ORCHESTRATION</text>
-    <text x="20" y="50" font-family="'Segoe UI', sans-serif" font-size="15" fill="#f5f5f5" font-weight="500">AWS Gateway → NATS → Service Daemons · SigV4, IAM, VPC-aware</text>
-
-    <g transform="translate(20, 66)" font-family="'Segoe UI', sans-serif">
-      <!-- EC2 -->
-      <rect x="0" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
-      <text x="59" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">EC2</text>
-      <text x="59" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">QEMU/KVM VMs</text>
-      <text x="59" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">cloud-init · AMIs</text>
-
-      <!-- EBS -->
-      <rect x="126" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
-      <text x="185" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">EBS</text>
-      <text x="185" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">Viperblock</text>
-      <text x="185" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">WAL · NBD · snapshots</text>
-
-      <!-- S3 -->
-      <rect x="252" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
-      <text x="311" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">S3</text>
-      <text x="311" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">Predastore</text>
-      <text x="311" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Reed–Solomon EC</text>
-
-      <!-- VPC -->
-      <rect x="378" y="0" width="118" height="58" rx="6" fill="#7F1D1D" fill-opacity="0.22" stroke="#F87171" stroke-opacity="0.55" stroke-width="1"/>
-      <text x="437" y="18" font-size="13" fill="#FCA5A5" text-anchor="middle" font-weight="600">VPC</text>
-      <text x="437" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">OVN / OVS</text>
-      <text x="437" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">subnets · SG · ALB</text>
-
-      <!-- IAM -->
-      <rect x="504" y="0" width="118" height="58" rx="6" fill="#7F1D1D" fill-opacity="0.22" stroke="#F87171" stroke-opacity="0.55" stroke-width="1"/>
-      <text x="563" y="18" font-size="13" fill="#FCA5A5" text-anchor="middle" font-weight="600">IAM</text>
-      <text x="563" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">SigV4 · policies</text>
-      <text x="563" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">users · roles · keys</text>
-
-      <!-- Control -->
-      <rect x="630" y="0" width="110" height="58" rx="6" fill="#27272a" fill-opacity="0.8" stroke="#71717a" stroke-opacity="0.6" stroke-width="1"/>
-      <text x="685" y="18" font-size="13" fill="#e4e4e7" text-anchor="middle" font-weight="600">Control</text>
-      <text x="685" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">NATS bus</text>
-      <text x="685" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">spx · daemons</text>
-    </g>
-  </g>
-
-  <!-- Down arrow -->
-  <g transform="translate(450, 360)" opacity="0.45">
-    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
-    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
-  </g>
-
-  <!-- Layer 2: OS -->
-  <g transform="translate(60, 387)">
-    <rect width="780" height="62" rx="10" fill="url(#os-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.5"/>
-    <text x="20" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" letter-spacing="1.5" font-weight="600">LAYER 2 · OPERATING SYSTEM</text>
-    <text x="20" y="44" font-family="'Segoe UI', sans-serif" font-size="14" fill="#f5f5f5">Debian / Ubuntu LTS · systemd · KVM · Linux bridge · headless autoinstall</text>
-  </g>
-
-  <!-- Down arrow -->
-  <g transform="translate(450, 454)" opacity="0.45">
-    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
-    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
-  </g>
-
-  <!-- Layer 1: Hardware -->
-  <g transform="translate(60, 481)">
-    <rect width="780" height="62" rx="10" fill="url(#hw-grad)" stroke="#71717a" stroke-width="1.2" stroke-opacity="0.6"/>
-    <text x="20" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#a1a1aa" letter-spacing="1.5" font-weight="600">LAYER 1 · HARDWARE</text>
-    <text x="20" y="44" font-family="'Segoe UI', sans-serif" font-size="14" fill="#f5f5f5">Single node · multi-node cluster · edge appliance · air-gapped rack · commodity x86 / ARM</text>
-  </g>
-</svg>
+<svg xmlns="http://www.w3.org/2000/svg" width="1000" height="664" viewBox="0 0 1000 664" role="img" aria-label="The Spinifex stack: applications and AWS tooling on top, the Spinifex AWS-compatible service layer, running on standard Linux on CPU/GPU hardware, deployable to Neocloud, on-prem or edge.">
+<rect x="0.0" y="0.0" width="1000.0" height="664.0" rx="20" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<rect x="28.0" y="204.0" width="950.0" height="438.0" rx="16" fill="none" fill-opacity="1.0" stroke="#9AA1A8" stroke-width="1" stroke-opacity="0.12" stroke-dasharray="2 4"/>
+<rect x="25.0" y="201.0" width="950.0" height="438.0" rx="16" fill="none" fill-opacity="1.0" stroke="#9AA1A8" stroke-width="1" stroke-opacity="0.2" stroke-dasharray="2 4"/>
+<rect x="22.0" y="198.0" width="950.0" height="438.0" rx="16" fill="#181D23" fill-opacity="0.25" stroke="#9AA1A8" stroke-width="1.3" stroke-opacity="0.45" stroke-dasharray="2 4"/>
+<text x="36.0" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="500" text-anchor="start">Apps</text>
+<rect x="136.0" y="34.0" width="836.0" height="54.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="156.0" y="66.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="15" fill="#F5F3EE" font-weight="700" text-anchor="start">Your infrastructure &amp; apps</text>
+<rect x="516.7" y="51.0" width="89.3" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.16" stroke="none" stroke-width="1" stroke-opacity="1.0"/>
+<text x="561.4" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#EE4B12" font-weight="500" text-anchor="middle">AI training</text>
+<rect x="613.0" y="51.0" width="76.7" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.16" stroke="none" stroke-width="1" stroke-opacity="1.0"/>
+<text x="651.4" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#EE4B12" font-weight="500" text-anchor="middle">Inference</text>
+<rect x="696.7" y="51.0" width="79.0" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="736.2" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Web &amp; APIs</text>
+<rect x="782.7" y="51.0" width="72.7" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="819.1" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Databases</text>
+<rect x="862.4" y="51.0" width="91.6" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="908.2" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Batch &amp; data</text>
+<text x="36.0" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="500" text-anchor="start">Tooling</text>
+<rect x="136.0" y="102.0" width="836.0" height="54.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="156.0" y="134.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="15" fill="#F5F3EE" font-weight="700" text-anchor="start">AWS tooling</text>
+<rect x="600.9" y="119.0" width="64.1" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.45"/>
+<text x="633.0" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.85">aws-cli</text>
+<rect x="672.0" y="119.0" width="76.7" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.45"/>
+<text x="710.4" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.85">Terraform</text>
+<rect x="755.7" y="119.0" width="64.1" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.45"/>
+<text x="787.8" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.85">kubectl</text>
+<rect x="826.8" y="119.0" width="66.4" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="860.0" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">AWS SDKs</text>
+<rect x="900.2" y="119.0" width="53.8" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="927.1" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Packer</text>
+<text x="484.0" y="184.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="end">↓  AWS API · SigV4</text>
+<text x="624.0" y="184.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">RESPONSE  ↑</text>
+<text x="136.0" y="231.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="15" fill="#F5F3EE" font-weight="700" text-anchor="start">One portable stack.</text>
+<text x="136.0" y="249.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="start">Runs the same on a single node, cluster, edge, or air-gapped rack.</text>
+<rect x="752.3" y="222.0" width="72.4" height="22.0" rx="6" fill="none" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="788.5" y="236.5" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="500" text-anchor="middle">NEOCLOUD</text>
+<rect x="832.7" y="222.0" width="66.1" height="22.0" rx="6" fill="none" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="865.7" y="236.5" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="500" text-anchor="middle">ON-PREM</text>
+<rect x="906.8" y="222.0" width="47.2" height="22.0" rx="6" fill="none" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="930.4" y="236.5" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="500" text-anchor="middle">EDGE</text>
+<text x="152.0" y="295.6" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="16" fill="#F5F3EE" font-weight="800" text-anchor="start">AWS orchestration</text>
+<text x="956.0" y="293.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#EE4B12" font-weight="400" text-anchor="end" fill-opacity="0.9">Gateway → NATS → daemons</text>
+<rect x="150.0" y="312.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="164.0" y="333.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">EC2</text>
+<text x="164.0" y="349.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Compute</text>
+<text x="164.0" y="363.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">QEMU / KVM</text>
+<rect x="422.7" y="312.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="436.7" y="333.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">EBS</text>
+<text x="436.7" y="349.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Block storage</text>
+<text x="436.7" y="363.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">Viperblock</text>
+<rect x="695.3" y="312.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="709.3" y="333.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">S3</text>
+<text x="709.3" y="349.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Object storage</text>
+<text x="709.3" y="363.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">Predastore</text>
+<rect x="150.0" y="382.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="164.0" y="403.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">VPC</text>
+<text x="164.0" y="419.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Networking</text>
+<text x="164.0" y="433.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">OVN / OVS</text>
+<rect x="422.7" y="382.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="436.7" y="403.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">IAM</text>
+<text x="436.7" y="419.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Identity</text>
+<text x="436.7" y="433.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">SigV4 · policies</text>
+<rect x="695.3" y="382.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="709.3" y="403.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">EKS</text>
+<text x="709.3" y="419.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Kubernetes</text>
+<text x="709.3" y="433.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">K8s-conformant</text>
+<rect x="150.0" y="452.0" width="808.0" height="32.0" rx="8" fill="#14181D" fill-opacity="0.6" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0" stroke-dasharray="3 3"/>
+<text x="164.0" y="472.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#9AA1A8" font-weight="400" text-anchor="start">+ ALB · ECS · ECR · RDS and more of the core AWS services</text>
+<text x="944.0" y="471.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#EE4B12" font-weight="400" text-anchor="end">See all →</text>
+<text x="36.0" y="386.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#EE4B12" font-weight="500" text-anchor="start">Spinifex</text>
+<rect x="136.0" y="268.0" width="836.0" height="230.0" rx="12" fill="#EE4B12" fill-opacity="0.07" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
+<text x="36.0" y="536.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="500" text-anchor="start">OS</text>
+<rect x="136.0" y="510.0" width="836.0" height="46.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="156.0" y="537.9" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="700" text-anchor="start">Operating system</text>
+<text x="952.0" y="536.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="end">Debian / Ubuntu LTS · systemd · KVM · Linux bridge · Open vSwitch</text>
+<text x="36.0" y="596.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="500" text-anchor="start">Hardware</text>
+<rect x="136.0" y="568.0" width="413.0" height="50.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="156.0" y="597.9" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="700" text-anchor="start">CPU</text>
+<text x="533.0" y="596.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="end">Intel · AMD · ARM</text>
+<rect x="559.0" y="568.0" width="413.0" height="50.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="579.0" y="597.9" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="700" text-anchor="start">GPU</text>
+<text x="956.0" y="596.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#EE4B12" font-weight="400" text-anchor="end">NVIDIA · AMD</text>
+</svg>

.github/resolve-sibling-refs.sh

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+# Resolve which branch of each sibling repo (viperblock, predastore) the CI
+# workspace should build against. A cross-repo change is developed on a branch
+# of the SAME name in each repo; build against that branch when it exists so the
+# set can be tested together, and fall back to dev otherwise. This mirrors the
+# resolution the e2e path already does in
+# mulga/scripts/tofu-cluster/build-install-artifacts.sh, so unit CI and e2e CI
+# agree on how sibling branches are selected.
+#
+# Reads $BRANCH (the triggering head/ref name) and writes viperblock=<ref> and
+# predastore=<ref> to $GITHUB_OUTPUT for the checkout steps to consume.
+set -euo pipefail
+
+: "${BRANCH:?BRANCH must be set to the triggering branch name}"
+: "${GITHUB_OUTPUT:?GITHUB_OUTPUT must be set}"
+
+for repo in viperblock predastore; do
+  if git ls-remote --exit-code --heads \
+      "https://github.com/mulgadc/${repo}.git" "$BRANCH" >/dev/null 2>&1; then
+    ref="$BRANCH"
+  else
+    ref="dev"
+  fi
+  echo "${repo}: building against ${ref}"
+  echo "${repo}=${ref}" >>"$GITHUB_OUTPUT"
+done

.github/scripts/suite-sets.sh

@@ -0,0 +1,58 @@
+# shellcheck shell=bash
+# shellcheck disable=SC2034  # every variable here is consumed by the sourcing workflow
+# Source from a workflow `run:` block:
+#   . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+#
+# Which e2e suites each topology can run. Sourced by every workflow that drives
+# suites (e2e.yml, e2e-suites.yml, e2e-nightly.yml) so the lists cannot drift
+# apart — they previously disagreed three ways.
+#
+# Eligibility is a property of the suite, not of the workflow: a suite belongs
+# to a topology when all of its assertions are reachable there. lb is
+# multi-only for exactly that reason — on a single node every internet-facing
+# subtest skips for want of a peer, so a single-node lb run reports green
+# having exercised a fraction of the suite.
+
+# Suites runnable against a one-node environment.
+#
+# partialblock gates merges because it earned it, not because it is new: on one
+# node, same binary and workload, it fails against the pre-fix engine (1758 of
+# 8192 halves reverted a generation) and passes against the fixed one (0 of
+# 8192). A suite that has never been shown to go red for the reason it claims
+# does not belong here.
+E2E_SUITES_SINGLE="single iam cert eks ecs storagegrowth partialblock"
+
+# Suites runnable against a multi-node environment.
+E2E_SUITES_MULTI="multinode lb cert"
+
+# grep -xE alternation form, for narrowing a requested set to the eligible one.
+E2E_SUITES_SINGLE_RE="$(printf '%s' "${E2E_SUITES_SINGLE}" | tr ' ' '|')"
+E2E_SUITES_MULTI_RE="$(printf '%s' "${E2E_SUITES_MULTI}" | tr ' ' '|')"
+
+# e2e-nightly runs a narrower set than the two sets above, deliberately: its
+# ~19 cells each have a 35-minute budget and exist to prove that every
+# install / network / host-OS permutation boots and serves. Suite breadth is
+# the PR gate's job, and it already runs on every push. These are subsets, and
+# e2e_suites_assert_subset below enforces that.
+E2E_SUITES_NIGHTLY_SINGLE="single cert"
+E2E_SUITES_NIGHTLY_MULTI="multinode cert lb"
+
+# Fail loudly when a narrowed list names a suite its topology cannot run. This
+# is the guard that stops the lists drifting again: a suite added to a nightly
+# subset without also being declared eligible above stops the job here rather
+# than silently skipping half its assertions in CI.
+#   e2e_suites_assert_subset <label> <subset> <eligible-set>
+e2e_suites_assert_subset() {
+  local label="$1" subset="$2" eligible="$3" suite found eligible_suite
+  for suite in ${subset}; do
+    found=0
+    for eligible_suite in ${eligible}; do
+      if [ "${suite}" = "${eligible_suite}" ]; then found=1; break; fi
+    done
+    if [ "${found}" -ne 1 ]; then
+      echo "::error::${label} names suite '${suite}', which is not eligible for that topology (eligible: ${eligible}). Fix the list in .github/scripts/suite-sets.sh." >&2
+      return 1
+    fi
+  done
+  return 0
+}

.github/workflows/e2e-baremetal.yml

@@ -9,6 +9,11 @@ on:
         required: false
         type: string
         default: ""
+      otel-attrs:
+        description: "OTEL_RESOURCE_ATTRIBUTES value for this run, e.g. ci.run_id=123,ci.cell=20 (caller-specific, so it's an input rather than a workflow-level default)"
+        required: false
+        type: string
+        default: ""
 
 permissions:
   contents: read
@@ -16,6 +21,10 @@ permissions:
 
 env:
   GIT_BRANCH: ${{ github.ref_name }}
+  # Same central ES/APM sink as the other four e2e workflows. Bare-metal had
+  # no default here before — this cell's telemetry never reached the sink at
+  # all (not just missing ci.cell), see bm-bootstrap.sh's telemetry.env step.
+  SPINIFEX_OTLP_ENDPOINT: ${{ vars.SPINIFEX_OTLP_ENDPOINT || 'http://192.168.1.13:8200' }}
 
 jobs:
   e2e_baremetal:
@@ -28,6 +37,9 @@ jobs:
     env:
       BM_ISO_CACHE: /var/cache/spinifex-iso-builder
       ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-20-${{ github.run_id }}
+      # Caller-supplied (workflow_call input); empty for a manual
+      # workflow_dispatch run, which bm-bootstrap.sh treats as "no attrs".
+      SPINIFEX_OTEL_ATTRS: ${{ inputs.otel-attrs }}
     steps:
       - name: Clean Workspace
         run: |

.github/workflows/e2e-ddil.yml

@@ -13,9 +13,16 @@ on:
 
 permissions:
   contents: read
+  # actions: read lets the otel-job-id action look up this job's numeric id via
+  # the Actions API — github.job is only the YAML key, no context exposes it.
+  actions: read
 
 env:
   GIT_BRANCH: ${{ github.ref_name }}
+  # OTLP export to the central Elastic sink (wattle). Repo variable overrides;
+  # empty string disables telemetry entirely.
+  SPINIFEX_OTLP_ENDPOINT: ${{ vars.SPINIFEX_OTLP_ENDPOINT || 'http://192.168.1.13:8200' }}
+  SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }}
 
 jobs:
   ddil:
@@ -24,6 +31,10 @@ jobs:
     timeout-minutes: 60
     env:
       TOFU_LOG: /tmp/e2e-tofu-ddil-${{ github.run_id }}.log
+      # Job-level env wins over the workflow-level default. ci.cell=ddil marks
+      # this workflow's telemetry so a DDIL run's partition-induced errors are
+      # never mistaken for a healthy multi-node run's in the sink.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=ddil
     steps:
       - name: Clean Workspace
         run: sudo rm -rf "$GITHUB_WORKSPACE"/*
@@ -68,6 +79,13 @@ jobs:
           echo "${BIN_DIR}" >> "${GITHUB_PATH}"
           "${BIN_DIR}/gotestsum" --version
 
+      # Adds ci.job_id alongside ci.cell, before Bootstrap bakes the attrs into
+      # the VMs, so a telemetry doc points back at this job's log directly.
+      - name: Resolve CI job id
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: DDIL Real Multi-Node
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -200,6 +218,16 @@ jobs:
           set -euo pipefail
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "telemetry flush"       "$TOFU_LOG" timeout 120 ./flush-telemetry.sh "${SPINIFEX_CI_ENV}" || true
           quiet_run "tofu destroy"          "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
           quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
           ok "teardown complete"
+
+      - name: Upload tofu log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: tofu-log-ddil-${{ github.run_id }}
+          path: ${{ env.TOFU_LOG }}
+          if-no-files-found: ignore
+          retention-days: 30

.github/workflows/e2e-nightly.yml

@@ -11,12 +11,20 @@ on:
 
 permissions:
   contents: read
+  # actions: read lets the "Resolve CI job id" step below look up this job's
+  # numeric id via the Actions API — github.job is only the YAML key, no
+  # context exposes the id itself.
+  actions: read
 
 # Scheduled runs are loaded from the default branch (main), so the schedule
 # trigger must stay in sync on main for cron to fire.
 env:
   GIT_BRANCH: ${{ github.event_name == 'schedule' && 'dev' || github.ref_name }}
   UBUNTU_GOLD_IMAGE: /mnt/disk1/gold-images/spinifex-image-builder-ubuntu-24.04.qcow2
+  # OTLP export to the central Elastic sink (wattle). Repo variable overrides;
+  # empty string disables telemetry entirely.
+  SPINIFEX_OTLP_ENDPOINT: ${{ vars.SPINIFEX_OTLP_ENDPOINT || 'http://192.168.1.13:8200' }}
+  SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }}
 
 jobs:
   matrix:
@@ -35,12 +43,18 @@ jobs:
           - { cell: 10, install: binary, topo: real-multi, net: dhcp,   host: ubuntu-24.04, arch: x86_64, runner: ci-multi,  extra: "" }
           - { cell: 17, install: binary, topo: single,     net: static, host: debian-13,    arch: x86_64, runner: ci-pseudo, extra: "tofu-examples" }
           - { cell: 18, install: binary, topo: single,     net: static, host: debian-13,    arch: x86_64, runner: ci-single, extra: "reboot-single" }
+          - { cell: 19, install: binary, topo: single,     net: nat,    host: debian-13,    arch: x86_64, runner: ci-single, extra: "nat-single" }
 
     runs-on: [ self-hosted, "${{ matrix.runner }}" ]
     timeout-minutes: 35
     env:
       TOFU_LOG: /tmp/e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}.log
       ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}
+      # Job-level env wins over the workflow-level default above, so telemetry
+      # from this cell carries ci.cell/ci.matrix. Nine cells run under one
+      # run id and every cell names its hosts node1/node2/node3, so without
+      # this every cell's telemetry collapses into one unattributable bucket.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=${{ matrix.cell }},ci.matrix=${{ matrix.topo }}/${{ matrix.net }}/${{ matrix.host }}
 
     steps:
       # blank or scheduled run = full matrix.
@@ -87,6 +101,16 @@ jobs:
         with:
           path: spinifex
 
+      # Nine cells share one run id and every cell names its hosts
+      # node1/node2/node3, so ci.cell above separates them and ci.job_id here
+      # links each cell's telemetry to the job log that produced it. Runs
+      # before Bootstrap bakes SPINIFEX_OTEL_ATTRS into the VM's telemetry.env.
+      - name: Resolve CI job id
+        if: steps.gate.outputs.selected == 'true'
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: "cell-${{ matrix.cell }} (${{ matrix.topo }}/${{ matrix.net }}/${{ matrix.host }}${{ matrix.extra && format('/{0}', matrix.extra) || '' }})"
+
       - name: Build Install Artifacts
         if: steps.gate.outputs.selected == 'true'
         working-directory: mulga/scripts/tofu-cluster
@@ -121,7 +145,12 @@ jobs:
         env:
           MATRIX_NET: ${{ matrix.net }}
         run: |
-          if [ "$MATRIX_NET" = "dhcp" ]; then
+          if [ "$MATRIX_NET" = "nat" ]; then
+            # Routed-NAT mode with a static public pool: exercises both the
+            # Tier-1 transit datapath and the Tier-2 EIP ingress lane.
+            export SPX_EXTERNAL_MODE=nat
+            export SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}"
+          elif [ "$MATRIX_NET" = "dhcp" ]; then
             unset SPX_EXTERNAL_POOL
             export SPX_EXTERNAL_SOURCE=dhcp
           else
@@ -137,13 +166,20 @@ jobs:
         if: steps.gate.outputs.selected == 'true' && matrix.topo == 'single' && matrix.extra == ''
         working-directory: mulga/scripts/tofu-cluster
         run: |
+          set -euo pipefail
+          # Nightly runs a documented subset of the single-node set; the assert
+          # fails the cell if that subset ever names a suite this topology
+          # cannot fully exercise.
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          e2e_suites_assert_subset "e2e-nightly single" "${E2E_SUITES_NIGHTLY_SINGLE}" "${E2E_SUITES_SINGLE}"
+
           INVENTORY=$(tofu output -json inventory)
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
           ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
             "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
-             for SUITE in single cert lb; do \
+             for SUITE in ${E2E_SUITES_NIGHTLY_SINGLE}; do \
                echo \"::group::suite \$SUITE\"; \
                date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
                GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
@@ -154,10 +190,37 @@ jobs:
                if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
              done; exit \$FAIL"
 
+      - name: Run NAT Uplink E2E
+        if: steps.gate.outputs.selected == 'true' && matrix.extra == 'nat-single'
+        working-directory: mulga/scripts/tofu-cluster
+        # Skips cleanly (exit 0) when the tested branch predates the natuplink
+        # suite so scheduled runs against dev stay green until the merge lands.
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; \
+             if [ ! -f natuplink.test ]; then echo 'natuplink.test not in tests tar (branch predates suite) — skipping'; exit 0; fi; \
+             echo '::group::suite natuplink'; \
+             date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-natuplink.start'; \
+             GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+               ARTIFACT_DIR='${ARTIFACT_DIR}' \
+               ./natuplink.test -test.v -test.timeout=30m \
+               2>&1 | tee '${ARTIFACT_DIR}/test-natuplink.log'; \
+             rc=\${PIPESTATUS[0]}; echo '::endgroup::'; \
+             if [ \$rc -ne 0 ]; then echo 'natuplink: FAILED' >> '${ARTIFACT_DIR}/suite-status.txt'; fi; exit \$rc"
+
       - name: Run Multi Node E2E
         if: steps.gate.outputs.selected == 'true' && matrix.topo == 'real-multi'
         working-directory: mulga/scripts/tofu-cluster
         run: |
+          set -euo pipefail
+          # See the single-node step: same subset guard, multi-node set.
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          e2e_suites_assert_subset "e2e-nightly multi" "${E2E_SUITES_NIGHTLY_MULTI}" "${E2E_SUITES_MULTI}"
+
           INVENTORY=$(tofu output -json inventory)
           NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
@@ -172,7 +235,7 @@ jobs:
 
           ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
             "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
-             for SUITE in multinode cert lb; do \
+             for SUITE in ${E2E_SUITES_NIGHTLY_MULTI}; do \
                echo \"::group::suite \$SUITE\"; \
                date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
                GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
@@ -312,9 +375,19 @@ jobs:
             TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE}"
           fi
           tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          timeout 120 ./flush-telemetry.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
           tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1
           ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
 
+      - name: Upload tofu log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: tofu-log-cell-${{ matrix.cell }}-${{ github.run_id }}
+          path: ${{ env.TOFU_LOG }}
+          if-no-files-found: ignore
+          retention-days: 14
+
   # Bare-metal single-node suite. Doesn't fit the tofu matrix shape, so it
   # runs as a sibling job via the reusable e2e-baremetal workflow. Gated like a
   # matrix cell ("cell 20"): every weeknight cron, or dispatch with 20 selected
@@ -331,4 +404,8 @@ jobs:
     uses: ./.github/workflows/e2e-baremetal.yml
     with:
       ref: ${{ github.event_name == 'schedule' && 'dev' || github.ref_name }}
+      # A reusable-workflow call gets none of this workflow's context (env:
+      # above, matrix.cell) for free — cell-20 needs its own attrs string.
+      # There's no matrix leg here, just the one fixed baremetal cell.
+      otel-attrs: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=20,ci.matrix=baremetal/pxe/real-hw
     secrets: inherit

.github/workflows/e2e-suites.yml

@@ -3,11 +3,14 @@
 # Restructures the looped e2e.yml into env-grouped job blocks so each suite
 # surfaces as its own node in the Actions graph (mulga-siv-283):
 #
-#   single_provision ─┬─ single_suite (matrix: single, cert, lb, eks, iam)
+#   single_provision ─┬─ single_suite (matrix: the single-node-eligible suites)
 #                     └─ single_teardown (if: always)
-#   multi_provision  ─┬─ multi_suite  (matrix: multinode)
+#   multi_provision  ─┬─ multi_suite  (matrix: the multi-node-eligible suites)
 #                     └─ multi_teardown (if: always)
 #
+# Both matrices come from .github/scripts/suite-sets.sh, the one definition of
+# which suites a topology can run, shared with e2e.yml and e2e-nightly.yml.
+#
 # Constraint: tofu state is runner-filesystem-local, so *_provision and
 # *_teardown must execute on the same self-hosted runner (one runner per label
 # today). Teardown reuses the persisted mulga/ tofu-cluster workspace and does
@@ -25,9 +28,13 @@ on:
   workflow_dispatch:
     inputs:
       single_suites:
-        description: "Single-node suites to run as matrix legs"
+        description: "Single-node suites to run as matrix legs (empty = every single-node-eligible suite, per .github/scripts/suite-sets.sh)"
         required: false
-        default: "single cert lb eks iam"
+        default: ""
+      multi_suites:
+        description: "Multi-node suites to run as matrix legs (empty = every multi-node-eligible suite, per .github/scripts/suite-sets.sh)"
+        required: false
+        default: ""
       run_multinode:
         description: "Also run the multinode env block"
         type: boolean
@@ -40,10 +47,18 @@ on:
 
 permissions:
   contents: read
+  # actions: read lets the otel-job-id action look up a provision job's numeric
+  # id via the Actions API — github.job is only the YAML key, no context
+  # exposes it.
+  actions: read
 
 env:
   GIT_BRANCH: ${{ github.ref_name }}
   TEST_RUN: ${{ inputs.test_run || '' }}
+  # OTLP export to the central Elastic sink (wattle). Repo variable overrides;
+  # empty string disables telemetry entirely.
+  SPINIFEX_OTLP_ENDPOINT: ${{ vars.SPINIFEX_OTLP_ENDPOINT || 'http://192.168.1.13:8200' }}
+  SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }}
 
 jobs:
   # ---------------------------------------------------------------------------
@@ -60,6 +75,13 @@ jobs:
       matrix: ${{ steps.suites.outputs.matrix }}
     env:
       TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
+      # This job (not single_suite) is where "Bootstrap via Binary Install"
+      # bakes SPINIFEX_OTEL_ATTRS into the VM's telemetry.env, before the
+      # suite matrix even exists — single_suite's legs share this one
+      # already-booted VM, so daemon telemetry can't be scoped any finer
+      # than "single env" per suite leg. ci.cell=single distinguishes it
+      # from multi_provision's VMs within the same run id.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=single
     steps:
       - name: Reap Orphan Procs
         run: |
@@ -103,7 +125,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -111,12 +133,22 @@ jobs:
       - name: Re-enable Go toolchain auto-download
         run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
 
+      # Default and allow-list both come from the shared definition, so this
+      # workflow cannot drift from e2e.yml about what a single node can run.
       - name: Compute suite matrix
         id: suites
+        env:
+          REQUESTED: ${{ inputs.single_suites }}
         run: |
-          LIST="${{ inputs.single_suites || 'single cert lb eks iam' }}"
+          set -euo pipefail
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          LIST="${REQUESTED:-${E2E_SUITES_SINGLE}}"
+          e2e_suites_assert_subset "single_suites" "${LIST}" "${E2E_SUITES_SINGLE}"
           echo "list=${LIST}" >> "$GITHUB_OUTPUT"
-          JSON=$(printf '%s\n' $LIST | jq -R . | jq -cs .)
+          # Unquoted on purpose: the space-separated list must word-split into
+          # one JSON element per suite.
+          # shellcheck disable=SC2086
+          JSON=$(printf '%s\n' ${LIST} | jq -R . | jq -cs .)
           echo "matrix=${JSON}" >> "$GITHUB_OUTPUT"
 
       - name: Reclaim docker build cache
@@ -126,6 +158,14 @@ jobs:
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
+      # ci.job_id names the provision job, not the suite job that later drives
+      # the env: the suite legs share this env and its baked telemetry.env, so
+      # the provisioner is the only job the daemon telemetry can belong to.
+      - name: Resolve CI job id
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: Single env · provision
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -263,6 +303,15 @@ jobs:
             "$VM":/tmp/start-net-capture.sh
           ssh "${SSH_OPTS[@]}" "$VM" "bash /tmp/start-net-capture.sh" || true
 
+      - name: Upload tofu log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: tofu-log-single-${{ github.run_id }}
+          path: ${{ env.TOFU_LOG }}
+          if-no-files-found: ignore
+          retention-days: 14
+
   single_suite:
     name: Single env · suite
     needs: single_provision
@@ -281,7 +330,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -458,10 +507,20 @@ jobs:
           reap TERM; sleep 2; reap KILL
           find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
           quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "telemetry flush"       "$TOFU_LOG" timeout 120 ./flush-telemetry.sh "${SPINIFEX_CI_ENV}" || true
           quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
           quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
           ok "teardown complete"
 
+      - name: Upload tofu log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: tofu-log-single-td-${{ github.run_id }}
+          path: ${{ env.TOFU_LOG }}
+          if-no-files-found: ignore
+          retention-days: 14
+
   # ---------------------------------------------------------------------------
   # Multinode env
   # ---------------------------------------------------------------------------
@@ -474,8 +533,13 @@ jobs:
       node1_ip: ${{ steps.cluster.outputs.node1_ip }}
       ssh_key: ${{ steps.cluster.outputs.ssh_key }}
       nodes_csv: ${{ steps.cluster.outputs.nodes_csv }}
+      suites: ${{ steps.suites.outputs.list }}
+      matrix: ${{ steps.suites.outputs.matrix }}
     env:
       TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
+      # See single_provision above — daemon telemetry is baked in here, at
+      # provision time, before the multi_suite matrix exists.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=multi
     steps:
       - name: Reap Orphan Procs
         run: |
@@ -516,7 +580,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -531,6 +595,12 @@ jobs:
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
+      # See the matching note on single_provision.
+      - name: Resolve CI job id
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: Multi env · provision
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -578,14 +648,36 @@ jobs:
           echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
           echo "nodes_csv=${NODES_CSV}" >> "$GITHUB_OUTPUT"
 
-      - name: Build multinode binary on runner
+      # Mirrors single_provision's step: the multi leg previously ran multinode
+      # alone, silently dropping the lb and cert coverage that only a
+      # multi-node env can complete.
+      - name: Compute suite matrix
+        id: suites
+        env:
+          REQUESTED: ${{ inputs.multi_suites }}
+        run: |
+          set -euo pipefail
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          LIST="${REQUESTED:-${E2E_SUITES_MULTI}}"
+          e2e_suites_assert_subset "multi_suites" "${LIST}" "${E2E_SUITES_MULTI}"
+          echo "list=${LIST}" >> "$GITHUB_OUTPUT"
+          # Unquoted on purpose: the space-separated list must word-split into
+          # one JSON element per suite.
+          # shellcheck disable=SC2086
+          JSON=$(printf '%s\n' ${LIST} | jq -R . | jq -cs .)
+          echo "matrix=${JSON}" >> "$GITHUB_OUTPUT"
+
+      - name: Build multinode binaries on runner
         working-directory: spinifex/tests/e2e
         run: |
           set -euo pipefail
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          banner "build multinode, teardown"
+          # teardown always builds: the run-scoped sweep binary runs in the
+          # teardown job, it is never a selectable suite leg.
+          SUITES_BUILD="${{ steps.suites.outputs.list }} teardown"
+          banner "build ${SUITES_BUILD// /, }"
           group_open "go test -c"
-          make e2e-build SUITES_BUILD="multinode teardown"
+          make e2e-build SUITES_BUILD="${SUITES_BUILD}"
           group_close
           ok "$(basename _bin/multinode.test) $(stat -c '%s' _bin/multinode.test | numfmt --to=iec)"
 
@@ -612,6 +704,15 @@ jobs:
             ssh "${SSH_OPTS[@]}" tf-user@"$ip" "bash /tmp/start-net-capture.sh" || true
           done
 
+      - name: Upload tofu log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: tofu-log-multi-${{ github.run_id }}
+          path: ${{ env.TOFU_LOG }}
+          if-no-files-found: ignore
+          retention-days: 14
+
   multi_suite:
     name: Multi env · suite
     needs: multi_provision
@@ -620,7 +721,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        suite: [multinode]
+        suite: ${{ fromJSON(needs.multi_provision.outputs.matrix) }}
     env:
       NODE1_IP: ${{ needs.multi_provision.outputs.node1_ip }}
       SSH_KEY: ${{ needs.multi_provision.outputs.ssh_key }}
@@ -631,7 +732,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -779,10 +880,20 @@ jobs:
           reap TERM; sleep 2; reap KILL
           find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
           quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "telemetry flush"       "$TOFU_LOG" timeout 120 ./flush-telemetry.sh "${SPINIFEX_CI_ENV}" || true
           quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
           quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
           ok "teardown complete"
 
+      - name: Upload tofu log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: tofu-log-multi-td-${{ github.run_id }}
+          path: ${{ env.TOFU_LOG }}
+          if-no-files-found: ignore
+          retention-days: 14
+
   e2e_suites_status:
     name: E2E suite blocks · summary
     if: always()

.github/workflows/e2e.yml

@@ -26,61 +26,42 @@ on:
 permissions:
   contents: read
   statuses: write
+  # actions: read lets the otel-job-id action look up a job's numeric id via
+  # the Actions API — github.job is only the YAML key, no context exposes it.
+  actions: read
 
 env:
   GIT_BRANCH: ${{ github.ref_name }}
-  SUITES: ${{ inputs.suites || 'e2e-single e2e-multinode e2e-cert e2e-lb e2e-iam' }}
+  SUITES: ${{ inputs.suites || 'e2e-single e2e-multinode e2e-cert e2e-lb e2e-iam e2e-partialblock' }}
   TEST_RUN: ${{ inputs.test_run || '' }}
+  # OTLP export to the central Elastic sink (wattle). Repo variable overrides;
+  # empty string disables telemetry entirely.
+  SPINIFEX_OTLP_ENDPOINT: ${{ vars.SPINIFEX_OTLP_ENDPOINT || 'http://192.168.1.13:8200' }}
+  SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }}
 
 jobs:
-  # Shadow-mode suite selector. Computes which suites the diff *would* run and logs it to the job summary.
-  # Does NOT gate or override the real SUITES env yet — it runs alongside the full matrix so we
-  # can compare predicted vs actual before flipping it on.
-  select_suites:
-    name: Select Suites (shadow)
-    runs-on: ubuntu-26.04
-    timeout-minutes: 5
-    steps:
-      - uses: actions/checkout@v7
-        with:
-          fetch-depth: 0
-      - uses: actions/setup-go@v6
-        with:
-          go-version-file: go.mod
-          cache: false
-      - name: Compute targeted suite selection
-        run: |
-          set -euo pipefail
-          BASE="${{ github.event.before }}"
-          HEAD="${{ github.sha }}"
-          # First push to a branch reports an all-zero before-SHA; fall back to
-          # the first parent so the diff is still meaningful.
-          if [ -z "$BASE" ] || [ "$BASE" = "0000000000000000000000000000000000000000" ]; then
-            BASE="${HEAD}~1"
-          fi
-          echo "## Targeted suite selection (shadow)" >> "$GITHUB_STEP_SUMMARY"
-          echo '```' >> "$GITHUB_STEP_SUMMARY"
-          echo "base=$BASE head=$HEAD" >> "$GITHUB_STEP_SUMMARY"
-          go run ./tests/e2e/select-suites/cmd/select-suites \
-            --manifest docs/service-interfaces.yaml \
-            --base-sha "$BASE" --head-sha "$HEAD" --repo-root . \
-            2>>"$GITHUB_STEP_SUMMARY" \
-            | tee -a "$GITHUB_STEP_SUMMARY"
-          echo '```' >> "$GITHUB_STEP_SUMMARY"
-          echo "Shadow only — actual run uses SUITES='${SUITES}'." >> "$GITHUB_STEP_SUMMARY"
-
   e2e_single:
     name: Single-Node E2E
-    # Skip when the dispatch input requests no in-VM suites (e.g.
-    # suites='e2e-multinode' only). Push triggers and the default SUITES list
-    # both include single/cert/lb so the inputs.suites null-check keeps the
-    # job running on plain PR pushes.
-    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-lb') || contains(github.event.inputs.suites, 'e2e-eks') || contains(github.event.inputs.suites, 'e2e-iam')
+    # The dispatch check gates on exactly the suites this job's Run step
+    # allow-lists, i.e. E2E_SUITES_SINGLE in .github/scripts/suite-sets.sh —
+    # an `if:` expression cannot source that file, so this one list is
+    # hand-mirrored and must be updated with it. Gating on a suite the job
+    # won't run — lb, which only executes in the multinode job — would
+    # provision a tofu VM and pay the Spinifex install only to run nothing and
+    # tear down. Push triggers keep running via the null-check on
+    # inputs.suites.
+    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-iam') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-eks') || contains(github.event.inputs.suites, 'e2e-ecs') || contains(github.event.inputs.suites, 'e2e-storagegrowth') || contains(github.event.inputs.suites, 'e2e-partialblock')
     runs-on: [self-hosted, ci-single]
     timeout-minutes: 50
     env:
       TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
       ARTIFACT_DIR: /tmp/spinifex-e2e-artifacts-${{ github.run_id }}
+      # Job-level env wins over the workflow-level default. This job and
+      # e2e_multi share one run id and both name their hosts node1/node2/node3,
+      # so without ci.cell their telemetry collapses into one bucket. This job
+      # is where "Bootstrap via Binary Install" bakes SPINIFEX_OTEL_ATTRS into
+      # the VM's telemetry.env, so the label has to be set here.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=single
     steps:
       - name: Reap Orphan Procs
         run: |
@@ -130,7 +111,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -145,6 +126,13 @@ jobs:
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
+      # Adds ci.job_id alongside ci.cell, before Bootstrap bakes the attrs into
+      # the VM, so a telemetry doc points back at this job's log directly.
+      - name: Resolve CI job id
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: Single-Node E2E
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -308,14 +296,23 @@ jobs:
         id: e2e
         run: |
           set +e
-          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(multinode)$' | tr '\n' ' ')
+          # Allow-list, not a deny-list: a suite runs here only if named. Deny-lists
+          # silently fan a newly added suite out to every job, which is how lb ended up
+          # running twice per e2e run. The allow-list itself is shared with the other
+          # e2e workflows so the three cannot disagree about what a topology can run;
+          # suite-sets.sh also carries the per-suite rationale.
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -xE "${E2E_SUITES_SINGLE_RE}" | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
             echo "no in-VM suites requested (SUITES=${SUITES}); skipping"
             echo "rc=0" >> "$GITHUB_OUTPUT"
             exit 0
           fi
           if [ -n "${TEST_RUN}" ]; then
-            TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+            # The value is expanded into a command that a remote shell re-parses over
+            # ssh, so single-quote it — otherwise regex metacharacters like ( | ) $ are
+            # taken as shell syntax on the VM and bash fails before the test binary runs.
+            TEST_RUN_FLAG="-test.run='${TEST_RUN}'"
           else
             TEST_RUN_FLAG=""
           fi
@@ -457,10 +454,20 @@ jobs:
           reap KILL
           find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
           quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "telemetry flush"       "$TOFU_LOG" timeout 120 ./flush-telemetry.sh "${SPINIFEX_CI_ENV}" || true
           quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
           quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
           ok "teardown complete"
 
+      - name: Upload tofu log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: tofu-log-single-${{ github.run_id }}
+          path: ${{ env.TOFU_LOG }}
+          if-no-files-found: ignore
+          retention-days: 14
+
       - name: Fail job if any suite failed
         if: always()
         run: |
@@ -472,12 +479,19 @@ jobs:
 
   e2e_multi:
     name: Multi-Node E2E
-    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-multinode')
+    # The dispatch check gates on exactly the suites this job's Run step
+    # allow-lists (multinode|lb|cert) so lb — which only runs here — still
+    # provisions the 3-node env when requested on its own, and the env stays
+    # unprovisioned when none of them are asked for.
+    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-multinode') || contains(github.event.inputs.suites, 'e2e-lb') || contains(github.event.inputs.suites, 'e2e-cert')
     runs-on: [self-hosted, ci-multi]
     timeout-minutes: 60
     env:
       TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
       ARTIFACT_DIR: /tmp/spinifex-e2e-multi-artifacts-${{ github.run_id }}
+      # See the matching note on e2e_single: ci.cell is what separates this
+      # job's 3-node telemetry from the single-node job's under one run id.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=multi
     steps:
       - name: Reap Orphan Procs
         run: |
@@ -527,7 +541,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -542,6 +556,12 @@ jobs:
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
+      # See the matching note on e2e_single.
+      - name: Resolve CI job id
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: Multi-Node E2E
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -683,14 +703,18 @@ jobs:
         id: e2e
         run: |
           set +e
-          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(single|eks|iam)$' | tr '\n' ' ')
+          # Allow-list, not a deny-list — see the single-node job for why.
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -xE "${E2E_SUITES_MULTI_RE}" | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
             echo "no multinode-eligible suites requested (SUITES=${SUITES}); skipping"
             echo "rc=0" >> "$GITHUB_OUTPUT"
             exit 0
           fi
           if [ -n "${TEST_RUN}" ]; then
-            TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+            # Single-quote for the remote shell — see the single-node job for the
+            # metacharacter-injection reason.
+            TEST_RUN_FLAG="-test.run='${TEST_RUN}'"
           else
             TEST_RUN_FLAG=""
           fi
@@ -871,10 +895,20 @@ jobs:
           reap KILL
           find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
           quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "telemetry flush"       "$TOFU_LOG" timeout 120 ./flush-telemetry.sh "${SPINIFEX_CI_ENV}" || true
           quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
           quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
           ok "teardown complete"
 
+      - name: Upload tofu log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: tofu-log-multi-${{ github.run_id }}
+          path: ${{ env.TOFU_LOG }}
+          if-no-files-found: ignore
+          retention-days: 14
+
       - name: Fail job if any suite failed
         if: always()
         run: |
@@ -911,3 +945,136 @@ jobs:
             -f context=e2e \
             -f description="$DESC" \
             -f target_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
+
+  # Emits one summary doc per run to the central Elastic sink, keyed by
+  # labels.ci_run_id, so Kibana can list "did run X pass, and where are its
+  # artifacts" without hand-aggregating traces-apm*/logs-apm.error*. This is
+  # a separate self-hosted job (not a step on e2e_status) because e2e_status
+  # runs on GitHub-hosted ubuntu-26.04, which has no route to the private
+  # 192.168.1.13 sink, and moving e2e_status off GH-hosted would risk the PR
+  # status-report check.
+  e2e_es_summary:
+    name: ES CI Run Summary
+    needs: [e2e_single, e2e_multi]
+    if: always()
+    runs-on: [self-hosted, ci-single]
+    timeout-minutes: 5
+    permissions:
+      actions: read
+    env:
+      # Repo var overrides; defaults to the LAN Elastic sink reachable from
+      # ci-single/ci-multi runners. Anonymous ES requests are superuser on
+      # this host, so no credentials are needed for the POST.
+      MULGA_ES_SINK: ${{ vars.MULGA_ES_SINK || '192.168.1.13:9200' }}
+    steps:
+      # Either leg may not have run (e.g. a workflow_dispatch that only
+      # requested single-node suites), so a missing artifact is tolerated —
+      # the aggregation script below treats an absent suite-status.txt as
+      # "no failures from that leg".
+      - name: Download single-node artifacts
+        uses: actions/download-artifact@v8
+        continue-on-error: true
+        with:
+          name: e2e-artifacts-${{ github.run_id }}
+          path: ${{ github.workspace }}/es-summary-single
+
+      - name: Download multi-node artifacts
+        uses: actions/download-artifact@v8
+        continue-on-error: true
+        with:
+          name: e2e-multi-artifacts-${{ github.run_id }}
+          path: ${{ github.workspace }}/es-summary-multi
+
+      - name: Build and post CI run summary
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+          RUN_ID: ${{ github.run_id }}
+          RUN_ATTEMPT: ${{ github.run_attempt }}
+          WORKFLOW: ${{ github.workflow }}
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+          SHA: ${{ github.sha }}
+          SERVER_URL: ${{ github.server_url }}
+          SINGLE_RESULT: ${{ needs.e2e_single.result }}
+          MULTI_RESULT: ${{ needs.e2e_multi.result }}
+          SINGLE_DIR: ${{ github.workspace }}/es-summary-single
+          MULTI_DIR: ${{ github.workspace }}/es-summary-multi
+        run: |
+          set -euo pipefail
+
+          SINGLE_STATUS="${SINGLE_DIR}/suite-status.txt"
+          MULTI_STATUS="${MULTI_DIR}/suite-status.txt"
+
+          # Collect "<suite>: FAILED" lines from both legs' suite-status.txt
+          # (either may be absent — a leg that didn't run, or a passing leg
+          # that never wrote the file). Accumulate as newline-separated text
+          # rather than a bash array so this stays correct under `set -u`
+          # even when nothing failed.
+          FAILED_SUITES=""
+          for f in "$SINGLE_STATUS" "$MULTI_STATUS"; do
+            [ -f "$f" ] || continue
+            while IFS= read -r line; do
+              case "$line" in
+                *": FAILED") FAILED_SUITES="${FAILED_SUITES}${line%: FAILED}"$'\n' ;;
+              esac
+            done < "$f"
+          done
+
+          FAILED_JSON=$(printf '%s' "$FAILED_SUITES" | jq -R -s 'split("\n") | map(select(length > 0))')
+          FAILED_COUNT=$(printf '%s' "$FAILED_JSON" | jq 'length')
+
+          # Mirror e2e_status's single_ok/multi_ok logic exactly so the PR
+          # status and this ES doc can never disagree on overall pass/fail.
+          single_ok=true
+          case "$SINGLE_RESULT" in success | skipped) ;; *) single_ok=false ;; esac
+          multi_ok=true
+          case "$MULTI_RESULT" in success | skipped) ;; *) multi_ok=false ;; esac
+
+          STATE=success
+          if [ "$single_ok" != true ] || [ "$multi_ok" != true ] || [ "$FAILED_COUNT" -gt 0 ]; then
+            STATE=failure
+          fi
+
+          RUN_STARTED_AT=$(gh api "repos/${REPO}/actions/runs/${RUN_ID}" --jq .run_started_at)
+          START_EPOCH=$(date -u -d "${RUN_STARTED_AT}" +%s)
+          NOW_EPOCH=$(date -u +%s)
+          DURATION=$((NOW_EPOCH - START_EPOCH))
+
+          DOC=$(jq -n \
+            --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
+            --arg run_id "${RUN_ID}" \
+            --arg workflow "${WORKFLOW}" \
+            --arg branch "${BRANCH}" \
+            --arg sha "${SHA}" \
+            --arg run_attempt "${RUN_ATTEMPT}" \
+            --arg state "${STATE}" \
+            --arg single_result "${SINGLE_RESULT}" \
+            --arg multi_result "${MULTI_RESULT}" \
+            --argjson duration "${DURATION}" \
+            --arg artifact_url "${SERVER_URL}/${REPO}/actions/runs/${RUN_ID}" \
+            --argjson failed_suites "${FAILED_JSON}" \
+            '{
+              "@timestamp": $ts,
+              labels: { ci_run_id: $run_id },
+              ci: { workflow: $workflow, branch: $branch, sha: $sha, run_attempt: $run_attempt },
+              state: $state,
+              single_result: $single_result,
+              multi_result: $multi_result,
+              failed_suites: $failed_suites,
+              duration_seconds: $duration,
+              artifact_url: $artifact_url
+            }')
+
+          echo "$DOC" | jq .
+
+          HTTP_CODE=$(curl -sS -o /tmp/es-summary-resp.json -w '%{http_code}' \
+            -X POST "http://${MULGA_ES_SINK}/mulga-ci-runs/_doc" \
+            -H 'Content-Type: application/json' \
+            -d "${DOC}")
+
+          if [ "$HTTP_CODE" != "200" ] && [ "$HTTP_CODE" != "201" ]; then
+            echo "::error::ES POST to ${MULGA_ES_SINK}/mulga-ci-runs failed (HTTP ${HTTP_CODE})"
+            cat /tmp/es-summary-resp.json
+            exit 1
+          fi
+          echo "CI run summary doc posted to ${MULGA_ES_SINK}/mulga-ci-runs (HTTP ${HTTP_CODE})"

.github/workflows/gpu-e2e.yml

@@ -0,0 +1,347 @@
+# GPU E2E — drives wattle's persistent, shared-host Spinifex stack over SSH to
+# exercise GPU passthrough (raw EC2, ECS task GPU exposure, EKS pod GPU
+# exposure). See docs/development/feature/wattle-deploy-and-ci.md (Part B).
+#
+# ---------------------------------------------------------------------------
+# ONE-TIME HOST-SIDE SETUP — NOT part of this repo, a human must do this:
+#
+#   1. Register ONE self-hosted GitHub Actions runner on a banksia host with
+#      the dedicated label `ci-gpu` (in addition to `self-hosted`). Do
+#      NOT add it to the `ci-single` / `ci-multi` pool and do NOT give any
+#      other runner the `ci-gpu` label — a second labelled runner would
+#      let two GPU jobs race the single RTX A1000. With exactly one runner
+#      carrying the label, `runs-on: [self-hosted, ci-gpu]` alone forces
+#      serialization; the `concurrency:` block below queues same-group runs
+#      when that runner is already busy with a live job.
+#   2. Drop the wattle SSH private key on that runner's filesystem and wire
+#      it into this repo as `secrets.WATTLE_SSH_KEY` (raw private key
+#      contents). Optionally override `vars.WATTLE_SSH_HOST` /
+#      `vars.WATTLE_SSH_USER` if they differ from the defaults below.
+#   3. Confirm the runner user can drive wattle's control plane over SSH. The
+#      job calls `spx` (read-only checks) and the `aws` CLI with the already-
+#      configured `spinifex` profile directly — no wrapper scripts, no install
+#      on wattle (tf-user is already in the `spinifex` group there):
+#        ssh tf-user@192.168.1.13 'AWS_PROFILE=spinifex aws ec2 describe-regions'
+#   4. Manual-work mutex: an agent doing hands-on GPU testing on wattle can
+#      block CI by touching a sentinel file in the RUNNER's home directory
+#      (NOT on wattle):
+#        touch ~/.ci-gpu-paused     # blocks CI
+#        rm ~/.ci-gpu-paused        # resumes CI
+#      or by setting the repo/environment variable `CI_GPU_PAUSED=true`
+#      (no runner filesystem access needed). Either one causes this workflow
+#      to skip cleanly (job succeeds, all work steps skipped) instead of
+#      queuing behind manual work indefinitely.
+#
+#   Runbook: docs/development/feature/wattle-deploy-and-ci.md (Part B).
+# ---------------------------------------------------------------------------
+
+name: GPU E2E
+
+on:
+  workflow_dispatch:
+    inputs:
+      test_run:
+        description: "Optional -test.run regex scoped to tests/e2e/gpu (empty = run all GPU tests)"
+        required: false
+        default: ""
+
+permissions:
+  contents: read
+
+# Single dedicated runner (see setup note 1 above) + this group together
+# enforce single-flight on the one RTX A1000: a second dispatch has nowhere
+# else to land (unique label) and, if it somehow overlaps in scheduling,
+# queues here rather than cancelling/stomping the in-flight run.
+concurrency:
+  group: ci-gpu
+  cancel-in-progress: false
+
+env:
+  # OTLP export to the central Elastic sink (wattle is the sink itself — see
+  # Part A.2). Repo variable overrides; carried for correlation even though
+  # the GPU suite's own daemon-side spans are emitted by wattle's already-
+  # configured obs-agent, not by this job.
+  SPINIFEX_OTLP_ENDPOINT: ${{ vars.SPINIFEX_OTLP_ENDPOINT || 'http://192.168.1.13:8200' }}
+  # ci.cell=gpu keeps this workflow's telemetry separable from the tofu-env
+  # workflows'. Nothing in this job bakes these attrs into a daemon (see the
+  # note above), so the label only takes effect once wattle's obs-agent is
+  # given them; there is deliberately no ci.job_id lookup for the same reason.
+  SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=gpu
+  WATTLE_SSH_HOST: ${{ vars.WATTLE_SSH_HOST || '192.168.1.13' }}
+  WATTLE_SSH_USER: ${{ vars.WATTLE_SSH_USER || 'tf-user' }}
+  TEST_RUN: ${{ inputs.test_run || '' }}
+
+jobs:
+  gpu_e2e:
+    name: GPU E2E (wattle)
+    runs-on: [self-hosted, ci-gpu]
+    timeout-minutes: 75
+    env:
+      ARTIFACT_DIR: /tmp/spinifex-gpu-e2e-artifacts-${{ github.run_id }}
+    steps:
+      # Manual-work mutex gate — see setup note 4 above. Checked BEFORE any
+      # SSH to wattle so a paused run touches the box zero times.
+      - name: Manual-work mutex gate
+        id: gate
+        env:
+          REPO_VAR_PAUSED: ${{ vars.CI_GPU_PAUSED }}
+        run: |
+          set -euo pipefail
+          PAUSED=false
+          if [ -f "$HOME/.ci-gpu-paused" ]; then
+            PAUSED=true
+            echo "::warning::ci-gpu runner paused via ~/.ci-gpu-paused sentinel on the runner — skipping this run (manual GPU work in progress on wattle). Remove the file to resume CI."
+          fi
+          if [ "$(echo "${REPO_VAR_PAUSED:-}" | tr '[:upper:]' '[:lower:]')" = "true" ]; then
+            PAUSED=true
+            echo "::warning::ci-gpu paused via repo/environment variable CI_GPU_PAUSED=true — skipping this run."
+          fi
+          echo "paused=${PAUSED}" >> "$GITHUB_OUTPUT"
+
+      - name: Paused — nothing to do
+        if: steps.gate.outputs.paused == 'true'
+        run: echo "ci-gpu is paused; job exits cleanly without touching wattle. See gate step above for the reason."
+
+      - name: Clean Workspace
+        if: steps.gate.outputs.paused != 'true'
+        run: sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/checkout@v7
+        if: steps.gate.outputs.paused != 'true'
+
+      - uses: actions/setup-go@v7
+        if: steps.gate.outputs.paused != 'true'
+        with:
+          go-version-file: go.mod
+          cache: false
+
+      - name: Install wattle SSH key
+        if: steps.gate.outputs.paused != 'true'
+        run: |
+          set -euo pipefail
+          install -m 0600 /dev/null "${RUNNER_TEMP}/wattle-ci"
+          echo "${{ secrets.WATTLE_SSH_KEY }}" > "${RUNNER_TEMP}/wattle-ci"
+          echo "ssh_opts=-o StrictHostKeyChecking=no -o ConnectTimeout=10 -i ${RUNNER_TEMP}/wattle-ci" >> "$GITHUB_ENV"
+
+      # Healthcheck: confirm wattle's persistent AWS-compat control plane is
+      # actually up before spending time building/shipping the test binary.
+      - name: Healthcheck — describe-regions
+        if: steps.gate.outputs.paused != 'true'
+        run: |
+          set -euo pipefail
+          ssh ${{ env.ssh_opts }} "${WATTLE_SSH_USER}@${WATTLE_SSH_HOST}" \
+            'AWS_PROFILE=spinifex aws ec2 describe-regions --output text'
+
+      # Ensure the base EC2 GPU AMI is present — the raw-EC2 GPU passthrough
+      # test is the always-on baseline, so CI self-provisions it rather than
+      # depending on a manual import. `--name` selects the catalog entry (pulls
+      # the prebuilt image from iso.mulgadc.com); `--ami-name` is REQUIRED to
+      # name the AMI ubuntu-26.04-nvidia-gpu-x86_64 — without it the import
+      # falls back to the generic ami-<distro>-<version>-<arch>, which the
+      # tests/e2e/gpu filter would never match. Idempotent — skipped once
+      # wattle's persistent loop-fs already has the named AMI (the 4.4G source
+      # qcow2 is cached in the images dir, so a re-import wouldn't re-download).
+      # `spx admin images import` writes local spinifex storage, so it needs
+      # the `spinifex` group active (sg); aws-CLI calls hit the gateway over
+      # HTTPS and don't.
+      - name: Ensure base GPU AMI present (import-if-absent)
+        if: steps.gate.outputs.paused != 'true'
+        run: |
+          set -euo pipefail
+          REMOTE='
+            set -e
+            BASE=ubuntu-26.04-nvidia-gpu-x86_64
+            present() {
+              AWS_PROFILE=spinifex aws ec2 describe-images \
+                --filters "Name=name,Values=$BASE" "Name=state,Values=available" \
+                --query "Images[0].ImageId" --output text 2>/dev/null
+            }
+            ID=$(present)
+            if [ -n "$ID" ] && [ "$ID" != "None" ]; then
+              echo "base GPU AMI already present: $BASE = $ID"; exit 0
+            fi
+            echo "base GPU AMI $BASE absent — importing prebuilt from catalog..."
+            sg spinifex -c "AWS_PROFILE=spinifex spx admin images import --name $BASE --ami-name $BASE"
+            for i in 1 2 3 4 5 6; do
+              ID=$(present)
+              [ -n "$ID" ] && [ "$ID" != "None" ] && break
+              sleep 5
+            done
+            [ -n "$ID" ] && [ "$ID" != "None" ] || { echo "import ran but $BASE still not available" >&2; exit 1; }
+            echo "imported base GPU AMI: $BASE = $ID"
+          '
+          ssh ${{ env.ssh_opts }} "${WATTLE_SSH_USER}@${WATTLE_SSH_HOST}" "$REMOTE"
+
+      # Preflight — the base GPU AMI is REQUIRED: the setup step above just
+      # ensured it, so a miss here means the import silently failed. Hard-fail
+      # rather than run a false-green all-skipped suite. ECS/EKS node AMIs are
+      # opt-in tiers (heavier — node AMIs + cluster spin-up); their tests
+      # self-skip when absent, so only warn.
+      - name: Preflight — verify GPU AMIs
+        if: steps.gate.outputs.paused != 'true'
+        run: |
+          set -euo pipefail
+          REMOTE='
+            set -e
+            present() {
+              AWS_PROFILE=spinifex aws ec2 describe-images \
+                --filters "Name=name,Values=$1" "Name=state,Values=available" \
+                --query "Images[0].ImageId" --output text 2>/dev/null
+            }
+            BASE=ubuntu-26.04-nvidia-gpu-x86_64
+            ID=$(present "$BASE")
+            if [ -z "$ID" ] || [ "$ID" = "None" ]; then
+              echo "required base GPU AMI $BASE missing — cannot run the baseline EC2 GPU test" >&2
+              exit 1
+            fi
+            echo "required: $BASE = $ID"
+            for OPT in spinifex-ecs-node-gpu spinifex-eks-node spinifex-eks-node-gpu; do
+              OID=$(present "$OPT")
+              if [ -z "$OID" ] || [ "$OID" = "None" ]; then
+                echo "::warning::$OPT not imported — its GPU test(s) will skip (opt-in tier)"
+              else
+                echo "opt-in: $OPT = $OID"
+              fi
+            done
+          '
+          ssh ${{ env.ssh_opts }} "${WATTLE_SSH_USER}@${WATTLE_SSH_HOST}" "$REMOTE"
+
+      # Same build command CI's other e2e workflows use to cross-compile a
+      # suite into a shippable binary (tests/e2e/Makefile e2e-build target,
+      # SUITES_BUILD scoped to just gpu). Building on the runner keeps go
+      # toolchain / module cache off wattle entirely.
+      - name: Build GPU e2e test binary on runner
+        if: steps.gate.outputs.paused != 'true'
+        working-directory: tests/e2e
+        run: |
+          set -euo pipefail
+          make e2e-build SUITES_BUILD=gpu
+          ls -la _bin/gpu.test
+
+      - name: Install go-junit-report on runner
+        if: steps.gate.outputs.paused != 'true'
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Ship GPU e2e test binary to wattle
+        if: steps.gate.outputs.paused != 'true'
+        run: |
+          set -euo pipefail
+          VM="${WATTLE_SSH_USER}@${WATTLE_SSH_HOST}"
+          ssh ${{ env.ssh_opts }} "$VM" "mkdir -p \$HOME/spinifex-gpu-e2e-bin"
+          scp ${{ env.ssh_opts }} tests/e2e/_bin/gpu.test "$VM":spinifex-gpu-e2e-bin/gpu.test
+
+      # Same invocation shape e2e.yml uses to run a pre-built suite binary
+      # (SPINIFEX_E2E=1 SPINIFEX_MODE=single, ./<suite>.test -test.v
+      # -test.timeout=NNm), plus SPINIFEX_WAN_IP: wattle's ~24 bridges make
+      # net.InterfaceAddrs() enumeration pick a co-tenant bridge instead of
+      # the advertised gateway (mulga-7suds), so the override is required
+      # here (proper fix tracked separately: tests/e2e/harness/env.go should
+      # prefer spinifex.toml's advertised bridge).
+      - name: Run GPU e2e suite on wattle
+        id: e2e
+        if: steps.gate.outputs.paused != 'true'
+        run: |
+          set +e
+          VM="${WATTLE_SSH_USER}@${WATTLE_SSH_HOST}"
+          if [ -n "${TEST_RUN}" ]; then
+            TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+          else
+            TEST_RUN_FLAG=""
+          fi
+          ssh -tt ${{ env.ssh_opts }} "$VM" \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/spinifex-gpu-e2e-bin; \
+             GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+               SPINIFEX_WAN_IP='${WATTLE_SSH_HOST}' \
+               ARTIFACT_DIR='${ARTIFACT_DIR}' \
+               ./gpu.test -test.v -test.timeout=60m ${TEST_RUN_FLAG} \
+               2>&1 | tee '${ARTIFACT_DIR}/test-gpu.log'; \
+             exit \${PIPESTATUS[0]}"
+          E2E_RC=$?
+          echo "rc=${E2E_RC}" >> "$GITHUB_OUTPUT"
+          exit 0
+
+      - name: Collect artifacts from wattle
+        if: always() && steps.gate.outputs.paused != 'true'
+        run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
+          VM="${WATTLE_SSH_USER}@${WATTLE_SSH_HOST}"
+          ssh ${{ env.ssh_opts }} "$VM" "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+          echo "collected: $(ls "${ARTIFACT_DIR}" 2>/dev/null | wc -l) entries"
+
+      - name: Convert test log to JUnit XML
+        if: always() && steps.gate.outputs.paused != 'true'
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log)
+            SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Publish test summary
+        if: always() && steps.gate.outputs.paused != 'true'
+        uses: ./.github/actions/junit-report
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          title: "GPU E2E results"
+
+      - name: Analyze e2e failures
+        if: always() && steps.gate.outputs.paused != 'true'
+        uses: ./.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "GPU E2E failure analysis"
+
+      - uses: actions/upload-artifact@v7
+        if: always() && steps.gate.outputs.paused != 'true'
+        with:
+          name: gpu-e2e-artifacts-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
+      # Cleanup is NEVER a host teardown: it never runs setup-ovn.sh,
+      # dev-reset, or cluster-reset. It only reconciles EC2-API-level state —
+      # terminate any GPU-typed instance still pending/running (the Go
+      # suite's own t.Cleanup already does this on normal pass/fail exit;
+      # this is the backstop for a hard `-test.timeout` kill or a workflow
+      # cancellation, which skip deferred Go cleanups). Termination triggers
+      # gpuManager.Release() on wattle, returning the A1000 to the pool for
+      # the next queued run or manual test.
+      - name: Cleanup — terminate leftover GPU instances, release GPU
+        if: always() && steps.gate.outputs.paused != 'true'
+        run: |
+          set +e
+          VM="${WATTLE_SSH_USER}@${WATTLE_SSH_HOST}"
+          REMOTE='
+            set +e
+            GPU_TYPES=$(AWS_PROFILE=spinifex aws ec2 describe-instance-types \
+              --query "InstanceTypes[?GpuInfo!=null].InstanceType" --output text | tr "\t" ",")
+            if [ -z "$GPU_TYPES" ]; then
+              echo "no GPU instance types advertised; nothing to reconcile"
+              exit 0
+            fi
+            IDS=$(AWS_PROFILE=spinifex aws ec2 describe-instances \
+              --filters "Name=instance-type,Values=${GPU_TYPES}" "Name=instance-state-name,Values=pending,running" \
+              --query "Reservations[].Instances[].InstanceId" --output text)
+            if [ -z "$IDS" ] || [ "$IDS" = "None" ]; then
+              echo "no leftover GPU instances — GPU already released"
+              exit 0
+            fi
+            echo "terminating leftover GPU instance(s): $IDS"
+            AWS_PROFILE=spinifex aws ec2 terminate-instances --instance-ids $IDS
+          '
+          ssh ${{ env.ssh_opts }} "$VM" "$REMOTE"
+
+      - name: Fail job if GPU suite failed
+        if: always() && steps.gate.outputs.paused != 'true'
+        run: |
+          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
+            echo "::error::GPU e2e suite failed (rc=${{ steps.e2e.outputs.rc }}) — see analysis above; raw ES/APM traces at ${SPINIFEX_OTLP_ENDPOINT} (ci.run_id=${{ github.run_id }})"
+            exit 1
+          fi

.github/workflows/publish-eks-ami.yml

@@ -1,16 +1,34 @@
-name: Publish EKS node AMI
+name: Publish EKS + ECS node AMIs
 
-# Builds the eks-node system AMI (libguestfs, on the runner) and publishes the
+# Builds the system AMIs (libguestfs, on the runner) and publishes each
 # compressed qcow2 + .sha256 sidecar to the Cloudflare R2 bucket that backs
-# iso.mulgadc.com/system-ami/. Operators then install it on any cluster with:
+# iso.mulgadc.com/system-ami/. Four images ship:
+#   spinifex-eks-node       Alpine EKS K3s node (server|agent, role at boot)
+#   spinifex-ecs-node       Alpine ECS container instance
+#   spinifex-eks-node-gpu   Ubuntu EKS K3s GPU worker (NVIDIA driver + toolkit)
+#   spinifex-ecs-node-gpu   Ubuntu ECS GPU container instance
+# Operators then install them on any cluster with:
 #   spx admin images import --name spinifex-eks-node
+#   spx admin images import --name spinifex-ecs-node
+#   spx admin images import --name spinifex-eks-node-gpu
+#   spx admin images import --name spinifex-ecs-node-gpu
 #
 # Manual-only: publishing overwrites the mirror artifact, so it is a deliberate
-# action, not something every push should trigger.
+# action, not something every push should trigger. The `variant` input scopes
+# which images build — the Ubuntu GPU legs are 16G and much slower (NVIDIA
+# driver download + DKMS build), so publishing only the CPU AMIs stays cheap.
 
 on:
   workflow_dispatch:
     inputs:
+      variant:
+        description: "Which images to build"
+        type: choice
+        default: all
+        options:
+          - all
+          - cpu
+          - gpu
       dry_run:
         description: "Build + convert only; skip the R2 upload"
         type: boolean
@@ -24,9 +42,37 @@ concurrency:
   cancel-in-progress: false
 
 jobs:
-  publish:
-    name: Build + publish eks-node AMI
+  matrix:
+    name: Resolve image matrix
     runs-on: [ self-hosted, ci-single ]
+    outputs:
+      images: ${{ steps.gen.outputs.images }}
+    steps:
+      - name: Select images for variant
+        id: gen
+        env:
+          VARIANT: ${{ inputs.variant }}
+        run: |
+          set -euo pipefail
+          cpu='{"name":"spinifex-eks-node","manifest":"scripts/images/eks-node/manifest.conf"},{"name":"spinifex-ecs-node","manifest":"scripts/images/ecs-agent/manifest.conf"}'
+          gpu='{"name":"spinifex-eks-node-gpu","manifest":"scripts/images/eks-node-gpu/manifest.conf"},{"name":"spinifex-ecs-node-gpu","manifest":"scripts/images/ecs-agent-gpu/manifest.conf"}'
+          case "${VARIANT}" in
+            cpu) items="${cpu}" ;;
+            gpu) items="${gpu}" ;;
+            *)   items="${cpu},${gpu}" ;;
+          esac
+          echo "images={\"include\":[${items}]}" >> "$GITHUB_OUTPUT"
+
+  publish:
+    name: Build + publish ${{ matrix.name }}
+    needs: matrix
+    runs-on: [ self-hosted, ci-single ]
+    strategy:
+      fail-fast: false
+      # ci-single is one runner and each GPU build needs ~16G of scratch disk;
+      # serialise the legs so they never contend for /_temp or the mirror.
+      max-parallel: 1
+      matrix: ${{ fromJSON(needs.matrix.outputs.images) }}
 
     steps:
       - name: Checkout Spinifex
@@ -49,7 +95,7 @@ jobs:
           path: predastore
 
       - name: Setup Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           go-version-file: "spinifex/go.mod"
           # Self-hosted runner: the Go module + build cache already persists on
@@ -59,7 +105,13 @@ jobs:
           cache: false
 
       - name: Setup Go Workspace
-        run: go work init ./spinifex ./viperblock ./predastore
+        # Idempotent: the ci-single runner reuses one workspace across matrix
+        # legs and runs, and checkout never scrubs the root go.work. Remove any
+        # stale file so each leg re-inits cleanly instead of failing on
+        # "go.work already exists".
+        run: |
+          rm -f go.work go.work.sum
+          go work init ./spinifex ./viperblock ./predastore
 
       - name: Install build + upload tooling
         run: |
@@ -73,7 +125,7 @@ jobs:
             rm -rf /tmp/awscliv2.zip /tmp/aws-install
           fi
 
-      - name: Build + publish to R2
+      - name: Build + publish ${{ matrix.name }} to R2
         working-directory: spinifex
         env:
           # Map the repo's R2 secrets onto the names publish-system-image.sh reads.
@@ -82,15 +134,20 @@ jobs:
           R2_ENDPOINT: ${{ secrets.CF_R2_ENDPOINT_URL }}
           R2_BUCKET: spinifex-iso
           DRY_RUN: ${{ inputs.dry_run }}
+          MANIFEST: ${{ matrix.manifest }}
+          # Ubuntu GPU images are 16G — a RAM-backed /tmp cannot hold the qcow2 +
+          # expanded raw, so point the build scratch at the runner's on-disk
+          # _temp. Harmless for the small Alpine legs too.
+          SYSTEM_IMAGE_BUILD_DIR: ${{ runner.temp }}/system-image-build
         run: |
           set -euo pipefail
           # libguestfs needs to read a host kernel; Ubuntu ships it mode 0600.
           # direct backend = no libvirt on the runner.
           export LIBGUESTFS_BACKEND=direct
           sudo chmod 0644 /boot/vmlinuz-* || true
+          mkdir -p "${SYSTEM_IMAGE_BUILD_DIR}"
           FLAGS=(--build)
           if [[ "${DRY_RUN}" == "true" ]]; then
             FLAGS+=(--dry-run)
           fi
-          ./scripts/publish-system-image.sh \
-            scripts/images/eks-node/manifest.conf "${FLAGS[@]}"
+          ./scripts/publish-system-image.sh "${MANIFEST}" "${FLAGS[@]}"

.github/workflows/release.yml

@@ -32,9 +32,9 @@ jobs:
         uses: docker/setup-buildx-action@v4
 
       - name: Set up Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
-          go-version: "1.26.4"
+          go-version: "1.26.5"
 
       - name: Build microVM artifacts (lb-agent + kernel + initramfs)
         run: cd spinifex && make build-microvm-image
@@ -280,7 +280,7 @@ jobs:
         run: sudo make -C mulga/scripts/iso-builder clean
 
       - name: Set up Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false

.github/workflows/spinifex.yml

@@ -32,22 +32,29 @@ jobs:
           path: spinifex
           fetch-depth: 2
 
+      - name: Resolve sibling refs
+        id: siblings
+        env:
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+        run: bash .github/resolve-sibling-refs.sh
+        working-directory: spinifex
+
       - name: Checkout Viperblock
         uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
-          ref: dev
+          ref: ${{ steps.siblings.outputs.viperblock }}
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
-          ref: dev
+          ref: ${{ steps.siblings.outputs.predastore }}
           path: predastore
 
       - name: Setup Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           cache-dependency-path: "spinifex/go.sum"
           go-version-file: "spinifex/go.mod"
@@ -70,6 +77,55 @@ jobs:
       - name: Check Diff Coverage
         run: cd spinifex && scripts/diff-coverage.sh coverage.out --base HEAD~1
 
+  # In-process integration tier (gateway.SetupRoutes over embedded NATS
+  # JetStream). Its own job so it runs in parallel with the unit tiers rather
+  # than adding to their wall time, and it still blocks a PR — unlike e2e.yml's
+  # "Integration (front gate)" job, which is push-triggered against main/dev on
+  # a self-hosted runner and so never gates a PR before merge.
+  integration_tests:
+    name: Integration Tests
+    runs-on: ubuntu-26.04
+
+    steps:
+      - name: Checkout Spinifex
+        uses: actions/checkout@v7
+        with:
+          path: spinifex
+          fetch-depth: 2
+
+      - name: Resolve sibling refs
+        id: siblings
+        env:
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+        run: bash .github/resolve-sibling-refs.sh
+        working-directory: spinifex
+
+      - name: Checkout Viperblock
+        uses: actions/checkout@v7
+        with:
+          repository: mulgadc/viperblock
+          ref: ${{ steps.siblings.outputs.viperblock }}
+          path: viperblock
+
+      - name: Checkout Predastore
+        uses: actions/checkout@v7
+        with:
+          repository: mulgadc/predastore
+          ref: ${{ steps.siblings.outputs.predastore }}
+          path: predastore
+
+      - name: Setup Go
+        uses: actions/setup-go@v7
+        with:
+          cache-dependency-path: "spinifex/go.sum"
+          go-version-file: "spinifex/go.mod"
+
+      - name: Setup Go Workspace
+        run: go work init ./spinifex ./viperblock ./predastore
+
+      - name: Run Integration Tests
+        run: make -C spinifex test-integration
+
   race_detection:
     name: Race Detection
     runs-on: ubuntu-26.04
@@ -80,22 +136,29 @@ jobs:
         with:
           path: spinifex
 
+      - name: Resolve sibling refs
+        id: siblings
+        env:
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+        run: bash .github/resolve-sibling-refs.sh
+        working-directory: spinifex
+
       - name: Checkout Viperblock
         uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
-          ref: dev
+          ref: ${{ steps.siblings.outputs.viperblock }}
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
-          ref: dev
+          ref: ${{ steps.siblings.outputs.predastore }}
           path: predastore
 
       - name: Setup Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           cache-dependency-path: "spinifex/go.sum"
           go-version-file: "spinifex/go.mod"
@@ -116,22 +179,29 @@ jobs:
         with:
           path: spinifex
 
+      - name: Resolve sibling refs
+        id: siblings
+        env:
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+        run: bash .github/resolve-sibling-refs.sh
+        working-directory: spinifex
+
       - name: Checkout Viperblock
         uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
-          ref: dev
+          ref: ${{ steps.siblings.outputs.viperblock }}
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
-          ref: dev
+          ref: ${{ steps.siblings.outputs.predastore }}
           path: predastore
 
       - name: Setup Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           cache-dependency-path: "spinifex/go.sum"
           go-version-file: "spinifex/go.mod"

.github/workflows/ui.yml

@@ -40,7 +40,7 @@ jobs:
           package_json_file: spinifex/services/spinifexui/frontend/package.json
 
       - name: Setup Node.js
-        uses: actions/setup-node@v6
+        uses: actions/setup-node@v7
         with:
           cache: "pnpm"
           cache-dependency-path: spinifex/services/spinifexui/frontend/pnpm-lock.yaml

.golangci.yml

@@ -1,5 +1,8 @@
 version: "2"
 
+issues:
+  max-same-issues: 10
+
 formatters:
   enable:
     - gofmt
@@ -14,24 +17,29 @@ linters:
     - bidichk # checks for dangerous unicode bidi character sequences
     - bodyclose # checks whether HTTP response body is closed successfully
     - canonicalheader # checks whether net/http.Header uses canonical header keys
-    - decorder # checks declaration order of types, consts, vars, and funcs
+    - depguard # blocks imports of packages with a preferred replacement
     - durationcheck # checks for two durations multiplied together
     - embeddedstructfieldcheck # checks embedded sync.Mutex/RWMutex in structs
     - errcheck # checks for unchecked errors in type assertions and I/O
     - errchkjson # checks types passed to json encoding functions, flags omittable error checks
     - errname # checks that sentinel errors are prefixed with Err, error types suffixed with Error
+    - errorlint # catches ==/!= on errors, type assertions on errors, and %v instead of %w — all break on wrapped errors
     - exptostd # detects golang.org/x/exp/ functions replaceable by stdlib
     - fatcontext # detects nested contexts in loops
-    - forcetypeassert # flags unchecked type assertions that can panic
+    - forbidigo # blocks the legacy JetStream/KV symbols
     - gocheckcompilerdirectives # validates go compiler directive comments (//go:)
+    - godot # checks that declaration comments end in a period
     - goprintffuncname # checks that printf-like functions are named with f at the end
     - gosec # inspects source code for security problems
     - govet # reports suspicious constructs (Printf alignment, shadow, etc.)
+    - iface # detects interface pollution — duplicate interface declarations
     - ineffassign # detects assignments to existing variables that are not used
+    - iotamixing # flags const blocks that mix iota with non-iota declarations
     - makezero # finds slice declarations with non-zero initial length followed by append
     - mirror # catches wrong mirror patterns in bytes/strings usage
-    - musttag # checks structs passed to marshal/unmarshal have proper tags
+    - misspell # finds commonly misspelled English words
     - modernize # modernizes code to use newer Go idioms (go fix analyzers)
+    - musttag # checks structs passed to marshal/unmarshal have proper tags
     - nakedret # finds naked returns in functions
     - nilerr # catches code that returns nil even after checking err != nil
     - nilnesserr # catches returning a different nil error variable after err != nil check
@@ -43,6 +51,8 @@ linters:
     - recvcheck # checks for receiver type consistency (pointer vs value)
     - sloglint # enforces consistent log/slog usage (key-value pairs, no globals)
     - staticcheck # extensive static analysis checks
+    - testifylint # checks common testify mistakes (assert.Len over assert.Equal on len, etc.)
+    - tparallel # detects inappropriate usage of t.Parallel()
     - unconvert # removes unnecessary type conversions
     - usestdlibvars # detects hardcoded values that should use stdlib constants
     - usetesting # catches os.TempDir() etc. in tests — should use t.TempDir()
@@ -50,14 +60,41 @@ linters:
     - whitespace # detects leading and trailing whitespace in functions
 
   settings:
+    depguard:
+      rules:
+        main:
+          deny:
+            - pkg: log$
+              desc: Use log/slog instead, see https://go.dev/blog/slog
+            - pkg: math/rand$
+              desc: Use math/rand/v2 instead, see https://go.dev/blog/randv2
+
     embeddedstructfieldcheck:
       forbid-mutex: true
 
     errcheck:
-      check-type-assertions: true
+      check-type-assertions: true # replaces forcetypeassert
       exclude-functions:
         - (*github.com/nats-io/nats.go.Subscription).Unsubscribe # cleanup-only, errors not actionable
 
+    forbidigo:
+      analyze-types: true
+      forbid:
+        - pattern: '\.JetStream$'
+          pkg: '^github\.com/nats-io/nats\.go$'
+          msg: legacy nc.JetStream() is superseded; construct with jetstream.New(nc).
+        - pattern: '^nats\.(JetStreamContext|KeyValue.*|KeyWatcher|Err(KeyNotFound|KeyExists|BucketNotFound|NoKeysFound|StreamNotFound))$'
+          pkg: '^github\.com/nats-io/nats\.go$'
+          msg: legacy nats KV types/sentinels are superseded; use github.com/nats-io/nats.go/jetstream.
+
+    gosec:
+      excludes:
+        - G104 # unchecked errors — errcheck is stricter
+        - G116 # bidi chars — bidichk is more thorough
+        - G117 # use of net/http serve functions (we handle TLS separately)
+        - G703 # path traversal taint analysis — false positives on config file loading
+        - G705 # XSS taint analysis — false positives on internal error responses
+
     govet:
       enable-all: true
       disable:
@@ -65,6 +102,28 @@ linters:
         - shadow # too noisy to retrofit — all 50 findings are err re-declarations
         - unusedwrite # false positives on struct field assignments before marshal/send
 
+    modernize:
+      disable:
+        - newexpr # don't replace aws.String() with new() — AWS SDK idiom
+        - stringsbuilder # don't replace small string concatenations with strings.Builder
+
+    nakedret:
+      max-func-lines: 0
+
+    nolintlint:
+      require-explanation: true
+      require-specific: true
+
+    perfsprint:
+      concat-loop: false
+      err-error: false # don't flag err.Error()
+      errorf: false # don't convert fmt.Errorf to errors.New
+      strconcat: false
+
+    reassign:
+      patterns:
+        - ".*"
+
     staticcheck:
       checks:
         - all
@@ -76,40 +135,14 @@ linters:
         - -ST1020 # godoc on exported types — not enforcing yet
         - -ST1021 # godoc on exported type groups — not enforcing yet
         - -ST1022 # godoc on exported vars/consts — not enforcing yet
-        - -SA1019 # deprecated usage (handle separately)
         - -SA4006 # value overwritten — ineffassign + wastedassign cover this better
-        - -SA9005 # ineffective break (staticcheck false positives)
+        - -SA9005 # marshaling structs with no exported fields — intended {} for empty API-response types
 
-    gosec:
-      excludes:
-        - G104 # unchecked errors — errcheck is stricter
-        - G116 # bidi chars — bidichk is more thorough
-        - G117 # use of net/http serve functions (we handle TLS separately)
-        - G703 # path traversal taint analysis — false positives on config file loading
-        - G705 # XSS taint analysis — false positives on internal error responses
-        - G706 # log injection taint analysis — false positives on structured slog usage
-
-    modernize:
+    testifylint:
       disable:
-        - newexpr # don't replace aws.String() with new() — AWS SDK idiom
-        - stringsbuilder # don't replace small string concatenations with strings.Builder
-
-    perfsprint:
-      concat-loop: false
-      err-error: false # don't flag err.Error()
-      errorf: false # don't convert fmt.Errorf to errors.New
-      strconcat: false
-
-    nakedret:
-      max-func-lines: 0
-
-    nolintlint:
-      require-explanation: true
-      require-specific: true
-
-    reassign:
-      patterns:
-        - ".*"
+        - encoded-compare # misidentifies non-JSON strings when identifiers contain "JSON"
+        - float-compare # false positives
+        - require-error # assert-on-error is deliberate where a test continues to collect further failures
 
   exclusions:
     presets:
@@ -133,3 +166,8 @@ linters:
         linters: [gosec] # subprocess execution — legitimate in VM orchestration
       - text: "G304"
         linters: [gosec] # file path from variable — necessary for config loading
+
+      # aws-sdk-go v1 is deprecated wholesale (EOL 2025-07-31); the v1->v2
+      # migration is tracked separately, so suppress its SA1019 noise for now.
+      - text: "aws-sdk-go is deprecated"
+        linters: [staticcheck]

Makefile

@@ -120,8 +120,13 @@ install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
 
-# Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
-# Use this before committing to catch CI failures locally.
+# Preflight — the pre-commit gate: manifest checks, lint, vuln, and the unit,
+# race and e2e-harness tiers.
+#
+# The integration tier is deliberately NOT here. CI runs it as its own parallel
+# job, so it is still gated before merge, and keeping it out leaves preflight
+# the fast common-path check — run `make test-integration` directly when a
+# change touches the gateway router or the NATS subjects it drives.
 preflight:
 	@$(MAKE) --no-print-directory QUIET=1 manifest-check manifest-lint lint govulncheck test-cover diff-coverage test-race test-harness
 	@echo -e "\n ✅ Preflight passed — safe to commit."
@@ -133,6 +138,18 @@ test-harness:
 	@echo -e "\n....Running e2e harness unit tests...."
 	$(_Q)LOG_IGNORE=1 go test -tags=e2e -timeout 60s ./tests/e2e/harness/... $(_RACEQ)
 
+# In-process integration tier: the real gateway router against embedded NATS
+# JetStream, with only the daemon-side NATS subjects stubbed. Build-tagged
+# `integration` so it's skipped by the default `go test ./spinifex/...` and by
+# `test-cover`/`test-race`. Nothing provisioned — no tofu, no docker, no
+# Spinifex daemons, so the whole package runs in well under a minute. It gets
+# its own PR-blocking CI job rather than the self-hosted, push-triggered live
+# e2e tiers, so a regression here is caught before it can be merged, not just
+# after.
+test-integration:
+	@echo -e "\n....Running in-process integration tests...."
+	$(_Q)LOG_IGNORE=1 go test -tags=integration -timeout 60s ./tests/integration/... $(_RACEQ)
+
 # Validate docs/service-interfaces.yaml. Schema check + cross-reference
 # of services/suites/fixtures + on-disk path existence. Subject content
 # vs source is enforced separately in Bead 5 drift lint.
@@ -218,9 +235,9 @@ install-system:
 		ovn-central ovn-host openvswitch-switch dhcpcd-base
 
 install-go:
-	@echo -e "\n....Installing Go 1.26.4 for $(ARCH) ($(GO_ARCH))...."
+	@echo -e "\n....Installing Go 1.26.5 for $(ARCH) ($(GO_ARCH))...."
 	@if [ ! -d "/usr/local/go" ]; then \
-		curl -L https://go.dev/dl/go1.26.4.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
+		curl -L https://go.dev/dl/go1.26.5.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
 	else \
 		echo "Go already installed in /usr/local/go"; \
 	fi
@@ -254,6 +271,13 @@ govulncheck:
 	$(_Q)go tool govulncheck ./...
 	@echo "  govulncheck ok"
 
+# NilAway — advisory nil-panic analysis. Not in preflight: it has a known
+# false-positive rate, so findings are triaged by hand rather than gating commits.
+nilaway:
+	@echo "Running nilaway..."
+	$(_Q)go tool nilaway -include-pkgs=github.com/mulgadc/spinifex -exclude-test-files ./...
+	@echo "  nilaway ok"
+
 # Build release tarballs — use distro-ARCH for single arch, distro for both
 distro: distro-amd64 distro-arm64
 	@echo ""
@@ -297,8 +321,8 @@ distro-arm64:
 distro-clean:
 	rm -rf dist/
 
-.PHONY: build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-harness manifest-check manifest-lint manifest-lint-update \
+.PHONY: build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-harness test-integration manifest-check manifest-lint manifest-lint-update \
 	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
-	lint fix govulncheck \
+	lint fix govulncheck nilaway \
 	distro distro-amd64 distro-arm64 distro-clean

README.md

@@ -1,5 +1,5 @@
 <p align="center">
-  <img src=".github/assets/banner.svg" alt="Spinifex — AWS-Compatible Infrastructure Stack" width="900">
+  <img src=".github/assets/banner.svg" alt="Spinifex: the open, AWS-compatible cloud you run yourself. EC2, EBS, S3, VPC, and IAM on your infrastructure, at the edge, or on a Neocloud." width="900">
 </p>
 
 <p align="center">
@@ -11,6 +11,8 @@
 <p align="center">
   <a href="#what-is-spinifex">What is Spinifex?</a> ·
   <a href="#the-platform">Platform</a> ·
+  <a href="#three-ways-to-deploy">Deploy</a> ·
+  <a href="#aws-compatibility">AWS services</a> ·
   <a href="#core-components">Components</a> ·
   <a href="#architecture-at-a-glance">Architecture</a> ·
   <a href="#installation">Installation</a> ·
@@ -19,31 +21,90 @@
 
 ---
 
-# Spinifex: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
+# Spinifex: the open, AWS-compatible cloud you run yourself
 
-Spinifex developed by [Mulga Defense Corporation](https://mulgadc.com) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
+Spinifex, built by [Mulga](https://mulgadc.com), recreates the AWS services your
+software already uses (EC2, EBS, S3, VPC, IAM) on infrastructure you control.
+Move workloads off the hyperscalers, run AI at a fraction of the cost, or take
+the cloud to the edge, without rewriting a line. The exact build you ship to AWS,
+Spinifex serves on your own hardware: same APIs, same Terraform, same SDKs.
 
 ## What is Spinifex?
 
-Spinifex replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
+Spinifex is an open-source infrastructure platform that recreates the AWS service
+surface (EC2, EBS, S3, VPC, IAM) on bare-metal, edge, and on-premises
+environments. Run cloud-native software without a hyperscaler.
 
-It's designed for developers and operators who need:
+Most AWS alternatives wrap an API and call it a cloud. We rebuilt the engineering
+underneath: distributed object storage with erasure coding, block storage built
+to survive failure, and compute on bare metal. That depth is the reason your
+software runs unchanged, instead of rewritten.
 
-- Cloud-like infrastructure without the cloud
-- Minimal dependencies, full control
-- Drop-in compatibility with tools like the AWS CLI, SDKs, and Terraform
-- A secure cloud environment you control and own the entire hardware, network and software stack
+It's built for teams that need:
 
-You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren't feasible.
+- The AWS tooling they already use (AWS CLI, SDKs, Terraform), with nothing to rewrite
+- Full control of the stack: your hardware, your network, your data, your keys
+- A cloud that keeps running offline, through disconnection, with no external control plane
+- An open core (AGPL-3.0), auditable and yours to keep, with no lock-in
+
+You change *where* your software runs, not *what* it is.
 
 ## The Platform
 
 <p align="center">
-  <img src=".github/assets/platform.svg" alt="The Mulga Platform — Hardware to Applications" width="900">
+  <img src=".github/assets/platform.svg" alt="The Spinifex stack: your apps and AWS tooling on top, the Spinifex AWS-compatible service layer, running on standard Linux on CPU and GPU hardware, deployable to Neocloud, on-premise, or the edge." width="900">
 </p>
 
 From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
 
+## Three ways to deploy
+
+One AWS-compatible surface, three deployment shapes. Pick the one that matches
+your reality; the platform on top is the same.
+
+### Neocloud
+
+Lift and shift onto a partner Neocloud. Move workloads off the hyperscalers and
+onto our Neocloud partner ecosystem without rewriting them. GPU capacity
+(H100 / H200 / B200), cheaper, available now. Change an endpoint, keep the
+software.
+
+### On-premise
+
+Bring your own hardware and host Spinifex in your own data centre. A real
+multi-node HA cluster integrated with your storage and networking, with full
+control of stack, data, and jurisdiction. A predictable bill, no egress
+surprises, an auditable open-source core.
+
+### Edge
+
+Cloud where the cloud can't reach. Air-gapped sites, vehicles, vessels,
+factories, and clinics. Compute next to the data, running through disconnection,
+on hardware you own. The same AWS APIs your software already uses.
+
+## AWS compatibility
+
+Speak the AWS API surface, natively. The AWS SDKs, AWS CLI, and Terraform:
+everything you deploy on AWS deploys on Spinifex unchanged. At the edge,
+on-premise, or on a partner Neocloud.
+
+| Service | What it is | Status |
+| --- | --- | --- |
+| **EC2** | Compute | Available |
+| **EBS** | Block storage | Available |
+| **S3** | Object storage | Available |
+| **VPC** | Networking | Available |
+| **IAM** | Identity & auth | Available |
+| **ALB / NLB** | Load balancers | Available |
+| **EKS** | Kubernetes | Available |
+| **ECR** | Container registry | Available |
+| **ECS** | Container service | Available |
+| **RDS** | Databases | Q3 2026 |
+
+Roadmap items (Q3 2026) ship under the same AWS API surface. Code written for AWS
+today keeps working the moment they land.
+[Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).
+
 ## Core Components
 
 ### Spinifex (Compute Service – EC2 Alternative)
@@ -75,19 +136,23 @@ Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs
 ## Architecture at a Glance
 
 <p align="center">
-  <img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture — AWS Gateway, NATS bus, service daemons" width="900">
+  <img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus answered by stateless ec2/ebs/s3/vpc/iam daemons over local backends." width="900">
 </p>
 
-Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless — scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
+Every AWS API call is authenticated at the gateway, published to a NATS subject,
+and answered by whichever daemon claims it. Daemons are stateless: scale
+horizontally by starting more. No etcd, no Kubernetes, no external control plane.
+Just systemd units, a NATS cluster, and your hardware. Deep dive:
+**[`docs/DESIGN.md`](docs/DESIGN.md)**.
 
 ## Key Features
 
-AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, or SDKs you already use.
-
-- Designed for Control – Run on your own terms, whether that's in a datacenter, remote site, or sensitive environment.
-- Minimal Dependencies – Each service is standalone and avoids complex orchestration layers.
-- Works Offline – No reliance on centralized cloud services or external networks.
-- Open Source – Licensed under AGPL 3.0. Fork it, extend it, or deploy it as-is.
+- **AWS-compatible APIs.** Use the AWS CLI, SDKs, and Terraform you already know. Repoint your endpoint and ship; nothing to rewrite.
+- **Zero cloud dependency.** Runs entirely on your hardware. No phone-home, no control plane, no external authority. Works fully offline.
+- **Bare-metal compute.** QEMU-based with direct hardware access. No hypervisor tax, no abstraction overhead.
+- **Built-in storage.** Block and object storage included: NVMe caching, Reed-Solomon erasure coding, and replication out of the box.
+- **Edge-first architecture.** Designed for disconnected, contested, and resource-constrained environments from day one.
+- **Open core, no lock-in.** AGPL-3.0 with a commercial option. Inspect it, modify it, deploy it. The platform is yours to keep.
 
 ## Installation
 
@@ -192,6 +257,10 @@ While Spinifex is architected and implemented by experienced engineers, we lever
 
 This hybrid approach ensures Spinifex benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
 
+## Trademarks
+
+"AWS", "Amazon Web Services", and all related service names (EC2, EBS, S3, VPC, IAM, and others) are trademarks of Amazon.com, Inc. or its affiliates. Mulga Defense Corporation and Spinifex are independent and not affiliated with, endorsed by, or sponsored by Amazon.com, Inc. or Amazon Web Services, Inc. References to AWS services describe interoperability and compatibility only.
+
 ## License
 
 Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE). You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.

build/systemd/spinifex-awsgw.service

@@ -5,6 +5,7 @@ After=spinifex-nats.service spinifex-daemon.service
 Wants=spinifex-nats.service
 
 [Service]
+EnvironmentFile=-/etc/spinifex/telemetry.env
 Type=simple
 Slice=spinifex-system.slice
 User=spinifex-gw

build/systemd/spinifex-daemon.service

@@ -6,6 +6,7 @@ After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.servi
 Wants=openvswitch-ipsec.service
 
 [Service]
+EnvironmentFile=-/etc/spinifex/telemetry.env
 Type=simple
 Slice=spinifex-system.slice
 User=spinifex-daemon

build/systemd/spinifex-nats.service

@@ -4,6 +4,7 @@ PartOf=spinifex.target
 Before=spinifex-predastore.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service
 
 [Service]
+EnvironmentFile=-/etc/spinifex/telemetry.env
 Type=simple
 Slice=spinifex-system.slice
 User=spinifex-nats

build/systemd/spinifex-northstar.service

@@ -0,0 +1,54 @@
+[Unit]
+Description=Spinifex Northstar (Authoritative DNS)
+PartOf=spinifex.target
+After=spinifex-nats.service spinifex-predastore.service
+Wants=spinifex-predastore.service
+
+[Service]
+EnvironmentFile=-/etc/spinifex/telemetry.env
+Type=simple
+Slice=spinifex-system.slice
+User=spinifex-northstar
+Group=spinifex
+ExecStart=/usr/local/bin/spx service northstar start
+Restart=on-failure
+RestartSec=5
+OOMScoreAdjust=-500
+Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
+Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/northstar/
+
+# Filesystem access
+NoNewPrivileges=yes
+ProtectSystem=strict
+PrivateDevices=yes
+ReadOnlyPaths=/etc/spinifex/northstar /etc/spinifex/spinifex.toml /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/northstar /var/log/spinifex
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+# RG-9: exactly CAP_NET_BIND_SERVICE so the unprivileged user binds :53 (no root).
+AmbientCapabilities=CAP_NET_BIND_SERVICE
+CapabilityBoundingSet=CAP_NET_BIND_SERVICE
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
+
+[Install]
+WantedBy=spinifex.target

build/systemd/spinifex-predastore.service

@@ -5,6 +5,10 @@ After=spinifex-nats.service
 Wants=spinifex-nats.service
 
 [Service]
+EnvironmentFile=-/etc/spinifex/telemetry.env
+# Guest block I/O arrives with no caller trace; sample those roots at 5% so
+# they don't drown request traces. Request-linked spans are always kept.
+Environment=MULGA_ROOT_TRACE_RATIO=0.05
 Type=simple
 Slice=spinifex-system.slice
 User=spinifex-storage

build/systemd/spinifex-qmp-collector.service

@@ -0,0 +1,56 @@
+[Unit]
+Description=Spinifex QMP Telemetry Collector (guest metrics)
+PartOf=spinifex.target
+After=spinifex-nats.service spinifex-daemon.service
+Wants=spinifex-nats.service
+
+[Service]
+EnvironmentFile=-/etc/spinifex/telemetry.env
+Type=simple
+Slice=spinifex-system.slice
+# spinifex-daemon user: the collector reads the daemon's telemetry QMP
+# sockets in /run/spinifex and the daemon-owned QEMU /proc entries.
+User=spinifex-daemon
+Group=spinifex
+ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
+ExecStart=/usr/local/bin/spx service qmp-collector start
+Restart=on-failure
+RestartSec=5
+Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
+Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex/
+# Same runtime dir as the daemon: telemetry sockets + metadata live there
+Environment=XDG_RUNTIME_DIR=/run/spinifex
+
+# RG-9: read-mostly telemetry tier — no capabilities, no device access
+ProtectSystem=full
+PrivateDevices=yes
+NoNewPrivileges=yes
+ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/spinifex /run/spinifex
+
+# RG-9: hardening baseline
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+# ProtectProc must stay default: hidepid=invisible hides the QEMU vCPU threads
+# (dumpable=0) this collector must read from /proc/<tid>/stat, even same-uid.
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=512
+
+[Install]
+WantedBy=spinifex.target

build/systemd/spinifex-ui.service

@@ -4,6 +4,7 @@ PartOf=spinifex.target
 After=spinifex-awsgw.service
 
 [Service]
+EnvironmentFile=-/etc/spinifex/telemetry.env
 Type=simple
 Slice=spinifex-system.slice
 User=spinifex-ui

build/systemd/spinifex-viperblock.service

@@ -5,6 +5,7 @@ After=spinifex-nats.service spinifex-predastore.service
 Wants=spinifex-nats.service
 
 [Service]
+EnvironmentFile=-/etc/spinifex/telemetry.env
 Type=simple
 Slice=spinifex-system.slice
 User=spinifex-viperblock

build/systemd/spinifex-vpcd.service

@@ -5,6 +5,7 @@ After=spinifex-nats.service spinifex-daemon.service systemd-networkd.service ope
 Wants=spinifex-nats.service ovn-controller.service
 
 [Service]
+EnvironmentFile=-/etc/spinifex/telemetry.env
 Type=simple
 Slice=spinifex-system.slice
 User=spinifex-vpcd

build/systemd/spinifex.target

@@ -2,7 +2,7 @@
 Description=Spinifex Infrastructure Platform
 After=network-online.target
 Wants=network-online.target
-Wants=spinifex-nats.service spinifex-predastore.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-shutdown.service spinifex-nats-watchdog.timer openvswitch-switch.service ovn-controller.service
+Wants=spinifex-nats.service spinifex-predastore.service spinifex-northstar.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-qmp-collector.service spinifex-shutdown.service spinifex-nats-watchdog.timer openvswitch-switch.service ovn-controller.service
 
 [Install]
 WantedBy=multi-user.target

cmd/dhcptest/main.go

@@ -6,7 +6,7 @@
 // Usage (must run as root — needs AF_PACKET):
 //
 //	sudo dhcptest --iface=br-wan
-//	sudo dhcptest --iface=br-wan --mac=02:00:00:7c:b7:b9 --promisc
+//	sudo dhcptest --iface=br-wan --mac=02:54:df:71:22:5c --promisc
 //	sudo dhcptest --iface=br-wan --spinifex-id=gateway-wan1
 package main
 
@@ -14,15 +14,15 @@ import (
 	"context"
 	"flag"
 	"fmt"
-	"log"
+	"log/slog"
 	"net"
 	"os"
 	"os/exec"
 	"strings"
 	"time"
 
-	"github.com/insomniacslk/dhcp/dhcpv4"
 	"github.com/insomniacslk/dhcp/dhcpv4/nclient4"
+	"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
 )
 
 func main() {
@@ -44,20 +44,25 @@ func main() {
 
 	switch {
 	case *spinifexID != "":
-		hwAddr, err = net.ParseMAC(generateMAC(*spinifexID))
+		// vpcd's own derivation, so the probe leases under the MAC the daemon
+		// would have used for this id.
+		hwAddr, err = dhcp.DeriveMAC(*spinifexID)
 		if err != nil {
-			log.Fatalf("generateMAC(%q): %v", *spinifexID, err)
+			slog.Error("Could not derive MAC", "spinifex-id", *spinifexID, "error", err)
+			os.Exit(1)
 		}
 		fmt.Printf("spinifex virtual MAC for %q: %s\n", *spinifexID, hwAddr)
 	case *macStr != "":
 		hwAddr, err = net.ParseMAC(*macStr)
 		if err != nil {
-			log.Fatalf("parse --mac %q: %v", *macStr, err)
+			slog.Error("Could not parse --mac", "mac", *macStr, "error", err)
+			os.Exit(1)
 		}
 	default:
 		hwAddr, err = ifaceMAC(*iface)
 		if err != nil {
-			log.Fatalf("get MAC for %s: %v", *iface, err)
+			slog.Error("Could not get MAC for interface", "iface", *iface, "error", err)
+			os.Exit(1)
 		}
 		fmt.Printf("using real interface MAC: %s\n", hwAddr)
 	}
@@ -83,7 +88,7 @@ func main() {
 
 	if *promisc {
 		if err := setPromisc(*iface, true); err != nil {
-			log.Printf("WARNING: failed to set promisc on %s: %v", *iface, err)
+			slog.Warn("Failed to set promisc on interface", "iface", *iface, "error", err)
 		} else {
 			fmt.Printf("set %s promisc ON\n\n", *iface)
 		}
@@ -96,26 +101,19 @@ func main() {
 		nclient4.WithRetry(*retries),
 	}
 	if *verbose {
-		opts = append(opts, nclient4.WithLogger(nclient4.ShortSummaryLogger{Printfer: log.New(os.Stderr, "[nclient4] ", 0)}))
+		opts = append(opts, nclient4.WithLogger(nclient4.ShortSummaryLogger{Printfer: slogPrintfer{}}))
 	}
 
 	client, err := nclient4.New(*iface, opts...)
 	if err != nil {
-		log.Fatalf("nclient4.New(%s): %v", *iface, err)
+		slog.Error("Could not create nclient4 client", "iface", *iface, "error", err)
+		os.Exit(1)
 	}
 	defer client.Close()
 
-	// Identity modifiers — client-id / hostname / vendor-class for upstream lease tagging.
-	var mods []dhcpv4.Modifier
-	if cid != "" {
-		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClientIdentifier([]byte(cid))))
-	}
-	if *hostname != "" {
-		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptHostName(*hostname)))
-	}
-	if *vendorClass != "" {
-		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClassIdentifier(*vendorClass)))
-	}
+	// Identity modifiers — client-id / hostname / vendor-class for upstream lease
+	// tagging. Built by the vpcd code path so the probe cannot drift from it.
+	mods := dhcp.IdentityModifiers(cid, *hostname, *vendorClass, hwAddr)
 
 	fmt.Println("Sending DHCP Discover...")
 	start := time.Now()
@@ -141,14 +139,12 @@ func main() {
 	}
 }
 
-// generateMAC is the same deterministic hash used by vpcd's topology.go.
-// It produces a locally-administered MAC (02:00:00:xx:xx:xx) from a resource ID.
-func generateMAC(resourceID string) string {
-	h := uint32(0)
-	for _, c := range resourceID {
-		h = h*31 + uint32(c)
-	}
-	return fmt.Sprintf("02:00:00:%02x:%02x:%02x", (h>>16)&0xff, (h>>8)&0xff, h&0xff)
+// slogPrintfer adapts nclient4's Printfer to slog. nclient4 only emits through
+// it under --verbose, so the packet summaries log at info level to stay visible.
+type slogPrintfer struct{}
+
+func (slogPrintfer) Printf(format string, v ...any) {
+	slog.Info(fmt.Sprintf(format, v...), "component", "nclient4")
 }
 
 func ifaceMAC(name string) (net.HardwareAddr, error) {

cmd/ecs-agent/agent.go

@@ -43,6 +43,11 @@ type Agent struct {
 	// cred serves task IAM role credentials at 169.254.170.2 (nil in unit tests).
 	cred *credEndpoint
 
+	// gpu is the local pinning ledger over the host's discovered GPU UUIDs.
+	// Always non-nil (newAgent seeds an empty one); New() reseeds it from
+	// nvidia-smi discovery.
+	gpu *gpuLedger
+
 	closers []func() error
 }
 
@@ -61,6 +66,7 @@ func newAgent(cfg config, id identity, cp controlPlane, puller ctrruntime.ImageP
 		reg:      newRegistrar(cp, id),
 		hb:       newHeartbeater(cp, id, cfg.Heartbeat),
 		netns:    newTaskNetns(execNetRunner{}),
+		gpu:      newGPULedger(id.Capacity.GPUIDs),
 	}
 }
 
@@ -84,7 +90,7 @@ func New(cfg config) (*Agent, error) {
 		InstanceID:   meta.InstanceID,
 		AZ:           meta.AZ,
 		Hostname:     host,
-		Capacity:     detectCapacity(),
+		Capacity:     detectCapacity(discoverNvidiaGPUs(execCommandRunner)),
 		AgentVersion: version,
 	}
 
@@ -153,11 +159,15 @@ func (a *Agent) Stop() error {
 	return firstErr
 }
 
-// detectCapacity reports the host's total CPU units (1 vCPU = 1024) and memory.
-func detectCapacity() bus.InstanceCapacity {
+// detectCapacity reports the host's total CPU units (1 vCPU = 1024), memory,
+// and GPU count/UUIDs from the caller's nvidia-smi discovery (nil on a
+// non-GPU host).
+func detectCapacity(gpuUUIDs []string) bus.InstanceCapacity {
 	return bus.InstanceCapacity{
 		CPU:       goruntime.NumCPU() * 1024,
 		MemoryMiB: memTotalMiB(),
+		GPU:       len(gpuUUIDs),
+		GPUIDs:    gpuUUIDs,
 	}
 }
 

cmd/ecs-agent/agent_test.go

@@ -42,7 +42,7 @@ func TestAgent_RunRegistersThenStopsOnContext(t *testing.T) {
 }
 
 func TestDetectCapacity_Positive(t *testing.T) {
-	c := detectCapacity()
+	c := detectCapacity(nil)
 	if c.CPU <= 0 {
 		t.Errorf("CPU = %d, want > 0", c.CPU)
 	}
@@ -50,4 +50,17 @@ func TestDetectCapacity_Positive(t *testing.T) {
 	if c.MemoryMiB < 0 {
 		t.Errorf("MemoryMiB = %d, want >= 0", c.MemoryMiB)
 	}
+	if c.GPU != 0 || c.GPUIDs != nil {
+		t.Errorf("no GPU UUIDs given: want GPU=0/GPUIDs=nil, got %d/%v", c.GPU, c.GPUIDs)
+	}
+}
+
+func TestDetectCapacity_WithGPUUUIDs(t *testing.T) {
+	c := detectCapacity([]string{"GPU-aaa", "GPU-bbb"})
+	if c.GPU != 2 {
+		t.Errorf("GPU = %d, want 2", c.GPU)
+	}
+	if len(c.GPUIDs) != 2 {
+		t.Errorf("GPUIDs = %v, want 2 entries", c.GPUIDs)
+	}
 }

cmd/ecs-agent/controlplane.go

@@ -25,9 +25,13 @@ type controlPlane interface {
 	Register(id identity) error
 	// SubmitTaskState reports a task's RUNNING/STOPPED transition.
 	SubmitTaskState(st bus.TaskState) error
-	// PollAssignments drains this instance's assignment inbox, acking the taskIDs
-	// accepted on the previous poll. Returns the still-pending assignments.
-	PollAssignments(cluster, instance string, ack []string) ([]bus.Assign, error)
+	// PollAssignments drains this instance's assignment + stop inboxes, acking the
+	// assign taskIDs and stop taskIDs accepted on the previous poll. Returns the
+	// still-pending assignments and stop directives.
+	PollAssignments(cluster, instance string, ackAssigns, ackStops []string) ([]bus.Assign, []bus.StopDirective, error)
+	// ReportTaskGPU reports the local ledger's pinned device UUIDs for a task's
+	// containers, so the control plane can populate DescribeTasks gpuIds.
+	ReportTaskGPU(cluster, task string, containers []handlers_ecs.ContainerGPUReport) error
 }
 
 // gatewayControlPlane implements controlPlane over the SigV4 ECS gateway client.
@@ -73,6 +77,11 @@ func (g *gatewayControlPlane) Register(id identity) error {
 		},
 		VersionInfo: &ecs.VersionInfo{AgentVersion: aws.String(id.AgentVersion)},
 	}
+	if len(id.Capacity.GPUIDs) > 0 {
+		in.TotalResources = append(in.TotalResources, &ecs.Resource{
+			Name: aws.String("GPU"), Type: aws.String("STRINGSET"), StringSetValue: aws.StringSlice(id.Capacity.GPUIDs),
+		})
+	}
 	body, err := json.Marshal(in)
 	if err != nil {
 		return fmt.Errorf("marshal register: %w", err)
@@ -111,19 +120,36 @@ func (g *gatewayControlPlane) SubmitTaskState(st bus.TaskState) error {
 	return nil
 }
 
-func (g *gatewayControlPlane) PollAssignments(cluster, instance string, ack []string) ([]bus.Assign, error) {
-	in := &handlers_ecs.PollAssignmentsInput{Cluster: cluster, ContainerInstance: instance, AckTaskIDs: ack}
+func (g *gatewayControlPlane) PollAssignments(cluster, instance string, ackAssigns, ackStops []string) ([]bus.Assign, []bus.StopDirective, error) {
+	in := &handlers_ecs.PollAssignmentsInput{
+		Cluster:           cluster,
+		ContainerInstance: instance,
+		AckTaskIDs:        ackAssigns,
+		AckStopIDs:        ackStops,
+	}
 	body, err := json.Marshal(in)
 	if err != nil {
-		return nil, fmt.Errorf("marshal poll: %w", err)
+		return nil, nil, fmt.Errorf("marshal poll: %w", err)
 	}
 	resp, err := g.client.Call("PollAssignments", body)
 	if err != nil {
-		return nil, fmt.Errorf("poll: %w", err)
+		return nil, nil, fmt.Errorf("poll: %w", err)
 	}
 	var out handlers_ecs.PollAssignmentsOutput
 	if err := json.Unmarshal(resp, &out); err != nil {
-		return nil, fmt.Errorf("decode poll: %w", err)
+		return nil, nil, fmt.Errorf("decode poll: %w", err)
 	}
-	return out.Assignments, nil
+	return out.Assignments, out.Stops, nil
+}
+
+func (g *gatewayControlPlane) ReportTaskGPU(cluster, task string, containers []handlers_ecs.ContainerGPUReport) error {
+	in := &handlers_ecs.ReportTaskGPUInput{Cluster: cluster, Task: task, Containers: containers}
+	body, err := json.Marshal(in)
+	if err != nil {
+		return fmt.Errorf("marshal report-task-gpu: %w", err)
+	}
+	if _, err := g.client.Call("ReportTaskGPU", body); err != nil {
+		return fmt.Errorf("report task gpu: %w", err)
+	}
+	return nil
 }

cmd/ecs-agent/controlplane_test.go

@@ -0,0 +1,136 @@
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/aws/aws-sdk-go/service/ecs"
+	"github.com/mulgadc/spinifex/cmd/ecs-agent/credentials"
+	handlers_ecs "github.com/mulgadc/spinifex/spinifex/handlers/ecs"
+)
+
+// fixedCredsProvider is a static credentials.CredentialsProvider for tests
+// that only need a valid SigV4 signature, not real IMDS rotation.
+type fixedCredsProvider struct{}
+
+func (fixedCredsProvider) Retrieve(context.Context) (credentials.Credentials, error) {
+	return credentials.Credentials{AccessKeyID: "AKIATEST", SecretAccessKey: "secret"}, nil
+}
+
+func newTestGatewayControlPlane(t *testing.T, srv *httptest.Server) *gatewayControlPlane {
+	t.Helper()
+	cp, err := newGatewayControlPlane(config{GatewayURL: srv.URL, Region: "us-east-1"}, fixedCredsProvider{})
+	if err != nil {
+		t.Fatalf("newGatewayControlPlane: %v", err)
+	}
+	return cp
+}
+
+// Register carries the discovered GPU UUIDs as a STRINGSET resource, matching
+// the control plane's RegisterContainerInstance GPU parsing (Epic C3).
+func TestGatewayControlPlane_Register_SendsGPUStringSet(t *testing.T) {
+	var body []byte
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		body, _ = io.ReadAll(r.Body)
+		_, _ = w.Write([]byte(`{}`))
+	}))
+	defer srv.Close()
+
+	cp := newTestGatewayControlPlane(t, srv)
+	id := identity{
+		ClusterName: "default", InstanceID: "i-1",
+		Capacity: detectCapacity([]string{"GPU-aaa", "GPU-bbb"}),
+	}
+	if err := cp.Register(id); err != nil {
+		t.Fatalf("Register: %v", err)
+	}
+
+	var in ecs.RegisterContainerInstanceInput
+	if err := json.Unmarshal(body, &in); err != nil {
+		t.Fatalf("decode body: %v", err)
+	}
+	var gpu *ecs.Resource
+	for _, r := range in.TotalResources {
+		if r.Name != nil && *r.Name == "GPU" {
+			gpu = r
+		}
+	}
+	if gpu == nil {
+		t.Fatal("no GPU resource in TotalResources")
+	}
+	if *gpu.Type != "STRINGSET" {
+		t.Errorf("GPU resource type = %q, want STRINGSET", *gpu.Type)
+	}
+	got := make([]string, len(gpu.StringSetValue))
+	for i, v := range gpu.StringSetValue {
+		got[i] = *v
+	}
+	if len(got) != 2 || got[0] != "GPU-aaa" || got[1] != "GPU-bbb" {
+		t.Errorf("GPU UUIDs = %v, want [GPU-aaa GPU-bbb]", got)
+	}
+}
+
+// A non-GPU host (no discovered UUIDs) registers with no GPU resource at all.
+func TestGatewayControlPlane_Register_NoGPUOmitsResource(t *testing.T) {
+	var body []byte
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		body, _ = io.ReadAll(r.Body)
+		_, _ = w.Write([]byte(`{}`))
+	}))
+	defer srv.Close()
+
+	cp := newTestGatewayControlPlane(t, srv)
+	id := identity{ClusterName: "default", InstanceID: "i-1", Capacity: detectCapacity(nil)}
+	if err := cp.Register(id); err != nil {
+		t.Fatalf("Register: %v", err)
+	}
+
+	var in ecs.RegisterContainerInstanceInput
+	if err := json.Unmarshal(body, &in); err != nil {
+		t.Fatalf("decode body: %v", err)
+	}
+	for _, r := range in.TotalResources {
+		if r.Name != nil && *r.Name == "GPU" {
+			t.Fatalf("unexpected GPU resource on a non-GPU host: %+v", r)
+		}
+	}
+}
+
+// ReportTaskGPU posts the per-container device UUIDs under the ReportTaskGPU
+// action, decodable as the handlers_ecs internal shape.
+func TestGatewayControlPlane_ReportTaskGPU(t *testing.T) {
+	var gotTarget string
+	var body []byte
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		gotTarget = r.Header.Get("X-Amz-Target")
+		body, _ = io.ReadAll(r.Body)
+		_, _ = w.Write([]byte(`{}`))
+	}))
+	defer srv.Close()
+
+	cp := newTestGatewayControlPlane(t, srv)
+	err := cp.ReportTaskGPU("web", "t-1", []handlers_ecs.ContainerGPUReport{
+		{Name: "trainer", GPUIDs: []string{"GPU-aaa"}},
+	})
+	if err != nil {
+		t.Fatalf("ReportTaskGPU: %v", err)
+	}
+	if gotTarget == "" {
+		t.Fatal("no X-Amz-Target header sent")
+	}
+
+	var in handlers_ecs.ReportTaskGPUInput
+	if err := json.Unmarshal(body, &in); err != nil {
+		t.Fatalf("decode body: %v", err)
+	}
+	if in.Cluster != "web" || in.Task != "t-1" {
+		t.Errorf("cluster/task = %q/%q, want web/t-1", in.Cluster, in.Task)
+	}
+	if len(in.Containers) != 1 || in.Containers[0].Name != "trainer" || len(in.Containers[0].GPUIDs) != 1 {
+		t.Errorf("containers = %+v, want one trainer entry with one UUID", in.Containers)
+	}
+}

cmd/ecs-agent/credendpoint.go

@@ -3,6 +3,7 @@ package main
 import (
 	"context"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"log/slog"
 	"net"
@@ -143,7 +144,7 @@ func (c *credEndpoint) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	}
 	creds, roleARN, err := c.fetch(r.Context(), credID)
 	if err != nil {
-		if err == errUnknownCredID {
+		if errors.Is(err, errUnknownCredID) {
 			http.Error(w, "not found", http.StatusNotFound)
 			return
 		}
@@ -192,6 +193,45 @@ func (c *credEndpoint) fetch(ctx context.Context, credID string) (stsauth.Creden
 	return creds, roleARN, nil
 }
 
+// AssumeProvider returns a CredentialsProvider that mints credentials by assuming
+// roleARN over the gateway — the execution-role path for ECR image pulls. It
+// caches the assumed set and re-assumes within the refresh margin. session scopes
+// the STS session name.
+func (c *credEndpoint) AssumeProvider(roleARN, session string) credentials.CredentialsProvider {
+	return &assumeProvider{ep: c, roleARN: roleARN, session: session}
+}
+
+// assumeProvider adapts a credEndpoint's AssumeRole path to the agent's
+// CredentialsProvider interface, caching the assumed credentials.
+type assumeProvider struct {
+	ep      *credEndpoint
+	roleARN string
+	session string
+
+	mu     sync.Mutex
+	cached stsauth.Credentials
+}
+
+var _ credentials.CredentialsProvider = (*assumeProvider)(nil)
+
+func (p *assumeProvider) Retrieve(ctx context.Context) (credentials.Credentials, error) {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	if !credValid(p.cached, credRefreshMargin) {
+		creds, err := p.ep.assume(ctx, p.roleARN, p.session)
+		if err != nil {
+			return credentials.Credentials{}, err
+		}
+		p.cached = creds
+	}
+	return credentials.Credentials{
+		AccessKeyID:     p.cached.AccessKeyID,
+		SecretAccessKey: p.cached.SecretAccessKey,
+		SessionToken:    p.cached.SessionToken,
+		Expiration:      p.cached.Expiration,
+	}, nil
+}
+
 // assumeOverGateway mints credentials for roleARN by SigV4-signing AssumeRole
 // against the gateway with the agent's instance credentials.
 func (c *credEndpoint) assumeOverGateway(ctx context.Context, roleARN, sessionName string) (stsauth.Credentials, error) {

cmd/ecs-agent/gpu_discovery.go

@@ -0,0 +1,42 @@
+package main
+
+import (
+	"os/exec"
+	"strings"
+)
+
+// commandRunner abstracts exec.Command so discovery is unit-testable without
+// shelling out. Production uses execCommandRunner; tests inject a fake.
+type commandRunner func(name string, args ...string) ([]byte, error)
+
+// execCommandRunner runs a real host command and returns its stdout.
+func execCommandRunner(name string, args ...string) ([]byte, error) {
+	return exec.Command(name, args...).Output()
+}
+
+// discoverNvidiaGPUs enumerates local NVIDIA GPU device UUIDs via nvidia-smi.
+// A missing binary or any run failure (no GPU host) yields an empty list, not
+// an error — the agent still registers with zero GPU capacity.
+func discoverNvidiaGPUs(run commandRunner) []string {
+	out, err := run("nvidia-smi", "--query-gpu=uuid", "--format=csv,noheader")
+	if err != nil {
+		return nil
+	}
+	return parseNvidiaSMIUUIDs(string(out))
+}
+
+// parseNvidiaSMIUUIDs splits nvidia-smi's newline-delimited UUID output,
+// dropping blank lines. Exported as a pure function so fixture strings can be
+// tested without exec.
+func parseNvidiaSMIUUIDs(out string) []string {
+	lines := strings.Split(strings.TrimSpace(out), "\n")
+	uuids := make([]string, 0, len(lines))
+	for _, line := range lines {
+		line = strings.TrimSpace(line)
+		if line == "" {
+			continue
+		}
+		uuids = append(uuids, line)
+	}
+	return uuids
+}

cmd/ecs-agent/gpu_discovery_test.go

@@ -0,0 +1,71 @@
+package main
+
+import (
+	"errors"
+	"reflect"
+	"testing"
+)
+
+func TestParseNvidiaSMIUUIDs(t *testing.T) {
+	cases := []struct {
+		name string
+		in   string
+		want []string
+	}{
+		{"empty", "", nil},
+		{"whitespace only", "   \n\n  ", nil},
+		{"single", "GPU-11111111-1111-1111-1111-111111111111\n", []string{"GPU-11111111-1111-1111-1111-111111111111"}},
+		{"multi", "GPU-aaa\nGPU-bbb\nGPU-ccc\n", []string{"GPU-aaa", "GPU-bbb", "GPU-ccc"}},
+		{"blank lines between", "GPU-aaa\n\nGPU-bbb\n", []string{"GPU-aaa", "GPU-bbb"}},
+		{"trailing whitespace per line", "GPU-aaa \r\nGPU-bbb\r\n", []string{"GPU-aaa", "GPU-bbb"}},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			got := parseNvidiaSMIUUIDs(c.in)
+			if !reflect.DeepEqual(got, c.want) && (len(got) != 0 || len(c.want) != 0) {
+				t.Errorf("parseNvidiaSMIUUIDs(%q) = %v, want %v", c.in, got, c.want)
+			}
+		})
+	}
+}
+
+func TestDiscoverNvidiaGPUs_AbsentBinaryReturnsEmptyNoError(t *testing.T) {
+	run := func(name string, args ...string) ([]byte, error) {
+		return nil, errors.New(`exec: "nvidia-smi": executable file not found in $PATH`)
+	}
+	got := discoverNvidiaGPUs(run)
+	if len(got) != 0 {
+		t.Errorf("want empty list on absent nvidia-smi, got %v", got)
+	}
+}
+
+func TestDiscoverNvidiaGPUs_ParsesFixtureOutput(t *testing.T) {
+	var gotName string
+	var gotArgs []string
+	run := func(name string, args ...string) ([]byte, error) {
+		gotName = name
+		gotArgs = args
+		return []byte("GPU-aaa\nGPU-bbb\n"), nil
+	}
+	got := discoverNvidiaGPUs(run)
+	want := []string{"GPU-aaa", "GPU-bbb"}
+	if !reflect.DeepEqual(got, want) {
+		t.Errorf("got %v, want %v", got, want)
+	}
+	if gotName != "nvidia-smi" {
+		t.Errorf("command = %q, want nvidia-smi", gotName)
+	}
+	if len(gotArgs) != 2 || gotArgs[0] != "--query-gpu=uuid" || gotArgs[1] != "--format=csv,noheader" {
+		t.Errorf("args = %v, want the uuid/csv,noheader query", gotArgs)
+	}
+}
+
+func TestDiscoverNvidiaGPUs_NoDevicesReturnsEmpty(t *testing.T) {
+	run := func(name string, args ...string) ([]byte, error) {
+		return []byte(""), nil
+	}
+	got := discoverNvidiaGPUs(run)
+	if len(got) != 0 {
+		t.Errorf("want empty list, got %v", got)
+	}
+}

cmd/ecs-agent/gpu_ledger.go

@@ -0,0 +1,62 @@
+package main
+
+import (
+	"fmt"
+	"strings"
+	"sync"
+)
+
+// gpuLedger is the agent's local source of truth for which of the host's
+// discovered GPU UUIDs are free vs pinned to a running task's container. C4
+// reads a task's pinned UUIDs to build the CDI device list for containerd.
+type gpuLedger struct {
+	mu     sync.Mutex
+	free   []string
+	pinned map[string][]string // key ("taskID/containerName") -> uuids
+}
+
+// newGPULedger seeds the ledger with the host's discovered device UUIDs. A nil
+// or empty slice is a valid non-GPU host: every Pin then fails, which callers
+// treat as best-effort (no device, no CDI injection later).
+func newGPULedger(uuids []string) *gpuLedger {
+	free := append([]string(nil), uuids...)
+	return &gpuLedger{free: free, pinned: make(map[string][]string)}
+}
+
+// gpuKey builds the ledger key for one container's pin/release.
+func gpuKey(taskID, containerName string) string {
+	return taskID + "/" + containerName
+}
+
+// Pin reserves n free UUIDs under key, returning them. An error means fewer
+// than n devices were free; the key is left unpinned so a retry (or a
+// best-effort caller) can proceed without a partial reservation.
+func (l *gpuLedger) Pin(key string, n int) ([]string, error) {
+	if n <= 0 {
+		return nil, nil
+	}
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	if len(l.free) < n {
+		return nil, fmt.Errorf("gpu ledger: requested %d GPUs, %d free", n, len(l.free))
+	}
+	uuids := append([]string(nil), l.free[:n]...)
+	l.free = l.free[n:]
+	l.pinned[key] = uuids
+	return uuids, nil
+}
+
+// ReleaseTask frees every UUID pinned under any container key belonging to
+// taskID, called once the task's terminal STOPPED state is reported.
+func (l *gpuLedger) ReleaseTask(taskID string) {
+	prefix := taskID + "/"
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	for key, uuids := range l.pinned {
+		if !strings.HasPrefix(key, prefix) {
+			continue
+		}
+		l.free = append(l.free, uuids...)
+		delete(l.pinned, key)
+	}
+}

cmd/ecs-agent/gpu_ledger_test.go

@@ -0,0 +1,155 @@
+package main
+
+import (
+	"sync"
+	"testing"
+)
+
+func TestGPULedger_PinReturnsDistinctUUIDs(t *testing.T) {
+	l := newGPULedger([]string{"GPU-a", "GPU-b", "GPU-c"})
+	got, err := l.Pin("t-1/web", 2)
+	if err != nil {
+		t.Fatalf("Pin: %v", err)
+	}
+	if len(got) != 2 {
+		t.Fatalf("want 2 UUIDs, got %v", got)
+	}
+	if got[0] == got[1] {
+		t.Errorf("pinned UUIDs must be distinct, got %v", got)
+	}
+	if len(l.free) != 1 {
+		t.Errorf("free = %d, want 1 remaining", len(l.free))
+	}
+}
+
+func TestGPULedger_PinInsufficientFreeErrors(t *testing.T) {
+	l := newGPULedger([]string{"GPU-a"})
+	_, err := l.Pin("t-1/web", 2)
+	if err == nil {
+		t.Fatal("want error when requesting more GPUs than free")
+	}
+	if len(l.free) != 1 {
+		t.Errorf("failed pin must not consume any device, free = %d", len(l.free))
+	}
+}
+
+func TestGPULedger_PinZeroIsNoop(t *testing.T) {
+	l := newGPULedger([]string{"GPU-a"})
+	got, err := l.Pin("t-1/web", 0)
+	if err != nil || got != nil {
+		t.Errorf("Pin(0) = %v, %v; want nil, nil", got, err)
+	}
+}
+
+func TestGPULedger_ReleaseTaskFreesOnlyThatTasksDevices(t *testing.T) {
+	l := newGPULedger([]string{"GPU-a", "GPU-b", "GPU-c", "GPU-d"})
+	if _, err := l.Pin(gpuKey("t-1", "web"), 2); err != nil {
+		t.Fatalf("pin t-1: %v", err)
+	}
+	if _, err := l.Pin(gpuKey("t-2", "worker"), 1); err != nil {
+		t.Fatalf("pin t-2: %v", err)
+	}
+	if len(l.free) != 1 {
+		t.Fatalf("free = %d, want 1 before release", len(l.free))
+	}
+
+	l.ReleaseTask("t-1")
+	if len(l.free) != 3 {
+		t.Errorf("free = %d, want 3 after releasing t-1's two devices", len(l.free))
+	}
+	if _, ok := l.pinned[gpuKey("t-2", "worker")]; !ok {
+		t.Error("t-2's pin should survive releasing t-1")
+	}
+	if _, ok := l.pinned[gpuKey("t-1", "web")]; ok {
+		t.Error("t-1's pin should be gone after release")
+	}
+}
+
+// Multi-device pinning across concurrent task assigns must hand out disjoint
+// UUID sets — no two tasks may ever share a device.
+func TestGPULedger_ConcurrentPinsHandOutDistinctUUIDs(t *testing.T) {
+	const nDevices = 20
+	uuids := make([]string, nDevices)
+	for i := range uuids {
+		uuids[i] = "GPU-" + string(rune('a'+i))
+	}
+	l := newGPULedger(uuids)
+
+	var wg sync.WaitGroup
+	var mu sync.Mutex
+	seen := make(map[string]bool)
+	errs := 0
+	for i := range nDevices {
+		wg.Add(1)
+		go func(n int) {
+			defer wg.Done()
+			key := gpuKey("t-concurrent", string(rune('A'+n)))
+			got, err := l.Pin(key, 1)
+			mu.Lock()
+			defer mu.Unlock()
+			if err != nil {
+				errs++
+				return
+			}
+			for _, u := range got {
+				if seen[u] {
+					t.Errorf("UUID %s handed out to more than one concurrent pin", u)
+				}
+				seen[u] = true
+			}
+		}(i)
+	}
+	wg.Wait()
+	if errs != 0 {
+		t.Errorf("want all %d pins to succeed (exactly enough devices), got %d errors", nDevices, errs)
+	}
+	if len(seen) != nDevices {
+		t.Errorf("distinct UUIDs handed out = %d, want %d", len(seen), nDevices)
+	}
+	if len(l.free) != 0 {
+		t.Errorf("free = %d, want 0 after all devices pinned", len(l.free))
+	}
+}
+
+// Releasing a task's devices concurrently with new pins never double-hands a
+// UUID: a device is only ever free or pinned to exactly one key at a time.
+func TestGPULedger_ConcurrentPinAndReleaseNoDoubleAssignment(t *testing.T) {
+	l := newGPULedger([]string{"GPU-a", "GPU-b"})
+	var wg sync.WaitGroup
+	for round := range 50 {
+		wg.Add(2)
+		taskID := "t-" + string(rune('a'+round%26))
+		go func(id string) {
+			defer wg.Done()
+			_, _ = l.Pin(gpuKey(id, "web"), 1)
+		}(taskID)
+		go func(id string) {
+			defer wg.Done()
+			l.ReleaseTask(id)
+		}(taskID)
+	}
+	wg.Wait()
+	// Invariant: every currently-pinned UUID appears in exactly one pinned entry
+	// and never simultaneously in free.
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	freeSet := make(map[string]bool, len(l.free))
+	for _, u := range l.free {
+		if freeSet[u] {
+			t.Errorf("duplicate UUID %s in free list", u)
+		}
+		freeSet[u] = true
+	}
+	pinnedSet := make(map[string]bool)
+	for _, uuids := range l.pinned {
+		for _, u := range uuids {
+			if pinnedSet[u] {
+				t.Errorf("UUID %s pinned under more than one key", u)
+			}
+			if freeSet[u] {
+				t.Errorf("UUID %s is both free and pinned", u)
+			}
+			pinnedSet[u] = true
+		}
+	}
+}

cmd/ecs-agent/registration_test.go

@@ -7,24 +7,36 @@ import (
 	"testing"
 	"time"
 
+	handlers_ecs "github.com/mulgadc/spinifex/spinifex/handlers/ecs"
 	"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
 )
 
-// fakeCP is a test double for the gateway control-plane. It records registers and
-// task-state reports, and replays a scripted queue of poll responses.
+// fakeCP is a test double for the gateway control-plane. It records registers,
+// task-state reports and GPU reports, and replays a scripted queue of poll
+// responses.
 type fakeCP struct {
 	mu sync.Mutex
 
-	registers   int
-	states      []bus.TaskState
-	pollAcks    [][]string
-	pollReplies [][]bus.Assign
-	pollCalls   int
+	registers    int
+	states       []bus.TaskState
+	gpuReports   []gpuReport
+	pollAcks     [][]string
+	pollStopAcks [][]string
+	pollReplies  [][]bus.Assign
+	stopReplies  [][]bus.StopDirective
+	pollCalls    int
 
 	registerErr bool
 	submitErr   bool
 }
 
+// gpuReport is one recorded ReportTaskGPU call.
+type gpuReport struct {
+	cluster    string
+	task       string
+	containers []handlers_ecs.ContainerGPUReport
+}
+
 var _ controlPlane = (*fakeCP)(nil)
 
 func (f *fakeCP) Register(_ identity) error {
@@ -47,17 +59,40 @@ func (f *fakeCP) SubmitTaskState(st bus.TaskState) error {
 	return nil
 }
 
-func (f *fakeCP) PollAssignments(_, _ string, ack []string) ([]bus.Assign, error) {
+func (f *fakeCP) PollAssignments(_, _ string, ackAssigns, ackStops []string) ([]bus.Assign, []bus.StopDirective, error) {
 	f.mu.Lock()
 	defer f.mu.Unlock()
-	cp := append([]string(nil), ack...)
-	f.pollAcks = append(f.pollAcks, cp)
+	f.pollAcks = append(f.pollAcks, append([]string(nil), ackAssigns...))
+	f.pollStopAcks = append(f.pollStopAcks, append([]string(nil), ackStops...))
 	var out []bus.Assign
 	if f.pollCalls < len(f.pollReplies) {
 		out = f.pollReplies[f.pollCalls]
 	}
+	var stops []bus.StopDirective
+	if f.pollCalls < len(f.stopReplies) {
+		stops = f.stopReplies[f.pollCalls]
+	}
 	f.pollCalls++
-	return out, nil
+	return out, stops, nil
+}
+
+func (f *fakeCP) ReportTaskGPU(cluster, task string, containers []handlers_ecs.ContainerGPUReport) error {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.gpuReports = append(f.gpuReports, gpuReport{cluster: cluster, task: task, containers: containers})
+	return nil
+}
+
+func (f *fakeCP) gpuReportsFor(taskID string) []gpuReport {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	var out []gpuReport
+	for _, r := range f.gpuReports {
+		if r.task == taskID {
+			out = append(out, r)
+		}
+	}
+	return out
 }
 
 func (f *fakeCP) registerCount() int {
@@ -78,6 +113,12 @@ func (f *fakeCP) acks() [][]string {
 	return append([][]string(nil), f.pollAcks...)
 }
 
+func (f *fakeCP) stopAcks() [][]string {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	return append([][]string(nil), f.pollStopAcks...)
+}
+
 func testIdentity() identity {
 	return identity{
 		AccountID:    "123456789012",
@@ -90,6 +131,15 @@ func testIdentity() identity {
 	}
 }
 
+// testIdentityWithGPUs is testIdentity seeded with discovered GPU UUIDs, so
+// newAgent's ledger has real devices to pin from.
+func testIdentityWithGPUs(uuids ...string) identity {
+	id := testIdentity()
+	id.Capacity.GPU = len(uuids)
+	id.Capacity.GPUIDs = uuids
+	return id
+}
+
 func TestRegistrar_Register(t *testing.T) {
 	cp := &fakeCP{}
 	if err := newRegistrar(cp, testIdentity()).Register(); err != nil {

cmd/ecs-agent/runtime/containerd_run.go

@@ -6,12 +6,41 @@ import (
 	"sort"
 
 	containerd "github.com/containerd/containerd/v2/client"
+	"github.com/containerd/containerd/v2/pkg/cdi"
 	"github.com/containerd/containerd/v2/pkg/cio"
 	"github.com/containerd/containerd/v2/pkg/namespaces"
 	"github.com/containerd/containerd/v2/pkg/oci"
 	specs "github.com/opencontainers/runtime-spec/specs-go"
 )
 
+// cdiVendorGPU is the NVIDIA CDI vendor/class prefix for whole-GPU devices, per
+// the NVIDIA CDI naming convention (nvidia.com/gpu=<uuid>).
+const cdiVendorGPU = "nvidia.com/gpu"
+
+// cdiDeviceNames maps a container's pinned GPU device UUIDs to CDI device
+// names. Empty input (non-GPU container) returns nil so no CDI opt is added.
+func cdiDeviceNames(uuids []string) []string {
+	if len(uuids) == 0 {
+		return nil
+	}
+	names := make([]string, len(uuids))
+	for i, id := range uuids {
+		names[i] = cdiVendorGPU + "=" + id
+	}
+	return names
+}
+
+// cdiSpecOpts returns the oci.SpecOpts needed to inject gpuIDs (via
+// containerd's CDI machinery) as devices into a container's OCI spec. It
+// returns nil for a non-GPU container so Run adds nothing to the opts list.
+func cdiSpecOpts(gpuIDs []string) []oci.SpecOpts {
+	devices := cdiDeviceNames(gpuIDs)
+	if len(devices) == 0 {
+		return nil
+	}
+	return []oci.SpecOpts{cdi.WithCDIDevices(devices...)}
+}
+
 var _ Runner = (*containerdPuller)(nil)
 
 // Run creates a container from an already-pulled image and starts its task. The
@@ -51,6 +80,7 @@ func (p *containerdPuller) Run(ctx context.Context, id string, spec RunSpec) (st
 	if len(spec.Env) > 0 {
 		specOpts = append(specOpts, oci.WithEnv(envSlice(spec.Env)))
 	}
+	specOpts = append(specOpts, cdiSpecOpts(spec.GPUIDs)...)
 
 	container, err := p.client.NewContainer(ctx, id,
 		containerd.WithNewSnapshot(id+"-snapshot", image),

cmd/ecs-agent/runtime/containerd_run_test.go

@@ -0,0 +1,47 @@
+package runtime
+
+import (
+	"reflect"
+	"testing"
+)
+
+// cdiDeviceNames maps a container's pinned GPU UUIDs to CDI device names
+// (nvidia.com/gpu=<uuid>); Run wires the result into oci.WithCDIDevices when
+// non-empty, so this is the injection logic's unit-testable core.
+func TestCDIDeviceNames(t *testing.T) {
+	cases := []struct {
+		name  string
+		uuids []string
+		want  []string
+	}{
+		{name: "empty", uuids: nil, want: nil},
+		{name: "single", uuids: []string{"GPU-aaaaaaaa-1111-2222-3333-444444444444"},
+			want: []string{"nvidia.com/gpu=GPU-aaaaaaaa-1111-2222-3333-444444444444"}},
+		{name: "multiple preserves order", uuids: []string{"GPU-aaa", "GPU-bbb"},
+			want: []string{"nvidia.com/gpu=GPU-aaa", "nvidia.com/gpu=GPU-bbb"}},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			got := cdiDeviceNames(tc.uuids)
+			if !reflect.DeepEqual(got, tc.want) {
+				t.Errorf("cdiDeviceNames(%v) = %v, want %v", tc.uuids, got, tc.want)
+			}
+		})
+	}
+}
+
+// TestCDISpecOpts_Wiring confirms Run's specOpts list picks up exactly one CDI
+// opt when GPUIDs is non-empty, and none for a non-GPU container — the wiring
+// Run relies on (a live oci.Spec is needed to unit test the opt's effect).
+func TestCDISpecOpts_Wiring(t *testing.T) {
+	if opts := cdiSpecOpts(nil); len(opts) != 0 {
+		t.Errorf("non-GPU container: want no spec opts, got %d", len(opts))
+	}
+	opts := cdiSpecOpts([]string{"GPU-aaa", "GPU-bbb"})
+	if len(opts) != 1 {
+		t.Fatalf("GPU container: want exactly one CDI spec opt, got %d", len(opts))
+	}
+	if opts[0] == nil {
+		t.Error("GPU container: spec opt is nil")
+	}
+}

cmd/ecs-agent/runtime/runtime.go

@@ -48,6 +48,12 @@ type RunSpec struct {
 	// (awsvpc task ENI). Empty keeps the container in the host (VM) netns —
 	// bridge/host mode behaviour.
 	NetnsPath string
+	// GPU is the whole-GPU count requested for this container.
+	GPU int
+	// GPUIDs are the ledger-pinned device UUIDs assigned to this container, one
+	// per GPU requested. Empty for non-GPU containers or if pinning fell short.
+	// The runner injects these as CDI devices into the OCI spec.
+	GPUIDs []string
 }
 
 // RunStatus is a finished container's outcome.

cmd/ecs-agent/taskrun.go

@@ -8,6 +8,7 @@ import (
 	"time"
 
 	ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
+	handlers_ecs "github.com/mulgadc/spinifex/spinifex/handlers/ecs"
 	"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
 )
 
@@ -22,31 +23,78 @@ func (a *Agent) pollAssignments(ctx context.Context, dispatched map[string]bool)
 	ticker := time.NewTicker(a.cfg.PollInterval)
 	defer ticker.Stop()
 
-	var ackNext []string
+	stopping := map[string]bool{}
+	var ackAssigns, ackStops []string
 	for {
 		select {
 		case <-ctx.Done():
 			return
 		case <-ticker.C:
-			assigns, err := a.cp.PollAssignments(a.id.ClusterName, a.id.InstanceID, ackNext)
+			assigns, stops, err := a.cp.PollAssignments(a.id.ClusterName, a.id.InstanceID, ackAssigns, ackStops)
 			if err != nil {
 				slog.Warn("ecs-agent: poll assignments failed", "err", err)
 				continue
 			}
-			ackNext = ackNext[:0]
+			// Dispatch stops first so a task stopped in the same poll it was assigned
+			// is never started.
+			ackStops = ackStops[:0]
+			for i := range stops {
+				sd := stops[i]
+				if !stopping[sd.TaskID] {
+					stopping[sd.TaskID] = true
+					slog.Info("ecs-agent: task stop requested", "task", sd.TaskID, "reason", sd.Reason)
+					go a.stopTask(ctx, sd)
+				}
+				ackStops = append(ackStops, sd.TaskID)
+			}
+			ackAssigns = ackAssigns[:0]
 			for i := range assigns {
 				as := assigns[i]
-				if !dispatched[as.TaskID] {
+				if !dispatched[as.TaskID] && !stopping[as.TaskID] {
 					dispatched[as.TaskID] = true
 					slog.Info("ecs-agent: task assigned", "task", as.TaskID, "containers", len(as.Containers))
 					go a.runTask(ctx, &as)
 				}
-				ackNext = append(ackNext, as.TaskID)
+				ackAssigns = append(ackAssigns, as.TaskID)
 			}
 		}
 	}
 }
 
+// stopTask reaps a task's containers on a control-plane stop directive: it lists
+// the runtime's containers, removes the ones labeled with this task (kill +
+// delete, releasing host ports/netns), then reports the task STOPPED with the
+// directive's reason. Idempotent — a task with no live containers still reports
+// STOPPED so the scheduler releases its capacity.
+func (a *Agent) stopTask(ctx context.Context, sd bus.StopDirective) {
+	if a.runner == nil {
+		return
+	}
+	containers, err := a.runner.List(ctx)
+	if err != nil {
+		slog.Warn("ecs-agent: stop list failed", "task", sd.TaskID, "err", err)
+		return
+	}
+	statuses := make([]bus.ContainerStatus, 0)
+	for _, c := range containers {
+		if c.Labels[labelTaskID] != sd.TaskID {
+			continue
+		}
+		if rerr := a.runner.Remove(ctx, c.ID); rerr != nil {
+			slog.Warn("ecs-agent: stop remove failed", "task", sd.TaskID, "container", c.ID, "err", rerr)
+		}
+		statuses = append(statuses, bus.ContainerStatus{
+			Name: c.Labels[labelContainerName], Status: bus.TaskStatusStopped, ContainerID: c.ID,
+		})
+	}
+	reason := sd.Reason
+	if reason == "" {
+		reason = "Task stopped"
+	}
+	a.reportTaskState(&bus.Assign{TaskID: sd.TaskID}, bus.TaskStatusStopped, reason, statuses)
+	slog.Info("ecs-agent: task reaped", "task", sd.TaskID, "containers", len(statuses))
+}
+
 // runTask pulls each container image, starts the containers, and reports task
 // state on the bus. A missing runtime or any per-container failure reports the
 // task STOPPED with a reason; success reports RUNNING and waits for exit.
@@ -65,10 +113,11 @@ func (a *Agent) runTask(ctx context.Context, as *bus.Assign) {
 	}
 
 	credID := a.registerTaskCreds(as)
+	resolver := a.pullResolver(as)
 
 	statuses := make([]bus.ContainerStatus, 0, len(as.Containers))
 	for _, c := range as.Containers {
-		if _, err := a.puller.Pull(ctx, ctrruntime.PullSpec{Ref: c.Image}, a.resolver); err != nil {
+		if _, err := a.puller.Pull(ctx, ctrruntime.PullSpec{Ref: c.Image}, resolver); err != nil {
 			slog.Error("ecs-agent: pull failed", "task", as.TaskID, "image", c.Image, "err", err)
 			a.teardownTaskNetns(as)
 			a.reportTaskState(as, bus.TaskStatusStopped, "image pull failed: "+err.Error(), statuses)
@@ -76,12 +125,15 @@ func (a *Agent) runTask(ctx context.Context, as *bus.Assign) {
 		}
 
 		cid := containerID(as.TaskID, c.Name)
+		gpuIDs := a.pinContainerGPUs(as.TaskID, c.Name, c.GPU)
 		spec := ctrruntime.RunSpec{
 			Image:     c.Image,
 			Command:   c.Command,
 			Env:       withCredEnv(c.Environment, credID),
 			Labels:    taskLabels(as, c.Name),
 			NetnsPath: netnsPath,
+			GPU:       c.GPU,
+			GPUIDs:    gpuIDs,
 		}
 		id, err := a.runner.Run(ctx, cid, spec)
 		if err != nil {
@@ -90,7 +142,9 @@ func (a *Agent) runTask(ctx context.Context, as *bus.Assign) {
 			a.reportTaskState(as, bus.TaskStatusStopped, "container start failed: "+err.Error(), statuses)
 			return
 		}
-		statuses = append(statuses, bus.ContainerStatus{Name: c.Name, Status: bus.TaskStatusRunning, ContainerID: id})
+		statuses = append(statuses, bus.ContainerStatus{
+			Name: c.Name, Status: bus.TaskStatusRunning, ContainerID: id, GPUIDs: gpuIDs,
+		})
 		go a.waitContainer(ctx, as, c.Name, id)
 	}
 
@@ -137,6 +191,18 @@ func (a *Agent) teardownTaskNetns(as *bus.Assign) {
 	}
 }
 
+// pullResolver returns the ECR resolver used for a task's image pulls. When the
+// assign carries an execution role, it authorizes pulls as that role (assumed
+// over the gateway); an empty role or a nil credential endpoint (unit tests)
+// falls back to the instance-role resolver, so a pull is never worse off.
+func (a *Agent) pullResolver(as *bus.Assign) ctrruntime.Resolver {
+	if as.ExecutionRoleARN == "" || a.cred == nil {
+		return a.resolver
+	}
+	prov := a.cred.AssumeProvider(as.ExecutionRoleARN, sessionName(as.TaskID))
+	return newLazyECRResolver(prov, a.cfg.Region, a.cfg.GatewayURL, a.cfg.GatewayCA)
+}
+
 // registerTaskCreds registers the task's credID -> role mapping with the
 // credential endpoint and returns the credID (empty when the task has no role).
 func (a *Agent) registerTaskCreds(as *bus.Assign) string {
@@ -174,8 +240,26 @@ func withCredEnv(env map[string]string, credID string) map[string]string {
 	return out
 }
 
+// pinContainerGPUs reserves n device UUIDs for a container from the local
+// ledger, before the container is started so the runner can inject them as
+// CDI devices. A short ledger (fewer free devices than requested, e.g.
+// discovery found none) is logged and treated as best-effort: the container
+// still runs without pinned UUIDs (and without GPU access) rather than
+// failing the task.
+func (a *Agent) pinContainerGPUs(taskID, containerName string, n int) []string {
+	if n <= 0 || a.gpu == nil {
+		return nil
+	}
+	uuids, err := a.gpu.Pin(gpuKey(taskID, containerName), n)
+	if err != nil {
+		slog.Warn("ecs-agent: gpu pin short", "task", taskID, "container", containerName, "err", err)
+	}
+	return uuids
+}
+
 // reportTaskState reports a task transition through the gateway's
-// SubmitTaskStateChange action.
+// SubmitTaskStateChange action, then (RUNNING with pinned GPUs) reports the
+// per-container device UUIDs, and (STOPPED) releases them back to the ledger.
 func (a *Agent) reportTaskState(as *bus.Assign, status, reason string, containers []bus.ContainerStatus) {
 	if a.cp == nil {
 		return
@@ -193,6 +277,28 @@ func (a *Agent) reportTaskState(as *bus.Assign, status, reason string, container
 	if err := a.cp.SubmitTaskState(msg); err != nil {
 		slog.Error("ecs-agent: report task-state failed", "task", as.TaskID, "err", err)
 	}
+	a.reportTaskGPUs(as.TaskID, containers)
+	if status == bus.TaskStatusStopped && a.gpu != nil {
+		a.gpu.ReleaseTask(as.TaskID)
+	}
+}
+
+// reportTaskGPUs sends the control plane the pinned device UUIDs for any
+// container in the report that has them, so DescribeTasks can surface gpuIds.
+// A no-op when nothing was pinned (the common non-GPU-task case).
+func (a *Agent) reportTaskGPUs(taskID string, containers []bus.ContainerStatus) {
+	reports := make([]handlers_ecs.ContainerGPUReport, 0, len(containers))
+	for _, c := range containers {
+		if len(c.GPUIDs) > 0 {
+			reports = append(reports, handlers_ecs.ContainerGPUReport{Name: c.Name, GPUIDs: c.GPUIDs})
+		}
+	}
+	if len(reports) == 0 {
+		return
+	}
+	if err := a.cp.ReportTaskGPU(a.id.ClusterName, taskID, reports); err != nil {
+		slog.Error("ecs-agent: report task-gpu failed", "task", taskID, "err", err)
+	}
 }
 
 // containerID composes a containerd-valid container ID from a task + container.

cmd/ecs-agent/taskrun_test.go

@@ -70,6 +70,160 @@ func TestRunTask_PullRunReportsRunning(t *testing.T) {
 	}
 }
 
+// TestRunTask_GPUCarriedToRunSpec covers mulga-11opz (Epic C task C1): the
+// AssignContainer's GPU count reaches the runtime RunSpec unchanged; a
+// container with no GPU request keeps RunSpec.GPU at zero (regression).
+func TestRunTask_GPUCarriedToRunSpec(t *testing.T) {
+	cp := &fakeCP{}
+	rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
+	a := newRunAgent(cp, rt)
+	as := testAssign()
+	as.Containers = append(as.Containers, bus.AssignContainer{
+		Name: "trainer", Image: "registry/trainer:1", GPU: 2,
+	})
+	a.runTask(context.Background(), as)
+
+	if len(rt.Runs) != 2 {
+		t.Fatalf("expected two runs, got %+v", rt.Runs)
+	}
+	if rt.Runs[0].GPU != 0 {
+		t.Errorf("web container: want GPU=0, got %d", rt.Runs[0].GPU)
+	}
+	if rt.Runs[1].GPU != 2 {
+		t.Errorf("trainer container: want GPU=2, got %d", rt.Runs[1].GPU)
+	}
+}
+
+// TestRunTask_GPUPinsAndReportsUUIDs covers C3: a container's GPU request is
+// pinned from the agent's local ledger, carried on the RUNNING report's
+// ContainerStatus, and forwarded to the control plane via ReportTaskGPU.
+func TestRunTask_GPUPinsAndReportsUUIDs(t *testing.T) {
+	cp := &fakeCP{}
+	rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
+	a := newAgent(config{}, testIdentityWithGPUs("GPU-aaa", "GPU-bbb"), cp, rt, rt, nil)
+	as := testAssign()
+	as.Containers = append(as.Containers, bus.AssignContainer{
+		Name: "trainer", Image: "registry/trainer:1", GPU: 1,
+	})
+	a.runTask(context.Background(), as)
+
+	st := cp.taskStates()
+	if len(st) == 0 || st[len(st)-1].LastStatus != bus.TaskStatusRunning {
+		t.Fatalf("want RUNNING final state, got %+v", st)
+	}
+	last := st[len(st)-1]
+	var trainer *bus.ContainerStatus
+	for i := range last.Containers {
+		if last.Containers[i].Name == "trainer" {
+			trainer = &last.Containers[i]
+		}
+	}
+	if trainer == nil || len(trainer.GPUIDs) != 1 {
+		t.Fatalf("trainer container missing pinned GPU UUID: %+v", last.Containers)
+	}
+	if trainer.GPUIDs[0] != "GPU-aaa" && trainer.GPUIDs[0] != "GPU-bbb" {
+		t.Errorf("unexpected pinned UUID %q", trainer.GPUIDs[0])
+	}
+
+	reports := cp.gpuReportsFor(as.TaskID)
+	if len(reports) != 1 {
+		t.Fatalf("want one ReportTaskGPU call, got %d", len(reports))
+	}
+	if reports[0].cluster != as.ClusterName {
+		t.Errorf("report cluster = %q, want %q", reports[0].cluster, as.ClusterName)
+	}
+	if len(reports[0].containers) != 1 || reports[0].containers[0].Name != "trainer" {
+		t.Errorf("report containers = %+v, want one trainer entry", reports[0].containers)
+	}
+}
+
+// TestRunTask_GPUIDsReachRunSpec covers C4: the ledger's pinned UUIDs for a
+// container reach RunSpec.GPUIDs (not just the RunSpec.GPU count), so the
+// runner can inject them as CDI devices; a non-GPU container's RunSpec
+// carries no GPUIDs.
+func TestRunTask_GPUIDsReachRunSpec(t *testing.T) {
+	cp := &fakeCP{}
+	rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
+	a := newAgent(config{}, testIdentityWithGPUs("GPU-aaa", "GPU-bbb"), cp, rt, rt, nil)
+	as := testAssign()
+	as.Containers = append(as.Containers, bus.AssignContainer{
+		Name: "trainer", Image: "registry/trainer:1", GPU: 1,
+	})
+	a.runTask(context.Background(), as)
+
+	if len(rt.Runs) != 2 {
+		t.Fatalf("expected two runs, got %+v", rt.Runs)
+	}
+	if len(rt.Runs[0].GPUIDs) != 0 {
+		t.Errorf("web container: want no GPUIDs, got %+v", rt.Runs[0].GPUIDs)
+	}
+	trainer := rt.Runs[1]
+	if len(trainer.GPUIDs) != 1 {
+		t.Fatalf("trainer container: want one pinned GPUID, got %+v", trainer.GPUIDs)
+	}
+	if trainer.GPUIDs[0] != "GPU-aaa" && trainer.GPUIDs[0] != "GPU-bbb" {
+		t.Errorf("unexpected pinned UUID %q", trainer.GPUIDs[0])
+	}
+}
+
+// A container with no GPU request never triggers a ReportTaskGPU call.
+func TestRunTask_NoGPUSkipsReport(t *testing.T) {
+	cp := &fakeCP{}
+	rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
+	a := newRunAgent(cp, rt)
+	as := testAssign()
+	a.runTask(context.Background(), as)
+
+	if reports := cp.gpuReportsFor(as.TaskID); len(reports) != 0 {
+		t.Errorf("want no GPU report for a non-GPU task, got %+v", reports)
+	}
+}
+
+// A GPU request exceeding the ledger's free devices (discovery found none, or
+// fewer than requested) is best-effort: the container still runs, it just
+// carries no pinned UUIDs and so no CDI devices are injected.
+func TestRunTask_GPUShortLedgerStillRuns(t *testing.T) {
+	cp := &fakeCP{}
+	rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
+	a := newRunAgent(cp, rt) // testIdentity() has no discovered GPU UUIDs
+	as := testAssign()
+	as.Containers = append(as.Containers, bus.AssignContainer{
+		Name: "trainer", Image: "registry/trainer:1", GPU: 2,
+	})
+	a.runTask(context.Background(), as)
+
+	if len(rt.Runs) != 2 {
+		t.Fatalf("expected both containers to run despite the short ledger, got %+v", rt.Runs)
+	}
+	if reports := cp.gpuReportsFor(as.TaskID); len(reports) != 0 {
+		t.Errorf("want no GPU report when nothing was pinned, got %+v", reports)
+	}
+}
+
+// stopTask releases the task's pinned GPU UUIDs back to the ledger so a later
+// task on the same instance can reuse them.
+func TestStopTask_ReleasesGPUsBackToLedger(t *testing.T) {
+	cp := &fakeCP{}
+	rt := &ctrruntime.FakePuller{Containers: []ctrruntime.Container{
+		{ID: "t-001-trainer", Running: true, Labels: map[string]string{
+			"mulga.ecs.taskID": "t-001", "mulga.ecs.containerName": "trainer",
+		}},
+	}}
+	a := newAgent(config{}, testIdentityWithGPUs("GPU-aaa"), cp, rt, rt, nil)
+	if _, err := a.gpu.Pin(gpuKey("t-001", "trainer"), 1); err != nil {
+		t.Fatalf("seed pin: %v", err)
+	}
+	if len(a.gpu.free) != 0 {
+		t.Fatalf("setup: want ledger fully pinned, free = %v", a.gpu.free)
+	}
+
+	a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "bye"})
+
+	if len(a.gpu.free) != 1 {
+		t.Errorf("want the GPU released back to the ledger, free = %v", a.gpu.free)
+	}
+}
+
 // A pull failure reports STOPPED and never starts the container.
 func TestRunTask_PullFailureReportsStopped(t *testing.T) {
 	cp := &fakeCP{}
@@ -172,6 +326,151 @@ func TestPollAssignments_DispatchesOnceAndAcks(t *testing.T) {
 	}
 }
 
+// stopTask reaps the task's labeled containers and reports STOPPED with the
+// directive reason and container status.
+func TestStopTask_ReapsLabeledContainersAndReportsStopped(t *testing.T) {
+	cp := &fakeCP{}
+	rt := &ctrruntime.FakePuller{Containers: []ctrruntime.Container{
+		{ID: "t-001-web", Running: true, Labels: map[string]string{
+			"mulga.ecs.taskID": "t-001", "mulga.ecs.containerName": "web",
+		}},
+		{ID: "t-999-other", Running: true, Labels: map[string]string{
+			"mulga.ecs.taskID": "t-999", "mulga.ecs.containerName": "web",
+		}},
+	}}
+	a := newRunAgent(cp, rt)
+
+	a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "bye"})
+
+	if len(rt.Removed) != 1 || rt.Removed[0] != "t-001-web" {
+		t.Fatalf("want only t-001-web removed, got %+v", rt.Removed)
+	}
+	st := cp.taskStates()
+	if len(st) != 1 || st[0].LastStatus != bus.TaskStatusStopped {
+		t.Fatalf("want one STOPPED report, got %+v", st)
+	}
+	if st[0].Reason != "bye" {
+		t.Errorf("want reason %q, got %q", "bye", st[0].Reason)
+	}
+	if len(st[0].Containers) != 1 || st[0].Containers[0].Status != bus.TaskStatusStopped {
+		t.Errorf("want one STOPPED container, got %+v", st[0].Containers)
+	}
+}
+
+// A stop for a task with no live containers still reports STOPPED so the
+// scheduler releases its capacity.
+func TestStopTask_NoContainersStillReportsStopped(t *testing.T) {
+	cp := &fakeCP{}
+	rt := &ctrruntime.FakePuller{}
+	a := newRunAgent(cp, rt)
+
+	a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "gone"})
+
+	if len(rt.Removed) != 0 {
+		t.Errorf("nothing to remove, got %+v", rt.Removed)
+	}
+	st := cp.taskStates()
+	if len(st) != 1 || st[0].LastStatus != bus.TaskStatusStopped {
+		t.Fatalf("want one STOPPED report, got %+v", st)
+	}
+}
+
+// A stop delivered by poll is dispatched once, acked in the stop-ack list, and
+// suppresses a same-poll assign for the same task.
+func TestPollAssignments_StopReapsAndSuppressesAssign(t *testing.T) {
+	as := testAssign()
+	cp := &fakeCP{
+		pollReplies: [][]bus.Assign{{*as}, {*as}, nil},
+		stopReplies: [][]bus.StopDirective{{{TaskID: as.TaskID, Reason: "bye"}}},
+	}
+	rt := &ctrruntime.FakePuller{Containers: []ctrruntime.Container{
+		{ID: "t-001-web", Running: true, Labels: map[string]string{"mulga.ecs.taskID": "t-001"}},
+	}}
+	a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, rt, nil)
+
+	ctx, cancel := context.WithCancel(context.Background())
+	go a.pollAssignments(ctx, map[string]bool{})
+
+	deadline := time.After(time.Second)
+	for {
+		if stopped := countStatus(cp.taskStates(), as.TaskID, bus.TaskStatusStopped); stopped >= 1 {
+			break
+		}
+		select {
+		case <-deadline:
+			t.Fatalf("stop never reaped; states=%+v", cp.taskStates())
+		case <-time.After(2 * time.Millisecond):
+		}
+	}
+	time.Sleep(20 * time.Millisecond)
+	cancel()
+
+	// The assign for the stopping task must never have started a container.
+	if len(rt.Runs) != 0 {
+		t.Errorf("assign for a stopping task should not run, got %+v", rt.Runs)
+	}
+	if running := countStatus(cp.taskStates(), as.TaskID, bus.TaskStatusRunning); running != 0 {
+		t.Errorf("stopping task should never report RUNNING, got %d", running)
+	}
+	acked := false
+	for _, ack := range cp.stopAcks() {
+		for _, id := range ack {
+			if id == as.TaskID {
+				acked = true
+			}
+		}
+	}
+	if !acked {
+		t.Errorf("stop %s was never acked; stopAcks=%v", as.TaskID, cp.stopAcks())
+	}
+}
+
+// stopTask with no runtime is a no-op: nothing to reap, no state reported.
+func TestStopTask_NilRunnerNoOp(t *testing.T) {
+	cp := &fakeCP{}
+	a := newAgent(config{}, testIdentity(), cp, nil, nil, nil)
+	a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "x"})
+	if len(cp.taskStates()) != 0 {
+		t.Fatalf("nil runner should report no state, got %+v", cp.taskStates())
+	}
+}
+
+// A List failure aborts the reap without reporting a (false) STOPPED state.
+func TestStopTask_ListErrorNoReport(t *testing.T) {
+	cp := &fakeCP{}
+	rt := &ctrruntime.FakePuller{ListErr: errors.New("list boom")}
+	a := newRunAgent(cp, rt)
+	a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "x"})
+	if len(cp.taskStates()) != 0 {
+		t.Fatalf("list error should report no state, got %+v", cp.taskStates())
+	}
+}
+
+// A directive with no reason reaps the matching container and reports STOPPED with
+// the default reason.
+func TestStopTask_EmptyReasonReapsWithDefault(t *testing.T) {
+	cp := &fakeCP{}
+	rt := &ctrruntime.FakePuller{Containers: []ctrruntime.Container{{
+		ID: "t-001-web",
+		Labels: map[string]string{
+			"mulga.ecs.taskID": "t-001", "mulga.ecs.containerName": "web",
+		},
+	}}}
+	a := newRunAgent(cp, rt)
+	a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001"})
+
+	if len(rt.Removed) != 1 || rt.Removed[0] != "t-001-web" {
+		t.Fatalf("want container t-001-web reaped, got %+v", rt.Removed)
+	}
+	st := cp.taskStates()
+	if len(st) != 1 || st[0].LastStatus != bus.TaskStatusStopped {
+		t.Fatalf("want one STOPPED report, got %+v", st)
+	}
+	if st[0].Reason != "Task stopped" {
+		t.Errorf("want default reason %q, got %q", "Task stopped", st[0].Reason)
+	}
+}
+
 func countStatus(states []bus.TaskState, taskID, status string) int {
 	n := 0
 	for _, st := range states {
@@ -227,3 +526,32 @@ func TestContainerID(t *testing.T) {
 		t.Errorf("containerID = %q, want t-1-web", got)
 	}
 }
+
+// sentinelResolver is an identity-comparable Resolver for pullResolver tests.
+type sentinelResolver struct{}
+
+func (sentinelResolver) Authorize(context.Context, string) (string, string, string, error) {
+	return "", "", "", nil
+}
+
+// TestPullResolver covers siv-459: an assign with no execution role (or no
+// credential endpoint) uses the instance resolver; with both, a distinct
+// exec-role resolver is built.
+func TestPullResolver(t *testing.T) {
+	fake := sentinelResolver{}
+	a := newAgent(config{Region: "us-east-1"}, testIdentity(), &fakeCP{}, nil, nil, fake)
+
+	if got := a.pullResolver(&bus.Assign{TaskID: "t1"}); got != fake {
+		t.Fatalf("no execution role: want instance resolver fallback")
+	}
+	execAssign := &bus.Assign{TaskID: "t1", ExecutionRoleARN: "arn:aws:iam::1:role/e"}
+	if got := a.pullResolver(execAssign); got != fake {
+		t.Fatalf("nil cred endpoint: want instance resolver fallback")
+	}
+
+	a.cred = &credEndpoint{}
+	got := a.pullResolver(execAssign)
+	if got == nil || got == fake {
+		t.Fatalf("execution role + cred endpoint: want distinct exec-role resolver, got %v", got)
+	}
+}

cmd/eks-gateway-fetch/main.go

@@ -17,10 +17,14 @@
 // Usage:
 //
 //	eks-gateway-fetch -resource addons   # GET /clusters/{cluster}/internal-addons/{accountId}
+//	eks-gateway-fetch -resource recovery -instance-id i-… # GET /clusters/{cluster}/internal-recovery/{accountId}/{instanceId}
+//
+// The recovery resource returns a single epoch\taction\tsnapshot line the on-VM
+// k3s-recovery agent applies before k3s starts (etcd cluster-reset / wipe-rejoin).
 //
 // Flags default to environment variables seeded by cloud-init:
 // EKS_GATEWAY_URL, EKS_GATEWAY_CA, EKS_ACCESS_KEY, EKS_SECRET_KEY, EKS_REGION,
-// EKS_ACCOUNT_ID, EKS_CLUSTER_NAME.
+// EKS_ACCOUNT_ID, EKS_CLUSTER_NAME, EKS_INSTANCE_ID.
 package main
 
 import (
@@ -68,6 +72,7 @@ func main() {
 		accountID   string
 		clusterName string
 		resource    string
+		instanceID  string
 	)
 
 	flag.StringVar(&gatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
@@ -77,7 +82,8 @@ func main() {
 	flag.StringVar(&region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
 	flag.StringVar(&accountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
 	flag.StringVar(&clusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
-	flag.StringVar(&resource, "resource", "addons", "Resource to fetch: addons")
+	flag.StringVar(&instanceID, "instance-id", os.Getenv("EKS_INSTANCE_ID"), "This member's EC2 instance ID (recovery resource)")
+	flag.StringVar(&resource, "resource", "addons", "Resource to fetch: addons | recovery")
 	flag.Parse()
 
 	switch {
@@ -85,15 +91,29 @@ func main() {
 		fatal("--account-id is required (or set EKS_ACCOUNT_ID)")
 	case clusterName == "":
 		fatal("--cluster is required (or set EKS_CLUSTER_NAME)")
-	case resource != "addons":
-		fatal("--resource must be addons")
+	case resource != "addons" && resource != "recovery":
+		fatal("--resource must be addons or recovery")
+	case resource == "recovery" && instanceID == "":
+		fatal("--instance-id is required for --resource recovery (or set EKS_INSTANCE_ID)")
 	}
 
 	client, err := eksgw.New(gatewayURL, gatewayCA, accessKey, secretKey, region)
 	if err != nil {
 		fatal(fmt.Sprintf("build gateway client: %v", err))
 	}
-	path := "/clusters/" + clusterName + "/internal-addons/" + accountID
+
+	var (
+		path string
+		emit func(io.Writer, []byte) error
+	)
+	switch resource {
+	case "recovery":
+		path = "/clusters/" + clusterName + "/internal-recovery/" + accountID + "/" + instanceID
+		emit = emitRecoveryTSV
+	default:
+		path = "/clusters/" + clusterName + "/internal-addons/" + accountID
+		emit = emitAddonsTSV
+	}
 
 	var lastErr error
 	for attempt := 1; attempt <= maxAttempts; attempt++ {
@@ -106,7 +126,7 @@ func main() {
 			}
 			continue
 		}
-		if werr := emitAddonsTSV(os.Stdout, body); werr != nil {
+		if werr := emit(os.Stdout, body); werr != nil {
 			fatal(fmt.Sprintf("render response: %v", werr))
 		}
 		return
@@ -133,6 +153,43 @@ func emitAddonsTSV(out io.Writer, body []byte) error {
 	return w.Flush()
 }
 
+// recoveryDirective mirrors the gateway's RecoveryDirective wire shape; duplicated
+// here to keep this tiny VM binary free of the handler package's dependency graph.
+type recoveryDirective struct {
+	Epoch            int64  `json:"epoch"`
+	Action           string `json:"action"`
+	Snapshot         string `json:"snapshot"`
+	SnapshotRequired bool   `json:"snapshotRequired"`
+}
+
+type internalRecoveryResponse struct {
+	Directive recoveryDirective `json:"directive"`
+}
+
+// emitRecoveryTSV decodes the recovery directive and writes a single
+// epoch\taction\tsnapshot\tsnapshotRequired line the on-VM k3s-recovery agent
+// consumes without a JSON parser. An unset action defaults to "none" (steady
+// state); snapshotRequired is 1 when the snapshot MUST restore (fresh DR seed
+// aborts boot on fetch failure) else 0.
+func emitRecoveryTSV(out io.Writer, body []byte) error {
+	var resp internalRecoveryResponse
+	if err := json.Unmarshal(body, &resp); err != nil {
+		return fmt.Errorf("unmarshal internal-recovery response: %w", err)
+	}
+	d := resp.Directive
+	if d.Action == "" {
+		d.Action = "none"
+	}
+	required := "0"
+	if d.SnapshotRequired {
+		required = "1"
+	}
+	if _, err := fmt.Fprintf(out, "%d\t%s\t%s\t%s\n", d.Epoch, d.Action, d.Snapshot, required); err != nil {
+		return err
+	}
+	return nil
+}
+
 func fatal(msg string) {
 	slog.Error("eks-gateway-fetch: " + msg)
 	os.Exit(1)

cmd/eks-konnectivity-cert/main.go

@@ -0,0 +1,245 @@
+// Command eks-konnectivity-cert mints, then reuses, an ECDSA-P256 serving
+// certificate for the on-node konnectivity-server, signed by the K3s cluster CA.
+//
+// The konnectivity-agent (DaemonSet on workers) verifies the server it dials
+// using the in-pod ServiceAccount CA (/var/run/secrets/.../ca.crt), which in
+// K3s is the same server-ca that signs the apiserver. Signing the konnectivity
+// serving cert with that CA is therefore what lets the agent validate the
+// reverse-tunnel endpoint with no separate CA distribution.
+//
+// The eks-node AMI carries no openssl, so this is the Go equivalent: load the
+// K3s server CA cert+key, issue a leaf (ServerAuth) cert carrying the cluster's
+// konnectivity SANs (the NLB private-endpoint IP, public endpoint IP, NLB DNS),
+// and cache it under -dir so the OpenRC service is byte-stable across restarts.
+//
+// Usage:
+//
+//	eks-konnectivity-cert -dir /var/lib/spinifex-eks/konnectivity \
+//	    -ca-cert /var/lib/rancher/k3s/server/tls/server-ca.crt \
+//	    -ca-key  /var/lib/rancher/k3s/server/tls/server-ca.key \
+//	    -cn konnectivity-server -sans 10.32.100.4,203.0.113.9,eks-toc.example
+//
+// Writes <dir>/tls.crt and <dir>/tls.key; prints the two paths (tab-separated).
+package main
+
+import (
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/pem"
+	"flag"
+	"fmt"
+	"log/slog"
+	"math/big"
+	"net"
+	"os"
+	"path/filepath"
+	"strings"
+	"time"
+)
+
+func main() {
+	dir := flag.String("dir", "", "directory to cache tls.crt/tls.key in")
+	caCert := flag.String("ca-cert", "", "PEM path of the signing (K3s server) CA cert")
+	caKey := flag.String("ca-key", "", "PEM path of the signing CA private key")
+	cn := flag.String("cn", "konnectivity-server", "certificate common name")
+	sansCSV := flag.String("sans", "", "comma-separated SANs (IP or DNS)")
+	flag.Parse()
+
+	if *dir == "" || *caCert == "" || *caKey == "" || *sansCSV == "" {
+		fatal("eks-konnectivity-cert: -dir, -ca-cert, -ca-key and -sans are required")
+	}
+	dns, ips := splitSANs(*sansCSV)
+	if len(dns) == 0 && len(ips) == 0 {
+		fatal("eks-konnectivity-cert: -sans lists no usable names")
+	}
+
+	certPath, keyPath, err := ensureCert(*dir, *caCert, *caKey, *cn, dns, ips)
+	if err != nil {
+		fatal("eks-konnectivity-cert: " + err.Error())
+	}
+	fmt.Printf("%s\t%s\n", certPath, keyPath)
+}
+
+// ensureCert returns the cached cert/key paths under dir, minting and persisting
+// a fresh CA-signed pair when none is present, readable, and still SAN-current.
+func ensureCert(dir, caCertPath, caKeyPath, cn string, dns []string, ips []net.IP) (string, string, error) {
+	certPath := filepath.Join(dir, "tls.crt")
+	keyPath := filepath.Join(dir, "tls.key")
+
+	if certCoversSANs(certPath, keyPath, dns, ips) {
+		return certPath, keyPath, nil
+	}
+
+	caCert, caKey, err := loadCA(caCertPath, caKeyPath)
+	if err != nil {
+		return "", "", err
+	}
+	certPEM, keyPEM, err := issueLeaf(caCert, caKey, cn, dns, ips)
+	if err != nil {
+		return "", "", err
+	}
+	if err = os.MkdirAll(dir, 0o700); err != nil {
+		return "", "", fmt.Errorf("mkdir %s: %w", dir, err)
+	}
+	if err = os.WriteFile(certPath, certPEM, 0o644); err != nil {
+		return "", "", fmt.Errorf("write %s: %w", certPath, err)
+	}
+	if err = os.WriteFile(keyPath, keyPEM, 0o600); err != nil {
+		return "", "", fmt.Errorf("write %s: %w", keyPath, err)
+	}
+	return certPath, keyPath, nil
+}
+
+// certCoversSANs reports whether the cached cert exists, parses, has a key, and
+// already carries every requested SAN — so a cert minted before the endpoint
+// set changed is re-issued rather than served stale.
+func certCoversSANs(certPath, keyPath string, dns []string, ips []net.IP) bool {
+	c, err := os.ReadFile(certPath)
+	if err != nil {
+		return false
+	}
+	if k, kerr := os.ReadFile(keyPath); kerr != nil || len(k) == 0 {
+		return false
+	}
+	der := decodeFirst(c)
+	if der == nil {
+		return false
+	}
+	crt, err := x509.ParseCertificate(der)
+	if err != nil {
+		return false
+	}
+	have := make(map[string]struct{}, len(crt.DNSNames)+len(crt.IPAddresses))
+	for _, n := range crt.DNSNames {
+		have[n] = struct{}{}
+	}
+	for _, ip := range crt.IPAddresses {
+		have[ip.String()] = struct{}{}
+	}
+	for _, n := range dns {
+		if _, ok := have[n]; !ok {
+			return false
+		}
+	}
+	for _, ip := range ips {
+		if _, ok := have[ip.String()]; !ok {
+			return false
+		}
+	}
+	return true
+}
+
+// loadCA parses the PEM CA cert + EC/PKCS8 private key used to sign the leaf.
+func loadCA(certPath, keyPath string) (*x509.Certificate, *ecdsa.PrivateKey, error) {
+	certPEM, err := os.ReadFile(certPath)
+	if err != nil {
+		return nil, nil, fmt.Errorf("read ca cert %s: %w", certPath, err)
+	}
+	der := decodeFirst(certPEM)
+	if der == nil {
+		return nil, nil, fmt.Errorf("ca cert %s: no PEM block", certPath)
+	}
+	caCert, err := x509.ParseCertificate(der)
+	if err != nil {
+		return nil, nil, fmt.Errorf("parse ca cert: %w", err)
+	}
+
+	keyPEM, err := os.ReadFile(keyPath)
+	if err != nil {
+		return nil, nil, fmt.Errorf("read ca key %s: %w", keyPath, err)
+	}
+	keyDER := decodeFirst(keyPEM)
+	if keyDER == nil {
+		return nil, nil, fmt.Errorf("ca key %s: no PEM block", keyPath)
+	}
+	caKey, err := parseECKey(keyDER)
+	if err != nil {
+		return nil, nil, err
+	}
+	return caCert, caKey, nil
+}
+
+// parseECKey accepts the EC SEC1 or PKCS8 encodings K3s may write for the CA key.
+func parseECKey(der []byte) (*ecdsa.PrivateKey, error) {
+	if k, err := x509.ParseECPrivateKey(der); err == nil {
+		return k, nil
+	}
+	k8, err := x509.ParsePKCS8PrivateKey(der)
+	if err != nil {
+		return nil, fmt.Errorf("parse ca key (sec1 and pkcs8 failed): %w", err)
+	}
+	ec, ok := k8.(*ecdsa.PrivateKey)
+	if !ok {
+		return nil, fmt.Errorf("ca key is %T, want ecdsa", k8)
+	}
+	return ec, nil
+}
+
+// issueLeaf mints an ECDSA-P256 ServerAuth leaf for cn + SANs, signed by caCert/caKey.
+func issueLeaf(caCert *x509.Certificate, caKey *ecdsa.PrivateKey, cn string, dns []string, ips []net.IP) (certPEM, keyPEM []byte, err error) {
+	priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		return nil, nil, fmt.Errorf("generate key: %w", err)
+	}
+	serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
+	if err != nil {
+		return nil, nil, fmt.Errorf("serial: %w", err)
+	}
+	now := time.Now().UTC()
+	tmpl := x509.Certificate{
+		SerialNumber:          serial,
+		Subject:               pkix.Name{CommonName: cn},
+		NotBefore:             now.Add(-time.Hour),
+		NotAfter:              now.AddDate(10, 0, 0),
+		KeyUsage:              x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment,
+		ExtKeyUsage:           []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+		BasicConstraintsValid: true,
+		DNSNames:              dns,
+		IPAddresses:           ips,
+	}
+	der, err := x509.CreateCertificate(rand.Reader, &tmpl, caCert, &priv.PublicKey, caKey)
+	if err != nil {
+		return nil, nil, fmt.Errorf("create cert: %w", err)
+	}
+	keyDER, err := x509.MarshalECPrivateKey(priv)
+	if err != nil {
+		return nil, nil, fmt.Errorf("marshal key: %w", err)
+	}
+	certPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
+	keyPEM = pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
+	return certPEM, keyPEM, nil
+}
+
+// splitSANs partitions the CSV into DNS names and IPs (an entry that parses as an
+// IP goes to IPAddresses, else DNSNames), matching how SAN matching is typed.
+func splitSANs(s string) (dns []string, ips []net.IP) {
+	for p := range strings.SplitSeq(s, ",") {
+		p = strings.TrimSpace(p)
+		if p == "" {
+			continue
+		}
+		if ip := net.ParseIP(p); ip != nil {
+			ips = append(ips, ip)
+			continue
+		}
+		dns = append(dns, p)
+	}
+	return dns, ips
+}
+
+// decodeFirst returns the DER bytes of the first PEM block, or nil.
+func decodeFirst(pemBytes []byte) []byte {
+	blk, _ := pem.Decode(pemBytes)
+	if blk == nil {
+		return nil
+	}
+	return blk.Bytes
+}
+
+func fatal(msg string) {
+	slog.Error(msg)
+	os.Exit(1)
+}

cmd/eks-konnectivity-cert/main_test.go

@@ -0,0 +1,153 @@
+package main
+
+import (
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/pem"
+	"math/big"
+	"net"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+)
+
+// writeTestCA mints a self-signed EC CA (cert + SEC1 key PEM) into dir, returning
+// the two paths — a stand-in for K3s' /var/lib/rancher/k3s/server/tls/server-ca.*.
+func writeTestCA(t *testing.T, dir string) (certPath, keyPath string) {
+	t.Helper()
+	priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		t.Fatalf("gen ca key: %v", err)
+	}
+	tmpl := x509.Certificate{
+		SerialNumber:          big.NewInt(1),
+		Subject:               pkix.Name{CommonName: "k3s-server-ca"},
+		NotBefore:             time.Now().Add(-time.Hour),
+		NotAfter:              time.Now().AddDate(10, 0, 0),
+		KeyUsage:              x509.KeyUsageCertSign | x509.KeyUsageDigitalSignature,
+		BasicConstraintsValid: true,
+		IsCA:                  true,
+	}
+	der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &priv.PublicKey, priv)
+	if err != nil {
+		t.Fatalf("create ca cert: %v", err)
+	}
+	keyDER, err := x509.MarshalECPrivateKey(priv)
+	if err != nil {
+		t.Fatalf("marshal ca key: %v", err)
+	}
+	certPath = filepath.Join(dir, "server-ca.crt")
+	keyPath = filepath.Join(dir, "server-ca.key")
+	if err := os.WriteFile(certPath, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}), 0o644); err != nil {
+		t.Fatalf("write ca cert: %v", err)
+	}
+	if err := os.WriteFile(keyPath, pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER}), 0o600); err != nil {
+		t.Fatalf("write ca key: %v", err)
+	}
+	return certPath, keyPath
+}
+
+func leafFrom(t *testing.T, certPath string) *x509.Certificate {
+	t.Helper()
+	b, err := os.ReadFile(certPath)
+	if err != nil {
+		t.Fatalf("read leaf: %v", err)
+	}
+	crt, err := x509.ParseCertificate(decodeFirst(b))
+	if err != nil {
+		t.Fatalf("parse leaf: %v", err)
+	}
+	return crt
+}
+
+func TestEnsureCertSignsLeafWithSANsThatChainsToCA(t *testing.T) {
+	dir := t.TempDir()
+	caCert, caKey := writeTestCA(t, dir)
+
+	dns, ips := splitSANs("10.32.100.4,203.0.113.9,eks-toc.example")
+	certPath, keyPath, err := ensureCert(dir, caCert, caKey, "konnectivity-server", dns, ips)
+	if err != nil {
+		t.Fatalf("ensureCert: %v", err)
+	}
+	if _, err := os.Stat(keyPath); err != nil {
+		t.Fatalf("key not written: %v", err)
+	}
+
+	leaf := leafFrom(t, certPath)
+	if leaf.IsCA {
+		t.Error("konnectivity serving cert must be a leaf, not a CA")
+	}
+	// The agent dials the NLB private-endpoint IP and verifies via the in-pod CA,
+	// so the IP must be a SAN or the tunnel TLS handshake fails.
+	if err := leaf.VerifyHostname("10.32.100.4"); err != nil {
+		t.Errorf("IP SAN 10.32.100.4 missing: %v", err)
+	}
+
+	// The leaf must chain to the K3s server CA — that CA is the in-pod ca.crt the
+	// agent trusts, so no separate CA distribution is needed.
+	caPEM, _ := os.ReadFile(caCert)
+	roots := x509.NewCertPool()
+	if !roots.AppendCertsFromPEM(caPEM) {
+		t.Fatal("append CA failed")
+	}
+	if _, err := leaf.Verify(x509.VerifyOptions{
+		Roots:     roots,
+		DNSName:   "eks-toc.example",
+		KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+	}); err != nil {
+		t.Errorf("leaf does not chain to the K3s server CA: %v", err)
+	}
+}
+
+func TestEnsureCertReusesThenReissuesOnSANChange(t *testing.T) {
+	dir := t.TempDir()
+	caCert, caKey := writeTestCA(t, dir)
+
+	dns, ips := splitSANs("10.32.100.4,eks-toc.example")
+	c1, _, err := ensureCert(dir, caCert, caKey, "konnectivity-server", dns, ips)
+	if err != nil {
+		t.Fatalf("first ensureCert: %v", err)
+	}
+	first, _ := os.ReadFile(c1)
+
+	// Same SANs → cached pair reused byte-for-byte (a churning cert would bounce
+	// the konnectivity-server on every restart).
+	if _, _, err := ensureCert(dir, caCert, caKey, "konnectivity-server", dns, ips); err != nil {
+		t.Fatalf("second ensureCert: %v", err)
+	}
+	if second, _ := os.ReadFile(c1); string(first) != string(second) {
+		t.Error("ensureCert re-minted instead of reusing the SAN-current cached cert")
+	}
+
+	// A new endpoint in the SAN set must force a re-issue.
+	dns2, ips2 := splitSANs("10.32.100.4,eks-toc.example,10.32.100.99")
+	if _, _, err := ensureCert(dir, caCert, caKey, "konnectivity-server", dns2, ips2); err != nil {
+		t.Fatalf("third ensureCert: %v", err)
+	}
+	if third, _ := os.ReadFile(c1); string(first) == string(third) {
+		t.Error("ensureCert served a stale cert that does not cover the new SAN")
+	}
+	if err := leafFrom(t, c1).VerifyHostname("10.32.100.99"); err != nil {
+		t.Errorf("re-issued cert missing new SAN: %v", err)
+	}
+}
+
+func TestSplitSANsPartitionsIPsAndDNS(t *testing.T) {
+	dns, ips := splitSANs(" 10.0.0.1 , host.example , , 2001:db8::1 ")
+	if len(dns) != 1 || dns[0] != "host.example" {
+		t.Errorf("dns = %v, want [host.example]", dns)
+	}
+	wantIPs := []net.IP{net.ParseIP("10.0.0.1"), net.ParseIP("2001:db8::1")}
+	if len(ips) != len(wantIPs) {
+		t.Fatalf("ips = %v, want %v", ips, wantIPs)
+	}
+	for i, ip := range ips {
+		if !ip.Equal(wantIPs[i]) {
+			t.Errorf("ips[%d] = %v, want %v", i, ip, wantIPs[i])
+		}
+	}
+}

cmd/eks-token-webhook/cert.go

@@ -18,14 +18,10 @@ import (
 	"github.com/mulgadc/spinifex/internal/tlsconfig"
 )
 
-// tlsCertificate is the webhook's loopback serving certificate.
-type tlsCertificate struct {
-	cert tls.Certificate
-}
-
-func (c tlsCertificate) serverTLSConfig() *tls.Config {
+// serverTLSConfig is the webhook's loopback serving TLS config.
+func serverTLSConfig(cert tls.Certificate) *tls.Config {
 	return &tls.Config{
-		Certificates:     []tls.Certificate{c.cert},
+		Certificates:     []tls.Certificate{cert},
 		MinVersion:       tls.VersionTLS13,
 		CurvePreferences: tlsconfig.Curves,
 	}
@@ -36,30 +32,30 @@ func (c tlsCertificate) serverTLSConfig() *tls.Config {
 // across restarts keeps the CA the apiserver loaded into its webhook kubeconfig
 // valid even if the webhook process is restarted. Returns the parsed cert and
 // the certificate PEM (for embedding as the kubeconfig CA bundle).
-func ensureServingCert(certPath, keyPath string) (tlsCertificate, []byte, error) {
+func ensureServingCert(certPath, keyPath string) (tls.Certificate, []byte, error) {
 	if certPEM, keyPEM, ok := readCertPair(certPath, keyPath); ok {
 		cert, err := tls.X509KeyPair(certPEM, keyPEM)
 		if err == nil {
-			return tlsCertificate{cert: cert}, certPEM, nil
+			return cert, certPEM, nil
 		}
 		// Fall through to regenerate on a corrupt/incompatible persisted pair.
 	}
 
 	certPEM, keyPEM, err := generateSelfSigned()
 	if err != nil {
-		return tlsCertificate{}, nil, err
+		return tls.Certificate{}, nil, err
 	}
 	if err := os.WriteFile(certPath, certPEM, 0o644); err != nil {
-		return tlsCertificate{}, nil, fmt.Errorf("write cert %s: %w", certPath, err)
+		return tls.Certificate{}, nil, fmt.Errorf("write cert %s: %w", certPath, err)
 	}
 	if err := os.WriteFile(keyPath, keyPEM, 0o600); err != nil {
-		return tlsCertificate{}, nil, fmt.Errorf("write key %s: %w", keyPath, err)
+		return tls.Certificate{}, nil, fmt.Errorf("write key %s: %w", keyPath, err)
 	}
 	cert, err := tls.X509KeyPair(certPEM, keyPEM)
 	if err != nil {
-		return tlsCertificate{}, nil, fmt.Errorf("parse minted keypair: %w", err)
+		return tls.Certificate{}, nil, fmt.Errorf("parse minted keypair: %w", err)
 	}
-	return tlsCertificate{cert: cert}, certPEM, nil
+	return cert, certPEM, nil
 }
 
 func readCertPair(certPath, keyPath string) (certPEM, keyPEM []byte, ok bool) {

cmd/eks-token-webhook/main.go

@@ -10,6 +10,9 @@
 // On startup it writes the apiserver webhook kubeconfig (with its self-signed
 // serving CA) so k3s can be pointed at it via
 // --authentication-token-webhook-config-file.
+//
+// SigV4 creds come from the AWS SDK chain (IMDS instance-role) when static
+// EKS_ACCESS_KEY/EKS_SECRET_KEY are absent, matching the sibling helpers.
 package main
 
 import (
@@ -56,11 +59,13 @@ func loadConfig() (config, error) {
 		keyPath:     envOr("EKS_WEBHOOK_KEY", "/etc/spinifex-eks/token-webhook.key"),
 		kubeconfig:  envOr("EKS_WEBHOOK_KUBECONFIG", "/etc/spinifex-eks/token-webhook.kubeconfig"),
 	}
+	// Static SigV4 creds are optional: when absent, eksgw.New signs with the
+	// AWS SDK chain (IMDS instance-role creds), the same path the CP VM's
+	// sibling helpers use. The CP VM launches on an instance profile, so
+	// buildK3sUserData omits EKS_ACCESS_KEY/EKS_SECRET_KEY by design.
 	switch {
 	case c.gatewayURL == "":
 		return c, errors.New("EKS_GATEWAY_URL not set")
-	case c.accessKey == "" || c.secretKey == "":
-		return c, errors.New("EKS_ACCESS_KEY / EKS_SECRET_KEY not set")
 	case c.accountID == "":
 		return c, errors.New("EKS_ACCOUNT_ID not set")
 	case c.clusterName == "":
@@ -125,7 +130,12 @@ func run(cfg config) error {
 	})
 	mux.HandleFunc("/authenticate", authr.handle)
 
-	server := newServer(cfg.addr, mux, tlsCert)
+	server := &http.Server{
+		Addr:              cfg.addr,
+		Handler:           mux,
+		ReadHeaderTimeout: 5 * time.Second,
+		TLSConfig:         serverTLSConfig(tlsCert),
+	}
 	slog.Info("eks-token-webhook listening", "addr", cfg.addr, "cluster", cfg.clusterName)
 	if err := server.ListenAndServeTLS("", ""); err != nil && !errors.Is(err, http.ErrServerClosed) {
 		return fmt.Errorf("serve: %w", err)
@@ -214,14 +224,3 @@ func (a *authenticator) handle(w http.ResponseWriter, r *http.Request) {
 		slog.Error("encode TokenReview response", "err", err)
 	}
 }
-
-// newServer builds the loopback TLS server. Split out so tests can construct it
-// without binding a port.
-func newServer(addr string, handler http.Handler, cert tlsCertificate) *http.Server {
-	return &http.Server{
-		Addr:              addr,
-		Handler:           handler,
-		ReadHeaderTimeout: 5 * time.Second,
-		TLSConfig:         cert.serverTLSConfig(),
-	}
-}

cmd/eks-token-webhook/main_test.go

@@ -99,7 +99,7 @@ func TestEnsureServingCert_PersistsAndReuses(t *testing.T) {
 	c1, pem1, err := ensureServingCert(certPath, keyPath)
 	require.NoError(t, err)
 	require.NotEmpty(t, pem1)
-	require.NotNil(t, c1.serverTLSConfig())
+	require.NotNil(t, serverTLSConfig(c1))
 
 	// Second call reuses the persisted pair (same cert bytes).
 	_, pem2, err := ensureServingCert(certPath, keyPath)
@@ -121,3 +121,41 @@ func TestWriteAPIServerKubeconfig(t *testing.T) {
 	assert.Contains(t, body, base64.StdEncoding.EncodeToString(certPEM))
 	assert.Contains(t, body, "current-context: webhook")
 }
+
+// The CP VM launches on an instance profile, so cloud-init omits the static
+// EKS_ACCESS_KEY/EKS_SECRET_KEY and eksgw.New signs with IMDS creds. loadConfig
+// must accept absent static keys; gating on them aborts every IMDS-mode boot,
+// leaving the apiserver webhook kubeconfig unwritten.
+func TestLoadConfig_AcceptsAbsentStaticCreds(t *testing.T) {
+	setEnv := func(t *testing.T, kv map[string]string) {
+		t.Helper()
+		for _, k := range []string{
+			"EKS_GATEWAY_URL", "EKS_ACCESS_KEY", "EKS_SECRET_KEY",
+			"EKS_ACCOUNT_ID", "EKS_CLUSTER_NAME",
+		} {
+			t.Setenv(k, "")
+		}
+		for k, v := range kv {
+			t.Setenv(k, v)
+		}
+	}
+
+	// IMDS mode: required vars set, static creds absent → accepted.
+	setEnv(t, map[string]string{
+		"EKS_GATEWAY_URL":  "https://gw:9999",
+		"EKS_ACCOUNT_ID":   "000000000001",
+		"EKS_CLUSTER_NAME": "toc",
+	})
+	cfg, err := loadConfig()
+	require.NoError(t, err)
+	assert.Empty(t, cfg.accessKey)
+	assert.Empty(t, cfg.secretKey)
+
+	// Genuinely-required vars are still enforced.
+	setEnv(t, map[string]string{
+		"EKS_ACCOUNT_ID":   "000000000001",
+		"EKS_CLUSTER_NAME": "toc",
+	})
+	_, err = loadConfig()
+	require.Error(t, err)
+}

cmd/installer/autoinstall/autoinstall.go

@@ -1,20 +1,3 @@
-/*
-Copyright © 2026 Mulga Defense Corporation
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
-
-You should have received a copy of the GNU Affero General Public License
-along with this program.  If not, see <https://www.gnu.org/licenses/>.
-*/
-
 // Package autoinstall reads headless install parameters from environment
 // variables exported by spinifex-init.sh from the kernel cmdline.
 // Parameters are set in the GRUB "Headless" menu entry in grub.cfg.

cmd/installer/branding/branding.go

@@ -1,20 +1,3 @@
-/*
-Copyright © 2026 Mulga Defense Corporation
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
-
-You should have received a copy of the GNU Affero General Public License
-along with this program.  If not, see <https://www.gnu.org/licenses/>.
-*/
-
 // Package branding centralises Mulga/Spinifex visual identity constants used
 // by the installer TUI.
 //

cmd/installer/firstboot/firstboot.go

@@ -1,20 +1,3 @@
-/*
-Copyright © 2026 Mulga Defense Corporation
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
-
-You should have received a copy of the GNU Affero General Public License
-along with this program.  If not, see <https://www.gnu.org/licenses/>.
-*/
-
 // Package firstboot writes the oneshot systemd service and configuration that
 // completes Spinifex provisioning on the first real boot after installation.
 package firstboot
@@ -91,6 +74,31 @@ func writeScript(root string, cfg Config) error {
 		setupOVN += fmt.Sprintf(" --encap-ip=%s", cfg.EncapIP)
 	}
 
+	// Pre-start OVS and OVN central so their databases are initialised before
+	// setup-ovn.sh runs. On physical hardware, first-boot DB initialisation takes
+	// longer than setup-ovn.sh's internal 15-second timeout allows. Starting them
+	// here and waiting until the NB DB is ready means setup-ovn.sh sees a live DB
+	// the moment it starts — no races, no timeout failures.
+	ovnPrestart := `systemctl start openvswitch-switch
+systemctl start ovn-central
+echo "Waiting for OVN NB DB to initialise..."
+for _i in $(seq 1 120); do
+    if ovn-nbctl --timeout=2 get-connection >/dev/null 2>&1; then
+        echo "OVN NB DB ready (${_i}s)"
+        break
+    fi
+    sleep 1
+done`
+
+	// When a provisioning controller owns formation (SkipFormation) it also owns
+	// the clustered OVN bring-up via setup-ovn.sh's RAFT flags. firstboot must
+	// not start a standalone ovn-central here: the single-node .db it would leave
+	// blocks ovn-ctl's create/join (which require a clean DB). Defer both.
+	if cfg.SkipFormation {
+		ovnPrestart = `echo "[firstboot] OVN bring-up deferred to provisioning controller"`
+		setupOVN = `echo "[firstboot] setup-ovn deferred to provisioning controller"`
+	}
+
 	callbackBlock := ""
 	if cfg.InstallCallback != "" {
 		callbackBlock = fmt.Sprintf(
@@ -123,21 +131,7 @@ fi
 # Set hostname
 hostnamectl set-hostname %s
 
-# Pre-start OVS and OVN central so their databases are initialised before
-# setup-ovn.sh runs. On physical hardware, first-boot DB initialisation takes
-# longer than setup-ovn.sh's internal 15-second timeout allows. Starting them
-# here and waiting until the NB DB is ready means setup-ovn.sh sees a live DB
-# the moment it starts — no races, no timeout failures.
-systemctl start openvswitch-switch
-systemctl start ovn-central
-echo "Waiting for OVN NB DB to initialise..."
-for _i in $(seq 1 120); do
-    if ovn-nbctl --timeout=2 get-connection >/dev/null 2>&1; then
-        echo "OVN NB DB ready (${_i}s)"
-        break
-    fi
-    sleep 1
-done
+%s
 
 # Create default nameservers if DHCP server returns blank
 printf "nameserver 1.1.1.1\nnameserver 8.8.8.8\n" > /etc/resolvconf/resolv.conf.d/base
@@ -229,7 +223,7 @@ if grep -q 'external_mode.*pool' /etc/spinifex/spinifex.toml 2>/dev/null; then
         echo "[firstboot] warning: br-ext not up after 30s — external networking may be delayed"
     fi
 fi
-`, cfg.Hostname, setupOVN, clusterCmd, callbackBlock)
+`, cfg.Hostname, ovnPrestart, setupOVN, clusterCmd, callbackBlock)
 
 	path := filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh")
 	return os.WriteFile(path, []byte(script), 0o755)

cmd/installer/firstboot/firstboot_test.go

@@ -66,6 +66,54 @@ func TestWriteScriptEmbedsCurlWhenCallbackSet(t *testing.T) {
 	}
 }
 
+func TestWriteScriptRunsOVNWhenFormationOwned(t *testing.T) {
+	root := t.TempDir()
+	makeRootDirs(t, root)
+
+	cfg := Config{Hostname: "test-node", ClusterRole: "init"}
+	if err := Write(root, cfg); err != nil {
+		t.Fatalf("Write: %v", err)
+	}
+	content := readScript(t, root)
+
+	if !strings.Contains(content, "setup-ovn.sh --management") {
+		t.Error("script should run setup-ovn --management when firstboot owns formation")
+	}
+	if !strings.Contains(content, "systemctl start ovn-central") {
+		t.Error("script should pre-start ovn-central when firstboot owns formation")
+	}
+}
+
+func TestWriteScriptDefersOVNWhenSkipFormation(t *testing.T) {
+	root := t.TempDir()
+	makeRootDirs(t, root)
+
+	cfg := Config{Hostname: "test-node", ClusterRole: "init", SkipFormation: true}
+	if err := Write(root, cfg); err != nil {
+		t.Fatalf("Write: %v", err)
+	}
+	content := readScript(t, root)
+
+	if strings.Contains(content, "setup-ovn.sh --management") {
+		t.Error("script must not run setup-ovn --management when a controller owns OVN")
+	}
+	if strings.Contains(content, "systemctl start ovn-central") {
+		t.Error("script must not pre-start ovn-central when a controller owns OVN")
+	}
+	if !strings.Contains(content, "setup-ovn deferred") {
+		t.Error("script should note setup-ovn is deferred under SkipFormation")
+	}
+}
+
+func readScript(t *testing.T, root string) string {
+	t.Helper()
+	b, err := os.ReadFile(filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh"))
+	if err != nil {
+		t.Fatalf("read script: %v", err)
+	}
+	return string(b)
+}
+
 func TestWriteScriptCallbackAfterDoneMarker(t *testing.T) {
 	root := t.TempDir()
 	makeRootDirs(t, root)

cmd/installer/install/config.go

@@ -1,20 +1,3 @@
-/*
-Copyright © 2026 Mulga Defense Corporation
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
-
-You should have received a copy of the GNU Affero General Public License
-along with this program.  If not, see <https://www.gnu.org/licenses/>.
-*/
-
 // Package install performs the disk installation steps after the UI has
 // collected configuration.
 package install

cmd/installer/install/install.go

@@ -1,20 +1,3 @@
-/*
-Copyright © 2026 Mulga Defense Corporation
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
-
-You should have received a copy of the GNU Affero General Public License
-along with this program.  If not, see <https://www.gnu.org/licenses/>.
-*/
-
 package install
 
 import (
@@ -556,7 +539,7 @@ func installBootloader(disk string) error {
 	); biosErr != nil {
 		if efiErr != nil {
 			// Both targets failed — the system will not boot.
-			return fmt.Errorf("both bootloader targets failed (EFI: %v; BIOS: %w)", efiErr, biosErr)
+			return fmt.Errorf("both bootloader targets failed (EFI: %w; BIOS: %w)", efiErr, biosErr)
 		}
 		return biosErr
 	}

cmd/installer/main.go

@@ -1,19 +1,3 @@
-/*
-Copyright © 2026 Mulga Defense Corporation
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
-
-You should have received a copy of the GNU Affero General Public License
-along with this program.  If not, see <https://www.gnu.org/licenses/>.
-*/
 package main
 
 import (

cmd/installer/systemd/systemd.go

@@ -1,20 +1,3 @@
-/*
-Copyright © 2026 Mulga Defense Corporation
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
-
-You should have received a copy of the GNU Affero General Public License
-along with this program. If not, see <https://www.gnu.org/licenses/>.
-*/
-
 // Package systemd writes systemd unit files and drop-ins into an installed
 // system root during the Spinifex ISO installation process.
 package systemd

cmd/installer/ui/ui.go

@@ -1,20 +1,3 @@
-/*
-Copyright © 2026 Mulga Defense Corporation
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
-
-You should have received a copy of the GNU Affero General Public License
-along with this program.  If not, see <https://www.gnu.org/licenses/>.
-*/
-
 // Package ui presents the interactive installer TUI using bubbletea and lipgloss.
 package ui
 

cmd/lb-agent/main.go

@@ -49,12 +49,13 @@ func main() {
 		slog.Error("--gateway is required (or set LB_GATEWAY_URL)")
 		os.Exit(1)
 	}
-	if accessKey == "" || secretKey == "" {
-		slog.Error("--access-key and --secret-key are required (or set LB_ACCESS_KEY / LB_SECRET_KEY)")
-		os.Exit(1)
+	// Empty keys select IMDS instance-role credentials (the SDK default chain
+	// falls back to the in-VM IMDS endpoint); static keys sign as-is.
+	credSource := "imds"
+	if accessKey != "" && secretKey != "" {
+		credSource = "static"
 	}
-
-	slog.Info("Starting LB agent", "lbId", lbID, "gateway", gatewayURL)
+	slog.Info("Starting LB agent", "lbId", lbID, "gateway", gatewayURL, "credSource", credSource)
 
 	agent, err := lbagent.New(lbID, gatewayURL, accessKey, secretKey, region)
 	if err != nil {

cmd/spinifex/LICENSE

@@ -1,202 +0,0 @@
-
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-   1. Definitions.
-
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-   END OF TERMS AND CONDITIONS
-
-   APPENDIX: How to apply the Apache License to your work.
-
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "[]"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-
-   Copyright [yyyy] [name of copyright owner]
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.

cmd/spinifex/cmd/admin_eks.go

@@ -0,0 +1,77 @@
+package cmd
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/admin"
+	handlers_eks "github.com/mulgadc/spinifex/spinifex/handlers/eks"
+	"github.com/spf13/cobra"
+)
+
+var adminEksCmd = &cobra.Command{
+	Use:   "eks",
+	Short: "Manage EKS control-plane disaster recovery",
+}
+
+var adminEksRestoreSnapshotCmd = &cobra.Command{
+	Use:   "restore-snapshot",
+	Short: "Rebuild a single-CP cluster's control plane from an etcd snapshot",
+	Long: `restore-snapshot drives the single-CP total-loss DR path: it launches a fresh
+control-plane VM as a cluster-init seed, directs its boot-time recovery agent to
+run k3s server --cluster-reset (restoring the given etcd snapshot, or the latest
+one found in predastore when --snapshot is omitted), and re-points the cluster
+NLB target groups at the new CP.
+
+Only single-CP clusters are supported. An HA cluster has a potentially
+surviving etcd quorum and should be recovered via quorum reformation instead
+of this destructive single-node rebuild.`,
+	Run: runAdminEksRestoreSnapshot,
+}
+
+func init() {
+	adminCmd.AddCommand(adminEksCmd)
+	adminEksCmd.AddCommand(adminEksRestoreSnapshotCmd)
+
+	adminEksRestoreSnapshotCmd.Flags().String("cluster", "", "Cluster name to restore (required)")
+	adminEksRestoreSnapshotCmd.Flags().String("snapshot", "", "Etcd snapshot key to restore; defaults to the latest snapshot found in predastore")
+	adminEksRestoreSnapshotCmd.Flags().String("account", "", "Account ID that owns the cluster; defaults to the bootstrap account")
+	_ = adminEksRestoreSnapshotCmd.MarkFlagRequired("cluster")
+}
+
+func runAdminEksRestoreSnapshot(cmd *cobra.Command, _ []string) {
+	clusterName, _ := cmd.Flags().GetString("cluster")
+	snapshot, _ := cmd.Flags().GetString("snapshot")
+	accountID, _ := cmd.Flags().GetString("account")
+	if accountID == "" {
+		accountID = admin.DefaultAccountID()
+	}
+
+	_, nc, err := loadConfigAndConnect()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		os.Exit(1)
+	}
+	defer nc.Close()
+
+	svc := handlers_eks.NewNATSEKSService(nc)
+	// The server side launches a VM (30-60s) then fences the old CP with retries,
+	// so the RPC needs generous headroom over a bare launch.
+	ctx, cancel := context.WithTimeout(context.Background(), 300*time.Second)
+	defer cancel()
+
+	fmt.Printf("Restoring control plane for cluster %q (account %s)...\n", clusterName, accountID)
+	out, err := svc.RestoreSnapshot(ctx, &handlers_eks.RestoreSnapshotInput{
+		ClusterName: clusterName,
+		Snapshot:    snapshot,
+	}, accountID)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		os.Exit(1)
+	}
+
+	fmt.Printf("New control plane %s launched (snapshot %s)\n", out.NewInstanceID, out.Snapshot)
+	fmt.Printf("Status: %s\n", out.Status)
+}

cmd/spinifex/cmd/admin_nat_test.go

@@ -0,0 +1,247 @@
+package cmd
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/admin"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestSpinifexTomlTemplate_NATMode(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node:      "node1",
+		Az:        "ap-southeast-2a",
+		Port:      "4432",
+		Region:    "ap-southeast-2",
+		BindIP:    "10.11.12.1",
+		AccessKey: "AKIATEST",
+		SecretKey: "SECRET",
+		AccountID: "123456789012",
+		NatsToken: "token",
+		ConfigDir: dir,
+		OVNNBAddr: "tcp:127.0.0.1:6641",
+		OVNSBAddr: "tcp:127.0.0.1:6642",
+
+		ExternalMode: "nat",
+		BridgeMode:   "nat",
+		Pools: []admin.PoolData{{
+			Name: "nat-transit", Gateway: "100.127.0.1", PrefixLen: 24,
+		}},
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	content := string(data)
+	assert.Contains(t, content, `external_mode = "nat"`)
+	assert.Contains(t, content, `bridge_mode = "nat"`)
+	assert.Contains(t, content, `name        = "nat-transit"`)
+	assert.Contains(t, content, `gateway     = "100.127.0.1"`)
+	assert.NotContains(t, content, "range_start", "nat mode has no public IP range")
+}
+
+func TestSpinifexTomlTemplate_GwLrpRange(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node: "node1", Az: "ap-southeast-2a", Port: "4432",
+		Region: "ap-southeast-2", BindIP: "10.11.12.1",
+		AccessKey: "AKIATEST", SecretKey: "SECRET", AccountID: "123456789012",
+		NatsToken: "token", ConfigDir: dir,
+		OVNNBAddr: "tcp:127.0.0.1:6641", OVNSBAddr: "tcp:127.0.0.1:6642",
+
+		ExternalMode: "nat", BridgeMode: "nat",
+		Pools: []admin.PoolData{{
+			Name: "nat-transit", Gateway: "100.127.0.1", PrefixLen: 24,
+			GwLrpRangeStart: "100.127.0.16", GwLrpRangeEnd: "100.127.0.254",
+		}},
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	content := string(data)
+	assert.Contains(t, content, `gw_lrp_range_start = "100.127.0.16"`)
+	assert.Contains(t, content, `gw_lrp_range_end   = "100.127.0.254"`)
+}
+
+func TestSpinifexTomlTemplate_GwLrpRangeOmittedWhenUnset(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node: "node1", Az: "ap-southeast-2a", Port: "4432",
+		Region: "ap-southeast-2", BindIP: "10.11.12.1",
+		AccessKey: "AKIATEST", SecretKey: "SECRET", AccountID: "123456789012",
+		NatsToken: "token", ConfigDir: dir,
+		OVNNBAddr: "tcp:127.0.0.1:6641", OVNSBAddr: "tcp:127.0.0.1:6642",
+
+		ExternalMode: "pool", ExternalIface: "enp0s3",
+		Pools: []admin.PoolData{{
+			Name: "wan", Source: "static",
+			Start: "192.168.1.150", End: "192.168.1.250",
+			Gateway: "192.168.1.1", PrefixLen: 24,
+		}},
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	assert.NotContains(t, string(data), "gw_lrp_range", "unset range must not render")
+}
+
+func TestSpinifexTomlTemplate_PoolModeOmitsBridgeMode(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node:      "node1",
+		Az:        "ap-southeast-2a",
+		Port:      "4432",
+		Region:    "ap-southeast-2",
+		BindIP:    "10.11.12.1",
+		AccessKey: "AKIATEST",
+		SecretKey: "SECRET",
+		AccountID: "123456789012",
+		NatsToken: "token",
+		ConfigDir: dir,
+		OVNNBAddr: "tcp:127.0.0.1:6641",
+		OVNSBAddr: "tcp:127.0.0.1:6642",
+
+		ExternalMode:  "pool",
+		ExternalIface: "enp0s3",
+		Pools: []admin.PoolData{{
+			Name: "wan", Source: "static",
+			Start: "192.168.1.150", End: "192.168.1.250",
+			Gateway: "192.168.1.1", PrefixLen: 24,
+		}},
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	assert.NotContains(t, string(data), "bridge_mode", "bridged modes keep runtime auto-detect")
+}
+
+func TestIsNonBridgeableUplink(t *testing.T) {
+	for _, name := range []string{"wlan0", "wlp3s0", "wlx001122334455", "wwan0", "wwp0s20f0u6", "ppp0"} {
+		assert.True(t, isNonBridgeableUplink(name), name)
+	}
+	for _, name := range []string{"eth0", "enp0s3", "eno1", "br-wan", "veth-wan-br"} {
+		assert.False(t, isNonBridgeableUplink(name), name)
+	}
+}
+
+func TestBridgeModeFor(t *testing.T) {
+	assert.Equal(t, "nat", bridgeModeFor("nat"))
+	assert.Empty(t, bridgeModeFor("pool"))
+	assert.Empty(t, bridgeModeFor(""))
+}
+
+func TestSpinifexTomlTemplate_NATModeWithPublicPool(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node:      "node1",
+		Az:        "ap-southeast-2a",
+		Port:      "4432",
+		Region:    "ap-southeast-2",
+		BindIP:    "10.11.12.1",
+		AccessKey: "AKIATEST",
+		SecretKey: "SECRET",
+		AccountID: "123456789012",
+		NatsToken: "token",
+		ConfigDir: dir,
+		OVNNBAddr: "tcp:127.0.0.1:6641",
+		OVNSBAddr: "tcp:127.0.0.1:6642",
+
+		ExternalMode: "nat",
+		BridgeMode:   "nat",
+		Pools: []admin.PoolData{
+			{Name: "nat-transit", Gateway: "100.127.0.1", PrefixLen: 24},
+			{
+				Name: "wan", Source: "dhcp", BindBridge: "wlan0",
+				DHCPMAC: "interface", PrefixLen: 24,
+			},
+		},
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	content := string(data)
+	assert.Contains(t, content, `name        = "nat-transit"`)
+	assert.Contains(t, content, `name        = "wan"`)
+	assert.Contains(t, content, `source      = "dhcp"`)
+	assert.Contains(t, content, `bind_bridge = "wlan0"`)
+	assert.Contains(t, content, `dhcp_mac    = "interface"`)
+	transitIdx := strings.Index(content, `name        = "nat-transit"`)
+	wanIdx := strings.Index(content, `name        = "wan"`)
+	assert.Less(t, transitIdx, wanIdx, "transit pool must render first (IGW allocator picks it)")
+}
+
+func TestResolvePublicPoolFlags(t *testing.T) {
+	tests := []struct {
+		name              string
+		source, poolRange string
+		bindBridge, gw    string
+		defaultBridge     string
+		wantSource        string
+		wantStart         string
+		wantEnd           string
+		wantBridge        string
+		wantErr           string
+	}{
+		{
+			name: "static pool range", source: "", poolRange: "192.168.1.150-192.168.1.250", gw: "192.168.1.1",
+			wantSource: "static", wantStart: "192.168.1.150", wantEnd: "192.168.1.250",
+		},
+		{
+			name: "dhcp defaults bind bridge", source: "dhcp", defaultBridge: "wlan0",
+			wantSource: "dhcp", wantBridge: "wlan0",
+		},
+		{
+			name: "dhcp explicit bind bridge", source: "dhcp", bindBridge: "br-wan", defaultBridge: "wlan0",
+			wantSource: "dhcp", wantBridge: "br-wan",
+		},
+		{
+			name: "dhcp rejects pool range", source: "dhcp", poolRange: "192.168.1.150-192.168.1.250",
+			wantErr: "--external-pool not allowed",
+		},
+		{
+			name: "static requires gateway", poolRange: "192.168.1.150-192.168.1.250",
+			wantErr: "--external-gateway is required",
+		},
+		{
+			name: "static rejects bind bridge", poolRange: "192.168.1.150-192.168.1.250", gw: "192.168.1.1", bindBridge: "br-wan",
+			wantErr: "--external-bind-bridge only valid",
+		},
+		{
+			name: "static requires range", source: "static", gw: "192.168.1.1",
+			wantErr: "--external-pool is required",
+		},
+		{
+			name: "bad range", poolRange: "not-a-range", gw: "192.168.1.1",
+			wantErr: "start-end IPs",
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			src, start, end, bb, err := resolvePublicPoolFlags(tt.source, tt.poolRange, tt.bindBridge, tt.gw, tt.defaultBridge)
+			if tt.wantErr != "" {
+				require.Error(t, err)
+				assert.Contains(t, err.Error(), tt.wantErr)
+				return
+			}
+			require.NoError(t, err)
+			assert.Equal(t, tt.wantSource, src)
+			assert.Equal(t, tt.wantStart, start)
+			assert.Equal(t, tt.wantEnd, end)
+			assert.Equal(t, tt.wantBridge, bb)
+		})
+	}
+}

cmd/spinifex/cmd/admin_test.go

@@ -10,7 +10,11 @@ import (
 	"testing"
 
 	"github.com/mulgadc/spinifex/spinifex/admin"
+	"github.com/mulgadc/spinifex/spinifex/config"
 	"github.com/mulgadc/spinifex/spinifex/formation"
+	handlers_dns "github.com/mulgadc/spinifex/spinifex/handlers/dns"
+	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
+	"github.com/mulgadc/spinifex/spinifex/hostdns"
 	"github.com/mulgadc/spinifex/spinifex/utils"
 	toml "github.com/pelletier/go-toml/v2"
 	"github.com/spf13/cobra"
@@ -50,7 +54,7 @@ func TestBuildRemoteNodes_AdvertiseFallback(t *testing.T) {
 		"node2": {Name: "node2", BindIP: "10.0.0.2"}, // legacy joiner
 		"node3": {Name: "node3", BindIP: "10.0.0.3", AdvertiseIP: "203.0.113.3"},
 	}
-	got := buildRemoteNodes(nodes, "node3")
+	got := buildRemoteNodes(nodes, "node3", "")
 	if assert.Len(t, got, 2) {
 		assert.Equal(t, "node1", got[0].Name)
 		assert.Equal(t, "203.0.113.1", got[0].Host, "advertise wins when set")
@@ -59,6 +63,29 @@ func TestBuildRemoteNodes_AdvertiseFallback(t *testing.T) {
 	}
 }
 
+// Every peer must carry the northstar stanza, since a node that describes only
+// its own northstar computes a single-entry seed set and pins the base zone's NS
+// records to whichever node won the create-if-absent race.
+func TestBuildRemoteNodes_NorthstarConfigPath(t *testing.T) {
+	nodes := map[string]formation.NodeInfo{
+		"node1": {Name: "node1", BindIP: "10.0.0.1"},
+		"node2": {Name: "node2", BindIP: "10.0.0.2"},
+	}
+
+	got := buildRemoteNodes(nodes, "node2", "/etc/spinifex/northstar/northstar.toml")
+	if assert.Len(t, got, 1) {
+		assert.Equal(t, "/etc/spinifex/northstar/northstar.toml", got[0].NorthstarConfigPath)
+	}
+
+	// No distributed credentials means no node runs northstar. Publishing peers
+	// as resolvers anyway would point guests at 169.254.169.253 for a service
+	// that never starts.
+	got = buildRemoteNodes(nodes, "node2", "")
+	if assert.Len(t, got, 1) {
+		assert.Empty(t, got[0].NorthstarConfigPath)
+	}
+}
+
 func TestResolveAdvertiseIP(t *testing.T) {
 	wan := &admin.DetectedNetwork{WAN: &admin.DetectedInterface{IP: "192.168.1.21"}}
 	noWAN := &admin.DetectedNetwork{}
@@ -145,6 +172,44 @@ func TestSpinifexTomlTemplate_AdvertiseOmittedWhenEmpty(t *testing.T) {
 	assert.NotContains(t, string(data), "advertise =")
 }
 
+// The northstar stanza carries the non-secret domains so confined services
+// (vpcd) resolve DNS names without reading the 0600 northstar.toml.
+func TestSpinifexTomlTemplate_NorthstarDomains(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node:                    "node1",
+		Az:                      "ap-southeast-2a",
+		Port:                    "4432",
+		Region:                  "ap-southeast-2",
+		BindIP:                  "10.11.12.1",
+		AccessKey:               "AKIATEST",
+		SecretKey:               "SECRET",
+		AccountID:               "123456789012",
+		NatsToken:               "token",
+		ConfigDir:               dir,
+		OVNNBAddr:               "tcp:127.0.0.1:6641",
+		OVNSBAddr:               "tcp:127.0.0.1:6642",
+		NorthstarConfigPath:     "/etc/spinifex/northstar/northstar.toml",
+		NorthstarDefaultDomain:  "spx3.net",
+		NorthstarInternalDomain: "compute.internal",
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	content := string(data)
+	assert.Contains(t, content, `config_path = "/etc/spinifex/northstar/northstar.toml"`)
+	assert.Contains(t, content, `default_domain = "spx3.net"`)
+	assert.Contains(t, content, `internal_domain = "compute.internal"`)
+
+	// The stanza must round-trip into NorthstarConfig so the resolvers read it.
+	cfg, err := config.LoadConfig(path)
+	require.NoError(t, err)
+	assert.Equal(t, "spx3.net", cfg.Nodes["node1"].Northstar.DefaultDomain)
+	assert.Equal(t, "compute.internal", cfg.Nodes["node1"].Northstar.InternalDomain)
+}
+
 // Legacy `wan_bridge` TOML key must fail-start vpcd with guidance, not silently
 // alias (per mulga-998 D3). Prevents the footgun where operators inherited the
 // old key pointing at 'br-ext' and got broken DHCP on veth-mode hosts.
@@ -209,7 +274,7 @@ func TestPredastoreMultinodeTemplate_NatsURLLoopbackShim(t *testing.T) {
 		{ID: 3, Host: "10.0.0.3"},
 	}
 	content, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "0.0.0.0", 0,
+		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "0.0.0.0", 0, admin.NorthstarCredentials{},
 	)
 	require.NoError(t, err)
 	assert.Contains(t, content, `nats_url = "nats://localhost:4222"`)
@@ -217,7 +282,7 @@ func TestPredastoreMultinodeTemplate_NatsURLLoopbackShim(t *testing.T) {
 
 	// Specific bind IP → stays as-is.
 	content2, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "10.11.12.1", 0,
+		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "10.11.12.1", 0, admin.NorthstarCredentials{},
 	)
 	require.NoError(t, err)
 	assert.Contains(t, content2, `nats_url = "nats://10.11.12.1:4222"`)
@@ -255,23 +320,6 @@ func TestWritePredastoreEncryptionKey(t *testing.T) {
 	assert.NotEqual(t, key1, key2, "per-node keys must differ")
 }
 
-// The viperblock key gates at-rest encryption for every volume on a fresh
-// install. masterkey.LoadShared rejects anything that isn't exactly 32 bytes at
-// 0640-or-stricter, so a regression in size/mode would silently fall back to
-// cleartext (empty path) or fail service startup.
-func TestWriteViperblockEncryptionKey(t *testing.T) {
-	configDir := t.TempDir()
-
-	keyPath, err := writeViperblockEncryptionKey(configDir)
-	require.NoError(t, err)
-	assert.Equal(t, filepath.Join(configDir, "viperblock", "encryption.key"), keyPath)
-
-	info, err := os.Stat(keyPath)
-	require.NoError(t, err)
-	assert.Equal(t, int64(32), info.Size(), "viperblock master key must be exactly 32 bytes")
-	assert.Equal(t, os.FileMode(0600), info.Mode().Perm(), "viperblock master key must be mode 0600")
-}
-
 // Joiners persist the leader's shared key via saveViperblockEncryptionKey
 // against a configDir with no viperblock/ subdir yet. The bytes round-trip
 // verbatim — the whole cluster shares one key, so any mutation would orphan
@@ -292,6 +340,181 @@ func TestSaveViperblockEncryptionKey_RoundTrip(t *testing.T) {
 	assert.Equal(t, key, got, "shared key must be written verbatim")
 }
 
+// A re-init must not rotate the per-node predastore key: rotating it would orphan
+// every fragment already sealed under the old key. The helper preserves an
+// existing key byte-for-byte.
+func TestWritePredastoreEncryptionKey_PreservesExisting(t *testing.T) {
+	configDir := t.TempDir()
+
+	keyPath, err := writePredastoreEncryptionKey(configDir)
+	require.NoError(t, err)
+	orig, err := os.ReadFile(keyPath)
+	require.NoError(t, err)
+
+	keyPath2, err := writePredastoreEncryptionKey(configDir)
+	require.NoError(t, err)
+	assert.Equal(t, keyPath, keyPath2)
+	after, err := os.ReadFile(keyPath2)
+	require.NoError(t, err)
+	assert.Equal(t, orig, after, "existing predastore key must be preserved on re-init")
+}
+
+// ensureViperblockEncryptionKey generates a shared 32-byte 0600 key on a fresh
+// dir and returns bytes that match the on-disk file; a second call preserves it
+// so a re-init keeps every encrypted volume readable.
+func TestEnsureViperblockEncryptionKey(t *testing.T) {
+	configDir := t.TempDir()
+
+	key, keyPath, err := ensureViperblockEncryptionKey(configDir)
+	require.NoError(t, err)
+	assert.Equal(t, filepath.Join(configDir, "viperblock", "encryption.key"), keyPath)
+	assert.Len(t, key, 32, "viperblock key must be 32 bytes")
+
+	info, err := os.Stat(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, int64(32), info.Size(), "viperblock master key must be exactly 32 bytes")
+	assert.Equal(t, os.FileMode(0600), info.Mode().Perm(), "viperblock master key must be mode 0600")
+
+	onDisk, err := os.ReadFile(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, key, onDisk, "returned bytes must match the file")
+
+	key2, _, err := ensureViperblockEncryptionKey(configDir)
+	require.NoError(t, err)
+	assert.Equal(t, key, key2, "existing viperblock key must be preserved on re-init")
+}
+
+// A present-but-corrupt viperblock key must fail loud rather than regenerate,
+// which would orphan every already-encrypted volume in the cluster.
+func TestEnsureViperblockEncryptionKey_CorruptFailsLoud(t *testing.T) {
+	configDir := t.TempDir()
+	keyDir := filepath.Join(configDir, "viperblock")
+	require.NoError(t, os.MkdirAll(keyDir, 0750))
+	keyPath := filepath.Join(keyDir, "encryption.key")
+	require.NoError(t, os.WriteFile(keyPath, []byte("bad"), 0600))
+
+	_, _, err := ensureViperblockEncryptionKey(configDir)
+	require.Error(t, err, "a truncated viperblock key must error, not regenerate")
+
+	after, err := os.ReadFile(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, []byte("bad"), after, "corrupt viperblock key must not be overwritten")
+}
+
+// ensureMasterKey generates a fresh in-memory key without touching disk (the
+// bootstrap writer persists it), and preserves an existing on-disk key so a
+// re-init never rotates the key that encrypts IAM secrets in NATS KV.
+func TestEnsureMasterKey(t *testing.T) {
+	t.Run("FreshGeneratesWithoutWriting", func(t *testing.T) {
+		configDir := t.TempDir()
+		key, existed, err := ensureMasterKey(configDir)
+		require.NoError(t, err)
+		assert.False(t, existed, "no master.key present yet")
+		assert.Len(t, key, 32)
+		assert.NoFileExists(t, filepath.Join(configDir, "master.key"),
+			"ensureMasterKey must not persist on the fresh path")
+	})
+
+	t.Run("PreservesExisting", func(t *testing.T) {
+		configDir := t.TempDir()
+		seed := make([]byte, 32)
+		for i := range seed {
+			seed[i] = byte(200 - i)
+		}
+		require.NoError(t, handlers_iam.SaveMasterKey(filepath.Join(configDir, "master.key"), seed))
+
+		key, existed, err := ensureMasterKey(configDir)
+		require.NoError(t, err)
+		assert.True(t, existed, "existing master.key must be detected")
+		assert.Equal(t, seed, key, "existing master key must be loaded verbatim")
+	})
+
+	// A present-but-corrupt master.key must fail loud, never be silently
+	// regenerated: rotating it would orphan every IAM secret in NATS KV.
+	t.Run("CorruptFailsLoud", func(t *testing.T) {
+		configDir := t.TempDir()
+		keyPath := filepath.Join(configDir, "master.key")
+		require.NoError(t, os.WriteFile(keyPath, []byte("too-short"), 0600))
+
+		_, _, err := ensureMasterKey(configDir)
+		require.Error(t, err, "a truncated master.key must error, not regenerate")
+
+		after, err := os.ReadFile(keyPath)
+		require.NoError(t, err)
+		assert.Equal(t, []byte("too-short"), after, "corrupt master key must not be overwritten")
+	})
+}
+
+// The identity bundle is preserved holistically on re-init: the load helpers
+// recover the exact system credentials and never rewrite bootstrap.json. Admin
+// credentials are not recovered from disk — awsgw consumes and deletes
+// bootstrap.json after first boot, so the preserve path must not read it back.
+func TestPreservedIdentityBundle(t *testing.T) {
+	configDir := t.TempDir()
+	bootstrapDir := filepath.Join(configDir, "awsgw")
+
+	masterKey, err := handlers_iam.GenerateMasterKey()
+	require.NoError(t, err)
+	const (
+		sysAccess   = "AKIASYSTEM0000000000"
+		sysSecret   = "system-secret-value"
+		adminAccess = "AKIAADMIN00000000000"
+		adminSecret = "admin-secret-value"
+		accountID   = "123456789012"
+	)
+	require.NoError(t, writeBootstrapFilesWithAdmin(configDir, bootstrapDir, masterKey,
+		sysAccess, sysSecret, accountID, adminAccess, adminSecret))
+	require.NoError(t, writeSystemCredentials(configDir, sysAccess, sysSecret))
+
+	bootstrapPath := filepath.Join(bootstrapDir, "bootstrap.json")
+	bootstrapBefore, err := os.ReadFile(bootstrapPath)
+	require.NoError(t, err)
+
+	// master.key round-trips and is flagged as pre-existing.
+	loadedKey, existed, err := ensureMasterKey(configDir)
+	require.NoError(t, err)
+	assert.True(t, existed)
+	assert.Equal(t, masterKey, loadedKey)
+
+	// System credentials come back verbatim (they must match the NATS KV seed).
+	gotSysAccess, gotSysSecret, err := loadSystemCredentials(configDir)
+	require.NoError(t, err)
+	assert.Equal(t, sysAccess, gotSysAccess)
+	assert.Equal(t, sysSecret, gotSysSecret)
+
+	// The preserve helpers must not rewrite the seed file.
+	bootstrapAfter, err := os.ReadFile(bootstrapPath)
+	require.NoError(t, err)
+	assert.Equal(t, bootstrapBefore, bootstrapAfter, "bootstrap.json must not be rewritten on re-init")
+}
+
+// loadSystemCredentials surfaces a clear error when the file is absent rather
+// than returning empty credentials that would silently break SigV4 auth.
+func TestLoadSystemCredentials_Missing(t *testing.T) {
+	_, _, err := loadSystemCredentials(t.TempDir())
+	require.Error(t, err)
+}
+
+// A present-but-empty or malformed system-credentials.json must error rather
+// than yield blank credentials that would silently break SigV4 between services.
+func TestLoadSystemCredentials_Invalid(t *testing.T) {
+	t.Run("EmptyValues", func(t *testing.T) {
+		configDir := t.TempDir()
+		require.NoError(t, os.WriteFile(filepath.Join(configDir, "system-credentials.json"),
+			[]byte(`{"access_key_id":"","secret_access_key":""}`), 0600))
+		_, _, err := loadSystemCredentials(configDir)
+		require.Error(t, err)
+	})
+
+	t.Run("MalformedJSON", func(t *testing.T) {
+		configDir := t.TempDir()
+		require.NoError(t, os.WriteFile(filepath.Join(configDir, "system-credentials.json"),
+			[]byte("not json"), 0600))
+		_, _, err := loadSystemCredentials(configDir)
+		require.Error(t, err)
+	})
+}
+
 // A fresh install must render encryption_key_file so viperblockd enables
 // at-rest encryption; an empty field (upgrade / legacy) must omit the line
 // entirely so the volume stays in cleartext legacy mode.
@@ -323,6 +546,32 @@ func TestSpinifexTomlTemplate_EncryptionKeyFile(t *testing.T) {
 	assert.NotContains(t, string(data2), "encryption_key_file", "blank key must omit the field")
 }
 
+// A rendered node config must turn chunk GC on. Asserting the template text
+// alone is not enough: the value has to survive parsing into a non-nil true
+// pointer, because every consumer treats nil as off. Rendering and loading
+// together is the only check that covers the whole path.
+func TestSpinifexTomlTemplate_GCEnabled(t *testing.T) {
+	dir := t.TempDir()
+	settings := admin.ConfigSettings{
+		Node: "node1", Az: "ap-southeast-2a", Port: "4432", Region: "ap-southeast-2",
+		BindIP: "0.0.0.0", AccessKey: "AKIATEST", SecretKey: "SECRET",
+		AccountID: "123456789012", NatsToken: "token",
+		OVNNBAddr: "tcp:127.0.0.1:6641", OVNSBAddr: "tcp:127.0.0.1:6642",
+		ConfigDir: dir,
+	}
+
+	path := filepath.Join(dir, "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	cfg, err := config.LoadConfig(path)
+	require.NoError(t, err)
+
+	node, ok := cfg.Nodes["node1"]
+	require.True(t, ok, "rendered config must define the node")
+	require.NotNil(t, node.Viperblock.GCEnabled, "gc_enabled must be rendered; nil leaves chunk GC off on every deployed node")
+	assert.True(t, *node.Viperblock.GCEnabled, "chunk GC must be enabled in a freshly rendered node config")
+}
+
 // Joiners run `spx admin join` against a configDir that doesn't yet contain
 // a predastore/ subdir. The helper must create it (predastore subdir is
 // owned by spinifex-storage in prod, but the helper just needs the dir to
@@ -365,15 +614,15 @@ const cobraRequiredAnnotation = "cobra_annotation_bash_completion_one_required_f
 func TestAdminInitCmd_ExternalDHCPFlagSchema(t *testing.T) {
 	sourceFlag := adminInitCmd.Flags().Lookup("external-source")
 	require.NotNil(t, sourceFlag, "--external-source must be defined")
-	assert.Equal(t, "", sourceFlag.DefValue)
+	assert.Empty(t, sourceFlag.DefValue)
 
 	bindFlag := adminInitCmd.Flags().Lookup("external-bind-bridge")
 	require.NotNil(t, bindFlag, "--external-bind-bridge must be defined")
-	assert.Equal(t, "", bindFlag.DefValue)
+	assert.Empty(t, bindFlag.DefValue)
 
 	poolFlag := adminInitCmd.Flags().Lookup("external-pool")
 	require.NotNil(t, poolFlag)
-	assert.Equal(t, "", poolFlag.DefValue)
+	assert.Empty(t, poolFlag.DefValue)
 }
 
 // DHCP-sourced pool must emit source + bind_bridge into spinifex.toml so the
@@ -382,20 +631,19 @@ func TestSpinifexTomlTemplate_ExternalPoolDHCPSource(t *testing.T) {
 	dir := t.TempDir()
 	path := filepath.Join(dir, "spinifex.toml")
 	settings := admin.ConfigSettings{
-		Node:           "node1",
-		Az:             "ap-southeast-2a",
-		Port:           "4432",
-		Region:         "ap-southeast-2",
-		BindIP:         "10.11.12.1",
-		ConfigDir:      dir,
-		OVNNBAddr:      "tcp:127.0.0.1:6641",
-		OVNSBAddr:      "tcp:127.0.0.1:6642",
-		ExternalMode:   "pool",
-		ExternalIface:  "eth0",
-		PoolName:       "wan",
-		PoolSource:     "dhcp",
-		PoolBindBridge: "br-wan",
-		PoolPrefixLen:  24,
+		Node:          "node1",
+		Az:            "ap-southeast-2a",
+		Port:          "4432",
+		Region:        "ap-southeast-2",
+		BindIP:        "10.11.12.1",
+		ConfigDir:     dir,
+		OVNNBAddr:     "tcp:127.0.0.1:6641",
+		OVNSBAddr:     "tcp:127.0.0.1:6642",
+		ExternalMode:  "pool",
+		ExternalIface: "eth0",
+		Pools: []admin.PoolData{{
+			Name: "wan", Source: "dhcp", BindBridge: "br-wan", PrefixLen: 24,
+		}},
 	}
 	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
 
@@ -424,12 +672,11 @@ func TestSpinifexTomlTemplate_ExternalPoolStaticSource(t *testing.T) {
 		OVNSBAddr:     "tcp:127.0.0.1:6642",
 		ExternalMode:  "pool",
 		ExternalIface: "eth0",
-		PoolName:      "wan",
-		PoolSource:    "static",
-		PoolStart:     "192.168.1.150",
-		PoolEnd:       "192.168.1.250",
-		PoolGateway:   "192.168.1.1",
-		PoolPrefixLen: 24,
+		Pools: []admin.PoolData{{
+			Name: "wan", Source: "static",
+			Start: "192.168.1.150", End: "192.168.1.250",
+			Gateway: "192.168.1.1", PrefixLen: 24,
+		}},
 	}
 	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
 
@@ -454,8 +701,9 @@ func TestApplyNetworkConfig_PropagatesPoolBindBridge(t *testing.T) {
 		PoolPrefixLen:  24,
 	}
 	applyNetworkConfig(settings, nc)
-	assert.Equal(t, "dhcp", settings.PoolSource)
-	assert.Equal(t, "br-wan", settings.PoolBindBridge)
+	require.Len(t, settings.Pools, 1)
+	assert.Equal(t, "dhcp", settings.Pools[0].Source)
+	assert.Equal(t, "br-wan", settings.Pools[0].BindBridge)
 }
 
 func renderSingleNodePredastore(t *testing.T, settings admin.ConfigSettings) string {
@@ -473,6 +721,297 @@ func predastoreMultinodeNodes() []admin.PredastoreNodeConfig {
 	}
 }
 
+// predastoreBucketAuth parses the buckets and auth entries a northstar-enabled
+// multi-node predastore must render.
+type predastoreBucketAuth struct {
+	Buckets []struct {
+		Name string `toml:"name"`
+		Type string `toml:"type"`
+	} `toml:"buckets"`
+	Auth []struct {
+		AccessKeyID     string `toml:"access_key_id"`
+		SecretAccessKey string `toml:"secret_access_key"`
+		Policy          []struct {
+			Bucket  string   `toml:"bucket"`
+			Actions []string `toml:"actions"`
+		} `toml:"policy"`
+	} `toml:"auth"`
+}
+
+// policyFor returns the actions the given key is granted on the given bucket,
+// and whether the key holds any policy for it at all.
+func (c predastoreBucketAuth) policyFor(accessKey, bucket string) ([]string, bool) {
+	for _, auth := range c.Auth {
+		if auth.AccessKeyID != accessKey {
+			continue
+		}
+		for _, p := range auth.Policy {
+			if p.Bucket == bucket {
+				return p.Actions, true
+			}
+		}
+	}
+	return nil, false
+}
+
+func (c predastoreBucketAuth) hasBucket(name string) bool {
+	for _, b := range c.Buckets {
+		if b.Name == name {
+			return true
+		}
+	}
+	return false
+}
+
+// Northstar on a multi-node cluster needs three things from predastore, all
+// gated on the same credential: the zone bucket, write access for the system
+// key (which is what BootstrapBaseZone seeds zones with), and the resolver's
+// own read-only key. The template rendered all three correctly, but the caller
+// never populated the credential, so multi-node clusters silently got none of
+// them.
+func TestPredastoreMultinodeTemplate_NorthstarProvisioned(t *testing.T) {
+	content, err := admin.GenerateMultiNodePredastoreConfig(
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0,
+		admin.NorthstarCredentials{AccessKey: "NSAK", SecretKey: "NSSK", Bucket: admin.NorthstarBucketName},
+	)
+	require.NoError(t, err)
+
+	var cfg predastoreBucketAuth
+	require.NoError(t, toml.Unmarshal([]byte(content), &cfg))
+
+	assert.True(t, cfg.hasBucket(admin.NorthstarBucketName), "northstar zone bucket not rendered")
+
+	// The zone bucket must be readable from every node's local predastore, so
+	// a resolver on any node can serve it.
+	for _, b := range cfg.Buckets {
+		if b.Name == admin.NorthstarBucketName {
+			assert.Equal(t, "distributed", b.Type)
+		}
+	}
+
+	// BootstrapBaseZone writes zone files with the system key, not the
+	// resolver's. Without this grant, seeding fails on every node.
+	systemActions, ok := cfg.policyFor("AK", admin.NorthstarBucketName)
+	require.True(t, ok, "system key has no policy for the northstar bucket")
+	assert.Contains(t, systemActions, "s3:GetObject", "bootstrap must check whether the zone already exists")
+	assert.Contains(t, systemActions, "s3:PutObject", "bootstrap must create a missing zone")
+
+	// The resolver's key stays read-only and scoped to the zone bucket alone.
+	nsActions, ok := cfg.policyFor("NSAK", admin.NorthstarBucketName)
+	require.True(t, ok, "northstar key has no policy for the northstar bucket")
+	assert.ElementsMatch(t, []string{"s3:ListBucket", "s3:GetObject"}, nsActions)
+	for _, auth := range cfg.Auth {
+		if auth.AccessKeyID == "NSAK" {
+			assert.Equal(t, "NSSK", auth.SecretAccessKey)
+		}
+	}
+
+	_, ok = cfg.policyFor("NSAK", "predastore")
+	assert.False(t, ok, "northstar key must not reach the predastore bucket")
+}
+
+// Without a credential the northstar stanzas must be omitted wholesale rather
+// than half-rendered: an empty-keyed [[auth]] entry would let anyone read the
+// zone bucket.
+func TestPredastoreMultinodeTemplate_NorthstarOmittedWithoutCredentials(t *testing.T) {
+	content, err := admin.GenerateMultiNodePredastoreConfig(
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0,
+		admin.NorthstarCredentials{},
+	)
+	require.NoError(t, err)
+
+	var cfg predastoreBucketAuth
+	require.NoError(t, toml.Unmarshal([]byte(content), &cfg))
+
+	assert.False(t, cfg.hasBucket(admin.NorthstarBucketName))
+	_, ok := cfg.policyFor("AK", admin.NorthstarBucketName)
+	assert.False(t, ok, "system key granted northstar access without a northstar credential")
+	assert.Len(t, cfg.Auth, 1, "only the system key should be rendered")
+}
+
+// A joiner must provision its predastore with the leader's pair verbatim, since
+// each node's predastore only honours the keys in its own config. Per-node
+// generation would have node A's predastore reject node B's resolver.
+func TestNorthstarFromFormation_UsesDistributedPair(t *testing.T) {
+	dirs := configDirs{Northstar: "/etc/spinifex/northstar"}
+	creds := &formation.SharedCredentials{NorthstarAccessKey: "NSAK", NorthstarSecretKey: "NSSK"}
+
+	got, path := northstarFromFormation(creds, dirs)
+
+	assert.Equal(t, "NSAK", got.AccessKey)
+	assert.Equal(t, "NSSK", got.SecretKey)
+	assert.Equal(t, admin.NorthstarBucketName, got.Bucket, "bucket is a constant, derived node-side")
+	assert.Equal(t, "/etc/spinifex/northstar/northstar.toml", path)
+}
+
+// A leader predating credential distribution, or returning only half a pair,
+// must yield no Northstar config. Advertising a resolver without usable keys
+// would send guests to a service that cannot read its zones.
+func TestNorthstarFromFormation_IncompletePairYieldsNoConfig(t *testing.T) {
+	tests := []struct {
+		name   string
+		access string
+		secret string
+	}{
+		{name: "legacy leader"},
+		{name: "access key only", access: "NSAK"},
+		{name: "secret key only", secret: "NSSK"},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got, path := northstarFromFormation(&formation.SharedCredentials{
+				NorthstarAccessKey: tt.access,
+				NorthstarSecretKey: tt.secret,
+			}, configDirs{Northstar: "/etc/spinifex/northstar"})
+
+			assert.Equal(t, admin.NorthstarCredentials{}, got)
+			assert.Empty(t, path, "config path without credentials advertises a resolver that cannot read its zones")
+		})
+	}
+}
+
+// multinodeNorthstarSettings mirrors what both formation paths hand
+// generateAndWriteConfigs once the distributed pair is wired through.
+func multinodeNorthstarSettings(configPath string) admin.ConfigSettings {
+	s := northstarSettings()
+	s.Node = "node1"
+	s.Az = "ap-southeast-2a"
+	s.Port = "4432"
+	s.AccountID = "123456789012"
+	s.OVNNBAddr = "tcp:127.0.0.1:6641"
+	s.OVNSBAddr = "tcp:127.0.0.1:6642"
+	s.NorthstarConfigPath = configPath
+	return s
+}
+
+// The multi-node formation paths must produce both halves of a working
+// resolver: the northstar.toml the service reads, and the spinifex.toml stanza
+// that makes configuredNorthstarNodeNames count this node. Rendering was
+// previously reachable only from the single-node path.
+func TestGenerateAndWriteConfigs_MultinodeRendersNorthstar(t *testing.T) {
+	dirs, err := createConfigSubdirs(t.TempDir())
+	require.NoError(t, err)
+	nsPath := filepath.Join(dirs.Northstar, "northstar.toml")
+	spinifexTomlPath := filepath.Join(t.TempDir(), "spinifex.toml")
+
+	require.NoError(t, generateAndWriteConfigs(dirs, spinifexTomlPath, multinodeNorthstarSettings(nsPath), true))
+
+	nsContent, err := os.ReadFile(nsPath)
+	require.NoError(t, err)
+	assert.Contains(t, string(nsContent), `access_key = "AKIANORTHSTAR"`)
+
+	// config_path is the sole predicate for "this node runs northstar", so its
+	// presence is what puts the node in the nameserver set and the resolver list.
+	cfg, err := config.LoadConfig(spinifexTomlPath)
+	require.NoError(t, err)
+	assert.Equal(t, nsPath, cfg.Nodes["node1"].Northstar.ConfigPath)
+}
+
+// Incomplete credentials must yield neither file nor stanza. A stanza without
+// a usable northstar.toml would point guests at a resolver that never starts.
+func TestGenerateAndWriteConfigs_NoNorthstarWithoutCompleteCredentials(t *testing.T) {
+	tests := []struct {
+		name   string
+		access string
+		secret string
+	}{
+		{name: "no credentials"},
+		{name: "access key only", access: "AKIANORTHSTAR"},
+		{name: "secret key only", secret: "northstar-secret"},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			dirs, err := createConfigSubdirs(t.TempDir())
+			require.NoError(t, err)
+			spinifexTomlPath := filepath.Join(t.TempDir(), "spinifex.toml")
+
+			settings := multinodeNorthstarSettings(filepath.Join(dirs.Northstar, "northstar.toml"))
+			settings.NorthstarAccessKey = tt.access
+			settings.NorthstarSecretKey = tt.secret
+			require.NoError(t, generateAndWriteConfigs(dirs, spinifexTomlPath, settings, false))
+
+			_, err = os.Stat(filepath.Join(dirs.Northstar, "northstar.toml"))
+			assert.True(t, os.IsNotExist(err), "northstar.toml rendered without complete credentials")
+
+			cfg, err := config.LoadConfig(spinifexTomlPath)
+			require.NoError(t, err)
+			assert.Empty(t, cfg.Nodes["node1"].Northstar.ConfigPath)
+
+			predastoreContent, err := os.ReadFile(filepath.Join(dirs.Predastore, "predastore.toml"))
+			require.NoError(t, err)
+			var predastoreCfg predastoreBucketAuth
+			require.NoError(t, toml.Unmarshal(predastoreContent, &predastoreCfg))
+			assert.False(t, predastoreCfg.hasBucket(admin.NorthstarBucketName))
+			assert.Len(t, predastoreCfg.Auth, 1, "incomplete pair rendered a Northstar auth entry")
+		})
+	}
+}
+
+// renderClusterNode renders one node's spinifex.toml as its own formation path
+// would: its local northstar stanza plus a stanza for every peer.
+func renderClusterNode(t *testing.T, node string, allNodes map[string]formation.NodeInfo) *config.ClusterConfig {
+	t.Helper()
+	const nsPath = "/etc/spinifex/northstar/northstar.toml"
+
+	settings := multinodeNorthstarSettings(nsPath)
+	settings.Node = node
+	settings.BindIP = allNodes[node].BindIP
+	settings.AdvertiseIP = allNodes[node].AdvertiseIP
+	settings.RemoteNodes = buildRemoteNodes(allNodes, node, nsPath)
+
+	path := filepath.Join(t.TempDir(), "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	cfg, err := config.LoadConfig(path)
+	require.NoError(t, err)
+	return cfg
+}
+
+// Every node must derive the same nameserver set from its own rendered config.
+// EnsureBaseZone is create-if-absent and never overwrites, so if each node saw
+// only itself the zone's NS records would be pinned to whichever node won the
+// startup race — permanently, for the life of the zone object. Identical sets
+// make that race benign: whoever writes, writes the same bytes.
+func TestSpinifexTomlTemplate_PeerNorthstarSeedsAreUniform(t *testing.T) {
+	allNodes := map[string]formation.NodeInfo{
+		"node1": {Name: "node1", BindIP: "10.0.0.1", AdvertiseIP: "192.168.1.31"},
+		"node2": {Name: "node2", BindIP: "10.0.0.2", AdvertiseIP: "192.168.1.32"},
+		"node3": {Name: "node3", BindIP: "10.0.0.3", AdvertiseIP: "192.168.1.33"},
+	}
+
+	wantIPs := []string{"192.168.1.31", "192.168.1.32", "192.168.1.33"}
+	wantHosts := []string{"ns1", "ns2", "ns3"}
+	for _, node := range []string{"node1", "node2", "node3"} {
+		cfg := renderClusterNode(t, node, allNodes)
+		seeds := handlers_dns.NameserverSeeds(cfg)
+		require.Len(t, seeds, 3)
+		for i, seed := range seeds {
+			assert.Equal(t, wantHosts[i], seed.Host)
+			assert.Equal(t, wantIPs[i], seed.IP)
+		}
+		assert.Equal(t, wantIPs, handlers_dns.ResolverNameserverIPs(cfg),
+			"%s derived a different nameserver set from its own config", node)
+	}
+}
+
+// A peer's stanza carries config_path alone: it is the only field ever read back
+// from a peer, and the domains would be fiction — the local node cannot know
+// what a peer configured.
+func TestSpinifexTomlTemplate_PeerNorthstarStanzaIsPathOnly(t *testing.T) {
+	allNodes := map[string]formation.NodeInfo{
+		"node1": {Name: "node1", BindIP: "10.0.0.1", AdvertiseIP: "192.168.1.31"},
+		"node2": {Name: "node2", BindIP: "10.0.0.2", AdvertiseIP: "192.168.1.32"},
+	}
+
+	cfg := renderClusterNode(t, "node1", allNodes)
+	peer := cfg.Nodes["node2"].Northstar
+	assert.Equal(t, "/etc/spinifex/northstar/northstar.toml", peer.ConfigPath)
+	assert.Empty(t, peer.DefaultDomain, "peer domains are unknowable from here")
+	assert.Empty(t, peer.InternalDomain)
+}
+
 // Unset knob must leave production config byte-identical: no [compaction] block
 // and the original trailing bytes unchanged (single-node has no trailing
 // newline, multinode keeps one).
@@ -486,7 +1025,7 @@ func TestPredastoreTemplates_UnsetCompactionEmitsNoBlock(t *testing.T) {
 		"unset single-node tail changed: %q", single[len(single)-40:])
 
 	multi, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0,
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0, admin.NorthstarCredentials{},
 	)
 	require.NoError(t, err)
 	assert.NotContains(t, multi, "[compaction]")
@@ -507,7 +1046,7 @@ func TestPredastoreTemplates_SetCompactionEmitsParseableBlock(t *testing.T) {
 	assert.Equal(t, interval, singleCfg.Compaction.IntervalSeconds)
 
 	multi, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", interval,
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", interval, admin.NorthstarCredentials{},
 	)
 	require.NoError(t, err)
 	var multiCfg compactionConfig
@@ -544,3 +1083,169 @@ func TestAdminInitFlag_CompactionIntervalReachesRenderedConfig(t *testing.T) {
 	})
 	assert.Contains(t, out, "interval_seconds = 45")
 }
+
+func TestFilterDNSServersExcludesLocalNorthstar(t *testing.T) {
+	servers := []string{
+		"10.0.0.5",
+		"127.0.0.53",
+		"1.1.1.1",
+		"1.1.1.1",
+		"not-an-ip",
+		"2001:0db8::1",
+		"8.8.8.8",
+		"9.9.9.9",
+		"4.4.4.4",
+	}
+
+	got := filterDNSServers(servers, "10.0.0.5", "2001:db8::1")
+	assert.Equal(t, []string{"1.1.1.1", "8.8.8.8", "9.9.9.9"}, got)
+}
+
+func TestDetectDNSServersContinuesAfterExcludedSource(t *testing.T) {
+	var queries []string
+	sources := dnsDetectionSources{
+		queryResolvectl: func(args ...string) (string, error) {
+			query := strings.Join(args, " ")
+			queries = append(queries, query)
+			switch query {
+			case "dns br-wan":
+				return "Link 2 (br-wan): 10.0.0.5 127.0.0.53\n", nil
+			case "dns":
+				return "Global: 10.0.0.5 10.0.0.5 127.0.0.53 10.20.0.53 8.8.8.8\n", nil
+			default:
+				return "", fmt.Errorf("unexpected query %q", query)
+			}
+		},
+		readResolvConf: func() (string, error) {
+			t.Fatal("resolv.conf read despite usable global resolvers")
+			return "", nil
+		},
+	}
+
+	got := detectDNSServersWithSources("br-wan", []string{"10.0.0.5"}, sources)
+	assert.Equal(t, []string{"10.20.0.53", "8.8.8.8"}, got)
+	assert.Equal(t, []string{"dns br-wan", "dns"}, queries)
+}
+
+func TestDetectDNSServersFallsBackToResolvConf(t *testing.T) {
+	sources := dnsDetectionSources{
+		queryResolvectl: func(_ ...string) (string, error) {
+			return "", errors.New("resolver manager unavailable")
+		},
+		readResolvConf: func() (string, error) {
+			return "nameserver 10.0.0.5\nnameserver 10.20.0.53\n", nil
+		},
+	}
+
+	got := detectDNSServersWithSources("br-wan", []string{"10.0.0.5"}, sources)
+	assert.Equal(t, []string{"10.20.0.53"}, got)
+}
+
+func TestHostDNSParams(t *testing.T) {
+	// northstarEnabled requires both credentials; the zones and IPs mirror what
+	// admin init/join hold in memory after rendering northstar.toml.
+	enabled := admin.ConfigSettings{
+		NorthstarAccessKey:      "AK",
+		NorthstarSecretKey:      "SK",
+		NorthstarDefaultDomain:  "spx3.net",
+		NorthstarInternalDomain: "compute.internal",
+		AdvertiseIP:             "10.0.0.5",
+		BindIP:                  "0.0.0.0",
+	}
+
+	tests := []struct {
+		name       string
+		settings   admin.ConfigSettings
+		skip       bool
+		wantOK     bool
+		wantParams hostdns.Params
+	}{
+		{
+			name:     "enabled selects AdvertiseIP over BindIP",
+			settings: enabled,
+			wantOK:   true,
+			wantParams: hostdns.Params{
+				ResolverIP:     "10.0.0.5",
+				BaseDomain:     "spx3.net",
+				InternalDomain: "compute.internal",
+			},
+		},
+		{
+			name: "falls back to BindIP when AdvertiseIP is unset",
+			settings: func() admin.ConfigSettings {
+				s := enabled
+				s.AdvertiseIP = ""
+				s.BindIP = "10.0.0.9"
+				return s
+			}(),
+			wantOK: true,
+			wantParams: hostdns.Params{
+				ResolverIP:     "10.0.0.9",
+				BaseDomain:     "spx3.net",
+				InternalDomain: "compute.internal",
+			},
+		},
+		{
+			name:     "opt-out skips even when enabled",
+			settings: enabled,
+			skip:     true,
+			wantOK:   false,
+		},
+		{
+			name: "northstar disabled skips",
+			settings: func() admin.ConfigSettings {
+				s := enabled
+				s.NorthstarSecretKey = ""
+				return s
+			}(),
+			wantOK: false,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got, ok := hostDNSParams(tt.settings, tt.skip)
+			assert.Equal(t, tt.wantOK, ok)
+			if tt.wantOK {
+				assert.Equal(t, tt.wantParams, got)
+			}
+		})
+	}
+}
+
+// The image is written into a root volume of exactly the size this returns, so
+// anything less than the image size truncates the image and the guest never
+// finds its root. The old flooring conversion undersized every non-round image.
+func TestAMIVolumeSizeGiB(t *testing.T) {
+	const giB int64 = 1024 * 1024 * 1024
+
+	tests := []struct {
+		name  string
+		bytes int64
+		want  uint64
+	}{
+		// The regression: a real 4.94 GiB mkosi image floored to 4 GiB and the
+		// guest hung with no root partition.
+		{name: "non-round image rounds up", bytes: 5306470400, want: 5},
+		// Exact multiples must not gain a spare GiB — the round sizes are the
+		// common case and were the only ones the old code got right.
+		{name: "exact multiple is unchanged", bytes: 16 * giB, want: 16},
+		{name: "one byte over a multiple rounds up", bytes: 16*giB + 1, want: 17},
+		{name: "one byte under a multiple rounds up", bytes: 16*giB - 1, want: 16},
+		// Sub-GiB images still need a whole GiB to live in.
+		{name: "sub-GiB image gets a full GiB", bytes: 1, want: 1},
+		{name: "empty image", bytes: 0, want: 0},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := amiVolumeSizeGiB(tt.bytes)
+			assert.Equal(t, tt.want, got)
+			// The invariant the guest depends on, stated directly.
+			if tt.bytes > 0 {
+				assert.GreaterOrEqual(t, int64(got)*giB, tt.bytes,
+					"volume must be large enough to hold the image")
+			}
+		})
+	}
+}

cmd/spinifex/cmd/cluster.go

@@ -123,7 +123,7 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
 		phaseStart := time.Now()
 		fmt.Printf("[%s] Sending to %d node(s)...\n", strings.ToUpper(phase), nodeCount)
 
-		acks, err := collectShutdownACKs(nc, topic, reqData, nodeCount, timeout)
+		acks, err := collectShutdownACKs(nc, topic, reqData, nodeCount, "", timeout)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "[%s] Error: %v\n", strings.ToUpper(phase), err)
 			if !force {
@@ -216,36 +216,14 @@ func runNodeDrainLocal(cmd *cobra.Command, args []string) {
 // from the local node, ignoring ACKs from any other node. Returns an error if
 // the local node does not respond within timeout.
 func collectLocalShutdownACK(nc *nats.Conn, topic string, reqData []byte, node string, timeout time.Duration) (daemon.ShutdownACK, error) {
-	inbox := nats.NewInbox()
-	sub, err := nc.SubscribeSync(inbox)
+	acks, err := collectShutdownACKs(nc, topic, reqData, 1, node, timeout)
 	if err != nil {
-		return daemon.ShutdownACK{}, fmt.Errorf("failed to subscribe to inbox: %w", err)
+		return daemon.ShutdownACK{}, err
 	}
-	defer sub.Unsubscribe()
-
-	if err := nc.PublishRequest(topic, inbox, reqData); err != nil {
-		return daemon.ShutdownACK{}, fmt.Errorf("failed to publish request: %w", err)
-	}
-	nc.Flush()
-
-	deadline := time.Now().Add(timeout)
-	for {
-		remaining := time.Until(deadline)
-		if remaining <= 0 {
-			return daemon.ShutdownACK{}, fmt.Errorf("timeout waiting for local node %q ACK", node)
-		}
-		msg, err := sub.NextMsg(remaining)
-		if err != nil {
-			return daemon.ShutdownACK{}, fmt.Errorf("timeout waiting for local node %q ACK", node)
-		}
-		var ack daemon.ShutdownACK
-		if err := json.Unmarshal(msg.Data, &ack); err != nil {
-			continue
-		}
-		if ack.Node == node {
-			return ack, nil
-		}
+	if len(acks) == 0 {
+		return daemon.ShutdownACK{}, fmt.Errorf("timeout waiting for local node %q ACK", node)
 	}
+	return acks[0], nil
 }
 
 // runClusterDrainDHCP asks every vpcd to DHCPRELEASE all external-pool leases
@@ -322,7 +300,9 @@ func drainDHCPLeases(nc *nats.Conn, timeout time.Duration) (released, responders
 }
 
 // collectShutdownACKs publishes a shutdown request and collects ACKs from nodes.
-func collectShutdownACKs(nc *nats.Conn, topic string, reqData []byte, nodeCount int, timeout time.Duration) ([]daemon.ShutdownACK, error) {
+// When nodeFilter is non-empty, only ACKs from that node count toward nodeCount
+// and all others are ignored (used by the single-node local-drain path).
+func collectShutdownACKs(nc *nats.Conn, topic string, reqData []byte, nodeCount int, nodeFilter string, timeout time.Duration) ([]daemon.ShutdownACK, error) {
 	inbox := nats.NewInbox()
 	sub, err := nc.SubscribeSync(inbox)
 	if err != nil {
@@ -350,6 +330,9 @@ func collectShutdownACKs(nc *nats.Conn, topic string, reqData []byte, nodeCount
 		if err := json.Unmarshal(msg.Data, &ack); err != nil {
 			continue
 		}
+		if nodeFilter != "" && ack.Node != nodeFilter {
+			continue
+		}
 		acks = append(acks, ack)
 	}
 	return acks, nil

cmd/spinifex/cmd/get.go

@@ -1,19 +1,3 @@
-/*
-Copyright © 2026 Mulga Defense Corporation
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
-
-You should have received a copy of the GNU Affero General Public License
-along with this program.  If not, see <https://www.gnu.org/licenses/>.
-*/
 package cmd
 
 import (
@@ -154,6 +138,12 @@ func formatRoles(resp types.NodeStatusResponse) string {
 	if resp.NATSRole != "" {
 		roles = append(roles, "nats:"+resp.NATSRole)
 	}
+	if resp.OVNNBRole != "" {
+		roles = append(roles, "ovn-nb:"+resp.OVNNBRole)
+	}
+	if resp.OVNSBRole != "" {
+		roles = append(roles, "ovn-sb:"+resp.OVNSBRole)
+	}
 	if resp.PredastoreRole != "" {
 		roles = append(roles, "predastore:"+resp.PredastoreRole)
 	}

cmd/spinifex/cmd/get_test.go

@@ -38,6 +38,26 @@ func TestFormatRoles(t *testing.T) {
 			resp: types.NodeStatusResponse{NATSRole: "leader", PredastoreRole: "leader"},
 			want: "nats:leader,predastore:leader",
 		},
+		{
+			name: "ovn roles only",
+			resp: types.NodeStatusResponse{OVNNBRole: "leader", OVNSBRole: "follower"},
+			want: "ovn-nb:leader,ovn-sb:follower",
+		},
+		{
+			name: "all four roles ordered",
+			resp: types.NodeStatusResponse{
+				NATSRole:       "leader",
+				OVNNBRole:      "leader",
+				OVNSBRole:      "follower",
+				PredastoreRole: "follower",
+			},
+			want: "nats:leader,ovn-nb:leader,ovn-sb:follower,predastore:follower",
+		},
+		{
+			name: "ovn-nb leader with nats, sb absent",
+			resp: types.NodeStatusResponse{NATSRole: "follower", OVNNBRole: "leader"},
+			want: "nats:follower,ovn-nb:leader",
+		},
 	}
 
 	for _, tt := range tests {

cmd/spinifex/cmd/northstar_template_test.go

@@ -0,0 +1,89 @@
+package cmd
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/admin"
+	toml "github.com/pelletier/go-toml/v2"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func renderTemplate(t *testing.T, tmpl string, settings admin.ConfigSettings) string {
+	t.Helper()
+	path := filepath.Join(t.TempDir(), "out.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, tmpl, settings))
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	var parsed map[string]any
+	require.NoError(t, toml.Unmarshal(data, &parsed), "rendered template must be valid TOML")
+	return string(data)
+}
+
+func northstarSettings() admin.ConfigSettings {
+	return admin.ConfigSettings{
+		Region:                  "ap-southeast-2",
+		BindIP:                  "0.0.0.0",
+		AdvertiseIP:             "192.168.1.31",
+		AccessKey:               "AKIASYSTEM",
+		SecretKey:               "SYSTEMSECRET",
+		NatsToken:               "NATSTOKEN",
+		PoolDNSServers:          []string{"1.1.1.1", "8.8.8.8"},
+		NorthstarAccessKey:      "AKIANORTHSTAR",
+		NorthstarSecretKey:      "NORTHSTARSECRET",
+		NorthstarBucket:         admin.NorthstarBucketName,
+		NorthstarDefaultDomain:  admin.NorthstarDefaultDomain,
+		NorthstarInternalDomain: admin.NorthstarInternalDomain,
+	}
+}
+
+func TestNorthstarTomlTemplate(t *testing.T) {
+	content := renderTemplate(t, northstarTomlTemplate, northstarSettings())
+
+	// Northstar binds :5300 on the wildcard and the privileged :53 on the node
+	// WAN IP (avoids the systemd-resolved stub on 127.0.0.53:53).
+	assert.Contains(t, content, `listen     = "0.0.0.0:5300,192.168.1.31:53"`)
+	assert.Contains(t, content, `default_domain = "spx3.net"`)
+	assert.Contains(t, content, `bucket     = "northstar"`)
+	assert.Contains(t, content, `access_key = "AKIANORTHSTAR"`)
+	assert.Contains(t, content, `secret_key = "NORTHSTARSECRET"`)
+	// 0.0.0.0 bind must resolve to a loopback S3 endpoint, not a dial to 0.0.0.0.
+	assert.Contains(t, content, `endpoint   = "https://127.0.0.1:8443"`)
+	assert.Contains(t, content, `nameservers = ["1.1.1.1:53", "8.8.8.8:53"]`)
+	assert.Contains(t, content, `internal_domain = "compute.internal"`)
+	var parsed struct {
+		Quotas struct {
+			Enabled              bool `toml:"enabled"`
+			RecordsPerHostedZone int  `toml:"records_per_hosted_zone"`
+		} `toml:"quotas"`
+	}
+	require.NoError(t, toml.Unmarshal([]byte(content), &parsed))
+	assert.False(t, parsed.Quotas.Enabled)
+	assert.Equal(t, 10_000, parsed.Quotas.RecordsPerHostedZone)
+	assert.NotContains(t, content, "nats_url")
+	assert.NotContains(t, content, "NATSTOKEN")
+}
+
+// When provisioned, predastore.toml gains the northstar bucket plus a scoped
+// read-only [[auth]] entry — the isolation guarantee.
+func TestPredastoreTemplateWithNorthstar(t *testing.T) {
+	content := renderTemplate(t, predastoreTomlTemplate, northstarSettings())
+
+	assert.Contains(t, content, `name = "northstar"`)
+	assert.Contains(t, content, `access_key_id = "AKIANORTHSTAR"`)
+	assert.Contains(t, content, `{ bucket = "northstar", actions = ["s3:ListBucket", "s3:GetObject"] }`)
+}
+
+// Without northstar creds (e.g. cluster-join paths), predastore.toml renders
+// exactly as before — no empty bucket/auth stanzas.
+func TestPredastoreTemplateWithoutNorthstar(t *testing.T) {
+	s := northstarSettings()
+	s.NorthstarAccessKey = ""
+	s.NorthstarSecretKey = ""
+	content := renderTemplate(t, predastoreTomlTemplate, s)
+
+	assert.NotContains(t, content, `name = "northstar"`)
+	assert.NotContains(t, content, "AKIANORTHSTAR")
+}

cmd/spinifex/cmd/paths.go

@@ -8,7 +8,7 @@ import (
 
 // DefaultConfigDir returns the default configuration directory.
 // Production: /etc/spinifex (when /etc/spinifex exists)
-// Development: ~/spinifex/config
+// Development: ~/spinifex/config.
 func DefaultConfigDir() string {
 	if isProductionLayout() {
 		return "/etc/spinifex"
@@ -18,7 +18,7 @@ func DefaultConfigDir() string {
 
 // DefaultDataDir returns the default data directory.
 // Production: /var/lib/spinifex
-// Development: ~/spinifex
+// Development: ~/spinifex.
 func DefaultDataDir() string {
 	if isProductionLayout() {
 		return "/var/lib/spinifex"
@@ -28,7 +28,7 @@ func DefaultDataDir() string {
 
 // LogDirFor returns the log directory for a given data directory.
 // Production: /var/log/spinifex (matches systemd ReadWritePaths)
-// Development: <dataDir>/logs (supports custom per-node data dirs)
+// Development: <dataDir>/logs (supports custom per-node data dirs).
 func LogDirFor(dataDir string) string {
 	if isProductionLayout() {
 		return "/var/log/spinifex"

cmd/spinifex/cmd/root.go

@@ -1,19 +1,3 @@
-/*
-Copyright © 2026 Mulga Defense Corporation
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
-
-You should have received a copy of the GNU Affero General Public License
-along with this program.  If not, see <https://www.gnu.org/licenses/>.
-*/
 package cmd
 
 import (
@@ -31,7 +15,7 @@ var (
 	appConfig *config.ClusterConfig
 )
 
-// rootCmd represents the base command when called without any subcommands
+// rootCmd represents the base command when called without any subcommands.
 var rootCmd = &cobra.Command{
 	Use:   "spx",
 	Short: "Spinifex - Open source AWS-compatible platform for secure edge deployments",

cmd/spinifex/cmd/service.go

@@ -1,32 +1,22 @@
-/*
-Copyright © 2026 Mulga Defense Corporation
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
-
-You should have received a copy of the GNU Affero General Public License
-along with this program.  If not, see <https://www.gnu.org/licenses/>.
-*/
 package cmd
 
 import (
+	"context"
 	"fmt"
 	"log/slog"
 	"os"
 	"strings"
 
 	"github.com/mulgadc/spinifex/spinifex/config"
+	handlers_dns "github.com/mulgadc/spinifex/spinifex/handlers/dns"
+	"github.com/mulgadc/spinifex/spinifex/network/external"
+	"github.com/mulgadc/spinifex/spinifex/otelsetup"
 	"github.com/mulgadc/spinifex/spinifex/service"
 	"github.com/mulgadc/spinifex/spinifex/services/awsgw"
 	"github.com/mulgadc/spinifex/spinifex/services/nats"
+	"github.com/mulgadc/spinifex/spinifex/services/northstar"
 	"github.com/mulgadc/spinifex/spinifex/services/predastore"
+	"github.com/mulgadc/spinifex/spinifex/services/qmpcollector"
 	"github.com/mulgadc/spinifex/spinifex/services/spinifexui"
 	"github.com/mulgadc/spinifex/spinifex/services/viperblockd"
 	"github.com/mulgadc/spinifex/spinifex/vpcd"
@@ -39,6 +29,28 @@ var serviceCmd = &cobra.Command{
 	Short: "Manage Spinifex services",
 }
 
+// initTelemetry installs the JSON slog default (trace-stamping) and the OTel
+// providers for a service process. The returned func flushes exporters and
+// must be deferred; with no OTLP endpoint configured both are no-ops.
+func initTelemetry(serviceName string, debug bool) func() {
+	level := slog.LevelInfo
+	if debug {
+		level = slog.LevelDebug
+	}
+	otelsetup.SetDefaultJSONLogger(level)
+
+	shutdown, err := otelsetup.Init(context.Background(), serviceName)
+	if err != nil {
+		slog.Warn("otel init", "service", serviceName, "error", err)
+		return func() {}
+	}
+	return func() {
+		if err := shutdown(context.Background()); err != nil {
+			slog.Warn("otel shutdown", "service", serviceName, "error", err)
+		}
+	}
+}
+
 var predastoreCmd = &cobra.Command{
 	Use:   "predastore",
 	Short: "Manage the predastore service",
@@ -126,6 +138,8 @@ var predastoreStartCmd = &cobra.Command{
 		pprofEnabled := viper.GetBool("pprof")
 		pprofOutput := viper.GetString("pprof-output")
 
+		defer initTelemetry("predastore", debug)()
+
 		service, err := service.New("predastore", &predastore.Config{
 			Port:       port,
 			Host:       host,
@@ -189,7 +203,7 @@ var predastoreStatusCmd = &cobra.Command{
 	},
 }
 
-// Repeat for viperblock
+// Repeat for viperblock.
 var viperblockStartCmd = &cobra.Command{
 	Use:   "start",
 	Short: "Start the viperblock service",
@@ -283,11 +297,21 @@ var viperblockStartCmd = &cobra.Command{
 			shardWAL = *nodeConfig.Viperblock.ShardWAL
 		}
 
+		// Resolve chunk GC setting: default false unless explicitly set to true.
+		// GC is not a per-volume toggle — flipping it on applies to every VB this
+		// node constructs (nbdkit plugin + volume service).
+		gcEnabled := false
+		if nodeConfig.Viperblock.GCEnabled != nil {
+			gcEnabled = *nodeConfig.Viperblock.GCEnabled
+		}
+
 		encryptionKeyFile := nodeConfig.Viperblock.EncryptionKeyFile
 		if envKey := viper.GetString("viperblock-encryption-key-file"); envKey != "" {
 			encryptionKeyFile = envKey
 		}
 
+		defer initTelemetry("viperblockd", false)()
+
 		service, err := service.New("viperblock", &viperblockd.Config{
 			NatsHost:          nodeConfig.NATS.Host,
 			NatsToken:         nodeConfig.NATS.ACL.Token,
@@ -301,6 +325,7 @@ var viperblockStartCmd = &cobra.Command{
 			BaseDir:           nodeConfig.Predastore.BaseDir,
 			NodeName:          clusterConfig.Node,
 			ShardWAL:          shardWAL,
+			GCEnabled:         gcEnabled,
 			EncryptionKeyFile: encryptionKeyFile,
 		})
 
@@ -351,7 +376,7 @@ var viperblockStatusCmd = &cobra.Command{
 	},
 }
 
-// Repeat for nats
+// Repeat for nats.
 var natsStartCmd = &cobra.Command{
 	Use:   "start",
 	Short: "Start the nats service",
@@ -366,6 +391,8 @@ var natsStartCmd = &cobra.Command{
 
 		cfgFile := viper.GetString("config")
 
+		defer initTelemetry("nats", debug)()
+
 		service, err := service.New("nats", &nats.Config{
 			ConfigFile: cfgFile,
 			Port:       port,
@@ -418,7 +445,7 @@ var natsStatusCmd = &cobra.Command{
 	},
 }
 
-// Repeat for spinifex
+// Repeat for spinifex.
 var spinifexStartCmd = &cobra.Command{
 	Use:   "start",
 	Short: "Start the spinifex service",
@@ -459,6 +486,8 @@ var spinifexStartCmd = &cobra.Command{
 		// Apply changes back to cluster config
 		clusterConfig.Nodes[clusterConfig.Node] = nodeConfig
 
+		defer initTelemetry("spinifex-daemon", false)()
+
 		svc, err := service.New("spinifex", clusterConfig)
 
 		if err != nil {
@@ -564,6 +593,8 @@ var awsgwStartCmd = &cobra.Command{
 		// Apply changes back to cluster config
 		clusterConfig.Nodes[clusterConfig.Node] = nodeConfig
 
+		defer initTelemetry("awsgw", viper.GetBool("debug"))()
+
 		awsgw.SetBuildInfo(Version, Commit)
 		service, err := service.New("awsgw", clusterConfig)
 
@@ -621,6 +652,8 @@ var spinifexUIStartCmd = &cobra.Command{
 		tlsCert := viper.GetString("spinifex-ui-tls-cert")
 		tlsKey := viper.GetString("spinifex-ui-tls-key")
 
+		defer initTelemetry("spinifex-ui", false)()
+
 		svc, err := service.New("spinifex-ui", &spinifexui.Config{
 			Port:    port,
 			Host:    host,
@@ -730,12 +763,13 @@ var vpcdStartCmd = &cobra.Command{
 		nodeConfig := clusterConfig.Nodes[clusterConfig.Node]
 
 		// Map cluster-wide external pools to vpcd config
-		var extPools []vpcd.ExternalPoolConfig
+		var extPools []external.ExternalPoolConfig
 		for _, p := range clusterConfig.Network.ExternalPools {
-			extPools = append(extPools, vpcd.ExternalPoolConfig{
+			extPools = append(extPools, external.ExternalPoolConfig{
 				Name:            p.Name,
 				Source:          p.Source,
 				BindBridge:      p.BindBridge,
+				DHCPMAC:         p.DHCPMAC,
 				RangeStart:      p.RangeStart,
 				RangeEnd:        p.RangeEnd,
 				Gateway:         p.Gateway,
@@ -767,20 +801,26 @@ var vpcdStartCmd = &cobra.Command{
 			nodeConfig.BaseDir = baseDir
 		}
 
+		defer initTelemetry("vpcd", false)()
+
 		svc, err := service.New("vpcd", &vpcd.Config{
-			NatsHost:          nodeConfig.NATS.Host,
-			NatsToken:         nodeConfig.NATS.ACL.Token,
-			NatsCACert:        nodeConfig.NATS.CACert,
-			OVNNBAddr:         nodeConfig.VPCD.OVNNBAddr,
-			OVNSBAddr:         nodeConfig.VPCD.OVNSBAddr,
-			BaseDir:           nodeConfig.BaseDir,
-			Debug:             false,
-			ExternalMode:      clusterConfig.Network.ExternalMode,
-			ExternalPools:     extPools,
-			Bootstrap:         bootstrap,
-			ExternalInterface: nodeConfig.VPCD.ExternalInterface,
-			BridgeMode:        nodeConfig.VPCD.BridgeMode,
-			AZ:                nodeConfig.AZ,
+			NatsHost:                nodeConfig.NATS.Host,
+			NatsToken:               nodeConfig.NATS.ACL.Token,
+			NatsCACert:              nodeConfig.NATS.CACert,
+			OVNNBAddr:               nodeConfig.VPCD.OVNNBAddr,
+			OVNSBAddr:               nodeConfig.VPCD.OVNSBAddr,
+			BaseDir:                 nodeConfig.BaseDir,
+			Debug:                   false,
+			ExternalMode:            clusterConfig.Network.ExternalMode,
+			ExternalPools:           extPools,
+			Bootstrap:               bootstrap,
+			ExternalInterface:       nodeConfig.VPCD.ExternalInterface,
+			BridgeMode:              nodeConfig.VPCD.BridgeMode,
+			AZ:                      nodeConfig.AZ,
+			NorthstarBaseDomain:     handlers_dns.ResolveBaseDomain(&nodeConfig),
+			NorthstarInternalDomain: handlers_dns.ResolveInternalDomain(&nodeConfig),
+			ResolverNameservers:     handlers_dns.ResolverNameserverIPs(clusterConfig),
+			NATExemptCIDRs:          clusterConfig.Network.NATExemptCIDRs,
 		})
 		if err != nil {
 			fmt.Println("Error starting vpcd service:", err)
@@ -823,6 +863,228 @@ var vpcdStatusCmd = &cobra.Command{
 	},
 }
 
+var northstarCmd = &cobra.Command{
+	Use:   "northstar",
+	Short: "Manage the northstar (DNS) service",
+}
+
+// northstarStartOptions contains the command inputs that select Northstar configuration.
+type northstarStartOptions struct {
+	configFile      string
+	configOverride  string
+	baseDirOverride string
+}
+
+// northstarStarter is the service operation required by Northstar activation.
+type northstarStarter interface {
+	Start() (int, error)
+}
+
+// northstarStartDependencies isolates startup side effects for behavioral tests.
+type northstarStartDependencies struct {
+	loadConfig        func(string) (*config.ClusterConfig, error)
+	bootstrapBaseZone func(string, *config.ClusterConfig) error
+	newService        func(*northstar.Config) (northstarStarter, error)
+}
+
+var northstarStartCmd = &cobra.Command{
+	Use:           "start",
+	Short:         "Start the northstar service",
+	SilenceErrors: true,
+	SilenceUsage:  true,
+	RunE: func(cmd *cobra.Command, args []string) error {
+		fmt.Println("Starting northstar service...")
+
+		defer initTelemetry("northstar", false)()
+
+		return runNorthstarStart(northstarStartOptions{
+			configFile:      viper.GetString("config"),
+			configOverride:  viper.GetString("northstar-config"),
+			baseDirOverride: viper.GetString("base-dir"),
+		}, northstarStartDependencies{
+			loadConfig:        loadRequiredClusterConfig,
+			bootstrapBaseZone: northstar.BootstrapBaseZone,
+			newService: func(cfg *northstar.Config) (northstarStarter, error) {
+				return service.New("northstar", cfg)
+			},
+		})
+	},
+}
+
+// loadRequiredClusterConfig loads the service's explicit cluster configuration.
+func loadRequiredClusterConfig(path string) (*config.ClusterConfig, error) {
+	if path == "" {
+		return nil, fmt.Errorf("config file is not set")
+	}
+	if _, err := os.Stat(path); err != nil {
+		return nil, fmt.Errorf("stat cluster config %s: %w", path, err)
+	}
+	clusterConfig, err := config.LoadConfig(path)
+	if err != nil {
+		return nil, fmt.Errorf("load cluster config %s: %w", path, err)
+	}
+	return clusterConfig, nil
+}
+
+// runNorthstarStart starts configured Northstar nodes and cleanly skips nodes where it is optional.
+func runNorthstarStart(options northstarStartOptions, deps northstarStartDependencies) error {
+	clusterConfig, err := deps.loadConfig(options.configFile)
+	if err != nil {
+		return err
+	}
+
+	nodeConfig, ok := clusterConfig.Nodes[clusterConfig.Node]
+	if !ok {
+		return fmt.Errorf("node %q not found in cluster config", clusterConfig.Node)
+	}
+
+	configPath := resolveNorthstarConfigPath(nodeConfig.Northstar.ConfigPath, options.configOverride)
+	if options.configOverride != "" {
+		fmt.Println("Overwriting northstar config path to:", options.configOverride)
+	}
+	if configPath == "" {
+		// An absent path disables this optional service and must not trigger systemd restarts.
+		slog.Info("northstar is not configured on this node; skipping service start")
+		return nil
+	}
+
+	baseDir := nodeConfig.BaseDir
+	if options.baseDirOverride != "" {
+		baseDir = options.baseDirOverride
+	}
+
+	// Seed the default_domain base zone before the read-only daemon starts.
+	// The S3 polling path remains the backstop when this best-effort seed fails.
+	if err := deps.bootstrapBaseZone(configPath, clusterConfig); err != nil {
+		slog.Warn("northstar base zone bootstrap failed (continuing)", "error", err)
+	}
+
+	svc, err := deps.newService(&northstar.Config{
+		ConfigPath: configPath,
+		BasePath:   baseDir,
+		NodeID:     nodeConfig.Predastore.NodeID,
+		NatsHost:   nodeConfig.NATS.Host,
+		NatsToken:  nodeConfig.NATS.ACL.Token,
+		NatsCACert: nodeConfig.NATS.CACert,
+	})
+	if err != nil {
+		return fmt.Errorf("create northstar service: %w", err)
+	}
+
+	if _, err := svc.Start(); err != nil {
+		return fmt.Errorf("start northstar service: %w", err)
+	}
+	fmt.Println("northstar service started")
+	return nil
+}
+
+// resolveNorthstarConfigPath applies the explicit override before the node config.
+func resolveNorthstarConfigPath(nodePath, override string) string {
+	if override != "" {
+		return override
+	}
+	return nodePath
+}
+
+var northstarStopCmd = &cobra.Command{
+	Use:   "stop",
+	Short: "Stop the northstar service",
+	Run: func(cmd *cobra.Command, args []string) {
+		fmt.Println("Stopping northstar service...")
+
+		svc, err := service.New("northstar", &northstar.Config{})
+		if err != nil {
+			fmt.Println("Error stopping northstar service:", err)
+			return
+		}
+
+		if err = svc.Stop(); err != nil {
+			fmt.Println("Error stopping northstar service:", err)
+			os.Exit(1)
+		}
+		fmt.Println("northstar service stopped")
+	},
+}
+
+var northstarStatusCmd = &cobra.Command{
+	Use:   "status",
+	Short: "Get status of the northstar service",
+	Run: func(cmd *cobra.Command, args []string) {
+		fmt.Println("northstar service status: ...")
+	},
+}
+
+var qmpCollectorCmd = &cobra.Command{
+	Use:   "qmp-collector",
+	Short: "Manage the qmp-collector (guest metrics) service",
+}
+
+var qmpCollectorStartCmd = &cobra.Command{
+	Use:   "start",
+	Short: "Start the qmp-collector service",
+	Run: func(cmd *cobra.Command, args []string) {
+		fmt.Println("Starting qmp-collector service...")
+
+		cfgFile := viper.GetString("config")
+		if cfgFile == "" {
+			fmt.Println("Config file is not set")
+			return
+		}
+		clusterConfig, err := config.LoadConfig(cfgFile)
+		if err != nil {
+			fmt.Println("Error loading config file:", err)
+			return
+		}
+		nodeConfig := clusterConfig.Nodes[clusterConfig.Node]
+
+		defer initTelemetry("qmp-collector", false)()
+
+		svc, err := service.New("qmp-collector", &qmpcollector.Config{
+			NatsHost:   nodeConfig.NATS.Host,
+			NatsToken:  nodeConfig.NATS.ACL.Token,
+			NatsCACert: nodeConfig.NATS.CACert,
+			BaseDir:    nodeConfig.BaseDir,
+			NodeName:   clusterConfig.Node,
+		})
+		if err != nil {
+			fmt.Println("Error starting qmp-collector service:", err)
+			return
+		}
+		if _, err = svc.Start(); err != nil {
+			fmt.Println("Error starting qmp-collector service:", err)
+			os.Exit(1)
+		}
+		fmt.Println("qmp-collector service started")
+	},
+}
+
+var qmpCollectorStopCmd = &cobra.Command{
+	Use:   "stop",
+	Short: "Stop the qmp-collector service",
+	Run: func(cmd *cobra.Command, args []string) {
+		fmt.Println("Stopping qmp-collector service...")
+
+		svc, err := service.New("qmp-collector", &qmpcollector.Config{})
+		if err != nil {
+			fmt.Println("Error stopping qmp-collector service:", err)
+			return
+		}
+		if err = svc.Stop(); err != nil {
+			fmt.Println("Error stopping qmp-collector service:", err)
+			os.Exit(1)
+		}
+		fmt.Println("qmp-collector service stopped")
+	},
+}
+
+var qmpCollectorStatusCmd = &cobra.Command{
+	Use:   "status",
+	Short: "Get status of the qmp-collector service",
+	Run: func(cmd *cobra.Command, args []string) {
+		fmt.Println("qmp-collector service status: ...")
+	},
+}
+
 func init() {
 	viper.SetEnvPrefix("SPINIFEX") // Prefix for environment variables
 	viper.SetEnvKeyReplacer(strings.NewReplacer("-", "_"))
@@ -873,11 +1135,6 @@ func init() {
 	viper.BindEnv("encryption-key-file", "SPINIFEX_PREDASTORE_ENCRYPTION_KEY_FILE")
 	viper.BindPFlag("encryption-key-file", predastoreCmd.PersistentFlags().Lookup("encryption-key-file"))
 
-	// Predastore Backend
-	predastoreCmd.PersistentFlags().String("backend", "distributed", "Predastore (S3) backend")
-	viper.BindEnv("backend", "SPINIFEX_PREDASTORE_BACKEND")
-	viper.BindPFlag("backend", predastoreCmd.PersistentFlags().Lookup("backend"))
-
 	// Predastore Node ID. Default -1 is dev mode (launch every configured
 	// QUIC node in-process). Production deployments set this to the node's
 	// real ID (>= 1) via SPINIFEX_PREDASTORE_NODE_ID or --node-id.
@@ -1022,4 +1279,22 @@ func init() {
 	vpcdCmd.AddCommand(vpcdStartCmd)
 	vpcdCmd.AddCommand(vpcdStopCmd)
 	vpcdCmd.AddCommand(vpcdStatusCmd)
+
+	// northstar
+	serviceCmd.AddCommand(northstarCmd)
+
+	northstarCmd.PersistentFlags().String("northstar-config", "", "Path to northstar.toml (overrides config file)")
+	viper.BindEnv("northstar-config", "SPINIFEX_NORTHSTAR_CONFIG")
+	viper.BindPFlag("northstar-config", northstarCmd.PersistentFlags().Lookup("northstar-config"))
+
+	northstarCmd.AddCommand(northstarStartCmd)
+	northstarCmd.AddCommand(northstarStopCmd)
+	northstarCmd.AddCommand(northstarStatusCmd)
+
+	// qmp-collector
+	serviceCmd.AddCommand(qmpCollectorCmd)
+
+	qmpCollectorCmd.AddCommand(qmpCollectorStartCmd)
+	qmpCollectorCmd.AddCommand(qmpCollectorStopCmd)
+	qmpCollectorCmd.AddCommand(qmpCollectorStatusCmd)
 }

cmd/spinifex/cmd/service_test.go

@@ -0,0 +1,207 @@
+package cmd
+
+import (
+	"errors"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/config"
+	"github.com/mulgadc/spinifex/spinifex/services/northstar"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+type stubNorthstarService struct {
+	starts   int
+	startErr error
+}
+
+func (s *stubNorthstarService) Start() (int, error) {
+	s.starts++
+	return 0, s.startErr
+}
+
+func TestRunNorthstarStartActivation(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name           string
+		nodePath       string
+		override       string
+		wantPath       string
+		wantConfigured bool
+	}{
+		{
+			name: "unconfigured node is skipped",
+		},
+		{
+			name:           "node config starts service",
+			nodePath:       "/etc/spinifex/northstar/northstar.toml",
+			wantPath:       "/etc/spinifex/northstar/northstar.toml",
+			wantConfigured: true,
+		},
+		{
+			name:           "override starts service",
+			override:       "/run/spinifex/northstar.toml",
+			wantPath:       "/run/spinifex/northstar.toml",
+			wantConfigured: true,
+		},
+		{
+			name:           "override takes precedence",
+			nodePath:       "/etc/spinifex/northstar/northstar.toml",
+			override:       "/run/spinifex/northstar.toml",
+			wantPath:       "/run/spinifex/northstar.toml",
+			wantConfigured: true,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			clusterConfig := &config.ClusterConfig{
+				Node: "node1",
+				Nodes: map[string]config.Config{
+					"node1": {
+						BaseDir:   "/var/lib/spinifex/northstar",
+						Northstar: config.NorthstarConfig{ConfigPath: tt.nodePath},
+					},
+				},
+			}
+			stub := &stubNorthstarService{}
+			bootstrapCalls := 0
+			factoryCalls := 0
+			var startedConfig *northstar.Config
+			deps := northstarStartDependencies{
+				loadConfig: func(string) (*config.ClusterConfig, error) {
+					return clusterConfig, nil
+				},
+				bootstrapBaseZone: func(path string, cfg *config.ClusterConfig) error {
+					bootstrapCalls++
+					assert.Equal(t, tt.wantPath, path)
+					assert.Same(t, clusterConfig, cfg)
+					return nil
+				},
+				newService: func(cfg *northstar.Config) (northstarStarter, error) {
+					factoryCalls++
+					startedConfig = cfg
+					return stub, nil
+				},
+			}
+
+			err := runNorthstarStart(northstarStartOptions{
+				configFile:     "/etc/spinifex/spinifex.toml",
+				configOverride: tt.override,
+			}, deps)
+			require.NoError(t, err)
+
+			if !tt.wantConfigured {
+				assert.Zero(t, bootstrapCalls)
+				assert.Zero(t, factoryCalls)
+				assert.Zero(t, stub.starts)
+				assert.Nil(t, startedConfig)
+				return
+			}
+			assert.Equal(t, 1, bootstrapCalls)
+			assert.Equal(t, 1, factoryCalls)
+			assert.Equal(t, 1, stub.starts)
+			require.NotNil(t, startedConfig)
+			assert.Equal(t, tt.wantPath, startedConfig.ConfigPath)
+		})
+	}
+}
+
+func TestLoadRequiredClusterConfigRejectsMissingFile(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name string
+		path string
+	}{
+		{name: "empty path"},
+		{name: "missing path", path: t.TempDir() + "/missing.toml"},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+			_, err := loadRequiredClusterConfig(tt.path)
+			require.Error(t, err)
+		})
+	}
+}
+
+func TestRunNorthstarStartReturnsLoadFailure(t *testing.T) {
+	t.Parallel()
+
+	loadErr := errors.New("cluster config unavailable")
+	deps := northstarStartDependencies{
+		loadConfig: func(string) (*config.ClusterConfig, error) {
+			return nil, loadErr
+		},
+	}
+
+	err := runNorthstarStart(northstarStartOptions{configFile: "/etc/spinifex/spinifex.toml"}, deps)
+	require.ErrorIs(t, err, loadErr)
+}
+
+func TestRunNorthstarStartContinuesAfterBootstrapFailure(t *testing.T) {
+	t.Parallel()
+
+	bootstrapErr := errors.New("predastore unavailable")
+	stub := &stubNorthstarService{}
+	deps := northstarStartDependencies{
+		loadConfig: func(string) (*config.ClusterConfig, error) {
+			return &config.ClusterConfig{
+				Node: "node1",
+				Nodes: map[string]config.Config{
+					"node1": {Northstar: config.NorthstarConfig{ConfigPath: "/etc/spinifex/northstar/northstar.toml"}},
+				},
+			}, nil
+		},
+		bootstrapBaseZone: func(string, *config.ClusterConfig) error { return bootstrapErr },
+		newService: func(*northstar.Config) (northstarStarter, error) {
+			return stub, nil
+		},
+	}
+
+	err := runNorthstarStart(northstarStartOptions{configFile: "/etc/spinifex/spinifex.toml"}, deps)
+	require.NoError(t, err)
+	assert.Equal(t, 1, stub.starts)
+}
+
+func TestRunNorthstarStartRejectsMissingNode(t *testing.T) {
+	t.Parallel()
+
+	deps := northstarStartDependencies{
+		loadConfig: func(string) (*config.ClusterConfig, error) {
+			return &config.ClusterConfig{Node: "missing", Nodes: map[string]config.Config{}}, nil
+		},
+	}
+
+	err := runNorthstarStart(northstarStartOptions{configFile: "/etc/spinifex/spinifex.toml"}, deps)
+	require.EqualError(t, err, `node "missing" not found in cluster config`)
+}
+
+func TestRunNorthstarStartReturnsConfiguredFailure(t *testing.T) {
+	t.Parallel()
+
+	startErr := errors.New("configured startup failed")
+	stub := &stubNorthstarService{startErr: startErr}
+	deps := northstarStartDependencies{
+		loadConfig: func(string) (*config.ClusterConfig, error) {
+			return &config.ClusterConfig{
+				Node: "node1",
+				Nodes: map[string]config.Config{
+					"node1": {Northstar: config.NorthstarConfig{ConfigPath: "/etc/spinifex/northstar/northstar.toml"}},
+				},
+			}, nil
+		},
+		bootstrapBaseZone: func(string, *config.ClusterConfig) error { return nil },
+		newService: func(*northstar.Config) (northstarStarter, error) {
+			return stub, nil
+		},
+	}
+
+	err := runNorthstarStart(northstarStartOptions{configFile: "/etc/spinifex/spinifex.toml"}, deps)
+	require.ErrorIs(t, err, startErr)
+	assert.Equal(t, 1, stub.starts)
+}

cmd/spinifex/cmd/telemetry_test.go

@@ -0,0 +1,24 @@
+package cmd
+
+import "testing"
+
+func TestInitTelemetryNoEndpoint(t *testing.T) {
+	t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "")
+
+	tests := []struct {
+		name  string
+		debug bool
+	}{
+		{"info level", false},
+		{"debug level", true},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			flush := initTelemetry("test-svc", tt.debug)
+			if flush == nil {
+				t.Fatal("initTelemetry returned nil flush func")
+			}
+			flush()
+		})
+	}
+}

cmd/spinifex/cmd/templates/northstar.toml

@@ -0,0 +1,40 @@
+# Northstar DNS service configuration (TOML).
+# Written by `spx admin init`; read by the spinifex northstar service.
+# Contains bucket-scoped S3 credentials — keep mode 0600.
+
+# UDP+TCP bind addresses (comma separated). :5300 binds the wildcard for local
+# tooling; :53 binds the node WAN IP specifically (via CAP_NET_BIND_SERVICE) so
+# instances resolve against the node directly. Binding the WAN IP rather than
+# 0.0.0.0 avoids the systemd-resolved stub on 127.0.0.53:53. A future per-VPC
+# resolver VIP (OVN) will front :53.
+listen     = "0.0.0.0:5300,{{if .AdvertiseIP}}{{.AdvertiseIP}}{{else}}{{.BindIP}}{{end}}:53"
+dot_listen = ""
+doh_listen = ""
+tls_cert   = ""
+tls_key    = ""
+
+# Authoritative base domain for internal resource names.
+default_domain = "{{.NorthstarDefaultDomain}}"
+
+# AWS-parity private DNS zone for instance ip-<addr> records (IMDS hostname).
+internal_domain = "{{.NorthstarInternalDomain}}"
+
+# Seconds between S3 zone re-scans (anti-entropy; live changes arrive via NATS).
+sync_interval = 30
+
+# DNS (Route 53) service quotas (default off, mirrors [quota] in awsgw.toml).
+# Applied at daemon start; restart to change.
+[quotas]
+enabled = false
+records_per_hosted_zone = 10000   # AWS default; enforced only when enabled
+
+[s3]
+endpoint   = "https://{{if eq .BindIP "0.0.0.0"}}127.0.0.1{{else}}{{.BindIP}}{{end}}:8443"
+bucket     = "{{.NorthstarBucket}}"
+region     = "{{.Region}}"
+access_key = "{{.NorthstarAccessKey}}"
+secret_key = "{{.NorthstarSecretKey}}"
+
+# Forwarders for non-authoritative (recursive) queries. Empty => REFUSED.
+[upstream]
+nameservers = [{{range $i, $s := .PoolDNSServers}}{{if $i}}, {{end}}"{{$s}}:53"{{end}}]

cmd/spinifex/cmd/templates/predastore-multinode.toml

@@ -46,13 +46,40 @@ public = false
 encryption = ""
 account_id = "000000000000"
 
+{{- if .NorthstarAccessKey}}
+# Northstar DNS zone storage. Read-only for the northstar service; the control
+# plane writes zone files with the system credentials.
+[[buckets]]
+name = "{{.NorthstarBucket}}"
+region = "{{.Region}}"
+type = "distributed"
+public = false
+encryption = ""
+account_id = "000000000000"
+{{- end}}
+
 [[auth]]
 access_key_id = "{{.AccessKey}}"
 secret_access_key = "{{.SecretKey}}"
 account_id = "000000000000"
 policy = [
   { bucket = "predastore", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
+{{- if .NorthstarAccessKey}}
+  { bucket = "{{.NorthstarBucket}}", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"] },
+{{- end}}
 ]
+{{- if .NorthstarAccessKey}}
+
+# Bucket-scoped, read-only credentials for the northstar service (isolation
+# guarantee: ListBucket + GetObject on the northstar bucket only).
+[[auth]]
+access_key_id = "{{.NorthstarAccessKey}}"
+secret_access_key = "{{.NorthstarSecretKey}}"
+account_id = "000000000000"
+policy = [
+  { bucket = "{{.NorthstarBucket}}", actions = ["s3:ListBucket", "s3:GetObject"] },
+]
+{{- end}}
 
 # API request throttling (per-account, per-action token bucket)
 [ratelimit]

cmd/spinifex/cmd/templates/predastore.toml

@@ -75,13 +75,40 @@ public = false
 encryption = ""
 account_id = "000000000000"
 
+{{- if .NorthstarAccessKey}}
+# Northstar DNS zone storage. Read-only for the northstar service; the control
+# plane writes zone files with the system credentials.
+[[buckets]]
+name = "{{.NorthstarBucket}}"
+region = "{{.Region}}"
+type = "distributed"
+public = false
+encryption = ""
+account_id = "000000000000"
+{{- end}}
+
 [[auth]]
 access_key_id = "{{.AccessKey}}"
 secret_access_key = "{{.SecretKey}}"
 account_id = "000000000000"
 policy = [
   { bucket = "predastore", actions = ["s3:ListBucket",  "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
+{{- if .NorthstarAccessKey}}
+  { bucket = "{{.NorthstarBucket}}", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"] },
+{{- end}}
 ]
+{{- if .NorthstarAccessKey}}
+
+# Bucket-scoped, read-only credentials for the northstar service (isolation
+# guarantee: ListBucket + GetObject on the northstar bucket only).
+[[auth]]
+access_key_id = "{{.NorthstarAccessKey}}"
+secret_access_key = "{{.NorthstarSecretKey}}"
+account_id = "000000000000"
+policy = [
+  { bucket = "{{.NorthstarBucket}}", actions = ["s3:ListBucket", "s3:GetObject"] },
+]
+{{- end}}
 
 # API request throttling (per-account, per-action token bucket)
 [ratelimit]

cmd/spinifex/cmd/templates/spinifex.toml

@@ -16,28 +16,37 @@ internal_suffix = "spinifex.internal"
 ipsec_enabled = {{.IPSecEnabled}}
 {{- if .ExternalMode}}
 external_mode = "{{.ExternalMode}}"
+{{- range .Pools}}
 
 [[network.external_pools]]
-name        = "{{.PoolName}}"
-{{- if .PoolSource}}
-source      = "{{.PoolSource}}"
+name        = "{{.Name}}"
+{{- if .Source}}
+source      = "{{.Source}}"
 {{- end}}
-{{- if .PoolBindBridge}}
-bind_bridge = "{{.PoolBindBridge}}"
+{{- if .BindBridge}}
+bind_bridge = "{{.BindBridge}}"
 {{- end}}
-{{- if .PoolStart}}
-range_start = "{{.PoolStart}}"
-range_end   = "{{.PoolEnd}}"
+{{- if .DHCPMAC}}
+dhcp_mac    = "{{.DHCPMAC}}"
 {{- end}}
-{{- if .PoolGateway}}
-gateway     = "{{.PoolGateway}}"
+{{- if .Start}}
+range_start = "{{.Start}}"
+range_end   = "{{.End}}"
 {{- end}}
-{{- if .PoolGatewayIP}}
-gateway_ip  = "{{.PoolGatewayIP}}"
+{{- if .Gateway}}
+gateway     = "{{.Gateway}}"
+{{- end}}
+{{- if .GatewayIP}}
+gateway_ip  = "{{.GatewayIP}}"
+{{- end}}
+{{- if .GwLrpRangeStart}}
+gw_lrp_range_start = "{{.GwLrpRangeStart}}"
+gw_lrp_range_end   = "{{.GwLrpRangeEnd}}"
+{{- end}}
+prefix_len  = {{if .PrefixLen}}{{.PrefixLen}}{{else}}24{{end}}
+{{- if .DNSServers}}
+dns_servers = [{{range $i, $s := .DNSServers}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
 {{- end}}
-prefix_len  = {{if .PoolPrefixLen}}{{.PoolPrefixLen}}{{else}}24{{end}}
-{{- if .PoolDNSServers}}
-dns_servers = [{{range $i, $s := .PoolDNSServers}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
 {{- end}}
 {{- end}}
 {{- if .BootstrapVpcId}}
@@ -83,7 +92,7 @@ gpu_passthrough = true
 host = "{{.BindIP}}:9999"
 tlskey = "config/server.key"
 tlscert = "config/server.pem"
-debug = true
+debug = false
 config = "awsgw/awsgw.toml"
 
 [nodes.{{.Node}}.nats]
@@ -97,6 +106,7 @@ token = "{{.NatsToken}}"
 
 [nodes.{{.Node}}.viperblock]
 shardwal = false
+gc_enabled = true
 {{- if .EncryptionKeyFile}}
 encryption_key_file = "{{.EncryptionKeyFile}}"
 {{- end}}
@@ -107,6 +117,9 @@ ovn_sb_addr = "{{.OVNSBAddr}}"
 {{- if .ExternalIface}}
 external_interface = "{{.ExternalIface}}"
 {{- end}}
+{{- if .BridgeMode}}
+bridge_mode = "{{.BridgeMode}}"
+{{- end}}
 
 [nodes.{{.Node}}.predastore]
 host = "{{.BindIP}}:8443"
@@ -117,6 +130,14 @@ secretkey = "{{.SecretKey}}"
 {{- if gt .PredastoreNodeID 0}}
 node_id = {{.PredastoreNodeID}}
 {{- end}}
+
+{{- if .NorthstarConfigPath}}
+
+[nodes.{{.Node}}.northstar]
+config_path = "{{.NorthstarConfigPath}}"
+default_domain = "{{.NorthstarDefaultDomain}}"
+internal_domain = "{{.NorthstarInternalDomain}}"
+{{- end}}
 {{- range .RemoteNodes}}
 
 [nodes.{{.Name}}]
@@ -127,4 +148,9 @@ az = "{{.AZ}}"
 {{- if .Services}}
 services = [{{range $i, $s := .Services}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
 {{- end}}
+{{- if .NorthstarConfigPath}}
+
+[nodes.{{.Name}}.northstar]
+config_path = "{{.NorthstarConfigPath}}"
+{{- end}}
 {{- end}}

cmd/spinifex/cmd/top.go

@@ -1,19 +1,3 @@
-/*
-Copyright © 2026 Mulga Defense Corporation
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
-
-You should have received a copy of the GNU Affero General Public License
-along with this program.  If not, see <https://www.gnu.org/licenses/>.
-*/
 package cmd
 
 import (

cmd/spinifex/cmd/version.go

@@ -8,7 +8,7 @@ import (
 )
 
 // Version variables set via ldflags at build time.
-// Example: go build -ldflags "-X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=v1.0.0"
+// Example: go build -ldflags "-X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=v1.0.0".
 var (
 	Version = "dev"
 	Commit  = "unknown"

cmd/spinifex/main.go

@@ -1,19 +1,3 @@
-/*
-Copyright © 2026 Mulga Defense Corporation
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
-
-You should have received a copy of the GNU Affero General Public License
-along with this program.  If not, see <https://www.gnu.org/licenses/>.
-*/
 package main
 
 import (

docs/COMMANDS.md

@@ -31,6 +31,12 @@ Service lifecycle commands for starting, stopping, and checking status of all Sp
 | `spx service vpcd start` | — | Loads cluster config → starts VPC daemon (subscribes to `vpc.*` NATS events, translates to OVN logical switches/ports/routers) |
 | `spx service vpcd stop` | — | Stops the vpcd service |
 | `spx service vpcd status` | — | Reports vpcd service status |
+| `spx service northstar start` | `--northstar-config` (overrides `nodes.<node>.northstar.config_path`) | Loads cluster config → reads `northstar.toml` → starts the northstar DNS server (authoritative for internal `*.spx3.net`, recursive via upstream forwarders), syncing zones from its S3 bucket. Guests resolve via `169.254.169.253`, served by vpcd's per-tap DNS shim which forwards to northstar |
+| `spx service northstar stop` | — | Stops the northstar service |
+| `spx service northstar status` | — | Reports northstar service status |
+| `spx service qmp-collector start` | — | Starts the guest-metrics collector (polls per-VM telemetry QMP sockets + tap counters, publishes CloudWatch-shaped series to NATS `metrics.ec2.*`) |
+| `spx service qmp-collector stop` | — | Stops the qmp-collector service |
+| `spx service qmp-collector status` | — | Reports qmp-collector service status |
 
 ### Cluster Inspection
 
@@ -51,8 +57,8 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Flags | Description |
 |---------|-------|-------------|
-| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--token-ttl`, `--force` | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` |
-| `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services` | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations |
+| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--skip-host-dns`, `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; add `--external-pool start-end --external-gateway <ip>` or `--external-source=dhcp [--external-bind-bridge <iface>]` for a public pool with full EIP support; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` → points the host resolver at this node's local northstar so the Spinifex zones (ELB/EKS names) resolve from the node itself (opt out with `--skip-host-dns`). `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
+| `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services`, `--skip-host-dns` | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations → points the host resolver at this node's local northstar so the Spinifex zones resolve from the node itself (opt out with `--skip-host-dns`) |
 
 ### Version
 
@@ -93,8 +99,69 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 |---------|-------|-------------|
 | `spx admin images import` | `--name`, `--file`, `--distro`, `--version`, `--arch`, `--platform`, `--boot-mode` (bios/uefi/uefi-preferred), `--tag`, `--force`, `--skip-verify` | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, verify the SHA-256/SHA-512 digest, and inherit `BootMode` from the catalog entry. `--boot-mode` overrides the catalog value when set. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports skip checksum verification (operator-supplied media is outside Spinifex's trust boundary, the skip is logged at INFO for audit) and require an explicit `--boot-mode` because there is no catalog metadata to inherit from. `--skip-verify` bypasses verification for catalog imports and emits a WARN slog + stderr notice; use only for debugging or when upstream mirrors are confirmed-broken. On every import a best-effort `virt-customize` step bakes the deployment CA (`<data-root>/config/ca.pem`) into the image's trust store before the block copy, so in-guest SDK-over-TLS calls to Spinifex endpoints trust the gateway from first boot; an image libguestfs cannot customize is imported as-is (CA-free) and logs a skip. Because the CA is fixed into the image, rotating the cluster CA requires re-importing affected images. |
 | `spx admin images list` | — | Lists available OS images that can be imported or downloaded |
+| `spx admin images promote` | `--image-id` (required), `--yes` | Reads `ami-<id>/config.json`, validates the AMI is account-owned, then rewrites `ImageOwnerAlias` to `"system"` in-place. No block data is copied. The change takes effect immediately — the AMI becomes visible to all accounts via `DescribeImages`. Prompts for confirmation (skipped with `--yes`). Already-system AMIs are refused. |
 | `spx admin images remove` | `--image-id` (required), `--force`, `--yes` | Loads `ami-<id>/config.json`, walks transitive dependents — copied snapshots whose `VolumeID == imageID`, volumes whose `SnapshotID` references the internal `snap-ami-<id>` or any derived snap, and account AMIs created via `CopyImage` whose `SnapshotID` is a derived snap — then prompts (skipped with `--yes`) before deleting `ami-<id>/config.json` (the DescribeImages barrier) followed by the rest of `ami-<id>/` and `snap-ami-<id>/`. Account-owned AMIs are refused with a hint pointing at `aws ec2 deregister-image` + `aws ec2 delete-snapshot`. `--force` bypasses the dependency, ownership and config-corrupt checks for salvage of orphaned blocks. |
 
+#### Image integrity verification (CMMC SI.L1-3.14.2)
+
+Catalog imports (`spx admin images import --name <name>`) verify the image
+against the catalog-declared SHA-256/SHA-512 digest before extraction. The sums
+file is fetched from the catalog `Checksum` URL over HTTPS only (cross-scheme
+redirects refused), and verification runs on both fresh downloads and cache
+hits so a poisoned cache is caught on the next import.
+
+On mismatch the import exits non-zero, the cached file is left on disk for
+inspection, and the printed guidance is `spx admin images import --name <name>
+--force` to re-download.
+
+`--file` imports skip verification by design: operator-supplied media is
+outside Spinifex's trust boundary and the operator is responsible for
+integrity (e.g. `sha256sum` against a trusted upstream digest before import).
+The skip is recorded as an INFO `slog` event with `reason=local-file-import`
+so a CMMC assessor can audit the decision from journald.
+
+`--skip-verify` bypasses the checksum step for catalog imports. The command
+still downloads via the catalog URL but does not compare the image digest
+against the sums file. Intended for narrow cases such as debugging upstream
+mirror issues or running against a transiently-broken `latest/` path; the
+skip is logged at WARN with `reason=skip-verify-flag` and printed to stderr
+so operators and assessors see it. Prefer `--file` with an out-of-band
+verified image over `--skip-verify` whenever possible.
+
+**Limitation:** verification confirms the image matches the digest the mirror
+served. A mirror compromise that swaps both image and sums file is not
+detected; closing that gap requires GPG signature verification of the sums
+file, deferred to a later phase.
+
+#### `spx admin images remove` caveats
+
+Admin-imported AMIs (`ImageOwnerAlias = "system"`) live
+under the `ami-<id>/` S3 prefix and use a viperblock-internal snap checkpoint
+at `snap-ami-<id>/` — there is no `snap-<id>/metadata.json`. The AWS handlers
+(`DeregisterImage`, `DeleteSnapshot`) reject system owners with
+`UnauthorizedOperation`, which is the right behaviour for tenant API callers
+but leaves no AWS-flow path to reclaim space. `spx admin images remove` is
+the admin-trust-boundary counterpart that performs the dependency walk and
+hard-deletes the blocks directly against predastore.
+
+`CopyImage` of a system AMI is metadata-only: it writes a fresh
+`snap-<acct>/metadata.json` whose `VolumeID` points at `ami-<sys>` and a new
+`ami-<acct>/config.json` referencing that snap. Volumes launched from the
+copied AMI read transitively from `ami-<sys>/chunks/...`. The remove command
+walks this transitive set and refuses if anything references the target.
+
+**TOCTOU window:** between the safety scan and the `config.json` delete a
+concurrent `RunInstances` against the AMI could create a new dependent
+volume. The window is sub-second on a healthy cluster. The admin running
+this command is expected to know the fleet's operational state; if the race
+fires the result is a `vol-<id>` with deleted backing blocks, recovered by
+terminating the orphaned instance.
+
+**`--force` bypasses every safety check** (dependents, ownership, missing /
+corrupt `config.json`). Use only for salvage of orphaned blocks. Running it
+against a live system AMI corrupts every dependent volume on the next disk
+read.
+
 ### GPU Management
 
 | Command | Flags | Description |
@@ -107,6 +174,11 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 | `spx admin gpu mig enable` | `--profile <name>` (required, e.g. `1g.10gb`), `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → enables MIG mode on each target (`gpu.EnableMIGMode`) → lists available profiles (`gpu.ListProfiles`) and validates requested profile name → destroys any existing instances (`gpu.DestroyAllInstances`) → creates new instances filling GPU capacity (`gpu.CreateInstances`) → writes `mig_profile` to `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. |
 | `spx admin gpu mig disable` | `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → destroys all GPU instances (`gpu.DestroyAllInstances`) → disables MIG mode (`gpu.DisableMIGMode`) → clears `mig_profile` in `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. |
 
+### EKS Control-Plane Disaster Recovery
+
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin eks restore-snapshot` | `--cluster` (required), `--snapshot` (optional, defaults to the latest snapshot in predastore), `--account` (optional, defaults to the bootstrap account) | Single-CP total-loss DR path (fail-safe): validates the snapshot exists in predastore BEFORE any mutation (a typo'd/missing key hard-fails, never resets into an empty datastore) → launches a fresh control-plane VM as a cluster-init seed (replaying the persisted create-time launch template) → sets a required-snapshot `RecoveryDirective` (`cluster-reset`) so the boot-time recovery agent aborts rather than resets-into-empty if it cannot fetch the snapshot → persists the replacement in cluster meta BEFORE re-pointing the NLB (so an NLB failure is convergeable by the reconciler, returned as a provisional status, not a hard error) → re-points the cluster NLB's apiserver and konnectivity target groups from the old CP's ENI to the new one → fences the old CP with retries, failing loudly if it cannot be confirmed terminated (split-brain guard). Any failure before the meta commit unwinds the fresh CP (terminate + clear directive) so a re-run does not stack a second resetting control plane. The returned status is provisional — success means the sequence completed, not that etcd is restored and serving; verify cluster health. HA clusters (a spread with a potentially surviving quorum) are rejected — recover those via quorum reformation instead. |
 
 ## AWS-Compatible API
 
@@ -114,7 +186,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
-| `run-instances` | `--image-id`, `--instance-type`, `--count`, `--key-name`, `--user-data`, `--subnet-id`, `--security-group-ids`, `--tag-specifications` (instance-scoped), `--block-device-mappings` (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination), `--placement` (GroupName), `--iam-instance-profile` (Name/Arn), `--capacity-reservation-specification` (CapacityReservationTarget.CapacityReservationId, targeted-by-id only), `--metadata-options` (HttpPutResponseHopLimit; IMDSv2-only enforced — rejects `http-tokens=optional`) | `--dry-run`, `--client-token`, `--disable-api-termination`, `--ebs-optimized`, `--network-interfaces`, `--private-ip-address`, `--monitoring`, `--credit-specification`, `--cpu-options`, `--launch-template`, `--hibernate-options` | **DONE** |
+| `run-instances` | `--image-id`, `--instance-type`, `--count`, `--key-name`, `--user-data`, `--subnet-id`, `--security-group-ids`, `--tag-specifications` (instance-scoped), `--block-device-mappings` (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination), `--placement` (GroupName), `--iam-instance-profile` (Name/Arn), `--capacity-reservation-specification` (CapacityReservationTarget.CapacityReservationId, targeted-by-id only), `--metadata-options` (HttpPutResponseHopLimit; IMDSv2-only enforced — rejects `http-tokens=optional`), `--launch-template` (LaunchTemplateId/LaunchTemplateName, Version — resolves `$Default`/`$Latest`; direct params override the template) | `--dry-run`, `--client-token`, `--disable-api-termination`, `--ebs-optimized`, `--network-interfaces`, `--private-ip-address`, `--monitoring`, `--credit-specification`, `--cpu-options`, `--hibernate-options` | **DONE** |
 | `describe-instances` | `--instance-ids`, `--filters` (instance-state-name, instance-id, instance-type, vpc-id, subnet-id, tag:*, tag-key, tag-value) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
 | `start-instances` | `--instance-ids` | `--dry-run`, `--force` | **DONE** |
 | `stop-instances` | `--instance-ids` | `--force`, `--hibernate`, `--dry-run` | **DONE** |
@@ -130,29 +202,20 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 | `monitor-instances` | — | `--instance-ids` | **NOT STARTED** |
 | `unmonitor-instances` | — | `--instance-ids` | **NOT STARTED** |
 
-`run-instances --iam-instance-profile` enforces `iam:PassRole` on the contained
-role ARN and rejects cross-account profile ARNs. Placement strategies: `spread`
-(1 per node, atomic CAS) and `cluster` (pin all to one node).
+### EC2 — Spot Instances
 
-`run-instances --capacity-reservation-specification` consumes a reserved slot on
-the node owning the targeted reservation (targeted-by-id only; `open`
-auto-matching is not supported). The combination with `--placement` (group) is
-rejected. `describe-instances` then reports the instance's `CapacityReservationId`.
+Spot Instance Requests (SIRs) are a **mock** over the on-demand `run-instances` path: a request synchronously launches real VMs on the operator's own compute and is then reported `active`/`fulfilled`. There is no spot market — no bidding, price rejection, interruption, or reclamation, and instances are never reclaimed.
 
-`modify-instance-metadata-options` is the only metadata-options mutator — AWS has
-no `get-instance-metadata-options` API, so the per-instance block is read through
-`describe-instances`. Only the hop limit is mutable; `--http-tokens optional` and
-the other posture-weakening values are refused with `UnsupportedOperation`, since
-the platform enforces IMDSv2 permanently. `run-instances --metadata-options` runs
-the same validation at launch.
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `request-spot-instances` | `--instance-count` (default 1), `--type` (`one-time`/`persistent` — stored, behaviour identical), `--spot-price` (echoed only), `--client-token`, `--launch-specification` (ImageId, InstanceType, KeyName, SubnetId, SecurityGroupIds, UserData, BlockDeviceMappings, IamInstanceProfile, Placement.GroupName, NetworkInterfaces), `--tag-specifications` (spot-instances-request) | `--valid-from`, `--valid-until`, `--launch-group`, `--availability-zone-group`, `--block-duration-minutes`, `--instance-interruption-behavior`, `--dry-run` | **DONE** (mock) |
+| `describe-spot-instance-requests` | `--spot-instance-request-ids`, `--filters` (spot-instance-request-id, state, instance-id, launch.image-id, launch.instance-type, launch.key-name, type, launched-availability-zone, tag-key, tag:*) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
+| `cancel-spot-instance-requests` | `--spot-instance-request-ids` | `--dry-run` | **DONE** |
+
+`describe-spot-price-history` is **unsupported** (returns `InvalidAction`): on owned hardware there is no spot/on-demand price differential, so any synthetic price would be misleading rather than helpful.
 
 ### EC2 — IAM Instance Profile Associations
 
-Stored as `vm.VM.IamInstanceProfileArn` + `IamInstanceProfileAssociationId`
-(one ARN per instance). Association IDs (`iip-assoc-`) are regenerated on every
-Associate/Replace and never reused. Auto-disassociated on terminate; preserved
-across stop/start.
-
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
 | `associate-iam-instance-profile` | `--instance-id`, `--iam-instance-profile` (Name/Arn) | `--dry-run` | **DONE** |
@@ -164,10 +227,10 @@ across stop/start.
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
-| `create-key-pair` | `--key-name`, `--key-type` (rsa/ed25519) | `--key-format`, `--tag-specifications`, `--dry-run` | **DONE** |
+| `create-key-pair` | `--key-name`, `--key-type` (rsa/ed25519), `--tag-specifications` | `--key-format`, `--dry-run` | **DONE** |
 | `describe-key-pairs` | `--key-names`, `--key-pair-ids`, `--filters` (key-pair-id, key-name, fingerprint, tag:*) | `--max-results`, `--dry-run` | **DONE** |
 | `delete-key-pair` | `--key-name`, `--key-pair-id` | `--dry-run` | **DONE** |
-| `import-key-pair` | `--key-name`, `--public-key-material` | `--tag-specifications`, `--dry-run` | **DONE** |
+| `import-key-pair` | `--key-name`, `--public-key-material`, `--tag-specifications` | `--dry-run` | **DONE** |
 
 ### EC2 — AMI Images
 
@@ -183,16 +246,14 @@ across stop/start.
 | `reset-image-attribute` | `--image-id`, `--attribute description` | `--attribute launchPermission`, `--dry-run` | **DONE** |
 | `import-image` | — | `--disk-containers`, `--description`, `--architecture`, `--platform` | **NOT STARTED** |
 
-`copy-image` is metadata-only — no block copy. The new snapshot inherits the
-source `VolumeID`. `register-image`/`copy-image` accept `uefi-preferred` as
-input but treat it as `uefi` at launch.
+`copy-image` is metadata-only — no block copy. The new snapshot inherits the. source `VolumeID`.
 
 ### EC2 — Volumes (EBS)
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
 | `describe-volumes` | `--volume-ids`, `--filters` (volume-id, status, size, volume-type, attachment.instance-id, attachment.status, attachment.device, availability-zone, tag:*), persisted `DeleteOnTermination` | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
-| `create-volume` | `--size`, `--availability-zone`, `--volume-type` (gp3), `--snapshot-id` | `--iops` (hardcoded 3000), `--encrypted` (hardcoded false), `--throughput`, `--tag-specifications` | **DONE** |
+| `create-volume` | `--size`, `--availability-zone`, `--volume-type` (gp3), `--snapshot-id`, `--tag-specifications` | `--iops` (hardcoded 3000), `--encrypted` (hardcoded false), `--throughput` | **DONE** |
 | `delete-volume` | `--volume-id` | `--dry-run` | **DONE** |
 | `modify-volume` | `--volume-id`, `--size`, `--volume-type`, `--iops` | `--throughput`, `--dry-run`, `--multi-attach-enabled` | **DONE** |
 | `attach-volume` | `--volume-id`, `--instance-id`, `--device` (auto-assigns `/dev/sd[f-p]`) | `--dry-run` | **DONE** |
@@ -228,8 +289,7 @@ input but treat it as `uefi` at launch.
 
 ### EC2 — Account Settings
 
-Persistence works (NATS JetStream KV) but stored values are not yet enforced
-by downstream services.
+Persistence works but stored values are not yet enforced by downstream services.
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
@@ -248,10 +308,7 @@ by downstream services.
 | `modify-instance-metadata-defaults` | — | `--http-tokens`, `--http-put-response-hop-limit`, `--http-endpoint`, `--instance-metadata-tags` | **NOT STARTED** |
 | `get-instance-metadata-defaults` | — | `--dry-run` | **NOT STARTED** |
 
-The account-level `modify-/get-instance-metadata-defaults` are blocked on an
-`aws-sdk-go` bump: the `InstanceMetadataDefaultsResponse` type is absent from the
-pinned v1.44.266 (these APIs landed ~March 2024), and the gateway's generic
-handler needs the typed input struct to route them.
+The account-level `modify-/get-instance-metadata-defaults` are blocked on an `aws-sdk-go` bump: the `InstanceMetadataDefaultsResponse` type is absent from our version (these APIs landed ~March 2024), and the gateway's generic handler needs the typed input struct to route them.
 
 ### EC2 — VPC Core
 
@@ -296,8 +353,7 @@ handler needs the typed input struct to route them.
 
 ### EC2 — Egress-Only Internet Gateway
 
-KV CRUD only — no OVN/OVS integration. EIGWs are stored but have no effect on
-network topology.
+KV CRUD only — no OVN/OVS integration. EIGWs are stored but have no effect on network topology.
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
@@ -309,7 +365,7 @@ network topology.
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
-| `create-route-table` | `--vpc-id` | `--tag-specifications`, `--dry-run` | **DONE** |
+| `create-route-table` | `--vpc-id`, `--tag-specifications` | `--dry-run` | **DONE** |
 | `delete-route-table` | `--route-table-id` | `--dry-run` | **DONE** |
 | `describe-route-tables` | `--route-table-ids`, `--filters` (vpc-id, route-table-id, association.main, association.route-table-association-id, association.subnet-id, route.destination-cidr-block, route.gateway-id) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
 | `create-route` | `--route-table-id`, `--destination-cidr-block`, `--gateway-id`, `--nat-gateway-id` | `--egress-only-internet-gateway-id`, `--vpc-peering-connection-id`, `--dry-run` | **DONE** |
@@ -337,7 +393,7 @@ termination. Standalone attach/detach API is internal-only.
 
 ### EC2 — Elastic IP
 
-EIP commands are only registered when an external IPAM pool is configured.
+EIP handlers are always registered. Without a public IPAM pool (external mode disabled, or `nat` without a public pool), `describe-addresses` returns an empty list and mutating commands return `UnsupportedOperation`. In `nat` mode a public pool (`--external-pool` or `--external-source=dhcp` at init) enables the full EIP surface with host-delivered ingress.
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
@@ -352,7 +408,7 @@ EIP commands are only registered when an external IPAM pool is configured.
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
-| `create-nat-gateway` | `--subnet-id`, `--allocation-id` | `--connectivity-type`, `--tag-specifications`, `--dry-run` | **DONE** |
+| `create-nat-gateway` | `--subnet-id`, `--allocation-id`, `--tag-specifications` | `--connectivity-type`, `--dry-run` | **DONE** |
 | `delete-nat-gateway` | `--nat-gateway-id` | `--dry-run` | **DONE** |
 | `describe-nat-gateways` | `--nat-gateway-ids`, `--filters` (vpc-id, state) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
 | `assign-private-nat-gateway-address` | — | `--nat-gateway-id`, `--private-ip-addresses` | **NOT STARTED** |
@@ -362,9 +418,9 @@ EIP commands are only registered when an external IPAM pool is configured.
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
-| `create-placement-group` | `--group-name`, `--strategy` (spread/cluster) | `--partition-count`, `--spread-level`, `--tag-specifications`, `--dry-run` | **DONE** |
+| `create-placement-group` | `--group-name`, `--strategy` (spread/cluster), `--tag-specifications` | `--partition-count`, `--spread-level`, `--dry-run` | **DONE** |
 | `delete-placement-group` | `--group-name` | `--dry-run` | **DONE** |
-| `describe-placement-groups` | `--group-names`, `--group-ids`, `--filters` (strategy, state, spread-level, group-name) | `--dry-run` | **DONE** |
+| `describe-placement-groups` | `--group-names`, `--group-ids`, `--filters` (strategy, state, spread-level, group-name, tag:*, tag-key, tag-value) | `--dry-run` | **DONE** |
 
 ### EC2 — VPC Peering
 
@@ -432,11 +488,13 @@ EIP commands are only registered when an external IPAM pool is configured.
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
-| `create-launch-template` | — | `--launch-template-name`, `--launch-template-data`, `--tag-specifications` | **NOT STARTED** |
-| `create-launch-template-version` | — | `--launch-template-id`/`--launch-template-name`, `--launch-template-data`, `--source-version` | **NOT STARTED** |
-| `delete-launch-template` | — | `--launch-template-id`/`--launch-template-name`, `--dry-run` | **NOT STARTED** |
-| `describe-launch-templates` | — | `--launch-template-ids`, `--launch-template-names`, `--filters` | **NOT STARTED** |
-| `describe-launch-template-versions` | — | `--launch-template-id`/`--launch-template-name`, `--versions`, `--min-version`, `--max-version` | **NOT STARTED** |
+| `create-launch-template` | `--launch-template-name`, `--launch-template-data` (full nested RequestLaunchTemplateData), `--version-description`, `--tag-specifications` (launch-template-scoped), `--dry-run` (no-op) | `--client-token` (idempotency) | **DONE** |
+| `create-launch-template-version` | `--launch-template-id`/`--launch-template-name`, `--launch-template-data`, `--source-version` (clone-and-override), `--version-description`, `--dry-run` (no-op) | `--client-token`, `--resolve-alias` | **DONE** |
+| `delete-launch-template` | `--launch-template-id`/`--launch-template-name`, `--dry-run` (no-op) | — | **DONE** |
+| `delete-launch-template-versions` | `--launch-template-id`/`--launch-template-name`, `--versions` (rejects the current default version), `--dry-run` (no-op) | — | **DONE** |
+| `modify-launch-template` | `--launch-template-id`/`--launch-template-name`, `--default-version`, `--dry-run` (no-op) | — | **DONE** |
+| `describe-launch-templates` | `--launch-template-ids`, `--launch-template-names`, `--filters` (launch-template-id, launch-template-name, create-time, tag:*, tag-key) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
+| `describe-launch-template-versions` | `--launch-template-id`/`--launch-template-name`, `--versions` (`$Default`/`$Latest`/numeric), `--min-version`, `--max-version`, `--filters` (is-default-version, image-id, instance-type, kernel-id, ram-disk-id, ebs-optimized) | `--max-results`, `--next-token`, `--dry-run`, `--resolve-alias` | **DONE** |
 
 ### EC2 — Dedicated Hosts, IPv4 Pools, DHCP, Capacity Reservations
 
@@ -480,8 +538,7 @@ EIP commands are only registered when an external IPAM pool is configured.
 
 ## IAM
 
-All IAM operations are account-scoped. Root user (account `000000000000`)
-bypasses policy evaluation entirely.
+All IAM operations are account-scoped. Root user (account `000000000000`) bypasses policy evaluation entirely.
 
 ### IAM — Users
 
@@ -491,6 +548,21 @@ bypasses policy evaluation entirely.
 | `get-user` | `--user-name` | — | **DONE** |
 | `list-users` | `--path-prefix` | `--max-items`, `--marker` | **DONE** |
 | `delete-user` | `--user-name` | — | **DONE** |
+| `update-user` | — | `--user-name`, `--new-path`, `--new-user-name` | **NOT STARTED** |
+| `put-user-policy` | `--user-name`, `--policy-name`, `--policy-document` | — | **DONE** |
+| `get-user-policy` | `--user-name`, `--policy-name` | — | **DONE** |
+| `delete-user-policy` | `--user-name`, `--policy-name` | — | **DONE** |
+| `list-user-policies` | `--user-name` | `--max-items`, `--marker` | **DONE** |
+| `tag-user` | `--user-name`, `--tags` | — | **DONE** |
+| `untag-user` | `--user-name`, `--tag-keys` | — | **DONE** |
+| `list-user-tags` | `--user-name` | `--max-items`, `--marker` | **DONE** |
+| `put-user-permissions-boundary` | — | `--user-name`, `--permissions-boundary` | **NOT STARTED** |
+| `delete-user-permissions-boundary` | — | `--user-name` | **NOT STARTED** |
+| `create-login-profile` | — | `--user-name`, `--password` | **NOT STARTED** |
+| `get-login-profile` | — | `--user-name` | **NOT STARTED** |
+| `update-login-profile` | — | `--user-name`, `--password` | **NOT STARTED** |
+| `delete-login-profile` | — | `--user-name` | **NOT STARTED** |
+| `change-password` | — | `--old-password`, `--new-password` | **NOT STARTED** |
 
 ### IAM — Access Keys
 
@@ -500,6 +572,7 @@ bypasses policy evaluation entirely.
 | `list-access-keys` | `--user-name` | `--max-items`, `--marker` | **DONE** |
 | `delete-access-key` | `--access-key-id`, `--user-name` | — | **DONE** |
 | `update-access-key` | `--access-key-id`, `--user-name`, `--status` (Active/Inactive) | — | **DONE** |
+| `get-access-key-last-used` | — | `--access-key-id` | **NOT STARTED** |
 
 ### IAM — Policies
 
@@ -514,6 +587,17 @@ bypasses policy evaluation entirely.
 | `attach-user-policy` | `--user-name`, `--policy-arn` | — | **DONE** |
 | `detach-user-policy` | `--user-name`, `--policy-arn` | — | **DONE** |
 | `list-attached-user-policies` | `--user-name` | `--path-prefix`, `--max-items`, `--marker` | **DONE** |
+| `create-policy-version` | — | `--policy-arn`, `--policy-document`, `--set-as-default` | **NOT STARTED** |
+| `delete-policy-version` | — | `--policy-arn`, `--version-id` | **NOT STARTED** |
+| `set-default-policy-version` | — | `--policy-arn`, `--version-id` | **NOT STARTED** |
+| `list-entities-for-policy` | — | `--policy-arn`, `--entity-filter`, `--path-prefix`, `--policy-usage-filter` | **NOT STARTED** |
+| `tag-policy` | `--policy-arn`, `--tags` | — | **DONE** |
+| `untag-policy` | `--policy-arn`, `--tag-keys` | — | **DONE** |
+| `list-policy-tags` | `--policy-arn` | `--max-items`, `--marker` | **DONE** |
+| `generate-service-last-accessed-details` | — | `--arn`, `--granularity` | **NOT STARTED** |
+| `get-service-last-accessed-details` | — | `--job-id` | **NOT STARTED** |
+| `get-service-last-accessed-details-with-entities` | — | `--job-id`, `--service-namespace` | **NOT STARTED** |
+| `list-policies-granting-service-access` | — | `--arn`, `--service-namespaces` | **NOT STARTED** |
 
 ### IAM — Roles
 
@@ -532,6 +616,12 @@ bypasses policy evaluation entirely.
 | `put-role-policy` | `--role-name`, `--policy-name`, `--policy-document` | — | **DONE** |
 | `get-role-policy` | `--role-name`, `--policy-name` | — | **DONE** (document returned as raw JSON, not URL-encoded) |
 | `delete-role-policy` | `--role-name`, `--policy-name` | — | **DONE** |
+| `put-role-permissions-boundary` | — | `--role-name`, `--permissions-boundary` | **NOT STARTED** |
+| `delete-role-permissions-boundary` | — | `--role-name` | **NOT STARTED** |
+| `tag-role` | `--role-name`, `--tags` | — | **DONE** |
+| `untag-role` | `--role-name`, `--tag-keys` | — | **DONE** |
+| `list-role-tags` | `--role-name` | `--max-items`, `--marker` | **DONE** |
+| `update-role-description` | — | `--role-name`, `--description` | **NOT STARTED** |
 
 ### IAM — Instance Profiles
 
@@ -544,6 +634,9 @@ bypasses policy evaluation entirely.
 | `delete-instance-profile` | `--instance-profile-name` | — | **DONE** |
 | `add-role-to-instance-profile` | `--instance-profile-name`, `--role-name` | — | **DONE** |
 | `remove-role-from-instance-profile` | `--instance-profile-name`, `--role-name` | — | **DONE** |
+| `tag-instance-profile` | `--instance-profile-name`, `--tags` | — | **DONE** |
+| `untag-instance-profile` | `--instance-profile-name`, `--tag-keys` | — | **DONE** |
+| `list-instance-profile-tags` | `--instance-profile-name` | `--max-items`, `--marker` | **DONE** |
 
 ### IAM — OIDC Providers
 
@@ -556,22 +649,34 @@ bypasses policy evaluation entirely.
 | `add-client-id-to-open-id-connect-provider` | — | `--open-id-connect-provider-arn`, `--client-id` | **NOT STARTED** |
 | `remove-client-id-from-open-id-connect-provider` | — | `--open-id-connect-provider-arn`, `--client-id` | **NOT STARTED** |
 | `update-open-id-connect-provider-thumbprint` | — | `--open-id-connect-provider-arn`, `--thumbprint-list` | **NOT STARTED** |
-| `tag-open-id-connect-provider` / `untag-open-id-connect-provider` | — | `--open-id-connect-provider-arn`, `--tags`/`--tag-keys` | **NOT STARTED** |
+| `tag-open-id-connect-provider` / `untag-open-id-connect-provider` | `--open-id-connect-provider-arn`, `--tags`/`--tag-keys` | — | **DONE** |
+| `list-open-id-connect-provider-tags` | `--open-id-connect-provider-arn` | `--max-items`, `--marker` | **DONE** |
 
 ### IAM — Groups
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
-| `create-group` | — | `--group-name`, `--path` | **NOT STARTED** |
-| `get-group` | — | `--group-name` | **NOT STARTED** |
-| `list-groups` | — | `--path-prefix`, `--max-items`, `--marker` | **NOT STARTED** |
-| `delete-group` | — | `--group-name` | **NOT STARTED** |
-| `add-user-to-group` | — | `--group-name`, `--user-name` | **NOT STARTED** |
-| `remove-user-from-group` | — | `--group-name`, `--user-name` | **NOT STARTED** |
-| `list-groups-for-user` | — | `--user-name`, `--max-items`, `--marker` | **NOT STARTED** |
-| `attach-group-policy` | — | `--group-name`, `--policy-arn` | **NOT STARTED** |
-| `detach-group-policy` | — | `--group-name`, `--policy-arn` | **NOT STARTED** |
-| `list-attached-group-policies` | — | `--group-name`, `--path-prefix`, `--max-items`, `--marker` | **NOT STARTED** |
+| `create-group` | `--group-name`, `--path` | — | **DONE** |
+| `get-group` | `--group-name` | — | **DONE** |
+| `list-groups` | `--path-prefix` | `--max-items`, `--marker` | **DONE** |
+| `delete-group` | `--group-name` | — | **DONE** |
+| `update-group` | — | `--group-name`, `--new-path`, `--new-group-name` | **NOT STARTED** |
+| `add-user-to-group` | `--group-name`, `--user-name` | — | **DONE** |
+| `remove-user-from-group` | `--group-name`, `--user-name` | — | **DONE** |
+| `list-groups-for-user` | `--user-name` | `--max-items`, `--marker` | **DONE** |
+| `attach-group-policy` | `--group-name`, `--policy-arn` | — | **DONE** |
+| `detach-group-policy` | `--group-name`, `--policy-arn` | — | **DONE** |
+| `list-attached-group-policies` | `--group-name`, `--path-prefix` | `--max-items`, `--marker` | **DONE** |
+| `put-group-policy` | `--group-name`, `--policy-name`, `--policy-document` | — | **DONE** |
+| `get-group-policy` | `--group-name`, `--policy-name` | — | **DONE** |
+| `delete-group-policy` | `--group-name`, `--policy-name` | — | **DONE** |
+| `list-group-policies` | `--group-name` | `--max-items`, `--marker` | **DONE** |
+
+### IAM — Account
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `get-account-summary` | — | — | **DONE** |
 
 ---
 
@@ -588,7 +693,7 @@ bypasses policy evaluation entirely.
 | `get-federation-token` | — | `--name`, `--policy`, `--policy-arns`, `--duration-seconds`, `--tags` | **NOT STARTED** |
 | `decode-authorization-message` | — | `--encoded-message` | **NOT STARTED** |
 
-Trust policies stored on roles (`AssumeRolePolicyDocument`) reject `Condition`, `NotPrincipal`, `NotAction`, empty-string `Action` elements, and empty `Principal` blocks at write time (`MalformedPolicyDocument`) — v1 has no Condition evaluator so accepting them would silently allow.
+Trust policies (`AssumeRolePolicyDocument`) reject `NotPrincipal`, `NotAction`, empty-string `Action` elements, and empty `Principal` blocks at write time (`MalformedPolicyDocument`). `Condition` blocks are rejected except on `sts:AssumeRoleWithWebIdentity` with `StringEquals` (IRSA), which v1 evaluates at assume time (`{iss}:sub`, `{iss}:aud`); anything wider is rejected to avoid silent over-grant.
 
 ---
 
@@ -625,7 +730,7 @@ curl -i http://169.254.169.254/latest/meta-data/instance-id
 | `/latest/meta-data/ami-id` | GET | launch `ImageId` | **DONE** |
 | `/latest/meta-data/ami-launch-index` | GET | Per-instance launch index (`0..n-1`), contiguous on partial failure | **DONE** |
 | `/latest/meta-data/reservation-id` | GET | `DescribeInstances` `Reservation.ReservationId` | **DONE** |
-| `/latest/meta-data/instance-life-cycle` | GET | `on-demand` (Spot not modelled) | **DONE** |
+| `/latest/meta-data/instance-life-cycle` | GET | `spot` for a spot-launched instance, else `on-demand`; defaults to `on-demand` on a resolution miss (never 404, the leaf is always advertised) | **DONE** |
 | `/latest/meta-data/local-ipv4` | GET | `ENIRecord.PrivateIpAddress` (== request source IP) | **DONE** |
 | `/latest/meta-data/public-ipv4` | GET | EIP, else instance public IP; empty body if none | **DONE** |
 | `/latest/meta-data/public-hostname` | GET | Mirrors `public-ipv4`; 404 when no public IP | **DONE** |
@@ -651,7 +756,7 @@ curl -i http://169.254.169.254/latest/meta-data/instance-id
 | `/latest/meta-data/block-device-mapping/...` | GET | `ami`/`root`/`ebsN`/`ephemeralN` device map | **NOT STARTED** (404) |
 | `/latest/meta-data/placement/{group-name,partition-number,availability-zone-id,host-id}` | GET | Placement extras beyond `availability-zone`/`region` | **NOT STARTED** (404) |
 | `/latest/meta-data/instance-action` | GET | `none` unless interruptible instances ship | **NOT STARTED** (404) |
-| `/latest/meta-data/spot/{instance-action,termination-time}` | GET | Only meaningful once Spot is modelled | **NOT STARTED** (404) |
+| `/latest/meta-data/spot/{instance-action,termination-time}` | GET | 404 is the faithful steady state for a never-interrupted spot instance ("no action scheduled"); a 200 body would trigger interruption handling in pollers (AWS Node Termination Handler / Karpenter). Not advertised in the `spot/` listing | **DONE** (404 by contract) |
 
 ---
 

docs/DESIGN.md

@@ -49,7 +49,7 @@ Request routing (`spinifex/gateway/gateway.go`):
 
 1. **Authentication**: SigV4 middleware validates AWS credentials and resolves the account ID
 2. **Throttling**: Per-account+action token bucket rejects bursts post-auth
-3. **Service Detection**: Extracts service name (ec2, iam, account, elasticloadbalancing, spinifex) from the Authorization header
+3. **Service Detection**: Extracts service name from the Authorization header
 4. **Action Dispatch**: Routes to service-specific handler
 
 ```go
@@ -60,8 +60,20 @@ case "account":
     err = gw.Account_Request(w, r)
 case "iam":
     err = gw.IAM_Request(w, r)
+case "sts":
+    err = gw.STS_Request(w, r)
 case "elasticloadbalancing":
     err = gw.ELBv2_Request(w, r)
+case "eks":
+    err = gw.EKS_Request(w, r)
+case "ecs":
+    err = gw.ECS_Request(w, r)
+case "ecr":
+    err = gw.ECR_Request(w, r)
+case "acm":
+    err = gw.ACM_Request(w, r)
+case "tagging":
+    err = gw.Tagging_Request(w, r)
 case "spinifex":
     err = gw.Spinifex_Request(w, r)
 }
@@ -79,18 +91,12 @@ The EC2 handler (`spinifex/gateway/ec2.go`) parses the `Action` parameter and de
     return gateway_ec2_instance.DescribeInstances(input, gw.NATSConn, gw.DiscoverActiveNodes(), accountID)
 }),
 // ... + volumes, snapshots, VPCs, subnets, route tables, IGWs, NAT gateways,
-// security groups, network interfaces, placement groups, key pairs, images, tags
+// security groups, network interfaces, elastic IPs, placement groups, key pairs,
+// images, tags, capacity reservations, spot instances
 ```
 
 ### 4. NATS Messaging
 
-The full service-to-subject map — which service subscribes to what, what
-each one publishes downstream, what each depends on — is captured in
-[`service-interfaces.yaml`](./service-interfaces.yaml). It is the source
-of truth for targeted e2e suite selection (see
-`docs/development/improvements/e2e-targeted-suite-selection.md`) and is
-validated by `make manifest-check`.
-
 The gateway communicates with daemons via NATS request/response. Most calls go through `utils.NATSRequest`, which marshals the input, attaches the account ID as a NATS header, and unmarshals the typed response:
 
 ```go
@@ -114,7 +120,7 @@ Daemons (`spinifex/daemon/daemon.go`) subscribe to NATS topics and handle reques
 {"ec2.DescribeInstanceTypes", d.handleEC2DescribeInstanceTypes, ""},
 ```
 
-`ec2.RunInstances` is dynamic: `ResourceManager` subscribes to `ec2.RunInstances.{type}` (and `ec2.RunInstances.{type}.{nodeId}` for targeted dispatch) only while the node has capacity for that type, and unsubscribes when full. Per-instance commands flow over `ec2.cmd.{instanceID}`, subscribed only by the owning node.
+`ec2.RunInstances` is dynamic: `ResourceManager` subscribes to `ec2.RunInstances.{type}` only while the node has capacity for that type, and unsubscribes when full. The node-targeted subject `ec2.RunInstances.{type}.{nodeId}` stays subscribed even at full capacity — the launch-time `allocate()` enforces limits there. Per-instance commands flow over `ec2.cmd.{instanceID}`, subscribed only by the owning node.
 
 **Queue Groups**: topics with the `spinifex-workers` queue group are load-balanced — only one daemon handles each request. Topics without a queue group fan out to all daemons.
 
@@ -151,7 +157,7 @@ type Daemon struct {
     config        *config.Config          // Node-specific configuration
     natsConn      *nats.Conn              // NATS connection
     resourceMgr   *ResourceManager        // CPU/Memory tracking + dynamic RunInstances subscriptions
-    Instances     vm.Instances            // Local VMs
+    vmMgr         *vm.Manager             // Local VMs
     // ... + one service struct per EC2/ELBv2 resource
     //       (instance, key, image, volume, snapshot, tags, vpc, subnet,
     //        igw, eigw, natgw, routetable, eip, placementgroup, elbv2, ...)
@@ -172,6 +178,8 @@ type ResourceManager struct {
     allocatedVCPU int
     allocatedMem  float64
     instanceTypes map[string]*ec2.InstanceTypeInfo   // t3.micro, t3.small, etc.
+    // ... + capacity-reservation accounting, GPU manager, nbdkit memory
+    //       overhead tracking, and the dynamic-subscription state
 }
 ```
 
@@ -236,22 +244,26 @@ Cluster configuration (`spinifex/config/config.go`):
 
 ```go
 type ClusterConfig struct {
-    Epoch   uint64            // Incremented on config changes
-    Node    string            // This node's identifier
-    Version string            // Spinifex version
-    Nodes   map[string]Config // Configuration for all nodes
+    Epoch     uint64            // Incremented on config changes
+    Node      string            // This node's identifier
+    Version   string            // Spinifex version
+    Network   NetworkConfig     // Cluster-wide external network settings (IP pools, IPsec)
+    Bootstrap BootstrapConfig   // Default VPC IDs written by admin init
+    AWS       AWSConfig         // Cluster-wide AWS-parity settings (region, endpoint suffix)
+    Nodes     map[string]Config // Configuration for all nodes
 }
 
 type Config struct {
-    Node, Host, Region, AZ string
-    BaseDir, DataDir       string
+    Node, Host, AdvertiseIP  string
+    Region, AZ               string
+    BaseDir, DataDir, WalDir string
+    Services   []string         // Which services this node runs locally
     Daemon     DaemonConfig
     NATS       NATSConfig       // Broker address, JetStream, ACL token, CA cert
     AWSGW      AWSGWConfig      // Gateway host, TLS certs, throttle config
     Predastore PredastoreConfig // S3 endpoint + service credentials
     Viperblock ViperblockConfig
     VPCD       VPCDConfig       // OVN/OVS endpoints
-    Network    NetworkConfig
 }
 ```
 
@@ -263,14 +275,14 @@ Configs are generated by `spx admin init` (leader) or `spx admin join` (follower
 spinifex/
 ├── spinifex/
 │   ├── services/                # Long-running services (awsgw, spinifex, viperblockd,
-│   │                            #   predastore, nats, vpcd, spinifexui)
+│   │                            #   predastore, nats, spinifexui)
 │   ├── gateway/                 # AWS API surface
 │   │   ├── gateway.go           # chi router, SigV4 auth, throttling
 │   │   ├── auth.go              # SigV4 verification + IAM lookup
 │   │   ├── ec2.go               # EC2 action dispatcher
 │   │   ├── ec2/                 # Per-resource handlers (instance, volume, vpc, ...)
 │   │   ├── elbv2/, iam/, policy/
-│   ├── handlers/                # NATS-side service implementations (ec2, elbv2, iam)
+│   ├── handlers/                # NATS-side service implementations (ec2, elbv2, iam, ...)
 │   ├── daemon/
 │   │   ├── daemon.go            # Daemon entry, NATS subscription table
 │   │   └── daemon_handlers_*.go # One file per resource family
@@ -279,11 +291,13 @@ spinifex/
 │   ├── instancetypes/           # t3.*, m5.*, sys.* definitions
 │   ├── lbagent/                 # In-VM agent for ELB target health
 │   ├── vm/                      # VM instance representation
+│   ├── vpcd/                    # VPC daemon (OVN/OVS network programming)
 │   └── qmp/                     # QEMU Machine Protocol client
 └── cmd/
     ├── spinifex/                # spx CLI
     ├── lb-agent/                # ELB target agent binary
-    └── installer/               # Platform installer
+    ├── installer/               # Platform installer
+    └── ...                      # ECS agent, ECR credential provider, EKS helper binaries
 ```
 
 ## Development

docs/TAG-CONVENTIONS.md

@@ -1,11 +1,7 @@
 # Spinifex Tag Conventions
 
-Spinifex uses a small set of reserved tag keys to mark resources that are
-owned and managed by the platform itself (as opposed to resources created
-directly by customer AWS API calls). The UI uses these tags to hide
-system-owned resources from customer-facing listings so operators do not
-mistake them for user workloads and accidentally terminate or modify
-them.
+Spinifex uses a small set of reserved tag keys to mark resources that are owned and managed by the platform itself (as opposed to resources created directly by customer AWS API calls). The UI uses these tags to hide system-owned resources from customer-facing listings so operators do not
+mistake them for user workloads and accidentally terminate or modify them.
 
 ## `spinifex:managed-by`
 
@@ -17,50 +13,20 @@ Identifies the Spinifex platform component that owns a resource.
 
 The tag is applied by the backend at create/launch time:
 
-- **Instance** — set in
-  `spinifex/spinifex/daemon/daemon_system_instance.go` when
-  `LaunchSystemInstance` builds the `RunInstancesInput`, and mirrored onto
-  the stored `vm.VM.ManagedBy` field for fast lookup.
-- **ENI** — set in
-  `spinifex/spinifex/handlers/elbv2/service_impl.go` when
-  `CreateLoadBalancer` calls `CreateNetworkInterface`.
-- **AMI** — declared on the catalog entry in
-  `spinifex/spinifex/utils/images.go` and copied into
-  `AMIMetadata.Tags` by the image importer in
-  `spinifex/cmd/spinifex/cmd/admin.go`.
+- **Instance** — set in `spinifex/spinifex/daemon/daemon_system_instance.go` when `LaunchSystemInstance` builds the `RunInstancesInput`, and mirrored onto the stored `vm.VM.ManagedBy` field for fast lookup.
+- **ENI** — set in `spinifex/spinifex/handlers/elbv2/service_impl.go` when `CreateLoadBalancer` calls `CreateNetworkInterface`.
+- **AMI** — declared on the catalog entry in `spinifex/spinifex/utils/images.go` and copied into `AMIMetadata.Tags` by the image importer in `spinifex/cmd/spinifex/cmd/admin.go`.
 
-Tag keys and values are defined once in `spinifex/spinifex/tags/tags.go`;
-new system components should register their value there and reuse the
-constants rather than hardcoding strings.
+Tag keys and values are defined once in `spinifex/spinifex/tags/tags.go`; new system components should register their value there and reuse the constants rather than hardcoding strings.
 
 ## UI behaviour
 
-The admin **Nodes** page and the **Images** (AMIs) page filter out
-resources that carry `spinifex:managed-by`. Engineers who need to see
-system resources for debugging can append `?system=1` to the URL — the
-filter is bypassed and a hidden-count banner is replaced with the full
-listing. Direct navigation to a system AMI detail page without
-`?system=1` renders the "Image not found" fallback.
+The admin **Nodes** page and the **Images** (AMIs) page filter out resources that carry `spinifex:managed-by`. Engineers who need to see system resources for debugging can append `?system=1` to the URL — the filter is bypassed and a hidden-count banner is replaced with the full listing. Direct navigation to a system AMI detail page without `?system=1` renders the "Image not found" fallback.
 
-The EC2 **Instances** page does not need UI-side filtering: backend
-SigV4 authentication scopes `DescribeInstances` by account ID, and
-system instances live under the global system account
+The EC2 **Instances** page does not need UI-side filtering: backend SigV4 authentication scopes `DescribeInstances` by account ID, and system instances live under the global system account
 (`000000000000`), so customers never see them.
 
 ## `spinifex:lb-arn`
 
-Applied to ELBv2-managed ENIs alongside `spinifex:managed-by=elbv2`.
-Stores the ARN of the parent load balancer so reconciliation can map an
-ENI back to its owner without a separate lookup. Customer-facing
-filtering keys off `spinifex:managed-by`; `spinifex:lb-arn` is
-informational.
-
-## Adding a new system component
-
-1. Add a new value constant next to `ManagedByELBv2` in
-   `spinifex/spinifex/tags/tags.go`.
-2. Tag every resource the component creates (instance, ENI, AMI,
-   volume, …) with `spinifex:managed-by=<value>`.
-3. Add the value to the table above so operators can recognise it.
-4. No frontend changes are required — the UI filters on the key, not
-   the value.
+Applied to ELBv2-managed ENIs alongside `spinifex:managed-by=elbv2`. Stores the ARN of the parent load balancer so reconciliation can map an ENI back to its owner without a separate lookup. Customer-facing
+filtering keys off `spinifex:managed-by`; `spinifex:lb-arn` is informational.

docs/admin/setting-up-your-cluster/README.md

@@ -1,7 +1,7 @@
 ---
 title: "Setting Up Your Cluster"
 description: "Import an AMI, create a VPC with a public subnet, and launch your first EC2 instance on Spinifex."
-category: "Getting Started"
+category: "Admin"
 tags:
   - setup
   - vpc
@@ -33,16 +33,14 @@ resources:
 
 ## Overview
 
-This guide assumes Spinifex is already installed and running. If not, follow one of the installation guides first:
-- [Single-Node Install](/docs/install)
-- [Multi-Node Install](/docs/install-multi-node)
-- [Source Install](/docs/install-source)
+This guide walks through the first steps on a freshly installed cluster: importing an AMI, creating an SSH key and a VPC with a public subnet, launching an instance, and connecting to it.
 
 ## Instructions
 
 ## Prerequisites
 
-Ensure the AWS profile is set:
+- Spinifex installed and running — see [Single-Node Install](/docs/install), [Multi-Node Install](/docs/install-multi-node), or [Source Install](/docs/install-source)
+- AWS CLI configured with the `spinifex` profile:
 
 ```bash
 export AWS_PROFILE=spinifex

docs/admin/spinifex-admin-cli/README.md

@@ -1,7 +1,7 @@
 ---
 title: "Spinifex Admin CLI"
 description: "Complete reference for the Spinifex administration CLI. Manage accounts, nodes, VMs, and services."
-category: "Administration"
+category: "Admin"
 tags:
   - cli
   - admin
@@ -25,30 +25,56 @@ resources:
 
 ## Overview
 
-The `spx` binary is the central administration tool for managing your Spinifex infrastructure. It provides commands for cluster initialization, account management, node operations, VM lifecycle, and service control.
+The `spx` binary is the central administration tool for managing your Spinifex infrastructure. It provides commands for cluster initialization, account management, node operations, VM lifecycle, and service control. All services in the Spinifex platform are managed through this single binary.
 
-**Binary location:** `/usr/local/bin/spx` (binary install) or `~/Development/mulga/spinifex/bin/spx` (source install)
-
-All services in the Spinifex platform are managed through this single binary.
+**Binary location:** `/usr/local/bin/spx`
 
 ## Instructions
 
 ## Account Management
 
-Create a new isolated account with an admin user and AdministratorAccess policy:
+Create a new isolated account. This provisions a sequential 12-digit account ID, an `admin` user with an `AdministratorAccess` policy attached, and an access key pair. The credentials are written to `~/.aws/credentials` and `~/.aws/config` under a `spinifex-<name>` profile automatically.
 
 ```bash
 spx admin account create --name myteam
+```
+
+```
+Account created successfully!
+  Account ID:        000000000002
+  Account Name:      myteam
+  Admin User:        admin
+  Access Key ID:     AKIA1A2B3C4D5E6F7890ABCD
+  Secret Access Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
+  AWS Profile:       spinifex-myteam
+
+Use with:
+  AWS_PROFILE=spinifex-myteam aws ec2 describe-instances
+```
+
+> **The secret access key is only shown once.** It is saved to `~/.aws/credentials` on the node where you ran the command; copy it from there if you need it elsewhere.
+
+Set the profile to start using the account:
+
+```bash
 export AWS_PROFILE=spinifex-myteam
 ```
 
+To create additional users and scoped permissions within the account, see [IAM Users and Policies](/docs/iam-users-and-policies).
+
 List all accounts:
 
 ```bash
 spx admin account list
 ```
 
-
+```
+ACCOUNT ID     NAME                 STATUS     CREATED
+----------     ----                 ------     -------
+000000000000   system               ACTIVE     2026-07-03 02:17
+000000000001   spinifex             ACTIVE     2026-07-03 02:17
+000000000002   myteam               ACTIVE     2026-07-03 03:32
+```
 
 ## Node Management
 
@@ -97,7 +123,7 @@ configuration is needed.
 
 ## Cluster Shutdown
 
-Coordinated, phased shutdown of the entire cluster (API/UI → VMs → viperblock → predastore → NATS/daemon):
+Coordinated, phased shutdown of the entire cluster — running VMs are stopped before storage and control-plane services:
 
 ```bash
 spx admin cluster shutdown

docs/admin/update/README.md

@@ -1,7 +1,7 @@
 ---
 title: "Updating Spinifex"
 description: "Upgrade an existing Spinifex installation to the latest release."
-category: "Administration"
+category: "Admin"
 tags:
   - update
   - upgrade

docs/administration/creating-accounts/README.md

@@ -1,55 +0,0 @@
----
-title: "Creating Accounts"
-description: "Create and manage isolated user accounts with their own resources."
-category: "Administration"
-tags:
-  - accounts
-  - iam
-  - admin
-resources:
-  - title: "Spinifex Repository"
-    url: "https://github.com/mulgadc/spinifex"
----
-
-# Creating Accounts
-
-> Create and manage isolated user accounts with their own resources.
-
-## Table of Contents
-
-- [Overview](#overview)
-- [Instructions](#instructions)
-- [Troubleshooting](#troubleshooting)
-
----
-
-## Overview
-
-Spinifex supports multi-tenant account isolation. Each account gets its own IAM credentials, AWS CLI profile, and isolated resource namespace.
-
-## Instructions
-
-## Create Account
-
-```bash
-spx admin account create --name myteam
-export AWS_PROFILE=spinifex-myteam
-```
-
-## Troubleshooting
-
-## Credentials Not Working
-
-Verify the AWS CLI configuration files exist and contain the correct profile:
-
-```bash
-cat ~/.aws/config
-cat ~/.aws/credentials
-```
-
-Ensure the `AWS_PROFILE` environment variable matches the profile name:
-
-```bash
-echo $AWS_PROFILE
-export AWS_PROFILE=spinifex-myteam
-```

docs/compute/gpu-passthrough/README.md

@@ -72,7 +72,7 @@ After setup and reboot, enable GPU passthrough:
 sudo spx admin gpu enable
 ```
 
-The command writes the configuration, signals the daemon (`SIGHUP`), and waits up to 30 seconds for the daemon to confirm the new state. On success it prints the current GPU status.
+The command writes the configuration, notifies the daemon and waits up to 30 seconds for the daemon to confirm the new state. On success it prints the current GPU status.
 
 <img src="../../../.github/assets/images/gpu-enabled.png" alt="GPU enabled">
 

docs/compute/launching-instances/README.md

@@ -22,11 +22,12 @@ resources:
 - [Overview](#overview)
 - [Launch](#launch)
 - [Manage](#manage)
-- [Reboot](#reboot)
 - [Modify Instance Attributes](#modify-instance-attributes)
+- [Instance Metadata Options](#instance-metadata-options)
+- [Spot Instances](#spot-instances)
 - [Console Output](#console-output)
 - [Instance Types](#instance-types)
-- [SSH (Development)](#ssh-development)
+- [SSH](#ssh)
 - [Troubleshooting](#troubleshooting)
   - [Instance Fails to Boot](#instance-fails-to-boot)
   - [Cannot SSH Into Instance](#cannot-ssh-into-instance)
@@ -37,22 +38,12 @@ resources:
 
 Spinifex provides EC2-compatible VM management built on QEMU/KVM. Instances support cloud-init, SSH key injection, VPC networking, and standard AWS lifecycle operations.
 
-**Supported operations:**
-
-- `run-instances` — Launch new VMs
-- `describe-instances` — Query state
-- `stop-instances` / `start-instances` — Lifecycle
-- `reboot-instances` — In-place restart (QMP reset)
-- `terminate-instances` — Permanent removal
-- `modify-instance-attribute` — Change type, user data, or source/dest check
-- `get-console-output` — Retrieve serial console log
-- `describe-instance-types` — List available instance types
-
 ## Instructions
 
 ## Prerequisites
 
-Ensure the AWS profile is set:
+- A running Spinifex cluster (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
+- AWS CLI configured with the `spinifex` profile:
 
 ```bash
 export AWS_PROFILE=spinifex
@@ -61,18 +52,20 @@ export AWS_PROFILE=spinifex
 ## Launch
 
 ```bash
-aws ec2 run-instances \
+INSTANCE_ID=$(aws ec2 run-instances \
   --image-id $SPINIFEX_AMI \
   --instance-type t3.small \
-  --key-name spinifex-key
-
-export INSTANCE_ID="i-XXX"
+  --key-name spinifex-key \
+  --query 'Instances[0].InstanceId' --output text)
 ```
 
+Other supported launch flags: `--subnet-id`, `--security-group-ids`, `--tag-specifications`, `--user-data`, `--iam-instance-profile`, `--block-device-mappings`, `--placement`, `--count`.
+
 ## Manage
 
 ```bash
 aws ec2 describe-instances --instance-ids $INSTANCE_ID
+aws ec2 describe-instance-status --instance-ids $INSTANCE_ID --include-all-instances
 aws ec2 stop-instances --instance-ids $INSTANCE_ID
 aws ec2 start-instances --instance-ids $INSTANCE_ID
 aws ec2 terminate-instances --instance-ids $INSTANCE_ID
@@ -81,7 +74,7 @@ aws ec2 reboot-instances --instance-ids $INSTANCE_ID
 
 ## Modify Instance Attributes
 
-Change instance type, user data, or source/dest check. Instance type and user data require the instance to be **stopped** first.
+Change instance type, user data, or termination protection. Instance type and user data require the instance to be **stopped** first.
 
 ### Change Instance Type
 
@@ -95,6 +88,37 @@ aws ec2 modify-instance-attribute \
 aws ec2 start-instances --instance-ids $INSTANCE_ID
 ```
 
+### Termination Protection
+
+```bash
+aws ec2 modify-instance-attribute \
+  --instance-id $INSTANCE_ID \
+  --disable-api-termination
+```
+
+## Instance Metadata Options
+
+IMDSv2 is always enforced — `--http-tokens optional` and `--http-endpoint disabled` are rejected with `UnsupportedOperation`, matching AWS account-level IMDSv2 enforcement. The hop limit is adjustable (raise it for containerised workloads that reach IMDS through an extra network hop):
+
+```bash
+aws ec2 modify-instance-metadata-options \
+  --instance-id $INSTANCE_ID \
+  --http-put-response-hop-limit 2
+```
+
+## Spot Instances
+
+Spot requests are supported as a compatibility layer over the on-demand path: requests launch real VMs on your own compute immediately and report `active`/`fulfilled`. There is no spot market — no bidding, interruption, or reclamation.
+
+```bash
+aws ec2 request-spot-instances \
+  --instance-count 1 \
+  --launch-specification '{"ImageId":"'$SPINIFEX_AMI'","InstanceType":"t3.small","KeyName":"spinifex-key"}'
+
+aws ec2 describe-spot-instance-requests
+aws ec2 cancel-spot-instance-requests --spot-instance-request-ids $SIR_ID
+```
+
 ## Console Output
 
 Retrieve the serial console log for a running instance. Output is base64-encoded.
@@ -121,7 +145,8 @@ aws ec2 describe-instance-types
 Filter to a specific type:
 
 ```bash
-aws ec2 describe-instance-types --instance-types t3.micro t3.small
+aws ec2 describe-instance-types \
+  --query "InstanceTypes[?InstanceType=='t3.small']"
 ```
 
 Show capacity (how many of each type can still be launched):
@@ -131,16 +156,9 @@ aws ec2 describe-instance-types \
   --filters Name=capacity,Values=true
 ```
 
-## SSH (Development)
+## SSH
 
-In development mode, find the QEMU port forward and connect via localhost:
-
-```bash
-ps auxw | grep $INSTANCE_ID
-ssh -i ~/.ssh/spinifex-key ubuntu@127.0.0.1 -p <port>
-```
-
-For production SSH via public IPs, see [Setting Up Your Cluster](/docs/setting-up-your-cluster).
+To SSH into instances via their public IPs, see [Setting Up Your Cluster](/docs/setting-up-your-cluster).
 
 ## Troubleshooting
 

docs/compute/placement-groups/README.md

@@ -50,7 +50,8 @@ The `partition` strategy is not supported.
 
 ## Prerequisites
 
-Ensure the AWS profile is set:
+- A running Spinifex cluster (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
+- AWS CLI configured with the `spinifex` profile:
 
 ```bash
 export AWS_PROFILE=spinifex

docs/compute/vpc-networking/README.md

@@ -191,11 +191,18 @@ is identical:
 Both support the same AWS features: public subnets, Elastic IPs, security groups,
 DescribeInstances showing public IPs.
 
-## `nat` — Outbound Only (Simple)
+## `nat` — Shared SNAT (Simple)
 
-All VMs share a single external IP for outbound SNAT. No public IPs, no Elastic
-IPs, no inbound from WAN. All subnets behave as private subnets with internet
-access.
+All VMs share a single external IP for outbound SNAT. By default there are no
+public IPs, no Elastic IPs, and no inbound from WAN — all subnets behave as
+private subnets with internet access. On routed-NAT nodes, adding a public pool
+restores full public IP parity (see below).
+
+> **Limitation (routed-NAT v1):** System instances (ECS/EKS/load-balancer
+> agents) source egress from ExternalIPAM pool IPs, which do not exist in
+> `external_mode=nat`. Those features require `external_mode=pool`. A v2 will
+> either allocate transit IPs for system instances or reject the feature at the
+> API level in nat mode.
 
 The `gateway_ip` is the IP that OVN uses for SNAT. You can set it statically or
 use `setup-ovn.sh --dhcp` to obtain one from the router. This is the router's
@@ -215,6 +222,90 @@ gateway_ip = "192.168.1.100"     # Single IP for all VM outbound SNAT
 prefix_len = 24
 ```
 
+### Host access to instances (jumpbox pattern)
+
+The spinifex host automatically reaches every instance's **private IP**: IGW
+attach installs a host route into OVN (`<vpc-cidr> via <gateway-transit-ip>
+dev spx-nat-host`) and exempts the transit net from SNAT, so replies to
+host-initiated connections come back un-NATted. No per-instance setup.
+
+Security groups still apply and the default SG is closed to the host, same as
+AWS — open SSH/ICMP from the transit net first:
+
+```bash
+aws ec2 authorize-security-group-ingress --group-id $SG \
+  --protocol tcp --port 22 --cidr 100.127.0.0/24
+```
+
+Then use the host as a jumpbox for remote access:
+
+```bash
+ssh -J admin@<spinifex-host> ubuntu@<instance-private-ip>
+```
+
+Extra networks that must reach instances without SNAT (e.g. a management LAN)
+can be added via `[network] nat_exempt_cidrs = ["192.168.50.0/24"]`.
+
+### Public IPs in NAT mode (public pool)
+
+A routed-NAT node (`setup-ovn.sh --nat-uplink` + `spx admin init
+--external-mode=nat`) can carry a public pool alongside the internal
+`nat-transit` pool. With one configured, nat mode behaves like pool mode for
+public IPs: `MapPublicIpOnLaunch` on the default subnet, auto-assigned public
+IPs, and Elastic IPs all work. Spinifex delivers each public IP at the host —
+a `/32` route steers it into OVN and a proxy-ARP neighbor entry answers for it
+on the uplink (L3 only, same MAC, so it works on WiFi and other non-bridgeable
+uplinks).
+
+Static range carved out of the router's DHCP scope:
+
+```bash
+spx admin init --external-mode=nat \
+  --external-pool 192.168.1.150-192.168.1.250 \
+  --external-gateway 192.168.1.1
+```
+
+Or lease public IPs from the upstream router's DHCP:
+
+```bash
+spx admin init --external-mode=nat --external-source=dhcp \
+  --external-bind-bridge wlan0
+```
+
+On WiFi/WWAN uplinks the leases are requested with the interface's own MAC
+(`dhcp_mac = "interface"`, written automatically) and distinguished by DHCP
+client-id. Some routers key leases by MAC and ignore the client-id — Spinifex
+detects this (the router hands the same IP to two client-ids) and fails the
+allocation with advice to switch to a static range.
+
+Resulting config:
+
+```toml
+[network]
+external_mode = "nat"
+bridge_mode   = "nat"
+
+[[network.external_pools]]
+name       = "nat-transit"        # internal transit net (auto-generated)
+gateway    = "100.127.0.1"
+prefix_len = 24
+
+[[network.external_pools]]
+name        = "wan"               # public pool
+range_start = "192.168.1.150"
+range_end   = "192.168.1.250"
+gateway     = "192.168.1.1"
+prefix_len  = 24
+```
+
+**Caveat — reaching an EIP from the spinifex host itself.** Host-sourced
+traffic enters OVN from the transit net, which is exempt from NAT (that is what
+makes the jumpbox pattern work) — so a host connection to an EIP that carries
+the transit source IP would skip DNAT. Spinifex stamps the EIP route with the
+uplink's LAN IP as source to avoid this, but if no uplink address can be
+determined, connect to the instance's **private IP** from the host instead.
+Other machines on the LAN are unaffected.
+
 ## Disabled (Empty/Omitted)
 
 VPC networking is overlay-only. No external connectivity. Instances can only
@@ -222,19 +313,21 @@ communicate within their VPC.
 
 ## Mode Comparison
 
-| Capability                        | `pool` (static) | `pool` (dhcp) | `nat`    | Disabled |
-| --------------------------------- | --------------- | ------------- | -------- | -------- |
-| Outbound internet                 | Yes             | Yes           | Yes      | No       |
-| Inbound from WAN                  | Yes (1:1 NAT)   | Yes (1:1 NAT) | No       | No       |
-| Public subnets                    | Yes             | Yes           | No       | No       |
-| Auto-assign public IPs            | Yes             | Yes           | No       | No       |
-| Elastic IPs                       | Yes             | Yes           | No       | No       |
-| DescribeInstances shows public IP | Yes             | Yes           | No       | No       |
-| Admin must reserve IP range       | Yes             | No            | No       | No       |
-| Needs router DHCP                 | No              | Yes           | Optional | No       |
+| Capability                        | `pool` (static) | `pool` (dhcp) | `nat`             | Disabled |
+| --------------------------------- | --------------- | ------------- | ----------------- | -------- |
+| Outbound internet                 | Yes             | Yes           | Yes               | No       |
+| Host reaches instance private IPs | No              | No            | Yes (routed)      | No       |
+| Inbound from WAN                  | Yes (1:1 NAT)   | Yes (1:1 NAT) | With public pool  | No       |
+| Public subnets                    | Yes             | Yes           | With public pool  | No       |
+| Auto-assign public IPs            | Yes             | Yes           | With public pool  | No       |
+| Elastic IPs                       | Yes             | Yes           | With public pool  | No       |
+| DescribeInstances shows public IP | Yes             | Yes           | With public pool  | No       |
+| Admin must reserve IP range       | Yes             | No            | Only static pool  | No       |
+| Needs router DHCP                 | No              | Yes           | Optional          | No       |
 
-If you start with `nat` and later need public subnets, switch to `pool` and
-define a range (or use `source = "dhcp"`) — no data migration needed.
+If you start with `nat` and later need public subnets: on a bridgeable uplink
+switch to `pool` and define a range (or use `source = "dhcp"`); on a routed-NAT
+node just add a public pool alongside `nat-transit` — no data migration needed.
 
 ## Bridge Setup — Physical Network Wiring
 
@@ -452,7 +545,7 @@ dns_servers = ["8.8.8.8"]           # DNS for VMs (optional)
 
 | Field         | Required    | Description                                                                                                                                                            |
 | ------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `name`        | Yes         | Unique pool name. Used as NATS KV key and in `AllocateAddress`.                                                                                                        |
+| `name`        | Yes         | Unique pool name. Referenced by `AllocateAddress` to target a specific pool.                                                                                           |
 | `source`      | No          | IP source: `"static"` (default) uses `range_start`/`range_end`. `"dhcp"` obtains IPs from the router's DHCP server on each VM launch.                                  |
 | `range_start` | Static only | First IP in the range. First IP is reserved for OVN gateway SNAT (unless `gateway_ip` overrides).                                                                      |
 | `range_end`   | Static only | Last IP in the range.                                                                                                                                                  |
@@ -462,6 +555,7 @@ dns_servers = ["8.8.8.8"]           # DNS for VMs (optional)
 | `region`      | No          | Scopes pool to a region. Instances in this region prefer this pool.                                                                                                    |
 | `az`          | No          | Scopes pool to an AZ. More specific than region.                                                                                                                       |
 | `dns_servers` | No          | DNS servers propagated to VMs via OVN DHCP.                                                                                                                            |
+| `gw_lrp_range_start` / `gw_lrp_range_end` | No | Reserve gateway-LRP IPs for per-VPC OVN routers. When unset, the allocator auto-derives the top 16 host IPs of the pool subnet (~15 concurrent VPCs). Widen to raise the concurrent-VPC ceiling — the `nat-transit` pool defaults to `100.127.0.16`-`100.127.0.254` (239 VPCs). Must not overlap `range_start`/`range_end`. |
 
 ### Why range_start/range_end Instead of CIDR?
 
@@ -483,13 +577,16 @@ These are different things:
 
 ```toml
 [nodes.spx1.vpcd]
-ovn_nb_addr        = "tcp:10.1.3.181:6641"   # OVN Northbound DB
-ovn_sb_addr        = "tcp:10.1.3.181:6642"   # OVN Southbound DB
+# Comma-separated list of the OVN NB/SB quorum endpoints (3 DB nodes). vpcd and
+# ovn-controller fail over across them and follow the RAFT leader.
+ovn_nb_addr        = "tcp:10.1.3.181:6641,tcp:10.1.3.182:6641,tcp:10.1.3.183:6641"
+ovn_sb_addr        = "tcp:10.1.3.181:6642,tcp:10.1.3.182:6642,tcp:10.1.3.183:6642"
 external_interface = "br-wan"                 # WAN Linux bridge name
 ```
 
 | Field                | Description                                                                                                                                                                          |
 | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `ovn_nb_addr` / `ovn_sb_addr` | OVN Northbound / Southbound DB endpoint(s). The NB and SB databases run clustered via OVSDB RAFT across the first three nodes (client ports 6641/6642, RAFT ports 6643/6644). Each node's config lists all three quorum endpoints so the loss of one DB node does not stall the control plane. A single `tcp:IP:6641` string remains valid for single-node dev. |
 | `external_interface` | Linux bridge that owns the WAN uplink on this node (e.g. `br-wan`, `br-public`). The physical NIC is enslaved to this bridge — not configured here. Different nodes may differ. |
 
 ## Pool Selection Logic
@@ -506,26 +603,10 @@ When an instance needs a public IP:
 
 ## IPAM Storage
 
-Each pool gets a NATS KV entry in bucket `spinifex-external-ipam`, keyed by pool
-name. Allocation uses CAS (Compare-And-Set) for lock-free concurrent access:
-
-```json
-{
-  "pool_name": "wan",
-  "range_start": "192.168.1.150",
-  "range_end": "192.168.1.250",
-  "allocated": {
-    "192.168.1.150": { "type": "gateway" },
-    "192.168.1.151": {
-      "type": "auto_assign",
-      "eni_id": "eni-abc",
-      "instance_id": "i-123"
-    }
-  }
-}
-```
-
-Pools are initialized from `spinifex.toml` on vpcd startup (idempotent).
+Pool allocation state is stored durably in the cluster (NATS KV bucket
+`spinifex-external-ipam`, one entry per pool) and survives restarts. Each
+allocation records the ENI and instance holding the address. Pools are
+initialized from `spinifex.toml` on vpcd startup (idempotent).
 
 ## Deployment Examples
 
@@ -1046,6 +1127,27 @@ sudo ovn-trace ext-vpc-{vpcId} \
    arp.op==1 && arp.spa==192.168.1.13 && arp.tpa==192.168.1.201'
 ```
 
+### OVN DB RAFT cluster (NB/SB replication)
+
+The NB and SB databases run clustered across the first three nodes via native
+OVSDB RAFT. A 3-node quorum tolerates the loss of one DB node; check status when
+the control plane stalls.
+
+```bash
+# NB cluster status: expect 3 servers and exactly one "Role: leader"
+sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
+
+# SB cluster status
+sudo ovn-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound
+
+# Drive a CLI at the whole quorum (fails over past a dead node)
+sudo ovn-nbctl --db=tcp:10.1.3.181:6641,tcp:10.1.3.182:6641,tcp:10.1.3.183:6641 show
+```
+
+A node showing only itself under `Servers:` never joined the cluster — confirm
+it was bootstrapped with `--db-cluster-remote-addr` pointing at the creator and
+that RAFT ports 6643/6644 are reachable between DB nodes.
+
 ### OVS (datapath / physical wiring)
 
 ```bash
@@ -1314,12 +1416,10 @@ OVNSB commit failed, force recompute next time.
 Repeated millions of times. Port binding never happens (`up: false`).
 
 **Cause:** Stale entries in the OVN Southbound DB (old chassis records, port
-bindings, datapath bindings) conflict with ovn-controller's expected state.
-Happens when `reset-dev-env.sh` cleans NB but not SB, or after ungraceful
-shutdowns.
+bindings, datapath bindings) conflict with ovn-controller's expected state,
+typically after an ungraceful shutdown.
 
-**Fix:** Delete both OVN DB files and restart. `reset-dev-env.sh` does this
-automatically:
+**Fix:** Delete both OVN DB files and restart:
 
 ```bash
 sudo systemctl stop ovn-central ovn-controller
@@ -1328,21 +1428,6 @@ sudo systemctl start ovn-central ovn-controller
 # vpcd reconcile will recreate the NB topology on next startup
 ```
 
-### Chassis Name Mismatch
-
-After mulga-999, vpcd uses the OVS-managed UUID (persisted at
-`/etc/openvswitch/system-id.conf` by the `openvswitch-switch` package and
-re-applied on every boot) as the canonical chassis identity. The chassis name
-across OVS `external_ids:system-id`, OVN SB `Chassis.name`, and NB
-`Gateway_Chassis.chassis_name` is always the same value, so the mismatch
-class that this section previously documented can no longer recur.
-
-`ReconcileFromKV` runs a `reconcileGatewayChassis` pre-step on every vpcd
-startup that deletes any stale `Gateway_Chassis` row left over from earlier
-installs (where `setup-ovn.sh` fabricated `chassis-$(hostname -s)`) and
-re-binds every gateway LRP against the live SBDB chassis. Pre-mulga-999
-brokenness recovers automatically on the first vpcd restart after upgrade —
-no manual `ovn-nbctl destroy` required.
 
 ### WAN NIC Not Enslaved to a Bridge