release: update go.mod for v1.14.0

* fix(eks): GPU node AMI declares uefi boot mode

The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but
the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under
legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at
firmware with zero serial console output.

* fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs

Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
initialize under the closed/proprietary kernel module: dmesg shows
RmInitAdapter failing with "requires use of the NVIDIA open kernel
modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to
nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia
recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no
separate open variant (userspace tools only), so it is unchanged.

* fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons

A cluster with a GPU-only nodegroup has no untainted node: the CP
carries CriticalAddonsOnly (EKS parity keeps user workloads off it)
and every worker carries nvidia.com/gpu. Without tolerations for both,
ebs-csi-controller and aws-load-balancer-controller have nowhere to
schedule and stay Pending forever. Real EKS never needs this since its
control plane isn't a schedulable node, so the stock upstream
manifests carry no such toleration.

* fix(images): give the libguestfs appliance a working resolver

The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
(127.0.0.53), which isn't running inside the appliance chroot, so DNS
is dead until package installs need it. Point it at a real resolver
before package installs, then restore the stub symlink afterward so
the shipped AMI still gets its DNS from systemd-resolved at boot like
a stock cloud image.

* fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind

A GPU's IOMMU group can include an upstream PCIe bridge when the
platform lacks ACS isolation between them. vfio-pci refuses to bind
non-endpoint bridge devices, so Claim/ReclaimByAddress must filter
bridge-class (0x06) members out of the group before touching driver
bindings, rather than failing the whole claim or leaving the bridge
rebound incorrectly.

* fix(ecs): GPU node AMI declares uefi boot mode

The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the
Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS
no OVMF/pflash firmware is attached, so the guest hangs at firmware with
zero serial console output.

Same defect and same fix as the eks-node-gpu entry: both images are built
by the same script from the same stock Ubuntu cloudimg, so both were
EFI-only and only one was corrected.

* fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch

ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary
kernel module. NVIDIA ships no proprietary module for Blackwell (GB202)
and later, so ECS GPU container-instances could not bind those cards on
any driver branch. It is the third verbatim copy of the routine and was
missed when the other two were fixed.

Also pins the driver branch across all three copies. The branch was
resolved at build time via apt-cache search | sort -rn | head -1, which
sorts on the number in the package name — but that number is a branch
alias, not a driver version (nvidia-dkms-535-server installs 580.126.20),
so the ordering key need not match what installs. Resolving also let
identical source produce different drivers on different days.

The pin lives in setup.sh rather than manifest.conf because manifest
variables are not passed into the guest: the setup script is uploaded and
executed by virt-customize --run, with no environment carried across.

* fix(eks): tolerate CP and GPU taints in argocd addon

All six argocd Deployments and its StatefulSet carried no tolerations. A
cluster with a GPU-only nodegroup has no untainted node to place them on:
the control plane carries CriticalAddonsOnly and every worker carries
nvidia.com/gpu, so the pods stay Pending forever.

Same defect the ebs-csi-controller and load-balancer-controller addons
already had. Real EKS never hits it because its control plane is not a
schedulable node, so the stock upstream manifests carry no such
toleration and the divergence is ours to maintain.

---------

Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>

.githooks/pre-commit

@@ -1,17 +0,0 @@
-#!/usr/bin/env bash
-set -e
-
-branch=$(git rev-parse --abbrev-ref HEAD)
-
-# Only enforce preflight on dev and main
-if [ "$branch" != "dev" ] && [ "$branch" != "main" ]; then
-  exit 0
-fi
-
-# Only run preflight if Go files were staged
-if git diff --cached --name-only | grep -q '\.go$'; then
-  echo "Go files staged on $branch — running make preflight..."
-  make preflight
-else
-  echo "No Go files staged, skipping preflight."
-fi

.github/actions/otel-job-id/action.yml

@@ -0,0 +1,52 @@
+name: Resolve OTEL job id
+description: >-
+  Append ci.job_id to SPINIFEX_OTEL_ATTRS so telemetry emitted by this job can
+  be pivoted straight back to its GitHub job log. Must run before the step that
+  bakes SPINIFEX_OTEL_ATTRS into a VM's telemetry.env, and requires
+  `actions: read` on the calling job.
+
+inputs:
+  job-name:
+    description: >-
+      Rendered name of the calling job, exactly as it appears in the Actions UI.
+      github.job is only the YAML key and no context exposes the numeric id, so
+      the id has to be looked up by name.
+    required: true
+  token:
+    description: Token used for the Actions jobs API lookup.
+    required: false
+    default: ${{ github.token }}
+
+runs:
+  using: composite
+  steps:
+    - name: Resolve CI job id (best-effort)
+      shell: bash
+      env:
+        GH_TOKEN: ${{ inputs.token }}
+        JOB_NAME: ${{ inputs.job-name }}
+        REPO: ${{ github.repository }}
+        RUN_ID: ${{ github.run_id }}
+      run: |
+        # Never fail the caller: on any lookup error or name mismatch ci.job_id
+        # is simply omitted and the run proceeds with the labels it already has.
+        set +e
+
+        if [ -z "${SPINIFEX_OTEL_ATTRS:-}" ]; then
+          echo "SPINIFEX_OTEL_ATTRS is unset or empty — telemetry is disabled, nothing to label"
+          exit 0
+        fi
+
+        # env.JOB_NAME rather than shell interpolation into the jq filter: job
+        # names carry quotes, slashes and non-ASCII separators that would
+        # otherwise break the expression.
+        JOB_ID=$(gh api "repos/${REPO}/actions/runs/${RUN_ID}/jobs" --paginate \
+          --jq '.jobs[] | select(.name == env.JOB_NAME) | .id' 2>/dev/null | head -n1)
+
+        if [ -n "$JOB_ID" ]; then
+          echo "SPINIFEX_OTEL_ATTRS=${SPINIFEX_OTEL_ATTRS},ci.job_id=${JOB_ID}" >> "$GITHUB_ENV"
+          echo "resolved ci.job_id=${JOB_ID} for job '${JOB_NAME}'"
+        else
+          echo "::warning::could not resolve numeric job id for '${JOB_NAME}' — continuing without ci.job_id"
+        fi
+        exit 0

.github/assets/architecture.svg

@@ -1,225 +1,114 @@
-<svg xmlns="http://www.w3.org/2000/svg" width="900" height="620" viewBox="0 0 900 620">
-  <defs>
-    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
-      <stop offset="0%" style="stop-color:#0a0a0a"/>
-      <stop offset="100%" style="stop-color:#0d1117"/>
-    </linearGradient>
-    <linearGradient id="bar-grad" x1="0%" y1="0%" x2="100%" y2="0%">
-      <stop offset="0%" style="stop-color:#161b22"/>
-      <stop offset="45%" style="stop-color:#1f2028"/>
-      <stop offset="100%" style="stop-color:#161b22"/>
-    </linearGradient>
-    <linearGradient id="client-grad" x1="0%" y1="0%" x2="100%" y2="0%">
-      <stop offset="0%" style="stop-color:#3fb95022"/>
-      <stop offset="100%" style="stop-color:#3fb95008"/>
-    </linearGradient>
-    <linearGradient id="gw-grad" x1="0%" y1="0%" x2="100%" y2="0%">
-      <stop offset="0%" style="stop-color:#DC262633"/>
-      <stop offset="100%" style="stop-color:#7F1D1D14"/>
-    </linearGradient>
-    <linearGradient id="nats-grad" x1="0%" y1="0%" x2="100%" y2="0%">
-      <stop offset="0%" style="stop-color:#F87171"/>
-      <stop offset="50%" style="stop-color:#DC2626"/>
-      <stop offset="100%" style="stop-color:#7F1D1D"/>
-    </linearGradient>
-  </defs>
-
-  <rect width="900" height="620" fill="url(#bg)" rx="12"/>
-
-  <!-- Title bar -->
-  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">SPINIFEX ARCHITECTURE</text>
-  <text x="24" y="52" font-family="'Segoe UI', system-ui, sans-serif" font-size="16" fill="#f5f5f5" font-weight="500">AWS SDK → Gateway → NATS → Service daemons</text>
-  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">stateless · message-driven · horizontally scalable</text>
-
-  <!-- ═══ CLIENT LAYER ════════════════════════════════════ -->
-  <rect x="16" y="72" width="868" height="78" rx="8" fill="url(#client-grad)" stroke="#3fb950" stroke-width="1" stroke-opacity="0.5"/>
-  <text x="32" y="93" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" font-weight="700" letter-spacing="1">CLIENTS</text>
-  <text x="32" y="108" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">Standard AWS tooling — unmodified, no shims, no custom SDK</text>
-
-  <g transform="translate(32, 120)" font-family="'SF Mono', ui-monospace, monospace" font-size="10">
-    <rect x="0" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="43" y="15" fill="#7ee787" text-anchor="middle">aws-cli</text>
-
-    <rect x="94" y="0" width="96" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="142" y="15" fill="#7ee787" text-anchor="middle">Terraform</text>
-
-    <rect x="198" y="0" width="76" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="236" y="15" fill="#7ee787" text-anchor="middle">boto3</text>
-
-    <rect x="282" y="0" width="98" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="331" y="15" fill="#7ee787" text-anchor="middle">aws-sdk-go</text>
-
-    <rect x="388" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="431" y="15" fill="#7ee787" text-anchor="middle">kubectl</text>
-
-    <rect x="482" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="525" y="15" fill="#7ee787" text-anchor="middle">Packer</text>
-
-    <rect x="576" y="0" width="120" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
-    <text x="636" y="15" fill="#7ee787" text-anchor="middle">your application</text>
-  </g>
-
-  <!-- ═══ HTTPS + SigV4 connector ═══════════════════════════ -->
-  <line x1="450" y1="150" x2="450" y2="178" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
-  <rect x="390" y="154" width="120" height="18" rx="9" fill="#0a0a0a" stroke="#DC2626" stroke-width="0.8" stroke-opacity="0.5"/>
-  <text x="450" y="166" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#F87171" text-anchor="middle">HTTPS · SigV4 · :9999</text>
-
-  <!-- ═══ AWS GATEWAY ════════════════════════════════════ -->
-  <rect x="16" y="178" width="868" height="68" rx="8" fill="url(#gw-grad)" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.75"/>
-  <text x="32" y="200" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" font-weight="700" letter-spacing="1">AWS GATEWAY</text>
-  <text x="32" y="215" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">spinifex/services/awsgw · TLS termination · SigV4 verification · IAM policy eval · NATS routing</text>
-
-  <g transform="translate(32, 224)" font-family="'SF Mono', ui-monospace, monospace" font-size="9">
-    <rect x="0" y="0" width="78" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="39" y="11" fill="#FCA5A5" text-anchor="middle">TLS 1.3</text>
-
-    <rect x="86" y="0" width="78" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="125" y="11" fill="#FCA5A5" text-anchor="middle">SigV4 auth</text>
-
-    <rect x="172" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="221" y="11" fill="#FCA5A5" text-anchor="middle">policy eval</text>
-
-    <rect x="278" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="327" y="11" fill="#FCA5A5" text-anchor="middle">req routing</text>
-
-    <rect x="384" y="0" width="88" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="428" y="11" fill="#FCA5A5" text-anchor="middle">rate-limit</text>
-
-    <rect x="480" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
-    <text x="529" y="11" fill="#FCA5A5" text-anchor="middle">audit log</text>
-  </g>
-
-  <!-- ═══ NATS BUS ═══════════════════════════════════════ -->
-  <line x1="450" y1="246" x2="450" y2="272" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
-  <rect x="16" y="272" width="868" height="34" rx="6" fill="url(#nats-grad)" opacity="0.92"/>
-  <text x="32" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#f5f5f5" font-weight="700" letter-spacing="1.5">NATS BUS</text>
-  <text x="140" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#fee2e2" opacity="0.95">ec2.*   ebs.*   s3.*   vpc.*   iam.*</text>
-  <text x="876" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#fee2e2" text-anchor="end" opacity="0.9">queue groups · fan-out · request/reply</text>
-
-  <!-- ═══ DAEMON FANOUT CONNECTORS ═══════════════════════ -->
-  <g stroke="#DC2626" stroke-width="0.9" stroke-dasharray="3,3" opacity="0.55" fill="none">
-    <line x1="100" y1="306" x2="100" y2="328"/>
-    <line x1="275" y1="306" x2="275" y2="328"/>
-    <line x1="450" y1="306" x2="450" y2="328"/>
-    <line x1="625" y1="306" x2="625" y2="328"/>
-    <line x1="800" y1="306" x2="800" y2="328"/>
-  </g>
-
-  <!-- ═══ SERVICE DAEMONS ROW ═══════════════════════════ -->
-  <!-- EC2 -->
-  <g transform="translate(28, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">ec2.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">RunInstances</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">TerminateInstances</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">QMP · cloud-init</text>
-  </g>
-
-  <!-- EBS -->
-  <g transform="translate(204, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">ebs.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">CreateVolume</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">AttachVolume</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">snapshots · WAL</text>
-  </g>
-
-  <!-- S3 -->
-  <g transform="translate(380, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">s3.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">PutObject · GetObject</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Multipart · presigned</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">erasure-coded</text>
-  </g>
-
-  <!-- VPC -->
-  <g transform="translate(556, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">vpc.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Subnets · SGs · RTs</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">ALB · NAT · IGW</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">OVN · OVS</text>
-  </g>
-
-  <!-- IAM -->
-  <g transform="translate(732, 328)">
-    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">iam.* daemon</text>
-    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
-    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Users · Roles · Keys</text>
-    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Policy evaluation</text>
-    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">audit trail</text>
-  </g>
-
-  <!-- ═══ DAEMON → BACKEND CONNECTORS ═══════════════════ -->
-  <g stroke="#71717a" stroke-width="0.8" stroke-dasharray="2,3" opacity="0.55" fill="none">
-    <line x1="100" y1="410" x2="100" y2="440"/>
-    <line x1="276" y1="410" x2="276" y2="440"/>
-    <line x1="452" y1="410" x2="452" y2="440"/>
-    <line x1="628" y1="410" x2="628" y2="440"/>
-    <line x1="804" y1="410" x2="804" y2="440"/>
-  </g>
-
-  <!-- ═══ BACKEND RESOURCES BAR ═════════════════════════ -->
-  <rect x="16" y="440" width="868" height="26" rx="5" fill="url(#bar-grad)"/>
-  <text x="450" y="458" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle" letter-spacing="1.5">BACKEND RESOURCES · local hardware · no cloud dependency</text>
-
-  <!-- ═══ BACKEND ROW ═══════════════════════════════════ -->
-  <!-- QEMU/KVM -->
-  <g transform="translate(28, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">hypervisor</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">host kernel · virtio</text>
-  </g>
-
-  <!-- Viperblock -->
-  <g transform="translate(204, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Viperblock</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">block storage</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">NVMe WAL · NBD</text>
-  </g>
-
-  <!-- Predastore -->
-  <g transform="translate(380, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Predastore</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">object storage</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">Reed–Solomon</text>
-  </g>
-
-  <!-- OVN/OVS -->
-  <g transform="translate(556, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">OVN / OVS</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">SDN fabric</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">overlays · flow rules</text>
-  </g>
-
-  <!-- IAM store -->
-  <g transform="translate(732, 474)">
-    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
-    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Identity store</text>
-    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">users · keys · rules</text>
-    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">local · audit-logged</text>
-  </g>
-
-  <!-- ═══ FOOTER NOTES ══════════════════════════════════ -->
-  <g transform="translate(16, 558)">
-    <rect width="868" height="48" rx="6" fill="#0a0a0a" stroke="#27272a" stroke-width="0.8"/>
-    <text x="24" y="20" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">
-      <tspan fill="#F87171" font-weight="600">Queue groups</tspan> load-balance ops across daemon replicas.
-      <tspan fill="#F87171" font-weight="600"> Fan-out</tspan> (no queue group) dispatches describe calls to every node.
-    </text>
-    <text x="24" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">
-      Daemons are <tspan fill="#F87171" font-weight="600">stateless</tspan> — scale horizontally by starting more.
-      No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.
-    </text>
-  </g>
-</svg>
+<svg xmlns="http://www.w3.org/2000/svg" width="1020" height="770" viewBox="0 0 1020 770" role="img" aria-label="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus; stateless ec2/ebs/s3/vpc/iam daemons answer and drive local backends (QEMU/KVM, Viperblock, Predastore, OVN/OVS, identity store).">
+<defs><linearGradient id="nats" x1="0" y1="0" x2="1" y2="0"><stop offset="0" stop-color="#F5652E"/><stop offset="1" stop-color="#C63C0E"/></linearGradient></defs>
+<rect x="0.0" y="0.0" width="1020.0" height="770.0" rx="20" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="28.0" y="42.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#EE4B12" font-weight="600" text-anchor="start" letter-spacing="2.5">SPINIFEX ARCHITECTURE</text>
+<text x="992.0" y="42.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#9AA1A8" font-weight="400" text-anchor="end">stateless · message-driven · horizontally scalable</text>
+<text x="28.0" y="72.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="23" fill="#F5F3EE" font-weight="800" text-anchor="start">AWS SDK → Gateway → NATS → service daemons</text>
+<rect x="28.0" y="92.0" width="964.0" height="92.0" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="48.0" y="119.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">CLIENTS</text>
+<text x="48.0" y="140.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#5B636C" font-weight="400" text-anchor="start">Standard AWS tooling, unmodified. No shims, no custom SDK.</text>
+<rect x="46.0" y="153.0" width="68.2" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="80.1" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">aws-cli</text>
+<rect x="123.2" y="153.0" width="81.4" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="163.9" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">Terraform</text>
+<rect x="213.6" y="153.0" width="55.0" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="241.1" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">boto3</text>
+<rect x="277.6" y="153.0" width="88.0" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="321.6" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">aws-sdk-go</text>
+<rect x="374.6" y="153.0" width="68.2" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="408.7" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">kubectl</text>
+<rect x="451.8" y="153.0" width="61.6" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="482.6" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">Packer</text>
+<rect x="522.4" y="153.0" width="74.8" height="22.0" rx="7" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="559.8" y="167.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">your app</text>
+<line x1="510.0" y1="186.0" x2="510.0" y2="192.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
+<rect x="429.9" y="191.0" width="160.3" height="22.0" rx="11" fill="#1E242B" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.4"/>
+<text x="510.0" y="205.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#EE4B12" font-weight="400" text-anchor="middle" fill-opacity="0.9">HTTPS · SigV4 · :9999</text>
+<line x1="510.0" y1="212.0" x2="510.0" y2="220.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
+<rect x="28.0" y="224.0" width="964.0" height="94.0" rx="13" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
+<text x="48.0" y="251.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#EE4B12" font-weight="700" text-anchor="start" letter-spacing="1.5">AWS GATEWAY</text>
+<text x="48.0" y="272.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">spinifex/services/awsgw · TLS termination · SigV4 verification · IAM policy eval · NATS routing</text>
+<rect x="46.0" y="287.0" width="68.2" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="80.1" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">TLS 1.3</text>
+<rect x="123.2" y="287.0" width="88.0" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="167.2" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">SigV4 auth</text>
+<rect x="220.2" y="287.0" width="94.6" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="267.5" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">policy eval</text>
+<rect x="323.8" y="287.0" width="94.6" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="371.1" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">req routing</text>
+<rect x="427.4" y="287.0" width="88.0" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="471.4" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">rate-limit</text>
+<rect x="524.4" y="287.0" width="81.4" height="22.0" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="565.1" y="301.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">audit log</text>
+<line x1="510.0" y1="320.0" x2="510.0" y2="332.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
+<rect x="28.0" y="334.0" width="964.0" height="52.0" rx="12" fill="url(#nats)" fill-opacity="1.0" stroke="none" stroke-width="1" stroke-opacity="1.0"/>
+<rect x="28.0" y="334.0" width="964.0" height="52.0" rx="12" fill="none" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55"/>
+<text x="50.0" y="365.6" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="16" fill="#F5F3EE" font-weight="800" text-anchor="start">NATS BUS</text>
+<text x="178.0" y="364.6" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#F5F3EE" font-weight="400" text-anchor="start" fill-opacity="0.92">ec2.*  ebs.*  s3.*  vpc.*  iam.*</text>
+<text x="972.0" y="363.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F5F3EE" font-weight="400" text-anchor="end" fill-opacity="0.9">queue groups · fan-out · request/reply</text>
+<line x1="119.6" y1="388.0" x2="119.6" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<line x1="314.8" y1="388.0" x2="314.8" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<line x1="510.0" y1="388.0" x2="510.0" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<line x1="705.2" y1="388.0" x2="705.2" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<line x1="900.4" y1="388.0" x2="900.4" y2="408.0" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
+<rect x="28.0" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="44.0" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">ec2.* daemon</text>
+<line x1="44.0" y1="448.0" x2="195.2" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="44.0" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">RunInstances</text>
+<text x="44.0" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">TerminateInstances</text>
+<text x="44.0" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">QMP · cloud-init</text>
+<rect x="223.2" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="239.2" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">ebs.* daemon</text>
+<line x1="239.2" y1="448.0" x2="390.4" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="239.2" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">CreateVolume</text>
+<text x="239.2" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">AttachVolume</text>
+<text x="239.2" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">snapshots · WAL</text>
+<rect x="418.4" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="434.4" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">s3.* daemon</text>
+<line x1="434.4" y1="448.0" x2="585.6" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="434.4" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">PutObject · GetObject</text>
+<text x="434.4" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">Multipart · presigned</text>
+<text x="434.4" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">erasure-coded</text>
+<rect x="613.6" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="629.6" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">vpc.* daemon</text>
+<line x1="629.6" y1="448.0" x2="780.8" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="629.6" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">Subnets · SGs · RTs</text>
+<text x="629.6" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">ALB · NAT · IGW</text>
+<text x="629.6" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">OVN · OVS</text>
+<rect x="808.8" y="412.0" width="183.2" height="104.0" rx="11" fill="#EE4B12" fill-opacity="0.045" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="824.8" y="438.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="700" text-anchor="start">iam.* daemon</text>
+<line x1="824.8" y1="448.0" x2="976.0" y2="448.0" stroke="#EE4B12" stroke-opacity="0.22" stroke-width="1"/>
+<text x="824.8" y="468.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">Users · Roles · Keys</text>
+<text x="824.8" y="486.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">Policy evaluation</text>
+<text x="824.8" y="502.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">audit trail</text>
+<line x1="119.6" y1="516.0" x2="119.6" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<line x1="314.8" y1="516.0" x2="314.8" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<line x1="510.0" y1="516.0" x2="510.0" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<line x1="705.2" y1="516.0" x2="705.2" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<line x1="900.4" y1="516.0" x2="900.4" y2="540.0" stroke="#5B636C" stroke-width="1.3" stroke-opacity="0.35" stroke-dasharray="2 5"/>
+<rect x="28.0" y="530.0" width="964.0" height="30.0" rx="9" fill="#1E242B" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="510.0" y="548.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle" letter-spacing="1">BACKEND RESOURCES  ·  local hardware  ·  no cloud dependency</text>
+<rect x="28.0" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="119.6" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">QEMU / KVM</text>
+<text x="119.6" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">hypervisor</text>
+<text x="119.6" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">host kernel · virtio</text>
+<rect x="223.2" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="314.8" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">Viperblock</text>
+<text x="314.8" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">block storage</text>
+<text x="314.8" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">NVMe WAL · NBD</text>
+<rect x="418.4" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="510.0" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">Predastore</text>
+<text x="510.0" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">object storage</text>
+<text x="510.0" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Reed-Solomon</text>
+<rect x="613.6" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="705.2" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">OVN / OVS</text>
+<text x="705.2" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">SDN fabric</text>
+<text x="705.2" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">overlays · flow rules</text>
+<rect x="808.8" y="572.0" width="183.2" height="84.0" rx="11" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="900.4" y="602.7" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13.5" fill="#F5F3EE" font-weight="700" text-anchor="middle">Identity store</text>
+<text x="900.4" y="624.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="middle">users · keys · rules</text>
+<text x="900.4" y="642.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">local · audit-logged</text>
+<rect x="28.0" y="670.0" width="964.0" height="72.0" rx="11" fill="#181D23" fill-opacity="0.6" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="48.0" y="697.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400"><tspan fill="#EE4B12" font-weight="700">Queue groups</tspan><tspan fill="#5B636C"> load-balance ops across daemon replicas. </tspan><tspan fill="#EE4B12" font-weight="700">Fan-out</tspan><tspan fill="#5B636C"> (no queue group) dispatches describe calls to every node.</tspan></text>
+<text x="48.0" y="720.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400"><tspan fill="#5B636C">Daemons are </tspan><tspan fill="#EE4B12" font-weight="700">stateless</tspan><tspan fill="#5B636C">: scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.</tspan></text>
+</svg>

.github/assets/platform.svg

@@ -1,144 +1,84 @@
-<svg xmlns="http://www.w3.org/2000/svg" width="900" height="560" viewBox="0 0 900 560">
-  <defs>
-    <linearGradient id="pf-bg" x1="0%" y1="0%" x2="100%" y2="100%">
-      <stop offset="0%" style="stop-color:#0a0a0a"/>
-      <stop offset="100%" style="stop-color:#0d1117"/>
-    </linearGradient>
-    <linearGradient id="hw-grad" x1="0%" y1="0%" x2="0%" y2="100%">
-      <stop offset="0%" style="stop-color:#52525b;stop-opacity:0.18"/>
-      <stop offset="100%" style="stop-color:#52525b;stop-opacity:0.06"/>
-    </linearGradient>
-    <linearGradient id="os-grad" x1="0%" y1="0%" x2="0%" y2="100%">
-      <stop offset="0%" style="stop-color:#a855f7;stop-opacity:0.18"/>
-      <stop offset="100%" style="stop-color:#a855f7;stop-opacity:0.05"/>
-    </linearGradient>
-    <linearGradient id="spx-grad" x1="0%" y1="0%" x2="0%" y2="100%">
-      <stop offset="0%" style="stop-color:#DC2626;stop-opacity:0.28"/>
-      <stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0.1"/>
-    </linearGradient>
-    <linearGradient id="app-grad" x1="0%" y1="0%" x2="0%" y2="100%">
-      <stop offset="0%" style="stop-color:#3fb950;stop-opacity:0.18"/>
-      <stop offset="100%" style="stop-color:#3fb950;stop-opacity:0.05"/>
-    </linearGradient>
-  </defs>
-
-  <rect width="900" height="560" fill="url(#pf-bg)" rx="12"/>
-
-  <!-- Subtle mesh dots background -->
-  <g opacity="0.12" fill="#7F1D1D">
-    <circle cx="40" cy="40" r="1"/><circle cx="120" cy="30" r="1"/><circle cx="220" cy="50" r="1"/>
-    <circle cx="320" cy="25" r="1"/><circle cx="440" cy="45" r="1"/><circle cx="560" cy="30" r="1"/>
-    <circle cx="680" cy="50" r="1"/><circle cx="780" cy="30" r="1"/><circle cx="860" cy="45" r="1"/>
-    <circle cx="60" cy="530" r="1"/><circle cx="160" cy="520" r="1"/><circle cx="260" cy="540" r="1"/>
-    <circle cx="380" cy="525" r="1"/><circle cx="500" cy="540" r="1"/><circle cx="620" cy="520" r="1"/>
-    <circle cx="740" cy="540" r="1"/><circle cx="840" cy="525" r="1"/>
-  </g>
-
-  <!-- Title -->
-  <text x="450" y="32" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" text-anchor="middle" letter-spacing="2.8">THE MULGA PLATFORM</text>
-  <text x="450" y="58" font-family="'Segoe UI', system-ui, sans-serif" font-size="20" font-weight="600" fill="#f5f5f5" text-anchor="middle">AWS-style workflows, on your own stack</text>
-
-  <!-- Layer 4: Applications (top) -->
-  <g transform="translate(60, 95)">
-    <rect width="780" height="88" rx="10" fill="url(#app-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
-    <text x="20" y="26" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" letter-spacing="1.5" font-weight="600">LAYER 4 · APPLICATIONS &amp; TOOLING</text>
-    <text x="20" y="48" font-family="'Segoe UI', sans-serif" font-size="15" fill="#f5f5f5" font-weight="500">Your workloads — unmodified AWS tooling</text>
-
-    <g transform="translate(20, 60)" font-family="'Segoe UI', sans-serif" font-size="10.5">
-      <rect x="0" y="0" width="70" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="35" y="14" fill="#7ee787" text-anchor="middle">aws-cli</text>
-
-      <rect x="78" y="0" width="78" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="117" y="14" fill="#7ee787" text-anchor="middle">Terraform</text>
-
-      <rect x="164" y="0" width="88" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="208" y="14" fill="#7ee787" text-anchor="middle">AWS SDKs</text>
-
-      <rect x="260" y="0" width="88" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="304" y="14" fill="#7ee787" text-anchor="middle">Kubernetes</text>
-
-      <rect x="356" y="0" width="70" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="391" y="14" fill="#7ee787" text-anchor="middle">Packer</text>
-
-      <rect x="434" y="0" width="98" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
-      <text x="483" y="14" fill="#7ee787" text-anchor="middle">Your Services</text>
-    </g>
-  </g>
-
-  <!-- Down arrow -->
-  <g transform="translate(450, 188)" opacity="0.45">
-    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
-    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
-  </g>
-
-  <!-- Layer 3: Spinifex (highlighted) -->
-  <g transform="translate(60, 215)">
-    <rect width="780" height="140" rx="10" fill="url(#spx-grad)" stroke="#DC2626" stroke-width="1.5" stroke-opacity="0.75"/>
-    <text x="20" y="26" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" letter-spacing="1.5" font-weight="600">LAYER 3 · SPINIFEX · AWS ORCHESTRATION</text>
-    <text x="20" y="50" font-family="'Segoe UI', sans-serif" font-size="15" fill="#f5f5f5" font-weight="500">AWS Gateway → NATS → Service Daemons · SigV4, IAM, VPC-aware</text>
-
-    <g transform="translate(20, 66)" font-family="'Segoe UI', sans-serif">
-      <!-- EC2 -->
-      <rect x="0" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
-      <text x="59" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">EC2</text>
-      <text x="59" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">QEMU/KVM VMs</text>
-      <text x="59" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">cloud-init · AMIs</text>
-
-      <!-- EBS -->
-      <rect x="126" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
-      <text x="185" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">EBS</text>
-      <text x="185" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">Viperblock</text>
-      <text x="185" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">WAL · NBD · snapshots</text>
-
-      <!-- S3 -->
-      <rect x="252" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
-      <text x="311" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">S3</text>
-      <text x="311" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">Predastore</text>
-      <text x="311" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Reed–Solomon EC</text>
-
-      <!-- VPC -->
-      <rect x="378" y="0" width="118" height="58" rx="6" fill="#7F1D1D" fill-opacity="0.22" stroke="#F87171" stroke-opacity="0.55" stroke-width="1"/>
-      <text x="437" y="18" font-size="13" fill="#FCA5A5" text-anchor="middle" font-weight="600">VPC</text>
-      <text x="437" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">OVN / OVS</text>
-      <text x="437" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">subnets · SG · ALB</text>
-
-      <!-- IAM -->
-      <rect x="504" y="0" width="118" height="58" rx="6" fill="#7F1D1D" fill-opacity="0.22" stroke="#F87171" stroke-opacity="0.55" stroke-width="1"/>
-      <text x="563" y="18" font-size="13" fill="#FCA5A5" text-anchor="middle" font-weight="600">IAM</text>
-      <text x="563" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">SigV4 · policies</text>
-      <text x="563" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">users · roles · keys</text>
-
-      <!-- Control -->
-      <rect x="630" y="0" width="110" height="58" rx="6" fill="#27272a" fill-opacity="0.8" stroke="#71717a" stroke-opacity="0.6" stroke-width="1"/>
-      <text x="685" y="18" font-size="13" fill="#e4e4e7" text-anchor="middle" font-weight="600">Control</text>
-      <text x="685" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">NATS bus</text>
-      <text x="685" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">spx · daemons</text>
-    </g>
-  </g>
-
-  <!-- Down arrow -->
-  <g transform="translate(450, 360)" opacity="0.45">
-    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
-    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
-  </g>
-
-  <!-- Layer 2: OS -->
-  <g transform="translate(60, 387)">
-    <rect width="780" height="62" rx="10" fill="url(#os-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.5"/>
-    <text x="20" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" letter-spacing="1.5" font-weight="600">LAYER 2 · OPERATING SYSTEM</text>
-    <text x="20" y="44" font-family="'Segoe UI', sans-serif" font-size="14" fill="#f5f5f5">Debian / Ubuntu LTS · systemd · KVM · Linux bridge · headless autoinstall</text>
-  </g>
-
-  <!-- Down arrow -->
-  <g transform="translate(450, 454)" opacity="0.45">
-    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
-    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
-  </g>
-
-  <!-- Layer 1: Hardware -->
-  <g transform="translate(60, 481)">
-    <rect width="780" height="62" rx="10" fill="url(#hw-grad)" stroke="#71717a" stroke-width="1.2" stroke-opacity="0.6"/>
-    <text x="20" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#a1a1aa" letter-spacing="1.5" font-weight="600">LAYER 1 · HARDWARE</text>
-    <text x="20" y="44" font-family="'Segoe UI', sans-serif" font-size="14" fill="#f5f5f5">Single node · multi-node cluster · edge appliance · air-gapped rack · commodity x86 / ARM</text>
-  </g>
-</svg>
+<svg xmlns="http://www.w3.org/2000/svg" width="1000" height="664" viewBox="0 0 1000 664" role="img" aria-label="The Spinifex stack: applications and AWS tooling on top, the Spinifex AWS-compatible service layer, running on standard Linux on CPU/GPU hardware, deployable to Neocloud, on-prem or edge.">
+<rect x="0.0" y="0.0" width="1000.0" height="664.0" rx="20" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<rect x="28.0" y="204.0" width="950.0" height="438.0" rx="16" fill="none" fill-opacity="1.0" stroke="#9AA1A8" stroke-width="1" stroke-opacity="0.12" stroke-dasharray="2 4"/>
+<rect x="25.0" y="201.0" width="950.0" height="438.0" rx="16" fill="none" fill-opacity="1.0" stroke="#9AA1A8" stroke-width="1" stroke-opacity="0.2" stroke-dasharray="2 4"/>
+<rect x="22.0" y="198.0" width="950.0" height="438.0" rx="16" fill="#181D23" fill-opacity="0.25" stroke="#9AA1A8" stroke-width="1.3" stroke-opacity="0.45" stroke-dasharray="2 4"/>
+<text x="36.0" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="500" text-anchor="start">Apps</text>
+<rect x="136.0" y="34.0" width="836.0" height="54.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="156.0" y="66.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="15" fill="#F5F3EE" font-weight="700" text-anchor="start">Your infrastructure &amp; apps</text>
+<rect x="516.7" y="51.0" width="89.3" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.16" stroke="none" stroke-width="1" stroke-opacity="1.0"/>
+<text x="561.4" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#EE4B12" font-weight="500" text-anchor="middle">AI training</text>
+<rect x="613.0" y="51.0" width="76.7" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.16" stroke="none" stroke-width="1" stroke-opacity="1.0"/>
+<text x="651.4" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#EE4B12" font-weight="500" text-anchor="middle">Inference</text>
+<rect x="696.7" y="51.0" width="79.0" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="736.2" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Web &amp; APIs</text>
+<rect x="782.7" y="51.0" width="72.7" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="819.1" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Databases</text>
+<rect x="862.4" y="51.0" width="91.6" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="908.2" y="64.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Batch &amp; data</text>
+<text x="36.0" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="500" text-anchor="start">Tooling</text>
+<rect x="136.0" y="102.0" width="836.0" height="54.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="156.0" y="134.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="15" fill="#F5F3EE" font-weight="700" text-anchor="start">AWS tooling</text>
+<rect x="600.9" y="119.0" width="64.1" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.45"/>
+<text x="633.0" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.85">aws-cli</text>
+<rect x="672.0" y="119.0" width="76.7" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.45"/>
+<text x="710.4" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.85">Terraform</text>
+<rect x="755.7" y="119.0" width="64.1" height="20.0" rx="10" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.45"/>
+<text x="787.8" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.85">kubectl</text>
+<rect x="826.8" y="119.0" width="66.4" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="860.0" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">AWS SDKs</text>
+<rect x="900.2" y="119.0" width="53.8" height="20.0" rx="10" fill="none" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1"/>
+<text x="927.1" y="132.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="middle">Packer</text>
+<text x="484.0" y="184.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="end">↓  AWS API · SigV4</text>
+<text x="624.0" y="184.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">RESPONSE  ↑</text>
+<text x="136.0" y="231.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="15" fill="#F5F3EE" font-weight="700" text-anchor="start">One portable stack.</text>
+<text x="136.0" y="249.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="start">Runs the same on a single node, cluster, edge, or air-gapped rack.</text>
+<rect x="752.3" y="222.0" width="72.4" height="22.0" rx="6" fill="none" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="788.5" y="236.5" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="500" text-anchor="middle">NEOCLOUD</text>
+<rect x="832.7" y="222.0" width="66.1" height="22.0" rx="6" fill="none" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="865.7" y="236.5" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="500" text-anchor="middle">ON-PREM</text>
+<rect x="906.8" y="222.0" width="47.2" height="22.0" rx="6" fill="none" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
+<text x="930.4" y="236.5" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="500" text-anchor="middle">EDGE</text>
+<text x="152.0" y="295.6" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="16" fill="#F5F3EE" font-weight="800" text-anchor="start">AWS orchestration</text>
+<text x="956.0" y="293.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#EE4B12" font-weight="400" text-anchor="end" fill-opacity="0.9">Gateway → NATS → daemons</text>
+<rect x="150.0" y="312.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="164.0" y="333.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">EC2</text>
+<text x="164.0" y="349.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Compute</text>
+<text x="164.0" y="363.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">QEMU / KVM</text>
+<rect x="422.7" y="312.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="436.7" y="333.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">EBS</text>
+<text x="436.7" y="349.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Block storage</text>
+<text x="436.7" y="363.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">Viperblock</text>
+<rect x="695.3" y="312.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="709.3" y="333.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">S3</text>
+<text x="709.3" y="349.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Object storage</text>
+<text x="709.3" y="363.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">Predastore</text>
+<rect x="150.0" y="382.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="164.0" y="403.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">VPC</text>
+<text x="164.0" y="419.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Networking</text>
+<text x="164.0" y="433.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">OVN / OVS</text>
+<rect x="422.7" y="382.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="436.7" y="403.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">IAM</text>
+<text x="436.7" y="419.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Identity</text>
+<text x="436.7" y="433.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">SigV4 · policies</text>
+<rect x="695.3" y="382.0" width="262.7" height="60.0" rx="9" fill="#14181D" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="709.3" y="403.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="13" fill="#EE4B12" font-weight="600" text-anchor="start">EKS</text>
+<text x="709.3" y="419.0" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12.5" fill="#F3F1EC" font-weight="400" text-anchor="start">Kubernetes</text>
+<text x="709.3" y="433.0" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">K8s-conformant</text>
+<rect x="150.0" y="452.0" width="808.0" height="32.0" rx="8" fill="#14181D" fill-opacity="0.6" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0" stroke-dasharray="3 3"/>
+<text x="164.0" y="472.2" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#9AA1A8" font-weight="400" text-anchor="start">+ ALB · ECS · ECR · RDS and more of the core AWS services</text>
+<text x="944.0" y="471.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#EE4B12" font-weight="400" text-anchor="end">See all →</text>
+<text x="36.0" y="386.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#EE4B12" font-weight="500" text-anchor="start">Spinifex</text>
+<rect x="136.0" y="268.0" width="836.0" height="230.0" rx="12" fill="#EE4B12" fill-opacity="0.07" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
+<text x="36.0" y="536.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="500" text-anchor="start">OS</text>
+<rect x="136.0" y="510.0" width="836.0" height="46.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="156.0" y="537.9" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="700" text-anchor="start">Operating system</text>
+<text x="952.0" y="536.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="end">Debian / Ubuntu LTS · systemd · KVM · Linux bridge · Open vSwitch</text>
+<text x="36.0" y="596.7" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="500" text-anchor="start">Hardware</text>
+<rect x="136.0" y="568.0" width="413.0" height="50.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="156.0" y="597.9" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="700" text-anchor="start">CPU</text>
+<text x="533.0" y="596.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="end">Intel · AMD · ARM</text>
+<rect x="559.0" y="568.0" width="413.0" height="50.0" rx="12" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
+<text x="579.0" y="597.9" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="700" text-anchor="start">GPU</text>
+<text x="956.0" y="596.9" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#EE4B12" font-weight="400" text-anchor="end">NVIDIA · AMD</text>
+</svg>

.github/resolve-sibling-refs.sh

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+# Resolve which branch of each sibling repo (viperblock, predastore) the CI
+# workspace should build against. A cross-repo change is developed on a branch
+# of the SAME name in each repo; build against that branch when it exists so the
+# set can be tested together, and fall back to dev otherwise. This mirrors the
+# resolution the e2e path already does in
+# mulga/scripts/tofu-cluster/build-install-artifacts.sh, so unit CI and e2e CI
+# agree on how sibling branches are selected.
+#
+# Reads $BRANCH (the triggering head/ref name) and writes viperblock=<ref> and
+# predastore=<ref> to $GITHUB_OUTPUT for the checkout steps to consume.
+set -euo pipefail
+
+: "${BRANCH:?BRANCH must be set to the triggering branch name}"
+: "${GITHUB_OUTPUT:?GITHUB_OUTPUT must be set}"
+
+for repo in viperblock predastore; do
+  if git ls-remote --exit-code --heads \
+      "https://github.com/mulgadc/${repo}.git" "$BRANCH" >/dev/null 2>&1; then
+    ref="$BRANCH"
+  else
+    ref="dev"
+  fi
+  echo "${repo}: building against ${ref}"
+  echo "${repo}=${ref}" >>"$GITHUB_OUTPUT"
+done

.github/scripts/suite-sets.sh

@@ -0,0 +1,58 @@
+# shellcheck shell=bash
+# shellcheck disable=SC2034  # every variable here is consumed by the sourcing workflow
+# Source from a workflow `run:` block:
+#   . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+#
+# Which e2e suites each topology can run. Sourced by every workflow that drives
+# suites (e2e.yml, e2e-suites.yml, e2e-nightly.yml) so the lists cannot drift
+# apart — they previously disagreed three ways.
+#
+# Eligibility is a property of the suite, not of the workflow: a suite belongs
+# to a topology when all of its assertions are reachable there. lb is
+# multi-only for exactly that reason — on a single node every internet-facing
+# subtest skips for want of a peer, so a single-node lb run reports green
+# having exercised a fraction of the suite.
+
+# Suites runnable against a one-node environment.
+#
+# partialblock gates merges because it earned it, not because it is new: on one
+# node, same binary and workload, it fails against the pre-fix engine (1758 of
+# 8192 halves reverted a generation) and passes against the fixed one (0 of
+# 8192). A suite that has never been shown to go red for the reason it claims
+# does not belong here.
+E2E_SUITES_SINGLE="single iam cert eks ecs storagegrowth partialblock"
+
+# Suites runnable against a multi-node environment.
+E2E_SUITES_MULTI="multinode lb cert"
+
+# grep -xE alternation form, for narrowing a requested set to the eligible one.
+E2E_SUITES_SINGLE_RE="$(printf '%s' "${E2E_SUITES_SINGLE}" | tr ' ' '|')"
+E2E_SUITES_MULTI_RE="$(printf '%s' "${E2E_SUITES_MULTI}" | tr ' ' '|')"
+
+# e2e-nightly runs a narrower set than the two sets above, deliberately: its
+# ~19 cells each have a 35-minute budget and exist to prove that every
+# install / network / host-OS permutation boots and serves. Suite breadth is
+# the PR gate's job, and it already runs on every push. These are subsets, and
+# e2e_suites_assert_subset below enforces that.
+E2E_SUITES_NIGHTLY_SINGLE="single cert"
+E2E_SUITES_NIGHTLY_MULTI="multinode cert lb"
+
+# Fail loudly when a narrowed list names a suite its topology cannot run. This
+# is the guard that stops the lists drifting again: a suite added to a nightly
+# subset without also being declared eligible above stops the job here rather
+# than silently skipping half its assertions in CI.
+#   e2e_suites_assert_subset <label> <subset> <eligible-set>
+e2e_suites_assert_subset() {
+  local label="$1" subset="$2" eligible="$3" suite found eligible_suite
+  for suite in ${subset}; do
+    found=0
+    for eligible_suite in ${eligible}; do
+      if [ "${suite}" = "${eligible_suite}" ]; then found=1; break; fi
+    done
+    if [ "${found}" -ne 1 ]; then
+      echo "::error::${label} names suite '${suite}', which is not eligible for that topology (eligible: ${eligible}). Fix the list in .github/scripts/suite-sets.sh." >&2
+      return 1
+    fi
+  done
+  return 0
+}

.github/workflows/e2e-baremetal.yml

@@ -9,6 +9,11 @@ on:
         required: false
         type: string
         default: ""
+      otel-attrs:
+        description: "OTEL_RESOURCE_ATTRIBUTES value for this run, e.g. ci.run_id=123,ci.cell=20 (caller-specific, so it's an input rather than a workflow-level default)"
+        required: false
+        type: string
+        default: ""
 
 permissions:
   contents: read
@@ -16,6 +21,10 @@ permissions:
 
 env:
   GIT_BRANCH: ${{ github.ref_name }}
+  # Same central ES/APM sink as the other four e2e workflows. Bare-metal had
+  # no default here before — this cell's telemetry never reached the sink at
+  # all (not just missing ci.cell), see bm-bootstrap.sh's telemetry.env step.
+  SPINIFEX_OTLP_ENDPOINT: ${{ vars.SPINIFEX_OTLP_ENDPOINT || 'http://192.168.1.13:8200' }}
 
 jobs:
   e2e_baremetal:
@@ -28,6 +37,9 @@ jobs:
     env:
       BM_ISO_CACHE: /var/cache/spinifex-iso-builder
       ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-20-${{ github.run_id }}
+      # Caller-supplied (workflow_call input); empty for a manual
+      # workflow_dispatch run, which bm-bootstrap.sh treats as "no attrs".
+      SPINIFEX_OTEL_ATTRS: ${{ inputs.otel-attrs }}
     steps:
       - name: Clean Workspace
         run: |

.github/workflows/e2e-ddil.yml

@@ -13,6 +13,9 @@ on:
 
 permissions:
   contents: read
+  # actions: read lets the otel-job-id action look up this job's numeric id via
+  # the Actions API — github.job is only the YAML key, no context exposes it.
+  actions: read
 
 env:
   GIT_BRANCH: ${{ github.ref_name }}
@@ -28,6 +31,10 @@ jobs:
     timeout-minutes: 60
     env:
       TOFU_LOG: /tmp/e2e-tofu-ddil-${{ github.run_id }}.log
+      # Job-level env wins over the workflow-level default. ci.cell=ddil marks
+      # this workflow's telemetry so a DDIL run's partition-induced errors are
+      # never mistaken for a healthy multi-node run's in the sink.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=ddil
     steps:
       - name: Clean Workspace
         run: sudo rm -rf "$GITHUB_WORKSPACE"/*
@@ -72,6 +79,13 @@ jobs:
           echo "${BIN_DIR}" >> "${GITHUB_PATH}"
           "${BIN_DIR}/gotestsum" --version
 
+      # Adds ci.job_id alongside ci.cell, before Bootstrap bakes the attrs into
+      # the VMs, so a telemetry doc points back at this job's log directly.
+      - name: Resolve CI job id
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: DDIL Real Multi-Node
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |

.github/workflows/e2e-nightly.yml

@@ -11,6 +11,10 @@ on:
 
 permissions:
   contents: read
+  # actions: read lets the "Resolve CI job id" step below look up this job's
+  # numeric id via the Actions API — github.job is only the YAML key, no
+  # context exposes the id itself.
+  actions: read
 
 # Scheduled runs are loaded from the default branch (main), so the schedule
 # trigger must stay in sync on main for cron to fire.
@@ -46,6 +50,11 @@ jobs:
     env:
       TOFU_LOG: /tmp/e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}.log
       ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}
+      # Job-level env wins over the workflow-level default above, so telemetry
+      # from this cell carries ci.cell/ci.matrix. Nine cells run under one
+      # run id and every cell names its hosts node1/node2/node3, so without
+      # this every cell's telemetry collapses into one unattributable bucket.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=${{ matrix.cell }},ci.matrix=${{ matrix.topo }}/${{ matrix.net }}/${{ matrix.host }}
 
     steps:
       # blank or scheduled run = full matrix.
@@ -92,6 +101,16 @@ jobs:
         with:
           path: spinifex
 
+      # Nine cells share one run id and every cell names its hosts
+      # node1/node2/node3, so ci.cell above separates them and ci.job_id here
+      # links each cell's telemetry to the job log that produced it. Runs
+      # before Bootstrap bakes SPINIFEX_OTEL_ATTRS into the VM's telemetry.env.
+      - name: Resolve CI job id
+        if: steps.gate.outputs.selected == 'true'
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: "cell-${{ matrix.cell }} (${{ matrix.topo }}/${{ matrix.net }}/${{ matrix.host }}${{ matrix.extra && format('/{0}', matrix.extra) || '' }})"
+
       - name: Build Install Artifacts
         if: steps.gate.outputs.selected == 'true'
         working-directory: mulga/scripts/tofu-cluster
@@ -147,13 +166,20 @@ jobs:
         if: steps.gate.outputs.selected == 'true' && matrix.topo == 'single' && matrix.extra == ''
         working-directory: mulga/scripts/tofu-cluster
         run: |
+          set -euo pipefail
+          # Nightly runs a documented subset of the single-node set; the assert
+          # fails the cell if that subset ever names a suite this topology
+          # cannot fully exercise.
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          e2e_suites_assert_subset "e2e-nightly single" "${E2E_SUITES_NIGHTLY_SINGLE}" "${E2E_SUITES_SINGLE}"
+
           INVENTORY=$(tofu output -json inventory)
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
           ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
             "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
-             for SUITE in single cert lb; do \
+             for SUITE in ${E2E_SUITES_NIGHTLY_SINGLE}; do \
                echo \"::group::suite \$SUITE\"; \
                date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
                GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
@@ -190,6 +216,11 @@ jobs:
         if: steps.gate.outputs.selected == 'true' && matrix.topo == 'real-multi'
         working-directory: mulga/scripts/tofu-cluster
         run: |
+          set -euo pipefail
+          # See the single-node step: same subset guard, multi-node set.
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          e2e_suites_assert_subset "e2e-nightly multi" "${E2E_SUITES_NIGHTLY_MULTI}" "${E2E_SUITES_MULTI}"
+
           INVENTORY=$(tofu output -json inventory)
           NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
@@ -204,7 +235,7 @@ jobs:
 
           ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
             "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
-             for SUITE in multinode cert lb; do \
+             for SUITE in ${E2E_SUITES_NIGHTLY_MULTI}; do \
                echo \"::group::suite \$SUITE\"; \
                date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
                GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
@@ -373,4 +404,8 @@ jobs:
     uses: ./.github/workflows/e2e-baremetal.yml
     with:
       ref: ${{ github.event_name == 'schedule' && 'dev' || github.ref_name }}
+      # A reusable-workflow call gets none of this workflow's context (env:
+      # above, matrix.cell) for free — cell-20 needs its own attrs string.
+      # There's no matrix leg here, just the one fixed baremetal cell.
+      otel-attrs: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=20,ci.matrix=baremetal/pxe/real-hw
     secrets: inherit

.github/workflows/e2e-suites.yml

@@ -3,11 +3,14 @@
 # Restructures the looped e2e.yml into env-grouped job blocks so each suite
 # surfaces as its own node in the Actions graph (mulga-siv-283):
 #
-#   single_provision ─┬─ single_suite (matrix: single, cert, lb, eks, iam)
+#   single_provision ─┬─ single_suite (matrix: the single-node-eligible suites)
 #                     └─ single_teardown (if: always)
-#   multi_provision  ─┬─ multi_suite  (matrix: multinode)
+#   multi_provision  ─┬─ multi_suite  (matrix: the multi-node-eligible suites)
 #                     └─ multi_teardown (if: always)
 #
+# Both matrices come from .github/scripts/suite-sets.sh, the one definition of
+# which suites a topology can run, shared with e2e.yml and e2e-nightly.yml.
+#
 # Constraint: tofu state is runner-filesystem-local, so *_provision and
 # *_teardown must execute on the same self-hosted runner (one runner per label
 # today). Teardown reuses the persisted mulga/ tofu-cluster workspace and does
@@ -25,9 +28,13 @@ on:
   workflow_dispatch:
     inputs:
       single_suites:
-        description: "Single-node suites to run as matrix legs"
+        description: "Single-node suites to run as matrix legs (empty = every single-node-eligible suite, per .github/scripts/suite-sets.sh)"
         required: false
-        default: "single cert lb eks iam"
+        default: ""
+      multi_suites:
+        description: "Multi-node suites to run as matrix legs (empty = every multi-node-eligible suite, per .github/scripts/suite-sets.sh)"
+        required: false
+        default: ""
       run_multinode:
         description: "Also run the multinode env block"
         type: boolean
@@ -40,6 +47,10 @@ on:
 
 permissions:
   contents: read
+  # actions: read lets the otel-job-id action look up a provision job's numeric
+  # id via the Actions API — github.job is only the YAML key, no context
+  # exposes it.
+  actions: read
 
 env:
   GIT_BRANCH: ${{ github.ref_name }}
@@ -64,6 +75,13 @@ jobs:
       matrix: ${{ steps.suites.outputs.matrix }}
     env:
       TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
+      # This job (not single_suite) is where "Bootstrap via Binary Install"
+      # bakes SPINIFEX_OTEL_ATTRS into the VM's telemetry.env, before the
+      # suite matrix even exists — single_suite's legs share this one
+      # already-booted VM, so daemon telemetry can't be scoped any finer
+      # than "single env" per suite leg. ci.cell=single distinguishes it
+      # from multi_provision's VMs within the same run id.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=single
     steps:
       - name: Reap Orphan Procs
         run: |
@@ -107,7 +125,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -115,12 +133,22 @@ jobs:
       - name: Re-enable Go toolchain auto-download
         run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
 
+      # Default and allow-list both come from the shared definition, so this
+      # workflow cannot drift from e2e.yml about what a single node can run.
       - name: Compute suite matrix
         id: suites
+        env:
+          REQUESTED: ${{ inputs.single_suites }}
         run: |
-          LIST="${{ inputs.single_suites || 'single cert lb eks iam' }}"
+          set -euo pipefail
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          LIST="${REQUESTED:-${E2E_SUITES_SINGLE}}"
+          e2e_suites_assert_subset "single_suites" "${LIST}" "${E2E_SUITES_SINGLE}"
           echo "list=${LIST}" >> "$GITHUB_OUTPUT"
-          JSON=$(printf '%s\n' $LIST | jq -R . | jq -cs .)
+          # Unquoted on purpose: the space-separated list must word-split into
+          # one JSON element per suite.
+          # shellcheck disable=SC2086
+          JSON=$(printf '%s\n' ${LIST} | jq -R . | jq -cs .)
           echo "matrix=${JSON}" >> "$GITHUB_OUTPUT"
 
       - name: Reclaim docker build cache
@@ -130,6 +158,14 @@ jobs:
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
+      # ci.job_id names the provision job, not the suite job that later drives
+      # the env: the suite legs share this env and its baked telemetry.env, so
+      # the provisioner is the only job the daemon telemetry can belong to.
+      - name: Resolve CI job id
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: Single env · provision
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -294,7 +330,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -497,8 +533,13 @@ jobs:
       node1_ip: ${{ steps.cluster.outputs.node1_ip }}
       ssh_key: ${{ steps.cluster.outputs.ssh_key }}
       nodes_csv: ${{ steps.cluster.outputs.nodes_csv }}
+      suites: ${{ steps.suites.outputs.list }}
+      matrix: ${{ steps.suites.outputs.matrix }}
     env:
       TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
+      # See single_provision above — daemon telemetry is baked in here, at
+      # provision time, before the multi_suite matrix exists.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=multi
     steps:
       - name: Reap Orphan Procs
         run: |
@@ -539,7 +580,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -554,6 +595,12 @@ jobs:
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
+      # See the matching note on single_provision.
+      - name: Resolve CI job id
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: Multi env · provision
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -601,14 +648,36 @@ jobs:
           echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
           echo "nodes_csv=${NODES_CSV}" >> "$GITHUB_OUTPUT"
 
-      - name: Build multinode binary on runner
+      # Mirrors single_provision's step: the multi leg previously ran multinode
+      # alone, silently dropping the lb and cert coverage that only a
+      # multi-node env can complete.
+      - name: Compute suite matrix
+        id: suites
+        env:
+          REQUESTED: ${{ inputs.multi_suites }}
+        run: |
+          set -euo pipefail
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          LIST="${REQUESTED:-${E2E_SUITES_MULTI}}"
+          e2e_suites_assert_subset "multi_suites" "${LIST}" "${E2E_SUITES_MULTI}"
+          echo "list=${LIST}" >> "$GITHUB_OUTPUT"
+          # Unquoted on purpose: the space-separated list must word-split into
+          # one JSON element per suite.
+          # shellcheck disable=SC2086
+          JSON=$(printf '%s\n' ${LIST} | jq -R . | jq -cs .)
+          echo "matrix=${JSON}" >> "$GITHUB_OUTPUT"
+
+      - name: Build multinode binaries on runner
         working-directory: spinifex/tests/e2e
         run: |
           set -euo pipefail
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          banner "build multinode, teardown"
+          # teardown always builds: the run-scoped sweep binary runs in the
+          # teardown job, it is never a selectable suite leg.
+          SUITES_BUILD="${{ steps.suites.outputs.list }} teardown"
+          banner "build ${SUITES_BUILD// /, }"
           group_open "go test -c"
-          make e2e-build SUITES_BUILD="multinode teardown"
+          make e2e-build SUITES_BUILD="${SUITES_BUILD}"
           group_close
           ok "$(basename _bin/multinode.test) $(stat -c '%s' _bin/multinode.test | numfmt --to=iec)"
 
@@ -652,7 +721,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        suite: [multinode]
+        suite: ${{ fromJSON(needs.multi_provision.outputs.matrix) }}
     env:
       NODE1_IP: ${{ needs.multi_provision.outputs.node1_ip }}
       SSH_KEY: ${{ needs.multi_provision.outputs.ssh_key }}
@@ -663,7 +732,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false

.github/workflows/e2e.yml

@@ -26,10 +26,13 @@ on:
 permissions:
   contents: read
   statuses: write
+  # actions: read lets the otel-job-id action look up a job's numeric id via
+  # the Actions API — github.job is only the YAML key, no context exposes it.
+  actions: read
 
 env:
   GIT_BRANCH: ${{ github.ref_name }}
-  SUITES: ${{ inputs.suites || 'e2e-single e2e-multinode e2e-cert e2e-lb e2e-iam' }}
+  SUITES: ${{ inputs.suites || 'e2e-single e2e-multinode e2e-cert e2e-lb e2e-iam e2e-partialblock' }}
   TEST_RUN: ${{ inputs.test_run || '' }}
   # OTLP export to the central Elastic sink (wattle). Repo variable overrides;
   # empty string disables telemetry entirely.
@@ -37,54 +40,28 @@ env:
   SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }}
 
 jobs:
-  # Shadow-mode suite selector. Computes which suites the diff *would* run and logs it to the job summary.
-  # Does NOT gate or override the real SUITES env yet — it runs alongside the full matrix so we
-  # can compare predicted vs actual before flipping it on.
-  select_suites:
-    name: Select Suites (shadow)
-    runs-on: ubuntu-26.04
-    timeout-minutes: 5
-    steps:
-      - uses: actions/checkout@v7
-        with:
-          fetch-depth: 0
-      - uses: actions/setup-go@v6
-        with:
-          go-version-file: go.mod
-          cache: false
-      - name: Compute targeted suite selection
-        run: |
-          set -euo pipefail
-          BASE="${{ github.event.before }}"
-          HEAD="${{ github.sha }}"
-          # First push to a branch reports an all-zero before-SHA; fall back to
-          # the first parent so the diff is still meaningful.
-          if [ -z "$BASE" ] || [ "$BASE" = "0000000000000000000000000000000000000000" ]; then
-            BASE="${HEAD}~1"
-          fi
-          echo "## Targeted suite selection (shadow)" >> "$GITHUB_STEP_SUMMARY"
-          echo '```' >> "$GITHUB_STEP_SUMMARY"
-          echo "base=$BASE head=$HEAD" >> "$GITHUB_STEP_SUMMARY"
-          go run ./tests/e2e/select-suites/cmd/select-suites \
-            --manifest docs/service-interfaces.yaml \
-            --base-sha "$BASE" --head-sha "$HEAD" --repo-root . \
-            2>>"$GITHUB_STEP_SUMMARY" \
-            | tee -a "$GITHUB_STEP_SUMMARY"
-          echo '```' >> "$GITHUB_STEP_SUMMARY"
-          echo "Shadow only — actual run uses SUITES='${SUITES}'." >> "$GITHUB_STEP_SUMMARY"
-
   e2e_single:
     name: Single-Node E2E
-    # Skip when the dispatch input requests no in-VM suites (e.g.
-    # suites='e2e-multinode' only). Push triggers and the default SUITES list
-    # both include single/cert/lb so the inputs.suites null-check keeps the
-    # job running on plain PR pushes.
-    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-lb') || contains(github.event.inputs.suites, 'e2e-eks') || contains(github.event.inputs.suites, 'e2e-iam')
+    # The dispatch check gates on exactly the suites this job's Run step
+    # allow-lists, i.e. E2E_SUITES_SINGLE in .github/scripts/suite-sets.sh —
+    # an `if:` expression cannot source that file, so this one list is
+    # hand-mirrored and must be updated with it. Gating on a suite the job
+    # won't run — lb, which only executes in the multinode job — would
+    # provision a tofu VM and pay the Spinifex install only to run nothing and
+    # tear down. Push triggers keep running via the null-check on
+    # inputs.suites.
+    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-iam') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-eks') || contains(github.event.inputs.suites, 'e2e-ecs') || contains(github.event.inputs.suites, 'e2e-storagegrowth') || contains(github.event.inputs.suites, 'e2e-partialblock')
     runs-on: [self-hosted, ci-single]
     timeout-minutes: 50
     env:
       TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
       ARTIFACT_DIR: /tmp/spinifex-e2e-artifacts-${{ github.run_id }}
+      # Job-level env wins over the workflow-level default. This job and
+      # e2e_multi share one run id and both name their hosts node1/node2/node3,
+      # so without ci.cell their telemetry collapses into one bucket. This job
+      # is where "Bootstrap via Binary Install" bakes SPINIFEX_OTEL_ATTRS into
+      # the VM's telemetry.env, so the label has to be set here.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=single
     steps:
       - name: Reap Orphan Procs
         run: |
@@ -134,7 +111,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -149,6 +126,13 @@ jobs:
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
+      # Adds ci.job_id alongside ci.cell, before Bootstrap bakes the attrs into
+      # the VM, so a telemetry doc points back at this job's log directly.
+      - name: Resolve CI job id
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: Single-Node E2E
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -312,14 +296,23 @@ jobs:
         id: e2e
         run: |
           set +e
-          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(multinode)$' | tr '\n' ' ')
+          # Allow-list, not a deny-list: a suite runs here only if named. Deny-lists
+          # silently fan a newly added suite out to every job, which is how lb ended up
+          # running twice per e2e run. The allow-list itself is shared with the other
+          # e2e workflows so the three cannot disagree about what a topology can run;
+          # suite-sets.sh also carries the per-suite rationale.
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -xE "${E2E_SUITES_SINGLE_RE}" | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
             echo "no in-VM suites requested (SUITES=${SUITES}); skipping"
             echo "rc=0" >> "$GITHUB_OUTPUT"
             exit 0
           fi
           if [ -n "${TEST_RUN}" ]; then
-            TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+            # The value is expanded into a command that a remote shell re-parses over
+            # ssh, so single-quote it — otherwise regex metacharacters like ( | ) $ are
+            # taken as shell syntax on the VM and bash fails before the test binary runs.
+            TEST_RUN_FLAG="-test.run='${TEST_RUN}'"
           else
             TEST_RUN_FLAG=""
           fi
@@ -486,12 +479,19 @@ jobs:
 
   e2e_multi:
     name: Multi-Node E2E
-    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-multinode')
+    # The dispatch check gates on exactly the suites this job's Run step
+    # allow-lists (multinode|lb|cert) so lb — which only runs here — still
+    # provisions the 3-node env when requested on its own, and the env stays
+    # unprovisioned when none of them are asked for.
+    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-multinode') || contains(github.event.inputs.suites, 'e2e-lb') || contains(github.event.inputs.suites, 'e2e-cert')
     runs-on: [self-hosted, ci-multi]
     timeout-minutes: 60
     env:
       TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
       ARTIFACT_DIR: /tmp/spinifex-e2e-multi-artifacts-${{ github.run_id }}
+      # See the matching note on e2e_single: ci.cell is what separates this
+      # job's 3-node telemetry from the single-node job's under one run id.
+      SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=multi
     steps:
       - name: Reap Orphan Procs
         run: |
@@ -541,7 +541,7 @@ jobs:
         with:
           path: spinifex
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false
@@ -556,6 +556,12 @@ jobs:
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
+      # See the matching note on e2e_single.
+      - name: Resolve CI job id
+        uses: ./spinifex/.github/actions/otel-job-id
+        with:
+          job-name: Multi-Node E2E
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -697,14 +703,18 @@ jobs:
         id: e2e
         run: |
           set +e
-          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(single|eks|iam)$' | tr '\n' ' ')
+          # Allow-list, not a deny-list — see the single-node job for why.
+          . "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
+          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -xE "${E2E_SUITES_MULTI_RE}" | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
             echo "no multinode-eligible suites requested (SUITES=${SUITES}); skipping"
             echo "rc=0" >> "$GITHUB_OUTPUT"
             exit 0
           fi
           if [ -n "${TEST_RUN}" ]; then
-            TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+            # Single-quote for the remote shell — see the single-node job for the
+            # metacharacter-injection reason.
+            TEST_RUN_FLAG="-test.run='${TEST_RUN}'"
           else
             TEST_RUN_FLAG=""
           fi
@@ -935,3 +945,136 @@ jobs:
             -f context=e2e \
             -f description="$DESC" \
             -f target_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
+
+  # Emits one summary doc per run to the central Elastic sink, keyed by
+  # labels.ci_run_id, so Kibana can list "did run X pass, and where are its
+  # artifacts" without hand-aggregating traces-apm*/logs-apm.error*. This is
+  # a separate self-hosted job (not a step on e2e_status) because e2e_status
+  # runs on GitHub-hosted ubuntu-26.04, which has no route to the private
+  # 192.168.1.13 sink, and moving e2e_status off GH-hosted would risk the PR
+  # status-report check.
+  e2e_es_summary:
+    name: ES CI Run Summary
+    needs: [e2e_single, e2e_multi]
+    if: always()
+    runs-on: [self-hosted, ci-single]
+    timeout-minutes: 5
+    permissions:
+      actions: read
+    env:
+      # Repo var overrides; defaults to the LAN Elastic sink reachable from
+      # ci-single/ci-multi runners. Anonymous ES requests are superuser on
+      # this host, so no credentials are needed for the POST.
+      MULGA_ES_SINK: ${{ vars.MULGA_ES_SINK || '192.168.1.13:9200' }}
+    steps:
+      # Either leg may not have run (e.g. a workflow_dispatch that only
+      # requested single-node suites), so a missing artifact is tolerated —
+      # the aggregation script below treats an absent suite-status.txt as
+      # "no failures from that leg".
+      - name: Download single-node artifacts
+        uses: actions/download-artifact@v8
+        continue-on-error: true
+        with:
+          name: e2e-artifacts-${{ github.run_id }}
+          path: ${{ github.workspace }}/es-summary-single
+
+      - name: Download multi-node artifacts
+        uses: actions/download-artifact@v8
+        continue-on-error: true
+        with:
+          name: e2e-multi-artifacts-${{ github.run_id }}
+          path: ${{ github.workspace }}/es-summary-multi
+
+      - name: Build and post CI run summary
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+          RUN_ID: ${{ github.run_id }}
+          RUN_ATTEMPT: ${{ github.run_attempt }}
+          WORKFLOW: ${{ github.workflow }}
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+          SHA: ${{ github.sha }}
+          SERVER_URL: ${{ github.server_url }}
+          SINGLE_RESULT: ${{ needs.e2e_single.result }}
+          MULTI_RESULT: ${{ needs.e2e_multi.result }}
+          SINGLE_DIR: ${{ github.workspace }}/es-summary-single
+          MULTI_DIR: ${{ github.workspace }}/es-summary-multi
+        run: |
+          set -euo pipefail
+
+          SINGLE_STATUS="${SINGLE_DIR}/suite-status.txt"
+          MULTI_STATUS="${MULTI_DIR}/suite-status.txt"
+
+          # Collect "<suite>: FAILED" lines from both legs' suite-status.txt
+          # (either may be absent — a leg that didn't run, or a passing leg
+          # that never wrote the file). Accumulate as newline-separated text
+          # rather than a bash array so this stays correct under `set -u`
+          # even when nothing failed.
+          FAILED_SUITES=""
+          for f in "$SINGLE_STATUS" "$MULTI_STATUS"; do
+            [ -f "$f" ] || continue
+            while IFS= read -r line; do
+              case "$line" in
+                *": FAILED") FAILED_SUITES="${FAILED_SUITES}${line%: FAILED}"$'\n' ;;
+              esac
+            done < "$f"
+          done
+
+          FAILED_JSON=$(printf '%s' "$FAILED_SUITES" | jq -R -s 'split("\n") | map(select(length > 0))')
+          FAILED_COUNT=$(printf '%s' "$FAILED_JSON" | jq 'length')
+
+          # Mirror e2e_status's single_ok/multi_ok logic exactly so the PR
+          # status and this ES doc can never disagree on overall pass/fail.
+          single_ok=true
+          case "$SINGLE_RESULT" in success | skipped) ;; *) single_ok=false ;; esac
+          multi_ok=true
+          case "$MULTI_RESULT" in success | skipped) ;; *) multi_ok=false ;; esac
+
+          STATE=success
+          if [ "$single_ok" != true ] || [ "$multi_ok" != true ] || [ "$FAILED_COUNT" -gt 0 ]; then
+            STATE=failure
+          fi
+
+          RUN_STARTED_AT=$(gh api "repos/${REPO}/actions/runs/${RUN_ID}" --jq .run_started_at)
+          START_EPOCH=$(date -u -d "${RUN_STARTED_AT}" +%s)
+          NOW_EPOCH=$(date -u +%s)
+          DURATION=$((NOW_EPOCH - START_EPOCH))
+
+          DOC=$(jq -n \
+            --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
+            --arg run_id "${RUN_ID}" \
+            --arg workflow "${WORKFLOW}" \
+            --arg branch "${BRANCH}" \
+            --arg sha "${SHA}" \
+            --arg run_attempt "${RUN_ATTEMPT}" \
+            --arg state "${STATE}" \
+            --arg single_result "${SINGLE_RESULT}" \
+            --arg multi_result "${MULTI_RESULT}" \
+            --argjson duration "${DURATION}" \
+            --arg artifact_url "${SERVER_URL}/${REPO}/actions/runs/${RUN_ID}" \
+            --argjson failed_suites "${FAILED_JSON}" \
+            '{
+              "@timestamp": $ts,
+              labels: { ci_run_id: $run_id },
+              ci: { workflow: $workflow, branch: $branch, sha: $sha, run_attempt: $run_attempt },
+              state: $state,
+              single_result: $single_result,
+              multi_result: $multi_result,
+              failed_suites: $failed_suites,
+              duration_seconds: $duration,
+              artifact_url: $artifact_url
+            }')
+
+          echo "$DOC" | jq .
+
+          HTTP_CODE=$(curl -sS -o /tmp/es-summary-resp.json -w '%{http_code}' \
+            -X POST "http://${MULGA_ES_SINK}/mulga-ci-runs/_doc" \
+            -H 'Content-Type: application/json' \
+            -d "${DOC}")
+
+          if [ "$HTTP_CODE" != "200" ] && [ "$HTTP_CODE" != "201" ]; then
+            echo "::error::ES POST to ${MULGA_ES_SINK}/mulga-ci-runs failed (HTTP ${HTTP_CODE})"
+            cat /tmp/es-summary-resp.json
+            exit 1
+          fi
+          echo "CI run summary doc posted to ${MULGA_ES_SINK}/mulga-ci-runs (HTTP ${HTTP_CODE})"

.github/workflows/gpu-e2e.yml

@@ -44,32 +44,6 @@ on:
         description: "Optional -test.run regex scoped to tests/e2e/gpu (empty = run all GPU tests)"
         required: false
         default: ""
-  push:
-    branches:
-      - main
-      - dev
-    paths: &gpu_paths
-      - "spinifex/gpu/**"
-      - "spinifex/instancetypes/**"
-      - "spinifex/daemon/gpu_adapter.go"
-      - "spinifex/daemon/gpu_pool_reconciler.go"
-      - "spinifex/handlers/ecs/gpu_report.go"
-      - "spinifex/handlers/eks/nodegroup.go"
-      - "spinifex/handlers/eks/nodegroup_userdata.go"
-      - "spinifex/vm/vm.go"
-      - "spinifex/vm/lifecycle.go"
-      - "cmd/ecs-agent/gpu_discovery.go"
-      - "cmd/ecs-agent/gpu_ledger.go"
-      - "cmd/spinifex/cmd/admin_gpu.go"
-      - "tests/e2e/gpu/**"
-      - "scripts/images/eks-node-gpu/**"
-      - "scripts/images/ecs-agent-gpu/**"
-      - ".github/workflows/gpu-e2e.yml"
-  pull_request:
-    branches:
-      - main
-      - dev
-    paths: *gpu_paths
 
 permissions:
   contents: read
@@ -88,7 +62,11 @@ env:
   # the GPU suite's own daemon-side spans are emitted by wattle's already-
   # configured obs-agent, not by this job.
   SPINIFEX_OTLP_ENDPOINT: ${{ vars.SPINIFEX_OTLP_ENDPOINT || 'http://192.168.1.13:8200' }}
-  SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }}
+  # ci.cell=gpu keeps this workflow's telemetry separable from the tofu-env
+  # workflows'. Nothing in this job bakes these attrs into a daemon (see the
+  # note above), so the label only takes effect once wattle's obs-agent is
+  # given them; there is deliberately no ci.job_id lookup for the same reason.
+  SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=gpu
   WATTLE_SSH_HOST: ${{ vars.WATTLE_SSH_HOST || '192.168.1.13' }}
   WATTLE_SSH_USER: ${{ vars.WATTLE_SSH_USER || 'tf-user' }}
   TEST_RUN: ${{ inputs.test_run || '' }}
@@ -131,7 +109,7 @@ jobs:
       - uses: actions/checkout@v7
         if: steps.gate.outputs.paused != 'true'
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@v7
         if: steps.gate.outputs.paused != 'true'
         with:
           go-version-file: go.mod

.github/workflows/publish-eks-ami.yml

@@ -95,7 +95,7 @@ jobs:
           path: predastore
 
       - name: Setup Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           go-version-file: "spinifex/go.mod"
           # Self-hosted runner: the Go module + build cache already persists on

.github/workflows/release.yml

@@ -32,7 +32,7 @@ jobs:
         uses: docker/setup-buildx-action@v4
 
       - name: Set up Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           go-version: "1.26.5"
 
@@ -280,7 +280,7 @@ jobs:
         run: sudo make -C mulga/scripts/iso-builder clean
 
       - name: Set up Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           go-version-file: spinifex/go.mod
           cache: false

.github/workflows/spinifex.yml

@@ -32,22 +32,29 @@ jobs:
           path: spinifex
           fetch-depth: 2
 
+      - name: Resolve sibling refs
+        id: siblings
+        env:
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+        run: bash .github/resolve-sibling-refs.sh
+        working-directory: spinifex
+
       - name: Checkout Viperblock
         uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
-          ref: dev
+          ref: ${{ steps.siblings.outputs.viperblock }}
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
-          ref: dev
+          ref: ${{ steps.siblings.outputs.predastore }}
           path: predastore
 
       - name: Setup Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           cache-dependency-path: "spinifex/go.sum"
           go-version-file: "spinifex/go.mod"
@@ -70,6 +77,55 @@ jobs:
       - name: Check Diff Coverage
         run: cd spinifex && scripts/diff-coverage.sh coverage.out --base HEAD~1
 
+  # In-process integration tier (gateway.SetupRoutes over embedded NATS
+  # JetStream). Its own job so it runs in parallel with the unit tiers rather
+  # than adding to their wall time, and it still blocks a PR — unlike e2e.yml's
+  # "Integration (front gate)" job, which is push-triggered against main/dev on
+  # a self-hosted runner and so never gates a PR before merge.
+  integration_tests:
+    name: Integration Tests
+    runs-on: ubuntu-26.04
+
+    steps:
+      - name: Checkout Spinifex
+        uses: actions/checkout@v7
+        with:
+          path: spinifex
+          fetch-depth: 2
+
+      - name: Resolve sibling refs
+        id: siblings
+        env:
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+        run: bash .github/resolve-sibling-refs.sh
+        working-directory: spinifex
+
+      - name: Checkout Viperblock
+        uses: actions/checkout@v7
+        with:
+          repository: mulgadc/viperblock
+          ref: ${{ steps.siblings.outputs.viperblock }}
+          path: viperblock
+
+      - name: Checkout Predastore
+        uses: actions/checkout@v7
+        with:
+          repository: mulgadc/predastore
+          ref: ${{ steps.siblings.outputs.predastore }}
+          path: predastore
+
+      - name: Setup Go
+        uses: actions/setup-go@v7
+        with:
+          cache-dependency-path: "spinifex/go.sum"
+          go-version-file: "spinifex/go.mod"
+
+      - name: Setup Go Workspace
+        run: go work init ./spinifex ./viperblock ./predastore
+
+      - name: Run Integration Tests
+        run: make -C spinifex test-integration
+
   race_detection:
     name: Race Detection
     runs-on: ubuntu-26.04
@@ -80,22 +136,29 @@ jobs:
         with:
           path: spinifex
 
+      - name: Resolve sibling refs
+        id: siblings
+        env:
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+        run: bash .github/resolve-sibling-refs.sh
+        working-directory: spinifex
+
       - name: Checkout Viperblock
         uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
-          ref: dev
+          ref: ${{ steps.siblings.outputs.viperblock }}
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
-          ref: dev
+          ref: ${{ steps.siblings.outputs.predastore }}
           path: predastore
 
       - name: Setup Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           cache-dependency-path: "spinifex/go.sum"
           go-version-file: "spinifex/go.mod"
@@ -116,22 +179,29 @@ jobs:
         with:
           path: spinifex
 
+      - name: Resolve sibling refs
+        id: siblings
+        env:
+          BRANCH: ${{ github.head_ref || github.ref_name }}
+        run: bash .github/resolve-sibling-refs.sh
+        working-directory: spinifex
+
       - name: Checkout Viperblock
         uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
-          ref: dev
+          ref: ${{ steps.siblings.outputs.viperblock }}
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
-          ref: dev
+          ref: ${{ steps.siblings.outputs.predastore }}
           path: predastore
 
       - name: Setup Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v7
         with:
           cache-dependency-path: "spinifex/go.sum"
           go-version-file: "spinifex/go.mod"

.github/workflows/ui.yml

@@ -40,7 +40,7 @@ jobs:
           package_json_file: spinifex/services/spinifexui/frontend/package.json
 
       - name: Setup Node.js
-        uses: actions/setup-node@v6
+        uses: actions/setup-node@v7
         with:
           cache: "pnpm"
           cache-dependency-path: spinifex/services/spinifexui/frontend/pnpm-lock.yaml

.golangci.yml

@@ -1,5 +1,8 @@
 version: "2"
 
+issues:
+  max-same-issues: 10
+
 formatters:
   enable:
     - gofmt
@@ -14,24 +17,29 @@ linters:
     - bidichk # checks for dangerous unicode bidi character sequences
     - bodyclose # checks whether HTTP response body is closed successfully
     - canonicalheader # checks whether net/http.Header uses canonical header keys
-    - decorder # checks declaration order of types, consts, vars, and funcs
+    - depguard # blocks imports of packages with a preferred replacement
     - durationcheck # checks for two durations multiplied together
     - embeddedstructfieldcheck # checks embedded sync.Mutex/RWMutex in structs
     - errcheck # checks for unchecked errors in type assertions and I/O
     - errchkjson # checks types passed to json encoding functions, flags omittable error checks
     - errname # checks that sentinel errors are prefixed with Err, error types suffixed with Error
+    - errorlint # catches ==/!= on errors, type assertions on errors, and %v instead of %w — all break on wrapped errors
     - exptostd # detects golang.org/x/exp/ functions replaceable by stdlib
     - fatcontext # detects nested contexts in loops
-    - forcetypeassert # flags unchecked type assertions that can panic
+    - forbidigo # blocks the legacy JetStream/KV symbols
     - gocheckcompilerdirectives # validates go compiler directive comments (//go:)
+    - godot # checks that declaration comments end in a period
     - goprintffuncname # checks that printf-like functions are named with f at the end
     - gosec # inspects source code for security problems
     - govet # reports suspicious constructs (Printf alignment, shadow, etc.)
+    - iface # detects interface pollution — duplicate interface declarations
     - ineffassign # detects assignments to existing variables that are not used
+    - iotamixing # flags const blocks that mix iota with non-iota declarations
     - makezero # finds slice declarations with non-zero initial length followed by append
     - mirror # catches wrong mirror patterns in bytes/strings usage
-    - musttag # checks structs passed to marshal/unmarshal have proper tags
+    - misspell # finds commonly misspelled English words
     - modernize # modernizes code to use newer Go idioms (go fix analyzers)
+    - musttag # checks structs passed to marshal/unmarshal have proper tags
     - nakedret # finds naked returns in functions
     - nilerr # catches code that returns nil even after checking err != nil
     - nilnesserr # catches returning a different nil error variable after err != nil check
@@ -43,6 +51,8 @@ linters:
     - recvcheck # checks for receiver type consistency (pointer vs value)
     - sloglint # enforces consistent log/slog usage (key-value pairs, no globals)
     - staticcheck # extensive static analysis checks
+    - testifylint # checks common testify mistakes (assert.Len over assert.Equal on len, etc.)
+    - tparallel # detects inappropriate usage of t.Parallel()
     - unconvert # removes unnecessary type conversions
     - usestdlibvars # detects hardcoded values that should use stdlib constants
     - usetesting # catches os.TempDir() etc. in tests — should use t.TempDir()
@@ -50,14 +60,41 @@ linters:
     - whitespace # detects leading and trailing whitespace in functions
 
   settings:
+    depguard:
+      rules:
+        main:
+          deny:
+            - pkg: log$
+              desc: Use log/slog instead, see https://go.dev/blog/slog
+            - pkg: math/rand$
+              desc: Use math/rand/v2 instead, see https://go.dev/blog/randv2
+
     embeddedstructfieldcheck:
       forbid-mutex: true
 
     errcheck:
-      check-type-assertions: true
+      check-type-assertions: true # replaces forcetypeassert
       exclude-functions:
         - (*github.com/nats-io/nats.go.Subscription).Unsubscribe # cleanup-only, errors not actionable
 
+    forbidigo:
+      analyze-types: true
+      forbid:
+        - pattern: '\.JetStream$'
+          pkg: '^github\.com/nats-io/nats\.go$'
+          msg: legacy nc.JetStream() is superseded; construct with jetstream.New(nc).
+        - pattern: '^nats\.(JetStreamContext|KeyValue.*|KeyWatcher|Err(KeyNotFound|KeyExists|BucketNotFound|NoKeysFound|StreamNotFound))$'
+          pkg: '^github\.com/nats-io/nats\.go$'
+          msg: legacy nats KV types/sentinels are superseded; use github.com/nats-io/nats.go/jetstream.
+
+    gosec:
+      excludes:
+        - G104 # unchecked errors — errcheck is stricter
+        - G116 # bidi chars — bidichk is more thorough
+        - G117 # use of net/http serve functions (we handle TLS separately)
+        - G703 # path traversal taint analysis — false positives on config file loading
+        - G705 # XSS taint analysis — false positives on internal error responses
+
     govet:
       enable-all: true
       disable:
@@ -65,6 +102,28 @@ linters:
         - shadow # too noisy to retrofit — all 50 findings are err re-declarations
         - unusedwrite # false positives on struct field assignments before marshal/send
 
+    modernize:
+      disable:
+        - newexpr # don't replace aws.String() with new() — AWS SDK idiom
+        - stringsbuilder # don't replace small string concatenations with strings.Builder
+
+    nakedret:
+      max-func-lines: 0
+
+    nolintlint:
+      require-explanation: true
+      require-specific: true
+
+    perfsprint:
+      concat-loop: false
+      err-error: false # don't flag err.Error()
+      errorf: false # don't convert fmt.Errorf to errors.New
+      strconcat: false
+
+    reassign:
+      patterns:
+        - ".*"
+
     staticcheck:
       checks:
         - all
@@ -76,40 +135,14 @@ linters:
         - -ST1020 # godoc on exported types — not enforcing yet
         - -ST1021 # godoc on exported type groups — not enforcing yet
         - -ST1022 # godoc on exported vars/consts — not enforcing yet
-        - -SA1019 # deprecated usage (handle separately)
         - -SA4006 # value overwritten — ineffassign + wastedassign cover this better
-        - -SA9005 # ineffective break (staticcheck false positives)
+        - -SA9005 # marshaling structs with no exported fields — intended {} for empty API-response types
 
-    gosec:
-      excludes:
-        - G104 # unchecked errors — errcheck is stricter
-        - G116 # bidi chars — bidichk is more thorough
-        - G117 # use of net/http serve functions (we handle TLS separately)
-        - G703 # path traversal taint analysis — false positives on config file loading
-        - G705 # XSS taint analysis — false positives on internal error responses
-        - G706 # log injection taint analysis — false positives on structured slog usage
-
-    modernize:
+    testifylint:
       disable:
-        - newexpr # don't replace aws.String() with new() — AWS SDK idiom
-        - stringsbuilder # don't replace small string concatenations with strings.Builder
-
-    perfsprint:
-      concat-loop: false
-      err-error: false # don't flag err.Error()
-      errorf: false # don't convert fmt.Errorf to errors.New
-      strconcat: false
-
-    nakedret:
-      max-func-lines: 0
-
-    nolintlint:
-      require-explanation: true
-      require-specific: true
-
-    reassign:
-      patterns:
-        - ".*"
+        - encoded-compare # misidentifies non-JSON strings when identifiers contain "JSON"
+        - float-compare # false positives
+        - require-error # assert-on-error is deliberate where a test continues to collect further failures
 
   exclusions:
     presets:
@@ -133,3 +166,8 @@ linters:
         linters: [gosec] # subprocess execution — legitimate in VM orchestration
       - text: "G304"
         linters: [gosec] # file path from variable — necessary for config loading
+
+      # aws-sdk-go v1 is deprecated wholesale (EOL 2025-07-31); the v1->v2
+      # migration is tracked separately, so suppress its SA1019 noise for now.
+      - text: "aws-sdk-go is deprecated"
+        linters: [staticcheck]

Makefile

@@ -120,8 +120,13 @@ install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
 
-# Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
-# Use this before committing to catch CI failures locally.
+# Preflight — the pre-commit gate: manifest checks, lint, vuln, and the unit,
+# race and e2e-harness tiers.
+#
+# The integration tier is deliberately NOT here. CI runs it as its own parallel
+# job, so it is still gated before merge, and keeping it out leaves preflight
+# the fast common-path check — run `make test-integration` directly when a
+# change touches the gateway router or the NATS subjects it drives.
 preflight:
 	@$(MAKE) --no-print-directory QUIET=1 manifest-check manifest-lint lint govulncheck test-cover diff-coverage test-race test-harness
 	@echo -e "\n ✅ Preflight passed — safe to commit."
@@ -133,6 +138,18 @@ test-harness:
 	@echo -e "\n....Running e2e harness unit tests...."
 	$(_Q)LOG_IGNORE=1 go test -tags=e2e -timeout 60s ./tests/e2e/harness/... $(_RACEQ)
 
+# In-process integration tier: the real gateway router against embedded NATS
+# JetStream, with only the daemon-side NATS subjects stubbed. Build-tagged
+# `integration` so it's skipped by the default `go test ./spinifex/...` and by
+# `test-cover`/`test-race`. Nothing provisioned — no tofu, no docker, no
+# Spinifex daemons, so the whole package runs in well under a minute. It gets
+# its own PR-blocking CI job rather than the self-hosted, push-triggered live
+# e2e tiers, so a regression here is caught before it can be merged, not just
+# after.
+test-integration:
+	@echo -e "\n....Running in-process integration tests...."
+	$(_Q)LOG_IGNORE=1 go test -tags=integration -timeout 60s ./tests/integration/... $(_RACEQ)
+
 # Validate docs/service-interfaces.yaml. Schema check + cross-reference
 # of services/suites/fixtures + on-disk path existence. Subject content
 # vs source is enforced separately in Bead 5 drift lint.
@@ -254,6 +271,13 @@ govulncheck:
 	$(_Q)go tool govulncheck ./...
 	@echo "  govulncheck ok"
 
+# NilAway — advisory nil-panic analysis. Not in preflight: it has a known
+# false-positive rate, so findings are triaged by hand rather than gating commits.
+nilaway:
+	@echo "Running nilaway..."
+	$(_Q)go tool nilaway -include-pkgs=github.com/mulgadc/spinifex -exclude-test-files ./...
+	@echo "  nilaway ok"
+
 # Build release tarballs — use distro-ARCH for single arch, distro for both
 distro: distro-amd64 distro-arm64
 	@echo ""
@@ -297,8 +321,8 @@ distro-arm64:
 distro-clean:
 	rm -rf dist/
 
-.PHONY: build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-harness manifest-check manifest-lint manifest-lint-update \
+.PHONY: build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-harness test-integration manifest-check manifest-lint manifest-lint-update \
 	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
-	lint fix govulncheck \
+	lint fix govulncheck nilaway \
 	distro distro-amd64 distro-arm64 distro-clean

README.md

@@ -1,5 +1,5 @@
 <p align="center">
-  <img src=".github/assets/banner.svg" alt="Spinifex — AWS-Compatible Infrastructure Stack" width="900">
+  <img src=".github/assets/banner.svg" alt="Spinifex: the open, AWS-compatible cloud you run yourself. EC2, EBS, S3, VPC, and IAM on your infrastructure, at the edge, or on a Neocloud." width="900">
 </p>
 
 <p align="center">
@@ -11,6 +11,8 @@
 <p align="center">
   <a href="#what-is-spinifex">What is Spinifex?</a> ·
   <a href="#the-platform">Platform</a> ·
+  <a href="#three-ways-to-deploy">Deploy</a> ·
+  <a href="#aws-compatibility">AWS services</a> ·
   <a href="#core-components">Components</a> ·
   <a href="#architecture-at-a-glance">Architecture</a> ·
   <a href="#installation">Installation</a> ·
@@ -19,31 +21,90 @@
 
 ---
 
-# Spinifex: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
+# Spinifex: the open, AWS-compatible cloud you run yourself
 
-Spinifex developed by [Mulga Defense Corporation](https://mulgadc.com) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
+Spinifex, built by [Mulga](https://mulgadc.com), recreates the AWS services your
+software already uses (EC2, EBS, S3, VPC, IAM) on infrastructure you control.
+Move workloads off the hyperscalers, run AI at a fraction of the cost, or take
+the cloud to the edge, without rewriting a line. The exact build you ship to AWS,
+Spinifex serves on your own hardware: same APIs, same Terraform, same SDKs.
 
 ## What is Spinifex?
 
-Spinifex replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
+Spinifex is an open-source infrastructure platform that recreates the AWS service
+surface (EC2, EBS, S3, VPC, IAM) on bare-metal, edge, and on-premises
+environments. Run cloud-native software without a hyperscaler.
 
-It's designed for developers and operators who need:
+Most AWS alternatives wrap an API and call it a cloud. We rebuilt the engineering
+underneath: distributed object storage with erasure coding, block storage built
+to survive failure, and compute on bare metal. That depth is the reason your
+software runs unchanged, instead of rewritten.
 
-- Cloud-like infrastructure without the cloud
-- Minimal dependencies, full control
-- Drop-in compatibility with tools like the AWS CLI, SDKs, and Terraform
-- A secure cloud environment you control and own the entire hardware, network and software stack
+It's built for teams that need:
 
-You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren't feasible.
+- The AWS tooling they already use (AWS CLI, SDKs, Terraform), with nothing to rewrite
+- Full control of the stack: your hardware, your network, your data, your keys
+- A cloud that keeps running offline, through disconnection, with no external control plane
+- An open core (AGPL-3.0), auditable and yours to keep, with no lock-in
+
+You change *where* your software runs, not *what* it is.
 
 ## The Platform
 
 <p align="center">
-  <img src=".github/assets/platform.svg" alt="The Mulga Platform — Hardware to Applications" width="900">
+  <img src=".github/assets/platform.svg" alt="The Spinifex stack: your apps and AWS tooling on top, the Spinifex AWS-compatible service layer, running on standard Linux on CPU and GPU hardware, deployable to Neocloud, on-premise, or the edge." width="900">
 </p>
 
 From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
 
+## Three ways to deploy
+
+One AWS-compatible surface, three deployment shapes. Pick the one that matches
+your reality; the platform on top is the same.
+
+### Neocloud
+
+Lift and shift onto a partner Neocloud. Move workloads off the hyperscalers and
+onto our Neocloud partner ecosystem without rewriting them. GPU capacity
+(H100 / H200 / B200), cheaper, available now. Change an endpoint, keep the
+software.
+
+### On-premise
+
+Bring your own hardware and host Spinifex in your own data centre. A real
+multi-node HA cluster integrated with your storage and networking, with full
+control of stack, data, and jurisdiction. A predictable bill, no egress
+surprises, an auditable open-source core.
+
+### Edge
+
+Cloud where the cloud can't reach. Air-gapped sites, vehicles, vessels,
+factories, and clinics. Compute next to the data, running through disconnection,
+on hardware you own. The same AWS APIs your software already uses.
+
+## AWS compatibility
+
+Speak the AWS API surface, natively. The AWS SDKs, AWS CLI, and Terraform:
+everything you deploy on AWS deploys on Spinifex unchanged. At the edge,
+on-premise, or on a partner Neocloud.
+
+| Service | What it is | Status |
+| --- | --- | --- |
+| **EC2** | Compute | Available |
+| **EBS** | Block storage | Available |
+| **S3** | Object storage | Available |
+| **VPC** | Networking | Available |
+| **IAM** | Identity & auth | Available |
+| **ALB / NLB** | Load balancers | Available |
+| **EKS** | Kubernetes | Available |
+| **ECR** | Container registry | Available |
+| **ECS** | Container service | Available |
+| **RDS** | Databases | Q3 2026 |
+
+Roadmap items (Q3 2026) ship under the same AWS API surface. Code written for AWS
+today keeps working the moment they land.
+[Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).
+
 ## Core Components
 
 ### Spinifex (Compute Service – EC2 Alternative)
@@ -75,19 +136,23 @@ Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs
 ## Architecture at a Glance
 
 <p align="center">
-  <img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture — AWS Gateway, NATS bus, service daemons" width="900">
+  <img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus answered by stateless ec2/ebs/s3/vpc/iam daemons over local backends." width="900">
 </p>
 
-Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless — scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
+Every AWS API call is authenticated at the gateway, published to a NATS subject,
+and answered by whichever daemon claims it. Daemons are stateless: scale
+horizontally by starting more. No etcd, no Kubernetes, no external control plane.
+Just systemd units, a NATS cluster, and your hardware. Deep dive:
+**[`docs/DESIGN.md`](docs/DESIGN.md)**.
 
 ## Key Features
 
-AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, or SDKs you already use.
-
-- Designed for Control – Run on your own terms, whether that's in a datacenter, remote site, or sensitive environment.
-- Minimal Dependencies – Each service is standalone and avoids complex orchestration layers.
-- Works Offline – No reliance on centralized cloud services or external networks.
-- Open Source – Licensed under AGPL 3.0. Fork it, extend it, or deploy it as-is.
+- **AWS-compatible APIs.** Use the AWS CLI, SDKs, and Terraform you already know. Repoint your endpoint and ship; nothing to rewrite.
+- **Zero cloud dependency.** Runs entirely on your hardware. No phone-home, no control plane, no external authority. Works fully offline.
+- **Bare-metal compute.** QEMU-based with direct hardware access. No hypervisor tax, no abstraction overhead.
+- **Built-in storage.** Block and object storage included: NVMe caching, Reed-Solomon erasure coding, and replication out of the box.
+- **Edge-first architecture.** Designed for disconnected, contested, and resource-constrained environments from day one.
+- **Open core, no lock-in.** AGPL-3.0 with a commercial option. Inspect it, modify it, deploy it. The platform is yours to keep.
 
 ## Installation
 

build/systemd/spinifex-northstar.service

@@ -0,0 +1,54 @@
+[Unit]
+Description=Spinifex Northstar (Authoritative DNS)
+PartOf=spinifex.target
+After=spinifex-nats.service spinifex-predastore.service
+Wants=spinifex-predastore.service
+
+[Service]
+EnvironmentFile=-/etc/spinifex/telemetry.env
+Type=simple
+Slice=spinifex-system.slice
+User=spinifex-northstar
+Group=spinifex
+ExecStart=/usr/local/bin/spx service northstar start
+Restart=on-failure
+RestartSec=5
+OOMScoreAdjust=-500
+Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
+Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/northstar/
+
+# Filesystem access
+NoNewPrivileges=yes
+ProtectSystem=strict
+PrivateDevices=yes
+ReadOnlyPaths=/etc/spinifex/northstar /etc/spinifex/spinifex.toml /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/northstar /var/log/spinifex
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+# RG-9: exactly CAP_NET_BIND_SERVICE so the unprivileged user binds :53 (no root).
+AmbientCapabilities=CAP_NET_BIND_SERVICE
+CapabilityBoundingSet=CAP_NET_BIND_SERVICE
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
+
+[Install]
+WantedBy=spinifex.target

build/systemd/spinifex.target

@@ -2,7 +2,7 @@
 Description=Spinifex Infrastructure Platform
 After=network-online.target
 Wants=network-online.target
-Wants=spinifex-nats.service spinifex-predastore.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-qmp-collector.service spinifex-shutdown.service spinifex-nats-watchdog.timer openvswitch-switch.service ovn-controller.service
+Wants=spinifex-nats.service spinifex-predastore.service spinifex-northstar.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-qmp-collector.service spinifex-shutdown.service spinifex-nats-watchdog.timer openvswitch-switch.service ovn-controller.service
 
 [Install]
 WantedBy=multi-user.target

cmd/dhcptest/main.go

@@ -6,7 +6,7 @@
 // Usage (must run as root — needs AF_PACKET):
 //
 //	sudo dhcptest --iface=br-wan
-//	sudo dhcptest --iface=br-wan --mac=02:00:00:7c:b7:b9 --promisc
+//	sudo dhcptest --iface=br-wan --mac=02:54:df:71:22:5c --promisc
 //	sudo dhcptest --iface=br-wan --spinifex-id=gateway-wan1
 package main
 
@@ -14,15 +14,15 @@ import (
 	"context"
 	"flag"
 	"fmt"
-	"log"
+	"log/slog"
 	"net"
 	"os"
 	"os/exec"
 	"strings"
 	"time"
 
-	"github.com/insomniacslk/dhcp/dhcpv4"
 	"github.com/insomniacslk/dhcp/dhcpv4/nclient4"
+	"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
 )
 
 func main() {
@@ -44,20 +44,25 @@ func main() {
 
 	switch {
 	case *spinifexID != "":
-		hwAddr, err = net.ParseMAC(generateMAC(*spinifexID))
+		// vpcd's own derivation, so the probe leases under the MAC the daemon
+		// would have used for this id.
+		hwAddr, err = dhcp.DeriveMAC(*spinifexID)
 		if err != nil {
-			log.Fatalf("generateMAC(%q): %v", *spinifexID, err)
+			slog.Error("Could not derive MAC", "spinifex-id", *spinifexID, "error", err)
+			os.Exit(1)
 		}
 		fmt.Printf("spinifex virtual MAC for %q: %s\n", *spinifexID, hwAddr)
 	case *macStr != "":
 		hwAddr, err = net.ParseMAC(*macStr)
 		if err != nil {
-			log.Fatalf("parse --mac %q: %v", *macStr, err)
+			slog.Error("Could not parse --mac", "mac", *macStr, "error", err)
+			os.Exit(1)
 		}
 	default:
 		hwAddr, err = ifaceMAC(*iface)
 		if err != nil {
-			log.Fatalf("get MAC for %s: %v", *iface, err)
+			slog.Error("Could not get MAC for interface", "iface", *iface, "error", err)
+			os.Exit(1)
 		}
 		fmt.Printf("using real interface MAC: %s\n", hwAddr)
 	}
@@ -83,7 +88,7 @@ func main() {
 
 	if *promisc {
 		if err := setPromisc(*iface, true); err != nil {
-			log.Printf("WARNING: failed to set promisc on %s: %v", *iface, err)
+			slog.Warn("Failed to set promisc on interface", "iface", *iface, "error", err)
 		} else {
 			fmt.Printf("set %s promisc ON\n\n", *iface)
 		}
@@ -96,26 +101,19 @@ func main() {
 		nclient4.WithRetry(*retries),
 	}
 	if *verbose {
-		opts = append(opts, nclient4.WithLogger(nclient4.ShortSummaryLogger{Printfer: log.New(os.Stderr, "[nclient4] ", 0)}))
+		opts = append(opts, nclient4.WithLogger(nclient4.ShortSummaryLogger{Printfer: slogPrintfer{}}))
 	}
 
 	client, err := nclient4.New(*iface, opts...)
 	if err != nil {
-		log.Fatalf("nclient4.New(%s): %v", *iface, err)
+		slog.Error("Could not create nclient4 client", "iface", *iface, "error", err)
+		os.Exit(1)
 	}
 	defer client.Close()
 
-	// Identity modifiers — client-id / hostname / vendor-class for upstream lease tagging.
-	var mods []dhcpv4.Modifier
-	if cid != "" {
-		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClientIdentifier([]byte(cid))))
-	}
-	if *hostname != "" {
-		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptHostName(*hostname)))
-	}
-	if *vendorClass != "" {
-		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClassIdentifier(*vendorClass)))
-	}
+	// Identity modifiers — client-id / hostname / vendor-class for upstream lease
+	// tagging. Built by the vpcd code path so the probe cannot drift from it.
+	mods := dhcp.IdentityModifiers(cid, *hostname, *vendorClass, hwAddr)
 
 	fmt.Println("Sending DHCP Discover...")
 	start := time.Now()
@@ -141,14 +139,12 @@ func main() {
 	}
 }
 
-// generateMAC is the same deterministic hash used by vpcd's topology.go.
-// It produces a locally-administered MAC (02:00:00:xx:xx:xx) from a resource ID.
-func generateMAC(resourceID string) string {
-	h := uint32(0)
-	for _, c := range resourceID {
-		h = h*31 + uint32(c)
-	}
-	return fmt.Sprintf("02:00:00:%02x:%02x:%02x", (h>>16)&0xff, (h>>8)&0xff, h&0xff)
+// slogPrintfer adapts nclient4's Printfer to slog. nclient4 only emits through
+// it under --verbose, so the packet summaries log at info level to stay visible.
+type slogPrintfer struct{}
+
+func (slogPrintfer) Printf(format string, v ...any) {
+	slog.Info(fmt.Sprintf(format, v...), "component", "nclient4")
 }
 
 func ifaceMAC(name string) (net.HardwareAddr, error) {

cmd/ecs-agent/credendpoint.go

@@ -3,6 +3,7 @@ package main
 import (
 	"context"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"log/slog"
 	"net"
@@ -143,7 +144,7 @@ func (c *credEndpoint) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	}
 	creds, roleARN, err := c.fetch(r.Context(), credID)
 	if err != nil {
-		if err == errUnknownCredID {
+		if errors.Is(err, errUnknownCredID) {
 			http.Error(w, "not found", http.StatusNotFound)
 			return
 		}

cmd/installer/install/install.go

@@ -539,7 +539,7 @@ func installBootloader(disk string) error {
 	); biosErr != nil {
 		if efiErr != nil {
 			// Both targets failed — the system will not boot.
-			return fmt.Errorf("both bootloader targets failed (EFI: %v; BIOS: %w)", efiErr, biosErr)
+			return fmt.Errorf("both bootloader targets failed (EFI: %w; BIOS: %w)", efiErr, biosErr)
 		}
 		return biosErr
 	}

cmd/spinifex/cmd/admin.go

@@ -34,6 +34,7 @@ import (
 	"github.com/mulgadc/spinifex/spinifex/gpu"
 	handlers_ec2_vpc "github.com/mulgadc/spinifex/spinifex/handlers/ec2/vpc"
 	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
+	"github.com/mulgadc/spinifex/spinifex/hostdns"
 	"github.com/mulgadc/spinifex/spinifex/network/host"
 	"github.com/mulgadc/spinifex/spinifex/objectstore"
 	"github.com/mulgadc/spinifex/spinifex/utils"
@@ -60,13 +61,16 @@ var natsConfTemplate string
 //go:embed templates/predastore-multinode.toml
 var predastoreMultiNodeTemplate string
 
+//go:embed templates/northstar.toml
+var northstarTomlTemplate string
+
 var supportedArchs = map[string]bool{
 	"x86_64":  true,
 	"aarch64": true, // alias for arm64
 	"arm64":   true,
 }
 
-// TODO: Confirm suppported platform types
+// TODO: Confirm suppported platform types.
 var supportedPlatforms = map[string]bool{
 	"Linux/UNIX": true,
 	"Windows":    true,
@@ -336,6 +340,7 @@ func init() {
 	adminInitCmd.Flags().Int("external-prefix-len", 24, "External pool subnet prefix length (auto-detected)")
 	adminInitCmd.Flags().Bool("gpu-passthrough", false, "Enable VFIO GPU passthrough (sets gpu_passthrough = true in daemon config)")
 	adminInitCmd.Flags().Bool("ipsec", true, "Encrypt intra-AZ Geneve via OVN native IPsec (cluster-wide); disable only for trusted single-rack lab")
+	adminInitCmd.Flags().Bool("skip-host-dns", false, "Do not point this node's host resolver at its local northstar (LB/EKS names then won't resolve from the node)")
 
 	// Flags for admin join
 	adminJoinCmd.Flags().String("region", "ap-southeast-2", "Region for this node")
@@ -352,6 +357,7 @@ func init() {
 	adminJoinCmd.Flags().StringSlice("services", nil, "Services this node runs (default: all)")
 	adminJoinCmd.Flags().Bool("no-telemetry", false, "Disable telemetry metrics sent during join (default: enabled)")
 	adminJoinCmd.Flags().String("email", "", "Operator email address (used for update and security notifications)")
+	adminJoinCmd.Flags().Bool("skip-host-dns", false, "Do not point this node's host resolver at its local northstar (LB/EKS names then won't resolve from the node)")
 	adminJoinCmd.Flags().Int("predastore-compaction-interval", 0, "Predastore compactor interval in seconds (0 = unset, uses built-in default). Test clusters set a short interval.")
 	adminJoinCmd.MarkFlagRequired("node")
 	adminJoinCmd.MarkFlagRequired("host")
@@ -385,6 +391,27 @@ func init() {
 	}
 }
 
+// amiVolumeSizeGiB returns the smallest whole GiB that still holds sizeBytes.
+//
+// Rounding up is load-bearing. The image is copied into a root volume of
+// exactly this size, so a volume smaller than the image truncates it and the
+// guest comes up with no root partition — it stalls on the root device until
+// systemd drops it to an emergency shell, and nothing on the way there reports
+// an undersized volume. Flooring (plain integer division) undersizes every
+// image that is not an exact multiple of a GiB, which is why this went unseen
+// while every system image happened to be a round 16 GiB.
+//
+// The downstream guard cannot cover for a wrong answer here: floorVolumeSizeToAMI
+// raises a caller's requested size to this value, so it inherits the mistake
+// rather than catching it.
+func amiVolumeSizeGiB(sizeBytes int64) uint64 {
+	const bytesPerGiB = 1024 * 1024 * 1024
+	if sizeBytes <= 0 {
+		return 0
+	}
+	return utils.SafeInt64ToUint64((sizeBytes + bytesPerGiB - 1) / bytesPerGiB)
+}
+
 func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	var image utils.Images
 
@@ -591,7 +618,7 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	manifest.AMIMetadata.RootDeviceType = "ebs"
 	manifest.AMIMetadata.Virtualization = "hvm"
 	manifest.AMIMetadata.ImageOwnerAlias = "system"
-	manifest.AMIMetadata.VolumeSizeGiB = utils.SafeInt64ToUint64(imageStat.Size() / 1024 / 1024 / 1024)
+	manifest.AMIMetadata.VolumeSizeGiB = amiVolumeSizeGiB(imageStat.Size())
 	manifest.AMIMetadata.BootMode = image.BootMode
 	manifest.AMIMetadata.Distro = image.Distro
 	manifest.AMIMetadata.DistroFamily = utils.DistroFamily(image.Distro)
@@ -655,6 +682,12 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 		os.Exit(1)
 	}
 
+	// Import is an interactive command whose output is a progress bar, so the
+	// volume's routine mount chatter is noise here: a fresh import has no prior
+	// state, making the "no state found" / 404 lines expected rather than
+	// notable. Scoping the logger to this VB (New copies it onto the backend
+	// too) keeps it off the process-wide default. Errors still surface, both
+	// through this logger and as the returned error the caller prints.
 	vbConfig := viperblock.VB{
 		VolumeName: volumeId,
 		VolumeSize: utils.SafeInt64ToUint64(imageStat.Size()),
@@ -667,6 +700,7 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 		VolumeConfig:      manifest,
 		MasterKey:         mkey,
 		EncryptionEnabled: mkey != nil,
+		Logger:            slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelError})),
 	}
 
 	// Bake the deployment CA into the image trust store so a stock cloud image
@@ -676,7 +710,25 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	// command, which would collapse the path to a relative config/ca.pem.
 	bakeCACertIntoImage(extractedImagePath, filepath.Join(baseDir, "config", "ca.pem"))
 
-	err = v_utils.ImportDiskImage(&s3Config, &vbConfig, extractedImagePath)
+	// Render the flush bar here rather than inside viperblock, which stays a
+	// pure storage library. The bar is built lazily on the first update so it is
+	// never drawn if the import fails before any bytes flush; viperblock
+	// throttles the callback to ≤101 invocations, so the human-readable title
+	// renders without the per-block render-frequency regression.
+	var flushBar *pterm.ProgressbarPrinter
+	var flushUpdate func(current uint64)
+	progress := func(current, total uint64) {
+		if flushBar == nil {
+			flushBar, flushUpdate = utils.NewByteProgressBar("Flushing image to storage", total)
+		}
+		flushUpdate(current)
+	}
+
+	err = v_utils.ImportDiskImage(&s3Config, &vbConfig, extractedImagePath, progress)
+
+	if flushBar != nil {
+		_, _ = flushBar.Stop()
+	}
 
 	if err != nil {
 		fmt.Fprintf(os.Stderr, "Could not import image to predastore: %v\n", err)
@@ -937,7 +989,7 @@ func runimagesPromoteCmd(cmd *cobra.Command, args []string) {
 	fmt.Printf("✅ Promoted %s to system image (owner: %s).\n", imageID, admin.SystemOwnerAlias)
 }
 
-// List remote images available
+// List remote images available.
 func runimagesListCmd(cmd *cobra.Command, args []string) {
 	//fmt.Println(availableImages)
 
@@ -964,7 +1016,7 @@ func runimagesListCmd(cmd *cobra.Command, args []string) {
 	pterm.Println("spx admin images import --name <image-name>")
 }
 
-// TODO: Move all logic to a module, use minimal application logic in viper commands
+// TODO: Move all logic to a module, use minimal application logic in viper commands.
 func runAdminInit(cmd *cobra.Command, args []string) {
 	if os.Getuid() != 0 {
 		fmt.Fprintln(os.Stderr, "⚠️  Warning: 'spx admin init' is not running as root.")
@@ -1159,10 +1211,31 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		os.Exit(1)
 	}
 
-	// Detect DNS servers from the host for VM DHCP
+	// Validate IP address format
+	if net.ParseIP(bindIP) == nil {
+		fmt.Fprintf(os.Stderr, "❌ Error: Invalid IP address for --bind: %s\n", bindIP)
+		os.Exit(1)
+	}
+
+	// Resolve the off-host advertise IP before detecting DNS. DetectNetwork may
+	// have failed earlier, so retry lazily when the listener still needs a WAN IP.
+	if advertiseFlag == "" && (bindIP == "0.0.0.0" || bindIP == "127.0.0.1") && detectedNet == nil {
+		if d, derr := admin.DetectNetwork(); derr == nil {
+			detectedNet = d
+		}
+	}
+	advertiseIP, err := resolveAdvertiseIP(bindIP, advertiseFlag, detectedNet)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "❌ Error: %v\n", err)
+		os.Exit(1)
+	}
+
+	// Exclude the node-local Northstar listener from its own upstreams. On a
+	// forced re-init, resolvconf may still list this address first; forwarding it
+	// back to Northstar creates a recursive DNS loop.
 	var dnsServers []string
 	if externalMode != "" {
-		dnsServers = detectDNSServers(externalIface)
+		dnsServers = detectDNSServers(externalIface, advertiseIP)
 		if len(dnsServers) > 0 {
 			fmt.Printf("  DNS servers: %s\n", strings.Join(dnsServers, ", "))
 		}
@@ -1198,25 +1271,6 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		})
 	}
 
-	// Validate IP address format
-	if net.ParseIP(bindIP) == nil {
-		fmt.Fprintf(os.Stderr, "❌ Error: Invalid IP address for --bind: %s\n", bindIP)
-		os.Exit(1)
-	}
-
-	// Resolve the off-host advertise IP. DetectNetwork may have failed earlier;
-	// detect lazily if we still need the WAN IP.
-	if advertiseFlag == "" && (bindIP == "0.0.0.0" || bindIP == "127.0.0.1") && detectedNet == nil {
-		if d, derr := admin.DetectNetwork(); derr == nil {
-			detectedNet = d
-		}
-	}
-	advertiseIP, err := resolveAdvertiseIP(bindIP, advertiseFlag, detectedNet)
-	if err != nil {
-		fmt.Fprintf(os.Stderr, "❌ Error: %v\n", err)
-		os.Exit(1)
-	}
-
 	// Validate port range
 	if port < 1 || port > 65535 {
 		fmt.Fprintf(os.Stderr, "❌ Error: Port must be between 1 and 65535, got: %d\n", port)
@@ -1379,6 +1433,29 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	}
 	spxRoot = filepath.Clean(spxRoot)
 
+	// Generate dedicated, bucket-scoped credentials for the northstar DNS
+	// service. Rendered into predastore.toml ([[auth]]) and northstar.toml so
+	// the resolver reads zone files read-only from its own S3 bucket.
+	//
+	// Generated above the multi-node dispatch because the pair is cluster-wide:
+	// a node's predastore only honours the keys rendered into its own config, so
+	// every node must present this same pair to read the distributed zone bucket.
+	northstarAccessKey, err := admin.GenerateAWSAccessKey()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error generating northstar access key: %v\n", err)
+		os.Exit(1)
+	}
+	northstarSecretKey, err := admin.GenerateAWSSecretKey()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error generating northstar secret key: %v\n", err)
+		os.Exit(1)
+	}
+	northstarCreds := admin.NorthstarCredentials{
+		AccessKey: northstarAccessKey,
+		SecretKey: northstarSecretKey,
+		Bucket:    admin.NorthstarBucketName,
+	}
+
 	// Determine if this is a multi-node formation. Operator intent comes from
 	// --nodes, not from whether --bind was left at the 0.0.0.0 default.
 	isMultiNode := nodes >= 2
@@ -1415,7 +1492,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		runAdminInitMultiNode(cmd, accessKey, secretKey, accountID, adminAccessKey, adminSecretKey,
 			masterKey, viperblockKey, natsToken, clusterName,
 			configDir, spxRoot, certPath, region, az, node, bindIP, advertiseIP, clusterBind, email,
-			port, nodes, formationTimeoutStr, tokenTTLStr, services, networkConfig)
+			port, nodes, formationTimeoutStr, tokenTTLStr, services, networkConfig, northstarCreds)
 		return
 	}
 
@@ -1432,6 +1509,10 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 
 	portStr := strconv.Itoa(port)
 
+	// The keys are cluster-wide and generated above the dispatch; the config path
+	// is node-local, so it is derived here from this node's own config dirs.
+	northstarConfigPath := filepath.Join(dirs.Northstar, "northstar.toml")
+
 	// Parse multi-node predastore configuration (legacy flag-based approach for single-node)
 	var predastoreNodeID int
 	if predastoreNodesStr != "" {
@@ -1461,7 +1542,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		}
 
 		// Generate multi-node predastore.toml
-		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP, compactionInterval)
+		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP, compactionInterval, northstarCreds)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Error generating multi-node predastore config: %v\n", err)
 			os.Exit(1)
@@ -1526,6 +1607,14 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		IPSecEnabled:   ipsecEnabled,
 
 		EncryptionKeyFile: viperblockKeyPath,
+
+		NorthstarAccessKey:      northstarCreds.AccessKey,
+		NorthstarSecretKey:      northstarCreds.SecretKey,
+		NorthstarBucket:         northstarCreds.Bucket,
+		NorthstarDefaultDomain:  admin.NorthstarDefaultDomain,
+		NorthstarInternalDomain: admin.NorthstarInternalDomain,
+		NorthstarConfigPath:     northstarConfigPath,
+		PoolDNSServers:          dnsServers,
 	}
 
 	// Print external networking summary
@@ -1567,6 +1656,9 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 
 	finalizeNodeSetup(spxRoot, certPath, adminAccessKey, adminSecretKey, region, bindIP)
 
+	skipHostDNS, _ := cmd.Flags().GetBool("skip-host-dns")
+	configureHostDNS(configSettings, skipHostDNS)
+
 	// Write node.conf so spx admin banner works on source installs (not just ISO).
 	nodeHostname, _ := os.Hostname()
 	if nodeHostname == "" {
@@ -1596,10 +1688,15 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 // runAdminInitMultiNode handles the multi-node formation path for admin init.
 // It starts a formation server, registers this node, waits for all nodes to join,
 // then generates configs with complete cluster topology.
+//
+// The northstar credentials are generated by the caller and provision this
+// node's predastore with the zone bucket. The same pair is distributed to
+// joiners, since each node's predastore only honours the keys in its own config.
 func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID, adminAccessKey, adminSecretKey string,
 	masterKey, viperblockKey []byte, natsToken, clusterName,
 	configDir, spxRoot, certPath, region, az, node, bindIP, advertiseIP, clusterBind, email string,
-	port, expectedNodes int, formationTimeoutStr, tokenTTLStr string, services []string, networkConfig *formation.NetworkConfig) {
+	port, expectedNodes int, formationTimeoutStr, tokenTTLStr string, services []string, networkConfig *formation.NetworkConfig,
+	northstarCreds admin.NorthstarCredentials) {
 	formationTimeout, err := time.ParseDuration(formationTimeoutStr)
 	if err != nil {
 		fmt.Fprintf(os.Stderr, "❌ Error: Invalid --formation-timeout: %v\n", err)
@@ -1651,6 +1748,12 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 		Region:         region,
 		AdminAccessKey: adminAccessKey,
 		AdminSecretKey: adminSecretKey,
+
+		// Joiners provision their own predastore with this pair, so every
+		// node's resolver can read the distributed zone bucket via its local
+		// endpoint. The bucket is a constant, so it is derived node-side.
+		NorthstarAccessKey: northstarCreds.AccessKey,
+		NorthstarSecretKey: northstarCreds.SecretKey,
 	}
 
 	fs := formation.NewFormationServer(expectedNodes, creds, string(caCertData), string(caKeyData), networkConfig, joinToken, tokenTTL)
@@ -1722,11 +1825,15 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 
 	portStr := strconv.Itoa(port)
 
+	// The keys are cluster-wide and generated above the dispatch; the config path
+	// is node-local, so it is derived here from this node's own config dirs.
+	northstarConfigPath := filepath.Join(dirs.Northstar, "northstar.toml")
+
 	// Generate multi-node predastore config
 	var predastoreNodeID int
 	hasPredastoreConfig := len(predastoreNodes) >= 2
 	if hasPredastoreConfig {
-		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP, compactionInterval)
+		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP, compactionInterval, northstarCreds)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Error generating multi-node predastore config: %v\n", err)
 			os.Exit(1)
@@ -1766,12 +1873,19 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 		PredastoreNodeID:          predastoreNodeID,
 		CompactionIntervalSeconds: compactionInterval,
 		Services:                  services,
-		RemoteNodes:               buildRemoteNodes(allNodes, node),
+		RemoteNodes:               buildRemoteNodes(allNodes, node, northstarConfigPath),
 
 		OperatorEmail: email,
 
 		EncryptionKeyFile: viperblockKeyPath,
 
+		NorthstarAccessKey:      northstarCreds.AccessKey,
+		NorthstarSecretKey:      northstarCreds.SecretKey,
+		NorthstarBucket:         northstarCreds.Bucket,
+		NorthstarDefaultDomain:  admin.NorthstarDefaultDomain,
+		NorthstarInternalDomain: admin.NorthstarInternalDomain,
+		NorthstarConfigPath:     northstarConfigPath,
+
 		// Multi-endpoint OVN NB/SB list across the RAFT quorum; the init node's
 		// own address leads, the rest provide failover.
 		OVNNBAddr: ovnNBAddr,
@@ -1789,6 +1903,9 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 
 	finalizeNodeSetup(spxRoot, certPath, adminAccessKey, adminSecretKey, region, bindIP)
 
+	skipHostDNS, _ := cmd.Flags().GetBool("skip-host-dns")
+	configureHostDNS(configSettings, skipHostDNS)
+
 	// Keep formation server running briefly so joining nodes can fetch complete status
 	fmt.Println("\n⏳ Waiting for joining nodes to fetch cluster data...")
 	time.Sleep(15 * time.Second)
@@ -2174,12 +2291,15 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 
 	portStr := strconv.Itoa(port)
 
+	northstarCreds, northstarConfigPath := northstarFromFormation(creds, dirs)
+
 	// Generate multi-node predastore config
 	var predastoreNodeID int
 	hasPredastoreConfig := len(predastoreNodes) >= 2
 
 	if hasPredastoreConfig {
-		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, creds.AccessKey, creds.SecretKey, creds.Region, creds.NatsToken, configDir, bindIP, compactionInterval)
+		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, creds.AccessKey, creds.SecretKey, creds.Region, creds.NatsToken, configDir, bindIP, compactionInterval,
+			northstarCreds)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Error generating multi-node predastore config: %v\n", err)
 			os.Exit(1)
@@ -2223,12 +2343,19 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 		PredastoreNodeID:          predastoreNodeID,
 		CompactionIntervalSeconds: compactionInterval,
 		Services:                  services,
-		RemoteNodes:               buildRemoteNodes(statusResp.Nodes, node),
+		RemoteNodes:               buildRemoteNodes(statusResp.Nodes, node, northstarConfigPath),
 
 		OperatorEmail: email,
 
 		EncryptionKeyFile: viperblockKeyPath,
 
+		NorthstarAccessKey:      northstarCreds.AccessKey,
+		NorthstarSecretKey:      northstarCreds.SecretKey,
+		NorthstarBucket:         northstarCreds.Bucket,
+		NorthstarDefaultDomain:  admin.NorthstarDefaultDomain,
+		NorthstarInternalDomain: admin.NorthstarInternalDomain,
+		NorthstarConfigPath:     northstarConfigPath,
+
 		// Multi-endpoint OVN NB/SB list across the RAFT quorum so the client
 		// fails over instead of pinning to a single init node.
 		OVNNBAddr: ovnNBAddr,
@@ -2246,6 +2373,9 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 
 	finalizeNodeSetup(dataDir, caCertPath, creds.AdminAccessKey, creds.AdminSecretKey, creds.Region, bindIP)
 
+	skipHostDNS, _ := cmd.Flags().GetBool("skip-host-dns")
+	configureHostDNS(configSettings, skipHostDNS)
+
 	// Print cluster summary
 	fmt.Println("\n🎉 Node successfully joined cluster!")
 	fmt.Printf("   Cluster: %s (%d nodes)\n", creds.ClusterName, len(statusResp.Nodes))
@@ -2285,7 +2415,16 @@ func resolveAdvertiseIP(bindIP, advertiseFlag string, detected *admin.DetectedNe
 // buildRemoteNodes converts formation NodeInfo into RemoteNode entries,
 // excluding the local node. This puts all cluster members into spinifex.toml
 // so config is the source of truth for expected cluster membership.
-func buildRemoteNodes(allNodes map[string]formation.NodeInfo, localNode string) []admin.RemoteNode {
+//
+// northstarConfigPath is the local node's own path, republished for every peer:
+// every node in a formed cluster runs northstar, and the seed set must be
+// identical on all of them or the base zone's NS records get pinned to whichever
+// node wins the create-if-absent race. --config-dir is per-node and does not
+// cross the wire, so the value is accurate whenever nodes share a config dir and
+// inert when they do not — only its emptiness is ever read back. Passing it
+// empty (no credentials distributed) leaves peers with no stanza, so no node is
+// advertised as a resolver it cannot be.
+func buildRemoteNodes(allNodes map[string]formation.NodeInfo, localNode, northstarConfigPath string) []admin.RemoteNode {
 	var remote []admin.RemoteNode
 	for name, n := range allNodes {
 		if name == localNode {
@@ -2298,11 +2437,12 @@ func buildRemoteNodes(allNodes map[string]formation.NodeInfo, localNode string)
 			host = n.BindIP
 		}
 		remote = append(remote, admin.RemoteNode{
-			Name:     name,
-			Host:     host,
-			Region:   n.Region,
-			AZ:       n.AZ,
-			Services: n.Services,
+			Name:                name,
+			Host:                host,
+			Region:              n.Region,
+			AZ:                  n.AZ,
+			Services:            n.Services,
+			NorthstarConfigPath: northstarConfigPath,
 		})
 	}
 	sort.Slice(remote, func(i, j int) bool {
@@ -2326,7 +2466,9 @@ func initIAMServiceFromConfig() (*handlers_iam.IAMServiceImpl, *config.ClusterCo
 		return nil, nil, nil, nil, fmt.Errorf("load master key: %w", err)
 	}
 
-	svc, err := handlers_iam.NewIAMServiceImpl(nc, masterKey, len(cfg.Nodes))
+	// Background: this runs at CLI top level, where there is no request to
+	// inherit a deadline from and the process exits after the one command.
+	svc, err := handlers_iam.NewIAMServiceImpl(context.Background(), nc, masterKey, len(cfg.Nodes))
 	if err != nil {
 		nc.Close()
 		return nil, nil, nil, nil, fmt.Errorf("init IAM service: %w", err)
@@ -2360,7 +2502,7 @@ func runAccountCreate(cmd *cobra.Command, args []string) {
 	// Create default VPC for the new account (belt-and-suspenders: daemon also
 	// does this via iam.account.created event, but daemon may not be running).
 	nodeConfig := cfg.Nodes[cfg.Node]
-	vpcSvc, vpcErr := handlers_ec2_vpc.NewVPCServiceImplWithNATS(&nodeConfig, nc)
+	vpcSvc, vpcErr := handlers_ec2_vpc.NewVPCServiceImplWithNATS(context.Background(), &nodeConfig, nc)
 	if vpcErr != nil {
 		fmt.Fprintf(os.Stderr, "Warning: could not create default VPC service: %v\n", vpcErr)
 	} else if _, vpcErr = vpcSvc.EnsureDefaultVPC(accountID); vpcErr != nil {
@@ -2518,6 +2660,7 @@ type configDirs struct {
 	Viperblock string
 	NATS       string
 	Spinifex   string
+	Northstar  string
 }
 
 // createConfigSubdirs creates the standard config subdirectories under configDir.
@@ -2528,8 +2671,9 @@ func createConfigSubdirs(configDir string) (configDirs, error) {
 		Viperblock: filepath.Join(configDir, "viperblock"),
 		NATS:       filepath.Join(configDir, "nats"),
 		Spinifex:   filepath.Join(configDir, "spinifex"),
+		Northstar:  filepath.Join(configDir, "northstar"),
 	}
-	for _, dir := range []string{dirs.AWSGW, dirs.Predastore, dirs.Viperblock, dirs.NATS, dirs.Spinifex} {
+	for _, dir := range []string{dirs.AWSGW, dirs.Predastore, dirs.Viperblock, dirs.NATS, dirs.Spinifex, dirs.Northstar} {
 		if err := os.MkdirAll(dir, 0700); err != nil {
 			return configDirs{}, fmt.Errorf("create directory %s: %w", dir, err)
 		}
@@ -2537,14 +2681,55 @@ func createConfigSubdirs(configDir string) (configDirs, error) {
 	return dirs, nil
 }
 
+// northstarFromFormation derives a joining node's northstar credentials and
+// node-local config path from the cluster-wide pair distributed at formation.
+//
+// The keys cross the wire because a node's predastore only honours the keys in
+// its own config, so every node must present the same pair to read the
+// distributed zone bucket. The bucket name and config path are node-local
+// constants, so they are derived here rather than carried.
+//
+// A leader that predates credential distribution sends no keys. That yields a
+// zero pair and an empty path, so the node renders no northstar config at all
+// rather than a resolver holding a key its own predastore would reject.
+func northstarFromFormation(creds *formation.SharedCredentials, dirs configDirs) (admin.NorthstarCredentials, string) {
+	if creds.NorthstarAccessKey == "" || creds.NorthstarSecretKey == "" {
+		return admin.NorthstarCredentials{}, ""
+	}
+	return admin.NorthstarCredentials{
+		AccessKey: creds.NorthstarAccessKey,
+		SecretKey: creds.NorthstarSecretKey,
+		Bucket:    admin.NorthstarBucketName,
+	}, filepath.Join(dirs.Northstar, "northstar.toml")
+}
+
 // generateAndWriteConfigs renders the standard config files (spinifex.toml,
 // awsgw.toml, nats.conf, and optionally predastore.toml) from templates.
 func generateAndWriteConfigs(dirs configDirs, spinifexTomlPath string, settings admin.ConfigSettings, skipPredastore bool) error {
+	// A one-sided pair must disable Northstar wholesale. Rendering only the
+	// public stanza or only the secret file advertises a resolver that cannot run.
+	northstarEnabled := settings.NorthstarAccessKey != "" && settings.NorthstarSecretKey != ""
+	if !northstarEnabled {
+		settings.NorthstarAccessKey = ""
+		settings.NorthstarSecretKey = ""
+		settings.NorthstarBucket = ""
+		settings.NorthstarConfigPath = ""
+	}
+
 	configs := []admin.ConfigFile{
 		{Name: "spinifex.toml", Path: spinifexTomlPath, Template: spinifexTomlTemplate},
 		{Name: filepath.Join(dirs.AWSGW, "awsgw.toml"), Path: filepath.Join(dirs.AWSGW, "awsgw.toml"), Template: awsgwTomlTemplate},
 		{Name: filepath.Join(dirs.NATS, "nats.conf"), Path: filepath.Join(dirs.NATS, "nats.conf"), Template: natsConfTemplate},
 	}
+	// northstar.toml is only rendered when scoped DNS credentials were
+	// provisioned. A cluster formed by a leader that predates their distribution
+	// leaves the keys empty, yielding no northstar config rather than a partial
+	// one pointing at a bucket the local predastore does not serve.
+	if northstarEnabled {
+		configs = append(configs, admin.ConfigFile{
+			Name: filepath.Join(dirs.Northstar, "northstar.toml"), Path: filepath.Join(dirs.Northstar, "northstar.toml"), Template: northstarTomlTemplate,
+		})
+	}
 	if !skipPredastore {
 		configs = append(configs, admin.ConfigFile{
 			Name: filepath.Join(dirs.Predastore, "predastore.toml"), Path: filepath.Join(dirs.Predastore, "predastore.toml"), Template: predastoreTomlTemplate,
@@ -2553,6 +2738,45 @@ func generateAndWriteConfigs(dirs configDirs, spinifexTomlPath string, settings
 	return admin.GenerateConfigFiles(configs, settings)
 }
 
+// hostDNSParams derives the resolver target from the settings just rendered into
+// northstar.toml and reports whether host DNS should be configured at all. Host
+// DNS is pointed at the node's own northstar only when a northstar config was
+// rendered (scoped DNS credentials present) and the operator did not opt out.
+// The resolver is the node AdvertiseIP, falling back to BindIP.
+func hostDNSParams(settings admin.ConfigSettings, skip bool) (hostdns.Params, bool) {
+	northstarEnabled := settings.NorthstarAccessKey != "" && settings.NorthstarSecretKey != ""
+	if !northstarEnabled || skip {
+		return hostdns.Params{}, false
+	}
+	resolverIP := settings.AdvertiseIP
+	if resolverIP == "" {
+		resolverIP = settings.BindIP
+	}
+	return hostdns.Params{
+		ResolverIP:     resolverIP,
+		BaseDomain:     settings.NorthstarDefaultDomain,
+		InternalDomain: settings.NorthstarInternalDomain,
+	}, true
+}
+
+// configureHostDNS points this node's host resolver at its own northstar so the
+// Spinifex authoritative zones (ELB/EKS names) resolve from the node itself on
+// every install path. Failures are surfaced loudly but never abort a formed
+// cluster, mirroring installCACertificate.
+func configureHostDNS(settings admin.ConfigSettings, skip bool) {
+	params, ok := hostDNSParams(settings, skip)
+	if !ok {
+		return
+	}
+	fmt.Println("\n🔧 Configuring host DNS for Spinifex zones...")
+	if err := hostdns.Configure(params); err != nil {
+		fmt.Fprintf(os.Stderr, "⚠️  Warning: could not configure host DNS: %v\n", err)
+		fmt.Fprintf(os.Stderr, "    LB/EKS names may not resolve from this node until the host resolver points at %s:53.\n", params.ResolverIP)
+		return
+	}
+	fmt.Printf("✅ Host DNS: %s + %s -> %s:53 (northstar)\n", params.BaseDomain, params.InternalDomain, params.ResolverIP)
+}
+
 // finalizeNodeSetup configures AWS credentials, creates service directories,
 // and sets ownership when running as root.
 func finalizeNodeSetup(dataDir, certPath, adminAccessKey, adminSecretKey, region, bindIP string) {
@@ -2575,6 +2799,7 @@ func finalizeNodeSetup(dataDir, certPath, adminAccessKey, adminSecretKey, region
 func applyNetworkConfig(settings *admin.ConfigSettings, nc *formation.NetworkConfig) {
 	settings.IPSecEnabled = nc.IPSecEnabled
 	settings.ExternalMode = nc.ExternalMode
+	settings.PoolDNSServers = nc.PoolDNSServers
 	if nc.ExternalMode != "" {
 		settings.Pools = []admin.PoolData{{
 			Name:       nc.PoolName,
@@ -2857,15 +3082,36 @@ func bridgeModeFor(externalMode string) string {
 	return ""
 }
 
-// detectDNSServers auto-detects DNS servers from the host for the specified
-// interface. Uses resolvectl (systemd-resolved) first, then falls back to
-// /etc/resolv.conf. Returns up to 3 servers. Falls back to public DNS if none found.
-func detectDNSServers(iface string) []string {
+// dnsDetectionCommandTimeout prevents a wedged resolver manager from blocking init.
+const dnsDetectionCommandTimeout = 5 * time.Second
+
+// dnsDetectionSources isolates host resolver discovery for deterministic tests.
+type dnsDetectionSources struct {
+	queryResolvectl func(args ...string) (string, error)
+	readResolvConf  func() (string, error)
+}
+
+// detectDNSServers auto-detects up to three host DNS servers, preferring
+// systemd-resolved before resolv.conf and public fallbacks.
+func detectDNSServers(iface string, excludedIPs ...string) []string {
+	return detectDNSServersWithSources(iface, excludedIPs, dnsDetectionSources{
+		queryResolvectl: func(args ...string) (string, error) {
+			return utils.RunCommandWithTimeout(dnsDetectionCommandTimeout, "resolvectl", args...)
+		},
+		readResolvConf: func() (string, error) {
+			data, err := os.ReadFile("/etc/resolv.conf")
+			return string(data), err
+		},
+	})
+}
+
+// detectDNSServersWithSources finds usable upstreams in resolver-manager order.
+func detectDNSServersWithSources(iface string, excludedIPs []string, sources dnsDetectionSources) []string {
 	// Try resolvectl for the specific link first (most reliable on modern systems)
 	if iface != "" {
-		out, err := exec.Command("resolvectl", "dns", iface).CombinedOutput()
+		out, err := sources.queryResolvectl("dns", iface)
 		if err == nil {
-			servers := parseDNSFromResolvectl(string(out))
+			servers := filterDNSServers(parseDNSFromResolvectl(out), excludedIPs...)
 			if len(servers) > 0 {
 				return servers
 			}
@@ -2873,42 +3119,70 @@ func detectDNSServers(iface string) []string {
 	}
 
 	// Try resolvectl global
-	out, err := exec.Command("resolvectl", "dns").CombinedOutput()
+	out, err := sources.queryResolvectl("dns")
 	if err == nil {
-		servers := parseDNSFromResolvectl(string(out))
+		servers := filterDNSServers(parseDNSFromResolvectl(out), excludedIPs...)
 		if len(servers) > 0 {
 			return servers
 		}
 	}
 
 	// Fall back to /etc/resolv.conf
-	data, err := os.ReadFile("/etc/resolv.conf")
+	resolvConf, err := sources.readResolvConf()
 	if err == nil {
 		var servers []string
-		for line := range strings.SplitSeq(string(data), "\n") {
+		for line := range strings.SplitSeq(resolvConf, "\n") {
 			line = strings.TrimSpace(line)
 			if strings.HasPrefix(line, "nameserver ") {
-				ip := strings.TrimSpace(strings.TrimPrefix(line, "nameserver"))
-				// Skip localhost (systemd-resolved stub)
-				if ip != "127.0.0.53" && ip != "127.0.0.1" && net.ParseIP(ip) != nil {
-					servers = append(servers, ip)
-				}
+				servers = append(servers, strings.TrimSpace(strings.TrimPrefix(line, "nameserver")))
 			}
 		}
-		if len(servers) > 0 {
-			if len(servers) > 3 {
-				servers = servers[:3]
-			}
+		if servers = filterDNSServers(servers, excludedIPs...); len(servers) > 0 {
 			return servers
 		}
 	}
 
 	// Fallback to well-known public DNS
-	return []string{"8.8.8.8", "1.1.1.1"}
+	return filterDNSServers([]string{"8.8.8.8", "1.1.1.1"}, excludedIPs...)
+}
+
+// filterDNSServers removes local, duplicate, and invalid resolver addresses and
+// caps the upstream list at the three entries supported by generated configs.
+func filterDNSServers(servers []string, excludedIPs ...string) []string {
+	excluded := make(map[string]struct{}, len(excludedIPs)+2)
+	excluded["127.0.0.1"] = struct{}{}
+	excluded["127.0.0.53"] = struct{}{}
+	for _, value := range excludedIPs {
+		if ip := net.ParseIP(strings.TrimSpace(value)); ip != nil {
+			excluded[ip.String()] = struct{}{}
+		}
+	}
+
+	filtered := make([]string, 0, min(len(servers), 3))
+	seen := make(map[string]struct{}, len(servers))
+	for _, value := range servers {
+		ip := net.ParseIP(strings.TrimSpace(value))
+		if ip == nil {
+			continue
+		}
+		normalized := ip.String()
+		if _, skip := excluded[normalized]; skip {
+			continue
+		}
+		if _, duplicate := seen[normalized]; duplicate {
+			continue
+		}
+		seen[normalized] = struct{}{}
+		filtered = append(filtered, normalized)
+		if len(filtered) == 3 {
+			break
+		}
+	}
+	return filtered
 }
 
 // parseDNSFromResolvectl extracts IP addresses from resolvectl dns output.
-// Format: "Link 2 (enp0s3): 192.168.1.1 8.8.8.8 1.1.1.1"
+// Format: "Link 2 (enp0s3): 192.168.1.1 8.8.8.8 1.1.1.1".
 func parseDNSFromResolvectl(output string) []string {
 	var servers []string
 	for line := range strings.SplitSeq(output, "\n") {
@@ -2919,14 +3193,11 @@ func parseDNSFromResolvectl(output string) []string {
 		}
 		fields := strings.FieldsSeq(after)
 		for f := range fields {
-			if net.ParseIP(f) != nil && f != "127.0.0.53" && f != "127.0.0.1" {
+			if net.ParseIP(f) != nil {
 				servers = append(servers, f)
 			}
 		}
 	}
-	if len(servers) > 3 {
-		servers = servers[:3]
-	}
 	return servers
 }
 

cmd/spinifex/cmd/admin_nat_test.go

@@ -138,8 +138,8 @@ func TestIsNonBridgeableUplink(t *testing.T) {
 
 func TestBridgeModeFor(t *testing.T) {
 	assert.Equal(t, "nat", bridgeModeFor("nat"))
-	assert.Equal(t, "", bridgeModeFor("pool"))
-	assert.Equal(t, "", bridgeModeFor(""))
+	assert.Empty(t, bridgeModeFor("pool"))
+	assert.Empty(t, bridgeModeFor(""))
 }
 
 func TestSpinifexTomlTemplate_NATModeWithPublicPool(t *testing.T) {

cmd/spinifex/cmd/admin_test.go

@@ -10,8 +10,11 @@ import (
 	"testing"
 
 	"github.com/mulgadc/spinifex/spinifex/admin"
+	"github.com/mulgadc/spinifex/spinifex/config"
 	"github.com/mulgadc/spinifex/spinifex/formation"
+	handlers_dns "github.com/mulgadc/spinifex/spinifex/handlers/dns"
 	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
+	"github.com/mulgadc/spinifex/spinifex/hostdns"
 	"github.com/mulgadc/spinifex/spinifex/utils"
 	toml "github.com/pelletier/go-toml/v2"
 	"github.com/spf13/cobra"
@@ -51,7 +54,7 @@ func TestBuildRemoteNodes_AdvertiseFallback(t *testing.T) {
 		"node2": {Name: "node2", BindIP: "10.0.0.2"}, // legacy joiner
 		"node3": {Name: "node3", BindIP: "10.0.0.3", AdvertiseIP: "203.0.113.3"},
 	}
-	got := buildRemoteNodes(nodes, "node3")
+	got := buildRemoteNodes(nodes, "node3", "")
 	if assert.Len(t, got, 2) {
 		assert.Equal(t, "node1", got[0].Name)
 		assert.Equal(t, "203.0.113.1", got[0].Host, "advertise wins when set")
@@ -60,6 +63,29 @@ func TestBuildRemoteNodes_AdvertiseFallback(t *testing.T) {
 	}
 }
 
+// Every peer must carry the northstar stanza, since a node that describes only
+// its own northstar computes a single-entry seed set and pins the base zone's NS
+// records to whichever node won the create-if-absent race.
+func TestBuildRemoteNodes_NorthstarConfigPath(t *testing.T) {
+	nodes := map[string]formation.NodeInfo{
+		"node1": {Name: "node1", BindIP: "10.0.0.1"},
+		"node2": {Name: "node2", BindIP: "10.0.0.2"},
+	}
+
+	got := buildRemoteNodes(nodes, "node2", "/etc/spinifex/northstar/northstar.toml")
+	if assert.Len(t, got, 1) {
+		assert.Equal(t, "/etc/spinifex/northstar/northstar.toml", got[0].NorthstarConfigPath)
+	}
+
+	// No distributed credentials means no node runs northstar. Publishing peers
+	// as resolvers anyway would point guests at 169.254.169.253 for a service
+	// that never starts.
+	got = buildRemoteNodes(nodes, "node2", "")
+	if assert.Len(t, got, 1) {
+		assert.Empty(t, got[0].NorthstarConfigPath)
+	}
+}
+
 func TestResolveAdvertiseIP(t *testing.T) {
 	wan := &admin.DetectedNetwork{WAN: &admin.DetectedInterface{IP: "192.168.1.21"}}
 	noWAN := &admin.DetectedNetwork{}
@@ -146,6 +172,44 @@ func TestSpinifexTomlTemplate_AdvertiseOmittedWhenEmpty(t *testing.T) {
 	assert.NotContains(t, string(data), "advertise =")
 }
 
+// The northstar stanza carries the non-secret domains so confined services
+// (vpcd) resolve DNS names without reading the 0600 northstar.toml.
+func TestSpinifexTomlTemplate_NorthstarDomains(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node:                    "node1",
+		Az:                      "ap-southeast-2a",
+		Port:                    "4432",
+		Region:                  "ap-southeast-2",
+		BindIP:                  "10.11.12.1",
+		AccessKey:               "AKIATEST",
+		SecretKey:               "SECRET",
+		AccountID:               "123456789012",
+		NatsToken:               "token",
+		ConfigDir:               dir,
+		OVNNBAddr:               "tcp:127.0.0.1:6641",
+		OVNSBAddr:               "tcp:127.0.0.1:6642",
+		NorthstarConfigPath:     "/etc/spinifex/northstar/northstar.toml",
+		NorthstarDefaultDomain:  "spx3.net",
+		NorthstarInternalDomain: "compute.internal",
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	content := string(data)
+	assert.Contains(t, content, `config_path = "/etc/spinifex/northstar/northstar.toml"`)
+	assert.Contains(t, content, `default_domain = "spx3.net"`)
+	assert.Contains(t, content, `internal_domain = "compute.internal"`)
+
+	// The stanza must round-trip into NorthstarConfig so the resolvers read it.
+	cfg, err := config.LoadConfig(path)
+	require.NoError(t, err)
+	assert.Equal(t, "spx3.net", cfg.Nodes["node1"].Northstar.DefaultDomain)
+	assert.Equal(t, "compute.internal", cfg.Nodes["node1"].Northstar.InternalDomain)
+}
+
 // Legacy `wan_bridge` TOML key must fail-start vpcd with guidance, not silently
 // alias (per mulga-998 D3). Prevents the footgun where operators inherited the
 // old key pointing at 'br-ext' and got broken DHCP on veth-mode hosts.
@@ -210,7 +274,7 @@ func TestPredastoreMultinodeTemplate_NatsURLLoopbackShim(t *testing.T) {
 		{ID: 3, Host: "10.0.0.3"},
 	}
 	content, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "0.0.0.0", 0,
+		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "0.0.0.0", 0, admin.NorthstarCredentials{},
 	)
 	require.NoError(t, err)
 	assert.Contains(t, content, `nats_url = "nats://localhost:4222"`)
@@ -218,7 +282,7 @@ func TestPredastoreMultinodeTemplate_NatsURLLoopbackShim(t *testing.T) {
 
 	// Specific bind IP → stays as-is.
 	content2, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "10.11.12.1", 0,
+		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "10.11.12.1", 0, admin.NorthstarCredentials{},
 	)
 	require.NoError(t, err)
 	assert.Contains(t, content2, `nats_url = "nats://10.11.12.1:4222"`)
@@ -482,6 +546,32 @@ func TestSpinifexTomlTemplate_EncryptionKeyFile(t *testing.T) {
 	assert.NotContains(t, string(data2), "encryption_key_file", "blank key must omit the field")
 }
 
+// A rendered node config must turn chunk GC on. Asserting the template text
+// alone is not enough: the value has to survive parsing into a non-nil true
+// pointer, because every consumer treats nil as off. Rendering and loading
+// together is the only check that covers the whole path.
+func TestSpinifexTomlTemplate_GCEnabled(t *testing.T) {
+	dir := t.TempDir()
+	settings := admin.ConfigSettings{
+		Node: "node1", Az: "ap-southeast-2a", Port: "4432", Region: "ap-southeast-2",
+		BindIP: "0.0.0.0", AccessKey: "AKIATEST", SecretKey: "SECRET",
+		AccountID: "123456789012", NatsToken: "token",
+		OVNNBAddr: "tcp:127.0.0.1:6641", OVNSBAddr: "tcp:127.0.0.1:6642",
+		ConfigDir: dir,
+	}
+
+	path := filepath.Join(dir, "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	cfg, err := config.LoadConfig(path)
+	require.NoError(t, err)
+
+	node, ok := cfg.Nodes["node1"]
+	require.True(t, ok, "rendered config must define the node")
+	require.NotNil(t, node.Viperblock.GCEnabled, "gc_enabled must be rendered; nil leaves chunk GC off on every deployed node")
+	assert.True(t, *node.Viperblock.GCEnabled, "chunk GC must be enabled in a freshly rendered node config")
+}
+
 // Joiners run `spx admin join` against a configDir that doesn't yet contain
 // a predastore/ subdir. The helper must create it (predastore subdir is
 // owned by spinifex-storage in prod, but the helper just needs the dir to
@@ -524,15 +614,15 @@ const cobraRequiredAnnotation = "cobra_annotation_bash_completion_one_required_f
 func TestAdminInitCmd_ExternalDHCPFlagSchema(t *testing.T) {
 	sourceFlag := adminInitCmd.Flags().Lookup("external-source")
 	require.NotNil(t, sourceFlag, "--external-source must be defined")
-	assert.Equal(t, "", sourceFlag.DefValue)
+	assert.Empty(t, sourceFlag.DefValue)
 
 	bindFlag := adminInitCmd.Flags().Lookup("external-bind-bridge")
 	require.NotNil(t, bindFlag, "--external-bind-bridge must be defined")
-	assert.Equal(t, "", bindFlag.DefValue)
+	assert.Empty(t, bindFlag.DefValue)
 
 	poolFlag := adminInitCmd.Flags().Lookup("external-pool")
 	require.NotNil(t, poolFlag)
-	assert.Equal(t, "", poolFlag.DefValue)
+	assert.Empty(t, poolFlag.DefValue)
 }
 
 // DHCP-sourced pool must emit source + bind_bridge into spinifex.toml so the
@@ -631,6 +721,297 @@ func predastoreMultinodeNodes() []admin.PredastoreNodeConfig {
 	}
 }
 
+// predastoreBucketAuth parses the buckets and auth entries a northstar-enabled
+// multi-node predastore must render.
+type predastoreBucketAuth struct {
+	Buckets []struct {
+		Name string `toml:"name"`
+		Type string `toml:"type"`
+	} `toml:"buckets"`
+	Auth []struct {
+		AccessKeyID     string `toml:"access_key_id"`
+		SecretAccessKey string `toml:"secret_access_key"`
+		Policy          []struct {
+			Bucket  string   `toml:"bucket"`
+			Actions []string `toml:"actions"`
+		} `toml:"policy"`
+	} `toml:"auth"`
+}
+
+// policyFor returns the actions the given key is granted on the given bucket,
+// and whether the key holds any policy for it at all.
+func (c predastoreBucketAuth) policyFor(accessKey, bucket string) ([]string, bool) {
+	for _, auth := range c.Auth {
+		if auth.AccessKeyID != accessKey {
+			continue
+		}
+		for _, p := range auth.Policy {
+			if p.Bucket == bucket {
+				return p.Actions, true
+			}
+		}
+	}
+	return nil, false
+}
+
+func (c predastoreBucketAuth) hasBucket(name string) bool {
+	for _, b := range c.Buckets {
+		if b.Name == name {
+			return true
+		}
+	}
+	return false
+}
+
+// Northstar on a multi-node cluster needs three things from predastore, all
+// gated on the same credential: the zone bucket, write access for the system
+// key (which is what BootstrapBaseZone seeds zones with), and the resolver's
+// own read-only key. The template rendered all three correctly, but the caller
+// never populated the credential, so multi-node clusters silently got none of
+// them.
+func TestPredastoreMultinodeTemplate_NorthstarProvisioned(t *testing.T) {
+	content, err := admin.GenerateMultiNodePredastoreConfig(
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0,
+		admin.NorthstarCredentials{AccessKey: "NSAK", SecretKey: "NSSK", Bucket: admin.NorthstarBucketName},
+	)
+	require.NoError(t, err)
+
+	var cfg predastoreBucketAuth
+	require.NoError(t, toml.Unmarshal([]byte(content), &cfg))
+
+	assert.True(t, cfg.hasBucket(admin.NorthstarBucketName), "northstar zone bucket not rendered")
+
+	// The zone bucket must be readable from every node's local predastore, so
+	// a resolver on any node can serve it.
+	for _, b := range cfg.Buckets {
+		if b.Name == admin.NorthstarBucketName {
+			assert.Equal(t, "distributed", b.Type)
+		}
+	}
+
+	// BootstrapBaseZone writes zone files with the system key, not the
+	// resolver's. Without this grant, seeding fails on every node.
+	systemActions, ok := cfg.policyFor("AK", admin.NorthstarBucketName)
+	require.True(t, ok, "system key has no policy for the northstar bucket")
+	assert.Contains(t, systemActions, "s3:GetObject", "bootstrap must check whether the zone already exists")
+	assert.Contains(t, systemActions, "s3:PutObject", "bootstrap must create a missing zone")
+
+	// The resolver's key stays read-only and scoped to the zone bucket alone.
+	nsActions, ok := cfg.policyFor("NSAK", admin.NorthstarBucketName)
+	require.True(t, ok, "northstar key has no policy for the northstar bucket")
+	assert.ElementsMatch(t, []string{"s3:ListBucket", "s3:GetObject"}, nsActions)
+	for _, auth := range cfg.Auth {
+		if auth.AccessKeyID == "NSAK" {
+			assert.Equal(t, "NSSK", auth.SecretAccessKey)
+		}
+	}
+
+	_, ok = cfg.policyFor("NSAK", "predastore")
+	assert.False(t, ok, "northstar key must not reach the predastore bucket")
+}
+
+// Without a credential the northstar stanzas must be omitted wholesale rather
+// than half-rendered: an empty-keyed [[auth]] entry would let anyone read the
+// zone bucket.
+func TestPredastoreMultinodeTemplate_NorthstarOmittedWithoutCredentials(t *testing.T) {
+	content, err := admin.GenerateMultiNodePredastoreConfig(
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0,
+		admin.NorthstarCredentials{},
+	)
+	require.NoError(t, err)
+
+	var cfg predastoreBucketAuth
+	require.NoError(t, toml.Unmarshal([]byte(content), &cfg))
+
+	assert.False(t, cfg.hasBucket(admin.NorthstarBucketName))
+	_, ok := cfg.policyFor("AK", admin.NorthstarBucketName)
+	assert.False(t, ok, "system key granted northstar access without a northstar credential")
+	assert.Len(t, cfg.Auth, 1, "only the system key should be rendered")
+}
+
+// A joiner must provision its predastore with the leader's pair verbatim, since
+// each node's predastore only honours the keys in its own config. Per-node
+// generation would have node A's predastore reject node B's resolver.
+func TestNorthstarFromFormation_UsesDistributedPair(t *testing.T) {
+	dirs := configDirs{Northstar: "/etc/spinifex/northstar"}
+	creds := &formation.SharedCredentials{NorthstarAccessKey: "NSAK", NorthstarSecretKey: "NSSK"}
+
+	got, path := northstarFromFormation(creds, dirs)
+
+	assert.Equal(t, "NSAK", got.AccessKey)
+	assert.Equal(t, "NSSK", got.SecretKey)
+	assert.Equal(t, admin.NorthstarBucketName, got.Bucket, "bucket is a constant, derived node-side")
+	assert.Equal(t, "/etc/spinifex/northstar/northstar.toml", path)
+}
+
+// A leader predating credential distribution, or returning only half a pair,
+// must yield no Northstar config. Advertising a resolver without usable keys
+// would send guests to a service that cannot read its zones.
+func TestNorthstarFromFormation_IncompletePairYieldsNoConfig(t *testing.T) {
+	tests := []struct {
+		name   string
+		access string
+		secret string
+	}{
+		{name: "legacy leader"},
+		{name: "access key only", access: "NSAK"},
+		{name: "secret key only", secret: "NSSK"},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got, path := northstarFromFormation(&formation.SharedCredentials{
+				NorthstarAccessKey: tt.access,
+				NorthstarSecretKey: tt.secret,
+			}, configDirs{Northstar: "/etc/spinifex/northstar"})
+
+			assert.Equal(t, admin.NorthstarCredentials{}, got)
+			assert.Empty(t, path, "config path without credentials advertises a resolver that cannot read its zones")
+		})
+	}
+}
+
+// multinodeNorthstarSettings mirrors what both formation paths hand
+// generateAndWriteConfigs once the distributed pair is wired through.
+func multinodeNorthstarSettings(configPath string) admin.ConfigSettings {
+	s := northstarSettings()
+	s.Node = "node1"
+	s.Az = "ap-southeast-2a"
+	s.Port = "4432"
+	s.AccountID = "123456789012"
+	s.OVNNBAddr = "tcp:127.0.0.1:6641"
+	s.OVNSBAddr = "tcp:127.0.0.1:6642"
+	s.NorthstarConfigPath = configPath
+	return s
+}
+
+// The multi-node formation paths must produce both halves of a working
+// resolver: the northstar.toml the service reads, and the spinifex.toml stanza
+// that makes configuredNorthstarNodeNames count this node. Rendering was
+// previously reachable only from the single-node path.
+func TestGenerateAndWriteConfigs_MultinodeRendersNorthstar(t *testing.T) {
+	dirs, err := createConfigSubdirs(t.TempDir())
+	require.NoError(t, err)
+	nsPath := filepath.Join(dirs.Northstar, "northstar.toml")
+	spinifexTomlPath := filepath.Join(t.TempDir(), "spinifex.toml")
+
+	require.NoError(t, generateAndWriteConfigs(dirs, spinifexTomlPath, multinodeNorthstarSettings(nsPath), true))
+
+	nsContent, err := os.ReadFile(nsPath)
+	require.NoError(t, err)
+	assert.Contains(t, string(nsContent), `access_key = "AKIANORTHSTAR"`)
+
+	// config_path is the sole predicate for "this node runs northstar", so its
+	// presence is what puts the node in the nameserver set and the resolver list.
+	cfg, err := config.LoadConfig(spinifexTomlPath)
+	require.NoError(t, err)
+	assert.Equal(t, nsPath, cfg.Nodes["node1"].Northstar.ConfigPath)
+}
+
+// Incomplete credentials must yield neither file nor stanza. A stanza without
+// a usable northstar.toml would point guests at a resolver that never starts.
+func TestGenerateAndWriteConfigs_NoNorthstarWithoutCompleteCredentials(t *testing.T) {
+	tests := []struct {
+		name   string
+		access string
+		secret string
+	}{
+		{name: "no credentials"},
+		{name: "access key only", access: "AKIANORTHSTAR"},
+		{name: "secret key only", secret: "northstar-secret"},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			dirs, err := createConfigSubdirs(t.TempDir())
+			require.NoError(t, err)
+			spinifexTomlPath := filepath.Join(t.TempDir(), "spinifex.toml")
+
+			settings := multinodeNorthstarSettings(filepath.Join(dirs.Northstar, "northstar.toml"))
+			settings.NorthstarAccessKey = tt.access
+			settings.NorthstarSecretKey = tt.secret
+			require.NoError(t, generateAndWriteConfigs(dirs, spinifexTomlPath, settings, false))
+
+			_, err = os.Stat(filepath.Join(dirs.Northstar, "northstar.toml"))
+			assert.True(t, os.IsNotExist(err), "northstar.toml rendered without complete credentials")
+
+			cfg, err := config.LoadConfig(spinifexTomlPath)
+			require.NoError(t, err)
+			assert.Empty(t, cfg.Nodes["node1"].Northstar.ConfigPath)
+
+			predastoreContent, err := os.ReadFile(filepath.Join(dirs.Predastore, "predastore.toml"))
+			require.NoError(t, err)
+			var predastoreCfg predastoreBucketAuth
+			require.NoError(t, toml.Unmarshal(predastoreContent, &predastoreCfg))
+			assert.False(t, predastoreCfg.hasBucket(admin.NorthstarBucketName))
+			assert.Len(t, predastoreCfg.Auth, 1, "incomplete pair rendered a Northstar auth entry")
+		})
+	}
+}
+
+// renderClusterNode renders one node's spinifex.toml as its own formation path
+// would: its local northstar stanza plus a stanza for every peer.
+func renderClusterNode(t *testing.T, node string, allNodes map[string]formation.NodeInfo) *config.ClusterConfig {
+	t.Helper()
+	const nsPath = "/etc/spinifex/northstar/northstar.toml"
+
+	settings := multinodeNorthstarSettings(nsPath)
+	settings.Node = node
+	settings.BindIP = allNodes[node].BindIP
+	settings.AdvertiseIP = allNodes[node].AdvertiseIP
+	settings.RemoteNodes = buildRemoteNodes(allNodes, node, nsPath)
+
+	path := filepath.Join(t.TempDir(), "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	cfg, err := config.LoadConfig(path)
+	require.NoError(t, err)
+	return cfg
+}
+
+// Every node must derive the same nameserver set from its own rendered config.
+// EnsureBaseZone is create-if-absent and never overwrites, so if each node saw
+// only itself the zone's NS records would be pinned to whichever node won the
+// startup race — permanently, for the life of the zone object. Identical sets
+// make that race benign: whoever writes, writes the same bytes.
+func TestSpinifexTomlTemplate_PeerNorthstarSeedsAreUniform(t *testing.T) {
+	allNodes := map[string]formation.NodeInfo{
+		"node1": {Name: "node1", BindIP: "10.0.0.1", AdvertiseIP: "192.168.1.31"},
+		"node2": {Name: "node2", BindIP: "10.0.0.2", AdvertiseIP: "192.168.1.32"},
+		"node3": {Name: "node3", BindIP: "10.0.0.3", AdvertiseIP: "192.168.1.33"},
+	}
+
+	wantIPs := []string{"192.168.1.31", "192.168.1.32", "192.168.1.33"}
+	wantHosts := []string{"ns1", "ns2", "ns3"}
+	for _, node := range []string{"node1", "node2", "node3"} {
+		cfg := renderClusterNode(t, node, allNodes)
+		seeds := handlers_dns.NameserverSeeds(cfg)
+		require.Len(t, seeds, 3)
+		for i, seed := range seeds {
+			assert.Equal(t, wantHosts[i], seed.Host)
+			assert.Equal(t, wantIPs[i], seed.IP)
+		}
+		assert.Equal(t, wantIPs, handlers_dns.ResolverNameserverIPs(cfg),
+			"%s derived a different nameserver set from its own config", node)
+	}
+}
+
+// A peer's stanza carries config_path alone: it is the only field ever read back
+// from a peer, and the domains would be fiction — the local node cannot know
+// what a peer configured.
+func TestSpinifexTomlTemplate_PeerNorthstarStanzaIsPathOnly(t *testing.T) {
+	allNodes := map[string]formation.NodeInfo{
+		"node1": {Name: "node1", BindIP: "10.0.0.1", AdvertiseIP: "192.168.1.31"},
+		"node2": {Name: "node2", BindIP: "10.0.0.2", AdvertiseIP: "192.168.1.32"},
+	}
+
+	cfg := renderClusterNode(t, "node1", allNodes)
+	peer := cfg.Nodes["node2"].Northstar
+	assert.Equal(t, "/etc/spinifex/northstar/northstar.toml", peer.ConfigPath)
+	assert.Empty(t, peer.DefaultDomain, "peer domains are unknowable from here")
+	assert.Empty(t, peer.InternalDomain)
+}
+
 // Unset knob must leave production config byte-identical: no [compaction] block
 // and the original trailing bytes unchanged (single-node has no trailing
 // newline, multinode keeps one).
@@ -644,7 +1025,7 @@ func TestPredastoreTemplates_UnsetCompactionEmitsNoBlock(t *testing.T) {
 		"unset single-node tail changed: %q", single[len(single)-40:])
 
 	multi, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0,
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0, admin.NorthstarCredentials{},
 	)
 	require.NoError(t, err)
 	assert.NotContains(t, multi, "[compaction]")
@@ -665,7 +1046,7 @@ func TestPredastoreTemplates_SetCompactionEmitsParseableBlock(t *testing.T) {
 	assert.Equal(t, interval, singleCfg.Compaction.IntervalSeconds)
 
 	multi, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", interval,
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", interval, admin.NorthstarCredentials{},
 	)
 	require.NoError(t, err)
 	var multiCfg compactionConfig
@@ -702,3 +1083,169 @@ func TestAdminInitFlag_CompactionIntervalReachesRenderedConfig(t *testing.T) {
 	})
 	assert.Contains(t, out, "interval_seconds = 45")
 }
+
+func TestFilterDNSServersExcludesLocalNorthstar(t *testing.T) {
+	servers := []string{
+		"10.0.0.5",
+		"127.0.0.53",
+		"1.1.1.1",
+		"1.1.1.1",
+		"not-an-ip",
+		"2001:0db8::1",
+		"8.8.8.8",
+		"9.9.9.9",
+		"4.4.4.4",
+	}
+
+	got := filterDNSServers(servers, "10.0.0.5", "2001:db8::1")
+	assert.Equal(t, []string{"1.1.1.1", "8.8.8.8", "9.9.9.9"}, got)
+}
+
+func TestDetectDNSServersContinuesAfterExcludedSource(t *testing.T) {
+	var queries []string
+	sources := dnsDetectionSources{
+		queryResolvectl: func(args ...string) (string, error) {
+			query := strings.Join(args, " ")
+			queries = append(queries, query)
+			switch query {
+			case "dns br-wan":
+				return "Link 2 (br-wan): 10.0.0.5 127.0.0.53\n", nil
+			case "dns":
+				return "Global: 10.0.0.5 10.0.0.5 127.0.0.53 10.20.0.53 8.8.8.8\n", nil
+			default:
+				return "", fmt.Errorf("unexpected query %q", query)
+			}
+		},
+		readResolvConf: func() (string, error) {
+			t.Fatal("resolv.conf read despite usable global resolvers")
+			return "", nil
+		},
+	}
+
+	got := detectDNSServersWithSources("br-wan", []string{"10.0.0.5"}, sources)
+	assert.Equal(t, []string{"10.20.0.53", "8.8.8.8"}, got)
+	assert.Equal(t, []string{"dns br-wan", "dns"}, queries)
+}
+
+func TestDetectDNSServersFallsBackToResolvConf(t *testing.T) {
+	sources := dnsDetectionSources{
+		queryResolvectl: func(_ ...string) (string, error) {
+			return "", errors.New("resolver manager unavailable")
+		},
+		readResolvConf: func() (string, error) {
+			return "nameserver 10.0.0.5\nnameserver 10.20.0.53\n", nil
+		},
+	}
+
+	got := detectDNSServersWithSources("br-wan", []string{"10.0.0.5"}, sources)
+	assert.Equal(t, []string{"10.20.0.53"}, got)
+}
+
+func TestHostDNSParams(t *testing.T) {
+	// northstarEnabled requires both credentials; the zones and IPs mirror what
+	// admin init/join hold in memory after rendering northstar.toml.
+	enabled := admin.ConfigSettings{
+		NorthstarAccessKey:      "AK",
+		NorthstarSecretKey:      "SK",
+		NorthstarDefaultDomain:  "spx3.net",
+		NorthstarInternalDomain: "compute.internal",
+		AdvertiseIP:             "10.0.0.5",
+		BindIP:                  "0.0.0.0",
+	}
+
+	tests := []struct {
+		name       string
+		settings   admin.ConfigSettings
+		skip       bool
+		wantOK     bool
+		wantParams hostdns.Params
+	}{
+		{
+			name:     "enabled selects AdvertiseIP over BindIP",
+			settings: enabled,
+			wantOK:   true,
+			wantParams: hostdns.Params{
+				ResolverIP:     "10.0.0.5",
+				BaseDomain:     "spx3.net",
+				InternalDomain: "compute.internal",
+			},
+		},
+		{
+			name: "falls back to BindIP when AdvertiseIP is unset",
+			settings: func() admin.ConfigSettings {
+				s := enabled
+				s.AdvertiseIP = ""
+				s.BindIP = "10.0.0.9"
+				return s
+			}(),
+			wantOK: true,
+			wantParams: hostdns.Params{
+				ResolverIP:     "10.0.0.9",
+				BaseDomain:     "spx3.net",
+				InternalDomain: "compute.internal",
+			},
+		},
+		{
+			name:     "opt-out skips even when enabled",
+			settings: enabled,
+			skip:     true,
+			wantOK:   false,
+		},
+		{
+			name: "northstar disabled skips",
+			settings: func() admin.ConfigSettings {
+				s := enabled
+				s.NorthstarSecretKey = ""
+				return s
+			}(),
+			wantOK: false,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got, ok := hostDNSParams(tt.settings, tt.skip)
+			assert.Equal(t, tt.wantOK, ok)
+			if tt.wantOK {
+				assert.Equal(t, tt.wantParams, got)
+			}
+		})
+	}
+}
+
+// The image is written into a root volume of exactly the size this returns, so
+// anything less than the image size truncates the image and the guest never
+// finds its root. The old flooring conversion undersized every non-round image.
+func TestAMIVolumeSizeGiB(t *testing.T) {
+	const giB int64 = 1024 * 1024 * 1024
+
+	tests := []struct {
+		name  string
+		bytes int64
+		want  uint64
+	}{
+		// The regression: a real 4.94 GiB mkosi image floored to 4 GiB and the
+		// guest hung with no root partition.
+		{name: "non-round image rounds up", bytes: 5306470400, want: 5},
+		// Exact multiples must not gain a spare GiB — the round sizes are the
+		// common case and were the only ones the old code got right.
+		{name: "exact multiple is unchanged", bytes: 16 * giB, want: 16},
+		{name: "one byte over a multiple rounds up", bytes: 16*giB + 1, want: 17},
+		{name: "one byte under a multiple rounds up", bytes: 16*giB - 1, want: 16},
+		// Sub-GiB images still need a whole GiB to live in.
+		{name: "sub-GiB image gets a full GiB", bytes: 1, want: 1},
+		{name: "empty image", bytes: 0, want: 0},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := amiVolumeSizeGiB(tt.bytes)
+			assert.Equal(t, tt.want, got)
+			// The invariant the guest depends on, stated directly.
+			if tt.bytes > 0 {
+				assert.GreaterOrEqual(t, int64(got)*giB, tt.bytes,
+					"volume must be large enough to hold the image")
+			}
+		})
+	}
+}

cmd/spinifex/cmd/northstar_template_test.go

@@ -0,0 +1,89 @@
+package cmd
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/admin"
+	toml "github.com/pelletier/go-toml/v2"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func renderTemplate(t *testing.T, tmpl string, settings admin.ConfigSettings) string {
+	t.Helper()
+	path := filepath.Join(t.TempDir(), "out.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, tmpl, settings))
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	var parsed map[string]any
+	require.NoError(t, toml.Unmarshal(data, &parsed), "rendered template must be valid TOML")
+	return string(data)
+}
+
+func northstarSettings() admin.ConfigSettings {
+	return admin.ConfigSettings{
+		Region:                  "ap-southeast-2",
+		BindIP:                  "0.0.0.0",
+		AdvertiseIP:             "192.168.1.31",
+		AccessKey:               "AKIASYSTEM",
+		SecretKey:               "SYSTEMSECRET",
+		NatsToken:               "NATSTOKEN",
+		PoolDNSServers:          []string{"1.1.1.1", "8.8.8.8"},
+		NorthstarAccessKey:      "AKIANORTHSTAR",
+		NorthstarSecretKey:      "NORTHSTARSECRET",
+		NorthstarBucket:         admin.NorthstarBucketName,
+		NorthstarDefaultDomain:  admin.NorthstarDefaultDomain,
+		NorthstarInternalDomain: admin.NorthstarInternalDomain,
+	}
+}
+
+func TestNorthstarTomlTemplate(t *testing.T) {
+	content := renderTemplate(t, northstarTomlTemplate, northstarSettings())
+
+	// Northstar binds :5300 on the wildcard and the privileged :53 on the node
+	// WAN IP (avoids the systemd-resolved stub on 127.0.0.53:53).
+	assert.Contains(t, content, `listen     = "0.0.0.0:5300,192.168.1.31:53"`)
+	assert.Contains(t, content, `default_domain = "spx3.net"`)
+	assert.Contains(t, content, `bucket     = "northstar"`)
+	assert.Contains(t, content, `access_key = "AKIANORTHSTAR"`)
+	assert.Contains(t, content, `secret_key = "NORTHSTARSECRET"`)
+	// 0.0.0.0 bind must resolve to a loopback S3 endpoint, not a dial to 0.0.0.0.
+	assert.Contains(t, content, `endpoint   = "https://127.0.0.1:8443"`)
+	assert.Contains(t, content, `nameservers = ["1.1.1.1:53", "8.8.8.8:53"]`)
+	assert.Contains(t, content, `internal_domain = "compute.internal"`)
+	var parsed struct {
+		Quotas struct {
+			Enabled              bool `toml:"enabled"`
+			RecordsPerHostedZone int  `toml:"records_per_hosted_zone"`
+		} `toml:"quotas"`
+	}
+	require.NoError(t, toml.Unmarshal([]byte(content), &parsed))
+	assert.False(t, parsed.Quotas.Enabled)
+	assert.Equal(t, 10_000, parsed.Quotas.RecordsPerHostedZone)
+	assert.NotContains(t, content, "nats_url")
+	assert.NotContains(t, content, "NATSTOKEN")
+}
+
+// When provisioned, predastore.toml gains the northstar bucket plus a scoped
+// read-only [[auth]] entry — the isolation guarantee.
+func TestPredastoreTemplateWithNorthstar(t *testing.T) {
+	content := renderTemplate(t, predastoreTomlTemplate, northstarSettings())
+
+	assert.Contains(t, content, `name = "northstar"`)
+	assert.Contains(t, content, `access_key_id = "AKIANORTHSTAR"`)
+	assert.Contains(t, content, `{ bucket = "northstar", actions = ["s3:ListBucket", "s3:GetObject"] }`)
+}
+
+// Without northstar creds (e.g. cluster-join paths), predastore.toml renders
+// exactly as before — no empty bucket/auth stanzas.
+func TestPredastoreTemplateWithoutNorthstar(t *testing.T) {
+	s := northstarSettings()
+	s.NorthstarAccessKey = ""
+	s.NorthstarSecretKey = ""
+	content := renderTemplate(t, predastoreTomlTemplate, s)
+
+	assert.NotContains(t, content, `name = "northstar"`)
+	assert.NotContains(t, content, "AKIANORTHSTAR")
+}

cmd/spinifex/cmd/paths.go

@@ -8,7 +8,7 @@ import (
 
 // DefaultConfigDir returns the default configuration directory.
 // Production: /etc/spinifex (when /etc/spinifex exists)
-// Development: ~/spinifex/config
+// Development: ~/spinifex/config.
 func DefaultConfigDir() string {
 	if isProductionLayout() {
 		return "/etc/spinifex"
@@ -18,7 +18,7 @@ func DefaultConfigDir() string {
 
 // DefaultDataDir returns the default data directory.
 // Production: /var/lib/spinifex
-// Development: ~/spinifex
+// Development: ~/spinifex.
 func DefaultDataDir() string {
 	if isProductionLayout() {
 		return "/var/lib/spinifex"
@@ -28,7 +28,7 @@ func DefaultDataDir() string {
 
 // LogDirFor returns the log directory for a given data directory.
 // Production: /var/log/spinifex (matches systemd ReadWritePaths)
-// Development: <dataDir>/logs (supports custom per-node data dirs)
+// Development: <dataDir>/logs (supports custom per-node data dirs).
 func LogDirFor(dataDir string) string {
 	if isProductionLayout() {
 		return "/var/log/spinifex"

cmd/spinifex/cmd/root.go

@@ -15,7 +15,7 @@ var (
 	appConfig *config.ClusterConfig
 )
 
-// rootCmd represents the base command when called without any subcommands
+// rootCmd represents the base command when called without any subcommands.
 var rootCmd = &cobra.Command{
 	Use:   "spx",
 	Short: "Spinifex - Open source AWS-compatible platform for secure edge deployments",

cmd/spinifex/cmd/service.go

@@ -8,11 +8,13 @@ import (
 	"strings"
 
 	"github.com/mulgadc/spinifex/spinifex/config"
+	handlers_dns "github.com/mulgadc/spinifex/spinifex/handlers/dns"
 	"github.com/mulgadc/spinifex/spinifex/network/external"
 	"github.com/mulgadc/spinifex/spinifex/otelsetup"
 	"github.com/mulgadc/spinifex/spinifex/service"
 	"github.com/mulgadc/spinifex/spinifex/services/awsgw"
 	"github.com/mulgadc/spinifex/spinifex/services/nats"
+	"github.com/mulgadc/spinifex/spinifex/services/northstar"
 	"github.com/mulgadc/spinifex/spinifex/services/predastore"
 	"github.com/mulgadc/spinifex/spinifex/services/qmpcollector"
 	"github.com/mulgadc/spinifex/spinifex/services/spinifexui"
@@ -201,7 +203,7 @@ var predastoreStatusCmd = &cobra.Command{
 	},
 }
 
-// Repeat for viperblock
+// Repeat for viperblock.
 var viperblockStartCmd = &cobra.Command{
 	Use:   "start",
 	Short: "Start the viperblock service",
@@ -295,6 +297,14 @@ var viperblockStartCmd = &cobra.Command{
 			shardWAL = *nodeConfig.Viperblock.ShardWAL
 		}
 
+		// Resolve chunk GC setting: default false unless explicitly set to true.
+		// GC is not a per-volume toggle — flipping it on applies to every VB this
+		// node constructs (nbdkit plugin + volume service).
+		gcEnabled := false
+		if nodeConfig.Viperblock.GCEnabled != nil {
+			gcEnabled = *nodeConfig.Viperblock.GCEnabled
+		}
+
 		encryptionKeyFile := nodeConfig.Viperblock.EncryptionKeyFile
 		if envKey := viper.GetString("viperblock-encryption-key-file"); envKey != "" {
 			encryptionKeyFile = envKey
@@ -315,6 +325,7 @@ var viperblockStartCmd = &cobra.Command{
 			BaseDir:           nodeConfig.Predastore.BaseDir,
 			NodeName:          clusterConfig.Node,
 			ShardWAL:          shardWAL,
+			GCEnabled:         gcEnabled,
 			EncryptionKeyFile: encryptionKeyFile,
 		})
 
@@ -365,7 +376,7 @@ var viperblockStatusCmd = &cobra.Command{
 	},
 }
 
-// Repeat for nats
+// Repeat for nats.
 var natsStartCmd = &cobra.Command{
 	Use:   "start",
 	Short: "Start the nats service",
@@ -434,7 +445,7 @@ var natsStatusCmd = &cobra.Command{
 	},
 }
 
-// Repeat for spinifex
+// Repeat for spinifex.
 var spinifexStartCmd = &cobra.Command{
 	Use:   "start",
 	Short: "Start the spinifex service",
@@ -793,20 +804,23 @@ var vpcdStartCmd = &cobra.Command{
 		defer initTelemetry("vpcd", false)()
 
 		svc, err := service.New("vpcd", &vpcd.Config{
-			NatsHost:          nodeConfig.NATS.Host,
-			NatsToken:         nodeConfig.NATS.ACL.Token,
-			NatsCACert:        nodeConfig.NATS.CACert,
-			OVNNBAddr:         nodeConfig.VPCD.OVNNBAddr,
-			OVNSBAddr:         nodeConfig.VPCD.OVNSBAddr,
-			BaseDir:           nodeConfig.BaseDir,
-			Debug:             false,
-			ExternalMode:      clusterConfig.Network.ExternalMode,
-			ExternalPools:     extPools,
-			Bootstrap:         bootstrap,
-			ExternalInterface: nodeConfig.VPCD.ExternalInterface,
-			BridgeMode:        nodeConfig.VPCD.BridgeMode,
-			AZ:                nodeConfig.AZ,
-			NATExemptCIDRs:    clusterConfig.Network.NATExemptCIDRs,
+			NatsHost:                nodeConfig.NATS.Host,
+			NatsToken:               nodeConfig.NATS.ACL.Token,
+			NatsCACert:              nodeConfig.NATS.CACert,
+			OVNNBAddr:               nodeConfig.VPCD.OVNNBAddr,
+			OVNSBAddr:               nodeConfig.VPCD.OVNSBAddr,
+			BaseDir:                 nodeConfig.BaseDir,
+			Debug:                   false,
+			ExternalMode:            clusterConfig.Network.ExternalMode,
+			ExternalPools:           extPools,
+			Bootstrap:               bootstrap,
+			ExternalInterface:       nodeConfig.VPCD.ExternalInterface,
+			BridgeMode:              nodeConfig.VPCD.BridgeMode,
+			AZ:                      nodeConfig.AZ,
+			NorthstarBaseDomain:     handlers_dns.ResolveBaseDomain(&nodeConfig),
+			NorthstarInternalDomain: handlers_dns.ResolveInternalDomain(&nodeConfig),
+			ResolverNameservers:     handlers_dns.ResolverNameserverIPs(clusterConfig),
+			NATExemptCIDRs:          clusterConfig.Network.NATExemptCIDRs,
 		})
 		if err != nil {
 			fmt.Println("Error starting vpcd service:", err)
@@ -849,6 +863,157 @@ var vpcdStatusCmd = &cobra.Command{
 	},
 }
 
+var northstarCmd = &cobra.Command{
+	Use:   "northstar",
+	Short: "Manage the northstar (DNS) service",
+}
+
+// northstarStartOptions contains the command inputs that select Northstar configuration.
+type northstarStartOptions struct {
+	configFile      string
+	configOverride  string
+	baseDirOverride string
+}
+
+// northstarStarter is the service operation required by Northstar activation.
+type northstarStarter interface {
+	Start() (int, error)
+}
+
+// northstarStartDependencies isolates startup side effects for behavioral tests.
+type northstarStartDependencies struct {
+	loadConfig        func(string) (*config.ClusterConfig, error)
+	bootstrapBaseZone func(string, *config.ClusterConfig) error
+	newService        func(*northstar.Config) (northstarStarter, error)
+}
+
+var northstarStartCmd = &cobra.Command{
+	Use:           "start",
+	Short:         "Start the northstar service",
+	SilenceErrors: true,
+	SilenceUsage:  true,
+	RunE: func(cmd *cobra.Command, args []string) error {
+		fmt.Println("Starting northstar service...")
+
+		defer initTelemetry("northstar", false)()
+
+		return runNorthstarStart(northstarStartOptions{
+			configFile:      viper.GetString("config"),
+			configOverride:  viper.GetString("northstar-config"),
+			baseDirOverride: viper.GetString("base-dir"),
+		}, northstarStartDependencies{
+			loadConfig:        loadRequiredClusterConfig,
+			bootstrapBaseZone: northstar.BootstrapBaseZone,
+			newService: func(cfg *northstar.Config) (northstarStarter, error) {
+				return service.New("northstar", cfg)
+			},
+		})
+	},
+}
+
+// loadRequiredClusterConfig loads the service's explicit cluster configuration.
+func loadRequiredClusterConfig(path string) (*config.ClusterConfig, error) {
+	if path == "" {
+		return nil, fmt.Errorf("config file is not set")
+	}
+	if _, err := os.Stat(path); err != nil {
+		return nil, fmt.Errorf("stat cluster config %s: %w", path, err)
+	}
+	clusterConfig, err := config.LoadConfig(path)
+	if err != nil {
+		return nil, fmt.Errorf("load cluster config %s: %w", path, err)
+	}
+	return clusterConfig, nil
+}
+
+// runNorthstarStart starts configured Northstar nodes and cleanly skips nodes where it is optional.
+func runNorthstarStart(options northstarStartOptions, deps northstarStartDependencies) error {
+	clusterConfig, err := deps.loadConfig(options.configFile)
+	if err != nil {
+		return err
+	}
+
+	nodeConfig, ok := clusterConfig.Nodes[clusterConfig.Node]
+	if !ok {
+		return fmt.Errorf("node %q not found in cluster config", clusterConfig.Node)
+	}
+
+	configPath := resolveNorthstarConfigPath(nodeConfig.Northstar.ConfigPath, options.configOverride)
+	if options.configOverride != "" {
+		fmt.Println("Overwriting northstar config path to:", options.configOverride)
+	}
+	if configPath == "" {
+		// An absent path disables this optional service and must not trigger systemd restarts.
+		slog.Info("northstar is not configured on this node; skipping service start")
+		return nil
+	}
+
+	baseDir := nodeConfig.BaseDir
+	if options.baseDirOverride != "" {
+		baseDir = options.baseDirOverride
+	}
+
+	// Seed the default_domain base zone before the read-only daemon starts.
+	// The S3 polling path remains the backstop when this best-effort seed fails.
+	if err := deps.bootstrapBaseZone(configPath, clusterConfig); err != nil {
+		slog.Warn("northstar base zone bootstrap failed (continuing)", "error", err)
+	}
+
+	svc, err := deps.newService(&northstar.Config{
+		ConfigPath: configPath,
+		BasePath:   baseDir,
+		NodeID:     nodeConfig.Predastore.NodeID,
+		NatsHost:   nodeConfig.NATS.Host,
+		NatsToken:  nodeConfig.NATS.ACL.Token,
+		NatsCACert: nodeConfig.NATS.CACert,
+	})
+	if err != nil {
+		return fmt.Errorf("create northstar service: %w", err)
+	}
+
+	if _, err := svc.Start(); err != nil {
+		return fmt.Errorf("start northstar service: %w", err)
+	}
+	fmt.Println("northstar service started")
+	return nil
+}
+
+// resolveNorthstarConfigPath applies the explicit override before the node config.
+func resolveNorthstarConfigPath(nodePath, override string) string {
+	if override != "" {
+		return override
+	}
+	return nodePath
+}
+
+var northstarStopCmd = &cobra.Command{
+	Use:   "stop",
+	Short: "Stop the northstar service",
+	Run: func(cmd *cobra.Command, args []string) {
+		fmt.Println("Stopping northstar service...")
+
+		svc, err := service.New("northstar", &northstar.Config{})
+		if err != nil {
+			fmt.Println("Error stopping northstar service:", err)
+			return
+		}
+
+		if err = svc.Stop(); err != nil {
+			fmt.Println("Error stopping northstar service:", err)
+			os.Exit(1)
+		}
+		fmt.Println("northstar service stopped")
+	},
+}
+
+var northstarStatusCmd = &cobra.Command{
+	Use:   "status",
+	Short: "Get status of the northstar service",
+	Run: func(cmd *cobra.Command, args []string) {
+		fmt.Println("northstar service status: ...")
+	},
+}
+
 var qmpCollectorCmd = &cobra.Command{
 	Use:   "qmp-collector",
 	Short: "Manage the qmp-collector (guest metrics) service",
@@ -1115,6 +1280,17 @@ func init() {
 	vpcdCmd.AddCommand(vpcdStopCmd)
 	vpcdCmd.AddCommand(vpcdStatusCmd)
 
+	// northstar
+	serviceCmd.AddCommand(northstarCmd)
+
+	northstarCmd.PersistentFlags().String("northstar-config", "", "Path to northstar.toml (overrides config file)")
+	viper.BindEnv("northstar-config", "SPINIFEX_NORTHSTAR_CONFIG")
+	viper.BindPFlag("northstar-config", northstarCmd.PersistentFlags().Lookup("northstar-config"))
+
+	northstarCmd.AddCommand(northstarStartCmd)
+	northstarCmd.AddCommand(northstarStopCmd)
+	northstarCmd.AddCommand(northstarStatusCmd)
+
 	// qmp-collector
 	serviceCmd.AddCommand(qmpCollectorCmd)
 

cmd/spinifex/cmd/service_test.go

@@ -0,0 +1,207 @@
+package cmd
+
+import (
+	"errors"
+	"testing"
+
+	"github.com/mulgadc/spinifex/spinifex/config"
+	"github.com/mulgadc/spinifex/spinifex/services/northstar"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+type stubNorthstarService struct {
+	starts   int
+	startErr error
+}
+
+func (s *stubNorthstarService) Start() (int, error) {
+	s.starts++
+	return 0, s.startErr
+}
+
+func TestRunNorthstarStartActivation(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name           string
+		nodePath       string
+		override       string
+		wantPath       string
+		wantConfigured bool
+	}{
+		{
+			name: "unconfigured node is skipped",
+		},
+		{
+			name:           "node config starts service",
+			nodePath:       "/etc/spinifex/northstar/northstar.toml",
+			wantPath:       "/etc/spinifex/northstar/northstar.toml",
+			wantConfigured: true,
+		},
+		{
+			name:           "override starts service",
+			override:       "/run/spinifex/northstar.toml",
+			wantPath:       "/run/spinifex/northstar.toml",
+			wantConfigured: true,
+		},
+		{
+			name:           "override takes precedence",
+			nodePath:       "/etc/spinifex/northstar/northstar.toml",
+			override:       "/run/spinifex/northstar.toml",
+			wantPath:       "/run/spinifex/northstar.toml",
+			wantConfigured: true,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			clusterConfig := &config.ClusterConfig{
+				Node: "node1",
+				Nodes: map[string]config.Config{
+					"node1": {
+						BaseDir:   "/var/lib/spinifex/northstar",
+						Northstar: config.NorthstarConfig{ConfigPath: tt.nodePath},
+					},
+				},
+			}
+			stub := &stubNorthstarService{}
+			bootstrapCalls := 0
+			factoryCalls := 0
+			var startedConfig *northstar.Config
+			deps := northstarStartDependencies{
+				loadConfig: func(string) (*config.ClusterConfig, error) {
+					return clusterConfig, nil
+				},
+				bootstrapBaseZone: func(path string, cfg *config.ClusterConfig) error {
+					bootstrapCalls++
+					assert.Equal(t, tt.wantPath, path)
+					assert.Same(t, clusterConfig, cfg)
+					return nil
+				},
+				newService: func(cfg *northstar.Config) (northstarStarter, error) {
+					factoryCalls++
+					startedConfig = cfg
+					return stub, nil
+				},
+			}
+
+			err := runNorthstarStart(northstarStartOptions{
+				configFile:     "/etc/spinifex/spinifex.toml",
+				configOverride: tt.override,
+			}, deps)
+			require.NoError(t, err)
+
+			if !tt.wantConfigured {
+				assert.Zero(t, bootstrapCalls)
+				assert.Zero(t, factoryCalls)
+				assert.Zero(t, stub.starts)
+				assert.Nil(t, startedConfig)
+				return
+			}
+			assert.Equal(t, 1, bootstrapCalls)
+			assert.Equal(t, 1, factoryCalls)
+			assert.Equal(t, 1, stub.starts)
+			require.NotNil(t, startedConfig)
+			assert.Equal(t, tt.wantPath, startedConfig.ConfigPath)
+		})
+	}
+}
+
+func TestLoadRequiredClusterConfigRejectsMissingFile(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name string
+		path string
+	}{
+		{name: "empty path"},
+		{name: "missing path", path: t.TempDir() + "/missing.toml"},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+			_, err := loadRequiredClusterConfig(tt.path)
+			require.Error(t, err)
+		})
+	}
+}
+
+func TestRunNorthstarStartReturnsLoadFailure(t *testing.T) {
+	t.Parallel()
+
+	loadErr := errors.New("cluster config unavailable")
+	deps := northstarStartDependencies{
+		loadConfig: func(string) (*config.ClusterConfig, error) {
+			return nil, loadErr
+		},
+	}
+
+	err := runNorthstarStart(northstarStartOptions{configFile: "/etc/spinifex/spinifex.toml"}, deps)
+	require.ErrorIs(t, err, loadErr)
+}
+
+func TestRunNorthstarStartContinuesAfterBootstrapFailure(t *testing.T) {
+	t.Parallel()
+
+	bootstrapErr := errors.New("predastore unavailable")
+	stub := &stubNorthstarService{}
+	deps := northstarStartDependencies{
+		loadConfig: func(string) (*config.ClusterConfig, error) {
+			return &config.ClusterConfig{
+				Node: "node1",
+				Nodes: map[string]config.Config{
+					"node1": {Northstar: config.NorthstarConfig{ConfigPath: "/etc/spinifex/northstar/northstar.toml"}},
+				},
+			}, nil
+		},
+		bootstrapBaseZone: func(string, *config.ClusterConfig) error { return bootstrapErr },
+		newService: func(*northstar.Config) (northstarStarter, error) {
+			return stub, nil
+		},
+	}
+
+	err := runNorthstarStart(northstarStartOptions{configFile: "/etc/spinifex/spinifex.toml"}, deps)
+	require.NoError(t, err)
+	assert.Equal(t, 1, stub.starts)
+}
+
+func TestRunNorthstarStartRejectsMissingNode(t *testing.T) {
+	t.Parallel()
+
+	deps := northstarStartDependencies{
+		loadConfig: func(string) (*config.ClusterConfig, error) {
+			return &config.ClusterConfig{Node: "missing", Nodes: map[string]config.Config{}}, nil
+		},
+	}
+
+	err := runNorthstarStart(northstarStartOptions{configFile: "/etc/spinifex/spinifex.toml"}, deps)
+	require.EqualError(t, err, `node "missing" not found in cluster config`)
+}
+
+func TestRunNorthstarStartReturnsConfiguredFailure(t *testing.T) {
+	t.Parallel()
+
+	startErr := errors.New("configured startup failed")
+	stub := &stubNorthstarService{startErr: startErr}
+	deps := northstarStartDependencies{
+		loadConfig: func(string) (*config.ClusterConfig, error) {
+			return &config.ClusterConfig{
+				Node: "node1",
+				Nodes: map[string]config.Config{
+					"node1": {Northstar: config.NorthstarConfig{ConfigPath: "/etc/spinifex/northstar/northstar.toml"}},
+				},
+			}, nil
+		},
+		bootstrapBaseZone: func(string, *config.ClusterConfig) error { return nil },
+		newService: func(*northstar.Config) (northstarStarter, error) {
+			return stub, nil
+		},
+	}
+
+	err := runNorthstarStart(northstarStartOptions{configFile: "/etc/spinifex/spinifex.toml"}, deps)
+	require.ErrorIs(t, err, startErr)
+	assert.Equal(t, 1, stub.starts)
+}

cmd/spinifex/cmd/templates/northstar.toml

@@ -0,0 +1,40 @@
+# Northstar DNS service configuration (TOML).
+# Written by `spx admin init`; read by the spinifex northstar service.
+# Contains bucket-scoped S3 credentials — keep mode 0600.
+
+# UDP+TCP bind addresses (comma separated). :5300 binds the wildcard for local
+# tooling; :53 binds the node WAN IP specifically (via CAP_NET_BIND_SERVICE) so
+# instances resolve against the node directly. Binding the WAN IP rather than
+# 0.0.0.0 avoids the systemd-resolved stub on 127.0.0.53:53. A future per-VPC
+# resolver VIP (OVN) will front :53.
+listen     = "0.0.0.0:5300,{{if .AdvertiseIP}}{{.AdvertiseIP}}{{else}}{{.BindIP}}{{end}}:53"
+dot_listen = ""
+doh_listen = ""
+tls_cert   = ""
+tls_key    = ""
+
+# Authoritative base domain for internal resource names.
+default_domain = "{{.NorthstarDefaultDomain}}"
+
+# AWS-parity private DNS zone for instance ip-<addr> records (IMDS hostname).
+internal_domain = "{{.NorthstarInternalDomain}}"
+
+# Seconds between S3 zone re-scans (anti-entropy; live changes arrive via NATS).
+sync_interval = 30
+
+# DNS (Route 53) service quotas (default off, mirrors [quota] in awsgw.toml).
+# Applied at daemon start; restart to change.
+[quotas]
+enabled = false
+records_per_hosted_zone = 10000   # AWS default; enforced only when enabled
+
+[s3]
+endpoint   = "https://{{if eq .BindIP "0.0.0.0"}}127.0.0.1{{else}}{{.BindIP}}{{end}}:8443"
+bucket     = "{{.NorthstarBucket}}"
+region     = "{{.Region}}"
+access_key = "{{.NorthstarAccessKey}}"
+secret_key = "{{.NorthstarSecretKey}}"
+
+# Forwarders for non-authoritative (recursive) queries. Empty => REFUSED.
+[upstream]
+nameservers = [{{range $i, $s := .PoolDNSServers}}{{if $i}}, {{end}}"{{$s}}:53"{{end}}]

cmd/spinifex/cmd/templates/predastore-multinode.toml

@@ -46,13 +46,40 @@ public = false
 encryption = ""
 account_id = "000000000000"
 
+{{- if .NorthstarAccessKey}}
+# Northstar DNS zone storage. Read-only for the northstar service; the control
+# plane writes zone files with the system credentials.
+[[buckets]]
+name = "{{.NorthstarBucket}}"
+region = "{{.Region}}"
+type = "distributed"
+public = false
+encryption = ""
+account_id = "000000000000"
+{{- end}}
+
 [[auth]]
 access_key_id = "{{.AccessKey}}"
 secret_access_key = "{{.SecretKey}}"
 account_id = "000000000000"
 policy = [
   { bucket = "predastore", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
+{{- if .NorthstarAccessKey}}
+  { bucket = "{{.NorthstarBucket}}", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"] },
+{{- end}}
 ]
+{{- if .NorthstarAccessKey}}
+
+# Bucket-scoped, read-only credentials for the northstar service (isolation
+# guarantee: ListBucket + GetObject on the northstar bucket only).
+[[auth]]
+access_key_id = "{{.NorthstarAccessKey}}"
+secret_access_key = "{{.NorthstarSecretKey}}"
+account_id = "000000000000"
+policy = [
+  { bucket = "{{.NorthstarBucket}}", actions = ["s3:ListBucket", "s3:GetObject"] },
+]
+{{- end}}
 
 # API request throttling (per-account, per-action token bucket)
 [ratelimit]

cmd/spinifex/cmd/templates/predastore.toml

@@ -75,13 +75,40 @@ public = false
 encryption = ""
 account_id = "000000000000"
 
+{{- if .NorthstarAccessKey}}
+# Northstar DNS zone storage. Read-only for the northstar service; the control
+# plane writes zone files with the system credentials.
+[[buckets]]
+name = "{{.NorthstarBucket}}"
+region = "{{.Region}}"
+type = "distributed"
+public = false
+encryption = ""
+account_id = "000000000000"
+{{- end}}
+
 [[auth]]
 access_key_id = "{{.AccessKey}}"
 secret_access_key = "{{.SecretKey}}"
 account_id = "000000000000"
 policy = [
   { bucket = "predastore", actions = ["s3:ListBucket",  "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
+{{- if .NorthstarAccessKey}}
+  { bucket = "{{.NorthstarBucket}}", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"] },
+{{- end}}
 ]
+{{- if .NorthstarAccessKey}}
+
+# Bucket-scoped, read-only credentials for the northstar service (isolation
+# guarantee: ListBucket + GetObject on the northstar bucket only).
+[[auth]]
+access_key_id = "{{.NorthstarAccessKey}}"
+secret_access_key = "{{.NorthstarSecretKey}}"
+account_id = "000000000000"
+policy = [
+  { bucket = "{{.NorthstarBucket}}", actions = ["s3:ListBucket", "s3:GetObject"] },
+]
+{{- end}}
 
 # API request throttling (per-account, per-action token bucket)
 [ratelimit]

cmd/spinifex/cmd/templates/spinifex.toml

@@ -92,7 +92,7 @@ gpu_passthrough = true
 host = "{{.BindIP}}:9999"
 tlskey = "config/server.key"
 tlscert = "config/server.pem"
-debug = true
+debug = false
 config = "awsgw/awsgw.toml"
 
 [nodes.{{.Node}}.nats]
@@ -106,6 +106,7 @@ token = "{{.NatsToken}}"
 
 [nodes.{{.Node}}.viperblock]
 shardwal = false
+gc_enabled = true
 {{- if .EncryptionKeyFile}}
 encryption_key_file = "{{.EncryptionKeyFile}}"
 {{- end}}
@@ -129,6 +130,14 @@ secretkey = "{{.SecretKey}}"
 {{- if gt .PredastoreNodeID 0}}
 node_id = {{.PredastoreNodeID}}
 {{- end}}
+
+{{- if .NorthstarConfigPath}}
+
+[nodes.{{.Node}}.northstar]
+config_path = "{{.NorthstarConfigPath}}"
+default_domain = "{{.NorthstarDefaultDomain}}"
+internal_domain = "{{.NorthstarInternalDomain}}"
+{{- end}}
 {{- range .RemoteNodes}}
 
 [nodes.{{.Name}}]
@@ -139,4 +148,9 @@ az = "{{.AZ}}"
 {{- if .Services}}
 services = [{{range $i, $s := .Services}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
 {{- end}}
+{{- if .NorthstarConfigPath}}
+
+[nodes.{{.Name}}.northstar]
+config_path = "{{.NorthstarConfigPath}}"
+{{- end}}
 {{- end}}

cmd/spinifex/cmd/version.go

@@ -8,7 +8,7 @@ import (
 )
 
 // Version variables set via ldflags at build time.
-// Example: go build -ldflags "-X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=v1.0.0"
+// Example: go build -ldflags "-X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=v1.0.0".
 var (
 	Version = "dev"
 	Commit  = "unknown"

docs/COMMANDS.md

@@ -31,6 +31,9 @@ Service lifecycle commands for starting, stopping, and checking status of all Sp
 | `spx service vpcd start` | — | Loads cluster config → starts VPC daemon (subscribes to `vpc.*` NATS events, translates to OVN logical switches/ports/routers) |
 | `spx service vpcd stop` | — | Stops the vpcd service |
 | `spx service vpcd status` | — | Reports vpcd service status |
+| `spx service northstar start` | `--northstar-config` (overrides `nodes.<node>.northstar.config_path`) | Loads cluster config → reads `northstar.toml` → starts the northstar DNS server (authoritative for internal `*.spx3.net`, recursive via upstream forwarders), syncing zones from its S3 bucket. Guests resolve via `169.254.169.253`, served by vpcd's per-tap DNS shim which forwards to northstar |
+| `spx service northstar stop` | — | Stops the northstar service |
+| `spx service northstar status` | — | Reports northstar service status |
 | `spx service qmp-collector start` | — | Starts the guest-metrics collector (polls per-VM telemetry QMP sockets + tap counters, publishes CloudWatch-shaped series to NATS `metrics.ec2.*`) |
 | `spx service qmp-collector stop` | — | Stops the qmp-collector service |
 | `spx service qmp-collector status` | — | Reports qmp-collector service status |
@@ -54,8 +57,8 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Flags | Description |
 |---------|-------|-------------|
-| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; add `--external-pool start-end --external-gateway <ip>` or `--external-source=dhcp [--external-bind-bridge <iface>]` for a public pool with full EIP support; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/`. `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
-| `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services` | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations |
+| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--skip-host-dns`, `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; add `--external-pool start-end --external-gateway <ip>` or `--external-source=dhcp [--external-bind-bridge <iface>]` for a public pool with full EIP support; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` → points the host resolver at this node's local northstar so the Spinifex zones (ELB/EKS names) resolve from the node itself (opt out with `--skip-host-dns`). `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
+| `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services`, `--skip-host-dns` | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations → points the host resolver at this node's local northstar so the Spinifex zones resolve from the node itself (opt out with `--skip-host-dns`) |
 
 ### Version
 

docs/install/install-airgapped/README.md

@@ -75,7 +75,7 @@ Mirror the cloud image you intend to run as guest VMs:
 
 ```bash
 mkdir -p images
-curl -fsSL "https://cloud.debian.org/images/cloud/trixie/20260518-2482/debian-13-genericcloud-amd64-20260518-2482.qcow2" \
+curl -fsSL "https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2" \
   -o images/debian-13-amd64.qcow2
 ```
 

docs/reference-architectures/RTX-Pro-6000-EKS/README.md

@@ -0,0 +1,453 @@
+---
+title: "EKS AI Platform on Bare Metal: llama.cpp + YOLO on Spinifex"
+description: "Deploy a GPU-accelerated AI inference platform — an OpenAI-compatible LLM API and a real-time CV stream — in Kubernetes on bare-metal hardware, managed entirely with standard AWS tooling."
+category: "Reference Architectures"
+tags:
+  - nvidia
+  - rtx-pro-6000
+  - eks
+  - kubernetes
+  - llama.cpp
+  - yolo
+  - ecr
+  - bare-metal
+  - terraform
+resources:
+  - title: "eks-ai-platform (Terraform workbook, Dockerfiles, scripts)"
+    url: "https://github.com/mulgadc/eks-ai-platform"
+  - title: "GPU Passthrough"
+    url: "/docs/gpu-passthrough"
+  - title: "Launching EKS Clusters"
+    url: "/docs/eks"
+  - title: "llama.cpp"
+    url: "https://github.com/ggml-org/llama.cpp"
+  - title: "Ultralytics YOLO"
+    url: "https://docs.ultralytics.com"
+sections:
+  - overview
+  - prerequisites
+  - instructions
+---
+
+## Overview
+
+Spinifex is an open-source infrastructure platform that brings core AWS services such as EC2, S3, EBS and EKS, to bare-metal, edge, and on-prem deployments. It exposes a fully AWS-compatible API, so any tooling that works against AWS — the `aws` CLI, OpenTofu, `kubectl` — works against a Spinifex node unchanged, with a single profile swap.
+
+This guide walks through the use of Spinifex to deploy a self-contained AI inference platform on Supermicro's bare metal X14 platform using only standard AWS tooling. We use Terraform to create resources (wrapped in simple `make` commands) in the exact same way you would create AWS resources.
+
+
+Specifically, we create an EKS cluster with two worker nodes, each consisting of a g7e.2xlarge EC2 instance with an attached GPU via VFIO passthrough, an ALB to route traffic to each node, ECR for storing and managing our workload images, and all of the associated security and certificate management requirements (IAM, ACM) you would expect from real AWS.
+
+### Platform
+
+| Component | Specification |
+|---|---|
+| **Chassis** | Supermicro X14 2U CloudDC with Intel Xeon 6730P |
+| **GPUs** | 2× NVIDIA RTX Pro 6000 Blackwell Server Edition (96 GiB GDDR7 each, 192 GiB total) |
+| **Storage** | 4× NVMe SSD: 2× 1.5 TB, 2× 880 GB — one 1.5 TB drive carries the OS; the remaining three back Predastore |
+| **Spinifex instance family** | `g7e` — one RTX Pro 6000 per instance via VFIO PCIe passthrough |
+| **Kubernetes** | k3s, managed via Spinifex EKS API |
+| **API endpoint** | `https://<host>:9999` (AWS-compatible) |
+
+### Workloads
+Three workloads run in the `inference` namespace of an EKS cluster backed by two GPU worker nodes:
+
+| Pod | Role | GPU | Model | Image source |
+|---|---|---|---|---|
+| `llm-server` | OpenAI-compatible chat API (llama.cpp) | 1× RTX Pro 6000 | Llama 3.2 3B Instruct Q4_K_M GGUF | ECR `llm-server:latest` |
+| `yolo-stream` | MJPEG object-detection stream (CUDA) | 1× RTX Pro 6000 | YOLO11x | ECR `yolo-stream:latest` |
+| `ai-dashboard` | Web UI: chat + live detection feed | CPU only | — | ECR `ai-dashboard:latest` |
+
+Both GPU workloads bake their model weights into the Docker image at build time — `llm-server`'s GGUF weights (~2 GiB) and `yolo-stream`'s YOLO11x checkpoint (~109 MB) are present in the image when it starts.
+
+`yolo-stream` renders a 1280×720 sample video through YOLO11x once on startup (~25 s), caches the annotated frames in memory, then serves `/stream` from that cache — smooth MJPEG playback decoupled from per-frame inference cost after the initial warm-up.
+
+The two GPU nodes are split one-per-workload. With one RTX Pro 6000 per node and both pods requesting `nvidia.com/gpu: "1"`, the scheduler cannot fit both on the same node, so the split happens naturally without explicit affinity rules.
+
+
+
+## Architecture
+
+<p align="center"><img src="../../../.github/assets/images/rtx6000/ai-platform-request-flow-2.png" alt="AI Platform request flow: HTTPS ingress → ALB routing → EKS ai-platform GPU workers"></p>
+
+### AWS services exercised
+
+| Service | Role |
+|---|---|
+| **IAM** | Cluster role, node role (with ECR, LBC, EBS-CSI permissions inline), viewer access entry |
+| **ECR** | Private registry for all three images; source of truth for builds, though nodes receive images via sideload rather than live OCI pull for this demo |
+| **EC2** | 2× GPU microVM (`g7e.2xlarge`), one RTX Pro 6000 each via VFIO PCIe passthrough |
+| **EKS** | Cluster, GPU nodegroup (`desired_size = 2`), LBC and EBS-CSI managed addons, access entries |
+| **ELBv2** | ALB provisioned by the LBC addon; single shared IngressGroup across all three services |
+| **ACM** | Self-signed cert (`ai-platform.spinifex.local`) imported and attached to the ALB HTTPS listener |
+| **EBS (Viperblock)** | 200 GB root volume per GPU worker node, provisioned by the nodegroup (`disk_size = 200`)|
+
+All permissions for the LBC and EBS-CSI addons are attached directly to the node role. Both addons support IRSA, but fall back to the node's instance profile when no `service_account_role_arn` is supplied — sufficient for a single-cluster deployment.
+
+## Prerequisites
+
+### On the Spinifex host
+
+**1. Configure host-local VPC networking**
+
+Spinifex uses bridged networking via OVN. The X14 exposes a single public IP on one physical NIC — before installing Spinifex, ensure `br-wan` exists with the physical NIC enslaved to it, and attach a host-local address range for EC2 instance communication:
+
+```yaml
+# /etc/netplan/…
+bridges:
+  br-wan:
+    addresses:
+      - 192.168.10.1/24             # VM pool gateway — host-local
+      - <existing-wan-ip>/<prefix>  # existing WAN IP — unchanged
+    routes:
+      - to: default
+        via: <upstream-gateway>
+```
+
+Apply with `sudo netplan apply`. For outbound connectivity from instances through the host's WAN interface:
+
+```bash
+sysctl -w net.ipv4.ip_forward=1
+iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o br-wan -j MASQUERADE
+```
+
+The full process is described in the [VPC Networking](/docs/vpc-networking#host-local-subnet-no-upstream-router) guide.
+
+**2. Install Spinifex**
+
+Follow the [Single Node Install](/docs/install) guide. This installs Spinifex and starts all services, however `spinifex.toml` needs to be edited to wire up the bridge created in the previous step, as described in the following section.
+
+**3. Configure spinifex.toml and restart services**
+
+Edit `/etc/spinifex/spinifex.toml` to point the external pool at the bridge address range created in step 1:
+
+```toml
+[network]
+external_mode = "pool"
+
+[[network.external_pools]]
+name        = "wan"
+source      = "static"
+range_start = "192.168.10.2"
+range_end   = "192.168.10.100"
+gateway     = "192.168.10.1"
+prefix_len  = 24
+dns_servers = ["8.8.8.8"]
+```
+
+Then restart all services:
+
+```bash
+sudo systemctl restart spinifex.target
+sudo systemctl status spinifex.target
+```
+
+**4. Bind the GPUs to VFIO**
+
+```bash
+sudo spx admin gpu setup
+# Reboot, then:
+sudo spx admin gpu enable
+```
+
+Confirm both GPUs are bound:
+
+```bash
+lspci -d 10de: -nn
+# Expect: NVIDIA Corporation Device [10de:2bb5] appearing twice
+```
+
+**5. Attach Predastore storage**
+
+The X14 has four NVMe drives: two 1.5 TB SSDs (one carries the OS) and two ~880 GB SSDs. Predastore is distributed across the three non-OS drives — one storage node per physical drive, with Reed–Solomon redundancy so a single drive failure is recoverable.
+
+Confirm drive assignments with `lsblk` before proceeding, as device names vary between systems.
+
+```bash
+lsblk   # identify the OS drive and the three data drives
+
+# Stop services so Predastore isn't writing while we relocate its data directories
+sudo systemctl stop spinifex.target
+
+# --- Repeat the block below for each data drive (node-1/nvme-1, node-2/nvme-2, node-3/nvme-3) ---
+
+# Drive 1: non-OS 1.5 TB (e.g. /dev/nvme1n1)
+sudo mkdir -p /mnt/nvme-1
+sudo mount /dev/nvme1n1 /mnt/nvme-1
+sudo mkdir -p /mnt/nvme-1/nodes /mnt/nvme-1/db
+
+sudo mv /var/lib/spinifex/predastore/distributed/nodes/node-1 /mnt/nvme-1/nodes/node-1
+sudo mv /var/lib/spinifex/predastore/distributed/db/node-1    /mnt/nvme-1/db/node-1
+
+sudo ln -s /mnt/nvme-1/nodes/node-1 /var/lib/spinifex/predastore/distributed/nodes/node-1
+sudo ln -s /mnt/nvme-1/db/node-1    /var/lib/spinifex/predastore/distributed/db/node-1
+
+echo "/dev/nvme1n1  /mnt/nvme-1  auto  defaults  0  2" | sudo tee -a /etc/fstab
+
+# Drive 2: first ~880 GB drive (e.g. /dev/nvme2n1)
+sudo mkdir -p /mnt/nvme-2
+sudo mount /dev/nvme2n1 /mnt/nvme-2
+sudo mkdir -p /mnt/nvme-2/nodes /mnt/nvme-2/db
+
+sudo mv /var/lib/spinifex/predastore/distributed/nodes/node-2 /mnt/nvme-2/nodes/node-2
+sudo mv /var/lib/spinifex/predastore/distributed/db/node-2    /mnt/nvme-2/db/node-2
+
+sudo ln -s /mnt/nvme-2/nodes/node-2 /var/lib/spinifex/predastore/distributed/nodes/node-2
+sudo ln -s /mnt/nvme-2/db/node-2    /var/lib/spinifex/predastore/distributed/db/node-2
+
+echo "/dev/nvme2n1  /mnt/nvme-2  auto  defaults  0  2" | sudo tee -a /etc/fstab
+
+# Drive 3: second ~880 GB drive (e.g. /dev/nvme3n1)
+sudo mkdir -p /mnt/nvme-3
+sudo mount /dev/nvme3n1 /mnt/nvme-3
+sudo mkdir -p /mnt/nvme-3/nodes /mnt/nvme-3/db
+
+sudo mv /var/lib/spinifex/predastore/distributed/nodes/node-3 /mnt/nvme-3/nodes/node-3
+sudo mv /var/lib/spinifex/predastore/distributed/db/node-3    /mnt/nvme-3/db/node-3
+
+sudo ln -s /mnt/nvme-3/nodes/node-3 /var/lib/spinifex/predastore/distributed/nodes/node-3
+sudo ln -s /mnt/nvme-3/db/node-3    /var/lib/spinifex/predastore/distributed/db/node-3
+
+echo "/dev/nvme3n1  /mnt/nvme-3  auto  defaults  0  2" | sudo tee -a /etc/fstab
+
+# --- End of per-drive block ---
+
+sudo systemctl start spinifex.target
+```
+
+Verify the nodes are healthy before proceeding:
+
+```bash
+export AWS_PROFILE=spinifex
+aws s3 ls
+# Should return without error (empty bucket list is fine)
+```
+
+**6. Verify the GPU instance type**
+
+```bash
+sudo spx admin gpu status
+```
+
+This will print confirmation that GPU passthrough has been configured correctly along with the available GPU instance types. The RTX Pro 6000 Blackwell Server Edition (PCI device `10de:2bb5`) maps to the `g7e` family. The workbook defaults to `g7e.2xlarge` (one GPU per node); override with `GPU_TYPE=g7e.4xlarge` (or the size your host reports) if needed. Do not use `g7e.12xlarge` — that is the 2× GPU size.
+
+### Local tooling
+
+- [OpenTofu](https://opentofu.org/) >= 1.6
+- [kubectl](https://kubernetes.io/docs/tasks/tools/)
+- [Docker](https://docs.docker.com/get-docker/)
+
+
+### Clone the workbook
+
+```bash
+git clone https://github.com/mulgadc/eks-ai-platform
+cd eks-ai-platform
+```
+
+## Instructions
+
+### 1. Import the EKS GPU node AMI
+
+GPU worker nodes require the `ecr-credential-provider` binary so kubelet can call `GetAuthorizationToken` against ECR. This binary is included in the dedicated EKS GPU node AMI in the Spinifex image catalogue. List available images and import it:
+
+```bash
+spx admin images list
+# Look for the EKS GPU node image
+
+spx admin images import --name spinifex-eks-node-gpu
+```
+
+Confirm the AMI is registered:
+
+```bash
+aws ec2 describe-images --query 'Images[*].[Name,ImageId]' --output table
+```
+
+### 2. Provision the cluster
+
+```bash
+make infra ENDPOINT=https://<host>:9999
+```
+
+This provisions the VPC (`10.33.0.0/16`, two public and two private subnets with a NAT gateway), IAM roles, ECR repositories, EKS cluster, GPU nodegroup (2× `g7e.2xlarge`, 200 GB disk each), LBC and EBS-CSI managed addons, a self-signed ACM certificate, and the NodePort security group rules the ALB needs to reach the worker nodes.
+
+Update your kubeconfig once the cluster reports `ACTIVE`:
+
+```bash
+$(tofu -chdir=workbook output -raw update_kubeconfig)
+kubectl get nodes
+# Expect: 2 Ready nodes in the gpu-workers nodegroup
+```
+
+The Makefile wraps `tofu -chdir=workbook apply -var spinifex_endpoint=... -var gpu_instance_type=...` — running Tofu directly is equivalent and lets you pass any additional variables. The workbook provisions `aws_vpc`, `aws_subnet` (two public, two private), `aws_eks_cluster`, `aws_eks_node_group` (two `g7e.2xlarge` nodes, each with a 200 GB Viperblock root volume via `disk_size = 200`), `aws_eks_addon` for LBC and EBS-CSI, three `aws_ecr_repository` resources, three IAM roles, and a self-signed `aws_acm_certificate` — all via Spinifex's AWS-compatible endpoint at `:9999`.
+
+### 3. Build and push container images
+
+```bash
+make images
+```
+
+This authenticates to ECR, then builds and pushes all three images:
+
+- **`llm-server`** — based on `ghcr.io/ggml-org/llama.cpp:server-cuda`; downloads Llama 3.2 3B Instruct Q4_K_M GGUF (~2 GiB) from Hugging Face at build time and bakes it into the image. Exposes an OpenAI-compatible `/v1/chat/completions` API.
+- **`yolo-stream`** — based on `pytorch/pytorch:2.7.1-cuda12.8-cudnn9-runtime`; installs Ultralytics and downloads YOLO11x weights at build time. PyTorch 2.7.1+cu128 is required: the RTX Pro 6000 Blackwell is compute capability sm_120, and earlier PyTorch releases ship no sm_120 kernels.
+- **`ai-dashboard`** — lightweight Flask proxy (`FROM python:3.11-slim`) that aggregates the LLM API and YOLO stream into a single page.
+
+The ECR registry URI always includes `:9999` — for example, `<account>.dkr.ecr.ap-southeast-2.<suffix>:9999`. Use the `ecr_registry` Tofu output directly in `docker login` and image references; do not construct the hostname manually.
+
+Image URIs come from `tofu -chdir=workbook output -raw ecr_registry`. ECR authentication uses the same API as AWS: `aws ecr get-login-password` calls `GetAuthorizationToken` against the Spinifex ECR endpoint and returns a short-lived JWT that Docker accepts as a registry password. The `make images` target is equivalent to running those `docker build` and `docker push` commands directly against `$REGISTRY` from that Tofu output.
+
+### 4. Sideload images onto GPU worker nodes
+
+```bash
+make sideload
+```
+
+ECR is the source of truth for all three images — authentication, push, and registry management all work identically to AWS. In a standard EKS deployment, nodes would pull images directly from ECR at scheduling time. In this demo, live pulls of these image sizes (~2 GiB for `llm-server`, ~4.5 GiB for `yolo-stream`) through the Spinifex ECR gateway proved unreliable — large transfers stalled or failed mid-stream, a combination of network conditions on this single-host setup and a rough edge in early Spinifex ECR support. This step works around that by staging the images directly into each node's containerd store before the pods are scheduled.
+
+It exports each image from the local Docker daemon, serves the tarballs over HTTP from the Spinifex host, and imports them directly into each worker node's containerd store via short-lived privileged pods. All three Deployments use `imagePullPolicy: IfNotPresent` and depend on the images already being present in containerd.
+
+The script also labels the two GPU nodes deterministically (`workload=llm-server` / `workload=yolo-stream`, sorted by node name), and the Deployments use matching `nodeSelector` values so each pod lands on the node that already has its image. `ai-dashboard` has no GPU requirement and is imported on both nodes since it can schedule onto either.
+
+Expect several minutes for `yolo-stream`'s image (CUDA + PyTorch + YOLO11x weights, ~4.5 GiB).
+
+This step has no Tofu equivalent — it operates directly on the running cluster via `kubectl`. The ECR registry URI is read from the Tofu state; node labelling uses `kubectl label` and image import uses short-lived privileged pods that run `ctr images import` into each node's containerd store.
+
+### 5. Deploy workloads
+
+```bash
+make workloads ENDPOINT=https://<host>:9999
+```
+
+This deploys the NVIDIA GPU Operator via Helm, then all three application Deployments, ClusterIP and NodePort services, and the shared ALB Ingresses.
+
+The GPU Operator requires two adjustments for k3s:
+
+- `driver.enabled=false` — the NVIDIA driver is pre-built into the GPU AMI at image creation time; the Operator installs only the toolkit and device plugin.
+- `CONTAINERD_SOCKET=/run/k3s/containerd/containerd.sock` — k3s bundles its own containerd at a different socket and config path than the standalone containerd default. Without this override the toolkit DaemonSet crash-loops with `no such file or directory`.
+
+Watch the GPU Operator complete, then the inference pods come up:
+
+```bash
+kubectl -n gpu-operator get pods -w
+kubectl -n inference get pods -w
+```
+
+Confirm each GPU node reports `nvidia.com/gpu: 1` in allocatables:
+
+```bash
+$(tofu -chdir=workbook output -raw gpu_verify_hint)
+```
+
+All three routes share a single ALB via `alb.ingress.kubernetes.io/group.name`. Explicit `group.order` values (`/v1` = 10, `/stream` = 20, `/` = 100) ensure the dashboard's catch-all path evaluates last — without them, the LBC sorts Ingress resources alphabetically, which places the catch-all first and swallows the other routes.
+
+Retrieve the ALB IP (the DNS name `*.elb.spinifex.local` is a label, not a resolvable entry):
+
+```bash
+$(tofu -chdir=workbook output -raw alb_ip_hint)
+ALB_IP=<output>
+```
+
+Validate the LLM endpoint:
+
+```bash
+curl -sk https://$ALB_IP/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{"model":"llama-3.2-3b-instruct","messages":[{"role":"user","content":"What is Spinifex?"}]}' \
+  | python3 -m json.tool
+```
+
+Open `https://$ALB_IP/` to reach the dashboard. If accessing from a remote machine:
+
+```bash
+ssh -L 8443:$ALB_IP:443 <spinifex-host>
+# Open https://ai-platform.spinifex.local:8443/
+# Add 127.0.0.1 ai-platform.spinifex.local to /etc/hosts if the browser requires hostname match
+```
+
+The workloads module reads cluster coordinates, ECR image URIs, NodePort values, and the ACM cert ARN from the parent module's Tofu state, then creates the NVIDIA GPU Operator `helm_release`, three `kubernetes_deployment_v1` resources, six `kubernetes_service_v1` resources (ClusterIP + NodePort per workload), and three `kubernetes_ingress_v1` resources — all through the Terraform Kubernetes and Helm providers, which authenticate to the cluster via `aws eks get-token`.
+
+### Dashboard
+
+After the successful completion of the `make workloads` step, the dashboard should be available and displaying the outputs of the two worker nodes; YOLO computer vision in the left pane, and a LLM chat in the right pane, as shown in the images below.
+
+<p align="center"><img src="../../../.github/assets/images/rtx6000/yolo.png" alt="YOLO real-time object detection stream"></p>
+
+<p align="center"><img src="../../../.github/assets/images/rtx6000/llm-chat.png" alt="LLM chat interface"></p>
+
+
+
+### 6. Teardown
+
+```bash
+make destroy ENDPOINT=https://<host>:9999
+```
+
+Workloads are destroyed before infra. Both GPU worker instances terminate, immediately returning their RTX Pro 6000s to the Spinifex pool.
+
+The Makefile runs the two-module destroy sequence — `tofu -chdir=workbook/workloads destroy` first, then `tofu -chdir=workbook destroy` — because the parent module's security group rules are referenced by the ALB created in the workloads layer. Running Tofu directly in that order is equivalent.
+
+## Troubleshooting
+
+### `llm-server` or `yolo-stream` pod stuck in `Pending`
+
+The GPU Operator DaemonSet must complete before `nvidia.com/gpu` appears in node allocatables:
+
+```bash
+kubectl -n gpu-operator get daemonset -w
+kubectl get node -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu'
+```
+
+If a pod shows `Insufficient nvidia.com/gpu` on a node that looks healthy, the `workload=` labels from `make sideload` may be stale — for example, after a nodegroup recreation that assigned new node names. Re-run `make sideload` to relabel the nodes and re-import the images.
+
+### GPU worker instances fail to launch (`bind ... to vfio-pci: invalid argument`)
+
+On hosts where the GPU's IOMMU group contains an upstream PCIe root-port bridge (no ACS isolation), Spinifex attempts to bind the bridge to `vfio-pci`. Bridges are non-endpoint devices that `vfio-pci` refuses to bind, causing the instance to crash immediately after launch:
+
+```
+GPU claim failed... bind IOMMU group member 0000:14:02.0: bind 0000:14:02.0 to vfio-pci: invalid argument
+```
+
+Check IOMMU group membership with `lspci -nnk` and `/sys/kernel/iommu_groups/*/devices/`. This is fixed in Spinifex by excluding bridge-class PCI devices from the bind lifecycle — ensure your Spinifex build includes that fix. If the failed bind left a bridge without a driver, restore it:
+
+```bash
+echo | sudo tee /sys/bus/pci/devices/<bridge-addr>/driver_override
+echo <bridge-addr> | sudo tee /sys/bus/pci/drivers/pcieport/bind
+```
+
+If the nodegroup got wedged in `CREATING` after hitting this, delete it and let Terraform recreate it:
+
+```bash
+aws eks delete-nodegroup --cluster-name ai-platform --nodegroup-name gpu-workers
+```
+
+### Nodegroup stuck `CREATING` with healthy nodes, Tofu times out
+
+`tofu apply` hangs for 20 minutes and fails with `workers did not become Ready: timed out`, even though `kubectl get nodes` shows both workers `Ready`. The cause is a missing `eks.amazonaws.com/nodegroup` label on the node — without it, the control plane never tallies the nodegroup as satisfied. Confirm with:
+
+```bash
+kubectl get node <worker> -o jsonpath='{.metadata.labels}'
+```
+
+This is fixed upstream in Spinifex. If stuck, delete both the nodegroup and cluster and let Terraform recreate them cleanly:
+
+```bash
+aws eks delete-nodegroup --cluster-name ai-platform --nodegroup-name gpu-workers
+aws eks delete-cluster --name ai-platform
+```
+
+### ALB returns 502 immediately after rollout
+
+Both `llm-server` and `yolo-stream` require a few seconds after container start before their readiness probes pass — CUDA and GPU Operator initialisation contribute to the delay. The ALB marks targets unhealthy during this window. Watch the pods become ready:
+
+```bash
+kubectl -n inference get pods -w
+kubectl -n inference logs -f deploy/llm-server
+```
+
+## Conclusion
+
+This guide demonstrates how Spinifex turns a single bare-metal chassis into a production-shaped AI serving platform managed entirely with standard AWS tooling. IAM roles, ECR repositories, an EKS cluster, GPU worker nodes, addons, and an ALB are all provisioned with the same Terraform resources and AWS CLI commands that work on real AWS — with a single `AWS_PROFILE` swap.
+
+The RTX Pro 6000 Blackwell Server Edition's 96 GiB GDDR7 fits substantial GPU workloads in a single PCIe slot, and Spinifex's `g7e` instance family exposes each GPU as a standard EC2 instance. Teams already operating AWS infrastructure can point their existing tooling at a Spinifex node and retain the full EKS workflow — from `aws ecr get-login-password` to `kubectl get ingress` — on hardware they own. ECR acts as the canonical registry throughout: image build, push, and authentication are identical to AWS, with direct kubelet pulls being the intended path as Spinifex's ECR gateway matures.

docs/security/network-connections/README.md

@@ -107,8 +107,10 @@ Spinifex nodes initiate a small, fixed set of outbound connections.
 
 | Destination | Purpose | Protocol | Verification |
 |-------------|---------|----------|--------------|
-| `https://cloud.debian.org/images/cloud/trixie/` | Debian 13 cloud image download | HTTPS | TLS + checksum verification |
-| `https://cloud-images.ubuntu.com/resolute/` | Ubuntu 26.04 LTS cloud image download | HTTPS | TLS + checksum verification |
+| `https://cloud.debian.org/images/cloud/trixie/latest/` | Debian 13 cloud image download | HTTPS | TLS + checksum verification |
+| `https://cloud-images.ubuntu.com/releases/resolute/release/` | Ubuntu 26.04 LTS cloud image download | HTTPS | TLS + checksum verification |
+| `https://dl.rockylinux.org/pub/rocky/10/images/` | Rocky Linux 10 cloud image download | HTTPS | TLS + checksum verification |
+| `https://dl-cdn.alpinelinux.org/alpine/` | Alpine Linux cloud image download | HTTPS | TLS + checksum verification |
 | `https://d2yp8ipz5jfqcw.cloudfront.net` | Alpine image for managed HAProxy load-balancer | HTTPS | TLS + checksum verification |
 | `https://install.mulgadc.com/install` | One-shot install telemetry POST on `spx admin init` / `join`. | HTTPS | TLS |
 
@@ -116,7 +118,7 @@ Spinifex nodes initiate a small, fixed set of outbound connections.
 
 **Update checks and metadata.** Spinifex does not check for updates and does not consume a cloud metadata service (`169.254.169.254` is served *by* the cluster to guest VMs). Node software updates come from the operator's OS package channel. The install-telemetry endpoint above is the only vendor-operated destination contacted by a node; closed-egress deployments should disable it and record the opt-out in the security plan.
 
-**Air-gapped deployments.** The three image URLs are the only destinations needed for the standard image catalogue. Mirror them locally and use `spx admin images import --file` with pre-staged files. Telemetry must also be disabled. See [Air-Gapped Install](/docs/install-airgapped).
+**Air-gapped deployments.** The image URLs above are the only destinations needed for the standard image catalogue. Mirror them locally and use `spx admin images import --file` with pre-staged files. Telemetry must also be disabled. See [Air-Gapped Install](/docs/install-airgapped).
 
 ## 3. Cross-Node (Internal) Connections
 

go.mod

@@ -4,9 +4,10 @@ go 1.26.5
 
 require (
 	github.com/aws/aws-sdk-go v1.55.8
-	github.com/aws/aws-sdk-go-v2 v1.42.1
-	github.com/aws/aws-sdk-go-v2/config v1.32.29
-	github.com/aws/aws-sdk-go-v2/credentials v1.19.28
+	github.com/aws/aws-sdk-go-v2 v1.43.0
+	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14
+	github.com/aws/aws-sdk-go-v2/config v1.32.31
+	github.com/aws/aws-sdk-go-v2/credentials v1.19.30
 	github.com/cenkalti/backoff/v4 v4.3.0
 	github.com/charmbracelet/bubbles v1.0.0
 	github.com/charmbracelet/bubbletea v1.3.10
@@ -18,8 +19,10 @@ require (
 	github.com/google/uuid v1.6.0
 	github.com/insomniacslk/dhcp v0.0.0-20260407060928-11b94ed970f2
 	github.com/klauspost/cpuid/v2 v2.4.0
-	github.com/mulgadc/predastore v1.12.1
-	github.com/mulgadc/viperblock v1.12.1
+	github.com/miekg/dns v1.1.72
+	github.com/mulgadc/northstar v1.14.0
+	github.com/mulgadc/predastore v1.14.0
+	github.com/mulgadc/viperblock v1.14.0
 	github.com/nats-io/nats-server/v2 v2.14.3
 	github.com/nats-io/nats.go v1.52.0
 	github.com/opencontainers/runtime-spec v1.3.0
@@ -59,17 +62,20 @@ require (
 	github.com/antithesishq/antithesis-sdk-go v0.7.0 // indirect
 	github.com/armon/go-metrics v0.4.1 // indirect
 	github.com/atotto/clipboard v0.1.4 // indirect
-	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.30 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.30 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.30 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.31 // indirect
+	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.31 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.31 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.31 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.32 // indirect
 	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.30 // indirect
-	github.com/aws/aws-sdk-go-v2/service/signin v1.4.0 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sso v1.32.0 // indirect
-	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.37.0 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sts v1.44.0 // indirect
-	github.com/aws/smithy-go v1.27.3 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.24 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.31 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.32 // indirect
+	github.com/aws/aws-sdk-go-v2/service/s3 v1.106.0 // indirect
+	github.com/aws/aws-sdk-go-v2/service/signin v1.5.0 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sso v1.33.0 // indirect
+	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.0 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sts v1.45.0 // indirect
+	github.com/aws/smithy-go v1.27.4 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/boltdb/bolt v1.3.1 // indirect
@@ -98,13 +104,13 @@ require (
 	github.com/containerd/ttrpc v1.2.8 // indirect
 	github.com/containerd/typeurl/v2 v2.2.3 // indirect
 	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
-	github.com/dgraph-io/badger/v4 v4.9.4 // indirect
+	github.com/dgraph-io/badger/v4 v4.9.5 // indirect
 	github.com/dgraph-io/ristretto/v2 v2.4.0 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
 	github.com/fatih/color v1.19.0 // indirect
 	github.com/felixge/httpsnoop v1.0.4 // indirect
-	github.com/fsnotify/fsnotify v1.9.0 // indirect
+	github.com/fsnotify/fsnotify v1.10.1 // indirect
 	github.com/gabriel-vasile/mimetype v1.4.13 // indirect
 	github.com/go-logr/logr v1.4.3 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
@@ -166,7 +172,7 @@ require (
 	github.com/prometheus/client_model v0.6.2 // indirect
 	github.com/prometheus/common v0.67.5 // indirect
 	github.com/prometheus/procfs v0.20.1 // indirect
-	github.com/quic-go/quic-go v0.60.0 // indirect
+	github.com/quic-go/quic-go v0.61.0 // indirect
 	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/sagikazarmark/locafero v0.12.0 // indirect
 	github.com/sirupsen/logrus v1.9.4 // indirect
@@ -181,7 +187,9 @@ require (
 	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
 	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.44.0 // indirect
 	go.opentelemetry.io/proto/otlp v1.10.0 // indirect
+	go.uber.org/nilaway v0.0.0-20260721205819-4a0f7653488a // indirect
 	go.yaml.in/yaml/v2 v2.4.4 // indirect
+	golang.org/x/exp/typeparams v0.0.0-20260611194520-c48552f49976 // indirect
 	golang.org/x/mod v0.37.0 // indirect
 	golang.org/x/net v0.56.0 // indirect
 	golang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57 // indirect
@@ -189,10 +197,10 @@ require (
 	golang.org/x/text v0.40.0 // indirect
 	golang.org/x/time v0.15.0 // indirect
 	golang.org/x/tools v0.47.0 // indirect
-	golang.org/x/vuln v1.1.4 // indirect
+	golang.org/x/vuln v1.4.0 // indirect
 	google.golang.org/genproto/googleapis/api v0.0.0-20260618152121-87f3d3e198d3 // indirect
 	google.golang.org/genproto/googleapis/rpc v0.0.0-20260618152121-87f3d3e198d3 // indirect
-	google.golang.org/grpc v1.81.1 // indirect
+	google.golang.org/grpc v1.82.1 // indirect
 	google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 	sigs.k8s.io/yaml v1.6.0 // indirect
@@ -209,4 +217,7 @@ require (
 // stop requiring the monolith.
 replace google.golang.org/genproto => google.golang.org/genproto v0.0.0-20260622175928-b703f567277d
 
-tool golang.org/x/vuln/cmd/govulncheck
+tool (
+	go.uber.org/nilaway/cmd/nilaway
+	golang.org/x/vuln/cmd/govulncheck
+)

go.sum

@@ -524,78 +524,78 @@ github.com/aws/aws-sdk-go-v2 v0.18.0/go.mod h1:JWVYvqSMppoMJC0x5wdwiImzgXTI9FuZw
 github.com/aws/aws-sdk-go-v2 v1.17.5/go.mod h1:uzbQtefpm44goOPmdKyAlXSNcwlRgF3ePWVW6EtJvvw=
 github.com/aws/aws-sdk-go-v2 v1.25.2/go.mod h1:Evoc5AsmtveRt1komDwIsjHFyrP5tDuF1D1U+6z6pNo=
 github.com/aws/aws-sdk-go-v2 v1.36.3/go.mod h1:LLXuLpgzEbD766Z5ECcRmi8AzSwfZItDtmABVkRLGzg=
-github.com/aws/aws-sdk-go-v2 v1.42.1 h1:9eOTgu1z/dVtYpNZ3/8/XbbaX0x/BqE3HUzAzs6K0ek=
-github.com/aws/aws-sdk-go-v2 v1.42.1/go.mod h1:5pKeft2eJj+gElQ38Jqg4ibCqh+/AK33/0X3hip7IjM=
+github.com/aws/aws-sdk-go-v2 v1.43.0 h1:fharf/WhbRAVZ1du0QL7roNFxZ6T/sWr+4Ni617bwSI=
+github.com/aws/aws-sdk-go-v2 v1.43.0/go.mod h1:5pKeft2eJj+gElQ38Jqg4ibCqh+/AK33/0X3hip7IjM=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 h1:3IZY0XAJquT3aHzbkHfPzy4ACPcEjVG0x87KOwtpqGY=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14/go.mod h1:zwM6veDkhGgQFqkBy+uT28AAYpLu+uFMlPl+rCg/73E=
 github.com/aws/aws-sdk-go-v2/config v1.18.14/go.mod h1:0pI6JQBHKwd0JnwAZS3VCapLKMO++UL2BOkWwyyzTnA=
 github.com/aws/aws-sdk-go-v2/config v1.27.4/go.mod h1:zq2FFXK3A416kiukwpsd+rD4ny6JC7QSkp4QdN1Mp2g=
 github.com/aws/aws-sdk-go-v2/config v1.29.12/go.mod h1:xse1YTjmORlb/6fhkWi8qJh3cvZi4JoVNhc+NbJt4kI=
-github.com/aws/aws-sdk-go-v2/config v1.32.29 h1:BcMHHnpiWKogf+gGfpj3K1w+Sktz29XDo/cPSAPO3FU=
-github.com/aws/aws-sdk-go-v2/config v1.32.29/go.mod h1:+Kbhn8Es4kPUph3F/0W7avykytc+Jh2Ld9/msv9ljV4=
+github.com/aws/aws-sdk-go-v2/config v1.32.31 h1:n4nY9O3QKoHIkL85EX+V8RcMFtOhlpTFhGArg915PXk=
+github.com/aws/aws-sdk-go-v2/config v1.32.31/go.mod h1:PN0NYDCCoOpGGsZ2+elDUidmHfQBPyYzN2GCgl8HEBs=
 github.com/aws/aws-sdk-go-v2/credentials v1.13.14/go.mod h1:85ckagDuzdIOnZRwws1eLKnymJs3ZM1QwVC1XcuNGOY=
 github.com/aws/aws-sdk-go-v2/credentials v1.17.4/go.mod h1:+30tpwrkOgvkJL1rUZuRLoxcJwtI/OkeBLYnHxJtVe0=
 github.com/aws/aws-sdk-go-v2/credentials v1.17.65/go.mod h1:4zyjAuGOdikpNYiSGpsGz8hLGmUzlY8pc8r9QQ/RXYQ=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.28 h1:zTXJSsNcoO91/mTXsZoYf0AK8dvNPiA58/VtyGXR+wM=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.28/go.mod h1:Kd9E0JzDBW/q1xbsHFrev/GnbAf5J0Ng8xoyc7HZ91Q=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.30 h1:TTCvvzFU6gXa4iJecNG/0F/B0oYTiazoRECr2XyLHrY=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.30/go.mod h1:jKxAp2AEncnliinzpgOSZDFv6+VjvWhjw/AtbfsWT9U=
 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.23/go.mod h1:mOtmAg65GT1HIL/HT/PynwPbS+UG0BgCZ6vhkPqnxWo=
 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.15.2/go.mod h1:iRlGzMix0SExQEviAyptRWRGdYNo3+ufW/lCzvKVTUc=
 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.30/go.mod h1:Jpne2tDnYiFascUEs2AWHJL9Yp7A5ZVy3TNyxaAjD6M=
-github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.30 h1:/hi1JADLEW9YYryEz1w4GQu0EtP23pP553Cf9KgsDV4=
-github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.30/go.mod h1:/3AOgy4K17Dm4ucMZVC/MJkzy5kmfKUcINRHZyo0koQ=
+github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.31 h1:kfVL5wAunCJycL6MOQ6aNh6PlAYEymflcjuKmrWUA0o=
+github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.31/go.mod h1:nWfRNDAppujCQgOUd43lKT4yeLv9z3nJ3bw1G3BgQKo=
 github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.29/go.mod h1:Dip3sIGv485+xerzVv24emnjX5Sg88utCL8fwGmCeWg=
 github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.2/go.mod h1:wRQv0nN6v9wDXuWThpovGQjqF1HFdcgWjporw14lS8k=
 github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.34/go.mod h1:p4VfIceZokChbA9FzMbRGz5OV+lekcVtHlPKEO0gSZY=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.30 h1:xM/Is9cKMHa8Jj8zkvWhvrFkZsXJV9E+BB4g0HW0duQ=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.30/go.mod h1:WueJeNDZvK1fMYEWJIkcivBfEzUkTpBhzlrUKKY8EuA=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.31 h1:Z8F3hfCY33IGpJjFAnv0wvtv1FIKj1GHmRDEYqy64tw=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.31/go.mod h1:aVyUoytEyOViR6jhq6jula0xkc5NfBE2hgeF6BvOrao=
 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.23/go.mod h1:mr6c4cHC+S/MMkrjtSlG4QA36kOznDep+0fga5L/fGQ=
 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.2/go.mod h1:tyF5sKccmDz0Bv4NrstEr+/9YkSPJHrcO7UsUKf7pWM=
 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.34/go.mod h1:dFZsC0BLo346mvKQLWmoJxT+Sjp+qcVR1tRVHQGOH9Q=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.30 h1:jn46zC9LdsVR/ZpMIJqMqb8hHv31BlLx3ulVqNspUOk=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.30/go.mod h1:1hTMsAgbdS/AtUi4bw8+gUuh1pceo+eXRLfpSuSQj3M=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.31 h1:hyOxUyXdh3AyjE93gBgsfziJag9ACwcs+ZpDBLzi8mw=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.31/go.mod h1:OERqI9k0draSLB8O8woxY3q25ZWTELRK4RRoLMuMZFo=
 github.com/aws/aws-sdk-go-v2/internal/ini v1.3.30/go.mod h1:vsbq62AOBwQ1LJ/GWKFxX8beUEYeRp/Agitrxee2/qM=
 github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0/go.mod h1:8tu/lYfQfFe6IGnaOdrpVgEL2IrrDOf6/m9RQum4NkY=
 github.com/aws/aws-sdk-go-v2/internal/ini v1.8.3/go.mod h1:H5O/EsxDWyU+LP/V8i5sm8cxoZgc2fdNR9bxlOFrQTo=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.31 h1:3GUprIsfmGcC5SACIyB0e7E0BM1O1b3Erl5CePYIAeQ=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.31/go.mod h1:7PuV1yl5e2xnUbm+RqvVg5i2iBM8EyijZNoI9wsOoOc=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.32 h1:0MrUL35H/Y4kdFfItoR5jCgtDQ4Z/8LudAoIHRfA4hE=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.32/go.mod h1:2tNZkuWz54arj8mHVf+8Y7cKkcD8Wr/fBpENgEXpjLc=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.1/go.mod h1:JKpmtYhhPs7D97NL/ltqz7yCkERFW5dOlHyVl66ZYF8=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.3/go.mod h1:0yKJC/kb8sAnmlYa6Zs3QVYqaC8ug2AbnNChv5Ox3uA=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13 h1:mbRIur/BiHK6SKPjoBIXSE/hJ6g6JGRLuxQy1jGjlN4=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13/go.mod h1:ITg9em2KbJx1s0y4aqRX5OYWG6HBZ5TVR//OdpEZ2CQ=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23 h1:9Fjh6fi/U5JEStVZijmaMpUwE/gvBJj7x2B/PjbO9To=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23/go.mod h1:iMoT2f1tClxrWAAnKCXjZQ6LOmfLrMG14wmnWpM+F14=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.24 h1:mdPwDQPqxlw9Sc62Nt15yjEcARaDbPXkjRYtXsUripo=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.24/go.mod h1:ls5ytnwLTcQaUu32fMYXFI3MjpKuTwL840PAm9iqyEg=
 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.23/go.mod h1:9uPh+Hrz2Vn6oMnQYiUi/zbh3ovbnQk19YKINkQny44=
 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.2/go.mod h1:Ru7vg1iQ7cR4i7SZ/JTLYN9kaXtbL69UdgG0OQWQxW0=
 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.15/go.mod h1:SwFBy2vjtA0vZbjjaFtfN045boopadnoVPhu4Fv66vY=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.30 h1:/Z5jmNrKsSD7EmDjzAPsm/3L9IuOkzaynklJZ1qX7S4=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.30/go.mod h1:lEzEZnOosE7zi8Z6royW1cFJTD9fpab4Ul1SBrllewk=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31 h1:uao4A3QZ5UmB326V6KF+qRpv9Tjz7IlnlnTbbANntlU=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31/go.mod h1:I/1+z0VwL1GhQyLgkoHDlygpUZ+iTAwOQ/NsftiUL2I=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0 h1:XptwLL+UHXgafYMIHTy59IRovLbhz3znkxY2uS/pbXU=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0/go.mod h1:zdmCoFO/dSI7GlrwsPqFJI+WlFnSU4Tc8TJnlXrM1Do=
-github.com/aws/aws-sdk-go-v2/service/signin v1.4.0 h1:sLzmJGCMv+C8KqiJgEqDLB6vxaJGmobRh4rr//ZpA3w=
-github.com/aws/aws-sdk-go-v2/service/signin v1.4.0/go.mod h1:mxC0nT/C8wMMS97DemZPzvUZxvIt+2Iq+eS3JdFZGgg=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.31 h1:w2SIhW92DZPFrSL4ksVCr8IYff5OZwIcxg8+95tzvAI=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.31/go.mod h1:wAhpCQbkov+IcvjozJbd2xRCoZybUEHNkcFunssNACg=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.32 h1:jWXtZdCnhXa9sGFixRaU2AxT4DIVse9HS4E2f+/KwV0=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.32/go.mod h1:9JS1UpfVvyD/ZPX8GsKb/Pq8scEM+7GP5fqh9SwH7po=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.106.0 h1:7QZWVJZWzHivHWIa+5TELLaBBkbuoj0GPwQtMlJ0sqk=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.106.0/go.mod h1:fcvq5L7dK+5cQFicEJwpI6e6Wn8NY2i6yT5wRLYVc7s=
+github.com/aws/aws-sdk-go-v2/service/signin v1.5.0 h1:OHH5iTQvVGmfHjX/5Q+vFuA/Rf2x6/95aJ/75QCQSm4=
+github.com/aws/aws-sdk-go-v2/service/signin v1.5.0/go.mod h1:mCF3AK9PpL49oOrhniUXWAfhVBVQ/XbytoE5eccZUIs=
 github.com/aws/aws-sdk-go-v2/service/sso v1.12.3/go.mod h1:jtLIhd+V+lft6ktxpItycqHqiVXrPIRjWIsFIlzMriw=
 github.com/aws/aws-sdk-go-v2/service/sso v1.20.1/go.mod h1:RsYqzYr2F2oPDdpy+PdhephuZxTfjHQe7SOBcZGoAU8=
 github.com/aws/aws-sdk-go-v2/service/sso v1.25.2/go.mod h1:qs4a9T5EMLl/Cajiw2TcbNt2UNo/Hqlyp+GiuG4CFDI=
-github.com/aws/aws-sdk-go-v2/service/sso v1.32.0 h1:qjMmry/cBDee1E/2gyvel0uRYCi3mwRZ2hf6N+GAodo=
-github.com/aws/aws-sdk-go-v2/service/sso v1.32.0/go.mod h1:u8af9Nqkmqnr96f7v9nHqzZT9XBwbXEkTiqT4ROuJSE=
+github.com/aws/aws-sdk-go-v2/service/sso v1.33.0 h1:CaJyYhxBE0M/HJX/YvSaSmQlsI91VHB0lKU8LtLxL3A=
+github.com/aws/aws-sdk-go-v2/service/sso v1.33.0/go.mod h1:+e6BMRMPjBQoCw/WovYR9GLy2IU0z4Q77smOB1DraSg=
 github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.3/go.mod h1:zVwRrfdSmbRZWkUkWjOItY7SOalnFnq/Yg2LVPqDjwc=
 github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.1/go.mod h1:YjAPFn4kGFqKC54VsHs5fn5B6d+PCY2tziEa3U/GB5Y=
 github.com/aws/aws-sdk-go-v2/service/ssooidc v1.30.0/go.mod h1:MlYRNmYu/fGPoxBQVvBYr9nyr948aY/WLUvwBMBJubs=
-github.com/aws/aws-sdk-go-v2/service/ssooidc v1.37.0 h1:fpOlDPI55HdszaxapEGk6HsGosOUaM2YPWJpjMgp8UI=
-github.com/aws/aws-sdk-go-v2/service/ssooidc v1.37.0/go.mod h1:DMPWJBjYs6+3+f/qhBFEFPPlQ6NlhWjai3dJNvipJ84=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.0 h1:tC323YV77QdafeBr6LUhLDTsboyuyHLNRwAyCP44kGU=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.0/go.mod h1:SfLK1sgviHmbI+MozR9iDwDjL4cdCVZtahsjoR+z7wg=
 github.com/aws/aws-sdk-go-v2/service/sts v1.18.4/go.mod h1:1mKZHLLpDMHTNSYPJ7qrcnCQdHCWsNQaT0xRvq2u80s=
 github.com/aws/aws-sdk-go-v2/service/sts v1.28.1/go.mod h1:uQ7YYKZt3adCRrdCBREm1CD3efFLOUNH77MrUCvx5oA=
 github.com/aws/aws-sdk-go-v2/service/sts v1.33.17/go.mod h1:cQnB8CUnxbMU82JvlqjKR2HBOm3fe9pWorWBza6MBJ4=
-github.com/aws/aws-sdk-go-v2/service/sts v1.44.0 h1:bLZ0PolJ8J+HkJHztcXORUpHXBye2U8298lCEMi6ZCU=
-github.com/aws/aws-sdk-go-v2/service/sts v1.44.0/go.mod h1:9gdl4RrflIdpDb2TlXshWgR1F9TeCkvqDx77Vpr4Z/Q=
+github.com/aws/aws-sdk-go-v2/service/sts v1.45.0 h1:Pd6PNlp4t8PTXxqzstICl52Wsy78vpjFZ7PRUj44mJc=
+github.com/aws/aws-sdk-go-v2/service/sts v1.45.0/go.mod h1:rmQ0TnHzuLPmabgjPcsywhsSOmaBDgzR4zvDxSPsGdg=
 github.com/aws/smithy-go v1.13.5/go.mod h1:Tg+OJXh4MB2R/uN61Ko2f6hTZwB/ZYGOtib8J3gBHzA=
 github.com/aws/smithy-go v1.20.1/go.mod h1:krry+ya/rV9RDcV/Q16kpu6ypI4K2czasz0NC3qS14E=
 github.com/aws/smithy-go v1.22.2/go.mod h1:irrKGvNn1InZwb2d7fkIRNucdfwR8R+Ts3wxYa/cJHg=
 github.com/aws/smithy-go v1.22.3/go.mod h1:t1ufH5HMublsJYulve2RKmHDC15xu1f26kHCp/HgceI=
-github.com/aws/smithy-go v1.27.3 h1:F3Zb497UhhskkfpJmfkXswyo+t0sh9OTBnIHjogWbVY=
-github.com/aws/smithy-go v1.27.3/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
+github.com/aws/smithy-go v1.27.4 h1:JQcphmBN4f0q/sPqXqROIItRNV/hy10cgu7CsFy616M=
+github.com/aws/smithy-go v1.27.4/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/bazelbuild/rules_go v0.49.0/go.mod h1:Dhcz716Kqg1RHNWos+N6MlXNkjNP2EwZQ0LukRKJfMs=
@@ -735,8 +735,8 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
 github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/dgraph-io/badger/v4 v4.9.4 h1:bcw+waCpzRZ2nmcSPbnPvDVhiEsn98TKmvnAhK7r7LM=
-github.com/dgraph-io/badger/v4 v4.9.4/go.mod h1:nJjaJTUOSsQEBhsq209FmwCvMJzEA3e74RjZw6V2pQI=
+github.com/dgraph-io/badger/v4 v4.9.5 h1:zT46OMrF3ntqsfI3ynKp7hUkQrGlcK2CX5psQmH0iW0=
+github.com/dgraph-io/badger/v4 v4.9.5/go.mod h1:Xa9dAupjbwAacupWFCpa6YEn9E1PjBXkfZYr2I/8aWg=
 github.com/dgraph-io/ristretto/v2 v2.4.0 h1:I/w09yLjhdcVD2QV192UJcq8dPBaAJb9pOuMyNy0XlU=
 github.com/dgraph-io/ristretto/v2 v2.4.0/go.mod h1:0KsrXtXvnv0EqnzyowllbVJB8yBonswa2lTCK2gGo9E=
 github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
@@ -814,8 +814,8 @@ github.com/franela/goreq v0.0.0-20171204163338-bcd34c9993f8/go.mod h1:ZhphrRTfi2
 github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
 github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
-github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
-github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
+github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
+github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
 github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9Ufpk2AcUM=
 github.com/gabriel-vasile/mimetype v1.4.13/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
@@ -1276,6 +1276,8 @@ github.com/mdlayher/packet v1.1.2/go.mod h1:GEu1+n9sG5VtiRE4SydOmX5GTwyyYlteZiFU
 github.com/mdlayher/socket v0.5.1 h1:VZaqt6RkGkt2OE9l3GcC6nZkqD3xKeQLyfleW/uBcos=
 github.com/mdlayher/socket v0.5.1/go.mod h1:TjPLHI1UgwEv5J1B5q0zTZq12A/6H7nKmtTanQE37IQ=
 github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
+github.com/miekg/dns v1.1.72 h1:vhmr+TF2A3tuoGNkLDFK9zi36F2LS+hKTRW0Uf8kbzI=
+github.com/miekg/dns v1.1.72/go.mod h1:+EuEPhdHOsfk6Wk5TT2CzssZdqkmFhf8r+aVyDEToIs=
 github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8/go.mod h1:mC1jAcsrzbxHt8iiaC+zU4b1ylILSosueou12R++wfY=
 github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3/go.mod h1:RagcQ7I8IeTMnF8JTXieKnO4Z6JCsikNEzj0DwauVzE=
 github.com/minio/crc64nvme v1.1.1 h1:8dwx/Pz49suywbO+auHCBpCtlW1OfpcLN7wYgVR6wAI=
@@ -1315,10 +1317,12 @@ github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELU
 github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
 github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
 github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
-github.com/mulgadc/predastore v1.12.1 h1:mkJqmUbxfI3kBRmtBecjvBsRLFpnKARGWtmx4b9XKKg=
-github.com/mulgadc/predastore v1.12.1/go.mod h1:kgTncuCaB11e3eCQeZF0ydEFG0biuOlrZ4DlHNzwvZY=
-github.com/mulgadc/viperblock v1.12.1 h1:DWMbzPhIkYzdhOy4sn/D0/BJWJ5uhBoKWbVMfS7DMl4=
-github.com/mulgadc/viperblock v1.12.1/go.mod h1:5MGciKEvqpwLv2pdjk9j7PR8NHtXPs4pJzJi4UJojeU=
+github.com/mulgadc/northstar v1.14.0 h1:8JzDcS4k3SN5PF3jBdB8x9FOWC24eITCClmOdwpoY8Q=
+github.com/mulgadc/northstar v1.14.0/go.mod h1:6EArbWIQUrCFQ+XZ5ZJrGS/ep/u/rF6l+dKZh5vVIRs=
+github.com/mulgadc/predastore v1.14.0 h1:vG+egKIP08SLql4hDUwke31ritValUYaDbyH39Ta6qw=
+github.com/mulgadc/predastore v1.14.0/go.mod h1:C/mlefzQgqRAXNMbKc/rmW5BVSoB3R0OPHsgOi/islI=
+github.com/mulgadc/viperblock v1.14.0 h1:Q6REPTPHh+3fnkjbyBSfLXWSppgiLsEcxwuvaHow0LY=
+github.com/mulgadc/viperblock v1.14.0/go.mod h1:5ENQb0OrjTplIRiLShRW/oNWOzbFvSCFgo+PkNH4858=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
@@ -1471,8 +1475,8 @@ github.com/pterm/pterm v0.12.83 h1:ie+YmGmA727VuhxBlyGr74Ks+7McV6kT99IB8EU80aA=
 github.com/pterm/pterm v0.12.83/go.mod h1:xlgc6bFWyJIMtmLJvGim+L7jhSReilOlOnodeIYe4Tk=
 github.com/quic-go/go-ossfuzz-seeds v0.1.0 h1:APacT+iIaNF6fd8AGEiN3bT/Jtkd2jz4v4TzM7MFjy0=
 github.com/quic-go/go-ossfuzz-seeds v0.1.0/go.mod h1:3IOHRbJIc+L6YKMwfDtJAM9Vj9k0YY4muhuyUYk5tbk=
-github.com/quic-go/quic-go v0.60.0 h1:xcQioE8OM66UQLeUMHltK1CCcOu3JbVB4JAQdDQSB+0=
-github.com/quic-go/quic-go v0.60.0/go.mod h1:wpKpjmPpftl30sL6pFh7REVpjbcCVy4zt2vDyK1TuJk=
+github.com/quic-go/quic-go v0.61.0 h1:ui88A53s8MSVYLC56en0KQ17HARk+9986Dn0SBfKNvA=
+github.com/quic-go/quic-go v0.61.0/go.mod h1:9So2anK4Tp22URSQq00k+Vo2PNkle96ycDPDHL4s9vs=
 github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
 github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
 github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
@@ -1542,6 +1546,7 @@ github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+
 github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
 github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
+github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
 github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
@@ -1808,6 +1813,8 @@ go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/
 go.uber.org/multierr v1.3.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4=
 go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU=
 go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU=
+go.uber.org/nilaway v0.0.0-20260721205819-4a0f7653488a h1:CsRd824LAzMmlpu9nYv2PHOqPQYv4Zjd4kGM+g0ahS0=
+go.uber.org/nilaway v0.0.0-20260721205819-4a0f7653488a/go.mod h1:i9Hi8GeZFLBtUCXP73ZNSHKZ4K2JtxT0pLmo3sbJzlo=
 go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA=
 go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
 go.uber.org/zap v1.13.0/go.mod h1:zwrFLgMcdUuIBviXEYEH1YKNaOBnKXsx2IPda5bBwHM=
@@ -1907,6 +1914,8 @@ golang.org/x/exp/shiny v0.0.0-20230801115018-d63ba01acd4b/go.mod h1:UH99kUObWAZk
 golang.org/x/exp/shiny v0.0.0-20230817173708-d852ddb80c63/go.mod h1:UH99kUObWAZkDnWqppdQe5ZhPYESUw8I0zVV1uWBR+0=
 golang.org/x/exp/shiny v0.0.0-20240707233637-46b078467d37/go.mod h1:3F+MieQB7dRYLTmnncoFbb1crS5lfQoTfDgQy6K4N0o=
 golang.org/x/exp/shiny v0.0.0-20241009180824-f66d83c29e7c/go.mod h1:3F+MieQB7dRYLTmnncoFbb1crS5lfQoTfDgQy6K4N0o=
+golang.org/x/exp/typeparams v0.0.0-20260611194520-c48552f49976 h1:GTD/WuaexTazIG/SxLOz4rEKZPDVilmVVC2nz4xhwfE=
+golang.org/x/exp/typeparams v0.0.0-20260611194520-c48552f49976/go.mod h1:PqrXSW65cXDZH0k4IeUbhmg/bcAZDbzNz3byBpKCsXo=
 golang.org/x/image v0.0.0-20180708004352-c73c2afc3b81/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs=
 golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
 golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
@@ -2570,8 +2579,8 @@ golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnps
 golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY=
 golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM=
 golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated/go.mod h1:RVAQXBGNv1ib0J382/DPCRS/BPnsGebyM1Gj5VSDpG8=
-golang.org/x/vuln v1.1.4 h1:Ju8QsuyhX3Hk8ma3CesTbO8vfJD9EvUBgHvkxHBzj0I=
-golang.org/x/vuln v1.1.4/go.mod h1:F+45wmU18ym/ca5PLTPLsSzr2KppzswxPP603ldA67s=
+golang.org/x/vuln v1.4.0 h1:FpmTZiV4PyqY3lFfuCkz1JftEXb/+8M2NEkjJM5TF4g=
+golang.org/x/vuln v1.4.0/go.mod h1:FJ7XyKs83nAdxQ7PMsia2PoynwZMJ/QajXVMBBIgFe8=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -3090,8 +3099,9 @@ google.golang.org/grpc v1.79.1/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhH
 google.golang.org/grpc v1.79.2/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ=
 google.golang.org/grpc v1.79.3/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ=
 google.golang.org/grpc v1.80.0/go.mod h1:ho/dLnxwi3EDJA4Zghp7k2Ec1+c2jqup0bFkw07bwF4=
-google.golang.org/grpc v1.81.1 h1:VnnIIZ88UzOOKLukQi+ImGz8O1Wdp8nAGGnvOfEIWQQ=
 google.golang.org/grpc v1.81.1/go.mod h1:xGH9GfzOyMTGIOXBJmXt+BX/V0kcdQbdcuwQ/zNw42I=
+google.golang.org/grpc v1.82.1 h1:NnAxzGRA0677vCa4BUkOAnO5+FfQqVl9iUXeD0IqcGE=
+google.golang.org/grpc v1.82.1/go.mod h1:yzTZ1TB1Z3SG+LIYaI+WiE8D5+PZ3ArnrSp8zF3+/ZA=
 google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0/go.mod h1:6Kw0yEErY5E/yWrBtf03jp27GLLJujG4z/JK95pnjjw=
 google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.3.0/go.mod h1:Dk1tviKTvMCz5tvh7t+fh94dhmQVHuCt2OzJB3CTW9Y=
 google.golang.org/grpc/examples v0.0.0-20201112215255-90f1b3ee835b/go.mod h1:IBqQ7wSUJ2Ep09a8rMWFsg4fmI2r38zwsq8a0GgxXpM=

scripts/build-system-image.sh

@@ -265,6 +265,15 @@ else
     CUST+=(--memsize 2048)
 fi
 
+# The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
+# (127.0.0.53), which isn't running inside the appliance chroot, so DNS is
+# dead until package installs need it. Point it at a real resolver first, then
+# restore the stub symlink after package installs so the shipped AMI still
+# gets its DNS from systemd-resolved at boot like a stock cloud image.
+if [[ -n "${APK_PACKAGES:-}${APT_PACKAGES:-}" ]]; then
+    CUST+=(--run-command 'rm -f /etc/resolv.conf; printf "nameserver 1.1.1.1\nnameserver 8.8.8.8\n" > /etc/resolv.conf')
+fi
+
 # Packages
 if [[ "$DISTRO" == "alpine" ]] && [[ -n "${APK_PACKAGES:-}" ]]; then
     echo "Will install packages: ${APK_PACKAGES}"
@@ -338,6 +347,12 @@ else
     CUST+=(--run-command 'export DEBIAN_FRONTEND=noninteractive; apt-get clean; rm -rf /var/lib/apt/lists/* /tmp/* 2>/dev/null || true')
 fi
 
+# Restore the stock systemd-resolved stub symlink so the shipped AMI gets its
+# DNS from the instance's own resolver at boot, not the static appliance one.
+if [[ "$DISTRO" == "ubuntu" ]] && [[ -n "${APK_PACKAGES:-}${APT_PACKAGES:-}${SETUP_SCRIPT:-}" ]]; then
+    CUST+=(--run-command 'rm -f /etc/resolv.conf; ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf')
+fi
+
 echo "Customizing image (libguestfs appliance)..."
 "${CUST[@]}"
 

scripts/images/ecs-agent-gpu/setup.sh

@@ -71,18 +71,21 @@ EOF
 
     apt-get install -y --no-install-recommends "linux-headers-${KVER}"
 
-    # Ubuntu 26.04+ no longer ships unversioned meta-packages; detect the
-    # latest available versioned NVIDIA server driver.
-    NVIDIA_VER=$(apt-cache search '^nvidia-dkms-[0-9]+-server$' 2>/dev/null \
-        | grep -oP '(?<=nvidia-dkms-)\d+(?=-server)' | sort -rn | head -1)
-    if [ -z "${NVIDIA_VER}" ]; then
-        echo "ERROR: No versioned nvidia-dkms-*-server package found in apt cache"
-        apt-cache search nvidia-dkms || true
-        exit 1
-    fi
-    echo "[nvidia-gpu-stack] installing NVIDIA server driver version: ${NVIDIA_VER}"
+    # Ubuntu 26.04+ no longer ships unversioned meta-packages, so the branch is
+    # named explicitly. Pinned rather than resolved at build time: the number in
+    # the package name is a branch alias, not a driver version (nvidia-dkms-535-server
+    # installs 580.126.20), so picking the highest number sorts on a value that
+    # need not match what gets installed. Resolving also let identical source
+    # produce different drivers on different days. Bumping is a deliberate edit.
+    NVIDIA_VER=595
+    echo "[nvidia-gpu-stack] installing NVIDIA server driver branch: ${NVIDIA_VER}"
+    # The -open (GSP/open-kernel-module) DKMS variant, not the closed/proprietary
+    # one: Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
+    # initialize under the closed kernel module — RmInitAdapter fails with
+    # "requires use of the NVIDIA open kernel modules" (dmesg NVRM error 0x22).
+    # nvidia-utils has no separate open variant (userspace tools only).
     apt-get install -y --no-install-recommends \
-        "nvidia-dkms-${NVIDIA_VER}-server" \
+        "nvidia-dkms-${NVIDIA_VER}-server-open" \
         "nvidia-utils-${NVIDIA_VER}-server"
 
     NVIDIA_DKMS_NAME=$(dkms status 2>/dev/null | awk -F'[,/ ]+' '/nvidia/{print $1; exit}')

scripts/images/eks-node-gpu/setup.sh

@@ -90,18 +90,21 @@ EOF
 
     apt-get install -y --no-install-recommends "linux-headers-${KVER}"
 
-    # Ubuntu 26.04+ no longer ships unversioned meta-packages; detect the
-    # latest available versioned NVIDIA server driver.
-    NVIDIA_VER=$(apt-cache search '^nvidia-dkms-[0-9]+-server$' 2>/dev/null \
-        | grep -oP '(?<=nvidia-dkms-)\d+(?=-server)' | sort -rn | head -1)
-    if [ -z "${NVIDIA_VER}" ]; then
-        echo "ERROR: No versioned nvidia-dkms-*-server package found in apt cache"
-        apt-cache search nvidia-dkms || true
-        exit 1
-    fi
-    echo "[nvidia-gpu-stack] installing NVIDIA server driver version: ${NVIDIA_VER}"
+    # Ubuntu 26.04+ no longer ships unversioned meta-packages, so the branch is
+    # named explicitly. Pinned rather than resolved at build time: the number in
+    # the package name is a branch alias, not a driver version (nvidia-dkms-535-server
+    # installs 580.126.20), so picking the highest number sorts on a value that
+    # need not match what gets installed. Resolving also let identical source
+    # produce different drivers on different days. Bumping is a deliberate edit.
+    NVIDIA_VER=595
+    echo "[nvidia-gpu-stack] installing NVIDIA server driver branch: ${NVIDIA_VER}"
+    # The -open (GSP/open-kernel-module) DKMS variant, not the closed/proprietary
+    # one: Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
+    # initialize under the closed kernel module — RmInitAdapter fails with
+    # "requires use of the NVIDIA open kernel modules" (dmesg NVRM error 0x22).
+    # nvidia-utils has no separate open variant (userspace tools only).
     apt-get install -y --no-install-recommends \
-        "nvidia-dkms-${NVIDIA_VER}-server" \
+        "nvidia-dkms-${NVIDIA_VER}-server-open" \
         "nvidia-utils-${NVIDIA_VER}-server"
 
     NVIDIA_DKMS_NAME=$(dkms status 2>/dev/null | awk -F'[,/ ]+' '/nvidia/{print $1; exit}')

scripts/images/eks-node/addons/argocd/3.0.23/argocd.yaml

@@ -24939,6 +24939,17 @@ spec:
       labels:
         app.kubernetes.io/name: argocd-applicationset-controller
     spec:
+      # Local delta over the vendored upstream manifest: a cluster with a
+      # GPU-only nodegroup has no untainted node — the CP carries
+      # CriticalAddonsOnly and every worker carries nvidia.com/gpu, so without
+      # these the pod stays Pending forever. Upstream never needs it because
+      # real EKS control planes are not schedulable nodes.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       containers:
       - args:
         - /usr/local/bin/argocd-applicationset-controller
@@ -25180,6 +25191,17 @@ spec:
       labels:
         app.kubernetes.io/name: argocd-dex-server
     spec:
+      # Local delta over the vendored upstream manifest: a cluster with a
+      # GPU-only nodegroup has no untainted node — the CP carries
+      # CriticalAddonsOnly and every worker carries nvidia.com/gpu, so without
+      # these the pod stays Pending forever. Upstream never needs it because
+      # real EKS control planes are not schedulable nodes.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
@@ -25304,6 +25326,17 @@ spec:
       labels:
         app.kubernetes.io/name: argocd-notifications-controller
     spec:
+      # Local delta over the vendored upstream manifest: a cluster with a
+      # GPU-only nodegroup has no untainted node — the CP carries
+      # CriticalAddonsOnly and every worker carries nvidia.com/gpu, so without
+      # these the pod stays Pending forever. Upstream never needs it because
+      # real EKS control planes are not schedulable nodes.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       containers:
       - args:
         - /usr/local/bin/argocd-notifications
@@ -25403,6 +25436,17 @@ spec:
       labels:
         app.kubernetes.io/name: argocd-redis
     spec:
+      # Local delta over the vendored upstream manifest: a cluster with a
+      # GPU-only nodegroup has no untainted node — the CP carries
+      # CriticalAddonsOnly and every worker carries nvidia.com/gpu, so without
+      # these the pod stays Pending forever. Upstream never needs it because
+      # real EKS control planes are not schedulable nodes.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
@@ -25486,6 +25530,17 @@ spec:
       labels:
         app.kubernetes.io/name: argocd-repo-server
     spec:
+      # Local delta over the vendored upstream manifest: a cluster with a
+      # GPU-only nodegroup has no untainted node — the CP carries
+      # CriticalAddonsOnly and every worker carries nvidia.com/gpu, so without
+      # these the pod stays Pending forever. Upstream never needs it because
+      # real EKS control planes are not schedulable nodes.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
@@ -25840,6 +25895,17 @@ spec:
       labels:
         app.kubernetes.io/name: argocd-server
     spec:
+      # Local delta over the vendored upstream manifest: a cluster with a
+      # GPU-only nodegroup has no untainted node — the CP carries
+      # CriticalAddonsOnly and every worker carries nvidia.com/gpu, so without
+      # these the pod stays Pending forever. Upstream never needs it because
+      # real EKS control planes are not schedulable nodes.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
@@ -26251,6 +26317,17 @@ spec:
       labels:
         app.kubernetes.io/name: argocd-application-controller
     spec:
+      # Local delta over the vendored upstream manifest: a cluster with a
+      # GPU-only nodegroup has no untainted node — the CP carries
+      # CriticalAddonsOnly and every worker carries nvidia.com/gpu, so without
+      # these the pod stays Pending forever. Upstream never needs it because
+      # real EKS control planes are not schedulable nodes.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:

scripts/images/eks-node/addons/aws-ebs-csi-driver/1.40.1/10-controller.yaml

@@ -44,6 +44,18 @@ spec:
         kubernetes.io/os: linux
       serviceAccountName: ebs-csi-controller-sa
       priorityClassName: system-cluster-critical
+      # A cluster with a GPU-only nodegroup has no untainted node: the CP carries
+      # CriticalAddonsOnly (EKS parity keeps user workloads off it) and every
+      # worker carries nvidia.com/gpu (nodegroup_userdata.go). Without these
+      # tolerations the controller has nowhere to schedule and stays Pending
+      # forever. Real EKS never needs this: its control plane isn't a schedulable
+      # node, so the stock upstream manifest carries no such toleration.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       affinity:
         nodeAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:

scripts/images/eks-node/addons/aws-load-balancer-controller/2.11.0/10-aws-load-balancer-controller.yaml

@@ -348,6 +348,18 @@ spec:
           readOnly: true
 {{IRSA_VOLUME_MOUNT}}
       priorityClassName: system-cluster-critical
+      # A cluster with a GPU-only nodegroup has no untainted node: the CP carries
+      # CriticalAddonsOnly (EKS parity keeps user workloads off it) and every
+      # worker carries nvidia.com/gpu (nodegroup_userdata.go). Without these
+      # tolerations the controller has nowhere to schedule and stays Pending
+      # forever. Real EKS never needs this: its control plane isn't a schedulable
+      # node, so the stock upstream manifest carries no such toleration.
+      tolerations:
+      - key: CriticalAddonsOnly
+        operator: Exists
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
       securityContext:
         fsGroup: 1337
       serviceAccountName: aws-load-balancer-controller

scripts/images/eks-node/mulga-vpc-mtu.sh

@@ -4,13 +4,27 @@
 #
 # Spinifex carries tenant traffic over an OVN geneve underlay (~1442B usable on a
 # 1500B host link). The EKS node's primary ENI (eth0) comes up at the default
-# 1500 — cloud-init's Ec2 datasource has no MTU leaf and Alpine's udhcpc ignores
-# DHCP option 26 — so flannel derives flannel.1 = 1500-50 = 1450. VXLAN frames up
-# to 1500 then exceed the underlay and are silently dropped: small packets (TCP
-# handshakes) pass while large ones (TLS, kubelet streams, pod-to-pod payloads)
-# blackhole. Pinning eth0 down to 1320 (flannel.1 = 1270) leaves margin under the
-# underlay and restores the datapath. This must run before k3s starts flannel,
-# which reads the iface MTU once at startup.
+# 1500 — cloud-init's Ec2 datasource has no MTU leaf — so flannel derives
+# flannel.1 = 1500-50 = 1450. VXLAN frames up to 1500 then exceed the underlay and
+# are silently dropped: small packets (TCP handshakes) pass while large ones (TLS,
+# kubelet streams, pod-to-pod payloads) blackhole. Pinning eth0 down to 1320
+# (flannel.1 = 1270) leaves margin under the underlay and restores the datapath.
+# This must run before k3s starts flannel, which reads the iface MTU once at
+# startup.
+#
+# Pinning the LINK is not sufficient on its own. This image runs dhcpcd, not
+# udhcpc (an earlier version of this comment had that backwards), and dhcpcd
+# honours DHCP option 26 by stamping the advertised MTU onto every route it
+# installs:
+#
+#   default via 10.32.101.1 dev eth0 proto dhcp src 10.32.101.7 metric 1002 mtu 1442
+#
+# Linux prefers a route's MTU metric over the device MTU (dst_mtu()), so the node
+# still advertises an MSS derived from 1442 no matter what the link says, and
+# large inbound segments blackhole in the HOST netns — which is where containerd
+# pulls run, so image pulls fail with `TLS handshake timeout` while pod traffic
+# (correctly sized off cni0) is fine. Pod netns working while the node netns does
+# not is the signature. Re-stamp the routes to match the pinned link.
 #
 # eth0 is always the primary data ENI (cloud-init set-name; mgmt0, when present,
 # is a separate NIC and is left at its default). The default-route interface is
@@ -30,9 +44,47 @@ pin() {
     fi
 }
 
+# Rewrite the MTU metric dhcpcd stamped onto the routes of one interface, so no
+# route can advertise a larger MSS than the pinned link carries. Routes with no
+# MTU metric inherit the device MTU and are left alone.
+#
+# `ip route show dev X` omits the "dev X" token from each line, so it is added
+# back before replacing; a "default" line already carries its own dev.
+restamp() {
+    iface="$1"
+    [ -n "$iface" ] || return 0
+    ip -4 route show dev "$iface" 2>/dev/null | while read -r line; do
+        case "$line" in
+            *" mtu "*) ;;
+            *) continue ;;
+        esac
+        cur=$(echo "$line" | sed -n 's/.* mtu \([0-9]*\).*/\1/p')
+        [ -n "$cur" ] || continue
+        [ "$cur" != "$MTU" ] || continue
+        spec=$(echo "$line" | sed "s/ mtu $cur/ mtu $MTU/")
+        case "$spec" in
+            *" dev "*) ;;
+            *) spec="$spec dev $iface" ;;
+        esac
+        # Word splitting is intended: $spec is a route specification, not a path.
+        # shellcheck disable=SC2086
+        if ip route replace $spec; then
+            echo "[mulga-vpc-mtu] $iface route MTU $cur -> $MTU ($line)"
+        else
+            echo "[mulga-vpc-mtu] WARNING: failed to restamp $iface route: $line" >&2
+        fi
+    done
+}
+
 pin eth0
 
 route_iface=$(ip -4 route show default 2>/dev/null | awk '{print $5; exit}')
 [ "$route_iface" = "eth0" ] || pin "$route_iface"
 
+# setup.sh writes `nooption interface_mtu` to dhcpcd.conf so no future lease can
+# re-stamp a route MTU; this pass fixes routes installed before that took effect
+# (an already-running dhcpcd, or an image built before the fix).
+restamp eth0
+[ "$route_iface" = "eth0" ] || restamp "$route_iface"
+
 exit 0

scripts/images/eks-node/mulga-vpc-mtu_test.sh

@@ -0,0 +1,118 @@
+#!/bin/sh
+# Self-contained POSIX test for mulga-vpc-mtu.sh route re-stamping. No root or
+# real interfaces needed: `ip` is stubbed on PATH, serving fixture route output
+# and recording every mutating call.
+#
+# Covers the fault this script exists to prevent: dhcpcd stamps the DHCP MTU
+# (option 26) onto its routes, Linux prefers a route's MTU metric over the device
+# MTU, so pinning only the link leaves the node advertising an MSS too large for
+# the path and large inbound segments blackhole in the host netns.
+#
+# Run: sh scripts/images/eks-node/mulga-vpc-mtu_test.sh
+set -eu
+
+SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)
+PINNER="${SCRIPT_DIR}/mulga-vpc-mtu.sh"
+WORK=$(mktemp -d)
+trap 'rm -rf "${WORK}"' EXIT
+
+STUBBIN="${WORK}/bin"
+mkdir -p "${STUBBIN}"
+
+# Stub `ip`: serves ${ROUTES_DEFAULT} / ${ROUTES_DEV} for show, records mutations
+# ("link set …" / "route replace …") to ${CALLS}. Anything else is a no-op.
+cat > "${STUBBIN}/ip" <<'STUB'
+#!/bin/sh
+args="$*"
+case "$args" in
+    *"route show default"*) cat "${ROUTES_DEFAULT}"; exit 0 ;;
+    *"route show dev "*)    cat "${ROUTES_DEV}"; exit 0 ;;
+    "link show dev "*)      exit 0 ;;
+    "link set dev "*)       echo "$args" >> "${CALLS}"; exit 0 ;;
+    "route replace "*)      echo "$args" >> "${CALLS}"; exit 0 ;;
+esac
+exit 0
+STUB
+chmod +x "${STUBBIN}/ip"
+
+PASS=0
+FAIL=0
+check() { # check <desc> <expected> <actual>
+    if [ "$2" = "$3" ]; then
+        PASS=$((PASS + 1))
+    else
+        FAIL=$((FAIL + 1))
+        echo "FAIL: $1"
+        echo "  expected: [$2]"
+        echo "  actual:   [$3]"
+    fi
+}
+
+# run_case <name> <default-routes> <dev-routes>: runs the pinner against fixture
+# route tables and leaves the recorded mutations in ${CALLS}.
+run_case() {
+    CASE="${WORK}/$1"
+    mkdir -p "${CASE}"
+    CALLS="${CASE}/calls"
+    ROUTES_DEFAULT="${CASE}/default"
+    ROUTES_DEV="${CASE}/dev"
+    : > "${CALLS}"
+    printf '%s\n' "$2" > "${ROUTES_DEFAULT}"
+    printf '%s\n' "$3" > "${ROUTES_DEV}"
+    export CALLS ROUTES_DEFAULT ROUTES_DEV
+    PATH="${STUBBIN}:${PATH}" sh "${PINNER}" >/dev/null 2>&1
+}
+
+replaces() { grep -c '^route replace' "${CALLS}" 2>/dev/null | head -1; }
+
+# 1. The real-world fault: dhcpcd's 1442 on the default route is rewritten to the
+#    pinned link MTU, and the rest of the route spec is preserved verbatim.
+run_case dhcpcd-1442 \
+    'default via 10.32.101.1 dev eth0 proto dhcp src 10.32.101.7 metric 1002 mtu 1442' \
+    'default via 10.32.101.1 proto dhcp src 10.32.101.7 metric 1002 mtu 1442'
+check "link is pinned to 1320" \
+    "link set dev eth0 mtu 1320" \
+    "$(grep '^link set' "${CALLS}" | head -1)"
+check "route MTU rewritten to the pinned link MTU" \
+    "route replace default via 10.32.101.1 proto dhcp src 10.32.101.7 metric 1002 mtu 1320 dev eth0" \
+    "$(grep '^route replace' "${CALLS}" | head -1)"
+
+# 2. `ip route show dev X` omits the dev token; it must be added back or the
+#    replace would be rejected (and the fault would silently persist).
+run_case dev-token-restored \
+    'default via 10.32.101.1 dev eth0 proto dhcp metric 1002 mtu 1442' \
+    '10.32.101.0/24 proto kernel scope link src 10.32.101.7 metric 1002 mtu 1442'
+check "dev token re-added to a dev-scoped route" \
+    "route replace 10.32.101.0/24 proto kernel scope link src 10.32.101.7 metric 1002 mtu 1320 dev eth0" \
+    "$(grep '^route replace' "${CALLS}" | head -1)"
+
+# 3. Idempotent: a route already at the pinned MTU is left alone, so a re-run (or
+#    a boot after the dhcpcd.conf fix) issues no route changes.
+run_case already-pinned \
+    'default via 10.32.101.1 dev eth0 proto dhcp metric 1002 mtu 1320' \
+    'default via 10.32.101.1 proto dhcp metric 1002 mtu 1320'
+check "no replace when the route MTU already matches" "0" "$(replaces)"
+
+# 4. A route with no MTU metric inherits the device MTU — leave it alone rather
+#    than pinning a value the link may later change.
+run_case no-mtu-metric \
+    'default via 10.32.101.1 dev eth0 proto dhcp metric 1002' \
+    'default via 10.32.101.1 proto dhcp metric 1002'
+check "no replace when the route carries no MTU metric" "0" "$(replaces)"
+
+# 5. The pinned value is overridable, and the override reaches the routes too.
+run_case custom-mtu \
+    'default via 10.32.101.1 dev eth0 proto dhcp metric 1002 mtu 1442' \
+    'default via 10.32.101.1 proto dhcp metric 1002 mtu 1442'
+CASE="${WORK}/custom-mtu-run"
+mkdir -p "${CASE}"
+CALLS="${CASE}/calls"
+: > "${CALLS}"
+export CALLS
+PATH="${STUBBIN}:${PATH}" MULGA_VPC_NIC_MTU=1408 sh "${PINNER}" >/dev/null 2>&1
+check "MULGA_VPC_NIC_MTU overrides both link and route" \
+    "route replace default via 10.32.101.1 proto dhcp metric 1002 mtu 1408 dev eth0" \
+    "$(grep '^route replace' "${CALLS}" | head -1)"
+
+echo "pass=${PASS} fail=${FAIL}"
+[ "${FAIL}" -eq 0 ]

scripts/images/eks-node/setup.sh

@@ -142,4 +142,16 @@ if ! grep -qxF 'noipv4ll' /etc/dhcpcd.conf 2>/dev/null; then
     echo 'noipv4ll' >> /etc/dhcpcd.conf
 fi
 
+# Ignore the DHCP MTU (option 26). dhcpcd honours it by stamping the advertised
+# value onto every route it installs, and Linux prefers a route's MTU metric over
+# the device MTU — so the subnet's advertised MTU wins over the link MTU that
+# mulga-vpc-mtu pins, and the node advertises an MSS too large for the real path.
+# Large inbound segments then blackhole in the host netns (image pulls fail with
+# `TLS handshake timeout`) while pod traffic, sized off cni0, is unaffected.
+# mulga-vpc-mtu is the single source of truth for this node's MTU; this stops
+# dhcpcd re-stamping a different one on every lease renewal.
+if ! grep -qxF 'nooption interface_mtu' /etc/dhcpcd.conf 2>/dev/null; then
+    echo 'nooption interface_mtu' >> /etc/dhcpcd.conf
+fi
+
 echo "[eks-node-setup] done"

scripts/images/ubuntu-gpu-nvidia-setup.sh

@@ -60,19 +60,22 @@ EOF
 apt-get install -y --no-install-recommends \
     "linux-headers-${KVER}"
 
-# Detect the latest available versioned NVIDIA server driver. Ubuntu 26.04+
-# no longer ships unversioned meta-packages (nvidia-dkms-server); the packages
-# are now versioned (e.g. nvidia-dkms-570-server).
-NVIDIA_VER=$(apt-cache search '^nvidia-dkms-[0-9]+-server$' 2>/dev/null \
-    | grep -oP '(?<=nvidia-dkms-)\d+(?=-server)' | sort -rn | head -1)
-if [[ -z "${NVIDIA_VER}" ]]; then
-    echo "ERROR: No versioned nvidia-dkms-*-server package found in apt cache"
-    apt-cache search nvidia-dkms || true
-    exit 1
-fi
-echo "Installing NVIDIA server driver version: ${NVIDIA_VER}"
+# Ubuntu 26.04+ no longer ships unversioned meta-packages (nvidia-dkms-server);
+# the packages are versioned (e.g. nvidia-dkms-570-server), so the branch is
+# named explicitly. Pinned rather than resolved at build time: the number in the
+# package name is a branch alias, not a driver version (nvidia-dkms-535-server
+# installs 580.126.20), so picking the highest number sorts on a value that need
+# not match what gets installed. Resolving also let identical source produce
+# different drivers on different days. Bumping is a deliberate edit.
+NVIDIA_VER=595
+echo "Installing NVIDIA server driver branch: ${NVIDIA_VER}"
+# The -open (GSP/open-kernel-module) DKMS variant, not the closed/proprietary
+# one: Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
+# initialize under the closed kernel module — RmInitAdapter fails with
+# "requires use of the NVIDIA open kernel modules" (dmesg NVRM error 0x22).
+# nvidia-utils has no separate open variant (userspace tools only).
 apt-get install -y --no-install-recommends \
-    "nvidia-dkms-${NVIDIA_VER}-server" \
+    "nvidia-dkms-${NVIDIA_VER}-server-open" \
     "nvidia-utils-${NVIDIA_VER}-server"
 
 # Detect the installed NVIDIA DKMS module name + version for explicit build.

scripts/reset-dev-env.sh

@@ -275,5 +275,41 @@ sleep 5
 echo "==> Building and installing microVM artifacts"
 cd "$PROJECT_ROOT" && make build-lb-agent install-microvm
 
+# --- Authorize SSH ingress on the default security group ---
+# Default SG denies all inbound, so smoke-test.sh's SSH probe times out.
+# Open port 22 on the default VPC's default SG before the smoke test runs.
+aws_as_user() { sudo -u "$INVOKING_USER" env HOME="$INVOKING_HOME" AWS_PROFILE=spinifex aws "$@"; }
+
+echo "==> Waiting for EC2 daemon"
+DAEMON_TIMEOUT=60
+DAEMON_ELAPSED=0
+while [ $DAEMON_ELAPSED -lt $DAEMON_TIMEOUT ]; do
+    if aws_as_user ec2 describe-security-groups --output text >/dev/null 2>&1; then
+        break
+    fi
+    sleep 2
+    DAEMON_ELAPSED=$((DAEMON_ELAPSED + 2))
+done
+if [ $DAEMON_ELAPSED -ge $DAEMON_TIMEOUT ]; then
+    echo "❌ EC2 daemon not ready after ${DAEMON_TIMEOUT}s"
+    exit 1
+fi
+
+echo "==> Authorizing SSH (port 22) on default security group"
+DEFAULT_VPC_ID=$(aws_as_user ec2 describe-vpcs \
+    --filters "Name=is-default,Values=true" \
+    --query 'Vpcs[0].VpcId' --output text)
+DEFAULT_SG_ID=$(aws_as_user ec2 describe-security-groups \
+    --filters "Name=vpc-id,Values=$DEFAULT_VPC_ID" "Name=group-name,Values=default" \
+    --query 'SecurityGroups[0].GroupId' --output text)
+if [ -z "$DEFAULT_SG_ID" ] || [ "$DEFAULT_SG_ID" = "None" ]; then
+    echo "❌ Could not locate default security group in VPC $DEFAULT_VPC_ID"
+    exit 1
+fi
+aws_as_user ec2 authorize-security-group-ingress \
+    --group-id "$DEFAULT_SG_ID" --protocol tcp --port 22 --cidr 0.0.0.0/0 \
+    >/dev/null 2>&1 || true
+echo "  Authorized SSH on $DEFAULT_SG_ID ($DEFAULT_VPC_ID)"
+
 # --- Smoke test ---
 sudo -u "$INVOKING_USER" "$SCRIPT_DIR/smoke-test.sh"

scripts/setup.sh

@@ -15,7 +15,8 @@
 #   VERBOSE                    Set to 1 to echo "[setup] <stage>" before each top-level step.
 #   SETUP_STAGES               Comma-separated subset of stages to run:
 #                                deps, aws, users, sudoers, files, directories,
-#                                env, systemd, logrotate, udev, fixown, migrations
+#                                env, systemd, logrotate, udev, fixown,
+#                                migrations
 #                              Unset = run every stage appropriate for the current mode.
 
 set -e
@@ -139,11 +140,12 @@ create_service_users() {
         [gw]="/var/lib/spinifex/awsgw"
         [daemon]="/var/lib/spinifex/spinifex"
         [storage]="/var/lib/spinifex/predastore"
+        [northstar]="/var/lib/spinifex/northstar"
         [viperblock]="/var/lib/spinifex/viperblock"
         [vpcd]="/var/lib/spinifex"
         [ui]="/var/lib/spinifex"
     )
-    for svc in nats gw daemon storage viperblock vpcd ui; do
+    for svc in nats gw daemon storage northstar viperblock vpcd ui; do
         local user="spinifex-${svc}"
         if ! id "$user" > /dev/null 2>&1; then
             $SUDO useradd --system --no-create-home \
@@ -175,7 +177,7 @@ create_service_users() {
     # Daemon consumes viperblock's NBD socket — join the producer-typed group.
     $SUDO usermod -aG spinifex-viperblock spinifex-daemon
 
-    info "Service users created (spinifex-{nats,gw,daemon,storage,viperblock,vpcd,ui})"
+    info "Service users created (spinifex-{nats,gw,daemon,storage,northstar,viperblock,vpcd,ui})"
 }
 
 # --- Install scoped sudoers rules ---
@@ -421,6 +423,12 @@ TMPEOF
     $SUDO chown "spinifex-storage:$SPINIFEX_GROUP" /etc/spinifex/predastore
     $SUDO chmod 0750 /etc/spinifex/predastore
 
+    # Northstar holds northstar.toml (bucket-scoped S3 creds, written 0600 by
+    # `spx admin init`); fix_file_ownership reassigns it to spinifex-northstar.
+    $SUDO mkdir -p /etc/spinifex/northstar
+    $SUDO chown "spinifex-northstar:$SPINIFEX_GROUP" /etc/spinifex/northstar
+    $SUDO chmod 0750 /etc/spinifex/northstar
+
     $SUDO mkdir -p /etc/spinifex/awsgw
     $SUDO chown "spinifex-gw:$SPINIFEX_GROUP" /etc/spinifex/awsgw
     $SUDO chmod 0750 /etc/spinifex/awsgw
@@ -446,6 +454,10 @@ TMPEOF
     $SUDO chown "spinifex-storage:$SPINIFEX_GROUP" /var/lib/spinifex/predastore
     $SUDO chmod 0700 /var/lib/spinifex/predastore
 
+    $SUDO mkdir -p /var/lib/spinifex/northstar
+    $SUDO chown "spinifex-northstar:$SPINIFEX_GROUP" /var/lib/spinifex/northstar
+    $SUDO chmod 0700 /var/lib/spinifex/northstar
+
     $SUDO mkdir -p /var/lib/spinifex/viperblock
     $SUDO chown "spinifex-viperblock:$SPINIFEX_GROUP" /var/lib/spinifex/viperblock
     $SUDO chmod 0700 /var/lib/spinifex/viperblock
@@ -517,6 +529,7 @@ fix_file_ownership() {
     for entry in \
         nats:spinifex-nats \
         predastore:spinifex-storage \
+        northstar:spinifex-northstar \
         spinifex:spinifex-daemon \
         viperblock:spinifex-viperblock \
         vpcd:spinifex-vpcd \
@@ -537,6 +550,10 @@ fix_file_ownership() {
         $SUDO chown -R "spinifex-storage:$SPINIFEX_GROUP" /etc/spinifex/predastore \
             || fatal "Failed to set ownership on /etc/spinifex/predastore"
     fi
+    if [ -d /etc/spinifex/northstar ]; then
+        $SUDO chown -R "spinifex-northstar:$SPINIFEX_GROUP" /etc/spinifex/northstar \
+            || fatal "Failed to set ownership on /etc/spinifex/northstar"
+    fi
     if [ -d /etc/spinifex/awsgw ]; then
         $SUDO chown -R "spinifex-gw:$SPINIFEX_GROUP" /etc/spinifex/awsgw \
             || fatal "Failed to set ownership on /etc/spinifex/awsgw"

spinifex/admin/admin.go

@@ -33,6 +33,12 @@ type RemoteNode struct {
 	Region   string
 	AZ       string
 	Services []string
+
+	// NorthstarConfigPath marks the peer as running northstar. Its emptiness is
+	// the only thing read from a peer's stanza — no code opens a peer's path —
+	// and it must be rendered for every northstar peer so each node derives the
+	// same nameserver set for the base zone seed.
+	NorthstarConfigPath string
 }
 
 type ConfigSettings struct {
@@ -108,6 +114,20 @@ type ConfigSettings struct {
 	// Empty means no key was provisioned and volumes are written cleartext
 	// (legacy mode); the template omits the field entirely in that case.
 	EncryptionKeyFile string
+
+	// Northstar (DNS) settings. The northstar service reads zones from a
+	// dedicated, read-only S3 bucket using bucket-scoped credentials rendered
+	// into predastore.toml ([[auth]]) and northstar.toml ([s3]).
+	NorthstarAccessKey      string
+	NorthstarSecretKey      string
+	NorthstarBucket         string // S3 bucket holding zone files (default "northstar")
+	NorthstarDefaultDomain  string // authoritative base domain (default "spx3.net")
+	NorthstarInternalDomain string // AWS-parity private zone (default "compute.internal")
+	NorthstarConfigPath     string // path to northstar.toml, rendered into spinifex.toml
+
+	// PoolDNSServers are the host-detected upstream DNS servers northstar forwards
+	// recursive queries to, rendered into northstar.toml's [recursion] nameservers.
+	PoolDNSServers []string
 }
 
 // PoolData is one [[network.external_pools]] block rendered into spinifex.toml.
@@ -143,7 +163,7 @@ type ConfigFile struct {
 func GenerateConfigFiles(configs []ConfigFile, configSettings ConfigSettings) error {
 	for _, cfg := range configs {
 		if err := GenerateConfigFile(cfg.Path, cfg.Template, configSettings); err != nil {
-			return fmt.Errorf("error creating %s: %v", cfg.Name, err)
+			return fmt.Errorf("error creating %s: %w", cfg.Name, err)
 		}
 		fmt.Printf("✅ Created: %s\n", cfg.Name)
 	}
@@ -356,6 +376,8 @@ func SetServiceOwnership() {
 		"/var/lib/spinifex/nats":       "spinifex-nats",
 		"/etc/spinifex/predastore":     "spinifex-storage",
 		"/var/lib/spinifex/predastore": "spinifex-storage",
+		"/etc/spinifex/northstar":      "spinifex-northstar",
+		"/var/lib/spinifex/northstar":  "spinifex-northstar",
 		"/etc/spinifex/viperblock":     "spinifex-viperblock",
 		"/var/lib/spinifex/spinifex":   "spinifex-daemon",
 		"/var/lib/spinifex/viperblock": "spinifex-viperblock",
@@ -487,6 +509,30 @@ func GenerateAWSSecretKey() (string, error) {
 	return base64.StdEncoding.EncodeToString(bytes), nil
 }
 
+// Northstar DNS defaults baked into config files at init time.
+const (
+	// NorthstarBucketName is the S3 bucket holding DNS zone files.
+	NorthstarBucketName = "northstar"
+	// NorthstarDefaultDomain is the authoritative base domain for internal names.
+	NorthstarDefaultDomain = "spx3.net"
+	// NorthstarInternalDomain is the AWS-parity private zone for ip-<addr> names.
+	NorthstarInternalDomain = "compute.internal"
+)
+
+// NorthstarCredentials is the bucket-scoped credential pair the northstar DNS
+// service reads zone files with, together with the bucket that scopes it.
+//
+// Every node in a cluster must be provisioned with the same pair: the zone
+// bucket is distributed, and each node's predastore only honours the keys
+// rendered into its own config, so a per-node pair would have node A's
+// predastore reject node B's resolver. A zero value provisions no northstar
+// credentials at all, which renders a config without any northstar stanza.
+type NorthstarCredentials struct {
+	AccessKey string
+	SecretKey string
+	Bucket    string
+}
+
 // SystemAccountID returns the system/root account ID (000000000000).
 // Used for service-to-service auth credentials baked into config files.
 func SystemAccountID() string {
@@ -504,7 +550,7 @@ func DefaultAccountName() string {
 	return "spinifex"
 }
 
-// GenerateNATSToken generates a secure random token for NATS
+// GenerateNATSToken generates a secure random token for NATS.
 func GenerateNATSToken() (string, error) {
 	bytes := make([]byte, 32)
 	if _, err := rand.Read(bytes); err != nil {
@@ -842,7 +888,12 @@ func SetupAWSCredentials(accessKey, secretKey, region, certPath, bindIP string)
 // GenerateMultiNodePredastoreConfig produces a complete predastore.toml for a
 // multi-node Predastore cluster. Each node gets its own DB entry (port 6660)
 // and shard entry (port 9991) on a distinct IP. Node ID 1 is the bootstrap leader.
-func GenerateMultiNodePredastoreConfig(templateStr string, nodes []PredastoreNodeConfig, accessKey, secretKey, region, natsToken, configDir, bindIP string, compactionIntervalSeconds int) (string, error) {
+//
+// A populated northstar credential provisions the zone bucket, grants the
+// system key write access to it, and adds the read-only entry the resolver
+// authenticates with. A zero value omits all three, yielding a config no
+// northstar service can use.
+func GenerateMultiNodePredastoreConfig(templateStr string, nodes []PredastoreNodeConfig, accessKey, secretKey, region, natsToken, configDir, bindIP string, compactionIntervalSeconds int, northstar NorthstarCredentials) (string, error) {
 	if len(nodes) < 2 {
 		return "", fmt.Errorf("multi-node predastore requires at least 2 nodes, got %d", len(nodes))
 	}
@@ -856,7 +907,17 @@ func GenerateMultiNodePredastoreConfig(templateStr string, nodes []PredastoreNod
 		ConfigDir                 string
 		BindIP                    string
 		CompactionIntervalSeconds int
-	}{nodes, accessKey, secretKey, region, natsToken, configDir, bindIP, compactionIntervalSeconds}
+		NorthstarAccessKey        string
+		NorthstarSecretKey        string
+		NorthstarBucket           string
+	}{
+		Nodes: nodes, AccessKey: accessKey, SecretKey: secretKey, Region: region,
+		NatsToken: natsToken, ConfigDir: configDir, BindIP: bindIP,
+		CompactionIntervalSeconds: compactionIntervalSeconds,
+		NorthstarAccessKey:        northstar.AccessKey,
+		NorthstarSecretKey:        northstar.SecretKey,
+		NorthstarBucket:           northstar.Bucket,
+	}
 
 	tmpl, err := template.New("predastore-multinode").Parse(templateStr)
 	if err != nil {

spinifex/admin/admin_test.go

@@ -450,7 +450,7 @@ func TestGenerateSignedCert(t *testing.T) {
 		baseBlock, _ := pem.Decode(basePEM)
 		baseParsed, _ := x509.ParseCertificate(baseBlock.Bytes)
 
-		assert.Equal(t, len(baseParsed.IPAddresses), len(cert.IPAddresses),
+		assert.Len(t, cert.IPAddresses, len(baseParsed.IPAddresses),
 			"passing duplicate/special IPs should not add extra entries")
 	})
 
@@ -640,6 +640,8 @@ func TestGenerateSignedCert_IncludesHostname(t *testing.T) {
 // --- Certificate orchestrator ---
 
 // TestGenerateCertificatesIfNeeded uses subtests to share the initial generation.
+//
+//nolint:tparallel // subtests mutate the shared dir in order: SkipsWhenAllExist pins the CA modtime that ForcePreservesCARegeneratesServerCert then regenerates against
 func TestGenerateCertificatesIfNeeded(t *testing.T) {
 	t.Parallel()
 	dir := t.TempDir()
@@ -710,6 +712,7 @@ func TestGenerateServerCertOnly(t *testing.T) {
 	require.NoError(t, GenerateCACert(filepath.Join(caDir, "ca.pem"), filepath.Join(caDir, "ca.key")))
 
 	t.Run("Success", func(t *testing.T) {
+		t.Parallel()
 		dir := t.TempDir()
 		// Copy CA files into test dir
 		caCert, _ := os.ReadFile(filepath.Join(caDir, "ca.pem"))
@@ -744,6 +747,7 @@ func TestGenerateServerCertOnly(t *testing.T) {
 	})
 
 	t.Run("MissingCA", func(t *testing.T) {
+		t.Parallel()
 		dir := t.TempDir()
 		err := GenerateServerCertOnly(dir, "10.0.0.5", "us-east-1", "spinifex.internal")
 		assert.Error(t, err)
@@ -806,18 +810,49 @@ host = "{{.Host}}"
 		{ID: 3, Host: "10.0.0.3"},
 	}
 
-	result, err := GenerateMultiNodePredastoreConfig(tmpl, nodes, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1", 0)
+	result, err := GenerateMultiNodePredastoreConfig(tmpl, nodes, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1", 0, NorthstarCredentials{})
 	require.NoError(t, err)
 	assert.Contains(t, result, `host = "10.0.0.1"`)
 	assert.Contains(t, result, `host = "10.0.0.3"`)
 }
 
+// The northstar template fields were declared but never assigned, so every
+// multi-node predastore config silently rendered the empty-key path: no zone
+// bucket and no credential the resolver could authenticate with.
+func TestGenerateMultiNodePredastoreConfig_NorthstarCredentialsReachTemplate(t *testing.T) {
+	tmpl := `access = "{{.NorthstarAccessKey}}" secret = "{{.NorthstarSecretKey}}" bucket = "{{.NorthstarBucket}}"`
+	nodes := []PredastoreNodeConfig{
+		{ID: 1, Host: "10.0.0.1"},
+		{ID: 2, Host: "10.0.0.2"},
+	}
+
+	result, err := GenerateMultiNodePredastoreConfig(tmpl, nodes, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1", 0,
+		NorthstarCredentials{AccessKey: "NSAK", SecretKey: "NSSK", Bucket: "northstar"})
+	require.NoError(t, err)
+	assert.Equal(t, `access = "NSAK" secret = "NSSK" bucket = "northstar"`, result)
+}
+
+// A zero credential must leave every northstar field empty, which is what the
+// production template's guards key off to omit the stanzas entirely.
+func TestGenerateMultiNodePredastoreConfig_NoNorthstarCredentials(t *testing.T) {
+	tmpl := `access = "{{.NorthstarAccessKey}}" secret = "{{.NorthstarSecretKey}}" bucket = "{{.NorthstarBucket}}"`
+	nodes := []PredastoreNodeConfig{
+		{ID: 1, Host: "10.0.0.1"},
+		{ID: 2, Host: "10.0.0.2"},
+	}
+
+	result, err := GenerateMultiNodePredastoreConfig(tmpl, nodes, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1", 0,
+		NorthstarCredentials{})
+	require.NoError(t, err)
+	assert.Equal(t, `access = "" secret = "" bucket = ""`, result)
+}
+
 func TestGenerateMultiNodePredastoreConfig_MinimumNodes(t *testing.T) {
 	tmpl := "{{range .Nodes}}{{.ID}}{{end}}"
 
 	_, err := GenerateMultiNodePredastoreConfig(tmpl, []PredastoreNodeConfig{
 		{ID: 1, Host: "10.0.0.1"},
-	}, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1", 0)
+	}, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1", 0, NorthstarCredentials{})
 	assert.Error(t, err)
 	assert.Contains(t, err.Error(), "at least 2 nodes")
 }
@@ -825,7 +860,7 @@ func TestGenerateMultiNodePredastoreConfig_MinimumNodes(t *testing.T) {
 func TestGenerateMultiNodePredastoreConfig_InvalidTemplate(t *testing.T) {
 	_, err := GenerateMultiNodePredastoreConfig("{{.Unclosed", []PredastoreNodeConfig{
 		{ID: 1, Host: "a"}, {ID: 2, Host: "b"}, {ID: 3, Host: "c"},
-	}, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1", 0)
+	}, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1", 0, NorthstarCredentials{})
 	assert.Error(t, err)
 	assert.Contains(t, err.Error(), "failed to parse")
 }

spinifex/admin/images_remove.go

@@ -320,7 +320,7 @@ func readAMIConfig(store objectstore.ObjectStore, bucket, imageID string) (viper
 
 	var state viperblock.VBState
 	if err := json.Unmarshal(viperblock.StateBody(body), &state); err != nil {
-		return viperblock.AMIMetadata{}, fmt.Errorf("%w: %s: %v", handlers_ec2_image.ErrCorruptAMIConfig, key, err)
+		return viperblock.AMIMetadata{}, fmt.Errorf("%w: %s: %w", handlers_ec2_image.ErrCorruptAMIConfig, key, err)
 	}
 	return state.VolumeConfig.AMIMetadata, nil
 }
@@ -352,7 +352,7 @@ func readVolumeConfig(store objectstore.ObjectStore, bucket, volumeID string) (*
 
 	var w volumeConfigWrapper
 	if err := json.Unmarshal(viperblock.StateBody(body), &w); err != nil {
-		return nil, fmt.Errorf("%w: %s: %v", errCorruptVolumeConfig, key, err)
+		return nil, fmt.Errorf("%w: %s: %w", errCorruptVolumeConfig, key, err)
 	}
 	return &w.VolumeConfig, nil
 }

spinifex/admin/images_remove_test.go

@@ -3,7 +3,6 @@ package admin
 import (
 	"bytes"
 	"encoding/json"
-	"errors"
 	"testing"
 
 	"github.com/aws/aws-sdk-go/aws"
@@ -157,7 +156,7 @@ func TestRemoveSystemImage_AccountOwned_ForceBypasses(t *testing.T) {
 
 	res, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id, Force: true})
 	require.NoError(t, err)
-	assert.Greater(t, res.ObjectsDeleted, 0)
+	assert.Positive(t, res.ObjectsDeleted)
 	assert.Equal(t, 0, store.Count())
 }
 
@@ -196,7 +195,7 @@ func TestRemoveSystemImage_DependentVolume_Direct_Refused(t *testing.T) {
 	_, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
 	require.Error(t, err)
 	var depErr *DependentError
-	require.True(t, errors.As(err, &depErr))
+	require.ErrorAs(t, err, &depErr)
 	assert.Contains(t, depErr.Dependents.Volumes, "vol-aaa")
 }
 
@@ -214,7 +213,7 @@ func TestRemoveSystemImage_DependentVolume_Transitive_Refused(t *testing.T) {
 	_, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
 	require.Error(t, err)
 	var depErr *DependentError
-	require.True(t, errors.As(err, &depErr))
+	require.ErrorAs(t, err, &depErr)
 	assert.Contains(t, depErr.Dependents.Volumes, "vol-bbb")
 	assert.Contains(t, depErr.Dependents.Snapshots, derivedSnap)
 }
@@ -232,7 +231,7 @@ func TestRemoveSystemImage_DependentAMI_Refused(t *testing.T) {
 	_, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
 	require.Error(t, err)
 	var depErr *DependentError
-	require.True(t, errors.As(err, &depErr))
+	require.ErrorAs(t, err, &depErr)
 	assert.Contains(t, depErr.Dependents.AMIs, "ami-acct-002")
 }
 
@@ -245,7 +244,7 @@ func TestRemoveSystemImage_Force_OverridesDependents(t *testing.T) {
 
 	res, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id, Force: true})
 	require.NoError(t, err)
-	assert.Greater(t, res.ObjectsDeleted, 0)
+	assert.Positive(t, res.ObjectsDeleted)
 	// vol-orphan/config.json remains; the AMI is gone.
 	_, err = store.GetObject(t.Context(), &awss3.GetObjectInput{
 		Bucket: aws.String(testRemoveBucket),

spinifex/admin/ipsec_test.go

@@ -14,6 +14,8 @@ import (
 
 // TestGenerateIPSecPeerCert shares a CA across subtests so the 4096-bit
 // CA-key generation runs once (~0.7s).
+//
+//nolint:tparallel // the parent cannot be parallel: it overrides the charonCATrustDir/charonRereadCAs package vars for the duration of the test
 func TestGenerateIPSecPeerCert(t *testing.T) {
 	// Redirect the charon trust-store symlink at a temp dir so the test
 	// doesn't try to write to /etc/ipsec.d/cacerts on the host. Restored

spinifex/admin/telemetry_test.go

@@ -10,7 +10,6 @@ import (
 	"time"
 
 	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
 )
 
 func TestReadMachineID_ReturnsNonEmpty(t *testing.T) {
@@ -29,8 +28,14 @@ func TestSendTelemetry_PostsCorrectPayload(t *testing.T) {
 		assert.Equal(t, "application/json", r.Header.Get("Content-Type"))
 
 		body, err := io.ReadAll(r.Body)
-		require.NoError(t, err)
-		require.NoError(t, json.Unmarshal(body, &received))
+		if !assert.NoError(t, err) {
+			http.Error(w, "read request body", http.StatusBadRequest)
+			return
+		}
+		if !assert.NoError(t, json.Unmarshal(body, &received)) {
+			http.Error(w, "decode request body", http.StatusBadRequest)
+			return
+		}
 
 		w.WriteHeader(http.StatusNoContent)
 	}))

spinifex/awsec2query/parser_test.go

@@ -110,7 +110,7 @@ func TestQueryParamsToStruct_StopInstances(t *testing.T) {
 	assert.Len(t, input.InstanceIds, 2)
 	assert.Equal(t, "i-1234567890abcdef0", aws.StringValue(input.InstanceIds[0]))
 	assert.Equal(t, "i-0987654321fedcba0", aws.StringValue(input.InstanceIds[1]))
-	assert.Equal(t, true, aws.BoolValue(input.Force))
+	assert.True(t, aws.BoolValue(input.Force))
 }
 
 func TestQueryParamsToStruct_StartInstances(t *testing.T) {
@@ -249,7 +249,7 @@ func TestQueryParamsToStruct_BlockDeviceMappingsWithEBS(t *testing.T) {
 	assert.NotNil(t, bdm.Ebs)
 	assert.Equal(t, int64(100), aws.Int64Value(bdm.Ebs.VolumeSize))
 	assert.Equal(t, "gp3", aws.StringValue(bdm.Ebs.VolumeType))
-	assert.Equal(t, true, aws.BoolValue(bdm.Ebs.DeleteOnTermination))
+	assert.True(t, aws.BoolValue(bdm.Ebs.DeleteOnTermination))
 }
 
 func TestQueryParamsToStruct_BlockDeviceMappingsWithEphemeral(t *testing.T) {
@@ -312,8 +312,8 @@ func TestQueryParamsToStruct_MultipleBlockDeviceMappings(t *testing.T) {
 	assert.Equal(t, "snap-1234567890abcdef0", aws.StringValue(bdm1.Ebs.SnapshotId))
 	assert.Equal(t, int64(30), aws.Int64Value(bdm1.Ebs.VolumeSize))
 	assert.Equal(t, "gp3", aws.StringValue(bdm1.Ebs.VolumeType))
-	assert.Equal(t, true, aws.BoolValue(bdm1.Ebs.Encrypted))
-	assert.Equal(t, true, aws.BoolValue(bdm1.Ebs.DeleteOnTermination))
+	assert.True(t, aws.BoolValue(bdm1.Ebs.Encrypted))
+	assert.True(t, aws.BoolValue(bdm1.Ebs.DeleteOnTermination))
 
 	// Second mapping - Empty EBS
 	bdm2 := input.BlockDeviceMappings[1]
@@ -472,7 +472,7 @@ func TestQueryParamsToStruct_LocationNameDryRun(t *testing.T) {
 	err := QueryParamsToStruct(args, input)
 
 	assert.NoError(t, err)
-	assert.Equal(t, true, aws.BoolValue(input.DryRun))
+	assert.True(t, aws.BoolValue(input.DryRun))
 }
 
 func TestQueryParamsToStruct_NonStructPointer(t *testing.T) {

spinifex/awserrors/awserrors.go

@@ -449,7 +449,7 @@ var (
 	ErrorVpnGatewayLimitExceeded                        = "VpnGatewayLimitExceeded"
 	ErrorZonesMismatched                                = "ZonesMismatched"
 
-	// IAM-specific error codes
+	// IAM-specific error codes.
 	ErrorIAMNoSuchEntity            = "NoSuchEntity"
 	ErrorIAMEntityAlreadyExists     = "EntityAlreadyExists"
 	ErrorIAMDeleteConflict          = "DeleteConflict"
@@ -458,7 +458,7 @@ var (
 	ErrorIAMMalformedPolicyDocument = "MalformedPolicyDocument"
 	ErrorAccessDenied               = "AccessDenied"
 
-	// ECR-specific error codes
+	// ECR-specific error codes.
 	ErrorRepositoryNotFound       = "RepositoryNotFoundException"
 	ErrorRepositoryPolicyNotFound = "RepositoryPolicyNotFoundException"
 	ErrorLifecyclePolicyNotFound  = "LifecyclePolicyNotFoundException"
@@ -473,7 +473,7 @@ var (
 	ErrorTooManyTags              = "TooManyTagsException"
 	ErrorOperationNotSupported    = "OperationNotSupportedException"
 
-	// ELBv2-specific error codes
+	// ELBv2-specific error codes.
 	ErrorELBv2LoadBalancerNotFound         = "LoadBalancerNotFound"
 	ErrorELBv2TargetGroupNotFound          = "TargetGroupNotFound"
 	ErrorELBv2ListenerNotFound             = "ListenerNotFound"
@@ -500,7 +500,7 @@ var (
 	ErrorELBv2CertificateNotFound          = "CertificateNotFound"
 	ErrorELBv2SSLPolicyNotFound            = "SSLPolicyNotFound"
 
-	// Bedrock/bedrock-runtime error codes
+	// Bedrock/bedrock-runtime error codes.
 	ErrorValidationException = "ValidationException"
 	// ErrorResourceNotFoundException aliases ErrorEKSResourceNotFound: both
 	// services use the same wire code, so it is not a second ErrorLookup entry.
@@ -522,6 +522,38 @@ func ValidErrorCode(code string) string {
 	return ErrorServerInternal
 }
 
+// ResolveErrorCode returns the first registered AWS error code in err's unwrap tree.
+func ResolveErrorCode(err error) (string, bool) {
+	if err == nil {
+		return "", false
+	}
+	code := err.Error()
+	if _, ok := ErrorLookup[code]; ok {
+		return code, true
+	}
+
+	if joined, ok := err.(interface{ Unwrap() []error }); ok {
+		for _, inner := range joined.Unwrap() {
+			if code, found := ResolveErrorCode(inner); found {
+				return code, true
+			}
+		}
+		return "", false
+	}
+	if wrapped, ok := err.(interface{ Unwrap() error }); ok {
+		return ResolveErrorCode(wrapped.Unwrap())
+	}
+	return "", false
+}
+
+// ValidErrorCodeFromError resolves err or returns ErrorServerInternal.
+func ValidErrorCodeFromError(err error) string {
+	if code, ok := ResolveErrorCode(err); ok {
+		return code
+	}
+	return ErrorServerInternal
+}
+
 // HasErrorCode reports whether s is exactly a registered AWS error code.
 // Use it to decide whether an error string may be surfaced to clients verbatim
 // (mapped to its real HTTP status) rather than wrapped/sanitized to 500.

spinifex/awserrors/awserrors_test.go

@@ -70,6 +70,47 @@ func TestValidErrorCode(t *testing.T) {
 	}
 }
 
+func TestResolveErrorCode(t *testing.T) {
+	known := errors.New(ErrorInsufficientAddressCapacity)
+	tests := []struct {
+		name   string
+		err    error
+		want   string
+		wantOK bool
+	}{
+		{name: "nil", err: nil},
+		{name: "exact code", err: known, want: ErrorInsufficientAddressCapacity, wantOK: true},
+		{name: "single wrapper", err: fmt.Errorf("allocate public address: %w", known), want: ErrorInsufficientAddressCapacity, wantOK: true},
+		{name: "nested wrappers", err: fmt.Errorf("launch on node: %w", fmt.Errorf("prepare instance: %w", known)), want: ErrorInsufficientAddressCapacity, wantOK: true},
+		{name: "joined errors", err: errors.Join(errors.New("opaque"), known), want: ErrorInsufficientAddressCapacity, wantOK: true},
+		{name: "first joined code wins", err: errors.Join(errors.New(ErrorAuthFailure), known), want: ErrorAuthFailure, wantOK: true},
+		{name: "unknown error", err: errors.New("opaque")},
+		{name: "code substring is not exact", err: errors.New("launch failed: " + ErrorInsufficientAddressCapacity)},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got, ok := ResolveErrorCode(tt.err)
+			if got != tt.want || ok != tt.wantOK {
+				t.Errorf("ResolveErrorCode() = (%q, %v), want (%q, %v)", got, ok, tt.want, tt.wantOK)
+			}
+		})
+	}
+}
+
+func TestValidErrorCodeFromError(t *testing.T) {
+	wrapped := fmt.Errorf("launch: %w", errors.New(ErrorInsufficientInstanceCapacity))
+	if got := ValidErrorCodeFromError(wrapped); got != ErrorInsufficientInstanceCapacity {
+		t.Errorf("ValidErrorCodeFromError() = %q, want %q", got, ErrorInsufficientInstanceCapacity)
+	}
+	if got := ValidErrorCodeFromError(errors.New("opaque")); got != ErrorServerInternal {
+		t.Errorf("ValidErrorCodeFromError() = %q, want %q", got, ErrorServerInternal)
+	}
+	if got := ValidErrorCodeFromError(nil); got != ErrorServerInternal {
+		t.Errorf("ValidErrorCodeFromError(nil) = %q, want %q", got, ErrorServerInternal)
+	}
+}
+
 func TestHasErrorCode(t *testing.T) {
 	tests := []struct {
 		name string

spinifex/config/config.go

@@ -84,7 +84,7 @@ type BootstrapConfig struct {
 	SubnetCidr string `mapstructure:"subnet_cidr"`
 }
 
-// Config holds all configuration for the application
+// Config holds all configuration for the application.
 type Config struct {
 	// Node config
 	Node string `json:"Node" mapstructure:"node"`
@@ -102,6 +102,7 @@ type Config struct {
 	Viperblock ViperblockConfig `json:"Viperblock" mapstructure:"viperblock"`
 	AWSGW      AWSGWConfig      `json:"AWSGW" mapstructure:"awsgw"`
 	VPCD       VPCDConfig       `json:"VPCD" mapstructure:"vpcd"`
+	Northstar  NorthstarConfig  `json:"Northstar" mapstructure:"northstar"`
 
 	BaseDir string `json:"BaseDir" mapstructure:"base_dir"`
 	WalDir  string `json:"WalDir" mapstructure:"wal_dir"`
@@ -123,6 +124,13 @@ type ViperblockConfig struct {
 	// EncryptionKeyFile is the path to the shared 32-byte AES-256 master key for viperblock at-rest encryption.
 	// Empty means cleartext. When set, all VB instances must load it via masterkey.LoadShared.
 	EncryptionKeyFile string `json:"EncryptionKeyFile" mapstructure:"encryption_key_file"`
+
+	// GCEnabled turns on viperblock chunk garbage collection (mark-and-sweep,
+	// snapshot-ancestry gated) on every VB this node constructs: the nbdkit
+	// plugin backing a mounted volume, and the short-lived VBs used by the
+	// volume service. Default false when nil so existing deployments keep
+	// today's behavior until explicitly opted in.
+	GCEnabled *bool `json:"GCEnabled" mapstructure:"gc_enabled"`
 }
 
 // VPCDConfig holds the VPC daemon (vpcd) configuration.
@@ -133,6 +141,17 @@ type VPCDConfig struct {
 	BridgeMode        string `json:"BridgeMode" mapstructure:"bridge_mode"`               // "direct" or "veth" (auto-detected if empty)
 }
 
+// NorthstarConfig holds the per-node northstar DNS service configuration.
+type NorthstarConfig struct {
+	// ConfigPath is the path to northstar.toml written by `spx admin init`.
+	ConfigPath string `json:"ConfigPath" mapstructure:"config_path"`
+	// DefaultDomain and InternalDomain mirror the northstar zone domains as
+	// non-secret values so producers (daemon, vpcd) can resolve DNS names
+	// without reading the credential-bearing northstar.toml.
+	DefaultDomain  string `json:"DefaultDomain" mapstructure:"default_domain"`
+	InternalDomain string `json:"InternalDomain" mapstructure:"internal_domain"`
+}
+
 // ParseEndpoints splits a comma-separated OVSDB endpoint list (NB/SB RAFT
 // cluster) into individual endpoints, trimming whitespace and dropping empties.
 // A single endpoint yields a one-element slice; empty input yields nil. Both the
@@ -174,7 +193,7 @@ type GPUModelOverride struct {
 	MIGProfile string `json:"MIGProfile" mapstructure:"mig_profile"`
 }
 
-// DaemonConfig holds the daemon configuration
+// DaemonConfig holds the daemon configuration.
 type DaemonConfig struct {
 	Host              string             `json:"Host" mapstructure:"host"`
 	TLSKey            string             `json:"TLSKey" mapstructure:"tlskey"`
@@ -187,7 +206,7 @@ type DaemonConfig struct {
 	MIGProfile string `json:"MIGProfile" mapstructure:"mig_profile"`
 }
 
-// NATSConfig holds the NATS configuration
+// NATSConfig holds the NATS configuration.
 type NATSConfig struct {
 	Host   string  `json:"Host" mapstructure:"host"`
 	CACert string  `json:"CACert" mapstructure:"cacert"`
@@ -195,12 +214,12 @@ type NATSConfig struct {
 	Sub    NATSSub `json:"Sub" mapstructure:"sub"`
 }
 
-// NATSACL holds the NATS ACL configuration
+// NATSACL holds the NATS ACL configuration.
 type NATSACL struct {
 	Token string `json:"Token" mapstructure:"token"`
 }
 
-// NATSSub holds the NATS subscription configuration
+// NATSSub holds the NATS subscription configuration.
 type NATSSub struct {
 	Subject string `json:"Subject" mapstructure:"subject"`
 }
@@ -242,7 +261,7 @@ func (c Config) GetServices() []string {
 	return c.Services
 }
 
-// LoadConfig loads the configuration from file and environment variables
+// LoadConfig loads the configuration from file and environment variables.
 func LoadConfig(configPath string) (*ClusterConfig, error) {
 	// Set environment variable prefix
 	viper.SetEnvPrefix("SPINIFEX")

spinifex/config/config_test.go

@@ -313,7 +313,7 @@ func TestNodeBaseDir_HappyPath(t *testing.T) {
 
 func TestNodeBaseDir_NilConfig(t *testing.T) {
 	var cc *ClusterConfig
-	assert.Equal(t, "", cc.NodeBaseDir())
+	assert.Empty(t, cc.NodeBaseDir())
 }
 
 func TestNodeBaseDir_EmptyNode(t *testing.T) {
@@ -323,7 +323,7 @@ func TestNodeBaseDir_EmptyNode(t *testing.T) {
 			"node1": {BaseDir: "/data/node1"},
 		},
 	}
-	assert.Equal(t, "", cc.NodeBaseDir())
+	assert.Empty(t, cc.NodeBaseDir())
 }
 
 func TestNodeBaseDir_NodeNotInMap(t *testing.T) {
@@ -333,7 +333,7 @@ func TestNodeBaseDir_NodeNotInMap(t *testing.T) {
 			"node1": {BaseDir: "/data/node1"},
 		},
 	}
-	assert.Equal(t, "", cc.NodeBaseDir())
+	assert.Empty(t, cc.NodeBaseDir())
 }
 
 func TestNodeBaseDir_EmptyBaseDir(t *testing.T) {
@@ -343,7 +343,7 @@ func TestNodeBaseDir_EmptyBaseDir(t *testing.T) {
 			"node1": {BaseDir: ""},
 		},
 	}
-	assert.Equal(t, "", cc.NodeBaseDir())
+	assert.Empty(t, cc.NodeBaseDir())
 }
 
 // Tests for NetworkConfig (external pools)
@@ -785,6 +785,50 @@ shardwal = true
 	assert.True(t, *n.Viperblock.ShardWAL)
 }
 
+func TestLoadConfig_ViperblockGCEnabled_DefaultNil(t *testing.T) {
+	resetViper(t)
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+
+	toml := `
+node = "n1"
+
+[nodes.n1]
+region = "us-east-1"
+`
+	require.NoError(t, os.WriteFile(path, []byte(toml), 0600))
+
+	cfg, err := LoadConfig(path)
+	require.NoError(t, err)
+
+	n := cfg.Nodes["n1"]
+	assert.Nil(t, n.Viperblock.GCEnabled, "GCEnabled should be nil when not configured (defaults to false in service)")
+}
+
+func TestLoadConfig_ViperblockGCEnabled_True(t *testing.T) {
+	resetViper(t)
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+
+	toml := `
+node = "n1"
+
+[nodes.n1]
+region = "us-east-1"
+
+[nodes.n1.viperblock]
+gc_enabled = true
+`
+	require.NoError(t, os.WriteFile(path, []byte(toml), 0600))
+
+	cfg, err := LoadConfig(path)
+	require.NoError(t, err)
+
+	n := cfg.Nodes["n1"]
+	require.NotNil(t, n.Viperblock.GCEnabled, "GCEnabled should be set when explicitly configured")
+	assert.True(t, *n.Viperblock.GCEnabled)
+}
+
 func TestParseEndpoints(t *testing.T) {
 	cases := []struct {
 		name string

spinifex/daemon/daemon.go

@@ -34,6 +34,7 @@ import (
 	"github.com/mulgadc/spinifex/spinifex/config"
 	"github.com/mulgadc/spinifex/spinifex/gpu"
 	handlers_acm "github.com/mulgadc/spinifex/spinifex/handlers/acm"
+	handlers_dns "github.com/mulgadc/spinifex/spinifex/handlers/dns"
 	handlers_ec2_account "github.com/mulgadc/spinifex/spinifex/handlers/ec2/account"
 	handlers_ec2_eigw "github.com/mulgadc/spinifex/spinifex/handlers/ec2/eigw"
 	handlers_ec2_eip "github.com/mulgadc/spinifex/spinifex/handlers/ec2/eip"
@@ -56,6 +57,7 @@ import (
 	handlers_elbv2 "github.com/mulgadc/spinifex/spinifex/handlers/elbv2"
 	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
 	"github.com/mulgadc/spinifex/spinifex/instancetypes"
+	"github.com/mulgadc/spinifex/spinifex/kvutil"
 	"github.com/mulgadc/spinifex/spinifex/network/external"
 	"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
 	"github.com/mulgadc/spinifex/spinifex/network/host"
@@ -65,6 +67,7 @@ import (
 	"github.com/mulgadc/spinifex/spinifex/utils"
 	"github.com/mulgadc/spinifex/spinifex/vm"
 	"github.com/nats-io/nats.go"
+	"github.com/nats-io/nats.go/jetstream"
 )
 
 // ResourceManager handles the allocation and tracking of system resources.
@@ -116,7 +119,7 @@ type ResourceManager struct {
 // handler's GatePublisher hook — the two services are wired together below.
 var _ handlers_ec2_igw.GatePublisher = (*handlers_ec2_routetable.RouteTableServiceImpl)(nil)
 
-// Daemon represents the main daemon service
+// Daemon represents the main daemon service.
 type Daemon struct {
 	node                  string
 	clusterConfig         *config.ClusterConfig
@@ -124,6 +127,10 @@ type Daemon struct {
 	natsConn              *nats.Conn
 	resourceMgr           *ResourceManager
 	instanceService       *handlers_ec2_instance.InstanceServiceImpl
+	dnsWriter             *handlers_dns.Writer
+	dnsReconciler         *handlers_dns.Reconciler
+	dnsBaseDomain         string
+	dnsInternalDomain     string
 	keyService            *handlers_ec2_key.KeyServiceImpl
 	imageService          *handlers_ec2_image.ImageServiceImpl
 	volumeService         *handlers_ec2_volume.VolumeServiceImpl
@@ -343,7 +350,7 @@ func (d *Daemon) onNATSReconnect(_ *nats.Conn) {
 // execCommand wraps exec.Command so tests can substitute a fake implementation.
 var execCommand = exec.Command
 
-// getSystemMemory returns the total system memory in GB
+// getSystemMemory returns the total system memory in GB.
 func getSystemMemory() (float64, error) {
 	switch runtime.GOOS {
 	case "darwin":
@@ -523,7 +530,7 @@ func (rm *ResourceManager) instanceMemChargeMiB(it *ec2.InstanceTypeInfo) int64
 		nbdkitChargeMiB(defaultMainVolumes, defaultAuxVolumes, rm.nbdkitMainMiB, rm.nbdkitAuxMiB)
 }
 
-// GetInstanceTypeInfos returns all instance types as ec2.InstanceTypeInfo for AWS API compatibility
+// GetInstanceTypeInfos returns all instance types as ec2.InstanceTypeInfo for AWS API compatibility.
 func (rm *ResourceManager) GetInstanceTypeInfos() []*ec2.InstanceTypeInfo {
 	rm.mu.RLock()
 	defer rm.mu.RUnlock()
@@ -666,12 +673,12 @@ func (rm *ResourceManager) GetResourceStats() (totalVCPU int, totalMemGB float64
 	return totalVCPU, totalMemGB, reservedVCPU, reservedMemGB, allocVCPU, allocMemGB, caps
 }
 
-// SetConfigPath sets the configuration file path for cluster management
+// SetConfigPath sets the configuration file path for cluster management.
 func (d *Daemon) SetConfigPath(path string) {
 	d.configPath = path
 }
 
-// NewDaemon creates a new daemon instance
+// NewDaemon creates a new daemon instance.
 func NewDaemon(cfg *config.ClusterConfig) (*Daemon, error) {
 	ctx, cancel := context.WithCancel(context.Background())
 
@@ -765,123 +772,125 @@ func (d *Daemon) subscribeAll() error {
 	subs := []natsSub{
 		// ec2.RunInstances is handled by dynamic per-instance-type subscriptions
 		// managed by ResourceManager.initSubscriptions()
-		{"ec2.CreateKeyPair", d.handleEC2CreateKeyPair, "spinifex-workers"},
-		{"ec2.DeleteKeyPair", d.handleEC2DeleteKeyPair, "spinifex-workers"},
-		{"ec2.DescribeKeyPairs", d.handleEC2DescribeKeyPairs, "spinifex-workers"},
-		{"ec2.ImportKeyPair", d.handleEC2ImportKeyPair, "spinifex-workers"},
+		{"ec2.CreateKeyPair", handleNATSRequest(d.keyService.CreateKeyPair), "spinifex-workers"},
+		{"ec2.DeleteKeyPair", handleNATSRequest(d.keyService.DeleteKeyPair), "spinifex-workers"},
+		{"ec2.DescribeKeyPairs", handleNATSRequest(d.keyService.DescribeKeyPairs), "spinifex-workers"},
+		{"ec2.ImportKeyPair", handleNATSRequest(d.keyService.ImportKeyPair), "spinifex-workers"},
 		{"imds.ec2.get_public_key", d.handleIMDSGetPublicKey, "spinifex-workers"},
-		{"ec2.DescribeImages", d.handleEC2DescribeImages, "spinifex-workers"},
+		{"ec2.DescribeImages", handleNATSRequest(d.imageService.DescribeImages), "spinifex-workers"},
 		{"ec2.CreateImage", d.handleEC2CreateImage, ""},
-		{"ec2.DeregisterImage", d.handleEC2DeregisterImage, "spinifex-workers"},
-		{"ec2.RegisterImage", d.handleEC2RegisterImage, "spinifex-workers"},
-		{"ec2.CopyImage", d.handleEC2CopyImage, "spinifex-workers"},
-		{"ec2.DescribeImageAttribute", d.handleEC2DescribeImageAttribute, "spinifex-workers"},
-		{"ec2.ModifyImageAttribute", d.handleEC2ModifyImageAttribute, "spinifex-workers"},
-		{"ec2.ResetImageAttribute", d.handleEC2ResetImageAttribute, "spinifex-workers"},
-		{"ec2.CreateVolume", d.handleEC2CreateVolume, "spinifex-workers"},
-		{"ec2.DescribeVolumes", d.handleEC2DescribeVolumes, "spinifex-workers"},
+		{"ec2.DeregisterImage", handleNATSRequest(d.imageService.DeregisterImage), "spinifex-workers"},
+		{"ec2.RegisterImage", handleNATSRequest(d.imageService.RegisterImage), "spinifex-workers"},
+		{"ec2.CopyImage", handleNATSRequest(d.imageService.CopyImage), "spinifex-workers"},
+		{"ec2.DescribeImageAttribute", handleNATSRequest(d.imageService.DescribeImageAttribute), "spinifex-workers"},
+		{"ec2.ModifyImageAttribute", handleNATSRequest(d.imageService.ModifyImageAttribute), "spinifex-workers"},
+		{"ec2.ResetImageAttribute", handleNATSRequest(d.imageService.ResetImageAttribute), "spinifex-workers"},
+		{"ec2.CreateVolume", handleNATSRequest(d.volumeService.CreateVolume), "spinifex-workers"},
+		{"ec2.DescribeVolumes", handleNATSRequest(d.volumeService.DescribeVolumes), "spinifex-workers"},
 		{"ec2.ModifyVolume", d.handleEC2ModifyVolume, "spinifex-workers"},
-		{"ec2.DeleteVolume", d.handleEC2DeleteVolume, "spinifex-workers"},
-		{"ec2.DescribeVolumeStatus", d.handleEC2DescribeVolumeStatus, "spinifex-workers"},
-		{"ec2.DescribeVolumesModifications", d.handleEC2DescribeVolumesModifications, "spinifex-workers"},
-		{"ec2.CreateSnapshot", d.handleEC2CreateSnapshot, "spinifex-workers"},
-		{"ec2.DescribeSnapshots", d.handleEC2DescribeSnapshots, "spinifex-workers"},
-		{"ec2.DeleteSnapshot", d.handleEC2DeleteSnapshot, "spinifex-workers"},
-		{"ec2.CopySnapshot", d.handleEC2CopySnapshot, "spinifex-workers"},
-		{"ec2.CreateTags", d.handleEC2CreateTags, "spinifex-workers"},
-		{"ec2.DeleteTags", d.handleEC2DeleteTags, "spinifex-workers"},
-		{"ec2.DescribeTags", d.handleEC2DescribeTags, "spinifex-workers"},
-		{"ec2.CreateEgressOnlyInternetGateway", d.handleEC2CreateEgressOnlyInternetGateway, "spinifex-workers"},
-		{"ec2.DeleteEgressOnlyInternetGateway", d.handleEC2DeleteEgressOnlyInternetGateway, "spinifex-workers"},
-		{"ec2.DescribeEgressOnlyInternetGateways", d.handleEC2DescribeEgressOnlyInternetGateways, "spinifex-workers"},
-		{"ec2.CreateInternetGateway", d.handleEC2CreateInternetGateway, "spinifex-workers"},
-		{"ec2.DeleteInternetGateway", d.handleEC2DeleteInternetGateway, "spinifex-workers"},
-		{"ec2.DescribeInternetGateways", d.handleEC2DescribeInternetGateways, "spinifex-workers"},
-		{"ec2.AttachInternetGateway", d.handleEC2AttachInternetGateway, "spinifex-workers"},
-		{"ec2.DetachInternetGateway", d.handleEC2DetachInternetGateway, "spinifex-workers"},
-		{"ec2.CreatePlacementGroup", d.handleEC2CreatePlacementGroup, "spinifex-workers"},
-		{"ec2.DeletePlacementGroup", d.handleEC2DeletePlacementGroup, "spinifex-workers"},
-		{"ec2.DescribePlacementGroups", d.handleEC2DescribePlacementGroups, "spinifex-workers"},
-		{"ec2.ReserveSpreadNodes", d.handleEC2ReserveSpreadNodes, "spinifex-workers"},
-		{"ec2.FinalizeSpreadInstances", d.handleEC2FinalizeSpreadInstances, "spinifex-workers"},
-		{"ec2.ReleaseSpreadNodes", d.handleEC2ReleaseSpreadNodes, "spinifex-workers"},
-		{"ec2.RemoveInstanceFromPlacementGroup", d.handleEC2RemoveInstanceFromPlacementGroup, "spinifex-workers"},
-		{"ec2.ReserveClusterNode", d.handleEC2ReserveClusterNode, "spinifex-workers"},
-		{"ec2.FinalizeClusterInstances", d.handleEC2FinalizeClusterInstances, "spinifex-workers"},
-		{"ec2.CreateLaunchTemplate", d.handleEC2CreateLaunchTemplate, "spinifex-workers"},
-		{"ec2.CreateLaunchTemplateVersion", d.handleEC2CreateLaunchTemplateVersion, "spinifex-workers"},
-		{"ec2.DeleteLaunchTemplate", d.handleEC2DeleteLaunchTemplate, "spinifex-workers"},
-		{"ec2.DeleteLaunchTemplateVersions", d.handleEC2DeleteLaunchTemplateVersions, "spinifex-workers"},
-		{"ec2.ModifyLaunchTemplate", d.handleEC2ModifyLaunchTemplate, "spinifex-workers"},
-		{"ec2.DescribeLaunchTemplates", d.handleEC2DescribeLaunchTemplates, "spinifex-workers"},
-		{"ec2.DescribeLaunchTemplateVersions", d.handleEC2DescribeLaunchTemplateVersions, "spinifex-workers"},
-		{"ec2.PutSpotInstanceRequests", d.handleEC2PutSpotInstanceRequests, "spinifex-workers"},
-		{"ec2.DescribeSpotInstanceRequests", d.handleEC2DescribeSpotInstanceRequests, "spinifex-workers"},
-		{"ec2.CancelSpotInstanceRequests", d.handleEC2CancelSpotInstanceRequests, "spinifex-workers"},
+		{"ec2.DeleteVolume", handleNATSRequest(d.volumeService.DeleteVolume), "spinifex-workers"},
+		{"ec2.DescribeVolumeStatus", handleNATSRequest(d.volumeService.DescribeVolumeStatus), "spinifex-workers"},
+		{"ec2.DescribeVolumesModifications", handleNATSRequest(d.volumeService.DescribeVolumesModifications), "spinifex-workers"},
+		{"ec2.CreateSnapshot", handleNATSRequest(d.snapshotService.CreateSnapshot), "spinifex-workers"},
+		{"ec2.DescribeSnapshots", handleNATSRequest(d.snapshotService.DescribeSnapshots), "spinifex-workers"},
+		{"ec2.DeleteSnapshot", handleNATSRequest(d.snapshotService.DeleteSnapshot), "spinifex-workers"},
+		{"ec2.CopySnapshot", handleNATSRequest(d.snapshotService.CopySnapshot), "spinifex-workers"},
+		{"ec2.CreateTags", handleNATSRequest(d.createTags), "spinifex-workers"},
+		{"ec2.DeleteTags", handleNATSRequest(d.deleteTags), "spinifex-workers"},
+		{"ec2.DescribeTags", handleNATSRequest(d.tagsService.DescribeTags), "spinifex-workers"},
+		{"ec2.CreateEgressOnlyInternetGateway", handleNATSRequest(d.eigwService.CreateEgressOnlyInternetGateway), "spinifex-workers"},
+		{"ec2.DeleteEgressOnlyInternetGateway", handleNATSRequest(d.eigwService.DeleteEgressOnlyInternetGateway), "spinifex-workers"},
+		{"ec2.DescribeEgressOnlyInternetGateways", handleNATSRequest(d.eigwService.DescribeEgressOnlyInternetGateways), "spinifex-workers"},
+		{"ec2.CreateInternetGateway", handleNATSRequest(d.igwService.CreateInternetGateway), "spinifex-workers"},
+		{"ec2.DeleteInternetGateway", handleNATSRequest(d.igwService.DeleteInternetGateway), "spinifex-workers"},
+		{"ec2.DescribeInternetGateways", handleNATSRequest(d.igwService.DescribeInternetGateways), "spinifex-workers"},
+		{"ec2.AttachInternetGateway", handleNATSRequest(d.igwService.AttachInternetGateway), "spinifex-workers"},
+		{"ec2.DetachInternetGateway", handleNATSRequest(d.igwService.DetachInternetGateway), "spinifex-workers"},
+		{"ec2.CreatePlacementGroup", handleNATSRequest(d.placementGroupService.CreatePlacementGroup), "spinifex-workers"},
+		{"ec2.DeletePlacementGroup", handleNATSRequest(d.placementGroupService.DeletePlacementGroup), "spinifex-workers"},
+		{"ec2.DescribePlacementGroups", handleNATSRequest(d.placementGroupService.DescribePlacementGroups), "spinifex-workers"},
+		{"ec2.ReserveSpreadNodes", handleNATSRequest(d.placementGroupService.ReserveSpreadNodes), "spinifex-workers"},
+		{"ec2.FinalizeSpreadInstances", handleNATSRequest(d.placementGroupService.FinalizeSpreadInstances), "spinifex-workers"},
+		{"ec2.ReleaseSpreadNodes", handleNATSRequest(d.placementGroupService.ReleaseSpreadNodes), "spinifex-workers"},
+		{"ec2.RemoveInstanceFromPlacementGroup", handleNATSRequest(d.placementGroupService.RemoveInstance), "spinifex-workers"},
+		{"ec2.ReserveClusterNode", handleNATSRequest(d.placementGroupService.ReserveClusterNode), "spinifex-workers"},
+		{"ec2.FinalizeClusterInstances", handleNATSRequest(d.placementGroupService.FinalizeClusterInstances), "spinifex-workers"},
+		{"ec2.CreateLaunchTemplate", handleNATSRequest(d.launchTemplateService.CreateLaunchTemplate), "spinifex-workers"},
+		{"ec2.CreateLaunchTemplateVersion", handleNATSRequest(d.launchTemplateService.CreateLaunchTemplateVersion), "spinifex-workers"},
+		{"ec2.DeleteLaunchTemplate", handleNATSRequest(d.launchTemplateService.DeleteLaunchTemplate), "spinifex-workers"},
+		{"ec2.DeleteLaunchTemplateVersions", handleNATSRequest(d.launchTemplateService.DeleteLaunchTemplateVersions), "spinifex-workers"},
+		{"ec2.ModifyLaunchTemplate", handleNATSRequest(d.launchTemplateService.ModifyLaunchTemplate), "spinifex-workers"},
+		{"ec2.DescribeLaunchTemplates", handleNATSRequest(d.launchTemplateService.DescribeLaunchTemplates), "spinifex-workers"},
+		{"ec2.DescribeLaunchTemplateVersions", handleNATSRequest(d.launchTemplateService.DescribeLaunchTemplateVersions), "spinifex-workers"},
+		{"ec2.PutSpotInstanceRequests", handleNATSRequest(d.spotInstanceService.PutSpotInstanceRequests), "spinifex-workers"},
+		{"ec2.DescribeSpotInstanceRequests", handleNATSRequest(d.spotInstanceService.DescribeSpotInstanceRequests), "spinifex-workers"},
+		{"ec2.CancelSpotInstanceRequests", handleNATSRequest(d.spotInstanceService.CancelSpotInstanceRequests), "spinifex-workers"},
 		// Capacity reservations: Create is node-targeted (gateway pins one node);
 		// Describe fans out and Cancel broadcasts, so both use plain Subscribe.
 		{fmt.Sprintf("ec2.CreateCapacityReservation.%s", d.node), d.handleEC2CreateCapacityReservation, ""},
 		{"ec2.DescribeCapacityReservations", d.handleEC2DescribeCapacityReservations, ""},
 		{"ec2.CancelCapacityReservation", d.handleEC2CancelCapacityReservation, ""},
-		{"ec2.CreateNatGateway", d.handleEC2CreateNatGateway, "spinifex-workers"},
-		{"ec2.DeleteNatGateway", d.handleEC2DeleteNatGateway, "spinifex-workers"},
-		{"ec2.DescribeNatGateways", d.handleEC2DescribeNatGateways, "spinifex-workers"},
-		{"ec2.CreateRouteTable", d.handleEC2CreateRouteTable, "spinifex-workers"},
-		{"ec2.DeleteRouteTable", d.handleEC2DeleteRouteTable, "spinifex-workers"},
-		{"ec2.DescribeRouteTables", d.handleEC2DescribeRouteTables, "spinifex-workers"},
-		{"ec2.CreateRoute", d.handleEC2CreateRoute, "spinifex-workers"},
-		{"ec2.DeleteRoute", d.handleEC2DeleteRoute, "spinifex-workers"},
-		{"ec2.ReplaceRoute", d.handleEC2ReplaceRoute, "spinifex-workers"},
-		{"ec2.AssociateRouteTable", d.handleEC2AssociateRouteTable, "spinifex-workers"},
-		{"ec2.DisassociateRouteTable", d.handleEC2DisassociateRouteTable, "spinifex-workers"},
-		{"ec2.ReplaceRouteTableAssociation", d.handleEC2ReplaceRouteTableAssociation, "spinifex-workers"},
-		{"ec2.CreateVpc", d.handleEC2CreateVpc, "spinifex-workers"},
-		{"ec2.DeleteVpc", d.handleEC2DeleteVpc, "spinifex-workers"},
-		{"ec2.DescribeVpcs", d.handleEC2DescribeVpcs, "spinifex-workers"},
-		{"ec2.CreateSubnet", d.handleEC2CreateSubnet, "spinifex-workers"},
-		{"ec2.DeleteSubnet", d.handleEC2DeleteSubnet, "spinifex-workers"},
-		{"ec2.DescribeSubnets", d.handleEC2DescribeSubnets, "spinifex-workers"},
-		{"ec2.ModifySubnetAttribute", d.handleEC2ModifySubnetAttribute, "spinifex-workers"},
-		{"ec2.ModifyVpcAttribute", d.handleEC2ModifyVpcAttribute, "spinifex-workers"},
-		{"ec2.DescribeVpcAttribute", d.handleEC2DescribeVpcAttribute, "spinifex-workers"},
-		{"ec2.CreateNetworkInterface", d.handleEC2CreateNetworkInterface, "spinifex-workers"},
-		{"ec2.DeleteNetworkInterface", d.handleEC2DeleteNetworkInterface, "spinifex-workers"},
-		{"ec2.DescribeNetworkInterfaces", d.handleEC2DescribeNetworkInterfaces, "spinifex-workers"},
-		{"ec2.ModifyNetworkInterfaceAttribute", d.handleEC2ModifyNetworkInterfaceAttribute, "spinifex-workers"},
-		{"ec2.CreateSecurityGroup", d.handleEC2CreateSecurityGroup, "spinifex-workers"},
-		{"ec2.DeleteSecurityGroup", d.handleEC2DeleteSecurityGroup, "spinifex-workers"},
-		{"ec2.DescribeSecurityGroups", d.handleEC2DescribeSecurityGroups, "spinifex-workers"},
-		{"ec2.DescribeSecurityGroupRules", d.handleEC2DescribeSecurityGroupRules, "spinifex-workers"},
-		{"ec2.AuthorizeSecurityGroupIngress", d.handleEC2AuthorizeSecurityGroupIngress, "spinifex-workers"},
-		{"ec2.AuthorizeSecurityGroupEgress", d.handleEC2AuthorizeSecurityGroupEgress, "spinifex-workers"},
-		{"ec2.RevokeSecurityGroupIngress", d.handleEC2RevokeSecurityGroupIngress, "spinifex-workers"},
-		{"ec2.RevokeSecurityGroupEgress", d.handleEC2RevokeSecurityGroupEgress, "spinifex-workers"},
-		{"ec2.ModifyInstanceAttribute", d.handleEC2ModifyInstanceAttribute, "spinifex-workers"},
-		{"ec2.ModifyInstanceMetadataOptions", d.handleEC2ModifyInstanceMetadataOptions, "spinifex-workers"},
+		{"ec2.CreateNatGateway", handleNATSRequest(d.natGatewayService.CreateNatGateway), "spinifex-workers"},
+		{"ec2.DeleteNatGateway", handleNATSRequest(d.natGatewayService.DeleteNatGateway), "spinifex-workers"},
+		{"ec2.DescribeNatGateways", handleNATSRequest(d.natGatewayService.DescribeNatGateways), "spinifex-workers"},
+		{"ec2.CreateRouteTable", handleNATSRequest(d.routeTableService.CreateRouteTable), "spinifex-workers"},
+		{"ec2.DeleteRouteTable", handleNATSRequest(d.routeTableService.DeleteRouteTable), "spinifex-workers"},
+		{"ec2.DescribeRouteTables", handleNATSRequest(d.routeTableService.DescribeRouteTables), "spinifex-workers"},
+		{"ec2.CreateRoute", handleNATSRequest(d.routeTableService.CreateRoute), "spinifex-workers"},
+		{"ec2.DeleteRoute", handleNATSRequest(d.routeTableService.DeleteRoute), "spinifex-workers"},
+		{"ec2.ReplaceRoute", handleNATSRequest(d.routeTableService.ReplaceRoute), "spinifex-workers"},
+		{"ec2.AssociateRouteTable", handleNATSRequest(d.routeTableService.AssociateRouteTable), "spinifex-workers"},
+		{"ec2.DisassociateRouteTable", handleNATSRequest(d.routeTableService.DisassociateRouteTable), "spinifex-workers"},
+		{"ec2.ReplaceRouteTableAssociation", handleNATSRequest(d.routeTableService.ReplaceRouteTableAssociation), "spinifex-workers"},
+		{"ec2.CreateVpc", handleNATSRequest(d.vpcService.CreateVpc), "spinifex-workers"},
+		{"ec2.DeleteVpc", handleNATSRequest(d.vpcService.DeleteVpc), "spinifex-workers"},
+		{"ec2.DescribeVpcs", handleNATSRequest(d.vpcService.DescribeVpcs), "spinifex-workers"},
+		{"ec2.CreateSubnet", handleNATSRequest(d.vpcService.CreateSubnet), "spinifex-workers"},
+		{"ec2.DeleteSubnet", handleNATSRequest(d.vpcService.DeleteSubnet), "spinifex-workers"},
+		{"ec2.DescribeSubnets", handleNATSRequest(d.vpcService.DescribeSubnets), "spinifex-workers"},
+		{"ec2.ModifySubnetAttribute", handleNATSRequest(d.vpcService.ModifySubnetAttribute), "spinifex-workers"},
+		{"ec2.ModifyVpcAttribute", handleNATSRequest(d.vpcService.ModifyVpcAttribute), "spinifex-workers"},
+		{"ec2.DescribeVpcAttribute", handleNATSRequest(d.vpcService.DescribeVpcAttribute), "spinifex-workers"},
+		{"ec2.CreateNetworkInterface", handleNATSRequest(d.vpcService.CreateNetworkInterface), "spinifex-workers"},
+		{"ec2.DeleteNetworkInterface", handleNATSRequest(d.vpcService.DeleteNetworkInterface), "spinifex-workers"},
+		{"ec2.DescribeNetworkInterfaces", handleNATSRequest(d.vpcService.DescribeNetworkInterfaces), "spinifex-workers"},
+		{"ec2.ModifyNetworkInterfaceAttribute", handleNATSRequest(d.vpcService.ModifyNetworkInterfaceAttribute), "spinifex-workers"},
+		{"ec2.CreateSecurityGroup", handleNATSRequest(d.vpcService.CreateSecurityGroup), "spinifex-workers"},
+		{"ec2.DeleteSecurityGroup", handleNATSRequest(d.vpcService.DeleteSecurityGroup), "spinifex-workers"},
+		{"ec2.DescribeSecurityGroups", handleNATSRequest(d.vpcService.DescribeSecurityGroups), "spinifex-workers"},
+		{"ec2.DescribeSecurityGroupRules", handleNATSRequest(d.vpcService.DescribeSecurityGroupRules), "spinifex-workers"},
+		{"ec2.AuthorizeSecurityGroupIngress", handleNATSRequest(d.vpcService.AuthorizeSecurityGroupIngress), "spinifex-workers"},
+		{"ec2.AuthorizeSecurityGroupEgress", handleNATSRequest(d.vpcService.AuthorizeSecurityGroupEgress), "spinifex-workers"},
+		{"ec2.RevokeSecurityGroupIngress", handleNATSRequest(d.vpcService.RevokeSecurityGroupIngress), "spinifex-workers"},
+		{"ec2.RevokeSecurityGroupEgress", handleNATSRequest(d.vpcService.RevokeSecurityGroupEgress), "spinifex-workers"},
+		{"ec2.ModifyInstanceAttribute", handleNATSRequest(d.instanceService.ModifyInstanceAttribute), "spinifex-workers"},
+		{"ec2.ModifyInstanceMetadataOptions", handleNATSRequest(d.instanceService.ModifyInstanceMetadataOptions), "spinifex-workers"},
 		{"ec2.start", d.handleEC2StartStoppedInstance, "spinifex-workers"},
-		{fmt.Sprintf("ec2.start.%s", d.node), d.handleEC2StartStoppedInstanceDirect, ""},
-		{"ec2.terminate", d.handleEC2TerminateStoppedInstance, "spinifex-workers"},
-		{"ec2.DescribeStoppedInstances", d.handleEC2DescribeStoppedInstances, "spinifex-workers"},
-		{"ec2.DescribeTerminatedInstances", d.handleEC2DescribeTerminatedInstances, "spinifex-workers"},
+		// ec2.start.{node} is the node-targeted variant: it always starts locally
+		// and never re-forwards, so it goes straight to the service (no routing loop).
+		{fmt.Sprintf("ec2.start.%s", d.node), handleNATSRequest(d.instanceService.StartStoppedInstance), ""},
+		{"ec2.terminate", handleNATSRequest(d.instanceService.TerminateStoppedInstance), "spinifex-workers"},
+		{"ec2.DescribeStoppedInstances", handleNATSRequest(d.instanceService.DescribeStoppedInstances), "spinifex-workers"},
+		{"ec2.DescribeTerminatedInstances", handleNATSRequest(d.instanceService.DescribeTerminatedInstances), "spinifex-workers"},
 		// these fan out to all nodes and gateway aggregates the results. The
 		// handler only sees per-daemon local state (vmMgr/stoppedStore), so
 		// any queue-grouped routing produces 1/N false NotFound responses.
-		{"ec2.DescribeInstances", d.handleEC2DescribeInstances, ""},
-		{"ec2.DescribeInstanceStatus", d.handleEC2DescribeInstanceStatus, ""},
-		{"ec2.DescribeInstanceTypes", d.handleEC2DescribeInstanceTypes, ""},
-		{"ec2.DescribeInstanceAttribute", d.handleEC2DescribeInstanceAttribute, ""},
+		{"ec2.DescribeInstances", handleNATSRequest(d.instanceService.DescribeInstances), ""},
+		{"ec2.DescribeInstanceStatus", handleNATSRequest(d.instanceService.DescribeInstanceStatus), ""},
+		{"ec2.DescribeInstanceTypes", handleNATSRequest(d.instanceService.DescribeInstanceTypes), ""},
+		{"ec2.DescribeInstanceAttribute", handleNATSRequest(d.instanceService.DescribeInstanceAttribute), ""},
 		// IAM instance profile associations: Disassociate/Replace mutate the
 		// owning daemon's vm.VM (non-owners NoOp with Found=false); Describe
 		// returns per-daemon matches that the gateway concatenates.
-		{"ec2.IamProfileAssociation.disassociate", d.handleIamProfileDisassociate, ""},
-		{"ec2.IamProfileAssociation.replace", d.handleIamProfileReplace, ""},
-		{"ec2.IamProfileAssociation.describe", d.handleIamProfileDescribe, ""},
-		{"ec2.EnableEbsEncryptionByDefault", d.handleEC2EnableEbsEncryptionByDefault, "spinifex-workers"},
-		{"ec2.DisableEbsEncryptionByDefault", d.handleEC2DisableEbsEncryptionByDefault, "spinifex-workers"},
-		{"ec2.GetEbsEncryptionByDefault", d.handleEC2GetEbsEncryptionByDefault, "spinifex-workers"},
-		{"ec2.GetSerialConsoleAccessStatus", d.handleEC2GetSerialConsoleAccessStatus, "spinifex-workers"},
-		{"ec2.EnableSerialConsoleAccess", d.handleEC2EnableSerialConsoleAccess, "spinifex-workers"},
-		{"ec2.DisableSerialConsoleAccess", d.handleEC2DisableSerialConsoleAccess, "spinifex-workers"},
+		{"ec2.IamProfileAssociation.disassociate", handleNATSRequest(d.instanceService.DisassociateIamProfileAssociation), ""},
+		{"ec2.IamProfileAssociation.replace", handleNATSRequest(d.instanceService.ReplaceIamProfileAssociation), ""},
+		{"ec2.IamProfileAssociation.describe", handleNATSRequest(d.instanceService.DescribeIamProfileAssociations), ""},
+		{"ec2.EnableEbsEncryptionByDefault", handleNATSRequest(d.accountService.EnableEbsEncryptionByDefault), "spinifex-workers"},
+		{"ec2.DisableEbsEncryptionByDefault", handleNATSRequest(d.accountService.DisableEbsEncryptionByDefault), "spinifex-workers"},
+		{"ec2.GetEbsEncryptionByDefault", handleNATSRequest(d.accountService.GetEbsEncryptionByDefault), "spinifex-workers"},
+		{"ec2.GetSerialConsoleAccessStatus", handleNATSRequest(d.accountService.GetSerialConsoleAccessStatus), "spinifex-workers"},
+		{"ec2.EnableSerialConsoleAccess", handleNATSRequest(d.accountService.EnableSerialConsoleAccess), "spinifex-workers"},
+		{"ec2.DisableSerialConsoleAccess", handleNATSRequest(d.accountService.DisableSerialConsoleAccess), "spinifex-workers"},
 		{fmt.Sprintf("spinifex.admin.%s.health", d.node), d.handleHealthCheck, ""},
 		{"spinifex.nodes.discover", d.handleNodeDiscover, ""},
 		{"spinifex.node.status", d.handleNodeStatus, ""},
@@ -902,41 +911,41 @@ func (d *Daemon) subscribeAll() error {
 	// Without a subscriber the gateway returns nats.ErrNoResponders → ServiceUnavailable.
 	if d.elbv2Service.GatewayURL != "" {
 		subs = append(subs,
-			natsSub{"elbv2.CreateLoadBalancer", d.handleELBv2CreateLoadBalancer, "spinifex-workers"},
-			natsSub{"elbv2.DeleteLoadBalancer", d.handleELBv2DeleteLoadBalancer, "spinifex-workers"},
-			natsSub{"elbv2.DescribeLoadBalancers", d.handleELBv2DescribeLoadBalancers, "spinifex-workers"},
-			natsSub{"elbv2.CreateTargetGroup", d.handleELBv2CreateTargetGroup, "spinifex-workers"},
-			natsSub{"elbv2.ModifyTargetGroup", d.handleELBv2ModifyTargetGroup, "spinifex-workers"},
-			natsSub{"elbv2.DeleteTargetGroup", d.handleELBv2DeleteTargetGroup, "spinifex-workers"},
-			natsSub{"elbv2.DescribeTargetGroups", d.handleELBv2DescribeTargetGroups, "spinifex-workers"},
-			natsSub{"elbv2.RegisterTargets", d.handleELBv2RegisterTargets, "spinifex-workers"},
-			natsSub{"elbv2.DeregisterTargets", d.handleELBv2DeregisterTargets, "spinifex-workers"},
-			natsSub{"elbv2.DescribeTargetHealth", d.handleELBv2DescribeTargetHealth, "spinifex-workers"},
-			natsSub{"elbv2.CreateListener", d.handleELBv2CreateListener, "spinifex-workers"},
-			natsSub{"elbv2.DeleteListener", d.handleELBv2DeleteListener, "spinifex-workers"},
-			natsSub{"elbv2.ModifyListener", d.handleELBv2ModifyListener, "spinifex-workers"},
-			natsSub{"elbv2.DescribeListeners", d.handleELBv2DescribeListeners, "spinifex-workers"},
-			natsSub{"elbv2.CreateRule", d.handleELBv2CreateRule, "spinifex-workers"},
-			natsSub{"elbv2.ModifyRule", d.handleELBv2ModifyRule, "spinifex-workers"},
-			natsSub{"elbv2.DeleteRule", d.handleELBv2DeleteRule, "spinifex-workers"},
-			natsSub{"elbv2.DescribeRules", d.handleELBv2DescribeRules, "spinifex-workers"},
-			natsSub{"elbv2.SetRulePriorities", d.handleELBv2SetRulePriorities, "spinifex-workers"},
-			natsSub{"elbv2.DescribeTags", d.handleELBv2DescribeTags, "spinifex-workers"},
-			natsSub{"elbv2.AddTags", d.handleELBv2AddTags, "spinifex-workers"},
-			natsSub{"elbv2.RemoveTags", d.handleELBv2RemoveTags, "spinifex-workers"},
-			natsSub{"elbv2.LBAgentHeartbeat", d.handleELBv2LBAgentHeartbeat, "spinifex-workers"},
-			natsSub{"elbv2.GetLBConfig", d.handleELBv2GetLBConfig, "spinifex-workers"},
-			natsSub{"elbv2.ModifyTargetGroupAttributes", d.handleELBv2ModifyTargetGroupAttributes, "spinifex-workers"},
-			natsSub{"elbv2.DescribeTargetGroupAttributes", d.handleELBv2DescribeTargetGroupAttributes, "spinifex-workers"},
-			natsSub{"elbv2.ModifyLoadBalancerAttributes", d.handleELBv2ModifyLoadBalancerAttributes, "spinifex-workers"},
-			natsSub{"elbv2.DescribeLoadBalancerAttributes", d.handleELBv2DescribeLoadBalancerAttributes, "spinifex-workers"},
-			natsSub{"elbv2.SetSecurityGroups", d.handleELBv2SetSecurityGroups, "spinifex-workers"},
-			natsSub{"elbv2.SetIpAddressType", d.handleELBv2SetIpAddressType, "spinifex-workers"},
-			natsSub{"elbv2.SetSubnets", d.handleELBv2SetSubnets, "spinifex-workers"},
-			natsSub{"elbv2.AddListenerCertificates", d.handleELBv2AddListenerCertificates, "spinifex-workers"},
-			natsSub{"elbv2.RemoveListenerCertificates", d.handleELBv2RemoveListenerCertificates, "spinifex-workers"},
-			natsSub{"elbv2.DescribeListenerCertificates", d.handleELBv2DescribeListenerCertificates, "spinifex-workers"},
-			natsSub{"elbv2.DescribeSSLPolicies", d.handleELBv2DescribeSSLPolicies, "spinifex-workers"},
+			natsSub{"elbv2.CreateLoadBalancer", handleNATSRequest(d.elbv2Service.CreateLoadBalancer), "spinifex-workers"},
+			natsSub{"elbv2.DeleteLoadBalancer", handleNATSRequest(d.elbv2Service.DeleteLoadBalancer), "spinifex-workers"},
+			natsSub{"elbv2.DescribeLoadBalancers", handleNATSRequest(d.elbv2Service.DescribeLoadBalancers), "spinifex-workers"},
+			natsSub{"elbv2.CreateTargetGroup", handleNATSRequest(d.elbv2Service.CreateTargetGroup), "spinifex-workers"},
+			natsSub{"elbv2.ModifyTargetGroup", handleNATSRequest(d.elbv2Service.ModifyTargetGroup), "spinifex-workers"},
+			natsSub{"elbv2.DeleteTargetGroup", handleNATSRequest(d.elbv2Service.DeleteTargetGroup), "spinifex-workers"},
+			natsSub{"elbv2.DescribeTargetGroups", handleNATSRequest(d.elbv2Service.DescribeTargetGroups), "spinifex-workers"},
+			natsSub{"elbv2.RegisterTargets", handleNATSRequest(d.elbv2Service.RegisterTargets), "spinifex-workers"},
+			natsSub{"elbv2.DeregisterTargets", handleNATSRequest(d.elbv2Service.DeregisterTargets), "spinifex-workers"},
+			natsSub{"elbv2.DescribeTargetHealth", handleNATSRequest(d.elbv2Service.DescribeTargetHealth), "spinifex-workers"},
+			natsSub{"elbv2.CreateListener", handleNATSRequest(d.elbv2Service.CreateListener), "spinifex-workers"},
+			natsSub{"elbv2.DeleteListener", handleNATSRequest(d.elbv2Service.DeleteListener), "spinifex-workers"},
+			natsSub{"elbv2.ModifyListener", handleNATSRequest(d.elbv2Service.ModifyListener), "spinifex-workers"},
+			natsSub{"elbv2.DescribeListeners", handleNATSRequest(d.elbv2Service.DescribeListeners), "spinifex-workers"},
+			natsSub{"elbv2.CreateRule", handleNATSRequest(d.elbv2Service.CreateRule), "spinifex-workers"},
+			natsSub{"elbv2.ModifyRule", handleNATSRequest(d.elbv2Service.ModifyRule), "spinifex-workers"},
+			natsSub{"elbv2.DeleteRule", handleNATSRequest(d.elbv2Service.DeleteRule), "spinifex-workers"},
+			natsSub{"elbv2.DescribeRules", handleNATSRequest(d.elbv2Service.DescribeRules), "spinifex-workers"},
+			natsSub{"elbv2.SetRulePriorities", handleNATSRequest(d.elbv2Service.SetRulePriorities), "spinifex-workers"},
+			natsSub{"elbv2.DescribeTags", handleNATSRequest(d.elbv2Service.DescribeTags), "spinifex-workers"},
+			natsSub{"elbv2.AddTags", handleNATSRequest(d.elbv2Service.AddTags), "spinifex-workers"},
+			natsSub{"elbv2.RemoveTags", handleNATSRequest(d.elbv2Service.RemoveTags), "spinifex-workers"},
+			natsSub{"elbv2.LBAgentHeartbeat", handleNATSRequest(d.elbv2Service.LBAgentHeartbeat), "spinifex-workers"},
+			natsSub{"elbv2.GetLBConfig", handleNATSRequest(d.elbv2Service.GetLBConfig), "spinifex-workers"},
+			natsSub{"elbv2.ModifyTargetGroupAttributes", handleNATSRequest(d.elbv2Service.ModifyTargetGroupAttributes), "spinifex-workers"},
+			natsSub{"elbv2.DescribeTargetGroupAttributes", handleNATSRequest(d.elbv2Service.DescribeTargetGroupAttributes), "spinifex-workers"},
+			natsSub{"elbv2.ModifyLoadBalancerAttributes", handleNATSRequest(d.elbv2Service.ModifyLoadBalancerAttributes), "spinifex-workers"},
+			natsSub{"elbv2.DescribeLoadBalancerAttributes", handleNATSRequest(d.elbv2Service.DescribeLoadBalancerAttributes), "spinifex-workers"},
+			natsSub{"elbv2.SetSecurityGroups", handleNATSRequest(d.elbv2Service.SetSecurityGroups), "spinifex-workers"},
+			natsSub{"elbv2.SetIpAddressType", handleNATSRequest(d.elbv2Service.SetIpAddressType), "spinifex-workers"},
+			natsSub{"elbv2.SetSubnets", handleNATSRequest(d.elbv2Service.SetSubnets), "spinifex-workers"},
+			natsSub{"elbv2.AddListenerCertificates", handleNATSRequest(d.elbv2Service.AddListenerCertificates), "spinifex-workers"},
+			natsSub{"elbv2.RemoveListenerCertificates", handleNATSRequest(d.elbv2Service.RemoveListenerCertificates), "spinifex-workers"},
+			natsSub{"elbv2.DescribeListenerCertificates", handleNATSRequest(d.elbv2Service.DescribeListenerCertificates), "spinifex-workers"},
+			natsSub{"elbv2.DescribeSSLPolicies", handleNATSRequest(d.elbv2Service.DescribeSSLPolicies), "spinifex-workers"},
 		)
 	}
 
@@ -945,97 +954,97 @@ func (d *Daemon) subscribeAll() error {
 	// stable while real bodies land.
 	if d.eksService != nil {
 		subs = append(subs,
-			natsSub{"eks.CreateCluster", d.handleEKSCreateCluster, "spinifex-workers"},
-			natsSub{"eks.DescribeCluster", d.handleEKSDescribeCluster, "spinifex-workers"},
-			natsSub{"eks.ListClusters", d.handleEKSListClusters, "spinifex-workers"},
-			natsSub{"eks.UpdateClusterConfig", d.handleEKSUpdateClusterConfig, "spinifex-workers"},
-			natsSub{"eks.UpdateClusterVersion", d.handleEKSUpdateClusterVersion, "spinifex-workers"},
-			natsSub{"eks.DeleteCluster", d.handleEKSDeleteCluster, "spinifex-workers"},
-			natsSub{"eks.CreateNodegroup", d.handleEKSCreateNodegroup, "spinifex-workers"},
-			natsSub{"eks.DescribeNodegroup", d.handleEKSDescribeNodegroup, "spinifex-workers"},
-			natsSub{"eks.ListNodegroups", d.handleEKSListNodegroups, "spinifex-workers"},
-			natsSub{"eks.UpdateNodegroupConfig", d.handleEKSUpdateNodegroupConfig, "spinifex-workers"},
-			natsSub{"eks.UpdateNodegroupVersion", d.handleEKSUpdateNodegroupVersion, "spinifex-workers"},
-			natsSub{"eks.DeleteNodegroup", d.handleEKSDeleteNodegroup, "spinifex-workers"},
-			natsSub{"eks.CreateAccessEntry", d.handleEKSCreateAccessEntry, "spinifex-workers"},
-			natsSub{"eks.DescribeAccessEntry", d.handleEKSDescribeAccessEntry, "spinifex-workers"},
-			natsSub{"eks.ListAccessEntries", d.handleEKSListAccessEntries, "spinifex-workers"},
-			natsSub{"eks.UpdateAccessEntry", d.handleEKSUpdateAccessEntry, "spinifex-workers"},
-			natsSub{"eks.DeleteAccessEntry", d.handleEKSDeleteAccessEntry, "spinifex-workers"},
-			natsSub{"eks.AssociateAccessPolicy", d.handleEKSAssociateAccessPolicy, "spinifex-workers"},
-			natsSub{"eks.DisassociateAccessPolicy", d.handleEKSDisassociateAccessPolicy, "spinifex-workers"},
-			natsSub{"eks.ListAssociatedAccessPolicies", d.handleEKSListAssociatedAccessPolicies, "spinifex-workers"},
-			natsSub{"eks.ListAccessPolicies", d.handleEKSListAccessPolicies, "spinifex-workers"},
-			natsSub{"eks.ListAddons", d.handleEKSListAddons, "spinifex-workers"},
-			natsSub{"eks.DescribeAddonVersions", d.handleEKSDescribeAddonVersions, "spinifex-workers"},
-			natsSub{"eks.CreateAddon", d.handleEKSCreateAddon, "spinifex-workers"},
-			natsSub{"eks.DeleteAddon", d.handleEKSDeleteAddon, "spinifex-workers"},
-			natsSub{"eks.DescribeAddon", d.handleEKSDescribeAddon, "spinifex-workers"},
-			natsSub{"eks.UpdateAddon", d.handleEKSUpdateAddon, "spinifex-workers"},
-			natsSub{"eks.ListStagedAddonManifests", d.handleEKSListStagedAddonManifests, "spinifex-workers"},
-			natsSub{"eks.GetRecoveryDirective", d.handleEKSGetRecoveryDirective, "spinifex-workers"},
-			natsSub{"eks.SetRecoveryDirective", d.handleEKSSetRecoveryDirective, "spinifex-workers"},
-			natsSub{"eks.RestoreSnapshot", d.handleEKSRestoreSnapshot, "spinifex-workers"},
-			natsSub{"eks.AssociateIdentityProviderConfig", d.handleEKSAssociateIdentityProviderConfig, "spinifex-workers"},
-			natsSub{"eks.DescribeIdentityProviderConfig", d.handleEKSDescribeIdentityProviderConfig, "spinifex-workers"},
-			natsSub{"eks.ListIdentityProviderConfigs", d.handleEKSListIdentityProviderConfigs, "spinifex-workers"},
-			natsSub{"eks.DisassociateIdentityProviderConfig", d.handleEKSDisassociateIdentityProviderConfig, "spinifex-workers"},
-			natsSub{"eks.TagResource", d.handleEKSTagResource, "spinifex-workers"},
-			natsSub{"eks.UntagResource", d.handleEKSUntagResource, "spinifex-workers"},
-			natsSub{"eks.ListTagsForResource", d.handleEKSListTagsForResource, "spinifex-workers"},
+			natsSub{"eks.CreateCluster", handleNATSRequestWithPrincipal(d.eksService.CreateCluster), "spinifex-workers"},
+			natsSub{"eks.DescribeCluster", handleNATSRequest(d.eksService.DescribeCluster), "spinifex-workers"},
+			natsSub{"eks.ListClusters", handleNATSRequest(d.eksService.ListClusters), "spinifex-workers"},
+			natsSub{"eks.UpdateClusterConfig", handleNATSRequest(d.eksService.UpdateClusterConfig), "spinifex-workers"},
+			natsSub{"eks.UpdateClusterVersion", handleNATSRequest(d.eksService.UpdateClusterVersion), "spinifex-workers"},
+			natsSub{"eks.DeleteCluster", handleNATSRequest(d.eksService.DeleteCluster), "spinifex-workers"},
+			natsSub{"eks.CreateNodegroup", handleNATSRequest(d.eksService.CreateNodegroup), "spinifex-workers"},
+			natsSub{"eks.DescribeNodegroup", handleNATSRequest(d.eksService.DescribeNodegroup), "spinifex-workers"},
+			natsSub{"eks.ListNodegroups", handleNATSRequest(d.eksService.ListNodegroups), "spinifex-workers"},
+			natsSub{"eks.UpdateNodegroupConfig", handleNATSRequest(d.eksService.UpdateNodegroupConfig), "spinifex-workers"},
+			natsSub{"eks.UpdateNodegroupVersion", handleNATSRequest(d.eksService.UpdateNodegroupVersion), "spinifex-workers"},
+			natsSub{"eks.DeleteNodegroup", handleNATSRequest(d.eksService.DeleteNodegroup), "spinifex-workers"},
+			natsSub{"eks.CreateAccessEntry", handleNATSRequest(d.eksService.CreateAccessEntry), "spinifex-workers"},
+			natsSub{"eks.DescribeAccessEntry", handleNATSRequest(d.eksService.DescribeAccessEntry), "spinifex-workers"},
+			natsSub{"eks.ListAccessEntries", handleNATSRequest(d.eksService.ListAccessEntries), "spinifex-workers"},
+			natsSub{"eks.UpdateAccessEntry", handleNATSRequest(d.eksService.UpdateAccessEntry), "spinifex-workers"},
+			natsSub{"eks.DeleteAccessEntry", handleNATSRequest(d.eksService.DeleteAccessEntry), "spinifex-workers"},
+			natsSub{"eks.AssociateAccessPolicy", handleNATSRequest(d.eksService.AssociateAccessPolicy), "spinifex-workers"},
+			natsSub{"eks.DisassociateAccessPolicy", handleNATSRequest(d.eksService.DisassociateAccessPolicy), "spinifex-workers"},
+			natsSub{"eks.ListAssociatedAccessPolicies", handleNATSRequest(d.eksService.ListAssociatedAccessPolicies), "spinifex-workers"},
+			natsSub{"eks.ListAccessPolicies", handleNATSRequest(d.eksService.ListAccessPolicies), "spinifex-workers"},
+			natsSub{"eks.ListAddons", handleNATSRequest(d.eksService.ListAddons), "spinifex-workers"},
+			natsSub{"eks.DescribeAddonVersions", handleNATSRequest(d.eksService.DescribeAddonVersions), "spinifex-workers"},
+			natsSub{"eks.CreateAddon", handleNATSRequest(d.eksService.CreateAddon), "spinifex-workers"},
+			natsSub{"eks.DeleteAddon", handleNATSRequest(d.eksService.DeleteAddon), "spinifex-workers"},
+			natsSub{"eks.DescribeAddon", handleNATSRequest(d.eksService.DescribeAddon), "spinifex-workers"},
+			natsSub{"eks.UpdateAddon", handleNATSRequest(d.eksService.UpdateAddon), "spinifex-workers"},
+			natsSub{"eks.ListStagedAddonManifests", handleNATSRequest(d.eksService.ListStagedAddonManifests), "spinifex-workers"},
+			natsSub{"eks.GetRecoveryDirective", handleNATSRequest(d.eksService.GetRecoveryDirective), "spinifex-workers"},
+			natsSub{"eks.SetRecoveryDirective", handleNATSRequest(d.eksService.SetRecoveryDirective), "spinifex-workers"},
+			natsSub{"eks.RestoreSnapshot", handleNATSRequest(d.eksService.RestoreSnapshot), "spinifex-workers"},
+			natsSub{"eks.AssociateIdentityProviderConfig", handleNATSRequest(d.eksService.AssociateIdentityProviderConfig), "spinifex-workers"},
+			natsSub{"eks.DescribeIdentityProviderConfig", handleNATSRequest(d.eksService.DescribeIdentityProviderConfig), "spinifex-workers"},
+			natsSub{"eks.ListIdentityProviderConfigs", handleNATSRequest(d.eksService.ListIdentityProviderConfigs), "spinifex-workers"},
+			natsSub{"eks.DisassociateIdentityProviderConfig", handleNATSRequest(d.eksService.DisassociateIdentityProviderConfig), "spinifex-workers"},
+			natsSub{"eks.TagResource", handleNATSRequest(d.eksService.TagResource), "spinifex-workers"},
+			natsSub{"eks.UntagResource", handleNATSRequest(d.eksService.UntagResource), "spinifex-workers"},
+			natsSub{"eks.ListTagsForResource", handleNATSRequest(d.eksService.ListTagsForResource), "spinifex-workers"},
 		)
 	}
 
 	// ECS gateway → daemon subscriptions (control plane; per-account KV).
 	if d.ecsService != nil {
 		subs = append(subs,
-			natsSub{"ecs.CreateCluster", d.handleECSCreateCluster, "spinifex-workers"},
-			natsSub{"ecs.DeleteCluster", d.handleECSDeleteCluster, "spinifex-workers"},
-			natsSub{"ecs.DescribeClusters", d.handleECSDescribeClusters, "spinifex-workers"},
-			natsSub{"ecs.ListClusters", d.handleECSListClusters, "spinifex-workers"},
-			natsSub{"ecs.RegisterTaskDefinition", d.handleECSRegisterTaskDefinition, "spinifex-workers"},
-			natsSub{"ecs.DeregisterTaskDefinition", d.handleECSDeregisterTaskDefinition, "spinifex-workers"},
-			natsSub{"ecs.DescribeTaskDefinition", d.handleECSDescribeTaskDefinition, "spinifex-workers"},
-			natsSub{"ecs.ListTaskDefinitions", d.handleECSListTaskDefinitions, "spinifex-workers"},
-			natsSub{"ecs.RegisterContainerInstance", d.handleECSRegisterContainerInstance, "spinifex-workers"},
-			natsSub{"ecs.DeregisterContainerInstance", d.handleECSDeregisterContainerInstance, "spinifex-workers"},
-			natsSub{"ecs.UpdateContainerInstancesState", d.handleECSUpdateContainerInstancesState, "spinifex-workers"},
-			natsSub{"ecs.DescribeContainerInstances", d.handleECSDescribeContainerInstances, "spinifex-workers"},
-			natsSub{"ecs.ListContainerInstances", d.handleECSListContainerInstances, "spinifex-workers"},
-			natsSub{"ecs.RunTask", d.handleECSRunTask, "spinifex-workers"},
-			natsSub{"ecs.StartTask", d.handleECSStartTask, "spinifex-workers"},
-			natsSub{"ecs.StopTask", d.handleECSStopTask, "spinifex-workers"},
-			natsSub{"ecs.DescribeTasks", d.handleECSDescribeTasks, "spinifex-workers"},
-			natsSub{"ecs.ListTasks", d.handleECSListTasks, "spinifex-workers"},
-			natsSub{"ecs.CreateService", d.handleECSCreateService, "spinifex-workers"},
-			natsSub{"ecs.UpdateService", d.handleECSUpdateService, "spinifex-workers"},
-			natsSub{"ecs.DeleteService", d.handleECSDeleteService, "spinifex-workers"},
-			natsSub{"ecs.DescribeServices", d.handleECSDescribeServices, "spinifex-workers"},
-			natsSub{"ecs.ListServices", d.handleECSListServices, "spinifex-workers"},
-			natsSub{"ecs.SubmitTaskStateChange", d.handleECSSubmitTaskStateChange, "spinifex-workers"},
-			natsSub{"ecs.PollAssignments", d.handleECSPollAssignments, "spinifex-workers"},
-			natsSub{"ecs.ReportTaskGPU", d.handleECSReportTaskGPU, "spinifex-workers"},
-			natsSub{"ecs.ProvisionCapacity", d.handleECSProvisionCapacity, "spinifex-workers"},
-			natsSub{"ecs.TagResource", d.handleECSTagResource, "spinifex-workers"},
-			natsSub{"ecs.UntagResource", d.handleECSUntagResource, "spinifex-workers"},
-			natsSub{"ecs.ListTagsForResource", d.handleECSListTagsForResource, "spinifex-workers"},
-			natsSub{"ecs.PutClusterCapacityProviders", d.handleECSPutClusterCapacityProviders, "spinifex-workers"},
-			natsSub{"ecs.CreateCapacityProvider", d.handleECSCreateCapacityProvider, "spinifex-workers"},
-			natsSub{"ecs.DescribeCapacityProviders", d.handleECSDescribeCapacityProviders, "spinifex-workers"},
-			natsSub{"ecs.DeleteCapacityProvider", d.handleECSDeleteCapacityProvider, "spinifex-workers"},
+			natsSub{"ecs.CreateCluster", handleNATSRequest(d.ecsService.CreateCluster), "spinifex-workers"},
+			natsSub{"ecs.DeleteCluster", handleNATSRequest(d.ecsService.DeleteCluster), "spinifex-workers"},
+			natsSub{"ecs.DescribeClusters", handleNATSRequest(d.ecsService.DescribeClusters), "spinifex-workers"},
+			natsSub{"ecs.ListClusters", handleNATSRequest(d.ecsService.ListClusters), "spinifex-workers"},
+			natsSub{"ecs.RegisterTaskDefinition", handleNATSRequest(d.ecsService.RegisterTaskDefinition), "spinifex-workers"},
+			natsSub{"ecs.DeregisterTaskDefinition", handleNATSRequest(d.ecsService.DeregisterTaskDefinition), "spinifex-workers"},
+			natsSub{"ecs.DescribeTaskDefinition", handleNATSRequest(d.ecsService.DescribeTaskDefinition), "spinifex-workers"},
+			natsSub{"ecs.ListTaskDefinitions", handleNATSRequest(d.ecsService.ListTaskDefinitions), "spinifex-workers"},
+			natsSub{"ecs.RegisterContainerInstance", handleNATSRequest(d.ecsService.RegisterContainerInstance), "spinifex-workers"},
+			natsSub{"ecs.DeregisterContainerInstance", handleNATSRequest(d.ecsService.DeregisterContainerInstance), "spinifex-workers"},
+			natsSub{"ecs.UpdateContainerInstancesState", handleNATSRequest(d.ecsService.UpdateContainerInstancesState), "spinifex-workers"},
+			natsSub{"ecs.DescribeContainerInstances", handleNATSRequest(d.ecsService.DescribeContainerInstances), "spinifex-workers"},
+			natsSub{"ecs.ListContainerInstances", handleNATSRequest(d.ecsService.ListContainerInstances), "spinifex-workers"},
+			natsSub{"ecs.RunTask", handleNATSRequest(d.ecsService.RunTask), "spinifex-workers"},
+			natsSub{"ecs.StartTask", handleNATSRequest(d.ecsService.StartTask), "spinifex-workers"},
+			natsSub{"ecs.StopTask", handleNATSRequest(d.ecsService.StopTask), "spinifex-workers"},
+			natsSub{"ecs.DescribeTasks", handleNATSRequest(d.ecsService.DescribeTasks), "spinifex-workers"},
+			natsSub{"ecs.ListTasks", handleNATSRequest(d.ecsService.ListTasks), "spinifex-workers"},
+			natsSub{"ecs.CreateService", handleNATSRequest(d.ecsService.CreateService), "spinifex-workers"},
+			natsSub{"ecs.UpdateService", handleNATSRequest(d.ecsService.UpdateService), "spinifex-workers"},
+			natsSub{"ecs.DeleteService", handleNATSRequest(d.ecsService.DeleteService), "spinifex-workers"},
+			natsSub{"ecs.DescribeServices", handleNATSRequest(d.ecsService.DescribeServices), "spinifex-workers"},
+			natsSub{"ecs.ListServices", handleNATSRequest(d.ecsService.ListServices), "spinifex-workers"},
+			natsSub{"ecs.SubmitTaskStateChange", handleNATSRequest(d.ecsService.SubmitTaskStateChange), "spinifex-workers"},
+			natsSub{"ecs.PollAssignments", handleNATSRequest(d.ecsService.PollAssignments), "spinifex-workers"},
+			natsSub{"ecs.ReportTaskGPU", handleNATSRequest(d.ecsService.ReportTaskGPU), "spinifex-workers"},
+			natsSub{"ecs.ProvisionCapacity", handleNATSRequest(d.ecsService.ProvisionCapacity), "spinifex-workers"},
+			natsSub{"ecs.TagResource", handleNATSRequest(d.ecsService.TagResource), "spinifex-workers"},
+			natsSub{"ecs.UntagResource", handleNATSRequest(d.ecsService.UntagResource), "spinifex-workers"},
+			natsSub{"ecs.ListTagsForResource", handleNATSRequest(d.ecsService.ListTagsForResource), "spinifex-workers"},
+			natsSub{"ecs.PutClusterCapacityProviders", handleNATSRequest(d.ecsService.PutClusterCapacityProviders), "spinifex-workers"},
+			natsSub{"ecs.CreateCapacityProvider", handleNATSRequest(d.ecsService.CreateCapacityProvider), "spinifex-workers"},
+			natsSub{"ecs.DescribeCapacityProviders", handleNATSRequest(d.ecsService.DescribeCapacityProviders), "spinifex-workers"},
+			natsSub{"ecs.DeleteCapacityProvider", handleNATSRequest(d.ecsService.DeleteCapacityProvider), "spinifex-workers"},
 		)
 	}
 
 	// ACM gateway → daemon subscriptions (minimal certificate store).
 	if d.acmService != nil {
 		subs = append(subs,
-			natsSub{"acm.ImportCertificate", d.handleACMImportCertificate, "spinifex-workers"},
-			natsSub{"acm.DescribeCertificate", d.handleACMDescribeCertificate, "spinifex-workers"},
-			natsSub{"acm.ListCertificates", d.handleACMListCertificates, "spinifex-workers"},
-			natsSub{"acm.DeleteCertificate", d.handleACMDeleteCertificate, "spinifex-workers"},
-			natsSub{"acm.ListTagsForCertificate", d.handleACMListTagsForCertificate, "spinifex-workers"},
-			natsSub{"acm.AddTagsToCertificate", d.handleACMAddTagsToCertificate, "spinifex-workers"},
-			natsSub{"acm.RemoveTagsFromCertificate", d.handleACMRemoveTagsFromCertificate, "spinifex-workers"},
+			natsSub{"acm.ImportCertificate", handleNATSRequest(d.acmService.ImportCertificate), "spinifex-workers"},
+			natsSub{"acm.DescribeCertificate", handleNATSRequest(d.acmService.DescribeCertificate), "spinifex-workers"},
+			natsSub{"acm.ListCertificates", handleNATSRequest(d.acmService.ListCertificates), "spinifex-workers"},
+			natsSub{"acm.DeleteCertificate", handleNATSRequest(d.acmService.DeleteCertificate), "spinifex-workers"},
+			natsSub{"acm.ListTagsForCertificate", handleNATSRequest(d.acmService.ListTagsForCertificate), "spinifex-workers"},
+			natsSub{"acm.AddTagsToCertificate", handleNATSRequest(d.acmService.AddTagsToCertificate), "spinifex-workers"},
+			natsSub{"acm.RemoveTagsFromCertificate", handleNATSRequest(d.acmService.RemoveTagsFromCertificate), "spinifex-workers"},
 		)
 	}
 
@@ -1043,28 +1052,28 @@ func (d *Daemon) subscribeAll() error {
 	// JetStream KV metadata; blob/manifest bytes never traverse these subjects.
 	if d.ecrMetaService != nil {
 		subs = append(subs,
-			natsSub{handlers_ecr.SubjectRepoCreate, d.handleECRRepoCreate, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectRepoDescribe, d.handleECRRepoDescribe, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectRepoList, d.handleECRRepoList, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectRepoDelete, d.handleECRRepoDelete, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectPolicyPut, d.handleECRPolicyPut, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectPolicyGet, d.handleECRPolicyGet, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectPolicyDelete, d.handleECRPolicyDelete, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectLifecyclePut, d.handleECRLifecyclePut, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectLifecycleGet, d.handleECRLifecycleGet, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectLifecycleDelete, d.handleECRLifecycleDelete, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectTagPut, d.handleECRTagPut, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectTagGet, d.handleECRTagGet, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectTagList, d.handleECRTagList, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectTagDelete, d.handleECRTagDelete, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectManifestPut, d.handleECRManifestPut, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectManifestDescribe, d.handleECRManifestDescribe, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectManifestList, d.handleECRManifestList, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectManifestDelete, d.handleECRManifestDelete, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectUploadCreate, d.handleECRUploadCreate, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectUploadGet, d.handleECRUploadGet, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectUploadUpdate, d.handleECRUploadUpdate, "spinifex-workers"},
-			natsSub{handlers_ecr.SubjectUploadDelete, d.handleECRUploadDelete, "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectRepoCreate, handleNATSRequest(d.ecrMetaService.RepoCreate), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectRepoDescribe, handleNATSRequest(d.ecrMetaService.RepoDescribe), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectRepoList, handleNATSRequest(d.ecrMetaService.RepoList), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectRepoDelete, handleNATSRequest(d.ecrMetaService.RepoDelete), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectPolicyPut, handleNATSRequest(d.ecrMetaService.PolicyPut), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectPolicyGet, handleNATSRequest(d.ecrMetaService.PolicyGet), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectPolicyDelete, handleNATSRequest(d.ecrMetaService.PolicyDelete), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectLifecyclePut, handleNATSRequest(d.ecrMetaService.LifecyclePut), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectLifecycleGet, handleNATSRequest(d.ecrMetaService.LifecycleGet), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectLifecycleDelete, handleNATSRequest(d.ecrMetaService.LifecycleDelete), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectTagPut, handleNATSRequest(d.ecrMetaService.TagPut), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectTagGet, handleNATSRequest(d.ecrMetaService.TagGet), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectTagList, handleNATSRequest(d.ecrMetaService.TagList), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectTagDelete, handleNATSRequest(d.ecrMetaService.TagDelete), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectManifestPut, handleNATSRequest(d.ecrMetaService.ManifestPut), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectManifestDescribe, handleNATSRequest(d.ecrMetaService.ManifestDescribe), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectManifestList, handleNATSRequest(d.ecrMetaService.ManifestList), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectManifestDelete, handleNATSRequest(d.ecrMetaService.ManifestDelete), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectUploadCreate, handleNATSRequest(d.ecrMetaService.UploadCreate), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectUploadGet, handleNATSRequest(d.ecrMetaService.UploadGet), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectUploadUpdate, handleNATSRequest(d.ecrMetaService.UploadUpdate), "spinifex-workers"},
+			natsSub{handlers_ecr.SubjectUploadDelete, handleNATSRequest(d.ecrMetaService.UploadDelete), "spinifex-workers"},
 		)
 	}
 
@@ -1072,12 +1081,12 @@ func (d *Daemon) subscribeAll() error {
 	// the disabled stub, so reads return empty lists and mutations a clean
 	// UnsupportedOperation instead of a NATS timeout.
 	subs = append(subs,
-		natsSub{"ec2.AllocateAddress", d.handleEC2AllocateAddress, "spinifex-workers"},
-		natsSub{"ec2.ReleaseAddress", d.handleEC2ReleaseAddress, "spinifex-workers"},
-		natsSub{"ec2.AssociateAddress", d.handleEC2AssociateAddress, "spinifex-workers"},
-		natsSub{"ec2.DisassociateAddress", d.handleEC2DisassociateAddress, "spinifex-workers"},
-		natsSub{"ec2.DescribeAddresses", d.handleEC2DescribeAddresses, "spinifex-workers"},
-		natsSub{"ec2.DescribeAddressesAttribute", d.handleEC2DescribeAddressesAttribute, "spinifex-workers"},
+		natsSub{"ec2.AllocateAddress", handleNATSRequest(d.eipService.AllocateAddress), "spinifex-workers"},
+		natsSub{"ec2.ReleaseAddress", handleNATSRequest(d.eipService.ReleaseAddress), "spinifex-workers"},
+		natsSub{"ec2.AssociateAddress", handleNATSRequest(d.eipService.AssociateAddress), "spinifex-workers"},
+		natsSub{"ec2.DisassociateAddress", handleNATSRequest(d.eipService.DisassociateAddress), "spinifex-workers"},
+		natsSub{"ec2.DescribeAddresses", handleNATSRequest(d.eipService.DescribeAddresses), "spinifex-workers"},
+		natsSub{"ec2.DescribeAddressesAttribute", handleNATSRequest(d.eipService.DescribeAddressesAttribute), "spinifex-workers"},
 	)
 
 	for _, s := range subs {
@@ -1289,8 +1298,8 @@ func (d *Daemon) startCluster() error {
 	}
 
 	// Remove the obsolete spinifex-dhcp-leases bucket (idempotent).
-	if js, jsErr := d.natsConn.JetStream(); jsErr == nil {
-		if err := utils.DeleteKVBucketIfExists(js, "spinifex-dhcp-leases"); err != nil {
+	if js, jsErr := jetstream.New(d.natsConn); jsErr == nil {
+		if err := kvutil.DeleteBucketIfExists(d.ctx, js, "spinifex-dhcp-leases"); err != nil {
 			slog.Warn("Failed to delete obsolete spinifex-dhcp-leases KV bucket", "err", err)
 		}
 	}
@@ -1317,15 +1326,19 @@ func (d *Daemon) startCluster() error {
 	// Create services before loading/launching instances, since LaunchInstance depends on them
 	store := objectstore.NewS3ObjectStoreFromConfig(admin.DialTarget(d.config.Predastore.Host), d.config.Predastore.Region, d.config.Predastore.AccessKey, d.config.Predastore.SecretKey)
 	d.instanceService = handlers_ec2_instance.NewInstanceServiceImpl(d.config, d.resourceMgr.instanceTypes, d.natsConn, store, d.vmMgr, d.resourceMgr, d.jsManager)
+	d.dnsWriter = handlers_dns.NewWriter(d.config, d.clusterConfig, d.natsConn)
+	d.dnsReconciler = handlers_dns.NewReconciler(d.config, d.natsConn, d.dnsDesiredSet)
+	d.dnsBaseDomain = handlers_dns.ResolveBaseDomain(d.config)
+	d.dnsInternalDomain = handlers_dns.ResolveInternalDomain(d.config)
 	d.keyService = handlers_ec2_key.NewKeyServiceImpl(d.config)
 	d.imageService = handlers_ec2_image.NewImageServiceImpl(d.config)
 
 	type snapResult struct {
 		svc *handlers_ec2_snapshot.SnapshotServiceImpl
-		kv  nats.KeyValue
+		kv  jetstream.KeyValue
 	}
 	snap, err := initServiceWithRetry("snapshot service", func() (snapResult, error) {
-		svc, kv, err := handlers_ec2_snapshot.NewSnapshotServiceImplWithNATS(d.config, d.natsConn)
+		svc, kv, err := handlers_ec2_snapshot.NewSnapshotServiceImplWithNATS(d.ctx, d.config, d.natsConn)
 		return snapResult{svc, kv}, err
 	})
 	if err != nil {
@@ -1337,42 +1350,42 @@ func (d *Daemon) startCluster() error {
 	d.tagsService = handlers_ec2_tags.NewTagsServiceImpl(d.config)
 
 	d.eigwService, err = initServiceWithRetry("EIGW service", func() (*handlers_ec2_eigw.EgressOnlyIGWServiceImpl, error) {
-		return handlers_ec2_eigw.NewEgressOnlyIGWServiceImplWithNATS(d.config, d.natsConn)
+		return handlers_ec2_eigw.NewEgressOnlyIGWServiceImplWithNATS(d.ctx, d.config, d.natsConn)
 	})
 	if err != nil {
 		return fmt.Errorf("failed to initialize EIGW service: %w", err)
 	}
 
 	d.igwService, err = initServiceWithRetry("IGW service", func() (*handlers_ec2_igw.IGWServiceImpl, error) {
-		return handlers_ec2_igw.NewIGWServiceImplWithNATS(d.config, d.natsConn)
+		return handlers_ec2_igw.NewIGWServiceImplWithNATS(d.ctx, d.config, d.natsConn)
 	})
 	if err != nil {
 		return fmt.Errorf("failed to initialize IGW service: %w", err)
 	}
 
 	d.placementGroupService, err = initServiceWithRetry("placement group service", func() (*handlers_ec2_placementgroup.PlacementGroupServiceImpl, error) {
-		return handlers_ec2_placementgroup.NewPlacementGroupServiceImplWithNATS(d.config, d.natsConn)
+		return handlers_ec2_placementgroup.NewPlacementGroupServiceImplWithNATS(d.ctx, d.config, d.natsConn)
 	})
 	if err != nil {
 		return fmt.Errorf("failed to initialize placement group service: %w", err)
 	}
 
 	d.launchTemplateService, err = initServiceWithRetry("launch template service", func() (*handlers_ec2_launchtemplate.LaunchTemplateServiceImpl, error) {
-		return handlers_ec2_launchtemplate.NewLaunchTemplateServiceImplWithNATS(d.config, d.natsConn)
+		return handlers_ec2_launchtemplate.NewLaunchTemplateServiceImplWithNATS(d.ctx, d.config, d.natsConn)
 	})
 	if err != nil {
 		return fmt.Errorf("failed to initialize launch template service: %w", err)
 	}
 
 	d.spotInstanceService, err = initServiceWithRetry("spot instance service", func() (*handlers_ec2_spotinstance.SpotInstanceServiceImpl, error) {
-		return handlers_ec2_spotinstance.NewSpotInstanceServiceImplWithNATS(d.config, d.natsConn)
+		return handlers_ec2_spotinstance.NewSpotInstanceServiceImplWithNATS(d.ctx, d.config, d.natsConn)
 	})
 	if err != nil {
 		return fmt.Errorf("failed to initialize spot instance service: %w", err)
 	}
 
 	d.vpcService, err = initServiceWithRetry("VPC service", func() (*handlers_ec2_vpc.VPCServiceImpl, error) {
-		return handlers_ec2_vpc.NewVPCServiceImplWithNATS(d.config, d.natsConn)
+		return handlers_ec2_vpc.NewVPCServiceImplWithNATS(d.ctx, d.config, d.natsConn)
 	})
 	if err != nil {
 		return fmt.Errorf("failed to initialize VPC service: %w", err)
@@ -1383,7 +1396,7 @@ func (d *Daemon) startCluster() error {
 	d.vpcService.SetDefaultPublicIPMapping(d.hasPublicIPPools())
 
 	d.routeTableService, err = initServiceWithRetry("RouteTable service", func() (*handlers_ec2_routetable.RouteTableServiceImpl, error) {
-		return handlers_ec2_routetable.NewRouteTableServiceImplWithNATS(d.config, d.natsConn)
+		return handlers_ec2_routetable.NewRouteTableServiceImplWithNATS(d.ctx, d.config, d.natsConn)
 	})
 	if err != nil {
 		return fmt.Errorf("failed to initialize RouteTable service: %w", err)
@@ -1393,7 +1406,7 @@ func (d *Daemon) startCluster() error {
 	d.igwService.SetGatePublisher(d.routeTableService)
 
 	d.natGatewayService, err = initServiceWithRetry("NatGateway service", func() (*handlers_ec2_natgw.NatGatewayServiceImpl, error) {
-		return handlers_ec2_natgw.NewNatGatewayServiceImplWithNATS(d.natsConn)
+		return handlers_ec2_natgw.NewNatGatewayServiceImplWithNATS(d.ctx, d.natsConn)
 	})
 	if err != nil {
 		return fmt.Errorf("failed to initialize NatGateway service: %w", err)
@@ -1403,7 +1416,7 @@ func (d *Daemon) startCluster() error {
 	// mode with a public pool alongside the transit segment). The transit pool
 	// never enters IPAM — its addresses are gateway-LRP plumbing, not EIPs.
 	if d.hasPublicIPPools() {
-		js, jsErr := d.natsConn.JetStream()
+		js, jsErr := jetstream.New(d.natsConn)
 		if jsErr != nil {
 			slog.Warn("Failed to get JetStream for external IPAM", "err", jsErr)
 		} else {
@@ -1429,7 +1442,7 @@ func (d *Daemon) startCluster() error {
 					anyDHCP = true
 				}
 			}
-			d.externalIPAM, err = handlers_ec2_vpc.NewExternalIPAM(js, pools)
+			d.externalIPAM, err = handlers_ec2_vpc.NewExternalIPAM(d.ctx, js, pools)
 			if err != nil {
 				slog.Warn("Failed to initialize external IPAM", "err", err)
 			} else {
@@ -1446,7 +1459,7 @@ func (d *Daemon) startCluster() error {
 
 	// Initialize EIP service if external IPAM is available
 	if d.externalIPAM != nil && d.vpcService != nil {
-		eipSvc, eipErr := handlers_ec2_eip.NewEIPServiceImpl(d.natsConn, d.externalIPAM, d.vpcService)
+		eipSvc, eipErr := handlers_ec2_eip.NewEIPServiceImpl(d.ctx, d.natsConn, d.externalIPAM, d.vpcService)
 		if eipErr != nil {
 			slog.Warn("Failed to initialize EIP service", "err", eipErr)
 		} else {
@@ -1456,11 +1469,11 @@ func (d *Daemon) startCluster() error {
 
 		// Inject external IPAM + EIP KV into VPC service so DeleteNetworkInterface
 		// can release auto-assigned public IPs and NAT rules.
-		eipJS, eipJSErr := d.natsConn.JetStream()
+		eipJS, eipJSErr := jetstream.New(d.natsConn)
 		if eipJSErr != nil {
 			slog.Warn("Failed to get JetStream for VPC external IPAM injection", "err", eipJSErr)
 		} else {
-			eipKV, eipKVErr := utils.GetOrCreateKVBucket(eipJS, handlers_ec2_eip.KVBucketEIPs, 10)
+			eipKV, eipKVErr := kvutil.GetOrCreateBucket(d.ctx, eipJS, handlers_ec2_eip.KVBucketEIPs, 10)
 			if eipKVErr != nil {
 				slog.Warn("Failed to get EIP KV bucket for VPC service", "err", eipKVErr)
 			} else {
@@ -1484,7 +1497,7 @@ func (d *Daemon) startCluster() error {
 	}
 
 	d.accountService, err = initServiceWithRetry("account settings service", func() (*handlers_ec2_account.AccountSettingsServiceImpl, error) {
-		return handlers_ec2_account.NewAccountSettingsServiceImplWithNATS(d.config, d.natsConn)
+		return handlers_ec2_account.NewAccountSettingsServiceImplWithNATS(d.ctx, d.config, d.natsConn)
 	})
 	if err != nil {
 		return fmt.Errorf("failed to initialize account settings service: %w", err)
@@ -1536,9 +1549,9 @@ func (d *Daemon) startCluster() error {
 	// reaper. The scheduler is disabled (handlers still serve) when JetStream is
 	// unavailable.
 	d.ecsService = handlers_ecs.NewService(d.natsConn, d.config.Region, d.clusterConfig.AWS.InternalSuffix).WithDeps(d.buildECSServiceDeps())
-	if js, jsErr := d.natsConn.JetStream(); jsErr != nil {
+	if js, jsErr := jetstream.New(d.natsConn); jsErr != nil {
 		slog.Warn("ECS scheduler disabled: JetStream unavailable", "err", jsErr)
-	} else if _, lbErr := handlers_ecs.InitLeaderBucket(js); lbErr != nil {
+	} else if _, lbErr := handlers_ecs.InitLeaderBucket(d.ctx, js); lbErr != nil {
 		slog.Warn("ECS scheduler disabled: leader bucket init failed", "err", lbErr)
 	} else {
 		d.ecsScheduler = handlers_ecs.NewScheduler(d.natsConn, d.ecsService, d.node)
@@ -1553,7 +1566,7 @@ func (d *Daemon) startCluster() error {
 	}
 
 	d.acmService, err = initServiceWithRetry("ACM service", func() (*handlers_acm.ACMServiceImpl, error) {
-		return handlers_acm.NewACMServiceImplWithNATS(d.config, d.natsConn)
+		return handlers_acm.NewACMServiceImplWithNATS(d.ctx, d.config, d.natsConn)
 	})
 	if err != nil {
 		return fmt.Errorf("failed to initialize ACM service: %w", err)
@@ -1562,7 +1575,7 @@ func (d *Daemon) startCluster() error {
 	// ECR metadata service: owns per-account JetStream KV for repos, tags,
 	// manifest records and upload-state CAS. Disabled (gateway returns NATS
 	// timeouts) when JetStream is unavailable.
-	if js, jsErr := d.natsConn.JetStream(); jsErr != nil {
+	if js, jsErr := jetstream.New(d.natsConn); jsErr != nil {
 		slog.Warn("ECR metadata service disabled: JetStream unavailable", "err", jsErr)
 	} else {
 		d.ecrMetaService = handlers_ecr.NewKVMetaService(js)
@@ -1604,8 +1617,13 @@ func (d *Daemon) startCluster() error {
 		return fmt.Errorf("restore instances: %w", err)
 	}
 
-	// Rebuild mgmt IP allocator so already-allocated IPs aren't reused.
-	if d.mgmtIPAllocator != nil {
+	// Bind the mgmt IP allocator to cluster KV now that JetStream is up (DDIL
+	// §1e-audit: startLocal built it KV-less). Rebuild then reconciles this
+	// node's already-running VMs into the cluster-wide record instead of
+	// just refreshing the local cache, so already-allocated IPs aren't
+	// reused by another node either.
+	if d.mgmtIPAllocator != nil && d.jsManager != nil {
+		d.mgmtIPAllocator.BindKV(d.jsManager, d.node)
 		d.mgmtIPAllocator.Rebuild(d.vmMgr.SnapshotMap())
 		slog.Info("Rebuilt mgmt IP allocator from restored instances", "allocated", d.mgmtIPAllocator.AllocatedCount())
 	}
@@ -1614,6 +1632,24 @@ func (d *Daemon) startCluster() error {
 		return fmt.Errorf("failed to subscribe to NATS topics: %w", err)
 	}
 
+	// DNS record writer: the single queue-group consumer of
+	// dns.recordset.change. No-op when northstar S3 is not configured.
+	if sub, err := d.dnsWriter.Subscribe(d.natsConn); err != nil {
+		return fmt.Errorf("failed to subscribe DNS record writer: %w", err)
+	} else if sub != nil {
+		d.natsSubscriptions[handlers_dns.SubjectRecordsetChange] = sub
+		slog.Info("Subscribed DNS record writer", "subject", handlers_dns.SubjectRecordsetChange, "queue", handlers_dns.QueueGroup)
+	}
+
+	// DNS drift backstop: periodically rebuild managed records
+	// from the live cross-tenant inventory and converge the zone. It publishes
+	// through the same queue-group writer, so every node running it serialises on
+	// one writer and never races the zone. No-op when northstar is not configured.
+	if d.dnsReconciler.Enabled() {
+		go d.dnsReconciler.Run(d.ctx)
+		slog.Info("Started DNS reconcile backstop", "interval", handlers_dns.DefaultReconcileInterval)
+	}
+
 	// Initialize per-instance-type NATS subscriptions for capacity-aware routing.
 	d.resourceMgr.initSubscriptions(d.natsConn, d.handleEC2RunInstances, d.handleSystemLaunchInstance, d.node)
 
@@ -1628,6 +1664,7 @@ func (d *Daemon) startCluster() error {
 		reapers := []vm.Reaper{
 			d.vmMgr.NewTerminatedTeardownReaper(),
 			d.vmMgr.NewOrphanQEMUReaper(),
+			d.vmMgr.NewStuckTerminateReaper(),
 		}
 		if eniRec := d.newENIReconciler(); eniRec != nil {
 			reapers = append(reapers, eniRec)
@@ -2340,7 +2377,7 @@ func (rm *ResourceManager) allocate(instanceType *ec2.InstanceTypeInfo) error {
 	return nil
 }
 
-// deallocate releases resources for an instance and updates NATS subscriptions
+// deallocate releases resources for an instance and updates NATS subscriptions.
 func (rm *ResourceManager) deallocate(instanceType *ec2.InstanceTypeInfo) {
 	rm.mu.Lock()
 	vCPUs := instanceTypeVCPUs(instanceType)

spinifex/daemon/daemon_handlers.go

@@ -66,63 +66,68 @@ func respondWithJSON(msg *nats.Msg, data any) {
 	}
 }
 
-// handleNATSRequest is a generic helper for the common unmarshal → service → marshal → respond pattern.
-// It opens a consumer span joining the producer's trace, extracts the account ID
-// from the NATS message header, and passes both to the service function.
-func handleNATSRequest[I any, O any](msg *nats.Msg, serviceFn func(context.Context, *I, string) (*O, error)) {
-	ctx, span := utils.StartConsumerSpan(msg)
-	defer span.End()
+// handleNATSRequest returns the nats.MsgHandler for the common unmarshal → service →
+// marshal → respond pattern. Per message the handler opens a consumer span joining the
+// producer's trace, extracts the account ID from the NATS message header, and passes
+// both to the service function.
+func handleNATSRequest[I any, O any](serviceFn func(context.Context, *I, string) (*O, error)) nats.MsgHandler {
+	return func(msg *nats.Msg) {
+		ctx, span := utils.StartConsumerSpan(msg)
+		defer span.End()
 
-	accountID := utils.AccountIDFromMsg(msg)
-	input := new(I)
-	if errResp := utils.UnmarshalJsonPayload(input, msg.Data); errResp != nil {
-		utils.MarkSpanError(span, errors.New(awserrors.ErrorInvalidParameterValue))
-		if err := msg.Respond(errResp); err != nil {
-			slog.ErrorContext(ctx, "Failed to respond to NATS request", "err", err)
+		accountID := utils.AccountIDFromMsg(msg)
+		input := new(I)
+		if errResp := utils.UnmarshalJsonPayload(input, msg.Data); errResp != nil {
+			utils.MarkSpanError(span, errors.New(awserrors.ErrorInvalidParameterValue))
+			if err := msg.Respond(errResp); err != nil {
+				slog.ErrorContext(ctx, "Failed to respond to NATS request", "err", err)
+			}
+			return
 		}
-		return
+		output, err := serviceFn(ctx, input, accountID)
+		if err != nil {
+			// The error was otherwise only recorded on the OTel span, invisible
+			// without a trace backend. Log at Error so handler failures show up
+			// in the journal too.
+			slog.ErrorContext(ctx, "handleNATSRequest: service call failed", "subject", msg.Subject, "err", err)
+			utils.MarkSpanError(span, err)
+			respondWithError(msg, awserrors.ValidErrorCodeFromError(err))
+			return
+		}
+		respondWithJSON(msg, output)
 	}
-	output, err := serviceFn(ctx, input, accountID)
-	if err != nil {
-		// The error was otherwise only recorded on the OTel span, invisible
-		// without a trace backend. Log at Error so handler failures show up
-		// in the journal too.
-		slog.ErrorContext(ctx, "handleNATSRequest: service call failed", "subject", msg.Subject, "err", err)
-		utils.MarkSpanError(span, err)
-		respondWithError(msg, awserrors.ValidErrorCode(err.Error()))
-		return
-	}
-	respondWithJSON(msg, output)
 }
 
 // handleNATSRequestWithPrincipal is handleNATSRequest for service methods that
 // also need the caller's IAM principal ARN (X-Principal-ARN header) — e.g. EKS
 // CreateCluster, which mints the bootstrap-creator-admin AccessEntry for the
 // caller.
-func handleNATSRequestWithPrincipal[I any, O any](msg *nats.Msg, serviceFn func(context.Context, *I, string, string) (*O, error)) {
-	ctx, span := utils.StartConsumerSpan(msg)
-	defer span.End()
+func handleNATSRequestWithPrincipal[I any, O any](serviceFn func(context.Context, *I, string, string) (*O, error)) nats.MsgHandler {
+	return func(msg *nats.Msg) {
+		ctx, span := utils.StartConsumerSpan(msg)
+		defer span.End()
 
-	accountID := utils.AccountIDFromMsg(msg)
-	principalARN := utils.PrincipalARNFromMsg(msg)
-	input := new(I)
-	if errResp := utils.UnmarshalJsonPayload(input, msg.Data); errResp != nil {
-		utils.MarkSpanError(span, errors.New(awserrors.ErrorInvalidParameterValue))
-		if err := msg.Respond(errResp); err != nil {
-			slog.ErrorContext(ctx, "Failed to respond to NATS request", "err", err)
+		accountID := utils.AccountIDFromMsg(msg)
+		principalARN := utils.PrincipalARNFromMsg(msg)
+		input := new(I)
+		if errResp := utils.UnmarshalJsonPayload(input, msg.Data); errResp != nil {
+			utils.MarkSpanError(span, errors.New(awserrors.ErrorInvalidParameterValue))
+			if err := msg.Respond(errResp); err != nil {
+				slog.ErrorContext(ctx, "Failed to respond to NATS request", "err", err)
+			}
+			return
 		}
-		return
+		output, err := serviceFn(ctx, input, accountID, principalARN)
+		if err != nil {
+			utils.MarkSpanError(span, err)
+			respondWithError(msg, awserrors.ValidErrorCodeFromError(err))
+			return
+		}
+		respondWithJSON(msg, output)
 	}
-	output, err := serviceFn(ctx, input, accountID, principalARN)
-	if err != nil {
-		utils.MarkSpanError(span, err)
-		respondWithError(msg, awserrors.ValidErrorCode(err.Error()))
-		return
-	}
-	respondWithJSON(msg, output)
 }
 
-// handleEC2Events processes incoming EC2 instance events (start, stop, terminate, attach-volume)
+// handleEC2Events processes incoming EC2 instance events (start, stop, terminate, attach-volume).
 func (d *Daemon) handleEC2Events(msg *nats.Msg) {
 	ctx, span := utils.StartConsumerSpan(msg)
 	defer span.End()
@@ -155,6 +160,13 @@ func (d *Daemon) handleEC2Events(msg *nats.Msg) {
 		d.handleAttachVolume(ctx, msg, command, instance)
 	case command.Attributes.DetachVolume:
 		d.handleDetachVolume(ctx, msg, command, instance)
+	case command.Attributes.DrainVolume:
+		// A drain flushes the volume's whole dirty set to S3, so it is bounded by
+		// the dirty set rather than by a unit of work. nats.go delivers a
+		// subscription's messages serially, so running it inline would stall
+		// every other command for this instance — stop, terminate, hot-plug —
+		// for the duration. It touches no VM state, so it is safe off-thread.
+		go d.handleDrainVolume(ctx, msg, command, instance)
 	case command.Attributes.AttachENI:
 		d.handleAttachNetworkInterface(ctx, msg, command, instance)
 	case command.Attributes.DetachENI:
@@ -171,7 +183,7 @@ func (d *Daemon) handleEC2Events(msg *nats.Msg) {
 		endOpSpan(opSpan, err)
 		if err != nil {
 			utils.MarkSpanError(span, err)
-			respondWithError(msg, awserrors.ValidErrorCode(err.Error()))
+			respondWithError(msg, awserrors.ValidErrorCodeFromError(err))
 			return
 		}
 		if err := msg.Respond(fmt.Appendf(nil, `{"status":"running","instanceId":"%s"}`, instance.ID)); err != nil {
@@ -183,7 +195,7 @@ func (d *Daemon) handleEC2Events(msg *nats.Msg) {
 		endOpSpan(opSpan, err)
 		if err != nil {
 			utils.MarkSpanError(span, err)
-			respondWithError(msg, awserrors.ValidErrorCode(err.Error()))
+			respondWithError(msg, awserrors.ValidErrorCodeFromError(err))
 			return
 		}
 		if err := msg.Respond([]byte(`{}`)); err != nil {
@@ -199,7 +211,7 @@ func (d *Daemon) handleEC2Events(msg *nats.Msg) {
 		endOpSpan(opSpan, err)
 		if err != nil {
 			utils.MarkSpanError(span, err)
-			respondWithError(msg, awserrors.ValidErrorCode(err.Error()))
+			respondWithError(msg, awserrors.ValidErrorCodeFromError(err))
 			return
 		}
 		if err := msg.Respond([]byte(`{}`)); err != nil {
@@ -213,7 +225,7 @@ func (d *Daemon) handleEC2Events(msg *nats.Msg) {
 
 // --- Admin / node management handlers ---
 
-// handleHealthCheck processes NATS health check requests
+// handleHealthCheck processes NATS health check requests.
 func (d *Daemon) handleHealthCheck(msg *nats.Msg) {
 	configHash, err := d.computeConfigHash()
 	if err != nil {
@@ -239,7 +251,7 @@ func (d *Daemon) handleHealthCheck(msg *nats.Msg) {
 }
 
 // handleNodeDiscover responds to node discovery requests with this node's ID
-// Used by the gateway to dynamically discover active spinifex nodes in the cluster
+// Used by the gateway to dynamically discover active spinifex nodes in the cluster.
 func (d *Daemon) handleNodeDiscover(msg *nats.Msg) {
 	response := types.NodeDiscoverResponse{
 		Node: d.node,

spinifex/daemon/daemon_handlers_account.go

@@ -1,29 +0,0 @@
-package daemon
-
-import (
-	"github.com/nats-io/nats.go"
-)
-
-func (d *Daemon) handleEC2EnableEbsEncryptionByDefault(msg *nats.Msg) {
-	handleNATSRequest(msg, d.accountService.EnableEbsEncryptionByDefault)
-}
-
-func (d *Daemon) handleEC2DisableEbsEncryptionByDefault(msg *nats.Msg) {
-	handleNATSRequest(msg, d.accountService.DisableEbsEncryptionByDefault)
-}
-
-func (d *Daemon) handleEC2GetEbsEncryptionByDefault(msg *nats.Msg) {
-	handleNATSRequest(msg, d.accountService.GetEbsEncryptionByDefault)
-}
-
-func (d *Daemon) handleEC2GetSerialConsoleAccessStatus(msg *nats.Msg) {
-	handleNATSRequest(msg, d.accountService.GetSerialConsoleAccessStatus)
-}
-
-func (d *Daemon) handleEC2EnableSerialConsoleAccess(msg *nats.Msg) {
-	handleNATSRequest(msg, d.accountService.EnableSerialConsoleAccess)
-}
-
-func (d *Daemon) handleEC2DisableSerialConsoleAccess(msg *nats.Msg) {
-	handleNATSRequest(msg, d.accountService.DisableSerialConsoleAccess)
-}

spinifex/daemon/daemon_handlers_acm.go

@@ -1,31 +0,0 @@
-package daemon
-
-import "github.com/nats-io/nats.go"
-
-func (d *Daemon) handleACMImportCertificate(msg *nats.Msg) {
-	handleNATSRequest(msg, d.acmService.ImportCertificate)
-}
-
-func (d *Daemon) handleACMDescribeCertificate(msg *nats.Msg) {
-	handleNATSRequest(msg, d.acmService.DescribeCertificate)
-}
-
-func (d *Daemon) handleACMListCertificates(msg *nats.Msg) {
-	handleNATSRequest(msg, d.acmService.ListCertificates)
-}
-
-func (d *Daemon) handleACMDeleteCertificate(msg *nats.Msg) {
-	handleNATSRequest(msg, d.acmService.DeleteCertificate)
-}
-
-func (d *Daemon) handleACMListTagsForCertificate(msg *nats.Msg) {
-	handleNATSRequest(msg, d.acmService.ListTagsForCertificate)
-}
-
-func (d *Daemon) handleACMAddTagsToCertificate(msg *nats.Msg) {
-	handleNATSRequest(msg, d.acmService.AddTagsToCertificate)
-}
-
-func (d *Daemon) handleACMRemoveTagsFromCertificate(msg *nats.Msg) {
-	handleNATSRequest(msg, d.acmService.RemoveTagsFromCertificate)
-}

spinifex/daemon/daemon_handlers_capacityreservation.go

@@ -58,7 +58,7 @@ func (d *Daemon) handleEC2CreateCapacityReservation(msg *nats.Msg) {
 	}
 
 	if err := d.resourceMgr.CreateReservation(rec); err != nil {
-		respondWithError(msg, awserrors.ValidErrorCode(err.Error()))
+		respondWithError(msg, awserrors.ValidErrorCodeFromError(err))
 		return
 	}
 

spinifex/daemon/daemon_handlers_ecr.go

@@ -1,91 +0,0 @@
-package daemon
-
-import "github.com/nats-io/nats.go"
-
-func (d *Daemon) handleECRRepoCreate(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.RepoCreate)
-}
-
-func (d *Daemon) handleECRRepoDescribe(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.RepoDescribe)
-}
-
-func (d *Daemon) handleECRRepoList(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.RepoList)
-}
-
-func (d *Daemon) handleECRRepoDelete(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.RepoDelete)
-}
-
-func (d *Daemon) handleECRPolicyPut(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.PolicyPut)
-}
-
-func (d *Daemon) handleECRPolicyGet(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.PolicyGet)
-}
-
-func (d *Daemon) handleECRPolicyDelete(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.PolicyDelete)
-}
-
-func (d *Daemon) handleECRLifecyclePut(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.LifecyclePut)
-}
-
-func (d *Daemon) handleECRLifecycleGet(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.LifecycleGet)
-}
-
-func (d *Daemon) handleECRLifecycleDelete(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.LifecycleDelete)
-}
-
-func (d *Daemon) handleECRTagPut(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.TagPut)
-}
-
-func (d *Daemon) handleECRTagGet(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.TagGet)
-}
-
-func (d *Daemon) handleECRTagList(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.TagList)
-}
-
-func (d *Daemon) handleECRTagDelete(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.TagDelete)
-}
-
-func (d *Daemon) handleECRManifestPut(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.ManifestPut)
-}
-
-func (d *Daemon) handleECRManifestDescribe(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.ManifestDescribe)
-}
-
-func (d *Daemon) handleECRManifestList(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.ManifestList)
-}
-
-func (d *Daemon) handleECRManifestDelete(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.ManifestDelete)
-}
-
-func (d *Daemon) handleECRUploadCreate(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.UploadCreate)
-}
-
-func (d *Daemon) handleECRUploadGet(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.UploadGet)
-}
-
-func (d *Daemon) handleECRUploadUpdate(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.UploadUpdate)
-}
-
-func (d *Daemon) handleECRUploadDelete(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecrMetaService.UploadDelete)
-}

spinifex/daemon/daemon_handlers_ecs.go

@@ -1,155 +0,0 @@
-package daemon
-
-import "github.com/nats-io/nats.go"
-
-// --- Cluster ---
-
-func (d *Daemon) handleECSCreateCluster(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.CreateCluster)
-}
-
-func (d *Daemon) handleECSDeleteCluster(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.DeleteCluster)
-}
-
-func (d *Daemon) handleECSDescribeClusters(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.DescribeClusters)
-}
-
-func (d *Daemon) handleECSListClusters(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.ListClusters)
-}
-
-// --- Task definition ---
-
-func (d *Daemon) handleECSRegisterTaskDefinition(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.RegisterTaskDefinition)
-}
-
-func (d *Daemon) handleECSDeregisterTaskDefinition(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.DeregisterTaskDefinition)
-}
-
-func (d *Daemon) handleECSDescribeTaskDefinition(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.DescribeTaskDefinition)
-}
-
-func (d *Daemon) handleECSListTaskDefinitions(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.ListTaskDefinitions)
-}
-
-// --- Container instance ---
-
-func (d *Daemon) handleECSRegisterContainerInstance(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.RegisterContainerInstance)
-}
-
-func (d *Daemon) handleECSDeregisterContainerInstance(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.DeregisterContainerInstance)
-}
-
-func (d *Daemon) handleECSUpdateContainerInstancesState(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.UpdateContainerInstancesState)
-}
-
-func (d *Daemon) handleECSDescribeContainerInstances(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.DescribeContainerInstances)
-}
-
-func (d *Daemon) handleECSListContainerInstances(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.ListContainerInstances)
-}
-
-// --- Task ---
-
-func (d *Daemon) handleECSRunTask(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.RunTask)
-}
-
-func (d *Daemon) handleECSStartTask(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.StartTask)
-}
-
-func (d *Daemon) handleECSStopTask(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.StopTask)
-}
-
-func (d *Daemon) handleECSDescribeTasks(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.DescribeTasks)
-}
-
-// --- Service ---
-
-func (d *Daemon) handleECSCreateService(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.CreateService)
-}
-
-func (d *Daemon) handleECSUpdateService(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.UpdateService)
-}
-
-func (d *Daemon) handleECSDeleteService(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.DeleteService)
-}
-
-func (d *Daemon) handleECSDescribeServices(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.DescribeServices)
-}
-
-func (d *Daemon) handleECSListServices(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.ListServices)
-}
-
-func (d *Daemon) handleECSSubmitTaskStateChange(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.SubmitTaskStateChange)
-}
-
-func (d *Daemon) handleECSListTasks(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.ListTasks)
-}
-
-func (d *Daemon) handleECSPollAssignments(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.PollAssignments)
-}
-
-func (d *Daemon) handleECSReportTaskGPU(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.ReportTaskGPU)
-}
-
-// --- Capacity ---
-
-func (d *Daemon) handleECSProvisionCapacity(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.ProvisionCapacity)
-}
-
-// --- Tags ---
-
-func (d *Daemon) handleECSTagResource(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.TagResource)
-}
-
-func (d *Daemon) handleECSUntagResource(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.UntagResource)
-}
-
-func (d *Daemon) handleECSListTagsForResource(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.ListTagsForResource)
-}
-
-// --- Capacity providers ---
-
-func (d *Daemon) handleECSPutClusterCapacityProviders(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.PutClusterCapacityProviders)
-}
-
-func (d *Daemon) handleECSCreateCapacityProvider(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.CreateCapacityProvider)
-}
-
-func (d *Daemon) handleECSDescribeCapacityProviders(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.DescribeCapacityProviders)
-}
-
-func (d *Daemon) handleECSDeleteCapacityProvider(msg *nats.Msg) {
-	handleNATSRequest(msg, d.ecsService.DeleteCapacityProvider)
-}

spinifex/daemon/daemon_handlers_eip.go

@@ -1,27 +0,0 @@
-package daemon
-
-import "github.com/nats-io/nats.go"
-
-func (d *Daemon) handleEC2AllocateAddress(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eipService.AllocateAddress)
-}
-
-func (d *Daemon) handleEC2ReleaseAddress(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eipService.ReleaseAddress)
-}
-
-func (d *Daemon) handleEC2AssociateAddress(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eipService.AssociateAddress)
-}
-
-func (d *Daemon) handleEC2DisassociateAddress(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eipService.DisassociateAddress)
-}
-
-func (d *Daemon) handleEC2DescribeAddresses(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eipService.DescribeAddresses)
-}
-
-func (d *Daemon) handleEC2DescribeAddressesAttribute(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eipService.DescribeAddressesAttribute)
-}

spinifex/daemon/daemon_handlers_eks.go

@@ -1,167 +0,0 @@
-package daemon
-
-import "github.com/nats-io/nats.go"
-
-// --- Cluster ---
-
-func (d *Daemon) handleEKSCreateCluster(msg *nats.Msg) {
-	handleNATSRequestWithPrincipal(msg, d.eksService.CreateCluster)
-}
-
-func (d *Daemon) handleEKSDescribeCluster(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.DescribeCluster)
-}
-
-func (d *Daemon) handleEKSListClusters(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.ListClusters)
-}
-
-func (d *Daemon) handleEKSUpdateClusterConfig(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.UpdateClusterConfig)
-}
-
-func (d *Daemon) handleEKSUpdateClusterVersion(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.UpdateClusterVersion)
-}
-
-func (d *Daemon) handleEKSDeleteCluster(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.DeleteCluster)
-}
-
-// --- Nodegroup ---
-
-func (d *Daemon) handleEKSCreateNodegroup(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.CreateNodegroup)
-}
-
-func (d *Daemon) handleEKSDescribeNodegroup(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.DescribeNodegroup)
-}
-
-func (d *Daemon) handleEKSListNodegroups(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.ListNodegroups)
-}
-
-func (d *Daemon) handleEKSUpdateNodegroupConfig(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.UpdateNodegroupConfig)
-}
-
-func (d *Daemon) handleEKSUpdateNodegroupVersion(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.UpdateNodegroupVersion)
-}
-
-func (d *Daemon) handleEKSDeleteNodegroup(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.DeleteNodegroup)
-}
-
-// --- AccessEntry + AccessPolicy ---
-
-func (d *Daemon) handleEKSCreateAccessEntry(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.CreateAccessEntry)
-}
-
-func (d *Daemon) handleEKSDescribeAccessEntry(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.DescribeAccessEntry)
-}
-
-func (d *Daemon) handleEKSListAccessEntries(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.ListAccessEntries)
-}
-
-func (d *Daemon) handleEKSUpdateAccessEntry(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.UpdateAccessEntry)
-}
-
-func (d *Daemon) handleEKSDeleteAccessEntry(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.DeleteAccessEntry)
-}
-
-func (d *Daemon) handleEKSAssociateAccessPolicy(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.AssociateAccessPolicy)
-}
-
-func (d *Daemon) handleEKSDisassociateAccessPolicy(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.DisassociateAccessPolicy)
-}
-
-func (d *Daemon) handleEKSListAssociatedAccessPolicies(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.ListAssociatedAccessPolicies)
-}
-
-func (d *Daemon) handleEKSListAccessPolicies(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.ListAccessPolicies)
-}
-
-// --- Addons ---
-
-func (d *Daemon) handleEKSListAddons(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.ListAddons)
-}
-
-func (d *Daemon) handleEKSDescribeAddonVersions(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.DescribeAddonVersions)
-}
-
-func (d *Daemon) handleEKSCreateAddon(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.CreateAddon)
-}
-
-func (d *Daemon) handleEKSDeleteAddon(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.DeleteAddon)
-}
-
-func (d *Daemon) handleEKSDescribeAddon(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.DescribeAddon)
-}
-
-func (d *Daemon) handleEKSUpdateAddon(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.UpdateAddon)
-}
-
-func (d *Daemon) handleEKSListStagedAddonManifests(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.ListStagedAddonManifests)
-}
-
-func (d *Daemon) handleEKSGetRecoveryDirective(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.GetRecoveryDirective)
-}
-
-func (d *Daemon) handleEKSSetRecoveryDirective(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.SetRecoveryDirective)
-}
-
-func (d *Daemon) handleEKSRestoreSnapshot(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.RestoreSnapshot)
-}
-
-// --- OIDC identity-provider configs ---
-
-func (d *Daemon) handleEKSAssociateIdentityProviderConfig(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.AssociateIdentityProviderConfig)
-}
-
-func (d *Daemon) handleEKSDescribeIdentityProviderConfig(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.DescribeIdentityProviderConfig)
-}
-
-func (d *Daemon) handleEKSListIdentityProviderConfigs(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.ListIdentityProviderConfigs)
-}
-
-func (d *Daemon) handleEKSDisassociateIdentityProviderConfig(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.DisassociateIdentityProviderConfig)
-}
-
-// --- Tags ---
-
-func (d *Daemon) handleEKSTagResource(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.TagResource)
-}
-
-func (d *Daemon) handleEKSUntagResource(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.UntagResource)
-}
-
-func (d *Daemon) handleEKSListTagsForResource(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eksService.ListTagsForResource)
-}

spinifex/daemon/daemon_handlers_eks_test.go

@@ -71,41 +71,41 @@ func TestDaemonHandleEKS_AllHandlersDispatchToService(t *testing.T) {
 		handler  nats.MsgHandler
 		wantCode string
 	}{
-		{"eks.UpdateClusterConfig", d.handleEKSUpdateClusterConfig, notImpl},
-		{"eks.UpdateClusterVersion", d.handleEKSUpdateClusterVersion, notImpl},
-		{"eks.CreateNodegroup", d.handleEKSCreateNodegroup, unavailable},
-		{"eks.DescribeNodegroup", d.handleEKSDescribeNodegroup, invalid},
-		{"eks.ListNodegroups", d.handleEKSListNodegroups, invalid},
-		{"eks.UpdateNodegroupConfig", d.handleEKSUpdateNodegroupConfig, unavailable},
-		{"eks.UpdateNodegroupVersion", d.handleEKSUpdateNodegroupVersion, notImpl},
-		{"eks.DeleteNodegroup", d.handleEKSDeleteNodegroup, unavailable},
-		{"eks.CreateAccessEntry", d.handleEKSCreateAccessEntry, invalid},
-		{"eks.DescribeAccessEntry", d.handleEKSDescribeAccessEntry, invalid},
-		{"eks.ListAccessEntries", d.handleEKSListAccessEntries, invalid},
-		{"eks.UpdateAccessEntry", d.handleEKSUpdateAccessEntry, invalid},
-		{"eks.DeleteAccessEntry", d.handleEKSDeleteAccessEntry, invalid},
-		{"eks.AssociateAccessPolicy", d.handleEKSAssociateAccessPolicy, invalid},
-		{"eks.DisassociateAccessPolicy", d.handleEKSDisassociateAccessPolicy, invalid},
-		{"eks.ListAssociatedAccessPolicies", d.handleEKSListAssociatedAccessPolicies, invalid},
-		{"eks.ListAccessPolicies", d.handleEKSListAccessPolicies, ""},
+		{"eks.UpdateClusterConfig", handleNATSRequest(d.eksService.UpdateClusterConfig), notImpl},
+		{"eks.UpdateClusterVersion", handleNATSRequest(d.eksService.UpdateClusterVersion), notImpl},
+		{"eks.CreateNodegroup", handleNATSRequest(d.eksService.CreateNodegroup), unavailable},
+		{"eks.DescribeNodegroup", handleNATSRequest(d.eksService.DescribeNodegroup), invalid},
+		{"eks.ListNodegroups", handleNATSRequest(d.eksService.ListNodegroups), invalid},
+		{"eks.UpdateNodegroupConfig", handleNATSRequest(d.eksService.UpdateNodegroupConfig), unavailable},
+		{"eks.UpdateNodegroupVersion", handleNATSRequest(d.eksService.UpdateNodegroupVersion), notImpl},
+		{"eks.DeleteNodegroup", handleNATSRequest(d.eksService.DeleteNodegroup), unavailable},
+		{"eks.CreateAccessEntry", handleNATSRequest(d.eksService.CreateAccessEntry), invalid},
+		{"eks.DescribeAccessEntry", handleNATSRequest(d.eksService.DescribeAccessEntry), invalid},
+		{"eks.ListAccessEntries", handleNATSRequest(d.eksService.ListAccessEntries), invalid},
+		{"eks.UpdateAccessEntry", handleNATSRequest(d.eksService.UpdateAccessEntry), invalid},
+		{"eks.DeleteAccessEntry", handleNATSRequest(d.eksService.DeleteAccessEntry), invalid},
+		{"eks.AssociateAccessPolicy", handleNATSRequest(d.eksService.AssociateAccessPolicy), invalid},
+		{"eks.DisassociateAccessPolicy", handleNATSRequest(d.eksService.DisassociateAccessPolicy), invalid},
+		{"eks.ListAssociatedAccessPolicies", handleNATSRequest(d.eksService.ListAssociatedAccessPolicies), invalid},
+		{"eks.ListAccessPolicies", handleNATSRequest(d.eksService.ListAccessPolicies), ""},
 		// Addon handlers are implemented (Sprint 6 P1): an empty body fails
 		// cluster validation with InvalidParameterValue, except
 		// DescribeAddonVersions which returns the static catalogue (success).
-		{"eks.ListAddons", d.handleEKSListAddons, invalid},
-		{"eks.DescribeAddonVersions", d.handleEKSDescribeAddonVersions, ""},
-		{"eks.CreateAddon", d.handleEKSCreateAddon, invalid},
-		{"eks.DeleteAddon", d.handleEKSDeleteAddon, invalid},
-		{"eks.DescribeAddon", d.handleEKSDescribeAddon, invalid},
-		{"eks.UpdateAddon", d.handleEKSUpdateAddon, invalid},
-		{"eks.AssociateIdentityProviderConfig", d.handleEKSAssociateIdentityProviderConfig, notImpl},
-		{"eks.DescribeIdentityProviderConfig", d.handleEKSDescribeIdentityProviderConfig, notImpl},
-		{"eks.ListIdentityProviderConfigs", d.handleEKSListIdentityProviderConfigs, notImpl},
-		{"eks.DisassociateIdentityProviderConfig", d.handleEKSDisassociateIdentityProviderConfig, notImpl},
-		{"eks.TagResource", d.handleEKSTagResource, notImpl},
-		{"eks.UntagResource", d.handleEKSUntagResource, notImpl},
-		{"eks.ListTagsForResource", d.handleEKSListTagsForResource, notImpl},
+		{"eks.ListAddons", handleNATSRequest(d.eksService.ListAddons), invalid},
+		{"eks.DescribeAddonVersions", handleNATSRequest(d.eksService.DescribeAddonVersions), ""},
+		{"eks.CreateAddon", handleNATSRequest(d.eksService.CreateAddon), invalid},
+		{"eks.DeleteAddon", handleNATSRequest(d.eksService.DeleteAddon), invalid},
+		{"eks.DescribeAddon", handleNATSRequest(d.eksService.DescribeAddon), invalid},
+		{"eks.UpdateAddon", handleNATSRequest(d.eksService.UpdateAddon), invalid},
+		{"eks.AssociateIdentityProviderConfig", handleNATSRequest(d.eksService.AssociateIdentityProviderConfig), notImpl},
+		{"eks.DescribeIdentityProviderConfig", handleNATSRequest(d.eksService.DescribeIdentityProviderConfig), notImpl},
+		{"eks.ListIdentityProviderConfigs", handleNATSRequest(d.eksService.ListIdentityProviderConfigs), notImpl},
+		{"eks.DisassociateIdentityProviderConfig", handleNATSRequest(d.eksService.DisassociateIdentityProviderConfig), notImpl},
+		{"eks.TagResource", handleNATSRequest(d.eksService.TagResource), notImpl},
+		{"eks.UntagResource", handleNATSRequest(d.eksService.UntagResource), notImpl},
+		{"eks.ListTagsForResource", handleNATSRequest(d.eksService.ListTagsForResource), notImpl},
 	}
-	require.Equal(t, 30, len(cases), "expected exactly one handler per non-lifecycle AWS EKS action")
+	require.Len(t, cases, 30, "expected exactly one handler per non-lifecycle AWS EKS action")
 
 	for _, c := range cases {
 		t.Run(c.subject, func(t *testing.T) {

spinifex/daemon/daemon_handlers_elbv2.go

@@ -1,143 +0,0 @@
-package daemon
-
-import "github.com/nats-io/nats.go"
-
-func (d *Daemon) handleELBv2CreateLoadBalancer(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.CreateLoadBalancer)
-}
-
-func (d *Daemon) handleELBv2DeleteLoadBalancer(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.DeleteLoadBalancer)
-}
-
-func (d *Daemon) handleELBv2DescribeLoadBalancers(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.DescribeLoadBalancers)
-}
-
-func (d *Daemon) handleELBv2CreateTargetGroup(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.CreateTargetGroup)
-}
-
-func (d *Daemon) handleELBv2ModifyTargetGroup(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.ModifyTargetGroup)
-}
-
-func (d *Daemon) handleELBv2DeleteTargetGroup(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.DeleteTargetGroup)
-}
-
-func (d *Daemon) handleELBv2DescribeTargetGroups(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.DescribeTargetGroups)
-}
-
-func (d *Daemon) handleELBv2RegisterTargets(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.RegisterTargets)
-}
-
-func (d *Daemon) handleELBv2DeregisterTargets(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.DeregisterTargets)
-}
-
-func (d *Daemon) handleELBv2DescribeTargetHealth(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.DescribeTargetHealth)
-}
-
-func (d *Daemon) handleELBv2CreateListener(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.CreateListener)
-}
-
-func (d *Daemon) handleELBv2DeleteListener(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.DeleteListener)
-}
-
-func (d *Daemon) handleELBv2ModifyListener(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.ModifyListener)
-}
-
-func (d *Daemon) handleELBv2DescribeListeners(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.DescribeListeners)
-}
-
-func (d *Daemon) handleELBv2CreateRule(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.CreateRule)
-}
-
-func (d *Daemon) handleELBv2ModifyRule(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.ModifyRule)
-}
-
-func (d *Daemon) handleELBv2DeleteRule(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.DeleteRule)
-}
-
-func (d *Daemon) handleELBv2DescribeRules(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.DescribeRules)
-}
-
-func (d *Daemon) handleELBv2SetRulePriorities(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.SetRulePriorities)
-}
-
-func (d *Daemon) handleELBv2DescribeTags(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.DescribeTags)
-}
-
-func (d *Daemon) handleELBv2AddTags(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.AddTags)
-}
-
-func (d *Daemon) handleELBv2RemoveTags(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.RemoveTags)
-}
-
-func (d *Daemon) handleELBv2LBAgentHeartbeat(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.LBAgentHeartbeat)
-}
-
-func (d *Daemon) handleELBv2GetLBConfig(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.GetLBConfig)
-}
-
-func (d *Daemon) handleELBv2ModifyTargetGroupAttributes(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.ModifyTargetGroupAttributes)
-}
-
-func (d *Daemon) handleELBv2DescribeTargetGroupAttributes(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.DescribeTargetGroupAttributes)
-}
-
-func (d *Daemon) handleELBv2ModifyLoadBalancerAttributes(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.ModifyLoadBalancerAttributes)
-}
-
-func (d *Daemon) handleELBv2DescribeLoadBalancerAttributes(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.DescribeLoadBalancerAttributes)
-}
-
-func (d *Daemon) handleELBv2SetSecurityGroups(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.SetSecurityGroups)
-}
-
-func (d *Daemon) handleELBv2SetIpAddressType(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.SetIpAddressType)
-}
-
-func (d *Daemon) handleELBv2SetSubnets(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.SetSubnets)
-}
-
-func (d *Daemon) handleELBv2AddListenerCertificates(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.AddListenerCertificates)
-}
-
-func (d *Daemon) handleELBv2RemoveListenerCertificates(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.RemoveListenerCertificates)
-}
-
-func (d *Daemon) handleELBv2DescribeListenerCertificates(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.DescribeListenerCertificates)
-}
-
-func (d *Daemon) handleELBv2DescribeSSLPolicies(msg *nats.Msg) {
-	handleNATSRequest(msg, d.elbv2Service.DescribeSSLPolicies)
-}

spinifex/daemon/daemon_handlers_eni.go

@@ -39,7 +39,7 @@ func (d *Daemon) handleAttachNetworkInterface(ctx context.Context, msg *nats.Msg
 
 	record, err := d.vpcService.GetENIRecord(accountID, eniID)
 	if err != nil {
-		respondWithError(msg, errorCodeFor(err))
+		respondWithError(msg, awserrors.ValidErrorCodeFromError(err))
 		return
 	}
 	if record.Status == "in-use" && record.InstanceId != instance.ID {
@@ -49,7 +49,7 @@ func (d *Daemon) handleAttachNetworkInterface(ctx context.Context, msg *nats.Msg
 
 	attachmentID, err := d.vpcService.AttachENI(accountID, eniID, instance.ID, deviceIndex)
 	if err != nil {
-		respondWithError(msg, errorCodeFor(err))
+		respondWithError(msg, awserrors.ValidErrorCodeFromError(err))
 		return
 	}
 	if err := d.vpcService.UpdateENI(accountID, eniID, func(r *handlers_ec2_vpc.ENIRecord) {
@@ -117,7 +117,7 @@ func (d *Daemon) handleDetachNetworkInterface(ctx context.Context, msg *nats.Msg
 
 	record, err := d.vpcService.FindENIByAttachment(accountID, attachmentID)
 	if err != nil {
-		respondWithError(msg, errorCodeFor(err))
+		respondWithError(msg, awserrors.ValidErrorCodeFromError(err))
 		return
 	}
 	if record.InstanceId != instance.ID {
@@ -191,15 +191,6 @@ func eniHotplugErrorCode(err error) string {
 	}
 }
 
-// errorCodeFor passes through errors whose message is already a valid AWS
-// code (the convention VPCServiceImpl uses for its sentinel errors).
-func errorCodeFor(err error) string {
-	if err == nil {
-		return ""
-	}
-	return awserrors.ValidErrorCode(err.Error())
-}
-
 // publishENIHotplugEvent emits the per-instance NATS event on best-effort
 // basis — observers (Phase 4 ecs-agent, future vpcd subscribers) read it
 // for fanout. A publish failure does not roll back the pipeline.

spinifex/daemon/daemon_handlers_iam_profile.go

@@ -15,29 +15,8 @@ import (
 func (d *Daemon) handleAssociateIamInstanceProfile(ctx context.Context, msg *nats.Msg, command types.EC2InstanceCommand, instance *vm.VM) {
 	result, err := d.instanceService.AssociateIamInstanceProfile(ctx, instance, command)
 	if err != nil {
-		respondWithError(msg, awserrors.ValidErrorCode(err.Error()))
+		respondWithError(msg, awserrors.ValidErrorCodeFromError(err))
 		return
 	}
 	respondWithJSON(msg, result)
 }
-
-// handleIamProfileDisassociate services the disassociate fan-out. The owning
-// daemon responds with the mutated association; non-owners respond with JSON
-// null so the gateway's expectedNodes collector can exit early.
-func (d *Daemon) handleIamProfileDisassociate(msg *nats.Msg) {
-	handleNATSRequest(msg, d.instanceService.DisassociateIamProfileAssociation)
-}
-
-// handleIamProfileReplace services the replace fan-out. Same contract as
-// handleIamProfileDisassociate: every daemon responds (JSON null on
-// non-owners) so the gateway collector can exit early.
-func (d *Daemon) handleIamProfileReplace(msg *nats.Msg) {
-	handleNATSRequest(msg, d.instanceService.ReplaceIamProfileAssociation)
-}
-
-// handleIamProfileDescribe services the describe fan-out. An empty slice is a
-// valid response (no matches on this daemon) and counts toward the gateway's
-// expectedNodes collector; the gateway concatenates per-daemon slices.
-func (d *Daemon) handleIamProfileDescribe(msg *nats.Msg) {
-	handleNATSRequest(msg, d.instanceService.DescribeIamProfileAssociations)
-}

spinifex/daemon/daemon_handlers_igw.go

@@ -1,37 +0,0 @@
-package daemon
-
-import (
-	"github.com/nats-io/nats.go"
-)
-
-func (d *Daemon) handleEC2CreateInternetGateway(msg *nats.Msg) {
-	handleNATSRequest(msg, d.igwService.CreateInternetGateway)
-}
-
-func (d *Daemon) handleEC2DeleteInternetGateway(msg *nats.Msg) {
-	handleNATSRequest(msg, d.igwService.DeleteInternetGateway)
-}
-
-func (d *Daemon) handleEC2DescribeInternetGateways(msg *nats.Msg) {
-	handleNATSRequest(msg, d.igwService.DescribeInternetGateways)
-}
-
-func (d *Daemon) handleEC2AttachInternetGateway(msg *nats.Msg) {
-	handleNATSRequest(msg, d.igwService.AttachInternetGateway)
-}
-
-func (d *Daemon) handleEC2DetachInternetGateway(msg *nats.Msg) {
-	handleNATSRequest(msg, d.igwService.DetachInternetGateway)
-}
-
-func (d *Daemon) handleEC2CreateEgressOnlyInternetGateway(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eigwService.CreateEgressOnlyInternetGateway)
-}
-
-func (d *Daemon) handleEC2DeleteEgressOnlyInternetGateway(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eigwService.DeleteEgressOnlyInternetGateway)
-}
-
-func (d *Daemon) handleEC2DescribeEgressOnlyInternetGateways(msg *nats.Msg) {
-	handleNATSRequest(msg, d.eigwService.DescribeEgressOnlyInternetGateways)
-}

spinifex/daemon/daemon_handlers_image.go

@@ -14,34 +14,6 @@ import (
 	"github.com/nats-io/nats.go"
 )
 
-func (d *Daemon) handleEC2DescribeImages(msg *nats.Msg) {
-	handleNATSRequest(msg, d.imageService.DescribeImages)
-}
-
-func (d *Daemon) handleEC2DeregisterImage(msg *nats.Msg) {
-	handleNATSRequest(msg, d.imageService.DeregisterImage)
-}
-
-func (d *Daemon) handleEC2RegisterImage(msg *nats.Msg) {
-	handleNATSRequest(msg, d.imageService.RegisterImage)
-}
-
-func (d *Daemon) handleEC2CopyImage(msg *nats.Msg) {
-	handleNATSRequest(msg, d.imageService.CopyImage)
-}
-
-func (d *Daemon) handleEC2DescribeImageAttribute(msg *nats.Msg) {
-	handleNATSRequest(msg, d.imageService.DescribeImageAttribute)
-}
-
-func (d *Daemon) handleEC2ModifyImageAttribute(msg *nats.Msg) {
-	handleNATSRequest(msg, d.imageService.ModifyImageAttribute)
-}
-
-func (d *Daemon) handleEC2ResetImageAttribute(msg *nats.Msg) {
-	handleNATSRequest(msg, d.imageService.ResetImageAttribute)
-}
-
 func (d *Daemon) handleSpinifexPromoteImage(msg *nats.Msg) {
 	promoteImage := func(_ context.Context, input *admin.PromoteImageOpts, _ string) (*admin.PromoteImageResult, error) {
 		store := objectstore.NewS3ObjectStoreFromConfig(
@@ -52,7 +24,7 @@ func (d *Daemon) handleSpinifexPromoteImage(msg *nats.Msg) {
 		)
 		return admin.PromoteSystemImage(store, d.config.Predastore.Bucket, *input)
 	}
-	handleNATSRequest(msg, promoteImage)
+	handleNATSRequest(promoteImage)(msg)
 }
 
 // handleEC2CreateImage is a stateful handler that extracts instance context
@@ -158,7 +130,7 @@ func (d *Daemon) handleEC2CreateImage(msg *nats.Msg) {
 	output, err := d.imageService.CreateImageFromInstance(params, accountID)
 	if err != nil {
 		slog.Error("CreateImage: service failed", "instanceId", instanceID, "err", err)
-		respondWithError(msg, awserrors.ValidErrorCode(err.Error()))
+		respondWithError(msg, awserrors.ValidErrorCodeFromError(err))
 		return
 	}
 

spinifex/daemon/daemon_handlers_instance.go

@@ -6,13 +6,11 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
-	"strings"
 	"time"
 
 	"github.com/aws/aws-sdk-go/aws"
 	"github.com/aws/aws-sdk-go/service/ec2"
 	"github.com/mulgadc/spinifex/spinifex/awserrors"
-	"github.com/mulgadc/spinifex/spinifex/filterutil"
 	handlers_ec2_instance "github.com/mulgadc/spinifex/spinifex/handlers/ec2/instance"
 	"github.com/mulgadc/spinifex/spinifex/types"
 	"github.com/mulgadc/spinifex/spinifex/utils"
@@ -71,7 +69,7 @@ func (d *Daemon) handleSetInstanceTags(ctx context.Context, msg *nats.Msg, comma
 		return true
 	})
 	if err != nil {
-		respondWithError(msg, awserrors.ValidErrorCode(err.Error()))
+		respondWithError(msg, awserrors.ValidErrorCodeFromError(err))
 		return
 	}
 	if !found {
@@ -119,7 +117,7 @@ func (d *Daemon) handleEC2RunInstances(msg *nats.Msg) {
 	if reservationID != "" {
 		if it, ok := d.resourceMgr.instanceTypes[aws.StringValue(input.InstanceType)]; ok {
 			if vErr := d.resourceMgr.ValidateReservationTarget(reservationID, accountID, it); vErr != nil {
-				respondWithError(msg, awserrors.ValidErrorCode(vErr.Error()))
+				respondWithError(msg, awserrors.ValidErrorCodeFromError(vErr))
 				return
 			}
 		}
@@ -129,7 +127,7 @@ func (d *Daemon) handleEC2RunInstances(msg *nats.Msg) {
 	reservation, instances, instanceType, err := d.instanceService.PrepareRunInstances(ctx, input, accountID, reservationID)
 	endOpSpan(prepSpan, err)
 	if err != nil {
-		respondWithError(msg, awserrors.ValidErrorCode(err.Error()))
+		respondWithError(msg, awserrors.ValidErrorCodeFromError(err))
 		return
 	}
 
@@ -208,271 +206,6 @@ func capacityReservationTargetID(input *ec2.RunInstancesInput) string {
 	return aws.StringValue(spec.CapacityReservationTarget.CapacityReservationId)
 }
 
-// describeInstancesValidFilters defines the set of filter names accepted by DescribeInstances.
-var describeInstancesValidFilters = map[string]bool{
-	"instance-state-name": true,
-	"instance-id":         true,
-	"instance-type":       true,
-	"vpc-id":              true,
-	"subnet-id":           true,
-	"tag-key":             true,
-	"tag-value":           true,
-}
-
-// instanceMatchesFilters checks whether a VM + its built ec2.Instance copy satisfy all parsed filters.
-func instanceMatchesFilters(inst *vm.VM, ic *ec2.Instance, filters map[string][]string) bool {
-	for name, values := range filters {
-		if strings.HasPrefix(name, "tag:") {
-			// tag:Key filters are handled after all field filters.
-			continue
-		}
-
-		var field string
-		switch name {
-		case "instance-state-name":
-			if ic.State != nil && ic.State.Name != nil {
-				field = *ic.State.Name
-			}
-		case "instance-id":
-			field = inst.ID
-		case "instance-type":
-			field = inst.InstanceType
-		case "vpc-id":
-			if ic.VpcId != nil {
-				field = *ic.VpcId
-			}
-		case "subnet-id":
-			if ic.SubnetId != nil {
-				field = *ic.SubnetId
-			}
-		case "tag-key":
-			if !matchTagKey(ic.Tags, values) {
-				return false
-			}
-			continue
-		case "tag-value":
-			if !matchTagValue(ic.Tags, values) {
-				return false
-			}
-			continue
-		default:
-			// Filter name passed ParseFilters but has no case — treat as non-match
-			// to avoid silently ignoring it.
-			return false
-		}
-
-		if !filterutil.MatchesAny(values, field) {
-			return false
-		}
-	}
-
-	// Check tag:Key filters via the instance's Tag slice.
-	tags := filterutil.EC2TagsToMap(ic.Tags)
-	return filterutil.MatchesTags(filters, tags)
-}
-
-// matchTagKey returns true if any tag key on the resource matches any of the filter values.
-func matchTagKey(tags []*ec2.Tag, values []string) bool {
-	for _, t := range tags {
-		if t.Key != nil && filterutil.MatchesAny(values, *t.Key) {
-			return true
-		}
-	}
-	return false
-}
-
-// matchTagValue returns true if any tag value on the resource matches any of the filter values.
-func matchTagValue(tags []*ec2.Tag, values []string) bool {
-	for _, t := range tags {
-		if t.Value != nil && filterutil.MatchesAny(values, *t.Value) {
-			return true
-		}
-	}
-	return false
-}
-
-// handleEC2DescribeInstances responds with instances running on this node visible to the caller.
-func (d *Daemon) handleEC2DescribeInstances(msg *nats.Msg) {
-	slog.Debug("Received message", "subject", msg.Subject, "data", string(msg.Data))
-
-	// Extract account ID from NATS header for scoping
-	accountID := utils.AccountIDFromMsg(msg)
-
-	// Initialize describeInstancesInput before unmarshaling into it
-	describeInstancesInput := &ec2.DescribeInstancesInput{}
-	errResp := utils.UnmarshalJsonPayload(describeInstancesInput, msg.Data)
-
-	if errResp != nil {
-		if err := msg.Respond(errResp); err != nil {
-			slog.Error("Failed to respond to NATS request", "err", err)
-		}
-		slog.Error("Request does not match DescribeInstancesInput")
-		return
-	}
-
-	slog.Info("Processing DescribeInstances request from this node", "accountID", accountID)
-
-	// Validate and filter instances if specific instance IDs were requested
-	instanceIDFilter := make(map[string]bool)
-	if len(describeInstancesInput.InstanceIds) > 0 {
-		for _, id := range describeInstancesInput.InstanceIds {
-			if id != nil && *id != "" {
-				if !strings.HasPrefix(*id, "i-") {
-					respondWithError(msg, awserrors.ErrorInvalidInstanceIDMalformed)
-					return
-				}
-				instanceIDFilter[*id] = true
-			}
-		}
-	}
-
-	// Parse filters (returns error for unknown filter names)
-	parsedFilters, err := filterutil.ParseFilters(describeInstancesInput.Filters, describeInstancesValidFilters)
-	if err != nil {
-		slog.Warn("DescribeInstances: invalid filter", "err", err)
-		respondWithError(msg, awserrors.ErrorInvalidParameterValue)
-		return
-	}
-
-	// Group instances by reservation ID (AWS returns instances grouped by reservation)
-	reservationMap := make(map[string]*ec2.Reservation)
-
-	// Iterate under the manager lock — VM fields (Status, Instance, Reservation,
-	// PublicIP, PlacementGroupName) are mutated through manager-locked
-	// Inspect/UpdateState elsewhere, so reading them lock-free would race.
-	d.vmMgr.View(func(vms map[string]*vm.VM) {
-		for _, instance := range vms {
-			// Skip instances not owned by the caller's account.
-			// Instances with an empty AccountID (legacy/migration data)
-			// are only visible to root.
-			if !isInstanceVisible(accountID, instance.AccountID) {
-				continue
-			}
-
-			// Skip if filtering by instance IDs and this instance is not in the filter
-			if len(instanceIDFilter) > 0 && !instanceIDFilter[instance.ID] {
-				continue
-			}
-
-			// Use stored reservation metadata if available
-			if instance.Reservation != nil && instance.Instance != nil {
-				resID := ""
-				if instance.Reservation.ReservationId != nil {
-					resID = *instance.Reservation.ReservationId
-				}
-
-				// Create reservation entry if it doesn't exist
-				if _, exists := reservationMap[resID]; !exists {
-					reservation := &ec2.Reservation{}
-					reservation.SetReservationId(resID)
-					if instance.Reservation.OwnerId != nil {
-						reservation.SetOwnerId(*instance.Reservation.OwnerId)
-					}
-					reservation.Instances = []*ec2.Instance{}
-					reservationMap[resID] = reservation
-				}
-
-				// Update the instance state to current state
-				instanceCopy := *instance.Instance
-				instanceCopy.State = &ec2.InstanceState{}
-
-				// Populate PublicIpAddress from VM if stored
-				if instance.PublicIP != "" && instanceCopy.PublicIpAddress == nil {
-					instanceCopy.PublicIpAddress = aws.String(instance.PublicIP)
-				}
-
-				// Map internal status to AWS state, projecting Spinifex-only states
-				// (e.g. error -> stopped) so SDK/UI clients see a valid label.
-				if info, ok := vm.EC2APIState(instance.Status); ok {
-					instanceCopy.State.SetCode(info.Code)
-					instanceCopy.State.SetName(info.Name)
-				} else {
-					slog.Warn("Instance has unmapped status, reporting as pending",
-						"instanceId", instance.ID, "status", string(instance.Status))
-					instanceCopy.State.SetCode(0)
-					instanceCopy.State.SetName("pending")
-				}
-
-				// Project IamInstanceProfile from vm.VM (single source of truth
-				// across Associate/Disassociate/Replace lifecycle). Id is left
-				// nil — the gateway resolves it via IAMService post-aggregation
-				// since daemons have no IAM access. Empty Arn clears any stale
-				// reference left on stored instance.Instance (e.g. after
-				// Disassociate or auto-clear on terminate).
-				if instance.IamInstanceProfileArn != "" {
-					instanceCopy.IamInstanceProfile = &ec2.IamInstanceProfile{
-						Arn: aws.String(instance.IamInstanceProfileArn),
-					}
-				} else {
-					instanceCopy.IamInstanceProfile = nil
-				}
-
-				// Populate Placement if instance belongs to a placement group
-				if instance.PlacementGroupName != "" {
-					instanceCopy.Placement = &ec2.Placement{
-						GroupName:        aws.String(instance.PlacementGroupName),
-						AvailabilityZone: aws.String(d.config.AZ),
-					}
-				}
-
-				// Echo the consumed capacity reservation so targeted-launch
-				// Terraform converges — without it the instance reports no
-				// reservation and the plan never settles.
-				if instance.CapacityReservationId != "" {
-					instanceCopy.CapacityReservationId = aws.String(instance.CapacityReservationId)
-					instanceCopy.CapacityReservationSpecification = &ec2.CapacityReservationSpecificationResponse{
-						CapacityReservationPreference: aws.String(ec2.CapacityReservationPreferenceOpen),
-						CapacityReservationTarget: &ec2.CapacityReservationTargetResponse{
-							CapacityReservationId: aws.String(instance.CapacityReservationId),
-						},
-					}
-				}
-
-				// Project spot lineage stamped by the post-launch write-back.
-				// Both empty for on-demand, so the fields stay absent there.
-				if instance.InstanceLifecycle != "" {
-					instanceCopy.InstanceLifecycle = aws.String(instance.InstanceLifecycle)
-				}
-				if instance.SpotInstanceRequestId != "" {
-					instanceCopy.SpotInstanceRequestId = aws.String(instance.SpotInstanceRequestId)
-				}
-
-				// Apply filters against the fully-built instance copy
-				if len(parsedFilters) > 0 && !instanceMatchesFilters(instance, &instanceCopy, parsedFilters) {
-					continue
-				}
-
-				// Add instance to its reservation
-				reservationMap[resID].Instances = append(reservationMap[resID].Instances, &instanceCopy)
-			}
-		}
-	})
-
-	// Convert map to slice
-	reservations := make([]*ec2.Reservation, 0, len(reservationMap))
-	for _, reservation := range reservationMap {
-		reservations = append(reservations, reservation)
-	}
-
-	// Create the response
-	output := &ec2.DescribeInstancesOutput{
-		Reservations: reservations,
-	}
-
-	respondWithJSON(msg, output)
-	slog.Info("handleEC2DescribeInstances completed", "count", len(reservations))
-}
-
-// handleEC2DescribeInstanceTypes responds with instance types provisionable on this node.
-func (d *Daemon) handleEC2DescribeInstanceTypes(msg *nats.Msg) {
-	handleNATSRequest(msg, d.instanceService.DescribeInstanceTypes)
-}
-
-// handleEC2DescribeInstanceStatus responds with status entries for VMs on this node visible to the caller.
-func (d *Daemon) handleEC2DescribeInstanceStatus(msg *nats.Msg) {
-	handleNATSRequest(msg, d.instanceService.DescribeInstanceStatus)
-}
-
 // handleEC2StartStoppedInstance handles the generic ec2.start queue-group topic.
 // It reads the stopped instance's LastNode from shared KV and forwards the
 // request to ec2.start.{lastNode} to keep instances on their original node.
@@ -489,7 +222,7 @@ func (d *Daemon) handleEC2StartStoppedInstance(msg *nats.Msg) {
 	if err := json.Unmarshal(msg.Data, &peek); err != nil || peek.InstanceID == "" {
 		// Can't determine target node — fall through to local start which will
 		// return the appropriate error (missing parameter / unmarshal failure).
-		handleNATSRequest(msg, d.instanceService.StartStoppedInstance)
+		handleNATSRequest(d.instanceService.StartStoppedInstance)(msg)
 		return
 	}
 
@@ -535,150 +268,5 @@ func (d *Daemon) handleEC2StartStoppedInstance(msg *nats.Msg) {
 		}
 	}
 
-	handleNATSRequest(msg, d.instanceService.StartStoppedInstance)
-}
-
-// handleEC2StartStoppedInstanceDirect handles ec2.start.{nodeId} — the
-// node-targeted topic used by handleEC2StartStoppedInstance to forward start
-// requests back to the original node. Always starts locally; never forwards
-// further, preventing routing loops.
-func (d *Daemon) handleEC2StartStoppedInstanceDirect(msg *nats.Msg) {
-	handleNATSRequest(msg, d.instanceService.StartStoppedInstance)
-}
-
-func (d *Daemon) handleEC2TerminateStoppedInstance(msg *nats.Msg) {
-	handleNATSRequest(msg, d.instanceService.TerminateStoppedInstance)
-}
-
-// handleEC2DescribeStoppedInstances returns stopped instances from shared KV.
-func (d *Daemon) handleEC2DescribeStoppedInstances(msg *nats.Msg) {
-	if d.stateStore == nil {
-		respondWithError(msg, awserrors.ErrorServerInternal)
-		return
-	}
-	d.describeInstancesFromKV(msg, d.stateStore.ListStoppedInstances, 80, "stopped", "handleEC2DescribeStoppedInstances")
-}
-
-// handleEC2DescribeTerminatedInstances returns terminated instances from the terminated KV bucket.
-func (d *Daemon) handleEC2DescribeTerminatedInstances(msg *nats.Msg) {
-	if d.stateStore == nil {
-		respondWithError(msg, awserrors.ErrorServerInternal)
-		return
-	}
-	d.describeInstancesFromKV(msg, d.stateStore.ListTerminatedInstances, 48, "terminated", "handleEC2DescribeTerminatedInstances")
-}
-
-// describeInstancesFromKV is a shared helper for DescribeStopped/TerminatedInstances handlers.
-// It lists instances from a KV bucket, filters by account/instance ID, and responds with reservations.
-func (d *Daemon) describeInstancesFromKV(msg *nats.Msg, listFn func() ([]*vm.VM, error), fallbackCode int64, fallbackName, handlerName string) {
-	accountID := utils.AccountIDFromMsg(msg)
-
-	describeInput := &ec2.DescribeInstancesInput{}
-	if len(msg.Data) > 0 {
-		if errResp := utils.UnmarshalJsonPayload(describeInput, msg.Data); errResp != nil {
-			if err := msg.Respond(errResp); err != nil {
-				slog.Error("Failed to respond to NATS request", "err", err)
-			}
-			return
-		}
-	}
-
-	instanceIDFilter := make(map[string]bool)
-	for _, id := range describeInput.InstanceIds {
-		if id != nil {
-			instanceIDFilter[*id] = true
-		}
-	}
-
-	parsedFilters, filterErr := filterutil.ParseFilters(describeInput.Filters, describeInstancesValidFilters)
-	if filterErr != nil {
-		slog.Warn(handlerName+": invalid filter", "err", filterErr)
-		respondWithError(msg, awserrors.ErrorInvalidParameterValue)
-		return
-	}
-
-	instances, err := listFn()
-	if err != nil {
-		slog.Error(handlerName+": failed to list instances", "err", err)
-		respondWithError(msg, awserrors.ErrorServerInternal)
-		return
-	}
-
-	reservationMap := make(map[string]*ec2.Reservation)
-
-	for _, instance := range instances {
-		if !isInstanceVisible(accountID, instance.AccountID) {
-			continue
-		}
-		if len(instanceIDFilter) > 0 && !instanceIDFilter[instance.ID] {
-			continue
-		}
-		if instance.Reservation == nil || instance.Instance == nil {
-			slog.Warn(handlerName+": skipping instance with nil Reservation/Instance (data integrity issue)",
-				"instanceId", instance.ID)
-			continue
-		}
-
-		resID := ""
-		if instance.Reservation.ReservationId != nil {
-			resID = *instance.Reservation.ReservationId
-		}
-
-		if _, exists := reservationMap[resID]; !exists {
-			reservation := &ec2.Reservation{}
-			reservation.SetReservationId(resID)
-			if instance.Reservation.OwnerId != nil {
-				reservation.SetOwnerId(*instance.Reservation.OwnerId)
-			}
-			reservation.Instances = []*ec2.Instance{}
-			reservationMap[resID] = reservation
-		}
-
-		instanceCopy := *instance.Instance
-		instanceCopy.State = &ec2.InstanceState{}
-		if info, ok := vm.EC2APIState(instance.Status); ok {
-			instanceCopy.State.SetCode(info.Code)
-			instanceCopy.State.SetName(info.Name)
-		} else {
-			instanceCopy.State.SetCode(fallbackCode)
-			instanceCopy.State.SetName(fallbackName)
-		}
-
-		// Project IamInstanceProfile from vm.VM (cleared on terminate;
-		// preserved across stop/start). Mirrors handleEC2DescribeInstances.
-		if instance.IamInstanceProfileArn != "" {
-			instanceCopy.IamInstanceProfile = &ec2.IamInstanceProfile{
-				Arn: aws.String(instance.IamInstanceProfileArn),
-			}
-		} else {
-			instanceCopy.IamInstanceProfile = nil
-		}
-
-		if len(parsedFilters) > 0 && !instanceMatchesFilters(instance, &instanceCopy, parsedFilters) {
-			continue
-		}
-
-		reservationMap[resID].Instances = append(reservationMap[resID].Instances, &instanceCopy)
-	}
-
-	reservations := make([]*ec2.Reservation, 0, len(reservationMap))
-	for _, reservation := range reservationMap {
-		reservations = append(reservations, reservation)
-	}
-
-	respondWithJSON(msg, &ec2.DescribeInstancesOutput{Reservations: reservations})
-	slog.Info(handlerName+" completed", "count", len(reservations))
-}
-
-func (d *Daemon) handleEC2ModifyInstanceAttribute(msg *nats.Msg) {
-	handleNATSRequest(msg, d.instanceService.ModifyInstanceAttribute)
-}
-
-func (d *Daemon) handleEC2ModifyInstanceMetadataOptions(msg *nats.Msg) {
-	handleNATSRequest(msg, d.instanceService.ModifyInstanceMetadataOptions)
-}
-
-// handleEC2DescribeInstanceAttribute returns a single requested attribute for an instance.
-func (d *Daemon) handleEC2DescribeInstanceAttribute(msg *nats.Msg) {
-	handleNATSRequest(msg, d.instanceService.DescribeInstanceAttribute)
+	handleNATSRequest(d.instanceService.StartStoppedInstance)(msg)
 }

spinifex/daemon/daemon_handlers_instance_tags_test.go

@@ -13,7 +13,7 @@ import (
 	"github.com/mulgadc/spinifex/spinifex/objectstore"
 	"github.com/mulgadc/spinifex/spinifex/types"
 	"github.com/mulgadc/spinifex/spinifex/vm"
-	"github.com/nats-io/nats.go"
+	"github.com/nats-io/nats.go/jetstream"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 )
@@ -54,11 +54,11 @@ func (m *memStoppedStore) DeleteStoppedInstance(id string) error {
 }
 
 // UpdateStoppedInstance mimics the real CAS semantics: a missing record
-// returns nats.ErrKeyNotFound instead of resurrecting it.
+// returns jetstream.ErrKeyNotFound instead of resurrecting it.
 func (m *memStoppedStore) UpdateStoppedInstance(id string, mutate func(*vm.VM)) (*vm.VM, error) {
 	v, ok := m.instances[id]
 	if !ok {
-		return nil, nats.ErrKeyNotFound
+		return nil, jetstream.ErrKeyNotFound
 	}
 	mutate(v)
 	return v, nil

spinifex/daemon/daemon_handlers_instance_test.go

@@ -15,6 +15,7 @@ import (
 	"github.com/mulgadc/spinifex/spinifex/utils"
 	"github.com/mulgadc/spinifex/spinifex/vm"
 	"github.com/nats-io/nats.go"
+	"github.com/nats-io/nats.go/jetstream"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 )
@@ -126,7 +127,7 @@ func (f *fakeStateStore) ClaimStoppedInstance(id string) (*vm.VM, error) {
 
 // UpdateStoppedInstance mimics the real CAS semantics for tests: mutate runs
 // under the store lock against the stored value, and a missing record
-// returns nats.ErrKeyNotFound (matching JetStreamManager's createIfAbsent=false
+// returns jetstream.ErrKeyNotFound (matching JetStreamManager's createIfAbsent=false
 // behavior) rather than resurrecting it.
 func (f *fakeStateStore) UpdateStoppedInstance(id string, mutate func(*vm.VM)) (*vm.VM, error) {
 	f.mu.Lock()
@@ -136,7 +137,7 @@ func (f *fakeStateStore) UpdateStoppedInstance(id string, mutate func(*vm.VM)) (
 	}
 	v, ok := f.stopped[id]
 	if !ok {
-		return nil, nats.ErrKeyNotFound
+		return nil, jetstream.ErrKeyNotFound
 	}
 	mutate(v)
 	return v, nil
@@ -391,7 +392,7 @@ func TestHandleEC2TerminateStoppedInstance_LoadError(t *testing.T) {
 	d := daemonWithFakeStateStore(t, store)
 
 	body, _ := json.Marshal(handlers_ec2_instance.TerminateStoppedInstanceInput{InstanceID: "i-load-fail"})
-	reply := requestHandler(t, d.natsConn, "ec2.terminate.test1", d.handleEC2TerminateStoppedInstance, testAccountID, body)
+	reply := requestHandler(t, d.natsConn, "ec2.terminate.test1", handleNATSRequest(d.instanceService.TerminateStoppedInstance), testAccountID, body)
 	assert.Equal(t, awserrors.ErrorServerInternal, decodeError(t, reply.Data)["Code"])
 }
 
@@ -399,7 +400,7 @@ func TestHandleEC2TerminateStoppedInstance_StateStoreNil(t *testing.T) {
 	d := createTestDaemon(t, sharedNATSURL)
 
 	body, _ := json.Marshal(handlers_ec2_instance.TerminateStoppedInstanceInput{InstanceID: "i-no-store"})
-	reply := requestHandler(t, d.natsConn, "ec2.terminate.test2", d.handleEC2TerminateStoppedInstance, testAccountID, body)
+	reply := requestHandler(t, d.natsConn, "ec2.terminate.test2", handleNATSRequest(d.instanceService.TerminateStoppedInstance), testAccountID, body)
 	assert.Equal(t, awserrors.ErrorServerInternal, decodeError(t, reply.Data)["Code"])
 }
 
@@ -413,7 +414,7 @@ func TestHandleEC2TerminateStoppedInstance_WriteTerminatedFailureAborts(t *testi
 	d := daemonWithFakeStateStore(t, store)
 
 	body, _ := json.Marshal(handlers_ec2_instance.TerminateStoppedInstanceInput{InstanceID: v.ID})
-	reply := requestHandler(t, d.natsConn, "ec2.terminate.test3", d.handleEC2TerminateStoppedInstance, testAccountID, body)
+	reply := requestHandler(t, d.natsConn, "ec2.terminate.test3", handleNATSRequest(d.instanceService.TerminateStoppedInstance), testAccountID, body)
 	assert.Equal(t, awserrors.ErrorServerInternal, decodeError(t, reply.Data)["Code"])
 
 	store.mu.Lock()
@@ -436,7 +437,7 @@ func TestHandleEC2TerminateStoppedInstance_DeleteRetrySucceeds(t *testing.T) {
 	d := daemonWithFakeStateStore(t, store)
 
 	body, _ := json.Marshal(handlers_ec2_instance.TerminateStoppedInstanceInput{InstanceID: v.ID})
-	reply := requestHandler(t, d.natsConn, "ec2.terminate.test4", d.handleEC2TerminateStoppedInstance, testAccountID, body)
+	reply := requestHandler(t, d.natsConn, "ec2.terminate.test4", handleNATSRequest(d.instanceService.TerminateStoppedInstance), testAccountID, body)
 
 	var resp map[string]string
 	require.NoError(t, json.Unmarshal(reply.Data, &resp))
@@ -463,7 +464,7 @@ func TestHandleEC2TerminateStoppedInstance_DeleteAlwaysFailsKeepsTerminated(t *t
 	d := daemonWithFakeStateStore(t, store)
 
 	body, _ := json.Marshal(handlers_ec2_instance.TerminateStoppedInstanceInput{InstanceID: v.ID})
-	reply := requestHandler(t, d.natsConn, "ec2.terminate.test5", d.handleEC2TerminateStoppedInstance, testAccountID, body)
+	reply := requestHandler(t, d.natsConn, "ec2.terminate.test5", handleNATSRequest(d.instanceService.TerminateStoppedInstance), testAccountID, body)
 
 	var resp map[string]string
 	require.NoError(t, json.Unmarshal(reply.Data, &resp))
@@ -481,7 +482,7 @@ func TestHandleEC2TerminateStoppedInstance_CrossTenantRejected(t *testing.T) {
 	d := daemonWithFakeStateStore(t, store)
 
 	body, _ := json.Marshal(handlers_ec2_instance.TerminateStoppedInstanceInput{InstanceID: "i-foreign-term"})
-	reply := requestHandler(t, d.natsConn, "ec2.terminate.test6", d.handleEC2TerminateStoppedInstance, testAccountID, body)
+	reply := requestHandler(t, d.natsConn, "ec2.terminate.test6", handleNATSRequest(d.instanceService.TerminateStoppedInstance), testAccountID, body)
 	assert.Equal(t, awserrors.ErrorInvalidInstanceIDNotFound, decodeError(t, reply.Data)["Code"])
 
 	store.mu.Lock()
@@ -506,7 +507,7 @@ func TestHandleEC2ModifyInstanceAttribute_WriteFailureReturnsServerInternal(t *t
 		InstanceType: &ec2.AttributeValue{Value: aws.String("t3.large")},
 	}
 	body, _ := json.Marshal(input)
-	reply := requestHandler(t, d.natsConn, "ec2.ModifyInstanceAttribute.test1", d.handleEC2ModifyInstanceAttribute, testAccountID, body)
+	reply := requestHandler(t, d.natsConn, "ec2.ModifyInstanceAttribute.test1", handleNATSRequest(d.instanceService.ModifyInstanceAttribute), testAccountID, body)
 	assert.Equal(t, awserrors.ErrorServerInternal, decodeError(t, reply.Data)["Code"])
 }
 
@@ -520,7 +521,7 @@ func TestHandleEC2ModifyInstanceAttribute_LoadFailureReturnsServerInternal(t *te
 		InstanceType: &ec2.AttributeValue{Value: aws.String("t3.large")},
 	}
 	body, _ := json.Marshal(input)
-	reply := requestHandler(t, d.natsConn, "ec2.ModifyInstanceAttribute.test2", d.handleEC2ModifyInstanceAttribute, testAccountID, body)
+	reply := requestHandler(t, d.natsConn, "ec2.ModifyInstanceAttribute.test2", handleNATSRequest(d.instanceService.ModifyInstanceAttribute), testAccountID, body)
 	assert.Equal(t, awserrors.ErrorServerInternal, decodeError(t, reply.Data)["Code"])
 }
 
@@ -544,7 +545,7 @@ func TestHandleEC2ModifyInstanceAttribute_NilInstanceFieldGuard(t *testing.T) {
 		InstanceType: &ec2.AttributeValue{Value: aws.String("t3.large")},
 	}
 	body, _ := json.Marshal(input)
-	reply := requestHandler(t, d.natsConn, "ec2.ModifyInstanceAttribute.test3", d.handleEC2ModifyInstanceAttribute, testAccountID, body)
+	reply := requestHandler(t, d.natsConn, "ec2.ModifyInstanceAttribute.test3", handleNATSRequest(d.instanceService.ModifyInstanceAttribute), testAccountID, body)
 	assert.Equal(t, awserrors.ErrorServerInternal, decodeError(t, reply.Data)["Code"])
 }
 
@@ -559,7 +560,7 @@ func TestHandleEC2ModifyInstanceAttribute_EmptyInstanceTypeRejected(t *testing.T
 		InstanceType: &ec2.AttributeValue{Value: aws.String("")},
 	}
 	body, _ := json.Marshal(input)
-	reply := requestHandler(t, d.natsConn, "ec2.ModifyInstanceAttribute.test4", d.handleEC2ModifyInstanceAttribute, testAccountID, body)
+	reply := requestHandler(t, d.natsConn, "ec2.ModifyInstanceAttribute.test4", handleNATSRequest(d.instanceService.ModifyInstanceAttribute), testAccountID, body)
 	assert.Equal(t, awserrors.ErrorInvalidInstanceAttributeValue, decodeError(t, reply.Data)["Code"])
 }
 
@@ -573,7 +574,7 @@ func TestHandleEC2ModifyInstanceAttribute_CrossTenantRejected(t *testing.T) {
 		InstanceType: &ec2.AttributeValue{Value: aws.String("t3.large")},
 	}
 	body, _ := json.Marshal(input)
-	reply := requestHandler(t, d.natsConn, "ec2.ModifyInstanceAttribute.test5", d.handleEC2ModifyInstanceAttribute, testAccountID, body)
+	reply := requestHandler(t, d.natsConn, "ec2.ModifyInstanceAttribute.test5", handleNATSRequest(d.instanceService.ModifyInstanceAttribute), testAccountID, body)
 	assert.Equal(t, awserrors.ErrorInvalidInstanceIDNotFound, decodeError(t, reply.Data)["Code"])
 }
 
@@ -591,7 +592,7 @@ func TestHandleEC2DescribeInstanceAttribute_StoppedFallback_LoadError(t *testing
 		Attribute:  aws.String(ec2.InstanceAttributeNameInstanceType),
 	}
 	body, _ := json.Marshal(input)
-	reply := requestHandler(t, d.natsConn, "ec2.DescribeInstanceAttribute.test1", d.handleEC2DescribeInstanceAttribute, testAccountID, body)
+	reply := requestHandler(t, d.natsConn, "ec2.DescribeInstanceAttribute.test1", handleNATSRequest(d.instanceService.DescribeInstanceAttribute), testAccountID, body)
 	assert.Equal(t, awserrors.ErrorServerInternal, decodeError(t, reply.Data)["Code"])
 }
 
@@ -608,7 +609,7 @@ func TestHandleEC2DescribeInstanceAttribute_StoppedFallback_HitsKV(t *testing.T)
 		Attribute:  aws.String(ec2.InstanceAttributeNameInstanceType),
 	}
 	body, _ := json.Marshal(input)
-	reply := requestHandler(t, d.natsConn, "ec2.DescribeInstanceAttribute.test2", d.handleEC2DescribeInstanceAttribute, testAccountID, body)
+	reply := requestHandler(t, d.natsConn, "ec2.DescribeInstanceAttribute.test2", handleNATSRequest(d.instanceService.DescribeInstanceAttribute), testAccountID, body)
 
 	var output ec2.DescribeInstanceAttributeOutput
 	require.NoError(t, json.Unmarshal(reply.Data, &output))
@@ -627,7 +628,7 @@ func TestHandleEC2DescribeInstanceAttribute_StateStoreNil(t *testing.T) {
 		Attribute:  aws.String(ec2.InstanceAttributeNameInstanceType),
 	}
 	body, _ := json.Marshal(input)
-	reply := requestHandler(t, d.natsConn, "ec2.DescribeInstanceAttribute.test3", d.handleEC2DescribeInstanceAttribute, testAccountID, body)
+	reply := requestHandler(t, d.natsConn, "ec2.DescribeInstanceAttribute.test3", handleNATSRequest(d.instanceService.DescribeInstanceAttribute), testAccountID, body)
 	assert.Equal(t, awserrors.ErrorServerInternal, decodeError(t, reply.Data)["Code"])
 }
 
@@ -638,7 +639,7 @@ func TestHandleEC2DescribeStoppedInstances_ListError(t *testing.T) {
 	store.listStoppedErr = errors.New("list failed")
 	d := daemonWithFakeStateStore(t, store)
 
-	reply := requestHandler(t, d.natsConn, "ec2.DescribeStoppedInstances.test1", d.handleEC2DescribeStoppedInstances, testAccountID, []byte("{}"))
+	reply := requestHandler(t, d.natsConn, "ec2.DescribeStoppedInstances.test1", handleNATSRequest(d.instanceService.DescribeStoppedInstances), testAccountID, []byte("{}"))
 	assert.Equal(t, awserrors.ErrorServerInternal, decodeError(t, reply.Data)["Code"])
 }
 
@@ -647,21 +648,7 @@ func TestHandleEC2DescribeTerminatedInstances_ListError(t *testing.T) {
 	store.listTerminatedErr = errors.New("list failed")
 	d := daemonWithFakeStateStore(t, store)
 
-	reply := requestHandler(t, d.natsConn, "ec2.DescribeTerminatedInstances.test1", d.handleEC2DescribeTerminatedInstances, testAccountID, []byte("{}"))
-	assert.Equal(t, awserrors.ErrorServerInternal, decodeError(t, reply.Data)["Code"])
-}
-
-func TestHandleEC2DescribeStoppedInstances_StateStoreNil(t *testing.T) {
-	d := createTestDaemon(t, sharedNATSURL)
-
-	reply := requestHandler(t, d.natsConn, "ec2.DescribeStoppedInstances.test2", d.handleEC2DescribeStoppedInstances, testAccountID, []byte("{}"))
-	assert.Equal(t, awserrors.ErrorServerInternal, decodeError(t, reply.Data)["Code"])
-}
-
-func TestHandleEC2DescribeTerminatedInstances_StateStoreNil(t *testing.T) {
-	d := createTestDaemon(t, sharedNATSURL)
-
-	reply := requestHandler(t, d.natsConn, "ec2.DescribeTerminatedInstances.test2", d.handleEC2DescribeTerminatedInstances, testAccountID, []byte("{}"))
+	reply := requestHandler(t, d.natsConn, "ec2.DescribeTerminatedInstances.test1", handleNATSRequest(d.instanceService.DescribeTerminatedInstances), testAccountID, []byte("{}"))
 	assert.Equal(t, awserrors.ErrorServerInternal, decodeError(t, reply.Data)["Code"])
 }
 
@@ -671,7 +658,7 @@ func TestHandleEC2DescribeStoppedInstances_CrossAccountIsolation(t *testing.T) {
 	store.stopped["i-yours"] = stoppedVMFixture("i-yours", "999988887777")
 	d := daemonWithFakeStateStore(t, store)
 
-	reply := requestHandler(t, d.natsConn, "ec2.DescribeStoppedInstances.test3", d.handleEC2DescribeStoppedInstances, testAccountID, []byte("{}"))
+	reply := requestHandler(t, d.natsConn, "ec2.DescribeStoppedInstances.test3", handleNATSRequest(d.instanceService.DescribeStoppedInstances), testAccountID, []byte("{}"))
 
 	var output ec2.DescribeInstancesOutput
 	require.NoError(t, json.Unmarshal(reply.Data, &output))
@@ -686,191 +673,3 @@ func TestHandleEC2DescribeStoppedInstances_CrossAccountIsolation(t *testing.T) {
 	}
 	assert.ElementsMatch(t, []string{"i-mine"}, seen, "caller must only see their own instances")
 }
-
-func TestHandleEC2DescribeStoppedInstances_EmptyReturnsValidShape(t *testing.T) {
-	store := newFakeStateStore()
-	d := daemonWithFakeStateStore(t, store)
-
-	reply := requestHandler(t, d.natsConn, "ec2.DescribeStoppedInstances.test4", d.handleEC2DescribeStoppedInstances, testAccountID, []byte("{}"))
-
-	var output ec2.DescribeInstancesOutput
-	require.NoError(t, json.Unmarshal(reply.Data, &output))
-	assert.Empty(t, output.Reservations, "empty store must produce empty reservation list")
-}
-
-// Two instances sharing a ReservationId must collapse into a single
-// reservation with both Instances attached.
-func TestHandleEC2DescribeStoppedInstances_ReservationGrouping(t *testing.T) {
-	store := newFakeStateStore()
-
-	a := stoppedVMFixture("i-grp-a", testAccountID)
-	b := stoppedVMFixture("i-grp-b", testAccountID)
-	a.Reservation.ReservationId = aws.String("r-shared")
-	b.Reservation.ReservationId = aws.String("r-shared")
-	store.stopped[a.ID] = a
-	store.stopped[b.ID] = b
-
-	d := daemonWithFakeStateStore(t, store)
-
-	reply := requestHandler(t, d.natsConn, "ec2.DescribeStoppedInstances.test5", d.handleEC2DescribeStoppedInstances, testAccountID, []byte("{}"))
-
-	var output ec2.DescribeInstancesOutput
-	require.NoError(t, json.Unmarshal(reply.Data, &output))
-	require.Len(t, output.Reservations, 1, "shared ReservationId must collapse into one reservation")
-	require.NotNil(t, output.Reservations[0].ReservationId)
-	assert.Equal(t, "r-shared", *output.Reservations[0].ReservationId)
-	assert.Len(t, output.Reservations[0].Instances, 2)
-}
-
-// DescribeInstances echoes the consumed capacity reservation so a targeted
-// launch reports a CapacityReservationId (without it, Terraform never converges).
-func TestHandleEC2DescribeInstances_CapacityReservationEcho(t *testing.T) {
-	daemon := createTestDaemon(t, sharedNATSURL)
-
-	reservation := &ec2.Reservation{}
-	reservation.SetReservationId("r-cr-echo")
-	reservation.SetOwnerId(testAccountID)
-	instance := &ec2.Instance{}
-	instance.SetInstanceId("i-cr-echo")
-	instance.SetInstanceType("t3.micro")
-
-	daemon.vmMgr.Insert(&vm.VM{
-		ID:                    "i-cr-echo",
-		Status:                vm.StateRunning,
-		AccountID:             testAccountID,
-		CapacityReservationId: "cr-0123456789abcdef0",
-		Reservation:           reservation,
-		Instance:              instance,
-	})
-
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstances", daemon.handleEC2DescribeInstances)
-	require.NoError(t, err)
-	defer func() { _ = sub.Unsubscribe() }()
-
-	reqData, _ := json.Marshal(&ec2.DescribeInstancesInput{})
-	reply, err := natsRequest(daemon.natsConn, "ec2.DescribeInstances", reqData, 5*time.Second)
-	require.NoError(t, err)
-	var out ec2.DescribeInstancesOutput
-	require.NoError(t, json.Unmarshal(reply.Data, &out))
-	require.Len(t, out.Reservations, 1)
-	require.Len(t, out.Reservations[0].Instances, 1)
-
-	got := out.Reservations[0].Instances[0]
-	assert.Equal(t, "cr-0123456789abcdef0", aws.StringValue(got.CapacityReservationId))
-	require.NotNil(t, got.CapacityReservationSpecification)
-	assert.Equal(t, ec2.CapacityReservationPreferenceOpen,
-		aws.StringValue(got.CapacityReservationSpecification.CapacityReservationPreference))
-	require.NotNil(t, got.CapacityReservationSpecification.CapacityReservationTarget)
-	assert.Equal(t, "cr-0123456789abcdef0",
-		aws.StringValue(got.CapacityReservationSpecification.CapacityReservationTarget.CapacityReservationId))
-}
-
-// An instance with no reservation reports no capacity-reservation fields.
-func TestHandleEC2DescribeInstances_NoCapacityReservationEcho(t *testing.T) {
-	daemon := createTestDaemon(t, sharedNATSURL)
-
-	reservation := &ec2.Reservation{}
-	reservation.SetReservationId("r-plain")
-	reservation.SetOwnerId(testAccountID)
-	instance := &ec2.Instance{}
-	instance.SetInstanceId("i-plain")
-	instance.SetInstanceType("t3.micro")
-
-	daemon.vmMgr.Insert(&vm.VM{
-		ID:          "i-plain",
-		Status:      vm.StateRunning,
-		AccountID:   testAccountID,
-		Reservation: reservation,
-		Instance:    instance,
-	})
-
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstances", daemon.handleEC2DescribeInstances)
-	require.NoError(t, err)
-	defer func() { _ = sub.Unsubscribe() }()
-
-	reqData, _ := json.Marshal(&ec2.DescribeInstancesInput{})
-	reply, err := natsRequest(daemon.natsConn, "ec2.DescribeInstances", reqData, 5*time.Second)
-	require.NoError(t, err)
-	var out ec2.DescribeInstancesOutput
-	require.NoError(t, json.Unmarshal(reply.Data, &out))
-	require.Len(t, out.Reservations, 1)
-	require.Len(t, out.Reservations[0].Instances, 1)
-
-	got := out.Reservations[0].Instances[0]
-	assert.Empty(t, aws.StringValue(got.CapacityReservationId))
-	assert.Nil(t, got.CapacityReservationSpecification)
-}
-
-// A spot-launched instance projects InstanceLifecycle + SpotInstanceRequestId.
-func TestHandleEC2DescribeInstances_SpotLineageEcho(t *testing.T) {
-	daemon := createTestDaemon(t, sharedNATSURL)
-
-	reservation := &ec2.Reservation{}
-	reservation.SetReservationId("r-spot-echo")
-	reservation.SetOwnerId(testAccountID)
-	instance := &ec2.Instance{}
-	instance.SetInstanceId("i-spot-echo")
-	instance.SetInstanceType("t3.micro")
-
-	daemon.vmMgr.Insert(&vm.VM{
-		ID:                    "i-spot-echo",
-		Status:                vm.StateRunning,
-		AccountID:             testAccountID,
-		InstanceLifecycle:     ec2.InstanceLifecycleTypeSpot,
-		SpotInstanceRequestId: "sir-0123456789abcdef0",
-		Reservation:           reservation,
-		Instance:              instance,
-	})
-
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstances", daemon.handleEC2DescribeInstances)
-	require.NoError(t, err)
-	defer func() { _ = sub.Unsubscribe() }()
-
-	reqData, _ := json.Marshal(&ec2.DescribeInstancesInput{})
-	reply, err := natsRequest(daemon.natsConn, "ec2.DescribeInstances", reqData, 5*time.Second)
-	require.NoError(t, err)
-	var out ec2.DescribeInstancesOutput
-	require.NoError(t, json.Unmarshal(reply.Data, &out))
-	require.Len(t, out.Reservations, 1)
-	require.Len(t, out.Reservations[0].Instances, 1)
-
-	got := out.Reservations[0].Instances[0]
-	assert.Equal(t, ec2.InstanceLifecycleTypeSpot, aws.StringValue(got.InstanceLifecycle))
-	assert.Equal(t, "sir-0123456789abcdef0", aws.StringValue(got.SpotInstanceRequestId))
-}
-
-// An on-demand instance reports no spot-lineage fields.
-func TestHandleEC2DescribeInstances_NoSpotLineageEcho(t *testing.T) {
-	daemon := createTestDaemon(t, sharedNATSURL)
-
-	reservation := &ec2.Reservation{}
-	reservation.SetReservationId("r-ondemand")
-	reservation.SetOwnerId(testAccountID)
-	instance := &ec2.Instance{}
-	instance.SetInstanceId("i-ondemand")
-	instance.SetInstanceType("t3.micro")
-
-	daemon.vmMgr.Insert(&vm.VM{
-		ID:          "i-ondemand",
-		Status:      vm.StateRunning,
-		AccountID:   testAccountID,
-		Reservation: reservation,
-		Instance:    instance,
-	})
-
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstances", daemon.handleEC2DescribeInstances)
-	require.NoError(t, err)
-	defer func() { _ = sub.Unsubscribe() }()
-
-	reqData, _ := json.Marshal(&ec2.DescribeInstancesInput{})
-	reply, err := natsRequest(daemon.natsConn, "ec2.DescribeInstances", reqData, 5*time.Second)
-	require.NoError(t, err)
-	var out ec2.DescribeInstancesOutput
-	require.NoError(t, json.Unmarshal(reply.Data, &out))
-	require.Len(t, out.Reservations, 1)
-	require.Len(t, out.Reservations[0].Instances, 1)
-
-	got := out.Reservations[0].Instances[0]
-	assert.Empty(t, aws.StringValue(got.InstanceLifecycle))
-	assert.Empty(t, aws.StringValue(got.SpotInstanceRequestId))
-}

spinifex/daemon/daemon_handlers_key.go

@@ -6,22 +6,6 @@ import (
 	"github.com/nats-io/nats.go"
 )
 
-func (d *Daemon) handleEC2CreateKeyPair(msg *nats.Msg) {
-	handleNATSRequest(msg, d.keyService.CreateKeyPair)
-}
-
-func (d *Daemon) handleEC2DeleteKeyPair(msg *nats.Msg) {
-	handleNATSRequest(msg, d.keyService.DeleteKeyPair)
-}
-
-func (d *Daemon) handleEC2DescribeKeyPairs(msg *nats.Msg) {
-	handleNATSRequest(msg, d.keyService.DescribeKeyPairs)
-}
-
-func (d *Daemon) handleEC2ImportKeyPair(msg *nats.Msg) {
-	handleNATSRequest(msg, d.keyService.ImportKeyPair)
-}
-
 // handleIMDSGetPublicKey answers the IMDS material-path RPC. The responder lives
 // here (not in vpcd) because the daemon holds the key service and its Predastore
 // credentials; vpcd carries only a nats.Conn.

spinifex/daemon/daemon_handlers_launchtemplate.go

@@ -1,31 +0,0 @@
-package daemon
-
-import "github.com/nats-io/nats.go"
-
-func (d *Daemon) handleEC2CreateLaunchTemplate(msg *nats.Msg) {
-	handleNATSRequest(msg, d.launchTemplateService.CreateLaunchTemplate)
-}
-
-func (d *Daemon) handleEC2CreateLaunchTemplateVersion(msg *nats.Msg) {
-	handleNATSRequest(msg, d.launchTemplateService.CreateLaunchTemplateVersion)
-}
-
-func (d *Daemon) handleEC2DeleteLaunchTemplate(msg *nats.Msg) {
-	handleNATSRequest(msg, d.launchTemplateService.DeleteLaunchTemplate)
-}
-
-func (d *Daemon) handleEC2DeleteLaunchTemplateVersions(msg *nats.Msg) {
-	handleNATSRequest(msg, d.launchTemplateService.DeleteLaunchTemplateVersions)
-}
-
-func (d *Daemon) handleEC2ModifyLaunchTemplate(msg *nats.Msg) {
-	handleNATSRequest(msg, d.launchTemplateService.ModifyLaunchTemplate)
-}
-
-func (d *Daemon) handleEC2DescribeLaunchTemplates(msg *nats.Msg) {
-	handleNATSRequest(msg, d.launchTemplateService.DescribeLaunchTemplates)
-}
-
-func (d *Daemon) handleEC2DescribeLaunchTemplateVersions(msg *nats.Msg) {
-	handleNATSRequest(msg, d.launchTemplateService.DescribeLaunchTemplateVersions)
-}

spinifex/daemon/daemon_handlers_natgw.go

@@ -1,17 +0,0 @@
-package daemon
-
-import (
-	"github.com/nats-io/nats.go"
-)
-
-func (d *Daemon) handleEC2CreateNatGateway(msg *nats.Msg) {
-	handleNATSRequest(msg, d.natGatewayService.CreateNatGateway)
-}
-
-func (d *Daemon) handleEC2DeleteNatGateway(msg *nats.Msg) {
-	handleNATSRequest(msg, d.natGatewayService.DeleteNatGateway)
-}
-
-func (d *Daemon) handleEC2DescribeNatGateways(msg *nats.Msg) {
-	handleNATSRequest(msg, d.natGatewayService.DescribeNatGateways)
-}

spinifex/daemon/daemon_handlers_placementgroup.go

@@ -1,39 +0,0 @@
-package daemon
-
-import "github.com/nats-io/nats.go"
-
-func (d *Daemon) handleEC2CreatePlacementGroup(msg *nats.Msg) {
-	handleNATSRequest(msg, d.placementGroupService.CreatePlacementGroup)
-}
-
-func (d *Daemon) handleEC2DeletePlacementGroup(msg *nats.Msg) {
-	handleNATSRequest(msg, d.placementGroupService.DeletePlacementGroup)
-}
-
-func (d *Daemon) handleEC2DescribePlacementGroups(msg *nats.Msg) {
-	handleNATSRequest(msg, d.placementGroupService.DescribePlacementGroups)
-}
-
-func (d *Daemon) handleEC2ReserveSpreadNodes(msg *nats.Msg) {
-	handleNATSRequest(msg, d.placementGroupService.ReserveSpreadNodes)
-}
-
-func (d *Daemon) handleEC2FinalizeSpreadInstances(msg *nats.Msg) {
-	handleNATSRequest(msg, d.placementGroupService.FinalizeSpreadInstances)
-}
-
-func (d *Daemon) handleEC2ReleaseSpreadNodes(msg *nats.Msg) {
-	handleNATSRequest(msg, d.placementGroupService.ReleaseSpreadNodes)
-}
-
-func (d *Daemon) handleEC2RemoveInstanceFromPlacementGroup(msg *nats.Msg) {
-	handleNATSRequest(msg, d.placementGroupService.RemoveInstance)
-}
-
-func (d *Daemon) handleEC2ReserveClusterNode(msg *nats.Msg) {
-	handleNATSRequest(msg, d.placementGroupService.ReserveClusterNode)
-}
-
-func (d *Daemon) handleEC2FinalizeClusterInstances(msg *nats.Msg) {
-	handleNATSRequest(msg, d.placementGroupService.FinalizeClusterInstances)
-}

spinifex/daemon/daemon_handlers_routetable.go

@@ -1,41 +0,0 @@
-package daemon
-
-import (
-	"github.com/nats-io/nats.go"
-)
-
-func (d *Daemon) handleEC2CreateRouteTable(msg *nats.Msg) {
-	handleNATSRequest(msg, d.routeTableService.CreateRouteTable)
-}
-
-func (d *Daemon) handleEC2DeleteRouteTable(msg *nats.Msg) {
-	handleNATSRequest(msg, d.routeTableService.DeleteRouteTable)
-}
-
-func (d *Daemon) handleEC2DescribeRouteTables(msg *nats.Msg) {
-	handleNATSRequest(msg, d.routeTableService.DescribeRouteTables)
-}
-
-func (d *Daemon) handleEC2CreateRoute(msg *nats.Msg) {
-	handleNATSRequest(msg, d.routeTableService.CreateRoute)
-}
-
-func (d *Daemon) handleEC2DeleteRoute(msg *nats.Msg) {
-	handleNATSRequest(msg, d.routeTableService.DeleteRoute)
-}
-
-func (d *Daemon) handleEC2ReplaceRoute(msg *nats.Msg) {
-	handleNATSRequest(msg, d.routeTableService.ReplaceRoute)
-}
-
-func (d *Daemon) handleEC2AssociateRouteTable(msg *nats.Msg) {
-	handleNATSRequest(msg, d.routeTableService.AssociateRouteTable)
-}
-
-func (d *Daemon) handleEC2DisassociateRouteTable(msg *nats.Msg) {
-	handleNATSRequest(msg, d.routeTableService.DisassociateRouteTable)
-}
-
-func (d *Daemon) handleEC2ReplaceRouteTableAssociation(msg *nats.Msg) {
-	handleNATSRequest(msg, d.routeTableService.ReplaceRouteTableAssociation)
-}

spinifex/daemon/daemon_handlers_sg.go

@@ -1,35 +0,0 @@
-package daemon
-
-import "github.com/nats-io/nats.go"
-
-func (d *Daemon) handleEC2CreateSecurityGroup(msg *nats.Msg) {
-	handleNATSRequest(msg, d.vpcService.CreateSecurityGroup)
-}
-
-func (d *Daemon) handleEC2DeleteSecurityGroup(msg *nats.Msg) {
-	handleNATSRequest(msg, d.vpcService.DeleteSecurityGroup)
-}
-
-func (d *Daemon) handleEC2DescribeSecurityGroups(msg *nats.Msg) {
-	handleNATSRequest(msg, d.vpcService.DescribeSecurityGroups)
-}
-
-func (d *Daemon) handleEC2DescribeSecurityGroupRules(msg *nats.Msg) {
-	handleNATSRequest(msg, d.vpcService.DescribeSecurityGroupRules)
-}
-
-func (d *Daemon) handleEC2AuthorizeSecurityGroupIngress(msg *nats.Msg) {
-	handleNATSRequest(msg, d.vpcService.AuthorizeSecurityGroupIngress)
-}
-
-func (d *Daemon) handleEC2AuthorizeSecurityGroupEgress(msg *nats.Msg) {
-	handleNATSRequest(msg, d.vpcService.AuthorizeSecurityGroupEgress)
-}
-
-func (d *Daemon) handleEC2RevokeSecurityGroupIngress(msg *nats.Msg) {
-	handleNATSRequest(msg, d.vpcService.RevokeSecurityGroupIngress)
-}
-
-func (d *Daemon) handleEC2RevokeSecurityGroupEgress(msg *nats.Msg) {
-	handleNATSRequest(msg, d.vpcService.RevokeSecurityGroupEgress)
-}

spinifex/daemon/daemon_handlers_snapshot.go

@@ -1,21 +0,0 @@
-package daemon
-
-import (
-	"github.com/nats-io/nats.go"
-)
-
-func (d *Daemon) handleEC2CreateSnapshot(msg *nats.Msg) {
-	handleNATSRequest(msg, d.snapshotService.CreateSnapshot)
-}
-
-func (d *Daemon) handleEC2DescribeSnapshots(msg *nats.Msg) {
-	handleNATSRequest(msg, d.snapshotService.DescribeSnapshots)
-}
-
-func (d *Daemon) handleEC2DeleteSnapshot(msg *nats.Msg) {
-	handleNATSRequest(msg, d.snapshotService.DeleteSnapshot)
-}
-
-func (d *Daemon) handleEC2CopySnapshot(msg *nats.Msg) {
-	handleNATSRequest(msg, d.snapshotService.CopySnapshot)
-}

spinifex/daemon/daemon_handlers_spotinstance.go

@@ -11,18 +11,6 @@ import (
 	"github.com/nats-io/nats.go"
 )
 
-func (d *Daemon) handleEC2PutSpotInstanceRequests(msg *nats.Msg) {
-	handleNATSRequest(msg, d.spotInstanceService.PutSpotInstanceRequests)
-}
-
-func (d *Daemon) handleEC2DescribeSpotInstanceRequests(msg *nats.Msg) {
-	handleNATSRequest(msg, d.spotInstanceService.DescribeSpotInstanceRequests)
-}
-
-func (d *Daemon) handleEC2CancelSpotInstanceRequests(msg *nats.Msg) {
-	handleNATSRequest(msg, d.spotInstanceService.CancelSpotInstanceRequests)
-}
-
 // handleSetSpotLineage stamps the spot lineage (InstanceLifecycle=spot +
 // SpotInstanceRequestId) onto a launched VM. Dispatched from handleEC2Events,
 // which already verified ownership; the SIR id is the only datum on the wire as
@@ -39,7 +27,7 @@ func (d *Daemon) handleSetSpotLineage(ctx context.Context, msg *nats.Msg, comman
 		return true
 	})
 	if err != nil {
-		respondWithError(msg, awserrors.ValidErrorCode(err.Error()))
+		respondWithError(msg, awserrors.ValidErrorCodeFromError(err))
 		return
 	}
 	if !found {

spinifex/daemon/daemon_handlers_tag.go

@@ -21,18 +21,6 @@ import (
 // only a partitioned-but-subscribed owner pays the full timeout.
 const instanceTagCommandTimeout = 5 * time.Second
 
-func (d *Daemon) handleEC2CreateTags(msg *nats.Msg) {
-	handleNATSRequest(msg, d.createTags)
-}
-
-func (d *Daemon) handleEC2DeleteTags(msg *nats.Msg) {
-	handleNATSRequest(msg, d.deleteTags)
-}
-
-func (d *Daemon) handleEC2DescribeTags(msg *nats.Msg) {
-	handleNATSRequest(msg, d.tagsService.DescribeTags)
-}
-
 // recordTagMirror is implemented by every service whose resource records
 // mirror the central tag store, so tag-filtered describes observe tags
 // written through create-tags/delete-tags.

spinifex/daemon/daemon_handlers_test.go

@@ -5,6 +5,7 @@ import (
 	"context"
 	"encoding/base64"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"log/slog"
 	"os"
@@ -39,6 +40,7 @@ import (
 	"github.com/mulgadc/viperblock/viperblock"
 	"github.com/nats-io/nats-server/v2/server"
 	"github.com/nats-io/nats.go"
+	"github.com/nats-io/nats.go/jetstream"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 )
@@ -131,7 +133,7 @@ func initAccountServiceForTest(t *testing.T, daemon *Daemon) {
 	require.NoError(t, err)
 	t.Cleanup(func() { nc.Close() })
 
-	svc, err := handlers_ec2_account.NewAccountSettingsServiceImplWithNATS(nil, nc)
+	svc, err := handlers_ec2_account.NewAccountSettingsServiceImplWithNATS(t.Context(), nil, nc)
 	require.NoError(t, err)
 	daemon.accountService = svc
 }
@@ -157,9 +159,7 @@ func TestHandleNATSRequest_ValidRequest(t *testing.T) {
 		return &testOutput{Greeting: "hello " + in.Name}, nil
 	}
 
-	sub, err := nc.Subscribe("test.greet", func(msg *nats.Msg) {
-		handleNATSRequest(msg, serviceFn)
-	})
+	sub, err := nc.Subscribe("test.greet", handleNATSRequest(serviceFn))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -184,9 +184,7 @@ func TestHandleNATSRequest_MalformedJSON(t *testing.T) {
 		return &testOutput{Greeting: "hello"}, nil
 	}
 
-	sub, err := nc.Subscribe("test.malformed", func(msg *nats.Msg) {
-		handleNATSRequest(msg, serviceFn)
-	})
+	sub, err := nc.Subscribe("test.malformed", handleNATSRequest(serviceFn))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -210,9 +208,7 @@ func TestHandleNATSRequest_ServiceError(t *testing.T) {
 		return nil, fmt.Errorf("something went wrong")
 	}
 
-	sub, err := nc.Subscribe("test.err", func(msg *nats.Msg) {
-		handleNATSRequest(msg, serviceFn)
-	})
+	sub, err := nc.Subscribe("test.err", handleNATSRequest(serviceFn))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -228,6 +224,30 @@ func TestHandleNATSRequest_ServiceError(t *testing.T) {
 	assert.Equal(t, awserrors.ErrorServerInternal, errResp["Code"])
 }
 
+func TestHandleNATSRequest_WrappedAWSServiceError(t *testing.T) {
+	nc, err := nats.Connect(sharedNATSURL)
+	require.NoError(t, err)
+	defer nc.Close()
+
+	serviceFn := func(_ context.Context, _ *testInput, _ string) (*testOutput, error) {
+		cause := errors.New(awserrors.ErrorInsufficientAddressCapacity)
+		return nil, fmt.Errorf("allocate public address: %w", cause)
+	}
+
+	sub, err := nc.Subscribe("test.err.wrapped", handleNATSRequest(serviceFn))
+	require.NoError(t, err)
+	defer sub.Unsubscribe()
+
+	reqData, err := json.Marshal(testInput{Name: "world"})
+	require.NoError(t, err)
+	reply, err := nc.Request("test.err.wrapped", reqData, 5*time.Second)
+	require.NoError(t, err)
+
+	var errResp map[string]any
+	require.NoError(t, json.Unmarshal(reply.Data, &errResp))
+	assert.Equal(t, awserrors.ErrorInsufficientAddressCapacity, errResp["Code"])
+}
+
 // --- Handler wrapper tests (representative set via NATS round-trip) ---
 
 func TestHandleEC2CreateKeyPair_RoundTrip(t *testing.T) {
@@ -235,7 +255,7 @@ func TestHandleEC2CreateKeyPair_RoundTrip(t *testing.T) {
 
 	daemon := createFullTestDaemon(t, natsURL)
 
-	sub, err := daemon.natsConn.QueueSubscribe("ec2.CreateKeyPair", "spinifex-workers", daemon.handleEC2CreateKeyPair)
+	sub, err := daemon.natsConn.QueueSubscribe("ec2.CreateKeyPair", "spinifex-workers", handleNATSRequest(daemon.keyService.CreateKeyPair))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -938,7 +958,7 @@ func TestRespondWithVolumeAttachment(t *testing.T) {
 	assert.Equal(t, "/dev/sdf", *attachment.Device)
 	assert.Equal(t, "attached", *attachment.State)
 	assert.NotNil(t, attachment.AttachTime)
-	assert.Equal(t, false, *attachment.DeleteOnTermination)
+	assert.False(t, *attachment.DeleteOnTermination)
 }
 
 // --- handleEC2ModifyVolume tests ---
@@ -1226,7 +1246,7 @@ func TestHandleEC2DescribeStoppedInstances_ReturnsStoppedInstances(t *testing.T)
 	err := daemon.jsManager.WriteStoppedInstance(stoppedVM.ID, stoppedVM)
 	require.NoError(t, err)
 
-	sub, err := daemon.natsConn.QueueSubscribe("ec2.DescribeStoppedInstances", "spinifex-workers", daemon.handleEC2DescribeStoppedInstances)
+	sub, err := daemon.natsConn.QueueSubscribe("ec2.DescribeStoppedInstances", "spinifex-workers", handleNATSRequest(daemon.instanceService.DescribeStoppedInstances))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -1280,7 +1300,7 @@ func TestHandleEC2DescribeStoppedInstances_WithFilter(t *testing.T) {
 		require.NoError(t, err)
 	}
 
-	sub, err := daemon.natsConn.QueueSubscribe("ec2.DescribeStoppedInstances", "spinifex-workers", daemon.handleEC2DescribeStoppedInstances)
+	sub, err := daemon.natsConn.QueueSubscribe("ec2.DescribeStoppedInstances", "spinifex-workers", handleNATSRequest(daemon.instanceService.DescribeStoppedInstances))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -1510,7 +1530,7 @@ func TestHandleEC2ModifyInstanceAttribute_WrapperRoundTrip(t *testing.T) {
 
 	daemon := createFullTestDaemonWithJetStream(t, natsURL)
 
-	sub, err := daemon.natsConn.QueueSubscribe("ec2.ModifyInstanceAttribute", "spinifex-workers", daemon.handleEC2ModifyInstanceAttribute)
+	sub, err := daemon.natsConn.QueueSubscribe("ec2.ModifyInstanceAttribute", "spinifex-workers", handleNATSRequest(daemon.instanceService.ModifyInstanceAttribute))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -1552,7 +1572,7 @@ func TestHandleEC2DescribeInstanceAttribute_RunningInstance_InstanceType(t *test
 
 	daemon := createFullTestDaemonWithJetStream(t, natsURL)
 
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", daemon.handleEC2DescribeInstanceAttribute)
+	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", handleNATSRequest(daemon.instanceService.DescribeInstanceAttribute))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -1594,7 +1614,7 @@ func TestHandleEC2DescribeInstanceAttribute_StoppedInstance_InstanceType(t *test
 
 	daemon := createFullTestDaemonWithJetStream(t, natsURL)
 
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", daemon.handleEC2DescribeInstanceAttribute)
+	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", handleNATSRequest(daemon.instanceService.DescribeInstanceAttribute))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -1634,7 +1654,7 @@ func TestHandleEC2DescribeInstanceAttribute_UserData(t *testing.T) {
 
 	daemon := createFullTestDaemonWithJetStream(t, natsURL)
 
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", daemon.handleEC2DescribeInstanceAttribute)
+	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", handleNATSRequest(daemon.instanceService.DescribeInstanceAttribute))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -1676,7 +1696,7 @@ func TestHandleEC2DescribeInstanceAttribute_DefaultAttribute_DisableApiTerminati
 
 	daemon := createFullTestDaemonWithJetStream(t, natsURL)
 
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", daemon.handleEC2DescribeInstanceAttribute)
+	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", handleNATSRequest(daemon.instanceService.DescribeInstanceAttribute))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -1712,7 +1732,7 @@ func TestHandleEC2DescribeInstanceAttribute_DefaultAttribute_DisableApiTerminati
 	err = json.Unmarshal(reply.Data, &output)
 	require.NoError(t, err)
 	require.NotNil(t, output.DisableApiTermination)
-	assert.Equal(t, true, *output.DisableApiTermination.Value)
+	assert.True(t, *output.DisableApiTermination.Value)
 }
 
 func TestHandleEC2DescribeInstanceAttribute_DefaultAttribute_ShutdownBehavior(t *testing.T) {
@@ -1720,7 +1740,7 @@ func TestHandleEC2DescribeInstanceAttribute_DefaultAttribute_ShutdownBehavior(t
 
 	daemon := createFullTestDaemonWithJetStream(t, natsURL)
 
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", daemon.handleEC2DescribeInstanceAttribute)
+	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", handleNATSRequest(daemon.instanceService.DescribeInstanceAttribute))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -1758,7 +1778,7 @@ func TestHandleEC2DescribeInstanceAttribute_GroupSet_WithSecurityGroups(t *testi
 
 	daemon := createFullTestDaemonWithJetStream(t, natsURL)
 
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", daemon.handleEC2DescribeInstanceAttribute)
+	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", handleNATSRequest(daemon.instanceService.DescribeInstanceAttribute))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -1804,7 +1824,7 @@ func TestHandleEC2DescribeInstanceAttribute_GroupSet_NilInstance(t *testing.T) {
 
 	daemon := createFullTestDaemonWithJetStream(t, natsURL)
 
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", daemon.handleEC2DescribeInstanceAttribute)
+	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", handleNATSRequest(daemon.instanceService.DescribeInstanceAttribute))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -1841,7 +1861,7 @@ func TestHandleEC2DescribeInstanceAttribute_InstanceNotFound(t *testing.T) {
 
 	daemon := createFullTestDaemonWithJetStream(t, natsURL)
 
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", daemon.handleEC2DescribeInstanceAttribute)
+	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", handleNATSRequest(daemon.instanceService.DescribeInstanceAttribute))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -1864,7 +1884,7 @@ func TestHandleEC2DescribeInstanceAttribute_UnsupportedAttribute(t *testing.T) {
 
 	daemon := createFullTestDaemonWithJetStream(t, natsURL)
 
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", daemon.handleEC2DescribeInstanceAttribute)
+	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", handleNATSRequest(daemon.instanceService.DescribeInstanceAttribute))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -1900,7 +1920,7 @@ func TestHandleEC2DescribeInstanceAttribute_InvalidJSON(t *testing.T) {
 
 	daemon := createFullTestDaemonWithJetStream(t, natsURL)
 
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", daemon.handleEC2DescribeInstanceAttribute)
+	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceAttribute", handleNATSRequest(daemon.instanceService.DescribeInstanceAttribute))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -1941,68 +1961,68 @@ func TestDelegateHandlers_RoundTrip(t *testing.T) {
 		{
 			name:    "DeleteKeyPair",
 			topic:   "ec2.test.DeleteKeyPair",
-			handler: daemon.handleEC2DeleteKeyPair,
+			handler: handleNATSRequest(daemon.keyService.DeleteKeyPair),
 			input:   &ec2.DeleteKeyPairInput{KeyName: aws.String("nonexistent-key")},
 			// Idempotent delete: missing key returns a success response.
 		},
 		{
 			name:         "ImportKeyPair",
 			topic:        "ec2.test.ImportKeyPair",
-			handler:      daemon.handleEC2ImportKeyPair,
+			handler:      handleNATSRequest(daemon.keyService.ImportKeyPair),
 			input:        &ec2.ImportKeyPairInput{KeyName: aws.String("imported-key"), PublicKeyMaterial: []byte("ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITest test@test")},
 			expectedCode: awserrors.ErrorInvalidKeyFormat,
 		},
 		{
 			name:         "CreateVolume",
 			topic:        "ec2.test.CreateVolume",
-			handler:      daemon.handleEC2CreateVolume,
+			handler:      handleNATSRequest(daemon.volumeService.CreateVolume),
 			input:        &ec2.CreateVolumeInput{AvailabilityZone: aws.String("us-east-1a"), Size: aws.Int64(10)},
 			expectedCode: awserrors.ErrorInvalidAvailabilityZone,
 		},
 		{
 			name:    "DescribeVolumeStatus",
 			topic:   "ec2.test.DescribeVolumeStatus",
-			handler: daemon.handleEC2DescribeVolumeStatus,
+			handler: handleNATSRequest(daemon.volumeService.DescribeVolumeStatus),
 			input:   &ec2.DescribeVolumeStatusInput{},
 		},
 		{
 			name:         "DeleteVolume",
 			topic:        "ec2.test.DeleteVolume",
-			handler:      daemon.handleEC2DeleteVolume,
+			handler:      handleNATSRequest(daemon.volumeService.DeleteVolume),
 			input:        &ec2.DeleteVolumeInput{VolumeId: aws.String("vol-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidVolumeNotFound,
 		},
 		{
 			name:         "CreateSnapshot",
 			topic:        "ec2.test.CreateSnapshot",
-			handler:      daemon.handleEC2CreateSnapshot,
+			handler:      handleNATSRequest(daemon.snapshotService.CreateSnapshot),
 			input:        &ec2.CreateSnapshotInput{VolumeId: aws.String("vol-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidVolumeNotFound,
 		},
 		{
 			name:    "DescribeSnapshots",
 			topic:   "ec2.test.DescribeSnapshots",
-			handler: daemon.handleEC2DescribeSnapshots,
+			handler: handleNATSRequest(daemon.snapshotService.DescribeSnapshots),
 			input:   &ec2.DescribeSnapshotsInput{},
 		},
 		{
 			name:         "DeleteSnapshot",
 			topic:        "ec2.test.DeleteSnapshot",
-			handler:      daemon.handleEC2DeleteSnapshot,
+			handler:      handleNATSRequest(daemon.snapshotService.DeleteSnapshot),
 			input:        &ec2.DeleteSnapshotInput{SnapshotId: aws.String("snap-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidSnapshotNotFound,
 		},
 		{
 			name:         "CopySnapshot",
 			topic:        "ec2.test.CopySnapshot",
-			handler:      daemon.handleEC2CopySnapshot,
+			handler:      handleNATSRequest(daemon.snapshotService.CopySnapshot),
 			input:        &ec2.CopySnapshotInput{SourceRegion: aws.String("us-east-1"), SourceSnapshotId: aws.String("snap-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidSnapshotNotFound,
 		},
 		{
 			name:    "DeleteTags",
 			topic:   "ec2.test.DeleteTags",
-			handler: daemon.handleEC2DeleteTags,
+			handler: handleNATSRequest(daemon.deleteTags),
 			// Instance IDs route to the owning daemon; with no owner
 			// subscribed the mutation is rejected rather than written blind.
 			input:        &ec2.DeleteTagsInput{Resources: []*string{aws.String("i-12345678")}},
@@ -2011,7 +2031,7 @@ func TestDelegateHandlers_RoundTrip(t *testing.T) {
 		{
 			name:    "DescribeTags",
 			topic:   "ec2.test.DescribeTags",
-			handler: daemon.handleEC2DescribeTags,
+			handler: handleNATSRequest(daemon.tagsService.DescribeTags),
 			input:   &ec2.DescribeTagsInput{},
 		},
 	}
@@ -2119,11 +2139,11 @@ func TestHandleNodeStatus(t *testing.T) {
 	assert.Equal(t, "us-west-2a", resp.AZ)
 	assert.GreaterOrEqual(t, resp.Uptime, int64(0))
 	assert.Equal(t, 2, resp.VMCount)
-	assert.Greater(t, resp.TotalVCPU, 0)
+	assert.Positive(t, resp.TotalVCPU)
 	assert.Greater(t, resp.TotalMemGB, 0.0)
 	assert.Equal(t, daemon.resourceMgr.reservedVCPU, resp.ReservedVCPU, "ReservedVCPU must be populated on wire")
 	assert.InDelta(t, daemon.resourceMgr.reservedMem, resp.ReservedMemGB, 0.001, "ReservedMemGB must be populated on wire")
-	assert.Greater(t, resp.ReservedVCPU, 0, "default reserve is non-zero")
+	assert.Positive(t, resp.ReservedVCPU, "default reserve is non-zero")
 	assert.Greater(t, resp.ReservedMemGB, 0.0, "default reserve is non-zero")
 }
 
@@ -2144,7 +2164,7 @@ func TestHandleNodeStatus_NoVMs(t *testing.T) {
 
 	assert.Equal(t, 0, resp.VMCount)
 	assert.Equal(t, "Ready", resp.Status)
-	assert.Greater(t, resp.ReservedVCPU, 0, "default reserve is exposed even when no VMs")
+	assert.Positive(t, resp.ReservedVCPU, "default reserve is exposed even when no VMs")
 	assert.Greater(t, resp.ReservedMemGB, 0.0, "default reserve is exposed even when no VMs")
 }
 
@@ -2195,7 +2215,7 @@ func TestHandleNodeVMs(t *testing.T) {
 	vm1 := vmsByID["i-vm-1"]
 	assert.Equal(t, "running", vm1.Status)
 	assert.Equal(t, instanceType, vm1.InstanceType)
-	assert.Greater(t, vm1.VCPU, 0)
+	assert.Positive(t, vm1.VCPU)
 	assert.Greater(t, vm1.MemoryGB, 0.0)
 	assert.Equal(t, launchTime.Unix(), vm1.LaunchTime)
 
@@ -2276,11 +2296,11 @@ func createVPCTestDaemon(t *testing.T) *Daemon {
 
 	testutil.StubVpcdSGResponder(t, nc)
 
-	vpcSvc, err := handlers_ec2_vpc.NewVPCServiceImplWithNATS(daemon.config, nc)
+	vpcSvc, err := handlers_ec2_vpc.NewVPCServiceImplWithNATS(t.Context(), daemon.config, nc)
 	require.NoError(t, err)
 	daemon.vpcService = vpcSvc
 
-	igwSvc, err := handlers_ec2_igw.NewIGWServiceImplWithNATS(daemon.config, nc)
+	igwSvc, err := handlers_ec2_igw.NewIGWServiceImplWithNATS(t.Context(), daemon.config, nc)
 	require.NoError(t, err)
 	daemon.igwService = igwSvc
 
@@ -2301,26 +2321,26 @@ func TestDelegateHandlers_VPC(t *testing.T) {
 		{
 			name:    "CreateVpc",
 			topic:   "ec2.test.CreateVpc",
-			handler: daemon.handleEC2CreateVpc,
+			handler: handleNATSRequest(daemon.vpcService.CreateVpc),
 			input:   &ec2.CreateVpcInput{CidrBlock: aws.String("10.0.0.0/16")},
 		},
 		{
 			name:         "DeleteVpc",
 			topic:        "ec2.test.DeleteVpc",
-			handler:      daemon.handleEC2DeleteVpc,
+			handler:      handleNATSRequest(daemon.vpcService.DeleteVpc),
 			input:        &ec2.DeleteVpcInput{VpcId: aws.String("vpc-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidVpcIDNotFound,
 		},
 		{
 			name:    "DescribeVpcs",
 			topic:   "ec2.test.DescribeVpcs",
-			handler: daemon.handleEC2DescribeVpcs,
+			handler: handleNATSRequest(daemon.vpcService.DescribeVpcs),
 			input:   &ec2.DescribeVpcsInput{},
 		},
 		{
 			name:    "CreateSubnet",
 			topic:   "ec2.test.CreateSubnet",
-			handler: daemon.handleEC2CreateSubnet,
+			handler: handleNATSRequest(daemon.vpcService.CreateSubnet),
 			input: &ec2.CreateSubnetInput{
 				VpcId:     aws.String("vpc-nonexistent"),
 				CidrBlock: aws.String("10.0.1.0/24"),
@@ -2330,34 +2350,34 @@ func TestDelegateHandlers_VPC(t *testing.T) {
 		{
 			name:         "DeleteSubnet",
 			topic:        "ec2.test.DeleteSubnet",
-			handler:      daemon.handleEC2DeleteSubnet,
+			handler:      handleNATSRequest(daemon.vpcService.DeleteSubnet),
 			input:        &ec2.DeleteSubnetInput{SubnetId: aws.String("subnet-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidSubnetIDNotFound,
 		},
 		{
 			name:    "DescribeSubnets",
 			topic:   "ec2.test.DescribeSubnets",
-			handler: daemon.handleEC2DescribeSubnets,
+			handler: handleNATSRequest(daemon.vpcService.DescribeSubnets),
 			input:   &ec2.DescribeSubnetsInput{},
 		},
 		{
 			name:         "CreateNetworkInterface",
 			topic:        "ec2.test.CreateNetworkInterface",
-			handler:      daemon.handleEC2CreateNetworkInterface,
+			handler:      handleNATSRequest(daemon.vpcService.CreateNetworkInterface),
 			input:        &ec2.CreateNetworkInterfaceInput{SubnetId: aws.String("subnet-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidSubnetIDNotFound,
 		},
 		{
 			name:         "DeleteNetworkInterface",
 			topic:        "ec2.test.DeleteNetworkInterface",
-			handler:      daemon.handleEC2DeleteNetworkInterface,
+			handler:      handleNATSRequest(daemon.vpcService.DeleteNetworkInterface),
 			input:        &ec2.DeleteNetworkInterfaceInput{NetworkInterfaceId: aws.String("eni-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidNetworkInterfaceIDNotFound,
 		},
 		{
 			name:    "DescribeNetworkInterfaces",
 			topic:   "ec2.test.DescribeNetworkInterfaces",
-			handler: daemon.handleEC2DescribeNetworkInterfaces,
+			handler: handleNATSRequest(daemon.vpcService.DescribeNetworkInterfaces),
 			input:   &ec2.DescribeNetworkInterfacesInput{},
 		},
 	}
@@ -2394,26 +2414,26 @@ func TestDelegateHandlers_IGW(t *testing.T) {
 		{
 			name:    "CreateInternetGateway",
 			topic:   "ec2.test.CreateInternetGateway",
-			handler: daemon.handleEC2CreateInternetGateway,
+			handler: handleNATSRequest(daemon.igwService.CreateInternetGateway),
 			input:   &ec2.CreateInternetGatewayInput{},
 		},
 		{
 			name:         "DeleteInternetGateway",
 			topic:        "ec2.test.DeleteInternetGateway",
-			handler:      daemon.handleEC2DeleteInternetGateway,
+			handler:      handleNATSRequest(daemon.igwService.DeleteInternetGateway),
 			input:        &ec2.DeleteInternetGatewayInput{InternetGatewayId: aws.String("igw-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidInternetGatewayIDNotFound,
 		},
 		{
 			name:    "DescribeInternetGateways",
 			topic:   "ec2.test.DescribeInternetGateways",
-			handler: daemon.handleEC2DescribeInternetGateways,
+			handler: handleNATSRequest(daemon.igwService.DescribeInternetGateways),
 			input:   &ec2.DescribeInternetGatewaysInput{},
 		},
 		{
 			name:    "AttachInternetGateway",
 			topic:   "ec2.test.AttachInternetGateway",
-			handler: daemon.handleEC2AttachInternetGateway,
+			handler: handleNATSRequest(daemon.igwService.AttachInternetGateway),
 			input: &ec2.AttachInternetGatewayInput{
 				InternetGatewayId: aws.String("igw-nonexistent"),
 				VpcId:             aws.String("vpc-nonexistent"),
@@ -2423,7 +2443,7 @@ func TestDelegateHandlers_IGW(t *testing.T) {
 		{
 			name:    "DetachInternetGateway",
 			topic:   "ec2.test.DetachInternetGateway",
-			handler: daemon.handleEC2DetachInternetGateway,
+			handler: handleNATSRequest(daemon.igwService.DetachInternetGateway),
 			input: &ec2.DetachInternetGatewayInput{
 				InternetGatewayId: aws.String("igw-nonexistent"),
 				VpcId:             aws.String("vpc-nonexistent"),
@@ -2453,7 +2473,7 @@ func TestDelegateHandlers_IGW(t *testing.T) {
 func TestHandleEC2CreateVpc_SuccessPath(t *testing.T) {
 	daemon := createVPCTestDaemon(t)
 
-	sub, err := daemon.natsConn.QueueSubscribe("ec2.CreateVpc", "spinifex-workers", daemon.handleEC2CreateVpc)
+	sub, err := daemon.natsConn.QueueSubscribe("ec2.CreateVpc", "spinifex-workers", handleNATSRequest(daemon.vpcService.CreateVpc))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -2474,11 +2494,11 @@ func TestHandleEC2CreateVpc_SuccessPath(t *testing.T) {
 func TestHandleEC2CreateAndDescribeVpc_RoundTrip(t *testing.T) {
 	daemon := createVPCTestDaemon(t)
 
-	createSub, err := daemon.natsConn.QueueSubscribe("ec2.CreateVpc", "spinifex-workers", daemon.handleEC2CreateVpc)
+	createSub, err := daemon.natsConn.QueueSubscribe("ec2.CreateVpc", "spinifex-workers", handleNATSRequest(daemon.vpcService.CreateVpc))
 	require.NoError(t, err)
 	defer createSub.Unsubscribe()
 
-	describeSub, err := daemon.natsConn.QueueSubscribe("ec2.DescribeVpcs", "spinifex-workers", daemon.handleEC2DescribeVpcs)
+	describeSub, err := daemon.natsConn.QueueSubscribe("ec2.DescribeVpcs", "spinifex-workers", handleNATSRequest(daemon.vpcService.DescribeVpcs))
 	require.NoError(t, err)
 	defer describeSub.Unsubscribe()
 
@@ -2514,7 +2534,7 @@ func TestHandleEC2CreateAndDescribeVpc_RoundTrip(t *testing.T) {
 func TestHandleEC2CreateInternetGateway_SuccessPath(t *testing.T) {
 	daemon := createVPCTestDaemon(t)
 
-	sub, err := daemon.natsConn.QueueSubscribe("ec2.CreateInternetGateway", "spinifex-workers", daemon.handleEC2CreateInternetGateway)
+	sub, err := daemon.natsConn.QueueSubscribe("ec2.CreateInternetGateway", "spinifex-workers", handleNATSRequest(daemon.igwService.CreateInternetGateway))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -2528,17 +2548,17 @@ func TestHandleEC2CreateInternetGateway_SuccessPath(t *testing.T) {
 	require.NoError(t, err)
 	require.NotNil(t, output.InternetGateway)
 	assert.NotEmpty(t, *output.InternetGateway.InternetGatewayId)
-	assert.True(t, len(*output.InternetGateway.InternetGatewayId) > 4)
+	assert.Greater(t, len(*output.InternetGateway.InternetGatewayId), 4)
 }
 
 func TestHandleEC2CreateSubnet_SuccessPath(t *testing.T) {
 	daemon := createVPCTestDaemon(t)
 
-	createVpcSub, err := daemon.natsConn.QueueSubscribe("ec2.CreateVpc", "spinifex-workers", daemon.handleEC2CreateVpc)
+	createVpcSub, err := daemon.natsConn.QueueSubscribe("ec2.CreateVpc", "spinifex-workers", handleNATSRequest(daemon.vpcService.CreateVpc))
 	require.NoError(t, err)
 	defer createVpcSub.Unsubscribe()
 
-	createSubnetSub, err := daemon.natsConn.QueueSubscribe("ec2.CreateSubnet", "spinifex-workers", daemon.handleEC2CreateSubnet)
+	createSubnetSub, err := daemon.natsConn.QueueSubscribe("ec2.CreateSubnet", "spinifex-workers", handleNATSRequest(daemon.vpcService.CreateSubnet))
 	require.NoError(t, err)
 	defer createSubnetSub.Unsubscribe()
 
@@ -2592,7 +2612,7 @@ func TestDelegateHandlers_EIGW(t *testing.T) {
 	require.NoError(t, err)
 	t.Cleanup(func() { nc.Close() })
 
-	eigwSvc, err := handlers_ec2_eigw.NewEgressOnlyIGWServiceImplWithNATS(daemon.config, nc)
+	eigwSvc, err := handlers_ec2_eigw.NewEgressOnlyIGWServiceImplWithNATS(t.Context(), daemon.config, nc)
 	require.NoError(t, err)
 	daemon.eigwService = eigwSvc
 
@@ -2607,21 +2627,21 @@ func TestDelegateHandlers_EIGW(t *testing.T) {
 		{
 			name:         "CreateEgressOnlyInternetGateway",
 			topic:        "ec2.test.CreateEgressOnlyIGW",
-			handler:      daemon.handleEC2CreateEgressOnlyInternetGateway,
+			handler:      handleNATSRequest(daemon.eigwService.CreateEgressOnlyInternetGateway),
 			input:        &ec2.CreateEgressOnlyInternetGatewayInput{VpcId: aws.String("vpc-123")},
 			expectedCode: awserrors.ErrorInvalidVpcIDNotFound,
 		},
 		{
 			name:         "DeleteEgressOnlyInternetGateway",
 			topic:        "ec2.test.DeleteEgressOnlyIGW",
-			handler:      daemon.handleEC2DeleteEgressOnlyInternetGateway,
+			handler:      handleNATSRequest(daemon.eigwService.DeleteEgressOnlyInternetGateway),
 			input:        &ec2.DeleteEgressOnlyInternetGatewayInput{EgressOnlyInternetGatewayId: aws.String("eigw-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidEgressOnlyInternetGatewayIdNotFound,
 		},
 		{
 			name:    "DescribeEgressOnlyInternetGateways",
 			topic:   "ec2.test.DescribeEgressOnlyIGWs",
-			handler: daemon.handleEC2DescribeEgressOnlyInternetGateways,
+			handler: handleNATSRequest(daemon.eigwService.DescribeEgressOnlyInternetGateways),
 			input:   &ec2.DescribeEgressOnlyInternetGatewaysInput{},
 		},
 	}
@@ -2704,7 +2724,7 @@ func TestHandleEC2ModifyVolume_Success(t *testing.T) {
 func TestHandleEC2DescribeInstanceTypes_CapacityFilter(t *testing.T) {
 	daemon := createFullTestDaemon(t, sharedNATSURL)
 
-	sub, err := daemon.natsConn.QueueSubscribe("ec2.DescribeInstanceTypes", "spinifex-workers", daemon.handleEC2DescribeInstanceTypes)
+	sub, err := daemon.natsConn.QueueSubscribe("ec2.DescribeInstanceTypes", "spinifex-workers", handleNATSRequest(daemon.instanceService.DescribeInstanceTypes))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -2731,7 +2751,7 @@ func TestHandleEC2DescribeInstanceTypes_CapacityFilter(t *testing.T) {
 func TestHandleEC2DescribeInstanceTypes_NoFilter(t *testing.T) {
 	daemon := createFullTestDaemon(t, sharedNATSURL)
 
-	sub, err := daemon.natsConn.QueueSubscribe("ec2.DescribeInstanceTypes.nofilter", "spinifex-workers", daemon.handleEC2DescribeInstanceTypes)
+	sub, err := daemon.natsConn.QueueSubscribe("ec2.DescribeInstanceTypes.nofilter", "spinifex-workers", handleNATSRequest(daemon.instanceService.DescribeInstanceTypes))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -2744,7 +2764,7 @@ func TestHandleEC2DescribeInstanceTypes_NoFilter(t *testing.T) {
 	err = json.Unmarshal(reply.Data, &output)
 	require.NoError(t, err)
 	assert.NotNil(t, output.InstanceTypes)
-	assert.Greater(t, len(output.InstanceTypes), 0)
+	assert.NotEmpty(t, output.InstanceTypes)
 }
 
 // --- handleEC2StartStoppedInstance: instance type not available ---
@@ -3321,7 +3341,7 @@ func TestHandleEC2RunInstances_MalformedInput(t *testing.T) {
 func TestHandleEC2DescribeInstances_MalformedInstanceID(t *testing.T) {
 	daemon := createFullTestDaemon(t, sharedNATSURL)
 
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstances.malformed", daemon.handleEC2DescribeInstances)
+	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstances.malformed", handleNATSRequest(daemon.instanceService.DescribeInstances))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -3353,7 +3373,7 @@ func TestHandleEC2DescribeInstanceStatus_RoundTrip(t *testing.T) {
 	daemon.vmMgr.Insert(runningVM)
 	t.Cleanup(func() { daemon.vmMgr.Delete(runningVM.ID) })
 
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceStatus.rt", daemon.handleEC2DescribeInstanceStatus)
+	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceStatus.rt", handleNATSRequest(daemon.instanceService.DescribeInstanceStatus))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -3379,7 +3399,7 @@ func TestHandleEC2DescribeInstanceStatus_RoundTrip(t *testing.T) {
 func TestHandleEC2DescribeInstanceStatus_MalformedInstanceID(t *testing.T) {
 	daemon := createFullTestDaemon(t, sharedNATSURL)
 
-	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceStatus.malformed", daemon.handleEC2DescribeInstanceStatus)
+	sub, err := daemon.natsConn.Subscribe("ec2.DescribeInstanceStatus.malformed", handleNATSRequest(daemon.instanceService.DescribeInstanceStatus))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -3414,7 +3434,7 @@ func TestHandleEC2DescribeTerminatedInstances_ReturnsTerminatedInstances(t *test
 	err := daemon.jsManager.WriteTerminatedInstance(terminatedVM.ID, terminatedVM)
 	require.NoError(t, err)
 
-	sub, err := daemon.natsConn.QueueSubscribe("ec2.DescribeTerminatedInstances", "spinifex-workers", daemon.handleEC2DescribeTerminatedInstances)
+	sub, err := daemon.natsConn.QueueSubscribe("ec2.DescribeTerminatedInstances", "spinifex-workers", handleNATSRequest(daemon.instanceService.DescribeTerminatedInstances))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -3463,7 +3483,7 @@ func TestHandleEC2DescribeTerminatedInstances_WithFilter(t *testing.T) {
 		require.NoError(t, daemon.jsManager.WriteTerminatedInstance(v.ID, v))
 	}
 
-	sub, err := daemon.natsConn.QueueSubscribe("ec2.DescribeTerminatedInstances", "spinifex-workers", daemon.handleEC2DescribeTerminatedInstances)
+	sub, err := daemon.natsConn.QueueSubscribe("ec2.DescribeTerminatedInstances", "spinifex-workers", handleNATSRequest(daemon.instanceService.DescribeTerminatedInstances))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -3499,7 +3519,7 @@ func TestHandleEC2TerminateStoppedInstance_WritesToTerminatedKV(t *testing.T) {
 	}
 	require.NoError(t, daemon.jsManager.WriteStoppedInstance(stoppedVM.ID, stoppedVM))
 
-	sub, err := daemon.natsConn.QueueSubscribe("ec2.terminate", "spinifex-workers", daemon.handleEC2TerminateStoppedInstance)
+	sub, err := daemon.natsConn.QueueSubscribe("ec2.terminate", "spinifex-workers", handleNATSRequest(daemon.instanceService.TerminateStoppedInstance))
 	require.NoError(t, err)
 	defer sub.Unsubscribe()
 
@@ -3544,14 +3564,14 @@ func TestDelegateHandlers_EIP(t *testing.T) {
 	require.NoError(t, err)
 	t.Cleanup(func() { nc.Close() })
 
-	js, err := nc.JetStream()
+	js, err := jetstream.New(nc)
 	require.NoError(t, err)
-	ipam, err := handlers_ec2_vpc.NewExternalIPAM(js, []external.ExternalPoolConfig{
+	ipam, err := handlers_ec2_vpc.NewExternalIPAM(t.Context(), js, []external.ExternalPoolConfig{
 		{Name: "test-pool", RangeStart: "192.168.100.2", RangeEnd: "192.168.100.254", Gateway: "192.168.100.1", PrefixLen: 24},
 	})
 	require.NoError(t, err)
 
-	eipSvc, err := handlers_ec2_eip.NewEIPServiceImpl(nc, ipam, daemon.vpcService)
+	eipSvc, err := handlers_ec2_eip.NewEIPServiceImpl(t.Context(), nc, ipam, daemon.vpcService)
 	require.NoError(t, err)
 	daemon.eipService = eipSvc
 
@@ -3566,34 +3586,34 @@ func TestDelegateHandlers_EIP(t *testing.T) {
 		{
 			name:    "AllocateAddress",
 			topic:   "ec2.test.AllocateAddress",
-			handler: daemon.handleEC2AllocateAddress,
+			handler: handleNATSRequest(daemon.eipService.AllocateAddress),
 			input:   &ec2.AllocateAddressInput{},
 		},
 		{
 			name:         "ReleaseAddress",
 			topic:        "ec2.test.ReleaseAddress",
-			handler:      daemon.handleEC2ReleaseAddress,
+			handler:      handleNATSRequest(daemon.eipService.ReleaseAddress),
 			input:        &ec2.ReleaseAddressInput{AllocationId: aws.String("eipalloc-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidAllocationIDNotFound,
 		},
 		{
 			name:         "AssociateAddress",
 			topic:        "ec2.test.AssociateAddress",
-			handler:      daemon.handleEC2AssociateAddress,
+			handler:      handleNATSRequest(daemon.eipService.AssociateAddress),
 			input:        &ec2.AssociateAddressInput{AllocationId: aws.String("eipalloc-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidAllocationIDNotFound,
 		},
 		{
 			name:         "DisassociateAddress",
 			topic:        "ec2.test.DisassociateAddress",
-			handler:      daemon.handleEC2DisassociateAddress,
+			handler:      handleNATSRequest(daemon.eipService.DisassociateAddress),
 			input:        &ec2.DisassociateAddressInput{AssociationId: aws.String("eipassoc-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidAssociationIDNotFound,
 		},
 		{
 			name:    "DescribeAddresses",
 			topic:   "ec2.test.DescribeAddresses",
-			handler: daemon.handleEC2DescribeAddresses,
+			handler: handleNATSRequest(daemon.eipService.DescribeAddresses),
 			input:   &ec2.DescribeAddressesInput{},
 		},
 	}
@@ -3622,7 +3642,7 @@ func TestDelegateHandlers_SecurityGroup(t *testing.T) {
 	daemon := createVPCTestDaemon(t)
 
 	// Create a VPC first so SG operations have a valid target
-	createVpcSub, err := daemon.natsConn.QueueSubscribe("ec2.CreateVpc", "spinifex-workers", daemon.handleEC2CreateVpc)
+	createVpcSub, err := daemon.natsConn.QueueSubscribe("ec2.CreateVpc", "spinifex-workers", handleNATSRequest(daemon.vpcService.CreateVpc))
 	require.NoError(t, err)
 	defer createVpcSub.Unsubscribe()
 
@@ -3646,7 +3666,7 @@ func TestDelegateHandlers_SecurityGroup(t *testing.T) {
 		{
 			name:    "CreateSecurityGroup",
 			topic:   "ec2.test.CreateSecurityGroup",
-			handler: daemon.handleEC2CreateSecurityGroup,
+			handler: handleNATSRequest(daemon.vpcService.CreateSecurityGroup),
 			input: &ec2.CreateSecurityGroupInput{
 				GroupName:   aws.String("test-sg"),
 				Description: aws.String("test security group"),
@@ -3656,41 +3676,41 @@ func TestDelegateHandlers_SecurityGroup(t *testing.T) {
 		{
 			name:    "DescribeSecurityGroups",
 			topic:   "ec2.test.DescribeSecurityGroups",
-			handler: daemon.handleEC2DescribeSecurityGroups,
+			handler: handleNATSRequest(daemon.vpcService.DescribeSecurityGroups),
 			input:   &ec2.DescribeSecurityGroupsInput{},
 		},
 		{
 			name:         "AuthorizeSecurityGroupIngress",
 			topic:        "ec2.test.AuthorizeSecurityGroupIngress",
-			handler:      daemon.handleEC2AuthorizeSecurityGroupIngress,
+			handler:      handleNATSRequest(daemon.vpcService.AuthorizeSecurityGroupIngress),
 			input:        &ec2.AuthorizeSecurityGroupIngressInput{GroupId: aws.String("sg-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidGroupNotFound,
 		},
 		{
 			name:         "AuthorizeSecurityGroupEgress",
 			topic:        "ec2.test.AuthorizeSecurityGroupEgress",
-			handler:      daemon.handleEC2AuthorizeSecurityGroupEgress,
+			handler:      handleNATSRequest(daemon.vpcService.AuthorizeSecurityGroupEgress),
 			input:        &ec2.AuthorizeSecurityGroupEgressInput{GroupId: aws.String("sg-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidGroupNotFound,
 		},
 		{
 			name:         "RevokeSecurityGroupIngress",
 			topic:        "ec2.test.RevokeSecurityGroupIngress",
-			handler:      daemon.handleEC2RevokeSecurityGroupIngress,
+			handler:      handleNATSRequest(daemon.vpcService.RevokeSecurityGroupIngress),
 			input:        &ec2.RevokeSecurityGroupIngressInput{GroupId: aws.String("sg-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidGroupNotFound,
 		},
 		{
 			name:         "RevokeSecurityGroupEgress",
 			topic:        "ec2.test.RevokeSecurityGroupEgress",
-			handler:      daemon.handleEC2RevokeSecurityGroupEgress,
+			handler:      handleNATSRequest(daemon.vpcService.RevokeSecurityGroupEgress),
 			input:        &ec2.RevokeSecurityGroupEgressInput{GroupId: aws.String("sg-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidGroupNotFound,
 		},
 		{
 			name:         "DeleteSecurityGroup",
 			topic:        "ec2.test.DeleteSecurityGroup",
-			handler:      daemon.handleEC2DeleteSecurityGroup,
+			handler:      handleNATSRequest(daemon.vpcService.DeleteSecurityGroup),
 			input:        &ec2.DeleteSecurityGroupInput{GroupId: aws.String("sg-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidGroupNotFound,
 		},
@@ -3737,7 +3757,7 @@ func TestDelegateHandlers_RouteTable(t *testing.T) {
 	require.NoError(t, err)
 	t.Cleanup(func() { nc.Close() })
 
-	rtbSvc, err := handlers_ec2_routetable.NewRouteTableServiceImplWithNATS(daemon.config, nc)
+	rtbSvc, err := handlers_ec2_routetable.NewRouteTableServiceImplWithNATS(t.Context(), daemon.config, nc)
 	require.NoError(t, err)
 	daemon.routeTableService = rtbSvc
 
@@ -3752,27 +3772,27 @@ func TestDelegateHandlers_RouteTable(t *testing.T) {
 		{
 			name:         "CreateRouteTable",
 			topic:        "ec2.test.CreateRouteTable",
-			handler:      daemon.handleEC2CreateRouteTable,
+			handler:      handleNATSRequest(daemon.routeTableService.CreateRouteTable),
 			input:        &ec2.CreateRouteTableInput{VpcId: aws.String("vpc-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidVpcIDNotFound,
 		},
 		{
 			name:         "DeleteRouteTable",
 			topic:        "ec2.test.DeleteRouteTable",
-			handler:      daemon.handleEC2DeleteRouteTable,
+			handler:      handleNATSRequest(daemon.routeTableService.DeleteRouteTable),
 			input:        &ec2.DeleteRouteTableInput{RouteTableId: aws.String("rtb-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidRouteTableIDNotFound,
 		},
 		{
 			name:    "DescribeRouteTables",
 			topic:   "ec2.test.DescribeRouteTables",
-			handler: daemon.handleEC2DescribeRouteTables,
+			handler: handleNATSRequest(daemon.routeTableService.DescribeRouteTables),
 			input:   &ec2.DescribeRouteTablesInput{},
 		},
 		{
 			name:    "CreateRoute",
 			topic:   "ec2.test.CreateRoute",
-			handler: daemon.handleEC2CreateRoute,
+			handler: handleNATSRequest(daemon.routeTableService.CreateRoute),
 			input: &ec2.CreateRouteInput{
 				RouteTableId:         aws.String("rtb-nonexistent"),
 				DestinationCidrBlock: aws.String("0.0.0.0/0"),
@@ -3783,7 +3803,7 @@ func TestDelegateHandlers_RouteTable(t *testing.T) {
 		{
 			name:    "DeleteRoute",
 			topic:   "ec2.test.DeleteRoute",
-			handler: daemon.handleEC2DeleteRoute,
+			handler: handleNATSRequest(daemon.routeTableService.DeleteRoute),
 			input: &ec2.DeleteRouteInput{
 				RouteTableId:         aws.String("rtb-nonexistent"),
 				DestinationCidrBlock: aws.String("0.0.0.0/0"),
@@ -3793,7 +3813,7 @@ func TestDelegateHandlers_RouteTable(t *testing.T) {
 		{
 			name:    "ReplaceRoute",
 			topic:   "ec2.test.ReplaceRoute",
-			handler: daemon.handleEC2ReplaceRoute,
+			handler: handleNATSRequest(daemon.routeTableService.ReplaceRoute),
 			input: &ec2.ReplaceRouteInput{
 				RouteTableId:         aws.String("rtb-nonexistent"),
 				DestinationCidrBlock: aws.String("0.0.0.0/0"),
@@ -3804,7 +3824,7 @@ func TestDelegateHandlers_RouteTable(t *testing.T) {
 		{
 			name:    "AssociateRouteTable",
 			topic:   "ec2.test.AssociateRouteTable",
-			handler: daemon.handleEC2AssociateRouteTable,
+			handler: handleNATSRequest(daemon.routeTableService.AssociateRouteTable),
 			input: &ec2.AssociateRouteTableInput{
 				RouteTableId: aws.String("rtb-nonexistent"),
 				SubnetId:     aws.String("subnet-nonexistent"),
@@ -3814,14 +3834,14 @@ func TestDelegateHandlers_RouteTable(t *testing.T) {
 		{
 			name:         "DisassociateRouteTable",
 			topic:        "ec2.test.DisassociateRouteTable",
-			handler:      daemon.handleEC2DisassociateRouteTable,
+			handler:      handleNATSRequest(daemon.routeTableService.DisassociateRouteTable),
 			input:        &ec2.DisassociateRouteTableInput{AssociationId: aws.String("rtbassoc-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidAssociationIDNotFound,
 		},
 		{
 			name:    "ReplaceRouteTableAssociation",
 			topic:   "ec2.test.ReplaceRouteTableAssociation",
-			handler: daemon.handleEC2ReplaceRouteTableAssociation,
+			handler: handleNATSRequest(daemon.routeTableService.ReplaceRouteTableAssociation),
 			input: &ec2.ReplaceRouteTableAssociationInput{
 				AssociationId: aws.String("rtbassoc-nonexistent"),
 				RouteTableId:  aws.String("rtb-nonexistent"),
@@ -3870,7 +3890,7 @@ func TestDelegateHandlers_PlacementGroup(t *testing.T) {
 	require.NoError(t, err)
 	t.Cleanup(func() { nc.Close() })
 
-	pgSvc, err := handlers_ec2_placementgroup.NewPlacementGroupServiceImplWithNATS(daemon.config, nc)
+	pgSvc, err := handlers_ec2_placementgroup.NewPlacementGroupServiceImplWithNATS(t.Context(), daemon.config, nc)
 	require.NoError(t, err)
 	daemon.placementGroupService = pgSvc
 
@@ -3885,7 +3905,7 @@ func TestDelegateHandlers_PlacementGroup(t *testing.T) {
 		{
 			name:    "CreatePlacementGroup",
 			topic:   "ec2.test.CreatePlacementGroup",
-			handler: daemon.handleEC2CreatePlacementGroup,
+			handler: handleNATSRequest(daemon.placementGroupService.CreatePlacementGroup),
 			input: &ec2.CreatePlacementGroupInput{
 				GroupName: aws.String("test-pg"),
 				Strategy:  aws.String("spread"),
@@ -3894,20 +3914,20 @@ func TestDelegateHandlers_PlacementGroup(t *testing.T) {
 		{
 			name:    "DescribePlacementGroups",
 			topic:   "ec2.test.DescribePlacementGroups",
-			handler: daemon.handleEC2DescribePlacementGroups,
+			handler: handleNATSRequest(daemon.placementGroupService.DescribePlacementGroups),
 			input:   &ec2.DescribePlacementGroupsInput{},
 		},
 		{
 			name:         "DeletePlacementGroup",
 			topic:        "ec2.test.DeletePlacementGroup",
-			handler:      daemon.handleEC2DeletePlacementGroup,
+			handler:      handleNATSRequest(daemon.placementGroupService.DeletePlacementGroup),
 			input:        &ec2.DeletePlacementGroupInput{GroupName: aws.String("pg-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidPlacementGroupUnknown,
 		},
 		{
 			name:    "ReserveSpreadNodes",
 			topic:   "ec2.test.ReserveSpreadNodes",
-			handler: daemon.handleEC2ReserveSpreadNodes,
+			handler: handleNATSRequest(daemon.placementGroupService.ReserveSpreadNodes),
 			input: &handlers_ec2_placementgroup.ReserveSpreadNodesInput{
 				GroupName:     "pg-nonexistent",
 				EligibleNodes: []string{"node-1"},
@@ -3919,7 +3939,7 @@ func TestDelegateHandlers_PlacementGroup(t *testing.T) {
 		{
 			name:    "FinalizeSpreadInstances",
 			topic:   "ec2.test.FinalizeSpreadInstances",
-			handler: daemon.handleEC2FinalizeSpreadInstances,
+			handler: handleNATSRequest(daemon.placementGroupService.FinalizeSpreadInstances),
 			input: &handlers_ec2_placementgroup.FinalizeSpreadInstancesInput{
 				GroupName:     "pg-nonexistent",
 				NodeInstances: map[string][]string{"node-1": {"i-123"}},
@@ -3929,7 +3949,7 @@ func TestDelegateHandlers_PlacementGroup(t *testing.T) {
 		{
 			name:    "ReleaseSpreadNodes",
 			topic:   "ec2.test.ReleaseSpreadNodes",
-			handler: daemon.handleEC2ReleaseSpreadNodes,
+			handler: handleNATSRequest(daemon.placementGroupService.ReleaseSpreadNodes),
 			input: &handlers_ec2_placementgroup.ReleaseSpreadNodesInput{
 				GroupName: "pg-nonexistent",
 				Nodes:     []string{"node-1"},
@@ -3939,7 +3959,7 @@ func TestDelegateHandlers_PlacementGroup(t *testing.T) {
 		{
 			name:    "RemoveInstanceFromPlacementGroup",
 			topic:   "ec2.test.RemoveInstanceFromPlacementGroup",
-			handler: daemon.handleEC2RemoveInstanceFromPlacementGroup,
+			handler: handleNATSRequest(daemon.placementGroupService.RemoveInstance),
 			input: &handlers_ec2_placementgroup.RemoveInstanceInput{
 				GroupName:  "pg-nonexistent",
 				NodeName:   "node-1",
@@ -3952,7 +3972,7 @@ func TestDelegateHandlers_PlacementGroup(t *testing.T) {
 		{
 			name:    "ReserveClusterNode",
 			topic:   "ec2.test.ReserveClusterNode",
-			handler: daemon.handleEC2ReserveClusterNode,
+			handler: handleNATSRequest(daemon.placementGroupService.ReserveClusterNode),
 			input: &handlers_ec2_placementgroup.ReserveClusterNodeInput{
 				GroupName:     "pg-nonexistent",
 				EligibleNodes: []string{"node-1"},
@@ -3962,7 +3982,7 @@ func TestDelegateHandlers_PlacementGroup(t *testing.T) {
 		{
 			name:    "FinalizeClusterInstances",
 			topic:   "ec2.test.FinalizeClusterInstances",
-			handler: daemon.handleEC2FinalizeClusterInstances,
+			handler: handleNATSRequest(daemon.placementGroupService.FinalizeClusterInstances),
 			input: &handlers_ec2_placementgroup.FinalizeClusterInstancesInput{
 				GroupName:     "pg-nonexistent",
 				NodeInstances: map[string][]string{"node-1": {"i-123"}},
@@ -3995,7 +4015,7 @@ func TestDelegateHandlers_VPCAttributes(t *testing.T) {
 	daemon := createVPCTestDaemon(t)
 
 	// Create a VPC first
-	createVpcSub, err := daemon.natsConn.QueueSubscribe("ec2.CreateVpc", "spinifex-workers", daemon.handleEC2CreateVpc)
+	createVpcSub, err := daemon.natsConn.QueueSubscribe("ec2.CreateVpc", "spinifex-workers", handleNATSRequest(daemon.vpcService.CreateVpc))
 	require.NoError(t, err)
 	defer createVpcSub.Unsubscribe()
 
@@ -4019,7 +4039,7 @@ func TestDelegateHandlers_VPCAttributes(t *testing.T) {
 		{
 			name:         "ModifySubnetAttribute",
 			topic:        "ec2.test.ModifySubnetAttribute",
-			handler:      daemon.handleEC2ModifySubnetAttribute,
+			handler:      handleNATSRequest(daemon.vpcService.ModifySubnetAttribute),
 			input:        &ec2.ModifySubnetAttributeInput{SubnetId: aws.String("subnet-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidSubnetIDNotFound,
 		},
@@ -4029,14 +4049,14 @@ func TestDelegateHandlers_VPCAttributes(t *testing.T) {
 			// the validation contract.
 			name:         "ModifyVpcAttribute",
 			topic:        "ec2.test.ModifyVpcAttribute",
-			handler:      daemon.handleEC2ModifyVpcAttribute,
+			handler:      handleNATSRequest(daemon.vpcService.ModifyVpcAttribute),
 			input:        &ec2.ModifyVpcAttributeInput{VpcId: aws.String(vpcID)},
 			expectedCode: awserrors.ErrorInvalidParameterValue,
 		},
 		{
 			name:    "DescribeVpcAttribute",
 			topic:   "ec2.test.DescribeVpcAttribute",
-			handler: daemon.handleEC2DescribeVpcAttribute,
+			handler: handleNATSRequest(daemon.vpcService.DescribeVpcAttribute),
 			input: &ec2.DescribeVpcAttributeInput{
 				VpcId:     aws.String(vpcID),
 				Attribute: aws.String("enableDnsSupport"),
@@ -4047,7 +4067,7 @@ func TestDelegateHandlers_VPCAttributes(t *testing.T) {
 			// "exactly one attribute" guard before the ENI lookup runs.
 			name:         "ModifyNetworkInterfaceAttribute",
 			topic:        "ec2.test.ModifyNetworkInterfaceAttribute",
-			handler:      daemon.handleEC2ModifyNetworkInterfaceAttribute,
+			handler:      handleNATSRequest(daemon.vpcService.ModifyNetworkInterfaceAttribute),
 			input:        &ec2.ModifyNetworkInterfaceAttributeInput{NetworkInterfaceId: aws.String("eni-nonexistent")},
 			expectedCode: awserrors.ErrorInvalidParameterValue,
 		},

spinifex/daemon/daemon_handlers_volume.go

@@ -9,6 +9,7 @@ import (
 
 	"github.com/aws/aws-sdk-go/service/ec2"
 	"github.com/mulgadc/spinifex/spinifex/awserrors"
+	handlers_ec2_snapshot "github.com/mulgadc/spinifex/spinifex/handlers/ec2/snapshot"
 	"github.com/mulgadc/spinifex/spinifex/types"
 	"github.com/mulgadc/spinifex/spinifex/utils"
 	"github.com/mulgadc/spinifex/spinifex/vm"
@@ -139,6 +140,51 @@ func (d *Daemon) handleDetachVolume(ctx context.Context, msg *nats.Msg, command
 	d.respondWithVolumeAttachment(msg, command.DetachVolumeData.VolumeID, command.ID, deviceName, "detaching")
 }
 
+// handleDrainVolume flushes the volume's in-flight writes to S3 by dialing the
+// drain socket the local NBD plugin serves. ec2.CreateSnapshot is answered by
+// any node in the cluster, but only the node hosting the instance subscribes
+// ec2.cmd.{instanceID}, so routing the drain here lands it where the writes are.
+//
+// Runs on its own goroutine (see handleEC2Events) so a long flush cannot hold
+// the instance's command subscription.
+func (d *Daemon) handleDrainVolume(ctx context.Context, msg *nats.Msg, command types.EC2InstanceCommand, instance *vm.VM) {
+	ctx, span := startOpSpan(ctx, "ec2.DrainVolume", command.ID)
+	var err error
+	defer func() { endOpSpan(span, err) }()
+
+	if command.DrainVolumeData == nil || command.DrainVolumeData.VolumeID == "" {
+		slog.ErrorContext(ctx, "DrainVolume: missing drain volume data", "instanceId", command.ID)
+		respondWithError(msg, awserrors.ErrorInvalidParameterValue)
+		return
+	}
+
+	volumeID := command.DrainVolumeData.VolumeID
+
+	// Only stable post-teardown states prove the volume was sealed. Transitional
+	// states still have a live or shutting-down writer, so they must drain or
+	// fail rather than acknowledge an older checkpoint as current.
+	status := d.vmMgr.Status(instance)
+	if status == vm.StateStopped || status == vm.StateTerminated {
+		slog.InfoContext(ctx, "DrainVolume: instance teardown is complete, nothing to drain",
+			"volumeId", volumeID, "instanceId", command.ID, "status", status)
+		respondWithJSON(msg, types.DrainVolumeResponse{VolumeID: volumeID, Status: types.DrainVolumeStatusNotRunning})
+		return
+	}
+
+	slog.InfoContext(ctx, "Draining volume before snapshot", "volumeId", volumeID, "instanceId", command.ID)
+
+	// A missing or unresponsive socket under a running instance means the writes
+	// cannot be made current, so the caller must fail its snapshot rather than
+	// read a stale checkpoint.
+	if err = handlers_ec2_snapshot.DrainVolumeSocket(d.config.DataDir, volumeID); err != nil {
+		slog.ErrorContext(ctx, "DrainVolume: drain failed", "volumeId", volumeID, "instanceId", command.ID, "err", err)
+		respondWithError(msg, awserrors.ErrorServerInternal)
+		return
+	}
+
+	respondWithJSON(msg, types.DrainVolumeResponse{VolumeID: volumeID, Status: types.DrainVolumeStatusDrained})
+}
+
 // attachDetachErrorCode maps a vm.Manager error returned by AttachVolume
 // or DetachVolume to the AWS API error code that the SDK expects.
 func attachDetachErrorCode(err error) string {
@@ -160,23 +206,7 @@ func attachDetachErrorCode(err error) string {
 	}
 }
 
-func (d *Daemon) handleEC2CreateVolume(msg *nats.Msg) {
-	handleNATSRequest(msg, d.volumeService.CreateVolume)
-}
-
-func (d *Daemon) handleEC2DescribeVolumes(msg *nats.Msg) {
-	handleNATSRequest(msg, d.volumeService.DescribeVolumes)
-}
-
-func (d *Daemon) handleEC2DescribeVolumeStatus(msg *nats.Msg) {
-	handleNATSRequest(msg, d.volumeService.DescribeVolumeStatus)
-}
-
-func (d *Daemon) handleEC2DescribeVolumesModifications(msg *nats.Msg) {
-	handleNATSRequest(msg, d.volumeService.DescribeVolumesModifications)
-}
-
-// handleEC2ModifyVolume processes incoming EC2 ModifyVolume requests
+// handleEC2ModifyVolume processes incoming EC2 ModifyVolume requests.
 func (d *Daemon) handleEC2ModifyVolume(msg *nats.Msg) {
 	ctx, span := utils.StartConsumerSpan(msg)
 	defer span.End()
@@ -205,7 +235,7 @@ func (d *Daemon) handleEC2ModifyVolume(msg *nats.Msg) {
 	if err != nil {
 		slog.ErrorContext(ctx, "handleEC2ModifyVolume service.ModifyVolume failed", "err", err)
 		utils.MarkSpanError(span, err)
-		respondWithError(msg, awserrors.ValidErrorCode(err.Error()))
+		respondWithError(msg, awserrors.ValidErrorCodeFromError(err))
 		return
 	}
 
@@ -227,7 +257,3 @@ func (d *Daemon) handleEC2ModifyVolume(msg *nats.Msg) {
 
 	slog.InfoContext(ctx, "handleEC2ModifyVolume completed", "volumeId", modifyVolumeInput.VolumeId)
 }
-
-func (d *Daemon) handleEC2DeleteVolume(msg *nats.Msg) {
-	handleNATSRequest(msg, d.volumeService.DeleteVolume)
-}

spinifex/daemon/daemon_handlers_volume_test.go

@@ -15,11 +15,15 @@ import (
 	"github.com/mulgadc/spinifex/spinifex/awserrors"
 	"github.com/mulgadc/spinifex/spinifex/objectstore"
 	"github.com/mulgadc/spinifex/spinifex/qmp"
+	"github.com/mulgadc/spinifex/spinifex/testutil"
 	"github.com/mulgadc/spinifex/spinifex/types"
+	"github.com/mulgadc/spinifex/spinifex/utils"
 	"github.com/mulgadc/spinifex/spinifex/vm"
 	"github.com/mulgadc/viperblock/viperblock"
+	"github.com/nats-io/nats.go"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
+	"go.uber.org/goleak"
 )
 
 // TestAttachDetachErrorCode locks down the manager-error → AWS-API-code
@@ -314,3 +318,254 @@ func TestAttachVolume_IdempotentSameInstance_DeviceMismatch(t *testing.T) {
 	require.NoError(t, err)
 	assert.Contains(t, string(resp.Data), awserrors.ErrorVolumeInUse)
 }
+
+// drainCommandFor builds the drain command a snapshot addresses to the node
+// hosting instanceID.
+func drainCommandFor(t *testing.T, instanceID, volumeID string) []byte {
+	t.Helper()
+	command := types.EC2InstanceCommand{
+		ID:              instanceID,
+		Attributes:      types.EC2CommandAttributes{DrainVolume: true},
+		DrainVolumeData: &types.DrainVolumeData{VolumeID: volumeID},
+	}
+	data, err := json.Marshal(command)
+	require.NoError(t, err)
+	return data
+}
+
+// drainTestDaemon returns a daemon owning a running instance, with a data dir
+// short enough to hold the drain socket path.
+func drainTestDaemon(t *testing.T, instanceID string) *Daemon {
+	t.Helper()
+	return drainTestDaemonInState(t, instanceID, vm.StateRunning)
+}
+
+// drainTestDaemonInState is drainTestDaemon for an instance the node still
+// holds in some other state.
+func drainTestDaemonInState(t *testing.T, instanceID string, status vm.InstanceState) *Daemon {
+	t.Helper()
+	daemon := createTestDaemon(t, sharedNATSURL)
+	daemon.config.DataDir = testutil.SocketTempDir(t)
+	daemon.vmMgr.Insert(&vm.VM{
+		ID:           instanceID,
+		Status:       status,
+		AccountID:    testAccountID,
+		InstanceType: getTestInstanceType(t),
+		Instance:     &ec2.Instance{},
+	})
+	return daemon
+}
+
+// drainRequest issues a drain on the instance's command subject, returning the
+// raw reply.
+func drainRequest(t *testing.T, daemon *Daemon, instanceID, volumeID string) *nats.Msg {
+	t.Helper()
+	reply, err := sendDrainCommand(daemon, instanceID, drainCommandFor(t, instanceID, volumeID))
+	require.NoError(t, err)
+	return reply
+}
+
+// sendDrainCommand is drainRequest without the assertions, so a test can issue
+// one from a goroutine and assert on the result back on the test goroutine.
+func sendDrainCommand(daemon *Daemon, instanceID string, command []byte) (*nats.Msg, error) {
+	msg := nats.NewMsg("ec2.cmd." + instanceID)
+	msg.Data = command
+	msg.Header.Set(utils.AccountIDHeader, testAccountID)
+	return daemon.natsConn.RequestMsg(msg, 5*time.Second)
+}
+
+// subscribeEC2Commands wires the daemon's real dispatcher onto the per-instance
+// command subject, so tests exercise the same serial delivery production has.
+func subscribeEC2Commands(t *testing.T, daemon *Daemon, instanceID string) {
+	t.Helper()
+	sub, err := daemon.natsConn.Subscribe("ec2.cmd."+instanceID, daemon.handleEC2Events)
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = sub.Unsubscribe() })
+}
+
+// A drain routed to the node hosting the instance reaches the local socket and
+// acks, which is what lets a snapshot answered elsewhere read a current
+// checkpoint.
+func TestDrainVolume_HostNodeAcksLocalSocket(t *testing.T) {
+	const instanceID, volumeID = "i-drain-ok", "vol-drain-ok"
+	daemon := drainTestDaemon(t, instanceID)
+	testutil.StartDrainSocket(t, daemon.config.DataDir, volumeID, "OK\n")
+
+	resp := requestHandler(t, daemon.natsConn, "ec2.cmd."+instanceID, daemon.handleEC2Events,
+		testAccountID, drainCommandFor(t, instanceID, volumeID))
+
+	var ack types.DrainVolumeResponse
+	require.NoError(t, json.Unmarshal(resp.Data, &ack))
+	assert.Equal(t, volumeID, ack.VolumeID)
+	assert.Equal(t, types.DrainVolumeStatusDrained, ack.Status)
+}
+
+// The owning node having no socket for the volume means the writes cannot be
+// flushed. That must reach the caller as an error, never as a silent success:
+// the caller would otherwise snapshot a stale checkpoint.
+func TestDrainVolume_HostNodeWithoutSocketFails(t *testing.T) {
+	const instanceID, volumeID = "i-drain-nosock", "vol-drain-nosock"
+	daemon := drainTestDaemon(t, instanceID)
+
+	resp := requestHandler(t, daemon.natsConn, "ec2.cmd."+instanceID, daemon.handleEC2Events,
+		testAccountID, drainCommandFor(t, instanceID, volumeID))
+
+	assert.Equal(t, awserrors.ErrorServerInternal, replyErrCode(t, resp.Data))
+}
+
+// A plugin that refuses the drain (ERR rather than OK) is a failure, not an ack.
+func TestDrainVolume_HostNodeRelaysSocketFailure(t *testing.T) {
+	const instanceID, volumeID = "i-drain-err", "vol-drain-err"
+	daemon := drainTestDaemon(t, instanceID)
+	testutil.StartDrainSocket(t, daemon.config.DataDir, volumeID, "ERR\n")
+
+	resp := requestHandler(t, daemon.natsConn, "ec2.cmd."+instanceID, daemon.handleEC2Events,
+		testAccountID, drainCommandFor(t, instanceID, volumeID))
+
+	assert.Equal(t, awserrors.ErrorServerInternal, replyErrCode(t, resp.Data))
+}
+
+// A drain command with no volume is a caller error, not a server fault.
+func TestDrainVolume_MissingVolumeDataIsInvalidParameter(t *testing.T) {
+	const instanceID = "i-drain-novol"
+	daemon := drainTestDaemon(t, instanceID)
+
+	command := types.EC2InstanceCommand{
+		ID:         instanceID,
+		Attributes: types.EC2CommandAttributes{DrainVolume: true},
+	}
+	data, err := json.Marshal(command)
+	require.NoError(t, err)
+
+	resp := requestHandler(t, daemon.natsConn, "ec2.cmd."+instanceID, daemon.handleEC2Events,
+		testAccountID, data)
+
+	assert.Equal(t, awserrors.ErrorInvalidParameterValue, replyErrCode(t, resp.Data))
+}
+
+// Only the instance's owner may drain its volume: the command carries the same
+// ownership gate as every other per-instance command.
+func TestDrainVolume_ForeignAccountRejected(t *testing.T) {
+	const instanceID, volumeID = "i-drain-foreign", "vol-drain-foreign"
+	daemon := drainTestDaemon(t, instanceID)
+	testutil.StartDrainSocket(t, daemon.config.DataDir, volumeID, "OK\n")
+
+	resp := requestHandler(t, daemon.natsConn, "ec2.cmd."+instanceID, daemon.handleEC2Events,
+		"999988887777", drainCommandFor(t, instanceID, volumeID))
+
+	assert.Equal(t, awserrors.ErrorInvalidInstanceIDNotFound, replyErrCode(t, resp.Data))
+}
+
+// Only stable post-teardown states prove the volume is sealed. These states
+// acknowledge not-running without touching the absent plugin socket.
+func TestDrainVolume_CompletedTeardownAcksNotRunning(t *testing.T) {
+	for _, status := range []vm.InstanceState{vm.StateStopped, vm.StateTerminated} {
+		t.Run(string(status), func(t *testing.T) {
+			instanceID := "i-drain-" + string(status)
+			volumeID := "vol-drain-" + string(status)
+			daemon := drainTestDaemonInState(t, instanceID, status)
+
+			resp := requestHandler(t, daemon.natsConn, "ec2.cmd."+instanceID, daemon.handleEC2Events,
+				testAccountID, drainCommandFor(t, instanceID, volumeID))
+
+			var ack types.DrainVolumeResponse
+			require.NoError(t, json.Unmarshal(resp.Data, &ack))
+			assert.Equal(t, volumeID, ack.VolumeID)
+			assert.Equal(t, types.DrainVolumeStatusNotRunning, ack.Status)
+		})
+	}
+}
+
+// Transitional and uncertain states are not proof of a sealed checkpoint. An
+// absent socket in any of them must fail rather than silently snapshot stale
+// data while launch or teardown may still own the volume.
+func TestDrainVolume_UnsealedStatesRequireDrain(t *testing.T) {
+	statuses := []vm.InstanceState{
+		vm.StateRunning,
+		vm.StateStopping,
+		vm.StateShuttingDown,
+		vm.StatePending,
+		vm.StateProvisioning,
+		vm.StateError,
+	}
+	for _, status := range statuses {
+		t.Run(string(status), func(t *testing.T) {
+			instanceID := "i-drain-" + string(status)
+			volumeID := "vol-drain-" + string(status)
+			daemon := drainTestDaemonInState(t, instanceID, status)
+
+			resp := requestHandler(t, daemon.natsConn, "ec2.cmd."+instanceID, daemon.handleEC2Events,
+				testAccountID, drainCommandFor(t, instanceID, volumeID))
+
+			assert.Equal(t, awserrors.ErrorServerInternal, replyErrCode(t, resp.Data))
+		})
+	}
+}
+
+// nats.go delivers a subscription's messages serially, so a drain run inline
+// would hold ec2.cmd.{instanceID} for the length of the flush — stalling stop,
+// terminate and hot-plug for that instance. A drain still flushing must not
+// keep the next command waiting.
+func TestDrainVolume_SlowDrainDoesNotBlockTheCommandSubject(t *testing.T) {
+	const instanceID = "i-drain-parallel"
+	daemon := drainTestDaemon(t, instanceID)
+	accepted, release := testutil.StartBlockingDrainSocket(t, daemon.config.DataDir, "vol-drain-slow", "OK\n")
+	testutil.StartDrainSocket(t, daemon.config.DataDir, "vol-drain-fast", "OK\n")
+	subscribeEC2Commands(t, daemon, instanceID)
+
+	type reply struct {
+		msg *nats.Msg
+		err error
+	}
+	slowCommand := drainCommandFor(t, instanceID, "vol-drain-slow")
+	slow := make(chan reply, 1)
+	go func() {
+		msg, err := sendDrainCommand(daemon, instanceID, slowCommand)
+		slow <- reply{msg, err}
+	}()
+
+	// Only assert once the slow drain is genuinely mid-flush.
+	select {
+	case <-accepted:
+	case <-time.After(5 * time.Second):
+		t.Fatal("the slow drain never reached its socket")
+	}
+
+	fast := drainRequest(t, daemon, instanceID, "vol-drain-fast")
+	var ack types.DrainVolumeResponse
+	require.NoError(t, json.Unmarshal(fast.Data, &ack))
+	assert.Equal(t, types.DrainVolumeStatusDrained, ack.Status)
+
+	release()
+	select {
+	case got := <-slow:
+		require.NoError(t, got.err)
+		require.NoError(t, json.Unmarshal(got.msg.Data, &ack))
+		assert.Equal(t, types.DrainVolumeStatusDrained, ack.Status)
+	case <-time.After(5 * time.Second):
+		t.Fatal("the released drain never replied")
+	}
+}
+
+// Running the drain off the delivery goroutine must not leak it: the goroutine
+// has to be gone once the ack is on the wire, or a snapshot loop against a
+// wedged plugin accumulates them.
+func TestDrainVolume_DispatchedGoroutineDoesNotLeak(t *testing.T) {
+	const instanceID, volumeID = "i-drain-leak", "vol-drain-leak"
+	daemon := drainTestDaemon(t, instanceID)
+	testutil.StartDrainSocket(t, daemon.config.DataDir, volumeID, "OK\n")
+	subscribeEC2Commands(t, daemon, instanceID)
+
+	// Take the baseline after one full round trip: the first request is what
+	// makes nats.go stand up its long-lived response-inbox goroutine, which
+	// would otherwise read as the leak.
+	var ack types.DrainVolumeResponse
+	require.NoError(t, json.Unmarshal(drainRequest(t, daemon, instanceID, volumeID).Data, &ack))
+	require.Equal(t, types.DrainVolumeStatusDrained, ack.Status)
+	ignoreExisting := goleak.IgnoreCurrent()
+
+	require.NoError(t, json.Unmarshal(drainRequest(t, daemon, instanceID, volumeID).Data, &ack))
+	require.Equal(t, types.DrainVolumeStatusDrained, ack.Status)
+
+	goleak.VerifyNone(t, ignoreExisting)
+}