release: update go.mod for v1.8.0

* feat: join nodegroup workers via cluster NLB endpoint

Under the Set-B managed-CP-VPC topology the control plane runs in the
system-account CP VPC while nodegroup workers launch in the customer
VPC. The agent user-data set K3S_URL to the in-VPC control-plane ENI IP
(:6443), which sits in a different, unpeered VPC and is unreachable, so
the join path was dead.

Point K3S_URL at the published cluster NLB endpoint (meta.Endpoint,
https://<frontend-ip>:443, the :443->:6443 listener kubectl already
uses). Guard CreateNodegroup on a non-empty endpoint.

mulga-siv-298

* feat: gate nodegroup ACTIVE on workers registering Ready

launchNodegroupInfra flipped the nodegroup to ACTIVE the moment
RunInstances returned, so a worker that booted but never joined left the
nodegroup falsely ACTIVE with no usable capacity.

Gate ACTIVE on the cluster's Ready-node count (meta.NodeCount, refreshed
by the cluster reconciler from the CP NATS state report — Ready nodes
only) rising by the desired worker count over the create-time baseline.
On timeout the nodegroup is marked CREATE_FAILED with its instance IDs
retained for the reclaim path. Mirrors the CP reconciler's
healthy-observe gating. Bounded by injectable timeout/poll knobs.

mulga-siv-299

* feat(eks): private endpoint access via cross-account Set A ENI

Make EndpointPrivateAccess real: in-VPC workers and kubectl reach the
control plane through a customer-VPC (Set A) resident ENI on the cluster
NLB's LB VM, removing the public-NLB hairpin and the NAT-GW egress path.

Teardown: the Set A ENI is an extra cross-account NIC on the LB VM, not
its primary, so the instance-terminate cascade never reclaims it. Detach
(store-clear) the stale attachment before deleting, so private clusters
delete cleanly instead of looping on InvalidNetworkInterface.InUse.

Refs: mulga-siv-301

* fix(vpc): honor SecurityGroupRuleIds in RevokeSecurityGroup ingress/egress

Modern Terraform (aws_vpc_security_group_ingress_rule) revokes per-rule
resources by SecurityGroupRuleId with empty IpPermissions. The handlers
only read IpPermissions, so they returned success while removing nothing;
the surviving SG-reference rule then pinned both SGs undeletable
(DependencyViolation), deadlocking destroy of mutually-referencing SGs.

Resolve SecurityGroupRuleIds to their stored rules and remove them
(malformed -> InvalidSecurityGroupRuleId.Malformed, absent -> NotFound).

Refs: mulga-siv-321

* fix(vpc): reap the main route table on DeleteVpc

CreateVpc auto-creates a main route table and DeleteRouteTable refuses to
delete a main RT (AWS-faithful), so nothing reclaimed it — every deleted
VPC leaked one orphaned main RT in the rtb KV bucket. DeleteVpc now reaps
the VPC's main route table before deleting the VPC record.

Refs: mulga-siv-322

* fix: make AttachRolePolicy and AssociateRouteTable atomic via KV CAS

Concurrent writes to a single role / route table record used a blind
read-modify-Put with no optimistic-concurrency guard, so parallel writers
racing from the same revision clobbered each other and lost updates.
Verified on env19: three parallel policy attachments to one node role kept
only one; two parallel subnet associations to one route table kept only
one (the provider then timed out waiting for state 'associated').

Route the mutating paths (AttachRolePolicy/DetachRolePolicy,
AssociateRouteTable/DisassociateRouteTable) through a revision-guarded
Update + bounded retry, matching the existing accountCounterBucket CAS
idiom. Add concurrent regression tests covering both lost updates.

* docs: add EKS Terraform workbooks (quickstart, addons-access, https-ingress)

Three OpenTofu workbooks demonstrating EKS on Spinifex, each ending in a
browsable load-balanced demo app:
- eks-quickstart: minimal single-node cluster + NodePort demo
- eks-addons-access: CoreDNS managed addon + access-entry RBAC
- eks-https-ingress: 3-node cluster behind an HTTPS ALB terminating TLS
  with an ACM-imported self-signed certificate

Each covers the Spinifex-specific endpoints, the auto-managed worker SG
NodePort pattern, and the omitted addon_version constraint.

.github/CODEOWNERS

@@ -0,0 +1,15 @@
+# CODEOWNERS — GitHub PR review enforcement
+#
+# Format: <path pattern>  <owner> [<owner> …]
+# Order matters — last matching pattern wins.
+#
+# Set the matching branch protection rule on `dev` and `main` to
+# "Require review from Code Owners" or this file is advisory only.
+
+# Network layer is governed by ADR-0006 invariant tests.
+# Changes here must respect those clauses
+# see spinifex/network/invariants/
+# docs/development/proposals/az-local-architecture/0006-spinifex-network-layer-contract.md
+/spinifex/network/          @joshsiv-mulga
+/spinifex/vpcd/             @joshsiv-mulga
+/spinifex/network/invariants/ @joshsiv-mulga

.github/scripts/capture-net-diag.sh

@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+# capture-net-diag.sh — write a snapshot of network / OVN / OVS / IPsec
+# state into $1 (a directory the caller already created on the local host).
+# Designed to run on a spinifex node after an E2E suite, before tofu destroy
+# pulls the rug out. Every command is independent — `|| true` so a missing
+# binary on one path doesn't abort the rest of the capture.
+#
+# Output layout (under $1/net-diag/):
+#   openvswitch-ipsec.status         systemctl status (output-only)
+#   openvswitch-ipsec.journal        last 200 journal lines
+#   ipsec.openvswitch_conf           ovs-vsctl get Open_vSwitch . other_config
+#   ipsec.xfrm_state                 ip xfrm state list
+#   ipsec.xfrm_policy                ip xfrm policy list
+#   ovn.sb_show                      ovn-sbctl show
+#   ovn.nb_show                      ovn-nbctl show
+#   ovs.show                         ovs-vsctl show
+#   ovs.flows_br_int                 ovs-ofctl dump-flows br-int
+#   ovs.tunnels                      ovs-appctl ofproto/list-tunnels br-int
+#   net.links                        ip -s link show
+#   net.routes                       ip route show table all
+#   net.addrs                        ip -br addr show
+#   systemd.failed_units             systemctl list-units --state=failed
+set +e
+OUT="${1:?usage: capture-net-diag.sh <output-dir>}"
+mkdir -p "$OUT"
+
+run() {
+    local label="$1"; shift
+    {
+        echo "# $*"
+        "$@" 2>&1
+    } > "$OUT/$label" || true
+}
+
+run openvswitch-ipsec.status            sudo systemctl status openvswitch-ipsec.service --no-pager --full --lines=50
+run openvswitch-ipsec.journal           sudo journalctl -u openvswitch-ipsec.service --no-pager --output=short-iso -n 200
+run ipsec.openvswitch_conf              sudo ovs-vsctl get Open_vSwitch . other_config
+run ipsec.xfrm_state                    sudo ip xfrm state list
+run ipsec.xfrm_policy                   sudo ip xfrm policy list
+
+run ovn.sb_show                         sudo ovn-sbctl show
+run ovn.nb_show                         sudo ovn-nbctl show
+run ovn.sb_chassis                      sudo ovn-sbctl list chassis
+
+run_per_lr() {
+    local label="$1"; shift
+    local subcmd="$1"; shift
+    {
+        echo "# ovn-nbctl $subcmd <each-logical-router>"
+        local lrs
+        lrs="$(sudo ovn-nbctl --bare --columns=name list Logical_Router 2>/dev/null)"
+        for lr in $lrs; do
+            echo
+            echo "=== $lr ==="
+            sudo ovn-nbctl "$subcmd" "$lr" 2>&1
+        done
+    } > "$OUT/$label" || true
+}
+
+run_per_lr ovn.lr_routes                lr-route-list
+run_per_lr ovn.lr_policies              lr-policy-list
+run_per_lr ovn.lr_nat                   lr-nat-list
+
+run ovs.show                            sudo ovs-vsctl show
+run ovs.flows_br_int                    sudo ovs-ofctl dump-flows br-int
+run ovs.tunnels                         sudo ovs-appctl ofproto/list-tunnels br-int
+
+run net.links                           ip -s link show
+run net.routes                          ip route show table all
+run net.addrs                           ip -br addr show
+run net.neigh                           ip neigh show
+run net.bridge_fdb                      bridge fdb show
+run net.conntrack                       sudo conntrack -L
+
+run systemd.failed_units                sudo systemctl list-units --state=failed --no-pager
+
+# Background pcap (started by start-net-capture.sh, if present).
+if [ -f /tmp/spinifex-e2e-tcpdump.pid ]; then
+    PID="$(cat /tmp/spinifex-e2e-tcpdump.pid 2>/dev/null)"
+    sudo kill -INT "$PID" 2>/dev/null || true
+    sleep 1
+    if [ -f /tmp/spinifex-e2e-wan.pcap ]; then
+        sudo cp /tmp/spinifex-e2e-wan.pcap "$OUT/wan.pcap" 2>/dev/null || true
+        sudo chmod 0644 "$OUT/wan.pcap" 2>/dev/null || true
+    fi
+fi
+
+exit 0

.github/scripts/start-net-capture.sh

@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+# start-net-capture.sh — kick off background tcpdump on br-wan.
+# Pair: capture-net-diag.sh stops it and copies the pcap into net-diag/.
+#
+# Filter: anything on br-wan involving the EIP pool (192.168.0.0/23) plus
+# ARP for the same range. Single ring buffer (-W 1 -G 0) → grows until stop;
+# capped at 64 MiB via -C 64. Snaplen 200 bytes (headers + a bit) to keep
+# size sane while preserving L2/L3/L4 + first payload bytes.
+set +e
+
+PCAP=/tmp/spinifex-e2e-wan.pcap
+PIDFILE=/tmp/spinifex-e2e-tcpdump.pid
+
+if [ -f "$PIDFILE" ] && kill -0 "$(cat "$PIDFILE" 2>/dev/null)" 2>/dev/null; then
+    echo "[start-net-capture] tcpdump already running pid=$(cat "$PIDFILE")"
+    exit 0
+fi
+
+sudo rm -f "$PCAP" "$PIDFILE"
+
+if ! command -v tcpdump >/dev/null 2>&1; then
+    echo "[start-net-capture] tcpdump not installed — skipping" >&2
+    exit 0
+fi
+if ! ip link show br-wan >/dev/null 2>&1; then
+    echo "[start-net-capture] br-wan absent — skipping" >&2
+    exit 0
+fi
+
+sudo nohup tcpdump -i br-wan -s 200 -C 64 -W 1 -U \
+    -w "$PCAP" \
+    '(net 192.168.0.0/23) or (arp and (arp[24:4]&0xfffffe00=0xc0a80000))' \
+    >/tmp/spinifex-e2e-tcpdump.log 2>&1 &
+PID=$!
+echo "$PID" | sudo tee "$PIDFILE" >/dev/null
+disown
+
+sleep 1
+if ! kill -0 "$PID" 2>/dev/null; then
+    echo "[start-net-capture] tcpdump died — see /tmp/spinifex-e2e-tcpdump.log" >&2
+    cat /tmp/spinifex-e2e-tcpdump.log >&2 || true
+    sudo rm -f "$PIDFILE"
+    exit 1
+fi
+
+echo "[start-net-capture] tcpdump pid=$PID pcap=$PCAP"
+exit 0

.github/workflows/e2e-baremetal.yml

@@ -2,6 +2,13 @@ name: E2E Bare-Metal
 
 on:
   workflow_dispatch:
+  workflow_call:
+    inputs:
+      ref:
+        description: "spinifex ref to checkout/test (default: triggering ref)"
+        required: false
+        type: string
+        default: ""
 
 permissions:
   contents: read
@@ -12,7 +19,7 @@ env:
 
 jobs:
   e2e_baremetal:
-    name: Bare-Metal Multi-Node E2E
+    name: Bare-Metal Single-Node E2E
     runs-on: [ self-hosted, ci-baremetal ]
     timeout-minutes: 120
     concurrency:
@@ -20,11 +27,17 @@ jobs:
       cancel-in-progress: false
     env:
       BM_ISO_CACHE: /var/cache/spinifex-iso-builder
+      ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-20-${{ github.run_id }}
     steps:
       - name: Clean Workspace
-        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          sudo umount -R -l "${BM_ISO_CACHE}/rootfs" 2>/dev/null || true
+          if [ ! -c /dev/null ]; then sudo rm -f /dev/null; sudo mknod -m 666 /dev/null c 1 3; fi
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
 
       - uses: actions/checkout@v6
+        with:
+          ref: ${{ inputs.ref || github.ref }}
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -91,6 +104,8 @@ jobs:
             echo "Ubuntu 26.04 cloud image cached ($(du -h "$UBUNTU_IMG" | cut -f1))"
           fi
 
+      - name: Install go-junit-report on runner
+        run: go install github.com/jstemmer/go-junit-report/v2@latest
 
       - name: Run Bare-Metal E2E
         env:
@@ -100,8 +115,34 @@ jobs:
           BM_UBUNTU_IMAGE: ${{ env.BM_ISO_CACHE }}/ubuntu-26.04-cloudimg-amd64.img
           SPX_EXTERNAL_POOL: ${{ env.SPINIFEX_CI_POOL }}
         run: |
+          mkdir -p "${ARTIFACT_DIR}"
+          set -o pipefail
           sudo -E mulga/scripts/baremetal/run-baremetal-e2e.sh \
             --vmlinuz   "${BM_ISO_CACHE}/vmlinuz" \
             --initrd    "${BM_ISO_CACHE}/initrd.img" \
             --squashfs  "${BM_ISO_CACHE}/spinifex.squashfs" \
-            --tests-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+            --tests-tar "${RUNNER_TEMP}/spinifex-tests.tar" \
+            2>&1 | tee "${ARTIFACT_DIR}/test-baremetal.log"
+
+      - name: Generate JUnit report
+        if: always()
+        run: |
+          LOG="${ARTIFACT_DIR}/test-baremetal.log"
+          [ -f "$LOG" ] || exit 0
+          go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-baremetal.xml" || true
+
+      - name: Analyze e2e failures
+        if: always()
+        uses: ./.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "E2E failure analysis (cell 20)"
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-artifacts-cell-20-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14

.github/workflows/e2e-ddil.yml

@@ -21,10 +21,6 @@ jobs:
   ddil:
     name: DDIL Real Multi-Node
     runs-on: [ self-hosted, ci-multi ]
-    # Budget: ~45m go test + tofu provision/bootstrap/teardown on ci-multi.
-    # In Phase 1 the scenarios all t.Skip, so the real-world runtime is
-    # dominated by provisioning; the 60m ceiling leaves headroom for
-    # Phase 2/3 when assertions come online.
     timeout-minutes: 60
     env:
       TOFU_LOG: /tmp/e2e-tofu-ddil-${{ github.run_id }}.log
@@ -39,10 +35,6 @@ jobs:
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
-      # Full spinifex checkout: the Go harness must run on the runner
-      # (outside the cluster) so its SSH path survives scenario partitions
-      # that drop peer-to-peer cluster traffic. Running inside the cluster
-      # would self-sever when Scenario C isolates a node.
       - uses: actions/checkout@v6
         name: Checkout Spinifex
         with:
@@ -88,7 +80,6 @@ jobs:
         working-directory: mulga/scripts/tofu-cluster
         run: |
           set -euo pipefail
-          # shellcheck source=/dev/null
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           export TF_VAR_git_branch="${GIT_BRANCH}"
           quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
@@ -140,10 +131,6 @@ jobs:
             exit 1
           fi
 
-          # Guest SSH key: bootstrap-install.sh imports spinifex-key into EC2;
-          # the matching private material was generated on the primary node at
-          # ~/.ssh/spinifex-key. Pull it to the runner so the harness can SSH
-          # the witness via its MapPublicIpOnLaunch-assigned EIP.
           GUEST_KEY="${RUNNER_TEMP}/spinifex-key"
           scp -i "${SSH_KEY}" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
             "${SSH_USER}@${PRIMARY_IP}:~/.ssh/spinifex-key" "${GUEST_KEY}"
@@ -151,11 +138,6 @@ jobs:
 
           mkdir -p "${RUNNER_TEMP}/ddil-results"
           cd "${GITHUB_WORKSPACE}/spinifex"
-          # gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout,
-          # plus machine-readable jsonfile + JUnit XML artifacts. The JUnit
-          # file feeds the junit-report composite action below so failures
-          # become summary table rows + workflow annotations.
-          # shellcheck source=/dev/null
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           banner "DDIL scenarios"
           DDIL_NODES="${NODE_IPS}" \
@@ -185,10 +167,6 @@ jobs:
         if: failure()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          # Pull spinifex-* journals + QEMU process listings from every
-          # node into the artifact bundle so failures can be triaged
-          # without re-provisioning the cluster. Best-effort — a missing
-          # node should not mask the original test failure.
           set +e
           INVENTORY=$(tofu output -json inventory)
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
@@ -197,7 +175,6 @@ jobs:
           LOG_DIR="${RUNNER_TEMP}/ddil-results/cluster-logs"
           mkdir -p "${LOG_DIR}"
 
-          # shellcheck source=/dev/null
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           mapfile -t NODE_IPS < <(echo "$INVENTORY" | jq -r '.nodes[].wan_ip')
           for ip in "${NODE_IPS[@]}"; do
@@ -221,7 +198,6 @@ jobs:
         working-directory: mulga/scripts/tofu-cluster
         run: |
           set -euo pipefail
-          # shellcheck source=/dev/null
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
           quiet_run "tofu destroy"          "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false

.github/workflows/e2e-nightly.yml

@@ -40,6 +40,7 @@ jobs:
     timeout-minutes: 35
     env:
       TOFU_LOG: /tmp/e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}.log
+      ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}
 
     steps:
       # blank or scheduled run = full matrix.
@@ -60,7 +61,10 @@ jobs:
 
       - name: Clean Workspace
         if: steps.gate.outputs.selected == 'true'
-        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          sudo umount -R -l /var/cache/spinifex-iso-builder/rootfs 2>/dev/null || true
+          if [ ! -c /dev/null ]; then sudo rm -f /dev/null; sudo mknod -m 666 /dev/null c 1 3; fi
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         if: steps.gate.outputs.selected == 'true'
@@ -78,6 +82,11 @@ jobs:
           sparse-checkout: scripts/tofu-cluster
           path: mulga
 
+      - uses: actions/checkout@v6
+        if: steps.gate.outputs.selected == 'true'
+        with:
+          path: spinifex
+
       - name: Build Install Artifacts
         if: steps.gate.outputs.selected == 'true'
         working-directory: mulga/scripts/tofu-cluster
@@ -132,8 +141,18 @@ jobs:
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
-          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
-            "cd ~/e2e-tests && ./run-e2e.sh && ./run-cert-e2e.sh && ENDPOINT=https://${WAN_IP}:9999 ./run-lb-e2e.sh"
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
+             for SUITE in single cert lb; do \
+               echo \"::group::suite \$SUITE\"; \
+               date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
+               GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+                 ARTIFACT_DIR='${ARTIFACT_DIR}' \
+                 ./\$SUITE.test -test.v -test.timeout=30m \
+                 2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log'; \
+               rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
+               if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
+             done; exit \$FAIL"
 
       - name: Run Multi Node E2E
         if: steps.gate.outputs.selected == 'true' && matrix.topo == 'real-multi'
@@ -144,32 +163,57 @@ jobs:
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
 
-          NODE_IPS=""
+          NODES_CSV=""
           NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
           for i in $(seq 0 $((NODE_COUNT - 1))); do
-            NODE_IPS="${NODE_IPS} $(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")"
+            ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
+            NODES_CSV="${NODES_CSV:+$NODES_CSV,}$ip"
           done
 
-          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
-            "cd ~/e2e-tests && ./run-multinode-e2e.sh ${NODE_IPS} && ./run-cert-e2e.sh --multinode ${NODE_IPS} && ENDPOINT=https://${NODE1_IP}:9999 ./run-lb-e2e.sh --nodes ${NODE_IPS}"
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
+             for SUITE in multinode cert lb; do \
+               echo \"::group::suite \$SUITE\"; \
+               date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
+               GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
+                 SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
+                 SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
+                 ARTIFACT_DIR='${ARTIFACT_DIR}' \
+                 ./\$SUITE.test -test.v -test.timeout=30m \
+                 2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log'; \
+               rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
+               if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
+             done; exit \$FAIL"
 
       - name: Run Tofu Workbooks
         if: steps.gate.outputs.selected == 'true' && matrix.extra == 'tofu-examples'
         working-directory: mulga/scripts/tofu-cluster
+        # The driver emits go test -v markers per workbook; teeing to
+        # test-tofu.log feeds the same go-junit-report -> e2e-analyze -> upload
+        # chain the Go suites use, so cell 17 produces an RCA bundle too.
         run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
           INVENTORY=$(tofu output -json inventory)
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          date -u +%Y-%m-%dT%H:%M:%SZ > "${ARTIFACT_DIR}/test-tofu.start"
           ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
-            "cd ~/e2e-tests && ./run-tofu-examples-e2e.sh"
+            "cd ~/e2e-tests && ./run-tofu-examples-e2e.sh" \
+            2>&1 | tee "${ARTIFACT_DIR}/test-tofu.log"
+          rc=${PIPESTATUS[0]}
+          if [ $rc -ne 0 ]; then echo "tofu: FAILED" >> "${ARTIFACT_DIR}/suite-status.txt"; fi
+          exit $rc
 
       - name: Run Reboot Resilience E2E
         if: steps.gate.outputs.selected == 'true' && matrix.extra == 'reboot-single'
         working-directory: mulga/scripts/tofu-cluster
-        # Driver runs from the runner (NOT via ssh-and-run) so it survives the
+        # reboot.test runs from the runner (NOT via ssh-and-run) so it survives the
         # systemctl-reboot-induced ssh disconnect mid-run.
         run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
           INVENTORY=$(tofu output -json inventory)
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
@@ -178,31 +222,85 @@ jobs:
           REBOOT_DIR="${RUNNER_TEMP}/reboot-e2e"
           mkdir -p "$REBOOT_DIR"
           tar xf "${RUNNER_TEMP}/spinifex-tests.tar" -C "$REBOOT_DIR"
-          chmod +x "$REBOOT_DIR/run-reboot-e2e.sh"
+          chmod +x "$REBOOT_DIR/_bin/reboot.test"
 
-          WAN_IP="$WAN_IP" SSH_KEY="$SSH_KEY" "$REBOOT_DIR/run-reboot-e2e.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i "$SSH_KEY")
+          CREDS=$(ssh "${SSH_OPTS[@]}" tf-user@${WAN_IP} 'cat ~/.aws/credentials')
+          AWS_AK=$(echo "$CREDS" | awk -F '= *' '/^aws_access_key_id/ {print $2; exit}')
+          AWS_SK=$(echo "$CREDS" | awk -F '= *' '/^aws_secret_access_key/ {print $2; exit}')
+          echo "::add-mask::${AWS_AK}"; echo "::add-mask::${AWS_SK}"
 
-      - name: Dump Cluster Journals on Failure
-        if: failure() && steps.gate.outputs.selected == 'true'
+          date -u +%Y-%m-%dT%H:%M:%SZ > "${ARTIFACT_DIR}/test-reboot.start"
+          GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+            SPINIFEX_WAN_IP="${WAN_IP}" SPINIFEX_SSH_KEY="${SSH_KEY}" \
+            SPINIFEX_AWS_ENDPOINT="https://${WAN_IP}:9999" \
+            SPINIFEX_AWS_ACCESS_KEY_ID="${AWS_AK}" SPINIFEX_AWS_SECRET_ACCESS_KEY="${AWS_SK}" \
+            ARTIFACT_DIR="${ARTIFACT_DIR}" \
+            "$REBOOT_DIR/_bin/reboot.test" -test.v -test.timeout=30m \
+            2>&1 | tee "${ARTIFACT_DIR}/test-reboot.log"
+          rc=${PIPESTATUS[0]}
+          if [ $rc -ne 0 ]; then echo "reboot: FAILED" >> "${ARTIFACT_DIR}/suite-status.txt"; fi
+          exit $rc
+
+      - name: Collect e2e artifacts
+        if: always() && steps.gate.outputs.selected == 'true'
         working-directory: mulga/scripts/tofu-cluster
         run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
           INVENTORY=$(tofu output -json inventory 2>/dev/null) || { echo "no inventory; skipping"; exit 0; }
-          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
-          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path'); SSH_KEY="${SSH_KEY/#\~/$HOME}"
           NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY")
+
+          ssh "${SSH_OPTS[@]}" tf-user@${NODE1_IP} "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+
           for i in $(seq 0 $((NODE_COUNT - 1))); do
-            IP=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
-            echo "===== node$((i+1)) ($IP) ====="
-            for svc in spinifex-nats spinifex-predastore spinifex-viperblock spinifex-daemon spinifex-awsgw spinifex-vpcd dhcpcd; do
-              echo "--- $svc ---"
-              ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" tf-user@${IP} \
-                "sudo journalctl -u $svc --no-pager -n 200 2>/dev/null || echo '(not found)'" 2>&1 || echo "(node unreachable)"
-            done
-            echo "--- DHCP client state ---"
-            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" tf-user@${IP} \
-              "systemctl is-enabled dhcpcd.service 2>/dev/null; systemctl is-active dhcpcd.service 2>/dev/null; sudo ss -lupn 'sport = 68' 2>/dev/null; pgrep -laf dhcpcd 2>/dev/null" 2>&1 || echo "(node unreachable)"
+            IP=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip"); idx=$((i + 1))
+            mkdir -p "${ARTIFACT_DIR}/vm-console-node${idx}"
+            ssh "${SSH_OPTS[@]}" tf-user@${IP} 'sudo sh -c "cd /run/spinifex 2>/dev/null && tar -cf - console-*.log 2>/dev/null"' \
+              | tar -C "${ARTIFACT_DIR}/vm-console-node${idx}" -xf - 2>/dev/null
+            ssh "${SSH_OPTS[@]}" tf-user@${IP} 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "60 min ago"' \
+              > "${ARTIFACT_DIR}/spinifex-journal-node${idx}.log" 2>/dev/null
           done
 
+          cat "${ARTIFACT_DIR}"/spinifex-journal-node*.log \
+            > "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || true
+          echo "collected: $(ls "${ARTIFACT_DIR}" 2>/dev/null | wc -l) entries"
+
+      - name: Install go-junit-report
+        if: always() && steps.gate.outputs.selected == 'true'
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Convert test logs to JUnit XML
+        if: always() && steps.gate.outputs.selected == 'true'
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log); SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Analyze e2e failures
+        if: always() && steps.gate.outputs.selected == 'true'
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "E2E failure analysis (cell ${{ matrix.cell }})"
+
+      - uses: actions/upload-artifact@v7
+        if: always() && steps.gate.outputs.selected == 'true'
+        with:
+          name: e2e-artifacts-cell-${{ matrix.cell }}-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
       - name: Teardown
         if: always()
         working-directory: mulga/scripts/tofu-cluster
@@ -216,3 +314,21 @@ jobs:
           tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
           tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1
           ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+
+  # Bare-metal single-node suite. Doesn't fit the tofu matrix shape, so it
+  # runs as a sibling job via the reusable e2e-baremetal workflow. Gated like a
+  # matrix cell ("cell 20"): every weeknight cron, or dispatch with 20 selected
+  # (blank cells = all). Tests `dev` on schedule to match GIT_BRANCH above.
+  baremetal:
+    name: cell-20 (baremetal/pxe/real-hw)
+    if: >-
+      github.event_name == 'schedule' ||
+      inputs.cells == '' ||
+      contains(format(',{0},', inputs.cells), ',20,')
+    permissions:
+      contents: read
+      statuses: write
+    uses: ./.github/workflows/e2e-baremetal.yml
+    with:
+      ref: ${{ github.event_name == 'schedule' && 'dev' || github.ref_name }}
+    secrets: inherit

.github/workflows/e2e-suites.yml

@@ -0,0 +1,799 @@
+# E2E (per-suite blocks) — EXPERIMENTAL, workflow_dispatch only.
+#
+# Restructures the looped e2e.yml into env-grouped job blocks so each suite
+# surfaces as its own node in the Actions graph (mulga-siv-283):
+#
+#   single_provision ─┬─ single_suite (matrix: single, cert, lb, eks)
+#                     └─ single_teardown (if: always)
+#   multi_provision  ─┬─ multi_suite  (matrix: multinode)
+#                     └─ multi_teardown (if: always)
+#
+# Constraint: tofu state is runner-filesystem-local, so *_provision and
+# *_teardown must execute on the same self-hosted runner (one runner per label
+# today). Teardown reuses the persisted mulga/ tofu-cluster workspace and does
+# NOT re-checkout mulga (that would wipe local state). Suite-matrix legs reach
+# the already-provisioned VM over its WAN IP using the key path the provision
+# job emits; they share the runner home dir, so the key file persists.
+#
+# This file does NOT post the gating `e2e` commit status — it runs alongside
+# e2e.yml for validation. Once green across a few manual runs, fold the
+# triggers from e2e.yml here and retire the loop.
+
+name: E2E (suite blocks)
+
+on:
+  workflow_dispatch:
+    inputs:
+      single_suites:
+        description: "Single-node suites to run as matrix legs"
+        required: false
+        default: "single cert lb eks"
+      run_multinode:
+        description: "Also run the multinode env block"
+        type: boolean
+        required: false
+        default: true
+      test_run:
+        description: "Optional -test.run regex (empty = all tests in suite)"
+        required: false
+        default: ""
+
+permissions:
+  contents: read
+
+env:
+  GIT_BRANCH: ${{ github.ref_name }}
+  TEST_RUN: ${{ inputs.test_run || '' }}
+
+jobs:
+  # ---------------------------------------------------------------------------
+  # Single-node env
+  # ---------------------------------------------------------------------------
+  single_provision:
+    name: Single env · provision
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 40
+    outputs:
+      wan_ip: ${{ steps.vm.outputs.wan_ip }}
+      ssh_key: ${{ steps.vm.outputs.ssh_key }}
+      suites: ${{ steps.suites.outputs.list }}
+      matrix: ${{ steps.suites.outputs.matrix }}
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
+    steps:
+      - name: Reap Orphan Procs
+        run: |
+          set +e
+          UID_SELF=$(id -u)
+          WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in
+                "$WS"|"$WS"/*) echo "reap $sig pid=$pid cwd=$cwd"; kill "-$sig" "$pid" 2>/dev/null ;;
+              esac
+            done
+          }
+          reap TERM; sleep 3; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          true
+
+      - name: Clean Workspace
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - uses: actions/checkout@v6
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/tofu-cluster
+          path: mulga
+
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Re-enable Go toolchain auto-download
+        run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
+
+      - name: Compute suite matrix
+        id: suites
+        run: |
+          LIST="${{ inputs.single_suites || 'single cert lb eks' }}"
+          echo "list=${LIST}" >> "$GITHUB_OUTPUT"
+          JSON=$(printf '%s\n' $LIST | jq -R . | jq -cs .)
+          echo "matrix=${JSON}" >> "$GITHUB_OUTPUT"
+
+      - name: Reclaim docker build cache
+        run: |
+          docker container prune -f || true
+          docker image prune -f || true
+          docker buildx prune -af --filter "until=24h" || true
+          docker system df || true
+
+      - name: Build Install Artifacts
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Provision VM
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          export TF_VAR_git_branch="${GIT_BRANCH}"
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "VM provisioned"
+
+      - name: Bootstrap via Binary Install
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Resolve VM SSH details
+        id: vm
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          echo "wan_ip=${WAN_IP}" >> "$GITHUB_OUTPUT"
+          echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
+
+      - name: Build eks-node AMI (libguestfs, runner)
+        if: contains(steps.suites.outputs.list, 'eks')
+        working-directory: spinifex
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "build eks-node AMI (libguestfs — no host devices)"
+          sudo apt-get update -qq
+          sudo apt-get install -y --no-install-recommends libguestfs-tools qemu-utils
+          export LIBGUESTFS_BACKEND=direct
+          sudo chmod 0644 /boot/vmlinuz-* || true
+          group_open "make build-eks-node-image"
+          make build-eks-node-image
+          group_close
+          ok "raw: $(du -h /tmp/eks-node-image-build/eks-node-alpine.raw | cut -f1)"
+
+      - name: Import eks-node AMI on VM
+        if: contains(steps.suites.outputs.list, 'eks')
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          eval "$(grep -E '^(ALPINE_VERSION|AMI_NAME|SYSTEM_TAG)=' \
+            spinifex/scripts/images/eks-node/manifest.conf)"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          banner "import eks-node AMI to VM image store"
+          scp "${SSH_OPTS[@]}" \
+            /tmp/eks-node-image-build/eks-node-alpine.raw \
+            "$VM":/tmp/eks-node-alpine.raw
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "spx admin images import --file /tmp/eks-node-alpine.raw \
+               --distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
+               --boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}" >/dev/null
+          ok "eks-node AMI imported"
+
+      - name: Install kubectl on VM
+        if: contains(steps.suites.outputs.list, 'eks')
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          banner "install kubectl on VM (eks data-plane subtest)"
+          ssh "${SSH_OPTS[@]}" "$VM" 'bash -s' <<'EOF'
+          set -euo pipefail
+          KVER=v1.32.5
+          case "$(uname -m)" in
+            x86_64)  ARCH=amd64 ;;
+            aarch64) ARCH=arm64 ;;
+            *) echo "unsupported arch $(uname -m)" >&2; exit 1 ;;
+          esac
+          cd /tmp
+          curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl"
+          curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl.sha256"
+          echo "$(cat kubectl.sha256)  kubectl" | sha256sum -c -
+          sudo install -m 0755 kubectl /usr/local/bin/kubectl
+          kubectl version --client
+          EOF
+          ok "kubectl installed on VM"
+
+      - name: Build E2E test binaries on runner
+        working-directory: spinifex/tests/e2e
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          # teardown always builds: the run-scoped sweep binary runs in the
+          # teardown job, it is never a selectable suite leg.
+          SUITES_BUILD="${{ steps.suites.outputs.list }} teardown"
+          banner "build ${SUITES_BUILD// /, }"
+          group_open "go test -c"
+          make e2e-build SUITES_BUILD="${SUITES_BUILD}"
+          group_close
+          for f in _bin/*.test; do
+            ok "$(basename "$f") $(stat -c '%s' "$f" | numfmt --to=iec)"
+          done
+
+      - name: Ship test binaries to VM
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          ssh "${SSH_OPTS[@]}" "$VM" "rm -rf \$HOME/spinifex-e2e-bin && mkdir -p \$HOME/spinifex-e2e-bin"
+          tar -C spinifex/tests/e2e/_bin -cf - . | ssh "${SSH_OPTS[@]}" "$VM" "tar -xf - -C \$HOME/spinifex-e2e-bin"
+          ok "shipped binaries to $VM"
+
+      - name: Start packet capture on VM br-wan
+        run: |
+          set +e
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          scp "${SSH_OPTS[@]}" \
+            "${GITHUB_WORKSPACE}/spinifex/.github/scripts/start-net-capture.sh" \
+            "$VM":/tmp/start-net-capture.sh
+          ssh "${SSH_OPTS[@]}" "$VM" "bash /tmp/start-net-capture.sh" || true
+
+  single_suite:
+    name: Single env · suite
+    needs: single_provision
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 45
+    strategy:
+      fail-fast: false
+      matrix:
+        suite: ${{ fromJSON(needs.single_provision.outputs.matrix) }}
+    env:
+      WAN_IP: ${{ needs.single_provision.outputs.wan_ip }}
+      SSH_KEY: ${{ needs.single_provision.outputs.ssh_key }}
+      ARTIFACT_DIR: /tmp/spinifex-e2e-single-${{ github.run_id }}-${{ matrix.suite }}
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Install go-junit-report
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Run ${{ matrix.suite }}
+        id: e2e
+        run: |
+          set +e
+          SUITE="${{ matrix.suite }}"
+          VM_HOST="tf-user@${WAN_IP}"
+          TEST_RUN_FLAG=""
+          [ -n "${TEST_RUN}" ] && TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+          cleanup_remote() {
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" "$VM_HOST" \
+              "pkill -u tf-user -TERM -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; true" \
+              2>/dev/null || true
+          }
+          trap cleanup_remote EXIT INT TERM
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" "$VM_HOST" \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/spinifex-e2e-bin; \
+             R=\$'\033[0m' B=\$'\033[1m' C=\$'\033[36m' G=\$'\033[32m' RED=\$'\033[31m'; \
+             printf '\n%s%s━━ suite %s ━━%s\n' \"\$B\" \"\$C\" \"${SUITE}\" \"\$R\"; \
+             date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-${SUITE}.start'; \
+             GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+               SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+               ARTIFACT_DIR='${ARTIFACT_DIR}' \
+               ./${SUITE}.test -test.v -test.timeout=30m ${TEST_RUN_FLAG} \
+               2>&1 | tee '${ARTIFACT_DIR}/test-${SUITE}.log' \
+                    | sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'; \
+             exit \${PIPESTATUS[0]}"
+          echo "rc=$?" >> "$GITHUB_OUTPUT"
+          exit 0
+
+      - name: Collect suite artifacts from VM
+        if: always()
+        run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${SSH_KEY}")
+          VM=tf-user@${WAN_IP}
+          ssh "${SSH_OPTS[@]}" "$VM" "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+          true
+
+      - name: Convert test log to JUnit XML
+        if: always()
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log); SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          title: "E2E ${{ matrix.suite }} results"
+
+      - name: Analyze e2e failures
+        if: always()
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "E2E ${{ matrix.suite }} failure analysis"
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-single-${{ matrix.suite }}-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
+      - name: Fail job if suite failed
+        if: always()
+        run: |
+          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
+            echo "::error::suite ${{ matrix.suite }} failed (rc=${{ steps.e2e.outputs.rc }})"
+            exit 1
+          fi
+
+  single_teardown:
+    name: Single env · teardown
+    needs: [ single_provision, single_suite ]
+    if: always()
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 20
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-single-td-${{ github.run_id }}.log
+      ARTIFACT_DIR: /tmp/spinifex-e2e-single-diag-${{ github.run_id }}
+      WAN_IP: ${{ needs.single_provision.outputs.wan_ip }}
+      SSH_KEY: ${{ needs.single_provision.outputs.ssh_key }}
+    steps:
+      # Reuse the persisted mulga/ tofu-cluster workspace from single_provision.
+      # Re-checkout only spinifex (for ci-fmt + diag scripts); checking out mulga
+      # again would wipe the local tofu state and orphan the VM.
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - name: Capture network/IPsec diagnostics on VM
+        if: ${{ needs.single_provision.result == 'success' && env.WAN_IP != '' }}
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}")
+          VM=tf-user@${WAN_IP}
+          scp "${SSH_OPTS[@]}" \
+            "${GITHUB_WORKSPACE}/spinifex/.github/scripts/capture-net-diag.sh" \
+            "$VM":/tmp/capture-net-diag.sh 2>/dev/null
+          ssh "${SSH_OPTS[@]}" "$VM" "bash /tmp/capture-net-diag.sh '${ARTIFACT_DIR}/net-diag'" 2>/dev/null
+          ssh "${SSH_OPTS[@]}" "$VM" 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "90 min ago"' \
+            > "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null
+          mkdir -p "${ARTIFACT_DIR}/lb-console"
+          ssh "${SSH_OPTS[@]}" "$VM" 'sudo sh -c "cd /run/spinifex 2>/dev/null && tar -cf - console-*.log 2>/dev/null"' \
+            | tar -C "${ARTIFACT_DIR}/lb-console" -xf - 2>/dev/null
+          true
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-single-diag-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: ignore
+          retention-days: 14
+
+      # Reclaim any resource leaked by a crashed/interrupted suite before the VM
+      # is destroyed. Scoped to this run's tag, best-effort, never fails the job.
+      - name: Sweep leaked run resources on VM
+        if: ${{ needs.single_provision.result == 'success' && env.WAN_IP != '' }}
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}")
+          VM=tf-user@${WAN_IP}
+          banner "sweep run-tagged resources (run ${{ github.run_id }})"
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "cd \$HOME/spinifex-e2e-bin 2>/dev/null && \
+             SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+             SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+             ./teardown.test -test.v -test.run TestSweepRunResources -test.timeout=10m" \
+            2>&1 || true
+          true
+
+      - name: Teardown VM
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          if [ -n "${WAN_IP}" ] && [ -n "${SSH_KEY}" ]; then
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}" \
+              tf-user@${WAN_IP} \
+              'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
+              2>/dev/null || true
+          fi
+          UID_SELF=$(id -u); WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in "$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;; esac
+            done
+          }
+          reap TERM; sleep 2; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
+          ok "teardown complete"
+
+  # ---------------------------------------------------------------------------
+  # Multinode env
+  # ---------------------------------------------------------------------------
+  multi_provision:
+    name: Multi env · provision
+    if: ${{ inputs.run_multinode }}
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 45
+    outputs:
+      node1_ip: ${{ steps.cluster.outputs.node1_ip }}
+      ssh_key: ${{ steps.cluster.outputs.ssh_key }}
+      nodes_csv: ${{ steps.cluster.outputs.nodes_csv }}
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
+    steps:
+      - name: Reap Orphan Procs
+        run: |
+          set +e
+          UID_SELF=$(id -u); WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in "$WS"|"$WS"/*) echo "reap $sig pid=$pid"; kill "-$sig" "$pid" 2>/dev/null ;; esac
+            done
+          }
+          reap TERM; sleep 3; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          true
+
+      - name: Clean Workspace
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - uses: actions/checkout@v6
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/tofu-cluster
+          path: mulga
+
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Re-enable Go toolchain auto-download
+        run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
+
+      - name: Reclaim docker build cache
+        run: |
+          docker container prune -f || true
+          docker image prune -f || true
+          docker buildx prune -af --filter "until=24h" || true
+          docker system df || true
+
+      - name: Build Install Artifacts
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Provision VMs
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          export TF_VAR_git_branch="${GIT_BRANCH}"
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "3-node cluster provisioned"
+
+      - name: Bootstrap via Binary Install
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Resolve cluster SSH details
+        id: cluster
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path'); SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          NODES_CSV=""
+          for i in $(seq 0 $((NODE_COUNT - 1))); do
+            ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
+            if [ -z "$NODES_CSV" ]; then NODES_CSV="$ip"; else NODES_CSV="$NODES_CSV,$ip"; fi
+          done
+          echo "node1_ip=${NODE1_IP}" >> "$GITHUB_OUTPUT"
+          echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
+          echo "nodes_csv=${NODES_CSV}" >> "$GITHUB_OUTPUT"
+
+      - name: Build multinode binary on runner
+        working-directory: spinifex/tests/e2e
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "build multinode, teardown"
+          group_open "go test -c"
+          make e2e-build SUITES_BUILD="multinode teardown"
+          group_close
+          ok "$(basename _bin/multinode.test) $(stat -c '%s' _bin/multinode.test | numfmt --to=iec)"
+
+      - name: Ship test binaries to node1
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.cluster.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.cluster.outputs.node1_ip }}
+          ssh "${SSH_OPTS[@]}" "$VM" "rm -rf \$HOME/spinifex-e2e-bin && mkdir -p \$HOME/spinifex-e2e-bin"
+          tar -C spinifex/tests/e2e/_bin -cf - . | ssh "${SSH_OPTS[@]}" "$VM" "tar -xf - -C \$HOME/spinifex-e2e-bin"
+          ok "shipped binaries to $VM"
+
+      - name: Start packet capture on all nodes
+        run: |
+          set +e
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.cluster.outputs.ssh_key }}")
+          IFS=',' read -ra NODE_LIST <<< "${{ steps.cluster.outputs.nodes_csv }}"
+          for ip in "${NODE_LIST[@]}"; do
+            [ -z "$ip" ] && continue
+            scp "${SSH_OPTS[@]}" \
+              "${GITHUB_WORKSPACE}/spinifex/.github/scripts/start-net-capture.sh" \
+              tf-user@"$ip":/tmp/start-net-capture.sh
+            ssh "${SSH_OPTS[@]}" tf-user@"$ip" "bash /tmp/start-net-capture.sh" || true
+          done
+
+  multi_suite:
+    name: Multi env · suite
+    needs: multi_provision
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 60
+    strategy:
+      fail-fast: false
+      matrix:
+        suite: [ multinode ]
+    env:
+      NODE1_IP: ${{ needs.multi_provision.outputs.node1_ip }}
+      SSH_KEY: ${{ needs.multi_provision.outputs.ssh_key }}
+      NODES_CSV: ${{ needs.multi_provision.outputs.nodes_csv }}
+      ARTIFACT_DIR: /tmp/spinifex-e2e-multi-${{ github.run_id }}-${{ matrix.suite }}
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Install go-junit-report
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Run ${{ matrix.suite }}
+        id: e2e
+        run: |
+          set +e
+          SUITE="${{ matrix.suite }}"
+          VM_HOST="tf-user@${NODE1_IP}"
+          TEST_RUN_FLAG=""
+          [ -n "${TEST_RUN}" ] && TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+          cleanup_remote() {
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" "$VM_HOST" \
+              "pkill -u tf-user -TERM -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; true" \
+              2>/dev/null || true
+          }
+          trap cleanup_remote EXIT INT TERM
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" "$VM_HOST" \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/spinifex-e2e-bin; \
+             printf '\n━━ suite %s ━━\n' \"${SUITE}\"; \
+             date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-${SUITE}.start'; \
+             GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
+               SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+               SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
+               SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
+               ARTIFACT_DIR='${ARTIFACT_DIR}' \
+               ./${SUITE}.test -test.v -test.timeout=30m ${TEST_RUN_FLAG} \
+               2>&1 | tee '${ARTIFACT_DIR}/test-${SUITE}.log' \
+                    | sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'; \
+             exit \${PIPESTATUS[0]}"
+          echo "rc=$?" >> "$GITHUB_OUTPUT"
+          exit 0
+
+      - name: Collect suite artifacts from node1
+        if: always()
+        run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${SSH_KEY}")
+          ssh "${SSH_OPTS[@]}" tf-user@${NODE1_IP} "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+          true
+
+      - name: Convert test log to JUnit XML
+        if: always()
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log); SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          title: "Multi-Node E2E ${{ matrix.suite }} results"
+
+      - name: Analyze e2e failures
+        if: always()
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "Multi-Node E2E ${{ matrix.suite }} failure analysis"
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-multi-${{ matrix.suite }}-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
+      - name: Fail job if suite failed
+        if: always()
+        run: |
+          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
+            echo "::error::suite ${{ matrix.suite }} failed (rc=${{ steps.e2e.outputs.rc }})"
+            exit 1
+          fi
+
+  multi_teardown:
+    name: Multi env · teardown
+    needs: [ multi_provision, multi_suite ]
+    if: always() && inputs.run_multinode
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 20
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-multi-td-${{ github.run_id }}.log
+      SSH_KEY: ${{ needs.multi_provision.outputs.ssh_key }}
+      NODES_CSV: ${{ needs.multi_provision.outputs.nodes_csv }}
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      # Reclaim leaked run-tagged resources before the cluster is destroyed.
+      - name: Sweep leaked run resources on node1
+        if: ${{ needs.multi_provision.result == 'success' && needs.multi_provision.outputs.node1_ip != '' }}
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}")
+          VM=tf-user@${{ needs.multi_provision.outputs.node1_ip }}
+          banner "sweep run-tagged resources (run ${{ github.run_id }})"
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "cd \$HOME/spinifex-e2e-bin 2>/dev/null && \
+             SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
+             SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+             ./teardown.test -test.v -test.run TestSweepRunResources -test.timeout=10m" \
+            2>&1 || true
+          true
+
+      - name: Teardown cluster
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          if [ -n "${NODES_CSV}" ] && [ -n "${SSH_KEY}" ]; then
+            IFS=',' read -ra NODE_LIST <<< "${NODES_CSV}"
+            for ip in "${NODE_LIST[@]}"; do
+              ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}" \
+                tf-user@"$ip" \
+                'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
+                2>/dev/null || true
+            done
+          fi
+          UID_SELF=$(id -u); WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in "$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;; esac
+            done
+          }
+          reap TERM; sleep 2; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
+          ok "teardown complete"
+
+  e2e_suites_status:
+    name: E2E suite blocks · summary
+    if: always()
+    needs: [ single_suite, multi_suite ]
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - name: Summarize
+        run: |
+          echo "## E2E suite-block results" >> "$GITHUB_STEP_SUMMARY"
+          echo "single_suite: ${{ needs.single_suite.result }}" >> "$GITHUB_STEP_SUMMARY"
+          echo "multi_suite:  ${{ needs.multi_suite.result }}" >> "$GITHUB_STEP_SUMMARY"
+          case "${{ needs.single_suite.result }}" in success|skipped) ;; *) exit 1 ;; esac
+          case "${{ needs.multi_suite.result }}" in success|skipped) ;; *) exit 1 ;; esac

.github/workflows/e2e.yml

@@ -17,6 +17,14 @@ on:
       sha:
         description: "PR head SHA to report status on"
         required: false
+      suites:
+        description: "Space-separated Make targets to run (e.g. 'e2e-single e2e-multinode e2e-cert e2e-lb e2e-eks')"
+        required: false
+        default: "e2e-single e2e-multinode e2e-cert e2e-lb"
+      test_run:
+        description: "Optional regex for -test.run (e.g. '^TestMultinodeIPSec$'). Empty = run all tests in suite."
+        required: false
+        default: ""
 
 permissions:
   contents: read
@@ -24,17 +32,84 @@ permissions:
 
 env:
   GIT_BRANCH: ${{ github.ref_name }}
+  SUITES: ${{ inputs.suites || 'e2e-single e2e-multinode e2e-cert e2e-lb' }}
+  TEST_RUN: ${{ inputs.test_run || '' }}
 
 jobs:
+  # Shadow-mode suite selector. Computes which suites the diff *would* run and logs it to the job summary.
+  # Does NOT gate or override the real SUITES env yet — it runs alongside the full matrix so we
+  # can compare predicted vs actual before flipping it on.
+  select_suites:
+    name: Select Suites (shadow)
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: go.mod
+          cache: false
+      - name: Compute targeted suite selection
+        run: |
+          set -euo pipefail
+          BASE="${{ github.event.before }}"
+          HEAD="${{ github.sha }}"
+          # First push to a branch reports an all-zero before-SHA; fall back to
+          # the first parent so the diff is still meaningful.
+          if [ -z "$BASE" ] || [ "$BASE" = "0000000000000000000000000000000000000000" ]; then
+            BASE="${HEAD}~1"
+          fi
+          echo "## Targeted suite selection (shadow)" >> "$GITHUB_STEP_SUMMARY"
+          echo '```' >> "$GITHUB_STEP_SUMMARY"
+          echo "base=$BASE head=$HEAD" >> "$GITHUB_STEP_SUMMARY"
+          go run ./tests/e2e/select-suites/cmd/select-suites \
+            --manifest docs/service-interfaces.yaml \
+            --base-sha "$BASE" --head-sha "$HEAD" --repo-root . \
+            2>>"$GITHUB_STEP_SUMMARY" \
+            | tee -a "$GITHUB_STEP_SUMMARY"
+          echo '```' >> "$GITHUB_STEP_SUMMARY"
+          echo "Shadow only — actual run uses SUITES='${SUITES}'." >> "$GITHUB_STEP_SUMMARY"
+
   e2e_single:
     name: Single-Node E2E
-    runs-on: [ self-hosted, ci-single ]
-    timeout-minutes: 35
+    # Skip when the dispatch input requests no in-VM suites (e.g.
+    # suites='e2e-multinode' only). Push triggers and the default SUITES list
+    # both include single/cert/lb so the inputs.suites null-check keeps the
+    # job running on plain PR pushes.
+    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-lb') || contains(github.event.inputs.suites, 'e2e-eks')
+    runs-on: [self-hosted, ci-single]
+    timeout-minutes: 45
     env:
       TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
+      ARTIFACT_DIR: /tmp/spinifex-e2e-artifacts-${{ github.run_id }}
     steps:
+      - name: Reap Orphan Procs
+        run: |
+          set +e
+          UID_SELF=$(id -u)
+          WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in
+                "$WS"|"$WS"/*) echo "reap $sig pid=$pid cwd=$cwd"; kill "-$sig" "$pid" 2>/dev/null ;;
+              esac
+            done
+          }
+          reap TERM
+          sleep 3
+          reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          true
+
       - name: Clean Workspace
-        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -50,6 +125,25 @@ jobs:
           sparse-checkout: scripts/tofu-cluster
           path: mulga
 
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Re-enable Go toolchain auto-download
+        run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
+
+      - name: Reclaim docker build cache
+        run: |
+          docker container prune -f || true
+          docker image prune -f || true
+          docker buildx prune -af --filter "until=24h" || true
+          docker system df || true
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -61,12 +155,15 @@ jobs:
       - name: Provision VM
         working-directory: mulga/scripts/tofu-cluster
         run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           export TF_VAR_git_branch="${GIT_BRANCH}"
-          tofu init -input=false >> "$TOFU_LOG" 2>&1
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
-          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "VM provisioned"
 
       - name: Bootstrap via Binary Install
         working-directory: mulga/scripts/tofu-cluster
@@ -77,97 +174,338 @@ jobs:
             "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
             --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
 
-      - name: Run E2E Tests
+      - name: Resolve VM SSH details
+        id: vm
         working-directory: mulga/scripts/tofu-cluster
         run: |
           INVENTORY=$(tofu output -json inventory)
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
-          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
-            "cd ~/e2e-tests && ./run-e2e.sh && ./run-cert-e2e.sh && ENDPOINT=https://${WAN_IP}:9999 ./run-lb-e2e.sh"
+          echo "wan_ip=${WAN_IP}" >> "$GITHUB_OUTPUT"
+          echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
+
+      # The eks suite's CreateCluster boots a control-plane VM from the
+      # eks-node AMI, so that AMI must exist in the VM's image store before the
+      # suite runs. build-system-image.sh customizes the image with libguestfs
+      # (an isolated userspace appliance) — it touches no host block device, so
+      # it is safe on the shared runner. The build needs the spinifex source +
+      # network (runner only); the import needs predastore/NATS (VM only), so
+      # build on the runner then ship the raw and import on the VM.
+      - name: Build eks-node AMI (libguestfs, runner)
+        if: contains(env.SUITES, 'e2e-eks')
+        working-directory: spinifex
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "build eks-node AMI (libguestfs — no host devices)"
+          sudo apt-get update -qq
+          sudo apt-get install -y --no-install-recommends libguestfs-tools qemu-utils
+          # Debian/Ubuntu ship /boot/vmlinuz-* mode 0600; the libguestfs
+          # appliance needs to read a host kernel. direct backend = no libvirt.
+          export LIBGUESTFS_BACKEND=direct
+          sudo chmod 0644 /boot/vmlinuz-* || true
+          group_open "make build-eks-node-image"
+          make build-eks-node-image
+          group_close
+          ok "raw: $(du -h /tmp/eks-node-image-build/eks-node-alpine.raw | cut -f1)"
+
+      - name: Import eks-node AMI on VM
+        if: contains(env.SUITES, 'e2e-eks')
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          eval "$(grep -E '^(ALPINE_VERSION|AMI_NAME|SYSTEM_TAG)=' \
+            spinifex/scripts/images/eks-node/manifest.conf)"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          banner "import eks-node AMI to VM image store"
+          scp "${SSH_OPTS[@]}" \
+            /tmp/eks-node-image-build/eks-node-alpine.raw \
+            "$VM":/tmp/eks-node-alpine.raw
+          # Drop the import progress bar (pterm redraws to stdout — thousands of
+          # lines over the SSH pipe). Errors go to stderr, so they survive; the
+          # exit code still gates the step.
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "spx admin images import --file /tmp/eks-node-alpine.raw \
+               --distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
+               --boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}" >/dev/null
+          ok "eks-node AMI imported"
+
+      # The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
+      # `aws eks get-token` → kubectl → apiserver → token-webhook → gateway
+      # broker → STS verify path. The e2e binaries run ON the VM, so kubectl
+      # must be on the VM's PATH or harness.NewKubectl t.Skips the subtest.
+      # Pin to the k3s server version (setup.sh K3S_VERSION) — kubectl is
+      # ±1-minor skew tolerant; matching is simplest. Checksum-verified.
+      - name: Install kubectl on VM
+        if: contains(env.SUITES, 'e2e-eks')
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          banner "install kubectl on VM (eks data-plane subtest)"
+          ssh "${SSH_OPTS[@]}" "$VM" 'bash -s' <<'EOF'
+          set -euo pipefail
+          KVER=v1.32.5
+          case "$(uname -m)" in
+            x86_64)  ARCH=amd64 ;;
+            aarch64) ARCH=arm64 ;;
+            *) echo "unsupported arch $(uname -m)" >&2; exit 1 ;;
+          esac
+          cd /tmp
+          curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl"
+          curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl.sha256"
+          echo "$(cat kubectl.sha256)  kubectl" | sha256sum -c -
+          sudo install -m 0755 kubectl /usr/local/bin/kubectl
+          kubectl version --client
+          EOF
+          ok "kubectl installed on VM"
+
+      - name: Build E2E test binaries on runner
+        working-directory: spinifex/tests/e2e
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SUITES_BUILD=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(multinode)$' | tr '\n' ' ')
+          if [ -z "$(echo "${SUITES_BUILD}" | tr -d ' ')" ]; then
+            echo "no in-VM suites requested (SUITES=${SUITES}); nothing to build"
+            exit 0
+          fi
+          banner "build ${SUITES_BUILD// /, }"
+          group_open "go test -c"
+          make e2e-build SUITES_BUILD="${SUITES_BUILD}"
+          group_close
+          for f in _bin/*.test; do
+            ok "$(basename "$f") $(stat -c '%s' "$f" | numfmt --to=iec)"
+          done
+
+      - name: Install go-junit-report on runner
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Ship test binaries to VM
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          ssh "${SSH_OPTS[@]}" "$VM" "rm -rf \$HOME/spinifex-e2e-bin && mkdir -p \$HOME/spinifex-e2e-bin"
+          tar -C spinifex/tests/e2e/_bin -cf - . | ssh "${SSH_OPTS[@]}" "$VM" "tar -xf - -C \$HOME/spinifex-e2e-bin"
+          ok "shipped binaries to $VM"
+
+      - name: Start packet capture on VM br-wan
+        run: |
+          set +e
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          scp "${SSH_OPTS[@]}" \
+            "${GITHUB_WORKSPACE}/spinifex/.github/scripts/start-net-capture.sh" \
+            "$VM":/tmp/start-net-capture.sh
+          ssh "${SSH_OPTS[@]}" "$VM" "bash /tmp/start-net-capture.sh" || true
+
+      - name: Run E2E suites
+        id: e2e
+        run: |
+          set +e
+          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(multinode)$' | tr '\n' ' ')
+          if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
+            echo "no in-VM suites requested (SUITES=${SUITES}); skipping"
+            echo "rc=0" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+          if [ -n "${TEST_RUN}" ]; then
+            TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+          else
+            TEST_RUN_FLAG=""
+          fi
+          VM_HOST="tf-user@${{ steps.vm.outputs.wan_ip }}"
+          SSH_KEY="${{ steps.vm.outputs.ssh_key }}"
+          cleanup_remote() {
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" "$VM_HOST" \
+              'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
+              2>/dev/null || true
+          }
+          trap cleanup_remote EXIT INT TERM
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" \
+            "$VM_HOST" \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; \
+             cd \$HOME/spinifex-e2e-bin; \
+             R=\$'\033[0m' B=\$'\033[1m' C=\$'\033[36m' G=\$'\033[32m' RED=\$'\033[31m'; \
+             FAIL=0; \
+             for SUITE in ${SUITES_RUN}; do \
+               echo \"::group::suite \$SUITE\"; \
+               printf '\n%s%s━━ suite %s ━━%s\n' \"\$B\" \"\$C\" \"\$SUITE\" \"\$R\"; \
+               date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
+               GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+                 ARTIFACT_DIR='${ARTIFACT_DIR}' \
+                 ./\$SUITE.test -test.v -test.timeout=30m ${TEST_RUN_FLAG} \
+                 2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log' \
+                      | sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'; \
+               rc=\${PIPESTATUS[0]}; \
+               echo \"::endgroup::\"; \
+               if [ \$rc -ne 0 ]; then \
+                 printf '%s✗%s suite %s failed (exit %d)\n' \"\$RED\" \"\$R\" \"\$SUITE\" \"\$rc\"; \
+                 echo \"::error::suite \$SUITE failed (exit \$rc)\"; \
+                 echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; \
+                 FAIL=1; \
+               else \
+                 printf '%s✓%s suite %s passed\n' \"\$G\" \"\$R\" \"\$SUITE\"; \
+               fi; \
+             done; \
+             exit \$FAIL"
+          E2E_RC=$?
+          echo "rc=${E2E_RC}" >> "$GITHUB_OUTPUT"
+          exit 0
+
+      - name: Capture network/IPsec diagnostics on VM
+        if: always()
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          scp "${SSH_OPTS[@]}" \
+            "${GITHUB_WORKSPACE}/spinifex/.github/scripts/capture-net-diag.sh" \
+            "$VM":/tmp/capture-net-diag.sh
+          ssh "${SSH_OPTS[@]}" "$VM" "bash /tmp/capture-net-diag.sh '${ARTIFACT_DIR}/net-diag'"
+          ok "net diagnostics captured"
+
+      - name: Collect artifacts from VM
+        if: always()
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          ssh "${SSH_OPTS[@]}" "$VM" "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+          ok "artifacts pulled to ${ARTIFACT_DIR}"
+          ssh "${SSH_OPTS[@]}" "$VM" 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "60 min ago"' \
+            > "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null
+          ok "journal: $(wc -l < "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || echo 0) lines"
+          # LB/system microVM serial consoles. The lb-agent logs its nginx (NLB)
+          # engine activation and any `nginx -t`/reload error + active-probe
+          # results to the guest serial console, which the host journal never
+          # captures. These files persist in /run/spinifex past VM teardown, so
+          # collecting them here makes an NLB "0/N healthy" timeout diagnosable.
+          mkdir -p "${ARTIFACT_DIR}/lb-console"
+          ssh "${SSH_OPTS[@]}" "$VM" 'sudo sh -c "cd /run/spinifex 2>/dev/null && tar -cf - console-*.log 2>/dev/null"' \
+            | tar -C "${ARTIFACT_DIR}/lb-console" -xf - 2>/dev/null
+          ok "lb-console: $(ls "${ARTIFACT_DIR}/lb-console" 2>/dev/null | wc -l) files"
+
+      - name: Convert test logs to JUnit XML
+        if: always()
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log)
+            SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          title: "E2E results"
+
+      - name: Analyze e2e failures
+        if: always()
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "E2E failure analysis"
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-artifacts-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
 
       - name: Teardown
         if: always()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          if [ -n "${{ steps.vm.outputs.wan_ip }}" ] && [ -n "${{ steps.vm.outputs.ssh_key }}" ]; then
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
+              -i "${{ steps.vm.outputs.ssh_key }}" \
+              tf-user@${{ steps.vm.outputs.wan_ip }} \
+              'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
+              2>/dev/null || true
+          fi
+          UID_SELF=$(id -u)
+          WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in
+                "$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;;
+              esac
+            done
+          }
+          reap TERM
+          sleep 2
+          reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
+          ok "teardown complete"
 
-  e2e_pseudo_multi:
-    name: Pseudo Multi-Node E2E
-    runs-on: [ self-hosted, ci-pseudo ]
-    timeout-minutes: 25
-    env:
-      TOFU_LOG: /tmp/e2e-tofu-pseudo-${{ github.run_id }}.log
-    steps:
-      - name: Clean Workspace
-        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
-
-      - uses: actions/create-github-app-token@v3
-        id: app-token
-        with:
-          client-id: ${{ secrets.APP_ID }}
-          private-key: ${{ secrets.APP_PRIVATE_KEY }}
-          repositories: mulga
-
-      - uses: actions/checkout@v6
-        with:
-          repository: mulgadc/mulga
-          token: ${{ steps.app-token.outputs.token }}
-          sparse-checkout: scripts/tofu-cluster
-          path: mulga
-
-      - name: Clone and Build
-        working-directory: mulga/scripts/tofu-cluster
-        run: ./build-artifacts.sh "${GIT_BRANCH}"
-          "${RUNNER_TEMP}/spinifex-artifacts.tar"
-
-      - name: Provision VM
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          export TF_VAR_git_branch="${GIT_BRANCH}"
-          tofu init -input=false >> "$TOFU_LOG" 2>&1
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
-          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-
-      - name: Bootstrap Cluster
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          SPX_PREBUILT=true \
-          SPX_ARTIFACT_TAR="${RUNNER_TEMP}/spinifex-artifacts.tar" \
-          ./bootstrap.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-
-      - name: Run E2E Tests
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          INVENTORY=$(tofu output -json inventory)
-          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
-          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
-          SSH_KEY="${SSH_KEY/#\~/$HOME}"
-          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
-            "cd ~/Development/mulga/spinifex && SPX_EXTERNAL_POOL=\"${SPINIFEX_CI_POOL}\" ./tests/e2e/run-pseudo-multinode-e2e.sh"
-
-      - name: Teardown
+      - name: Fail job if any suite failed
         if: always()
-        working-directory: mulga/scripts/tofu-cluster
         run: |
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          if [ "${{ steps.e2e.outputs.rc }}" != "0" ] || [ -s "${ARTIFACT_DIR}/suite-status.txt" ]; then
+            echo "::error::E2E suites failed (rc=${{ steps.e2e.outputs.rc }})"
+            cat "${ARTIFACT_DIR}/suite-status.txt" 2>/dev/null || true
+            exit 1
+          fi
 
   e2e_real_multi:
     name: Real Multi-Node E2E
-    runs-on: [ self-hosted, ci-multi ]
-    timeout-minutes: 30
+    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-multinode')
+    runs-on: [self-hosted, ci-multi]
+    timeout-minutes: 60
     env:
       TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
+      ARTIFACT_DIR: /tmp/spinifex-e2e-multi-artifacts-${{ github.run_id }}
     steps:
+      - name: Reap Orphan Procs
+        run: |
+          set +e
+          UID_SELF=$(id -u)
+          WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in
+                "$WS"|"$WS"/*) echo "reap $sig pid=$pid cwd=$cwd"; kill "-$sig" "$pid" 2>/dev/null ;;
+              esac
+            done
+          }
+          reap TERM
+          sleep 3
+          reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          true
+
       - name: Clean Workspace
-        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -183,6 +521,25 @@ jobs:
           sparse-checkout: scripts/tofu-cluster
           path: mulga
 
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Re-enable Go toolchain auto-download
+        run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
+
+      - name: Reclaim docker build cache
+        run: |
+          docker container prune -f || true
+          docker image prune -f || true
+          docker buildx prune -af --filter "until=24h" || true
+          docker system df || true
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -194,12 +551,15 @@ jobs:
       - name: Provision VMs
         working-directory: mulga/scripts/tofu-cluster
         run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           export TF_VAR_git_branch="${GIT_BRANCH}"
-          tofu init -input=false >> "$TOFU_LOG" 2>&1
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
-          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "3-node cluster provisioned"
 
       - name: Bootstrap via Binary Install
         working-directory: mulga/scripts/tofu-cluster
@@ -210,55 +570,345 @@ jobs:
             "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
             --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
 
-      - name: Run E2E Tests
+      - name: Resolve cluster SSH details
+        id: cluster
         working-directory: mulga/scripts/tofu-cluster
         run: |
           INVENTORY=$(tofu output -json inventory)
           NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
-
-          # Build node IP list
-          NODE_IPS=""
           NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          NODES_CSV=""
           for i in $(seq 0 $((NODE_COUNT - 1))); do
-            NODE_IPS="${NODE_IPS} $(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")"
+            ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
+            if [ -z "$NODES_CSV" ]; then NODES_CSV="$ip"; else NODES_CSV="$NODES_CSV,$ip"; fi
+          done
+          echo "node1_ip=${NODE1_IP}" >> "$GITHUB_OUTPUT"
+          echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
+          echo "nodes_csv=${NODES_CSV}" >> "$GITHUB_OUTPUT"
+
+      - name: Build E2E test binaries on runner
+        working-directory: spinifex/tests/e2e
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          # eks is excluded on multinode: EKS KV is a clustered (Raft) JetStream
+          # stream and create-cluster needs an elected leader, but multinode
+          # JetStream clustering isn't solid yet (mulga-omq/z2z/5yg). EKS is
+          # validated single-node only for now; re-enable here once clustered
+          # JetStream + multinode EKS are a deliberate target.
+          SUITES_BUILD=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(single|eks)$' | tr '\n' ' ')
+          if [ -z "$(echo "${SUITES_BUILD}" | tr -d ' ')" ]; then
+            echo "no multinode-eligible suites requested (SUITES=${SUITES}); nothing to build"
+            exit 0
+          fi
+          banner "build ${SUITES_BUILD// /, }"
+          group_open "go test -c"
+          make e2e-build SUITES_BUILD="${SUITES_BUILD}"
+          group_close
+          for f in _bin/*.test; do
+            ok "$(basename "$f") $(stat -c '%s' "$f" | numfmt --to=iec)"
           done
 
-          NODE2_IP=$(echo "$INVENTORY" | jq -r '.nodes[1].wan_ip')
+      - name: Install go-junit-report on runner
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
 
-          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
-            "cd ~/e2e-tests && ./run-multinode-e2e.sh ${NODE_IPS} && ./run-cert-e2e.sh --multinode ${NODE_IPS} && ENDPOINT=https://${NODE1_IP}:9999 ./run-lb-e2e.sh --nodes ${NODE_IPS}"
+      - name: Ship test binaries to node1
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.cluster.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.cluster.outputs.node1_ip }}
+          ssh "${SSH_OPTS[@]}" "$VM" "rm -rf \$HOME/spinifex-e2e-bin && mkdir -p \$HOME/spinifex-e2e-bin"
+          tar -C spinifex/tests/e2e/_bin -cf - . | ssh "${SSH_OPTS[@]}" "$VM" "tar -xf - -C \$HOME/spinifex-e2e-bin"
+          ok "shipped binaries to $VM"
+
+      - name: Start packet capture on all nodes br-wan
+        run: |
+          set +e
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.cluster.outputs.ssh_key }}")
+          IFS=',' read -ra NODE_LIST <<< "${{ steps.cluster.outputs.nodes_csv }}"
+          for ip in "${NODE_LIST[@]}"; do
+              [ -z "$ip" ] && continue
+              scp "${SSH_OPTS[@]}" \
+                  "${GITHUB_WORKSPACE}/spinifex/.github/scripts/start-net-capture.sh" \
+                  tf-user@"$ip":/tmp/start-net-capture.sh
+              ssh "${SSH_OPTS[@]}" tf-user@"$ip" "bash /tmp/start-net-capture.sh" || true
+          done
+
+      - name: Pre-flight diagnostics on node1
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh" 2>/dev/null || true
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.cluster.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.cluster.outputs.node1_ip }}
+          mkdir -p "${ARTIFACT_DIR}"
+          DIAG="${ARTIFACT_DIR}/preflight-diag.log"
+          {
+            echo "=== id (tf-user groups) ==="
+            ssh "${SSH_OPTS[@]}" "$VM" 'id; echo; getent group spinifex || true'
+            echo
+            echo "=== ls -la /etc/spinifex/ ==="
+            ssh "${SSH_OPTS[@]}" "$VM" 'ls -la /etc/spinifex/ 2>&1; echo; ls -la /etc/spinifex/nats/ 2>&1'
+            echo
+            echo "=== spinifex.toml NATS section ==="
+            ssh "${SSH_OPTS[@]}" "$VM" 'grep -A4 "\[nodes\\..*\\.nats" /etc/spinifex/spinifex.toml 2>&1 | head -40'
+            echo
+            echo "=== systemctl is-active spinifex-* ==="
+            ssh "${SSH_OPTS[@]}" "$VM" 'for s in spinifex-nats spinifex-predastore spinifex-awsgw spinifex-vpcd spinifex-daemon; do printf "%-22s %s\n" "$s" "$(systemctl is-active $s 2>&1)"; done'
+            echo
+            echo "=== which spx + version ==="
+            ssh "${SSH_OPTS[@]}" "$VM" 'which spx; spx --version 2>&1 || true'
+            echo
+            echo "=== spx get nodes (direct ssh, tf-user) ==="
+            ssh "${SSH_OPTS[@]}" "$VM" 'spx get nodes --timeout 5s 2>&1; echo "exit=$?"'
+            echo
+            echo "=== spx top nodes (direct ssh, tf-user) ==="
+            ssh "${SSH_OPTS[@]}" "$VM" 'spx top nodes 2>&1 | head -30; echo "exit=$?"'
+            echo
+            echo "=== NATS routez (curl 127.0.0.1:8222) ==="
+            ssh "${SSH_OPTS[@]}" "$VM" 'curl -fsS http://127.0.0.1:8222/routez 2>&1 | head -40'
+            echo
+            echo "=== aws describe-instance-types via gateway (tf-user) ==="
+            ssh "${SSH_OPTS[@]}" "$VM" 'AWS_PROFILE=spinifex AWS_CA_BUNDLE=/etc/spinifex/ca.pem aws --endpoint-url https://${{ steps.cluster.outputs.node1_ip }}:9999 ec2 describe-instance-types --query "InstanceTypes[*].InstanceType" --output text 2>&1 | head -5'
+          } 2>&1 | tee "$DIAG"
+          true
+
+      - name: Run E2E suites
+        id: e2e
+        run: |
+          set +e
+          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(single|eks)$' | tr '\n' ' ')
+          if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
+            echo "no multinode-eligible suites requested (SUITES=${SUITES}); skipping"
+            echo "rc=0" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+          if [ -n "${TEST_RUN}" ]; then
+            TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+          else
+            TEST_RUN_FLAG=""
+          fi
+          VM_HOST="tf-user@${{ steps.cluster.outputs.node1_ip }}"
+          SSH_KEY="${{ steps.cluster.outputs.ssh_key }}"
+          NODES_CSV="${{ steps.cluster.outputs.nodes_csv }}"
+          cleanup_remote() {
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" "$VM_HOST" \
+              'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
+              2>/dev/null || true
+          }
+          trap cleanup_remote EXIT INT TERM
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" \
+            "$VM_HOST" \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; \
+             cd \$HOME/spinifex-e2e-bin; \
+             R=\$'\033[0m' B=\$'\033[1m' C=\$'\033[36m' G=\$'\033[32m' RED=\$'\033[31m'; \
+             FAIL=0; \
+             for SUITE in ${SUITES_RUN}; do \
+               echo \"::group::suite \$SUITE\"; \
+               printf '\n%s%s━━ suite %s ━━%s\n' \"\$B\" \"\$C\" \"\$SUITE\" \"\$R\"; \
+               date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
+               GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
+                 SPINIFEX_NODES='${NODES_CSV}' \
+                 SPINIFEX_NODE_IPS='${NODES_CSV}' \
+                 SPINIFEX_SSH_USER=tf-user \
+                 SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
+                 ARTIFACT_DIR='${ARTIFACT_DIR}' \
+                 ./\$SUITE.test -test.v -test.timeout=30m ${TEST_RUN_FLAG} \
+                 2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log' \
+                      | sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'; \
+               rc=\${PIPESTATUS[0]}; \
+               echo \"::endgroup::\"; \
+               if [ \$rc -ne 0 ]; then \
+                 printf '%s✗%s suite %s failed (exit %d)\n' \"\$RED\" \"\$R\" \"\$SUITE\" \"\$rc\"; \
+                 echo \"::error::suite \$SUITE failed (exit \$rc)\"; \
+                 echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; \
+                 FAIL=1; \
+               else \
+                 printf '%s✓%s suite %s passed\n' \"\$G\" \"\$R\" \"\$SUITE\"; \
+               fi; \
+             done; \
+             exit \$FAIL"
+          E2E_RC=$?
+          echo "rc=${E2E_RC}" >> "$GITHUB_OUTPUT"
+          exit 0
+
+      - name: Capture network/IPsec diagnostics on all nodes
+        if: always()
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.cluster.outputs.ssh_key }}")
+          IFS=',' read -ra NODE_LIST <<< "${{ steps.cluster.outputs.nodes_csv }}"
+          NODE1_IP="${NODE_LIST[0]}"
+          scp "${SSH_OPTS[@]}" \
+            "${GITHUB_WORKSPACE}/spinifex/.github/scripts/capture-net-diag.sh" \
+            tf-user@"$NODE1_IP":/tmp/capture-net-diag.sh
+          for i in "${!NODE_LIST[@]}"; do
+            idx=$((i + 1))
+            ip="${NODE_LIST[$i]}"
+            if [ "$idx" -ne 1 ]; then
+              scp "${SSH_OPTS[@]}" \
+                "${GITHUB_WORKSPACE}/spinifex/.github/scripts/capture-net-diag.sh" \
+                tf-user@"$ip":/tmp/capture-net-diag.sh
+            fi
+            ssh "${SSH_OPTS[@]}" tf-user@"$ip" \
+              "bash /tmp/capture-net-diag.sh '${ARTIFACT_DIR}/net-diag-node${idx}'"
+            if [ "$idx" -ne 1 ]; then
+              ssh "${SSH_OPTS[@]}" tf-user@"$ip" \
+                "tar -C '${ARTIFACT_DIR}' -cf - 'net-diag-node${idx}' 2>/dev/null || true" \
+                | ssh "${SSH_OPTS[@]}" tf-user@"$NODE1_IP" \
+                  "tar -C '${ARTIFACT_DIR}' -xf - 2>/dev/null"
+            fi
+            ok "net diagnostics node${idx} captured"
+          done
+
+      - name: Collect artifacts from node1
+        if: always()
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.cluster.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.cluster.outputs.node1_ip }}
+          ssh "${SSH_OPTS[@]}" "$VM" "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+          ok "artifacts pulled to ${ARTIFACT_DIR}"
+          IFS=',' read -ra NODE_LIST <<< "${{ steps.cluster.outputs.nodes_csv }}"
+          for i in "${!NODE_LIST[@]}"; do
+            idx=$((i + 1))
+            ip="${NODE_LIST[$i]}"
+            ssh "${SSH_OPTS[@]}" tf-user@"$ip" 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "60 min ago"' \
+              > "${ARTIFACT_DIR}/spinifex-journal-node${idx}.log" 2>/dev/null
+            ok "journal node${idx}: $(wc -l < "${ARTIFACT_DIR}/spinifex-journal-node${idx}.log" 2>/dev/null || echo 0) lines"
+          done
+          for i in "${!NODE_LIST[@]}"; do
+            idx=$((i + 1))
+            ip="${NODE_LIST[$i]}"
+            ssh "${SSH_OPTS[@]}" tf-user@"$ip" 'sudo journalctl -u openvswitch-ipsec.service --no-pager --output=short-iso --since "60 min ago"' \
+              > "${ARTIFACT_DIR}/openvswitch-ipsec-node${idx}.log" 2>/dev/null
+          done
+          # LB/system microVM serial consoles, per node. The lb-agent logs its
+          # nginx (NLB) engine activation and any `nginx -t`/reload error +
+          # active-probe results to the guest serial console (never in the host
+          # journal). Files persist in /run/spinifex past VM teardown.
+          for i in "${!NODE_LIST[@]}"; do
+            idx=$((i + 1))
+            ip="${NODE_LIST[$i]}"
+            mkdir -p "${ARTIFACT_DIR}/lb-console/node${idx}"
+            ssh "${SSH_OPTS[@]}" tf-user@"$ip" 'sudo sh -c "cd /run/spinifex 2>/dev/null && tar -cf - console-*.log 2>/dev/null"' \
+              | tar -C "${ARTIFACT_DIR}/lb-console/node${idx}" -xf - 2>/dev/null
+            ok "lb-console node${idx}: $(ls "${ARTIFACT_DIR}/lb-console/node${idx}" 2>/dev/null | wc -l) files"
+          done
+
+      - name: Convert test logs to JUnit XML
+        if: always()
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log)
+            SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          title: "Multi-Node E2E results"
+
+      - name: Analyze e2e failures
+        if: always()
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "Multi-Node E2E failure analysis"
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-multi-artifacts-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
 
       - name: Teardown
         if: always()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          if [ -n "${{ steps.cluster.outputs.node1_ip }}" ] && [ -n "${{ steps.cluster.outputs.ssh_key }}" ]; then
+            IFS=',' read -ra NODE_LIST <<< "${{ steps.cluster.outputs.nodes_csv }}"
+            for ip in "${NODE_LIST[@]}"; do
+              ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
+                -i "${{ steps.cluster.outputs.ssh_key }}" \
+                tf-user@"$ip" \
+                'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
+                2>/dev/null || true
+            done
+          fi
+          UID_SELF=$(id -u)
+          WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in
+                "$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;;
+              esac
+            done
+          }
+          reap TERM
+          sleep 2
+          reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
+          ok "teardown complete"
+
+      - name: Fail job if any suite failed
+        if: always()
+        run: |
+          if [ "${{ steps.e2e.outputs.rc }}" != "0" ] || [ -s "${ARTIFACT_DIR}/suite-status.txt" ]; then
+            echo "::error::Multi-node E2E suites failed (rc=${{ steps.e2e.outputs.rc }})"
+            cat "${ARTIFACT_DIR}/suite-status.txt" 2>/dev/null || true
+            exit 1
+          fi
 
   e2e_status:
     name: Report E2E Status
-    if: always() && inputs.sha
-    needs: [ e2e_single, e2e_pseudo_multi, e2e_real_multi ]
-    runs-on: [ self-hosted, ci-single ]
+    if: always()
+    needs: [e2e_single, e2e_real_multi]
+    runs-on: ubuntu-latest
     timeout-minutes: 5
     steps:
       - name: Report Status to PR
         env:
           GH_TOKEN: ${{ github.token }}
+          REPORT_SHA: ${{ inputs.sha || github.sha }}
         run: |
-          if [[ "${{ needs.e2e_single.result }}" == "success" && \
-                "${{ needs.e2e_pseudo_multi.result }}" == "success" && \
-                "${{ needs.e2e_real_multi.result }}" == "success" ]]; then
+          if [[ -z "${REPORT_SHA}" ]]; then
+            echo "::error::no SHA available (inputs.sha and github.sha both empty)"
+            exit 1
+          fi
+          single_ok=true; multi_ok=true
+          case "${{ needs.e2e_single.result }}" in success|skipped) ;; *) single_ok=false ;; esac
+          case "${{ needs.e2e_real_multi.result }}" in success|skipped) ;; *) multi_ok=false ;; esac
+          if $single_ok && $multi_ok; then
             STATE="success"
-            DESC="All E2E suites passed"
+            DESC="E2E suites passed"
           else
             STATE="failure"
-            DESC="E2E failed: single=${{ needs.e2e_single.result }} pseudo=${{ needs.e2e_pseudo_multi.result }} multi=${{ needs.e2e_real_multi.result }}"
+            DESC="E2E failed: single=${{ needs.e2e_single.result }} multi=${{ needs.e2e_real_multi.result }}"
           fi
-          gh api "repos/${{ github.repository }}/statuses/${{ inputs.sha }}" \
+          gh api "repos/${{ github.repository }}/statuses/${REPORT_SHA}" \
             -f state="$STATE" \
             -f context=e2e \
             -f description="$DESC" \

.github/workflows/publish-eks-ami.yml

@@ -0,0 +1,96 @@
+name: Publish EKS node AMI
+
+# Builds the eks-node system AMI (libguestfs, on the runner) and publishes the
+# compressed qcow2 + .sha256 sidecar to the Cloudflare R2 bucket that backs
+# iso.mulgadc.com/system-ami/. Operators then install it on any cluster with:
+#   spx admin images import --name spinifex-eks-node
+#
+# Manual-only: publishing overwrites the mirror artifact, so it is a deliberate
+# action, not something every push should trigger.
+
+on:
+  workflow_dispatch:
+    inputs:
+      dry_run:
+        description: "Build + convert only; skip the R2 upload"
+        type: boolean
+        default: false
+
+permissions:
+  contents: read
+
+concurrency:
+  group: publish-eks-ami
+  cancel-in-progress: false
+
+jobs:
+  publish:
+    name: Build + publish eks-node AMI
+    runs-on: [ self-hosted, ci-single ]
+
+    steps:
+      - name: Checkout Spinifex
+        uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - name: Checkout Viperblock
+        uses: actions/checkout@v6
+        with:
+          repository: mulgadc/viperblock
+          ref: dev
+          path: viperblock
+
+      - name: Checkout Predastore
+        uses: actions/checkout@v6
+        with:
+          repository: mulgadc/predastore
+          ref: dev
+          path: predastore
+
+      - name: Setup Go
+        uses: actions/setup-go@v6
+        with:
+          go-version-file: "spinifex/go.mod"
+          # Self-hosted runner: the Go module + build cache already persists on
+          # local disk between runs. setup-go's cache-save ("Post Setup Go")
+          # otherwise re-uploads hundreds of MB to the Actions cache backend,
+          # taking longer than the build itself.
+          cache: false
+
+      - name: Setup Go Workspace
+        run: go work init ./spinifex ./viperblock ./predastore
+
+      - name: Install build + upload tooling
+        run: |
+          set -euo pipefail
+          sudo apt-get update -qq
+          sudo apt-get install -y --no-install-recommends libguestfs-tools qemu-utils
+          if ! command -v aws &>/dev/null; then
+            curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o /tmp/awscliv2.zip
+            unzip -q /tmp/awscliv2.zip -d /tmp/aws-install
+            sudo /tmp/aws-install/aws/install
+            rm -rf /tmp/awscliv2.zip /tmp/aws-install
+          fi
+
+      - name: Build + publish to R2
+        working-directory: spinifex
+        env:
+          # Map the repo's R2 secrets onto the names publish-system-image.sh reads.
+          AWS_ACCESS_KEY_ID: ${{ secrets.CF_R2_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.CF_R2_SECRET_ACCESS_KEY }}
+          R2_ENDPOINT: ${{ secrets.CF_R2_ENDPOINT_URL }}
+          R2_BUCKET: spinifex-iso
+          DRY_RUN: ${{ inputs.dry_run }}
+        run: |
+          set -euo pipefail
+          # libguestfs needs to read a host kernel; Ubuntu ships it mode 0600.
+          # direct backend = no libvirt on the runner.
+          export LIBGUESTFS_BACKEND=direct
+          sudo chmod 0644 /boot/vmlinuz-* || true
+          FLAGS=(--build)
+          if [[ "${DRY_RUN}" == "true" ]]; then
+            FLAGS+=(--dry-run)
+          fi
+          ./scripts/publish-system-image.sh \
+            scripts/images/eks-node/manifest.conf "${FLAGS[@]}"

.github/workflows/release.yml

@@ -34,7 +34,7 @@ jobs:
       - name: Set up Go
         uses: actions/setup-go@v6
         with:
-          go-version: '1.26.3'
+          go-version: "1.26.4"
 
       - name: Build microVM artifacts (lb-agent + kernel + initramfs)
         run: cd spinifex && make build-microvm-image
@@ -131,7 +131,10 @@ jobs:
       REPO: ${{ github.repository }}
     steps:
       - name: Clean Workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -169,8 +172,30 @@ jobs:
           TARBALL=$(find "${GITHUB_WORKSPACE}/release-assets" -name "*-linux-amd64.tar.gz" | head -1)
           ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" "$SETUP_SH" "$TARBALL"
 
+      - name: Teardown
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+
+  baremetal:
+    needs: release
+    permissions:
+      contents: read
+      statuses: write
+    uses: ./.github/workflows/e2e-baremetal.yml
+    secrets: inherit
+
+  promote:
+    needs: [ verify, baremetal ]
+    runs-on: ubuntu-24.04
+    env:
+      TAG: ${{ github.ref_name }}
+      REPO: ${{ github.repository }}
+    steps:
       - name: Promote Release
-        if: success()
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
@@ -182,41 +207,41 @@ jobs:
             echo "Release ${TAG} promoted to latest"
           fi
 
+  # Annotate the (still-draft) release if either gate failed.
+  mark-failed:
+    needs: [ verify, baremetal ]
+    if: failure()
+    runs-on: ubuntu-24.04
+    env:
+      TAG: ${{ github.ref_name }}
+      REPO: ${{ github.repository }}
+    steps:
       - name: Mark Release Failed
-        if: failure()
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
           EXISTING_BODY=$(gh release view "$TAG" --repo "$REPO" --json body -q .body)
           NOTES_FILE=$(mktemp)
-          printf 'Verification failed — see workflow run for details.\n\n%s' "$EXISTING_BODY" > "$NOTES_FILE"
+          printf 'Verification failed (tofu install or bare-metal boot) — see workflow run for details.\n\n%s' "$EXISTING_BODY" > "$NOTES_FILE"
           gh release edit "$TAG" --repo "$REPO" --notes-file "$NOTES_FILE"
           rm -f "$NOTES_FILE"
 
-      - name: Teardown
-        if: always()
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
-
   iso:
     needs: release
     runs-on: [ self-hosted, ci-single ]
-    timeout-minutes: 90
+    # ISO build (~60-90m) overlaps the parallel baremetal run (~120m); the
+    # upload gate then waits for baremetal to finish, so the job must outlast it.
+    timeout-minutes: 180
     steps:
       - name: Clean Workspace
         run: |
           ROOTFS="${GITHUB_WORKSPACE}/mulga/scripts/iso-builder/.cache/rootfs"
-          # Mirror make clean's umount sequence unconditionally — lazy unmount
-          # detaches immediately even if the mount is busy, so rm -rf is safe.
           sudo umount -l -R "${ROOTFS}" 2>/dev/null || true
           sudo umount -l "${ROOTFS}/dev/pts" 2>/dev/null || true
           sudo umount -l "${ROOTFS}/dev" 2>/dev/null || true
           sudo umount -l "${ROOTFS}/sys" 2>/dev/null || true
           sudo umount -l "${ROOTFS}/proc" 2>/dev/null || true
-          sudo rm -rf "$GITHUB_WORKSPACE"/*
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -289,24 +314,25 @@ jobs:
             SPINIFEX_BIN="${GITHUB_WORKSPACE}/spinifex/bin/spx" \
             INSTALLER_BIN="${GITHUB_WORKSPACE}/spinifex/bin/spinifex-installer"
 
-      - name: Wait for verify
+      - name: Wait for verify + baremetal
         id: wait-verify
         if: steps.build-iso.outcome == 'success'
         continue-on-error: true
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
-          echo "Waiting for verify job to complete..."
+          echo "Waiting for verify + baremetal jobs to complete..."
           while true; do
-            CONCLUSION=$(gh api \
-              "repos/${{ github.repository }}/actions/runs/${{ github.run_id }}/jobs" \
-              --jq '.jobs[] | select(.name == "verify") | .conclusion // empty')
-            if [ -n "$CONCLUSION" ]; then
-              echo "Verify conclusion: $CONCLUSION"
-              [ "$CONCLUSION" = "success" ] && exit 0
+            JOBS=$(gh api \
+              "repos/${{ github.repository }}/actions/runs/${{ github.run_id }}/jobs" --paginate)
+            VERIFY=$(echo "$JOBS" | jq -r '.jobs[] | select(.name == "verify") | .conclusion // empty')
+            BAREMETAL=$(echo "$JOBS" | jq -r '.jobs[] | select(.name | startswith("baremetal")) | .conclusion // empty')
+            if [ -n "$VERIFY" ] && [ -n "$BAREMETAL" ]; then
+              echo "verify=$VERIFY baremetal=$BAREMETAL"
+              { [ "$VERIFY" = "success" ] && [ "$BAREMETAL" = "success" ]; } && exit 0
               exit 1
             fi
-            echo "Verify still running, retrying in 30s..."
+            echo "Still waiting (verify=${VERIFY:-pending} baremetal=${BAREMETAL:-pending}), retry 30s..."
             sleep 30
           done
 
@@ -327,14 +353,13 @@ jobs:
           AWS_ACCESS_KEY_ID: ${{ secrets.CF_R2_ACCESS_KEY_ID }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.CF_R2_SECRET_ACCESS_KEY }}
           CF_R2_ENDPOINT_URL: ${{ secrets.CF_R2_ENDPOINT_URL }}
-          VERSION: ${{ github.ref_name }}
         run: |
           aws --endpoint-url="${CF_R2_ENDPOINT_URL}" \
-            s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso" \
-                  "s3://spinifex-iso/spinifex-${VERSION}.iso"
+            s3 cp "mulga/scripts/iso-builder/dist/spinifex.iso" \
+                  "s3://spinifex-iso/spinifex.iso"
           aws --endpoint-url="${CF_R2_ENDPOINT_URL}" \
-            s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso.sha256" \
-                  "s3://spinifex-iso/spinifex-${VERSION}.iso.sha256"
+            s3 cp "mulga/scripts/iso-builder/dist/spinifex.iso.sha256" \
+                  "s3://spinifex-iso/spinifex.iso.sha256"
 
       - name: Mark ISO failed
         if: always() && (steps.build-iso.outcome == 'failure' || steps.upload.outcome == 'failure')

.github/workflows/spinifex.yml

@@ -58,6 +58,9 @@ jobs:
       - name: Build Spinifex
         run: make -C spinifex build
 
+      - name: Manifest check + drift lint
+        run: make -C spinifex manifest-check manifest-lint
+
       - name: Run Tests with Coverage
         run: make -C spinifex test-cover
 

INSTALL.md

@@ -1,3 +0,0 @@
-# Spinifex Installation
-
-For documentation on using and installing Spinifex see [https://docs.mulgadc.com](https://docs.mulgadc.com/)

Makefile

@@ -6,17 +6,12 @@ ARCH := $(shell uname -m)
 ifeq ($(ARCH),x86_64)
   GO_ARCH := amd64
   AWS_ARCH := x86_64
-  # On x86, we can run ARM VMs via emulation
-  QEMU_PACKAGES := qemu-system-x86 qemu-system-arm
 else ifeq ($(ARCH),aarch64)
   GO_ARCH := arm64
   AWS_ARCH := aarch64
-  # On ARM, we can run x86 VMs via emulation
-  QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
 else ifeq ($(ARCH),arm64)
   GO_ARCH := arm64
   AWS_ARCH := aarch64
-  QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
 else
   $(error Unsupported architecture: $(ARCH). Only x86_64 and aarch64/arm64 are supported.)
 endif
@@ -73,11 +68,27 @@ build-lb-agent:
 	@echo -e "\n....Building lb-agent (static)"
 	CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
 
-build-system-image: ## Build a system image from manifest (use IMAGE=lb)
+build-system-image: ## Build a system image from manifest (use IMAGE=lb or IMAGE=eks-node)
 ifndef IMAGE
 	$(error IMAGE is required. Usage: make build-system-image IMAGE=lb)
 endif
-	./scripts/build-system-image.sh scripts/images/$(IMAGE).conf
+	@if [ -f scripts/images/$(IMAGE).conf ]; then \
+		./scripts/build-system-image.sh scripts/images/$(IMAGE).conf $(if $(IMPORT),--import); \
+	elif [ -f scripts/images/$(IMAGE)/manifest.conf ]; then \
+		./scripts/build-system-image.sh scripts/images/$(IMAGE)/manifest.conf $(if $(IMPORT),--import); \
+	else \
+		echo "ERROR: no manifest at scripts/images/$(IMAGE).conf or scripts/images/$(IMAGE)/manifest.conf"; \
+		exit 1; \
+	fi
+
+build-eks-node-image: ## Build the unified eks-node AMI (K3s server+agent; role at first boot; IMPORT=1 to register)
+	$(MAKE) build-system-image IMAGE=eks-node
+
+import-eks-node-image: ## Build + register the eks-node AMI (requires a running cluster)
+	$(MAKE) build-system-image IMAGE=eks-node IMPORT=1
+
+publish-eks-node-image: ## Build + publish the eks-node AMI to Cloudflare R2 (needs R2_ENDPOINT + AWS_* env)
+	./scripts/publish-system-image.sh scripts/images/eks-node/manifest.conf --build
 
 MICROVM_OUT_DIR := build/microvm
 MICROVM_ARTIFACTS := $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz
@@ -99,14 +110,10 @@ install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
 
-go_run:
-	@echo -e "\n....Running $(GO_PROJECT_NAME)...."
-	$(GOPATH)/bin/$(GO_PROJECT_NAME)
-
 # Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
 # Use this before committing to catch CI failures locally.
 preflight:
-	@$(MAKE) --no-print-directory QUIET=1 manifest-check lint govulncheck test-cover diff-coverage test-race test-harness
+	@$(MAKE) --no-print-directory QUIET=1 manifest-check manifest-lint lint govulncheck test-cover diff-coverage test-race test-harness
 	@echo -e "\n ✅ Preflight passed — safe to commit."
 
 # E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
@@ -123,6 +130,17 @@ manifest-check:
 	@echo -e "\n....Checking service-interfaces.yaml...."
 	@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml
 
+# Drift guards (Bead 5): direct-create fixture lint + NATS subject lint,
+# ratcheted against tests/e2e/manifest-lint/baseline.txt. Fails only on NEW
+# drift beyond the baseline.
+manifest-lint:
+	@echo -e "\n....Linting manifest drift (fixtures + subjects)...."
+	@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root .
+
+# Accept current drift into the baseline. Run after an intentional change.
+manifest-lint-update:
+	@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root . -update
+
 # Run unit tests
 test:
 	@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
@@ -152,13 +170,8 @@ diff-coverage: test-cover
 
 bench:
 	@echo -e "\n....Running benchmarks for $(GO_PROJECT_NAME)...."
-	$(MAKE) easyjson
 	LOG_IGNORE=1 go test -benchmem -run=. -bench=. ./...
 
-run:
-	$(MAKE) go_build
-	$(MAKE) go_run
-
 # Fast iteration: build + install binary + restart all services.
 # Microvm artifacts are reinstalled when they already exist on disk — the rule's
 # input timestamps drive a rebuild only if anything actually changed. On a fresh
@@ -185,20 +198,19 @@ clean:
 
 install-system:
 	@echo -e "\n....Installing system dependencies for $(ARCH)...."
-	@echo "QEMU packages: $(QEMU_PACKAGES)"
 	sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
 		-o Dpkg::Options::="--force-confdef" \
 		-o Dpkg::Options::="--force-confold" \
-		nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
+		nbdkit nbdkit-plugin-dev pkg-config qemu-system-x86 qemu-system-arm qemu-utils qemu-kvm \
 		ovmf qemu-efi-aarch64 \
 		libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
 		iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
 		ovn-central ovn-host openvswitch-switch dhcpcd-base
 
 install-go:
-	@echo -e "\n....Installing Go 1.26.3 for $(ARCH) ($(GO_ARCH))...."
+	@echo -e "\n....Installing Go 1.26.4 for $(ARCH) ($(GO_ARCH))...."
 	@if [ ! -d "/usr/local/go" ]; then \
-		curl -L https://go.dev/dl/go1.26.3.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
+		curl -L https://go.dev/dl/go1.26.4.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
 	else \
 		echo "Go already installed in /usr/local/go"; \
 	fi
@@ -218,19 +230,15 @@ install-aws:
 quickinstall: install-system install-go install-aws
 	@echo -e "\n✅ Quickinstall complete for $(ARCH)."
 	@echo "   Please ensure /usr/local/go/bin is in your PATH."
-	@echo "   Installed: Go ($(GO_ARCH)), AWS CLI ($(AWS_ARCH)), QEMU ($(QEMU_PACKAGES))"
 
-# Lint all Go code via golangci-lint (replaces check-format, vet, gosec, staticcheck)
 lint:
 	@echo "Running golangci-lint..."
 	$(_Q)golangci-lint run ./...
 	@echo "  golangci-lint ok"
 
-# Auto-fix all linter issues that have fixers
 fix:
 	golangci-lint run --fix ./...
 
-# Govulncheck — dependency vulnerability scanning (not covered by golangci-lint)
 govulncheck:
 	@echo "Running govulncheck..."
 	$(_Q)go tool govulncheck ./...
@@ -279,26 +287,8 @@ distro-arm64:
 distro-clean:
 	rm -rf dist/
 
-# Ansible dev lifecycle (experimental, parallel to dev-*.sh / reset-dev-env.sh).
-# See scripts/ansible/README.md and docs/development/improvements/ansible-dev-lifecycle.md.
-ansible-dev-preflight:
-	cd scripts/ansible && ansible-playbook playbooks/dev-preflight.yml
-
-ansible-dev-teardown:
-	cd scripts/ansible && ansible-playbook playbooks/dev-teardown.yml
-
-ansible-dev-install:
-	cd scripts/ansible && ansible-playbook playbooks/dev-install.yml
-
-ansible-dev-reset:
-	cd scripts/ansible && ansible-playbook playbooks/dev-reset.yml
-
-ansible-dev-deploy:
-	cd scripts/ansible && ansible-playbook playbooks/dev-deploy.yml
-
-.PHONY: build build-ui build-installer build-lb-agent build-system-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race test-actions test-harness manifest-check diff-coverage bench run \
+.PHONY: build build-ui build-installer build-lb-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-harness manifest-check manifest-lint manifest-lint-update \
 	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
 	lint fix govulncheck \
-	distro distro-amd64 distro-arm64 distro-clean \
-	ansible-dev-preflight ansible-dev-teardown ansible-dev-install ansible-dev-reset ansible-dev-deploy
+	distro distro-amd64 distro-arm64 distro-clean

build/microvm/init.sh

@@ -6,20 +6,58 @@ mount -t devtmpfs dev /dev
 # virtio_mmio is built-in (CONFIG_VIRTIO_MMIO=y) so virtio-net devices are
 # already enumerated by the time PID 1 runs — no modprobe needed.
 # qemu_fw_cfg and virtio_net are =m in stock Alpine linux-virt, so load them.
-modprobe qemu_fw_cfg 2>/dev/null || modprobe fw_cfg_sysfs 2>/dev/null || true
-modprobe virtio_net 2>/dev/null || true
+#
+# modprobe needs a usable modules.dep; when the image is built on a host whose
+# depmod can't produce one, busybox modprobe silently no-ops and the module
+# never loads — fw_cfg then never appears and init panics. insmod by explicit
+# path needs no modules.dep, so fall back to it. Handles .ko and .ko.gz.
+load_mod() {
+    modprobe "$1" 2>/dev/null && return 0
+    for ko in $(find /lib/modules -name "$1.ko" -o -name "$1.ko.gz" 2>/dev/null); do
+        insmod "$ko" 2>/dev/null && return 0
+        case "$ko" in
+            *.gz) gunzip -c "$ko" >/tmp/mod.ko 2>/dev/null \
+                  && insmod /tmp/mod.ko 2>/dev/null && return 0 ;;
+        esac
+    done
+    return 1
+}
+load_mod qemu_fw_cfg || load_mod fw_cfg_sysfs || true
+# virtio_net depends on net_failover, which depends on failover. The load_mod
+# insmod fallback loads a single .ko with no dependency resolution, and
+# modules.dep is absent when the image is built on a kmod-less host (depmod
+# skipped), so modprobe can't resolve the chain either. Load it explicitly
+# bottom-up so the NIC comes up regardless of how the image was built —
+# without this the agent has no network and the LB hangs in provisioning.
+load_mod failover || true
+load_mod net_failover || true
+load_mod virtio_net || true
 
-for i in 1 2 3 4 5; do
-    [ -f /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw ] && break
-    sleep 0.02
+NETCFG=/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw
+i=0
+while [ "$i" -lt 25 ]; do
+    [ -f "$NETCFG" ] && break
+    sleep 0.04
+    i=$((i + 1))
 done
 
-if [ ! -f /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw ]; then
+if [ ! -f "$NETCFG" ]; then
+    # Dump host/guest fw_cfg state to the serial console before init exits so a
+    # "netcfg not available" panic is diagnosable from the captured console log
+    # (module-not-loaded vs qemu-not-exposing-by_name vs wrong machine type).
     echo "[init] FATAL: fw_cfg netcfg not available" >&2
+    echo "[init] dmi: vendor=$(cat /sys/class/dmi/id/sys_vendor 2>/dev/null) product=$(cat /sys/class/dmi/id/product_name 2>/dev/null) bios=$(cat /sys/class/dmi/id/bios_version 2>/dev/null)" >&2
+    echo "[init] qemu_fw_cfg dir present: $([ -d /sys/firmware/qemu_fw_cfg ] && echo yes || echo NO)" >&2
+    echo "[init] modules: $(lsmod 2>/dev/null | awk 'NR>1{print $1}' | tr '\n' ',')" >&2
+    echo "[init] by_name listing:" >&2
+    ls -la /sys/firmware/qemu_fw_cfg/by_name/ 2>&1 | sed 's/^/[init]   /' >&2
+    ls -la /sys/firmware/qemu_fw_cfg/by_name/opt/ 2>&1 | sed 's/^/[init]   /' >&2
+    echo "[init] dmesg fw_cfg:" >&2
+    dmesg 2>/dev/null | grep -i fw_cfg | sed 's/^/[init]   /' >&2
     exit 1
 fi
 
-. /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw
+. "$NETCFG"
 
 for n in 0 1 2 3 4 5; do
     eval mac=\$NIC${n}_MAC

build/systemd/spinifex-awsgw.service

@@ -6,6 +6,7 @@ Wants=spinifex-nats.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-gw
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh

build/systemd/spinifex-daemon.service

@@ -1,60 +1,30 @@
 [Unit]
 Description=Spinifex Daemon (VM Orchestration)
 PartOf=spinifex.target
-After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.service ovn-controller.service
-# No Wants=spinifex-nats.service: Wants= reactivates a stopped unit during
-# `systemctl restart spinifex-daemon`, which would mask the standalone-startup
-# path the Tier 1 DDIL Scenario B exercises. spinifex.target Wants= covers
-# cold boot.
+After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.service ovn-controller.service openvswitch-ipsec.service
+# Wants (not Requires): stay bootable when openvswitch-ipsec is absent
+Wants=openvswitch-ipsec.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-daemon
 Group=spinifex
-# DDIL Tier 1 (siv-17): no ExecStartPre wait-for-nats. The daemon starts in
-# standalone mode and reconnects to NATS in the background — process-exiting
-# on a missing NATS would kill the local VM management plane.
-#
-# Opt-in strict-startup (§1d-strict, siv-44): set SPINIFEX_REQUIRE_NATS=1 to
-# restore pre-DDIL fail-fast UX. First NATS connect uses a ~30s bounded retry
-# and the daemon aborts on timeout. Intended for dev/test/single-node deploys
-# where silent NATS misconfig is more confusing than helpful. Do NOT set this
-# in the shipped unit — it re-introduces the SPOF that Tier 1 removed.
 ExecStart=/usr/local/bin/spx service spinifex start
 Restart=on-failure
 RestartSec=5
-# DDIL Tier 1 (mulga-siv-58): KillMode=process so systemd only signals the
-# daemon's main PID, not the cgroup. QEMU children stay alive across
-# `systemctl restart spinifex-daemon` and the new daemon reattaches to them
-# via the persisted local state file + pidfile + QMP socket. With the
-# default control-group mode, restart would terminate every guest VM —
-# defeating Scenario B's "daemon restart preserves running instances".
+# KillMode=process: guest QEMU survives daemon restart; the new daemon reattaches (DDIL)
 KillMode=process
-# 15s upper bound on the SIGTERM grace window. The new SIGTERM handler
-# skips VM teardown (it now just persists state and exits), so 15s is
-# plenty of slack for the cluster-manager + NATS unsubscribe path.
 TimeoutStopSec=15
+# RG-4: infra tier — reaped only after every guest
 OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
 Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex/
 Environment=SPINIFEX_WAL_DIR=/var/lib/spinifex/spinifex/
-# DDIL Tier 1 (mulga-siv-58): QEMU pidfiles, QMP sockets, and the
-# shared qemu.log all resolve via utils.pidPath() — which falls back
-# to os.TempDir() (=/tmp) when $HOME/spinifex doesn't exist. With
-# PrivateTmp=yes, each daemon start gets a fresh /tmp namespace, so
-# pidfiles written by QEMU (running in the OLD daemon's namespace)
-# are invisible to the new daemon. The /local/instances handler then
-# reports PID=0 and Scenario B fails its "live QEMU PID" assertion.
-# Pin XDG_RUNTIME_DIR to the host-wide tmpfs at /run/spinifex (created
-# by tmpfiles.d, already in ReadWritePaths) so pidPath() returns a
-# stable, namespace-independent path that both daemon incarnations and
-# QEMU children resolve identically.
+# Pin to host tmpfs: QEMU pidfiles/QMP sockets must survive PrivateTmp namespace churn (DDIL)
 Environment=XDG_RUNTIME_DIR=/run/spinifex
 
-# Filesystem access — NoNewPrivileges omitted (daemon uses sudo for ip, ovs-vsctl, dhcpcd)
-# CAP_SYS_ADMIN: required for GPU passthrough sysfs operations (vfio group management).
-# CAP_DAC_OVERRIDE: sysfs driver bind/unbind paths (/sys/bus/pci/drivers/vfio-pci/bind,
-# unbind) are root:root 0200; non-root processes need CAP_DAC_OVERRIDE to open them.
+# RG-9: daemon tier — privileged by necessity (GPU vfio). NoNewPrivileges omitted (RG-10: sudo ip/ovs-vsctl/dhcpcd)
 AmbientCapabilities=CAP_SYS_ADMIN CAP_DAC_OVERRIDE
 ProtectSystem=full
 PrivateDevices=no
@@ -66,7 +36,7 @@ DeviceAllow=/dev/vfio/vfio rw
 ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
 ReadWritePaths=/var/lib/spinifex/spinifex /var/log/spinifex /run/spinifex /run/spinifex/nbd
 
-# Hardening
+# RG-9: hardening baseline
 ProtectHome=yes
 PrivateTmp=yes
 RestrictSUIDSGID=yes
@@ -81,8 +51,6 @@ ProtectControlGroups=yes
 ProtectProc=invisible
 SystemCallArchitectures=native
 MemoryDenyWriteExecute=yes
-# DHCP lease acquisition moved to spinifex-vpcd (see dhcp-allocate-error.md);
-# the daemon no longer needs AF_PACKET or the softened kernel protections.
 RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
 ProtectHostname=yes
 KeyringMode=private
@@ -90,8 +58,7 @@ RemoveIPC=yes
 LimitCORE=0
 LimitNOFILE=65536
 LimitNPROC=4096
-# VFIO GPU passthrough requires locking all guest RAM into physical memory.
-# Without this, QEMU fails with RLIMIT_MEMLOCK exceeded when pinning guest pages.
+# LimitMEMLOCK=infinity: VFIO pins all guest RAM
 LimitMEMLOCK=infinity
 
 [Install]

build/systemd/spinifex-guests.slice

@@ -0,0 +1,8 @@
+[Unit]
+Description=Spinifex guest VM tier
+Before=slices.target
+
+[Slice]
+# RG-3: child of spinifex.slice (spinifex-guests -> spinifex), sibling of
+# spinifex-system. Parent for the per-instance spinifex-instance-<id>.slice
+# leaves (0002 RG-7). Inherits the spinifex.slice ceiling; no extra limit here.

build/systemd/spinifex-nats.service

@@ -5,6 +5,7 @@ Before=spinifex-predastore.service spinifex-daemon.service spinifex-awsgw.servic
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-nats
 Group=spinifex
 ExecStart=/usr/local/bin/spx service nats start

build/systemd/spinifex-predastore.service

@@ -6,6 +6,7 @@ Wants=spinifex-nats.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-storage
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh

build/systemd/spinifex-system.slice

@@ -0,0 +1,9 @@
+[Unit]
+Description=Spinifex control/infra tier (NATS, predastore, viperblock, daemon, awsgw, vpcd, ui)
+Before=slices.target
+
+[Slice]
+# RG-3: child of spinifex.slice (spinifex-system -> spinifex), inherits the
+# parent ceiling, no extra limit. Holds the 7 control/infra services. Guest QEMU
+# moves to the sibling spinifex-guests.slice (per-instance placement lands in
+# 0002 RG-7); until then the daemon-forked guests stay under spinifex-daemon.service.

build/systemd/spinifex-ui.service

@@ -5,6 +5,7 @@ After=spinifex-awsgw.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-ui
 Group=spinifex
 ExecStart=/usr/local/bin/spx service spinifex-ui start

build/systemd/spinifex-viperblock.service

@@ -6,6 +6,7 @@ Wants=spinifex-nats.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-viperblock
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh

build/systemd/spinifex-vpcd.service

@@ -6,6 +6,7 @@ Wants=spinifex-nats.service ovn-controller.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-vpcd
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
@@ -16,37 +17,25 @@ OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
 Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/vpcd/
 
-# Filesystem access — NoNewPrivileges omitted (vpcd uses sudo for ovs-vsctl, ovn-nbctl, systemctl)
-ProtectSystem=full
+# RG-9: network tier. vpcd MUST share the host mount namespace — IMDS enters
+# per-VPC netns via setns(), and any filesystem-sandbox directive (ProtectSystem/
+# Home, PrivateTmp, ProtectKernel*, ProtectControlGroups/Proc, Read*Paths) forks a
+# private mount ns that traps the /run/netns bind-mount and breaks setns(2). Those,
+# plus MemoryDenyWriteExecute, are omitted by design — not regressions.
 PrivateDevices=no
-ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/ca.pem
-ReadWritePaths=/var/lib/spinifex/vpcd /var/log/spinifex
-
-# Hardening
-ProtectHome=yes
-PrivateTmp=yes
 RestrictSUIDSGID=no
 LockPersonality=yes
 RestrictRealtime=yes
-RestrictNamespaces=yes
 ProtectClock=yes
-ProtectKernelTunables=yes
-ProtectKernelModules=yes
-ProtectKernelLogs=yes
-ProtectControlGroups=yes
-ProtectProc=invisible
+# mnt alongside net: in-netns `ip` unshares a mount ns for its own /sys
+RestrictNamespaces=net mnt
 SystemCallArchitectures=native
 MemoryDenyWriteExecute=no
-# AF_PACKET: required by the in-process DHCP client (github.com/insomniacslk/dhcp
-# / nclient4) for raw L2 BOOTP on br-wan before an IP is configured on the
-# interface. See docs/development/bugs/dhcp-allocate-error.md.
+# AF_PACKET: raw L2 BOOTP for the in-process DHCP client before the iface has an IP
 RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK AF_PACKET
-# CAP_NET_RAW: open AF_PACKET sockets.
-# CAP_NET_BIND_SERVICE: covers unicast RENEW to UDP :68.
-# CAP_NET_ADMIN: already used for ovs-vsctl / ip link wiring on the bridge.
-# Note, disabled for now, sudo issue with ovs
-AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID
-CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID
+# Caps: NET_ADMIN (ovs/ip), NET_RAW (AF_PACKET), NET_BIND_SERVICE (:68), SETUID/SETGID, SYS_ADMIN (setns into VPC netns)
+AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID CAP_SYS_ADMIN
+CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID CAP_SYS_ADMIN
 NoNewPrivileges=no
 ProtectHostname=yes
 KeyringMode=private

build/systemd/spinifex.slice

@@ -0,0 +1,13 @@
+[Unit]
+Description=Spinifex resource-bounded slice (host ceiling)
+Before=slices.target
+
+[Slice]
+# RG-1: no CPUWeight — slice roots are siblings; capping below the default would
+# let user.slice out-prioritize the VM fleet. Operator CPU priority lives on
+# system.slice (system.slice.d/spinifex-reserve.conf).
+# RG-1/RG-5: throttle+reclaim at High, hard wall at Max (% of physical RAM).
+MemoryHigh=80%
+MemoryMax=90%
+# RG-5: spill to the full swap device before the MemoryMax wall; matched to swap size.
+MemorySwapMax=8G

cmd/dhcptest/main.go

@@ -89,7 +89,7 @@ func main() {
 		}
 	}
 
-	// Build nclient4 options — identical to spinifex/services/vpcd/dhcp/nclient4.go.
+	// Build nclient4 options — standalone DHCP probe for upstream-router debugging.
 	opts := []nclient4.ClientOpt{
 		nclient4.WithHWAddr(hwAddr),
 		nclient4.WithTimeout(*timeout),
@@ -105,7 +105,7 @@ func main() {
 	}
 	defer client.Close()
 
-	// Identity modifiers — identical to identityModifiers() in vpcd dhcp package.
+	// Identity modifiers — client-id / hostname / vendor-class for upstream lease tagging.
 	var mods []dhcpv4.Modifier
 	if cid != "" {
 		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClientIdentifier([]byte(cid))))

cmd/eks-gateway-fetch/main.go

@@ -0,0 +1,139 @@
+// eks-gateway-fetch runs inside the EKS K3s control-plane VM and fetches a
+// control-plane resource from the host through the AWS gateway, instead of
+// dialing core NATS directly. It is the read-side companion to
+// eks-gateway-publish.
+//
+// It SigV4-signs (service "eks") an HTTPS GET to the gateway, retrying with
+// backoff until the gateway returns 2xx or the attempt budget is exhausted (so
+// a degraded link surfaces as a non-zero exit), then emits the staged add-ons
+// as tab-separated lines the on-VM addon-sync shell agent consumes without a
+// JSON parser:
+//
+//	<addonName>\t<addonVersion>\t<serviceAccountRoleArn>\t<base64(configurationValues)>
+//
+// The configuration values are base64-encoded so embedded tabs/newlines cannot
+// break the line framing.
+//
+// Usage:
+//
+//	eks-gateway-fetch -resource addons   # GET /clusters/{cluster}/internal-addons/{accountId}
+//
+// Flags default to environment variables seeded by cloud-init:
+// EKS_GATEWAY_URL, EKS_GATEWAY_CA, EKS_ACCESS_KEY, EKS_SECRET_KEY, EKS_REGION,
+// EKS_ACCOUNT_ID, EKS_CLUSTER_NAME.
+package main
+
+import (
+	"bufio"
+	"encoding/base64"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"log/slog"
+	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/eksgw"
+
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
+)
+
+// stagedAddon mirrors the gateway's StagedAddonManifest wire shape; duplicated
+// here to keep this tiny VM binary free of the handler package's dependency
+// graph.
+type stagedAddon struct {
+	AddonName             string `json:"addonName"`
+	AddonVersion          string `json:"addonVersion"`
+	ServiceAccountRoleArn string `json:"serviceAccountRoleArn"`
+	ConfigurationValues   string `json:"configurationValues"`
+}
+
+type internalAddonsResponse struct {
+	Addons []stagedAddon `json:"addons"`
+}
+
+const (
+	maxAttempts = 30
+	retryDelay  = 5 * time.Second
+)
+
+func main() {
+	var (
+		gatewayURL  string
+		gatewayCA   string
+		accessKey   string
+		secretKey   string
+		region      string
+		accountID   string
+		clusterName string
+		resource    string
+	)
+
+	flag.StringVar(&gatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
+	flag.StringVar(&gatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
+	flag.StringVar(&accessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
+	flag.StringVar(&secretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
+	flag.StringVar(&region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
+	flag.StringVar(&accountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
+	flag.StringVar(&clusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
+	flag.StringVar(&resource, "resource", "addons", "Resource to fetch: addons")
+	flag.Parse()
+
+	switch {
+	case accountID == "":
+		fatal("--account-id is required (or set EKS_ACCOUNT_ID)")
+	case clusterName == "":
+		fatal("--cluster is required (or set EKS_CLUSTER_NAME)")
+	case resource != "addons":
+		fatal("--resource must be addons")
+	}
+
+	client, err := eksgw.New(gatewayURL, gatewayCA, accessKey, secretKey, region)
+	if err != nil {
+		fatal(fmt.Sprintf("build gateway client: %v", err))
+	}
+	path := "/clusters/" + clusterName + "/internal-addons/" + accountID
+
+	var lastErr error
+	for attempt := 1; attempt <= maxAttempts; attempt++ {
+		body, err := client.Get(path)
+		if err != nil {
+			lastErr = err
+			slog.Warn("eks-gateway-fetch: attempt failed", "resource", resource, "attempt", attempt, "err", err)
+			if attempt < maxAttempts {
+				time.Sleep(retryDelay)
+			}
+			continue
+		}
+		if werr := emitAddonsTSV(os.Stdout, body); werr != nil {
+			fatal(fmt.Sprintf("render response: %v", werr))
+		}
+		return
+	}
+	fatal(fmt.Sprintf("fetch failed after %d attempts: %v", maxAttempts, lastErr))
+}
+
+// emitAddonsTSV decodes the internal-addons response and writes one TSV line per
+// staged add-on to w. An empty add-on set writes nothing (the agent then GCs
+// every locally-rendered manifest), which is the correct steady state for a
+// cluster with no managed add-ons.
+func emitAddonsTSV(out io.Writer, body []byte) error {
+	var resp internalAddonsResponse
+	if err := json.Unmarshal(body, &resp); err != nil {
+		return fmt.Errorf("unmarshal internal-addons response: %w", err)
+	}
+	w := bufio.NewWriter(out)
+	for _, a := range resp.Addons {
+		cfg := base64.StdEncoding.EncodeToString([]byte(a.ConfigurationValues))
+		if _, err := fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", a.AddonName, a.AddonVersion, a.ServiceAccountRoleArn, cfg); err != nil {
+			return err
+		}
+	}
+	return w.Flush()
+}
+
+func fatal(msg string) {
+	slog.Error("eks-gateway-fetch: " + msg)
+	os.Exit(1)
+}

cmd/eks-gateway-fetch/main_test.go

@@ -0,0 +1,44 @@
+package main
+
+import (
+	"bytes"
+	"encoding/base64"
+	"testing"
+)
+
+func TestEmitAddonsTSV(t *testing.T) {
+	body := []byte(`{"addons":[
+		{"addonName":"spinifex-noop","addonVersion":"0.1.0"},
+		{"addonName":"aws-load-balancer-controller","addonVersion":"2.8.1",
+		 "serviceAccountRoleArn":"arn:aws:iam::111122223333:role/alb",
+		 "configurationValues":"{\"replicaCount\":2}"}
+	]}`)
+
+	var buf bytes.Buffer
+	if err := emitAddonsTSV(&buf, body); err != nil {
+		t.Fatalf("emitAddonsTSV: %v", err)
+	}
+	cfg := base64.StdEncoding.EncodeToString([]byte(`{"replicaCount":2}`))
+	want := "spinifex-noop\t0.1.0\t\t\n" +
+		"aws-load-balancer-controller\t2.8.1\tarn:aws:iam::111122223333:role/alb\t" + cfg + "\n"
+	if buf.String() != want {
+		t.Fatalf("TSV mismatch:\n got: %q\nwant: %q", buf.String(), want)
+	}
+}
+
+func TestEmitAddonsTSV_Empty(t *testing.T) {
+	var buf bytes.Buffer
+	if err := emitAddonsTSV(&buf, []byte(`{"addons":[]}`)); err != nil {
+		t.Fatalf("emitAddonsTSV: %v", err)
+	}
+	if buf.Len() != 0 {
+		t.Fatalf("empty addon set must emit nothing, got %q", buf.String())
+	}
+}
+
+func TestEmitAddonsTSV_BadJSON(t *testing.T) {
+	var buf bytes.Buffer
+	if err := emitAddonsTSV(&buf, []byte(`{not-json`)); err == nil {
+		t.Fatal("expected error on malformed JSON")
+	}
+}

cmd/eks-gateway-publish/main.go

@@ -0,0 +1,134 @@
+// eks-gateway-publish runs inside the EKS K3s control-plane VM and relays a
+// single publication to the host through the AWS gateway, instead of dialing
+// core NATS directly.
+//
+// It reads a JSON payload on stdin, wraps it as
+// {accountId, channel, kind, payload}, SigV4-signs (service "eks") an HTTPS
+// POST to {gateway}/clusters/{cluster}/internal-publish, and retries with
+// backoff until the gateway returns 2xx or the attempt budget is exhausted —
+// so a degraded link surfaces as a non-zero exit rather than a silently
+// dropped message (the failure mode of fire-and-forget `nats pub`).
+//
+// Usage:
+//
+//	echo '{"token":"..."}' | eks-gateway-publish -channel bootstrap -kind k3s-bootstrap-token
+//	kubectl ... | eks-gateway-publish -channel state
+//
+// Flags default to environment variables seeded by cloud-init:
+// EKS_GATEWAY_URL, EKS_GATEWAY_CA, EKS_ACCESS_KEY, EKS_SECRET_KEY, EKS_REGION,
+// EKS_ACCOUNT_ID, EKS_CLUSTER_NAME.
+package main
+
+import (
+	"bytes"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"log/slog"
+	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/eksgw"
+
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
+)
+
+const (
+	// Retry budget: the control plane reaches readiness before this runs, so a
+	// failing POST means a degraded link, not a cold start. Bounded so a stuck
+	// boot still terminates the OpenRC service.
+	maxAttempts = 30
+	retryDelay  = 5 * time.Second
+)
+
+type publishBody struct {
+	AccountID string          `json:"accountId"`
+	Channel   string          `json:"channel"`
+	Kind      string          `json:"kind"`
+	Payload   json.RawMessage `json:"payload"`
+}
+
+func main() {
+	var (
+		gatewayURL  string
+		gatewayCA   string
+		accessKey   string
+		secretKey   string
+		region      string
+		accountID   string
+		clusterName string
+		channel     string
+		kind        string
+	)
+
+	flag.StringVar(&gatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
+	flag.StringVar(&gatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
+	flag.StringVar(&accessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
+	flag.StringVar(&secretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
+	flag.StringVar(&region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
+	flag.StringVar(&accountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
+	flag.StringVar(&clusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
+	flag.StringVar(&channel, "channel", "", "Publish channel: bootstrap|state|addon")
+	flag.StringVar(&kind, "kind", "", "Bootstrap subject kind (bootstrap channel only)")
+	flag.Parse()
+
+	switch {
+	case accountID == "":
+		fatal("--account-id is required (or set EKS_ACCOUNT_ID)")
+	case clusterName == "":
+		fatal("--cluster is required (or set EKS_CLUSTER_NAME)")
+	case channel != "bootstrap" && channel != "state" && channel != "addon":
+		fatal("--channel must be bootstrap, state or addon")
+	case channel == "bootstrap" && kind == "":
+		fatal("--kind is required for the bootstrap channel")
+	}
+
+	payload, err := io.ReadAll(os.Stdin)
+	if err != nil {
+		fatal(fmt.Sprintf("read stdin payload: %v", err))
+	}
+	if len(bytes.TrimSpace(payload)) == 0 {
+		fatal("empty stdin payload")
+	}
+	if !json.Valid(payload) {
+		fatal("stdin payload is not valid JSON")
+	}
+
+	body, err := json.Marshal(publishBody{
+		AccountID: accountID,
+		Channel:   channel,
+		Kind:      kind,
+		Payload:   json.RawMessage(payload),
+	})
+	if err != nil {
+		fatal(fmt.Sprintf("marshal request body: %v", err))
+	}
+
+	client, err := eksgw.New(gatewayURL, gatewayCA, accessKey, secretKey, region)
+	if err != nil {
+		fatal(fmt.Sprintf("build gateway client: %v", err))
+	}
+	path := "/clusters/" + clusterName + "/internal-publish"
+
+	var lastErr error
+	for attempt := 1; attempt <= maxAttempts; attempt++ {
+		if _, err := client.Post(path, body); err != nil {
+			lastErr = err
+			slog.Warn("eks-gateway-publish: attempt failed",
+				"channel", channel, "kind", kind, "attempt", attempt, "err", err)
+			if attempt < maxAttempts {
+				time.Sleep(retryDelay)
+			}
+			continue
+		}
+		slog.Info("eks-gateway-publish: published", "channel", channel, "kind", kind)
+		return
+	}
+	fatal(fmt.Sprintf("publish failed after %d attempts: %v", maxAttempts, lastErr))
+}
+
+func fatal(msg string) {
+	slog.Error("eks-gateway-publish: " + msg)
+	os.Exit(1)
+}

cmd/eks-token-webhook/cert.go

@@ -0,0 +1,137 @@
+package main
+
+import (
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/tls"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/base64"
+	"encoding/pem"
+	"fmt"
+	"math/big"
+	"net"
+	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/tlsconfig"
+)
+
+// tlsCertificate is the webhook's loopback serving certificate.
+type tlsCertificate struct {
+	cert tls.Certificate
+}
+
+func (c tlsCertificate) serverTLSConfig() *tls.Config {
+	return &tls.Config{
+		Certificates:     []tls.Certificate{c.cert},
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
+	}
+}
+
+// ensureServingCert loads the persisted self-signed serving cert/key, or mints
+// a fresh ECDSA-P256 pair (SAN 127.0.0.1 / localhost) and persists it. Reusing
+// across restarts keeps the CA the apiserver loaded into its webhook kubeconfig
+// valid even if the webhook process is restarted. Returns the parsed cert and
+// the certificate PEM (for embedding as the kubeconfig CA bundle).
+func ensureServingCert(certPath, keyPath string) (tlsCertificate, []byte, error) {
+	if certPEM, keyPEM, ok := readCertPair(certPath, keyPath); ok {
+		cert, err := tls.X509KeyPair(certPEM, keyPEM)
+		if err == nil {
+			return tlsCertificate{cert: cert}, certPEM, nil
+		}
+		// Fall through to regenerate on a corrupt/incompatible persisted pair.
+	}
+
+	certPEM, keyPEM, err := generateSelfSigned()
+	if err != nil {
+		return tlsCertificate{}, nil, err
+	}
+	if err := os.WriteFile(certPath, certPEM, 0o644); err != nil {
+		return tlsCertificate{}, nil, fmt.Errorf("write cert %s: %w", certPath, err)
+	}
+	if err := os.WriteFile(keyPath, keyPEM, 0o600); err != nil {
+		return tlsCertificate{}, nil, fmt.Errorf("write key %s: %w", keyPath, err)
+	}
+	cert, err := tls.X509KeyPair(certPEM, keyPEM)
+	if err != nil {
+		return tlsCertificate{}, nil, fmt.Errorf("parse minted keypair: %w", err)
+	}
+	return tlsCertificate{cert: cert}, certPEM, nil
+}
+
+func readCertPair(certPath, keyPath string) (certPEM, keyPEM []byte, ok bool) {
+	c, err := os.ReadFile(certPath)
+	if err != nil {
+		return nil, nil, false
+	}
+	k, err := os.ReadFile(keyPath)
+	if err != nil {
+		return nil, nil, false
+	}
+	return c, k, true
+}
+
+func generateSelfSigned() (certPEM, keyPEM []byte, err error) {
+	priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		return nil, nil, fmt.Errorf("generate key: %w", err)
+	}
+	serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
+	if err != nil {
+		return nil, nil, fmt.Errorf("serial: %w", err)
+	}
+	now := time.Now().UTC()
+	tmpl := x509.Certificate{
+		SerialNumber:          serial,
+		Subject:               pkix.Name{CommonName: "eks-token-webhook"},
+		NotBefore:             now.Add(-time.Hour),
+		NotAfter:              now.AddDate(10, 0, 0),
+		KeyUsage:              x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment | x509.KeyUsageCertSign,
+		ExtKeyUsage:           []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+		BasicConstraintsValid: true,
+		IsCA:                  true,
+		IPAddresses:           []net.IP{net.IPv4(127, 0, 0, 1), net.IPv6loopback},
+		DNSNames:              []string{"localhost"},
+	}
+	der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &priv.PublicKey, priv)
+	if err != nil {
+		return nil, nil, fmt.Errorf("create cert: %w", err)
+	}
+	keyDER, err := x509.MarshalECPrivateKey(priv)
+	if err != nil {
+		return nil, nil, fmt.Errorf("marshal key: %w", err)
+	}
+	certPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
+	keyPEM = pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
+	return certPEM, keyPEM, nil
+}
+
+// writeAPIServerKubeconfig writes the kubeconfig kube-apiserver loads via
+// --authentication-token-webhook-config-file. It points the apiserver at the
+// loopback webhook and trusts its self-signed serving cert as the CA bundle.
+func writeAPIServerKubeconfig(path, addr string, certPEM []byte) error {
+	caB64 := base64.StdEncoding.EncodeToString(certPEM)
+	kubeconfig := fmt.Sprintf(`apiVersion: v1
+kind: Config
+clusters:
+- name: mulga-eks-token-webhook
+  cluster:
+    certificate-authority-data: %s
+    server: https://%s/authenticate
+users:
+- name: kube-apiserver
+contexts:
+- name: webhook
+  context:
+    cluster: mulga-eks-token-webhook
+    user: kube-apiserver
+current-context: webhook
+`, caB64, addr)
+	if err := os.WriteFile(path, []byte(kubeconfig), 0o600); err != nil {
+		return fmt.Errorf("write %s: %w", path, err)
+	}
+	return nil
+}

cmd/eks-token-webhook/main.go

@@ -0,0 +1,227 @@
+// eks-token-webhook is the K8s TokenReview webhook authenticator baked into
+// the eks-server AMI. Per kubectl call the kube-apiserver POSTs the bearer
+// token (produced by `aws eks get-token`); the webhook relays it through the
+// AWS gateway broker (SigV4 HTTPS POST to /clusters/{name}/token-review), which
+// host-side runs the STS verify (cross-cluster x-k8s-aws-id pin) + AccessEntry
+// KV lookup and returns the resolved identity. The webhook never speaks core
+// NATS — same gateway-broker model as ELBv2's lb-agent and eks-gateway-publish.
+//
+// It binds 127.0.0.1 only; the kube-apiserver is the sole client (loopback).
+// On startup it writes the apiserver webhook kubeconfig (with its self-signed
+// serving CA) so k3s can be pointed at it via
+// --authentication-token-webhook-config-file.
+package main
+
+import (
+	"encoding/json"
+	"errors"
+	"flag"
+	"fmt"
+	"log/slog"
+	"net/http"
+	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/eksgw"
+	handlers_eks "github.com/mulgadc/spinifex/spinifex/handlers/eks"
+)
+
+// config is the webhook's runtime configuration, sourced from the first-boot
+// env (/etc/spinifex-eks/first-boot.env) cloud-init seeds onto the VM.
+type config struct {
+	addr        string
+	gatewayURL  string
+	gatewayCA   string
+	accessKey   string
+	secretKey   string
+	region      string
+	accountID   string
+	clusterName string
+	certPath    string
+	keyPath     string
+	kubeconfig  string
+}
+
+func loadConfig() (config, error) {
+	c := config{
+		addr:        flagAddr,
+		gatewayURL:  os.Getenv("EKS_GATEWAY_URL"),
+		gatewayCA:   os.Getenv("EKS_GATEWAY_CA"),
+		accessKey:   os.Getenv("EKS_ACCESS_KEY"),
+		secretKey:   os.Getenv("EKS_SECRET_KEY"),
+		region:      os.Getenv("EKS_REGION"),
+		accountID:   os.Getenv("EKS_ACCOUNT_ID"),
+		clusterName: os.Getenv("EKS_CLUSTER_NAME"),
+		certPath:    envOr("EKS_WEBHOOK_CERT", "/etc/spinifex-eks/token-webhook.crt"),
+		keyPath:     envOr("EKS_WEBHOOK_KEY", "/etc/spinifex-eks/token-webhook.key"),
+		kubeconfig:  envOr("EKS_WEBHOOK_KUBECONFIG", "/etc/spinifex-eks/token-webhook.kubeconfig"),
+	}
+	switch {
+	case c.gatewayURL == "":
+		return c, errors.New("EKS_GATEWAY_URL not set")
+	case c.accessKey == "" || c.secretKey == "":
+		return c, errors.New("EKS_ACCESS_KEY / EKS_SECRET_KEY not set")
+	case c.accountID == "":
+		return c, errors.New("EKS_ACCOUNT_ID not set")
+	case c.clusterName == "":
+		return c, errors.New("EKS_CLUSTER_NAME not set")
+	}
+	return c, nil
+}
+
+func envOr(key, def string) string {
+	if v := os.Getenv(key); v != "" {
+		return v
+	}
+	return def
+}
+
+var flagAddr string
+
+func main() {
+	flag.StringVar(&flagAddr, "addr", "127.0.0.1:8443", "listen address (loopback only)")
+	flag.Parse()
+
+	slog.SetDefault(slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelInfo})))
+
+	cfg, err := loadConfig()
+	if err != nil {
+		slog.Error("eks-token-webhook config error", "err", err)
+		os.Exit(1)
+	}
+
+	if err := run(cfg); err != nil {
+		slog.Error("eks-token-webhook fatal", "err", err)
+		os.Exit(1)
+	}
+}
+
+func run(cfg config) error {
+	// Serving cert + apiserver kubeconfig. Persisted and reused across restarts
+	// so a webhook crash does not invalidate the CA the apiserver already loaded.
+	tlsCert, certPEM, err := ensureServingCert(cfg.certPath, cfg.keyPath)
+	if err != nil {
+		return fmt.Errorf("serving cert: %w", err)
+	}
+	if err := writeAPIServerKubeconfig(cfg.kubeconfig, cfg.addr, certPEM); err != nil {
+		return fmt.Errorf("write apiserver kubeconfig: %w", err)
+	}
+
+	client, err := eksgw.New(cfg.gatewayURL, cfg.gatewayCA, cfg.accessKey, cfg.secretKey, cfg.region)
+	if err != nil {
+		return fmt.Errorf("build gateway client: %w", err)
+	}
+
+	authr := &authenticator{
+		accountID:   cfg.accountID,
+		clusterName: cfg.clusterName,
+		review:      gatewayReviewer(client, cfg.clusterName, cfg.accountID),
+	}
+
+	mux := http.NewServeMux()
+	mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("ok"))
+	})
+	mux.HandleFunc("/authenticate", authr.handle)
+
+	server := newServer(cfg.addr, mux, tlsCert)
+	slog.Info("eks-token-webhook listening", "addr", cfg.addr, "cluster", cfg.clusterName)
+	if err := server.ListenAndServeTLS("", ""); err != nil && !errors.Is(err, http.ErrServerClosed) {
+		return fmt.Errorf("serve: %w", err)
+	}
+	return nil
+}
+
+// tokenReviewRequest is the body POSTed to /clusters/{name}/token-review. The
+// webhook holds system SigV4 creds (system account, not the cluster account),
+// so accountId names the cluster account explicitly — same as PublishInternal.
+type tokenReviewRequest struct {
+	AccountID string `json:"accountId"`
+	Token     string `json:"token"`
+}
+
+// gatewayReviewer returns the per-call relay: SigV4-POST {accountId,token} to
+// the gateway and decode the resolved identity. A non-2xx / transport error is
+// surfaced (handle() turns it into a 5xx so the apiserver retries) rather than a
+// silent deny, distinguishing "webhook broker down" from "token rejected".
+func gatewayReviewer(client *eksgw.Client, clusterName, accountID string) func(string) (handlers_eks.WebhookTokenReviewResult, error) {
+	path := "/clusters/" + clusterName + "/token-review"
+	return func(token string) (handlers_eks.WebhookTokenReviewResult, error) {
+		body, err := json.Marshal(tokenReviewRequest{AccountID: accountID, Token: token})
+		if err != nil {
+			return handlers_eks.WebhookTokenReviewResult{}, fmt.Errorf("marshal request: %w", err)
+		}
+		respBody, err := client.Post(path, body)
+		if err != nil {
+			return handlers_eks.WebhookTokenReviewResult{}, err
+		}
+		var res handlers_eks.WebhookTokenReviewResult
+		if err := json.Unmarshal(respBody, &res); err != nil {
+			return handlers_eks.WebhookTokenReviewResult{}, fmt.Errorf("decode response: %w", err)
+		}
+		return res, nil
+	}
+}
+
+// authenticator resolves a TokenReview by relaying to the gateway broker.
+type authenticator struct {
+	accountID   string
+	clusterName string
+	review      func(token string) (handlers_eks.WebhookTokenReviewResult, error)
+}
+
+func (a *authenticator) handle(w http.ResponseWriter, r *http.Request) {
+	if r.Method != http.MethodPost {
+		http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
+		return
+	}
+	var review tokenReview
+	if err := json.NewDecoder(r.Body).Decode(&review); err != nil {
+		http.Error(w, "bad request", http.StatusBadRequest)
+		return
+	}
+
+	res, err := a.review(review.Spec.Token)
+	if err != nil {
+		// Broker fault (gateway down, transport error): a 5xx tells the apiserver
+		// to retry rather than treating a transient outage as a hard deny.
+		slog.Error("TokenReview relay failed", "cluster", a.clusterName, "err", err)
+		http.Error(w, "token review unavailable", http.StatusServiceUnavailable)
+		return
+	}
+
+	status := tokenReviewStatus{Authenticated: res.Authenticated}
+	if res.Authenticated {
+		status.User = userInfo{
+			Username: res.Username,
+			UID:      res.UID,
+			Groups:   res.Groups,
+		}
+		// Confirm the token for the audiences the apiserver asked about; an empty
+		// status.audiences is rejected when --api-audiences is configured.
+		status.Audiences = review.Spec.Audiences
+	}
+	slog.Info("TokenReview decision", "authenticated", status.Authenticated, "username", status.User.Username, "audiences", status.Audiences)
+
+	resp := tokenReview{
+		APIVersion: "authentication.k8s.io/v1",
+		Kind:       "TokenReview",
+		Status:     status,
+	}
+	w.Header().Set("Content-Type", "application/json")
+	if err := json.NewEncoder(w).Encode(resp); err != nil {
+		slog.Error("encode TokenReview response", "err", err)
+	}
+}
+
+// newServer builds the loopback TLS server. Split out so tests can construct it
+// without binding a port.
+func newServer(addr string, handler http.Handler, cert tlsCertificate) *http.Server {
+	return &http.Server{
+		Addr:              addr,
+		Handler:           handler,
+		ReadHeaderTimeout: 5 * time.Second,
+		TLSConfig:         cert.serverTLSConfig(),
+	}
+}

cmd/eks-token-webhook/main_test.go

@@ -0,0 +1,123 @@
+package main
+
+import (
+	"encoding/base64"
+	"encoding/json"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	handlers_eks "github.com/mulgadc/spinifex/spinifex/handlers/eks"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// postReview drives authenticator.handle with a TokenReview carrying token and
+// audiences, returning the decoded response status and HTTP code.
+func postReview(t *testing.T, a *authenticator, token string, audiences []string) (tokenReviewStatus, int) {
+	t.Helper()
+	body, err := json.Marshal(tokenReview{
+		APIVersion: "authentication.k8s.io/v1",
+		Kind:       "TokenReview",
+		Spec:       tokenReviewSpec{Token: token, Audiences: audiences},
+	})
+	require.NoError(t, err)
+
+	req := httptest.NewRequest(http.MethodPost, "/authenticate", strings.NewReader(string(body)))
+	w := httptest.NewRecorder()
+	a.handle(w, req)
+
+	if w.Code != http.StatusOK {
+		return tokenReviewStatus{}, w.Code
+	}
+	var resp tokenReview
+	require.NoError(t, json.Unmarshal(w.Body.Bytes(), &resp))
+	return resp.Status, w.Code
+}
+
+func TestHandle_GrantsAndStampsAudiences(t *testing.T) {
+	a := &authenticator{
+		accountID:   "111122223333",
+		clusterName: "alpha",
+		review: func(string) (handlers_eks.WebhookTokenReviewResult, error) {
+			return handlers_eks.WebhookTokenReviewResult{
+				Authenticated: true,
+				Username:      "arn:aws:iam::111122223333:role/admin",
+				UID:           "AROAEXAMPLE:session",
+				Groups:        []string{"system:masters"},
+			}, nil
+		},
+	}
+
+	status, code := postReview(t, a, "tok", []string{"https://kubernetes.default.svc"})
+	require.Equal(t, http.StatusOK, code)
+	require.True(t, status.Authenticated)
+	assert.Equal(t, "arn:aws:iam::111122223333:role/admin", status.User.Username)
+	assert.Equal(t, "AROAEXAMPLE:session", status.User.UID)
+	assert.Equal(t, []string{"system:masters"}, status.User.Groups)
+	assert.Equal(t, []string{"https://kubernetes.default.svc"}, status.Audiences)
+}
+
+func TestHandle_DeniesWithoutStampingAudiences(t *testing.T) {
+	a := &authenticator{
+		clusterName: "alpha",
+		review: func(string) (handlers_eks.WebhookTokenReviewResult, error) {
+			return handlers_eks.WebhookTokenReviewResult{Authenticated: false}, nil
+		},
+	}
+
+	status, code := postReview(t, a, "tok", []string{"https://kubernetes.default.svc"})
+	require.Equal(t, http.StatusOK, code)
+	assert.False(t, status.Authenticated)
+	assert.Empty(t, status.User.Username)
+	assert.Empty(t, status.Audiences, "unauthenticated review must not stamp audiences")
+}
+
+// A broker fault is a 5xx (retryable), not a silent deny — a transient gateway
+// outage must not look identical to a rejected token.
+func TestHandle_BrokerFaultReturns5xx(t *testing.T) {
+	a := &authenticator{
+		clusterName: "alpha",
+		review: func(string) (handlers_eks.WebhookTokenReviewResult, error) {
+			return handlers_eks.WebhookTokenReviewResult{}, errors.New("gateway returned 503")
+		},
+	}
+
+	_, code := postReview(t, a, "tok", nil)
+	assert.Equal(t, http.StatusServiceUnavailable, code)
+}
+
+func TestEnsureServingCert_PersistsAndReuses(t *testing.T) {
+	dir := t.TempDir()
+	certPath := filepath.Join(dir, "wh.crt")
+	keyPath := filepath.Join(dir, "wh.key")
+
+	c1, pem1, err := ensureServingCert(certPath, keyPath)
+	require.NoError(t, err)
+	require.NotEmpty(t, pem1)
+	require.NotNil(t, c1.serverTLSConfig())
+
+	// Second call reuses the persisted pair (same cert bytes).
+	_, pem2, err := ensureServingCert(certPath, keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, pem1, pem2)
+}
+
+func TestWriteAPIServerKubeconfig(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "wh.kubeconfig")
+	certPEM := []byte("-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----\n")
+
+	require.NoError(t, writeAPIServerKubeconfig(path, "127.0.0.1:8443", certPEM))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	body := string(data)
+	assert.Contains(t, body, "server: https://127.0.0.1:8443/authenticate")
+	assert.Contains(t, body, base64.StdEncoding.EncodeToString(certPEM))
+	assert.Contains(t, body, "current-context: webhook")
+}

cmd/eks-token-webhook/tokenreview.go

@@ -0,0 +1,34 @@
+package main
+
+// Minimal authentication.k8s.io/v1 TokenReview wire types. Hand-rolled to avoid
+// pulling k8s.io/api into the spinifex module for these few fields; the shapes
+// match what kube-apiserver sends and expects on the webhook contract.
+
+type tokenReview struct {
+	APIVersion string            `json:"apiVersion"`
+	Kind       string            `json:"kind"`
+	Spec       tokenReviewSpec   `json:"spec,omitzero"`
+	Status     tokenReviewStatus `json:"status"`
+}
+
+type tokenReviewSpec struct {
+	Token     string   `json:"token"`
+	Audiences []string `json:"audiences,omitempty"`
+}
+
+type tokenReviewStatus struct {
+	Authenticated bool     `json:"authenticated"`
+	User          userInfo `json:"user,omitzero"`
+	// Audiences echoes the request's spec.audiences on success. When the
+	// apiserver runs with --api-audiences, it sends spec.audiences and discards
+	// an authenticated=true response that does not confirm a matching audience —
+	// so an empty Audiences here surfaces as a 401 despite a valid principal.
+	Audiences []string `json:"audiences,omitempty"`
+	Error     string   `json:"error,omitempty"`
+}
+
+type userInfo struct {
+	Username string   `json:"username,omitempty"`
+	UID      string   `json:"uid,omitempty"`
+	Groups   []string `json:"groups,omitempty"`
+}

cmd/installer/install/install.go

@@ -304,9 +304,10 @@ func installSpinifex(cfg *Config) error {
 		return err
 	}
 
-	// dhcpcd-base is present on the installed system (used by setup-ovn.sh for
-	// macvlan mode). Mask the standalone dhcpcd.service so it never auto-starts
-	// and races with systemd-networkd's built-in DHCP client on br-wan.
+	// dhcpcd-base is present on the installed system (used by setup-ovn.sh
+	// for WAN DHCP acquisition). Mask the standalone dhcpcd.service so it
+	// never auto-starts and races with systemd-networkd's built-in DHCP
+	// client on br-wan.
 	if err := maskSystemdUnit(mountRoot, "dhcpcd.service"); err != nil {
 		slog.Warn("installSpinifex: failed to mask dhcpcd.service", "err", err)
 	}

cmd/spinifex/cmd/admin.go

@@ -115,6 +115,16 @@ Each phase waits for all nodes to ACK before proceeding to the next.`,
 	Run: runClusterShutdown,
 }
 
+var clusterDrainDHCPCmd = &cobra.Command{
+	Use:   "drain-dhcp",
+	Short: "Release all upstream DHCP leases held by vpcd",
+	Long: `Ask each vpcd to DHCPRELEASE every external-pool DHCP lease it currently
+holds, returning them to the upstream DHCP server. Run this on teardown before
+stopping services: an env reset otherwise strands held leases on the upstream
+server until their TTL expires, eventually exhausting the upstream scope.`,
+	Run: runClusterDrainDHCP,
+}
+
 var adminInitCmd = &cobra.Command{
 	Use:   "init",
 	Short: "Initialize Spinifex platform configuration",
@@ -245,6 +255,9 @@ func init() {
 	clusterShutdownCmd.Flags().Duration("timeout", 120*time.Second, "Maximum time to wait per phase")
 	clusterShutdownCmd.Flags().Bool("dry-run", false, "Print phase plan without executing")
 
+	clusterCmd.AddCommand(clusterDrainDHCPCmd)
+	clusterDrainDHCPCmd.Flags().Duration("timeout", 30*time.Second, "Reply-collection window for vpcd drain responders")
+
 	adminCmd.AddCommand(imagesCmd)
 	imagesCmd.AddCommand(imagesImportCmd)
 	imagesCmd.AddCommand(imagesListCmd)
@@ -294,12 +307,14 @@ func init() {
 	adminInitCmd.Flags().String("external-mode", "", "External network mode: 'pool' (default when WAN detected) or '' (disabled)")
 	adminInitCmd.Flags().String("external-iface", "", "WAN NIC for br-external (auto-detected from default route)")
 	adminInitCmd.Flags().String("external-source", "", "Pool IP source: 'dhcp' (default when no --external-pool) or 'static' (uses --external-pool range)")
+	adminInitCmd.Flags().String("external-bind-bridge", "", "Linux bridge for upstream DHCP DORA (default 'br-wan' when --external-source=dhcp)")
 	adminInitCmd.Flags().String("external-pool", "", "External IP pool range as start-end (e.g., 192.168.1.150-192.168.1.250)")
 	adminInitCmd.Flags().String("external-gateway", "", "WAN gateway IP (auto-detected from default route)")
 	adminInitCmd.Flags().String("gateway-ip", "", "OVN gateway router's external IP for SNAT (default: pool range_start for pool mode, required for nat mode without DHCP)")
 	adminInitCmd.Flags().Int("external-prefix-len", 24, "External pool subnet prefix length (auto-detected)")
 	adminInitCmd.Flags().Bool("no-external", false, "Disable external networking (overlay-only, no internet for VMs)")
 	adminInitCmd.Flags().Bool("gpu-passthrough", false, "Enable VFIO GPU passthrough (sets gpu_passthrough = true in daemon config)")
+	adminInitCmd.Flags().Bool("ipsec", true, "Encrypt intra-AZ Geneve via OVN native IPsec (cluster-wide); disable only for trusted single-rack lab")
 
 	// Flags for admin join
 	adminJoinCmd.Flags().String("region", "ap-southeast-2", "Region for this node")
@@ -323,6 +338,7 @@ func init() {
 	imagesImportCmd.Flags().String("tmp-dir", os.TempDir(), "Temporary directory for image import processing")
 
 	imagesImportCmd.Flags().String("name", "", "Import specified image by name")
+	imagesImportCmd.Flags().String("ami-name", "", "Override the registered AMI name (DescribeImages name). Defaults to ami-{distro}-{version}-{arch}. Use for locally-built appliances (e.g. --ami-name spinifex-eks-node).")
 	imagesImportCmd.Flags().String("file", "", "Import file from specified path (raw, qcow2, compressed)")
 	imagesImportCmd.Flags().String("distro", "", "Specified distro name (e.g debian)")
 	imagesImportCmd.Flags().String("version", "", "Specified distro version (e.g 12)")
@@ -378,6 +394,7 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	// and the file is used directly (no download). When only --file is set,
 	// metadata comes from flags and no catalog tags are applied.
 	imageName, _ := cmd.Flags().GetString("name")
+	amiNameOverride, _ := cmd.Flags().GetString("ami-name")
 	localFile, _ := cmd.Flags().GetString("file")
 
 	if imageName == "" && localFile == "" {
@@ -533,6 +550,9 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	// Calculate the size
 
 	manifest.AMIMetadata.Name = fmt.Sprintf("ami-%s-%s-%s", image.Distro, image.Version, image.Arch)
+	if amiNameOverride != "" {
+		manifest.AMIMetadata.Name = amiNameOverride
+	}
 	volumeId := utils.GenerateResourceID("ami")
 	manifest.AMIMetadata.ImageID = volumeId
 
@@ -601,6 +621,12 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 		Host:       appConfig.Nodes[appConfig.Node].Predastore.Host,
 	}
 
+	mkey, err := utils.LoadViperblockMasterKey(appConfig.Nodes[appConfig.Node].Viperblock.EncryptionKeyFile)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Could not load viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+
 	vbConfig := viperblock.VB{
 		VolumeName: volumeId,
 		VolumeSize: utils.SafeInt64ToUint64(imageStat.Size()),
@@ -610,7 +636,9 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 				Size: 0,
 			},
 		},
-		VolumeConfig: manifest,
+		VolumeConfig:      manifest,
+		MasterKey:         mkey,
+		EncryptionEnabled: mkey != nil,
 	}
 
 	err = v_utils.ImportDiskImage(&s3Config, &vbConfig, extractedImagePath)
@@ -838,12 +866,14 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	externalMode, _ := cmd.Flags().GetString("external-mode")
 	externalIface, _ := cmd.Flags().GetString("external-iface")
 	externalSource, _ := cmd.Flags().GetString("external-source")
+	externalBindBridge, _ := cmd.Flags().GetString("external-bind-bridge")
 	externalPool, _ := cmd.Flags().GetString("external-pool")
 	externalGateway, _ := cmd.Flags().GetString("external-gateway")
 	externalPrefixLen, _ := cmd.Flags().GetInt("external-prefix-len")
 	gatewayIP, _ := cmd.Flags().GetString("gateway-ip")
 	noExternal, _ := cmd.Flags().GetBool("no-external")
 	gpuPassthrough, _ := cmd.Flags().GetBool("gpu-passthrough")
+	ipsecEnabled, _ := cmd.Flags().GetBool("ipsec")
 
 	// Fire telemetry in background (completes during init work, waited at end)
 	noTelemetry, _ := cmd.Flags().GetBool("no-telemetry")
@@ -893,9 +923,9 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 				fmt.Printf("  %-14s %-18s %-20s %-16s %s\n", iface.Name, iface.IP, iface.Subnet, gw, strings.ToUpper(iface.Role))
 			}
 			if detected.LANCount == 0 {
-				fmt.Println("\n  Mode: single-NIC (macvlan for external bridge)")
+				fmt.Println("\n  Mode: single-NIC (veth-bridged external)")
 			} else {
-				fmt.Printf("\n  Mode: %d LAN + 1 WAN (macvlan for external bridge)\n", detected.LANCount)
+				fmt.Printf("\n  Mode: %d LAN + 1 WAN (veth-bridged external)\n", detected.LANCount)
 			}
 
 			// Apply auto-detected values when flags not explicitly set
@@ -910,13 +940,11 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 					externalPrefixLen = detected.WAN.PrefixLen
 				}
 
-				// Default mode: always "pool" — with static range if --external-pool
-				// is given, otherwise with source=dhcp (gateway IP from router DHCP)
+				// Default mode: always "pool". Source defaults to "static"; if
+				// --external-pool is omitted the validator below will error with
+				// a SuggestPoolRange hint.
 				if externalMode == "" && !cmd.Flags().Changed("external-mode") {
 					externalMode = "pool"
-					if externalPool == "" && externalSource == "" {
-						externalSource = "dhcp"
-					}
 				}
 			}
 		}
@@ -928,25 +956,30 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		os.Exit(1)
 	}
 	if externalMode == "pool" {
-		// Resolve source: if not explicitly set, infer from whether a pool range was given
+		// Default source: dhcp when no --external-pool, else static.
 		if externalSource == "" {
-			if externalPool != "" {
-				externalSource = "static"
-			} else {
+			if externalPool == "" {
 				externalSource = "dhcp"
+			} else {
+				externalSource = "static"
 			}
 		}
-		if externalSource == "dhcp" {
-			// DHCP source: no static range needed, just gateway
-			if externalGateway == "" {
-				fmt.Fprintf(os.Stderr, "❌ Error: --external-gateway is required when --external-mode=pool\n")
+		switch externalSource {
+		case "dhcp":
+			if externalPool != "" {
+				fmt.Fprintf(os.Stderr, "❌ Error: --external-pool not allowed with --external-source=dhcp (addresses come from upstream DHCP server)\n")
+				os.Exit(1)
+			}
+			if externalBindBridge == "" {
+				externalBindBridge = "br-wan"
+			}
+		case "static":
+			if externalBindBridge != "" {
+				fmt.Fprintf(os.Stderr, "❌ Error: --external-bind-bridge only valid with --external-source=dhcp\n")
 				os.Exit(1)
 			}
-		} else {
-			// Static source: need pool range
 			if externalPool == "" {
-				fmt.Fprintf(os.Stderr, "❌ Error: --external-pool is required when --external-mode=pool (e.g., 192.168.1.150-192.168.1.250)\n")
-				fmt.Fprintf(os.Stderr, "   Or use --external-source=dhcp to get IPs from router DHCP\n")
+				fmt.Fprintf(os.Stderr, "❌ Error: --external-pool is required with --external-source=static (e.g., 192.168.1.150-192.168.1.250)\n")
 				if detectedNet != nil && detectedNet.WAN != nil {
 					sugStart, sugEnd := admin.SuggestPoolRange(detectedNet.WAN)
 					fmt.Fprintf(os.Stderr, "   Suggested: --external-pool=%s-%s\n", sugStart, sugEnd)
@@ -954,7 +987,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 				os.Exit(1)
 			}
 			if externalGateway == "" {
-				fmt.Fprintf(os.Stderr, "❌ Error: --external-gateway is required when --external-mode=pool\n")
+				fmt.Fprintf(os.Stderr, "❌ Error: --external-gateway is required with --external-source=static\n")
 				os.Exit(1)
 			}
 			parts := strings.SplitN(externalPool, "-", 2)
@@ -963,6 +996,9 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 				os.Exit(1)
 			}
 			poolStart, poolEnd = parts[0], parts[1]
+		default:
+			fmt.Fprintf(os.Stderr, "❌ Error: --external-source must be 'static' or 'dhcp', got: %s\n", externalSource)
+			os.Exit(1)
 		}
 	}
 	if externalGateway != "" && net.ParseIP(externalGateway) == nil {
@@ -983,10 +1019,6 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		}
 	}
 
-	// For pool/dhcp source, the gateway IP is obtained via DHCP on the
-	// macvlan/bridge interface (no static pool range or explicit gateway-ip).
-	useExternalDHCP := externalMode == "pool" && externalSource == "dhcp" && gatewayIP == ""
-
 	// Validate IP address format
 	if net.ParseIP(bindIP) == nil {
 		fmt.Fprintf(os.Stderr, "❌ Error: Invalid IP address for --bind: %s\n", bindIP)
@@ -1095,6 +1127,17 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	fmt.Println("\n🔐 Generated predastore encryption key (per-node, never transmitted)")
 	fmt.Printf("   Key: %s\n", predastoreKeyPath)
 
+	// Viperblock at-rest encryption key is cluster-wide; on a single-node init
+	// there are no joiners, so generate it locally and enable encryption by
+	// default for all volumes created on this install.
+	viperblockKeyPath, err := writeViperblockEncryptionKey(configDir)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error generating viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	fmt.Println("\n🔐 Generated viperblock at-rest encryption key")
+	fmt.Printf("   Key: %s\n", viperblockKeyPath)
+
 	fmt.Printf("\n🔑 Generated admin credentials (save these — they won't be shown again):\n")
 	fmt.Printf("   Access Key:  %s\n", bootstrapResult.AdminAccessKey)
 	fmt.Printf("   Secret Key:  %s\n", bootstrapResult.AdminSecretKey)
@@ -1104,6 +1147,18 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	// Generate SSL certificates (with bind IP in SANs for multi-node support)
 	certPath := admin.GenerateCertificatesIfNeeded(configDir, force, bindIP)
 
+	// Generate per-node IPsec peer cert when cluster-wide IPsec is enabled
+	// (default true). Reuses the cluster CA — no intermediate strongSwan PKI.
+	if ipsecEnabled {
+		caCertPath := filepath.Join(configDir, "ca.pem")
+		caKeyPath := filepath.Join(configDir, "ca.key")
+		if err := admin.GenerateIPSecPeerCert(configDir, caCertPath, caKeyPath, node, bindIP); err != nil {
+			fmt.Fprintf(os.Stderr, "Error generating IPsec peer certificate: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Println("🔐 IPsec peer certificate generated (intra-AZ Geneve encryption ON)")
+	}
+
 	// Install CA certificate into system trust store
 	installCACertificate(filepath.Join(configDir, "ca.pem"))
 
@@ -1125,30 +1180,32 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	isMultiNode := nodes >= 2
 
 	if isMultiNode {
-		// Build cluster-wide network config for propagation to joining nodes
-		var networkConfig *formation.NetworkConfig
+		// Build cluster-wide network config for propagation to joining nodes.
+		// Always emit so the IPSecEnabled flag reaches joiners even when
+		// external networking is disabled.
+		networkConfig := &formation.NetworkConfig{
+			IPSecEnabled: ipsecEnabled,
+		}
 		if externalMode != "" {
 			bootstrapVpcId := utils.GenerateResourceID("vpc")
 			bootstrapSubnetId := utils.GenerateResourceID("subnet")
 			bootstrapIgwId := utils.GenerateResourceID("igw")
-			networkConfig = &formation.NetworkConfig{
-				ExternalMode:        externalMode,
-				ExternalDHCP:        useExternalDHCP,
-				PoolName:            "wan",
-				PoolSource:          externalSource,
-				PoolStart:           poolStart,
-				PoolEnd:             poolEnd,
-				PoolGateway:         externalGateway,
-				PoolGatewayIP:       gatewayIP,
-				PoolPrefixLen:       externalPrefixLen,
-				PoolDNSServers:      dnsServers,
-				BootstrapAccountId:  admin.DefaultAccountID(),
-				BootstrapVpcId:      bootstrapVpcId,
-				BootstrapSubnetId:   bootstrapSubnetId,
-				BootstrapIgwId:      bootstrapIgwId,
-				BootstrapCidr:       handlers_ec2_vpc.DefaultVPCCidr,
-				BootstrapSubnetCidr: handlers_ec2_vpc.DefaultSubnetCidr,
-			}
+			networkConfig.ExternalMode = externalMode
+			networkConfig.PoolName = "wan"
+			networkConfig.PoolSource = externalSource
+			networkConfig.PoolBindBridge = externalBindBridge
+			networkConfig.PoolStart = poolStart
+			networkConfig.PoolEnd = poolEnd
+			networkConfig.PoolGateway = externalGateway
+			networkConfig.PoolGatewayIP = gatewayIP
+			networkConfig.PoolPrefixLen = externalPrefixLen
+			networkConfig.PoolDNSServers = dnsServers
+			networkConfig.BootstrapAccountId = admin.DefaultAccountID()
+			networkConfig.BootstrapVpcId = bootstrapVpcId
+			networkConfig.BootstrapSubnetId = bootstrapSubnetId
+			networkConfig.BootstrapIgwId = bootstrapIgwId
+			networkConfig.BootstrapCidr = handlers_ec2_vpc.DefaultVPCCidr
+			networkConfig.BootstrapSubnetCidr = handlers_ec2_vpc.DefaultSubnetCidr
 		}
 
 		runAdminInitMultiNode(cmd, accessKey, secretKey, accountID, natsToken, clusterName,
@@ -1248,10 +1305,9 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 
 		ExternalMode:   externalMode,
 		ExternalIface:  externalIface,
-		DhcpBindBridge: detectedDhcpBindBridge(detectedNet),
-		ExternalDHCP:   useExternalDHCP,
 		PoolName:       "wan",
 		PoolSource:     externalSource,
+		PoolBindBridge: externalBindBridge,
 		PoolStart:      poolStart,
 		PoolEnd:        poolEnd,
 		PoolGateway:    externalGateway,
@@ -1268,20 +1324,25 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		BootstrapSubnetCidr: handlers_ec2_vpc.DefaultSubnetCidr,
 
 		GPUPassthrough: gpuPassthrough,
+		IPSecEnabled:   ipsecEnabled,
+
+		EncryptionKeyFile: viperblockKeyPath,
 	}
 
 	// Print external networking summary
 	if externalMode != "" {
 		fmt.Printf("\n📡 External networking: %s\n", externalMode)
 		fmt.Printf("  WAN interface: %s\n", externalIface)
-		if poolStart != "" {
+		switch externalSource {
+		case "static":
 			fmt.Printf("  Source:        static (IP range)\n")
 			fmt.Printf("  IP pool:       %s - %s\n", poolStart, poolEnd)
 			fmt.Printf("  ⚠️  Ensure %s-%s is excluded from your router's DHCP range.\n", poolStart, poolEnd)
-		} else if useExternalDHCP {
-			fmt.Printf("  Source:        dhcp (from router DHCP)\n")
-			fmt.Println("  Gateway IP:    DHCP (obtained during OVN setup)")
-		} else if gatewayIP != "" {
+		case "dhcp":
+			fmt.Printf("  Source:        dhcp (upstream DHCP server)\n")
+			fmt.Printf("  Bind bridge:   %s\n", externalBindBridge)
+		}
+		if gatewayIP != "" {
 			fmt.Printf("  Gateway IP:    %s (static)\n", gatewayIP)
 		}
 	}
@@ -1374,6 +1435,22 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	fmt.Println("\n🔐 Generated predastore encryption key (per-node, never transmitted)")
 	fmt.Printf("   Key: %s\n", predastoreKeyPath)
 
+	// Viperblock at-rest encryption key is cluster-wide and shared: the leader
+	// generates it once and distributes it to joiners via the formation server
+	// so a volume sealed on any node can be opened on any other.
+	viperblockKey, err := handlers_iam.GenerateMasterKey()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "❌ Error generating viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	viperblockKeyPath, err := saveViperblockEncryptionKey(configDir, viperblockKey)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "❌ Error saving viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	fmt.Println("\n🔐 Generated viperblock at-rest encryption key (cluster-wide, shared with joiners)")
+	fmt.Printf("   Key: %s\n", viperblockKeyPath)
+
 	// Read CA cert/key for distribution to joining nodes
 	caCertData, err := os.ReadFile(filepath.Join(configDir, "ca.pem"))
 	if err != nil {
@@ -1402,6 +1479,9 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	// Include master key in formation server for distribution to joining nodes
 	fs.SetMasterKey(base64.StdEncoding.EncodeToString(masterKey))
 
+	// Distribute the cluster-wide viperblock encryption key to joiners
+	fs.SetViperblockKey(base64.StdEncoding.EncodeToString(viperblockKey))
+
 	// Register self (init node) as the first node
 	selfNode := formation.NodeInfo{
 		Name:        node,
@@ -1509,6 +1589,8 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 
 		OperatorEmail: email,
 
+		EncryptionKeyFile: viperblockKeyPath,
+
 		// Init node runs ovn-central locally
 		OVNNBAddr: "tcp:127.0.0.1:6641",
 		OVNSBAddr: "tcp:127.0.0.1:6642",
@@ -1550,7 +1632,7 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	}
 	fmt.Println("\n📋 Next steps:")
 	fmt.Println("   1. Start services on ALL nodes:")
-	fmt.Println("      ./scripts/start-dev.sh")
+	fmt.Println("      sudo systemctl start spinifex.target")
 	fmt.Println()
 }
 
@@ -1854,6 +1936,28 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 	}
 	fmt.Printf("✅ Predastore encryption key generated: %s\n", predastoreKeyPath)
 
+	// Viperblock at-rest encryption key is cluster-wide: receive it from the
+	// leader rather than generating one, so this node can open volumes sealed
+	// elsewhere. Lenient on absence — an older leader that predates key
+	// distribution returns nothing; fall back to cleartext rather than fail
+	// the join.
+	var viperblockKeyPath string
+	if statusResp.ViperblockKey == "" {
+		fmt.Println("⚠️  Leader did not provide a viperblock encryption key; at-rest encryption disabled on this node")
+	} else {
+		viperblockKeyBytes, err := base64.StdEncoding.DecodeString(statusResp.ViperblockKey)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "❌ Error decoding viperblock encryption key: %v\n", err)
+			os.Exit(1)
+		}
+		viperblockKeyPath, err = saveViperblockEncryptionKey(configDir, viperblockKeyBytes)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "❌ Error saving viperblock encryption key: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Printf("✅ Viperblock encryption key received from leader: %s\n", viperblockKeyPath)
+	}
+
 	// Generate server cert signed by CA with this node's bind IP
 	if err := admin.GenerateServerCertOnly(configDir, bindIP); err != nil {
 		fmt.Fprintf(os.Stderr, "Error generating server certificate: %v\n", err)
@@ -1861,6 +1965,24 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 	}
 	fmt.Printf("✅ Server certificate generated with bind IP: %s\n\n", bindIP)
 
+	// Match the leader's intra-AZ IPsec posture: NetworkConfig.IPSecEnabled is
+	// authoritative. When the formation response omits NetworkConfig entirely
+	// (no current code path; defensive guard against future regressions) fall
+	// back to the AWS-parity default of ON.
+	ipsecEnabled := true
+	if statusResp.NetworkConfig != nil {
+		ipsecEnabled = statusResp.NetworkConfig.IPSecEnabled
+	}
+	if ipsecEnabled {
+		caCertPath := filepath.Join(configDir, "ca.pem")
+		caKeyPath := filepath.Join(configDir, "ca.key")
+		if err := admin.GenerateIPSecPeerCert(configDir, caCertPath, caKeyPath, node, bindIP); err != nil {
+			fmt.Fprintf(os.Stderr, "Error generating IPsec peer certificate: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Println("🔐 IPsec peer certificate generated (intra-AZ Geneve encryption ON)")
+	}
+
 	// Build cluster topology from formation data
 	clusterRoutes := formation.BuildClusterRoutes(statusResp.Nodes)
 	predastoreNodes := formation.BuildPredastoreNodes(statusResp.Nodes)
@@ -1927,6 +2049,8 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 
 		OperatorEmail: email,
 
+		EncryptionKeyFile: viperblockKeyPath,
+
 		// Joining nodes connect to the init node's OVN NB/SB DB
 		OVNNBAddr: fmt.Sprintf("tcp:%s:6641", leaderIP),
 		OVNSBAddr: fmt.Sprintf("tcp:%s:6642", leaderIP),
@@ -2212,6 +2336,7 @@ func runCertRenew(cmd *cobra.Command, _ []string) {
 type configDirs struct {
 	AWSGW      string
 	Predastore string
+	Viperblock string
 	NATS       string
 	Spinifex   string
 }
@@ -2221,10 +2346,11 @@ func createConfigSubdirs(configDir string) (configDirs, error) {
 	dirs := configDirs{
 		AWSGW:      filepath.Join(configDir, "awsgw"),
 		Predastore: filepath.Join(configDir, "predastore"),
+		Viperblock: filepath.Join(configDir, "viperblock"),
 		NATS:       filepath.Join(configDir, "nats"),
 		Spinifex:   filepath.Join(configDir, "spinifex"),
 	}
-	for _, dir := range []string{dirs.AWSGW, dirs.Predastore, dirs.NATS, dirs.Spinifex} {
+	for _, dir := range []string{dirs.AWSGW, dirs.Predastore, dirs.Viperblock, dirs.NATS, dirs.Spinifex} {
 		if err := os.MkdirAll(dir, 0700); err != nil {
 			return configDirs{}, fmt.Errorf("create directory %s: %w", dir, err)
 		}
@@ -2270,10 +2396,11 @@ func finalizeNodeSetup(dataDir, certPath, adminAccessKey, adminSecretKey, region
 // applyNetworkConfig copies cluster-wide network settings from a formation
 // NetworkConfig into ConfigSettings and auto-detects the local WAN interface.
 func applyNetworkConfig(settings *admin.ConfigSettings, nc *formation.NetworkConfig) {
+	settings.IPSecEnabled = nc.IPSecEnabled
 	settings.ExternalMode = nc.ExternalMode
-	settings.ExternalDHCP = nc.ExternalDHCP
 	settings.PoolName = nc.PoolName
 	settings.PoolSource = nc.PoolSource
+	settings.PoolBindBridge = nc.PoolBindBridge
 	settings.PoolStart = nc.PoolStart
 	settings.PoolEnd = nc.PoolEnd
 	settings.PoolGateway = nc.PoolGateway
@@ -2292,7 +2419,6 @@ func applyNetworkConfig(settings *admin.ConfigSettings, nc *formation.NetworkCon
 		detected, err := admin.DetectNetwork()
 		if err == nil && detected.WAN != nil {
 			settings.ExternalIface = detected.WAN.Name
-			settings.DhcpBindBridge = detectedDhcpBindBridge(detected)
 		}
 	}
 }
@@ -2346,6 +2472,43 @@ func writePredastoreEncryptionKey(configDir string) (string, error) {
 	return keyPath, nil
 }
 
+// writeViperblockEncryptionKey generates a fresh 32-byte AES-256 master key for
+// viperblock at-rest encryption and writes it to
+// <configDir>/viperblock/encryption.key with mode 0600. Unlike the predastore
+// key, this key is cluster-wide and shared: the leader generates it once at init
+// and distributes it to joiners via the formation server, so a volume sealed on
+// any node can be opened on any other. Loaded at startup via masterkey.LoadShared.
+func writeViperblockEncryptionKey(configDir string) (string, error) {
+	viperblockDir := filepath.Join(configDir, "viperblock")
+	if err := os.MkdirAll(viperblockDir, 0750); err != nil {
+		return "", fmt.Errorf("create viperblock config dir: %w", err)
+	}
+	key, err := handlers_iam.GenerateMasterKey()
+	if err != nil {
+		return "", fmt.Errorf("generate viperblock encryption key: %w", err)
+	}
+	keyPath := filepath.Join(viperblockDir, "encryption.key")
+	if err := handlers_iam.SaveMasterKey(keyPath, key); err != nil {
+		return "", fmt.Errorf("save viperblock encryption key: %w", err)
+	}
+	return keyPath, nil
+}
+
+// saveViperblockEncryptionKey writes an already-generated 32-byte viperblock
+// master key (received from the formation leader) to
+// <configDir>/viperblock/encryption.key with mode 0600.
+func saveViperblockEncryptionKey(configDir string, key []byte) (string, error) {
+	viperblockDir := filepath.Join(configDir, "viperblock")
+	if err := os.MkdirAll(viperblockDir, 0750); err != nil {
+		return "", fmt.Errorf("create viperblock config dir: %w", err)
+	}
+	keyPath := filepath.Join(viperblockDir, "encryption.key")
+	if err := handlers_iam.SaveMasterKey(keyPath, key); err != nil {
+		return "", fmt.Errorf("save viperblock encryption key: %w", err)
+	}
+	return keyPath, nil
+}
+
 // writeSystemCredentials writes the system access key to a plaintext JSON file.
 // The daemon reads this at startup to inject credentials into ALB VM cloud-init
 // for SigV4-authenticated communication with the AWS gateway.
@@ -2401,29 +2564,6 @@ func writeBootstrapFilesWithAdmin(configDir, bootstrapDir string, masterKey []by
 	return handlers_iam.SaveBootstrapData(filepath.Join(bootstrapDir, "bootstrap.json"), bd)
 }
 
-// detectedDhcpBindBridge returns the bridge name where the vpcd DHCP client
-// should bind its AF_PACKET socket — i.e. the interface that physically sees
-// LAN DHCP traffic.
-//
-//   - Default route on a bridge (Linux or OVS, `br-*` prefix): return the
-//     bridge name verbatim. In veth mode this is the Linux bridge holding the
-//     WAN NIC; in direct mode this is the OVS bridge holding the WAN NIC.
-//   - Default route on a bare physical NIC: return "br-wan" as the name the
-//     installer will create.
-//
-// Never returns "br-ext". br-ext is the OVN-side bridge owned by
-// setup-ovn.sh's ovn-bridge-mappings and never sees LAN DHCP frames.
-func detectedDhcpBindBridge(detected *admin.DetectedNetwork) string {
-	if detected == nil || detected.WAN == nil {
-		return ""
-	}
-	name := detected.WAN.Name
-	if strings.HasPrefix(name, "br-") {
-		return name
-	}
-	return "br-wan"
-}
-
 // detectDNSServers auto-detects DNS servers from the host for the specified
 // interface. Uses resolvectl (systemd-resolved) first, then falls back to
 // /etc/resolv.conf. Returns up to 3 servers. Falls back to public DNS if none found.

cmd/spinifex/cmd/admin_gpu.go

@@ -52,16 +52,62 @@ Must be run as root.`,
 	Run: runAdminGpuSetup,
 }
 
+var adminGpuMigCmd = &cobra.Command{
+	Use:   "mig",
+	Short: "Manage NVIDIA MIG partitioning on this node",
+}
+
+var adminGpuMigStatusCmd = &cobra.Command{
+	Use:   "status",
+	Short: "Show MIG capability, mode, and slice state for all GPUs on this node",
+	Run:   runAdminGpuMigStatus,
+}
+
+var adminGpuMigEnableCmd = &cobra.Command{
+	Use:   "enable",
+	Short: "Enable MIG mode on all MIG-capable GPUs (or a specific one)",
+	Long: `Enable NVIDIA MIG mode on all MIG-capable GPUs (or a specific one).
+
+Without --profile the daemon handles slice creation dynamically: each
+RunInstances call for a mig.* instance type carves the right profile on
+demand and destroys the slices when the last instance stops.
+
+With --profile, slices are pre-carved immediately. The daemon recovers
+these slices on restart and serves them as a static pool for that profile.
+
+Must be run as root.`,
+	Run: runAdminGpuMigEnable,
+}
+
+var adminGpuMigDisableCmd = &cobra.Command{
+	Use:   "disable",
+	Short: "Destroy all MIG instances and disable MIG mode",
+	Long: `Destroy all GPU instances and disable NVIDIA MIG mode on all MIG-capable
+GPUs (or a specific one). Blocked while GPU instances are running.
+
+Must be run as root.`,
+	Run: runAdminGpuMigDisable,
+}
+
 func init() {
 	adminCmd.AddCommand(adminGpuCmd)
 	adminGpuCmd.AddCommand(adminGpuStatusCmd)
 	adminGpuCmd.AddCommand(adminGpuEnableCmd)
 	adminGpuCmd.AddCommand(adminGpuDisableCmd)
 	adminGpuCmd.AddCommand(adminGpuSetupCmd)
+	adminGpuCmd.AddCommand(adminGpuMigCmd)
+	adminGpuMigCmd.AddCommand(adminGpuMigStatusCmd)
+	adminGpuMigCmd.AddCommand(adminGpuMigEnableCmd)
+	adminGpuMigCmd.AddCommand(adminGpuMigDisableCmd)
 
 	adminGpuStatusCmd.Flags().String("node", "", "Target node name (default: local node)")
-	// enable, disable, and setup write to the local spinifex.toml and signal the
-	// local daemon — they must be run directly on the target host.
+
+	adminGpuMigEnableCmd.Flags().String("profile", "", "MIG profile to pre-carve (e.g. \"1g.10gb\"); omit for fully dynamic slice management")
+	adminGpuMigEnableCmd.Flags().String("gpu", "", "PCI address of GPU to configure (default: all MIG-capable GPUs)")
+
+	adminGpuMigDisableCmd.Flags().String("gpu", "", "PCI address of GPU to configure (default: all MIG-capable GPUs)")
+	// enable, disable, setup, and mig subcommands write to the local spinifex.toml
+	// and signal the local daemon — they must be run directly on the target host.
 }
 
 // gpuNodeStatus queries NATS and returns the NodeStatusResponse for the target node.
@@ -417,6 +463,277 @@ func runAdminGpuSetup(_ *cobra.Command, _ []string) {
 	gpuToggle(nil, true)
 }
 
+func runAdminGpuMigStatus(_ *cobra.Command, _ []string) {
+	devices, err := gpu.Discover()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "GPU discovery failed: %v\n", err)
+		os.Exit(1)
+	}
+	if len(devices) == 0 {
+		fmt.Println("No GPUs detected on this node.")
+		return
+	}
+
+	anyMIG := false
+	for _, d := range devices {
+		if d.MIGCapable {
+			anyMIG = true
+		}
+		migMode := "N/A"
+		if d.MIGCapable {
+			if d.MIGEnabled {
+				migMode = "enabled"
+			} else {
+				migMode = "disabled"
+			}
+		}
+		fmt.Printf("GPU:           %s\n", d.PCIAddress)
+		fmt.Printf("  Model:       %s\n", d.Model)
+		fmt.Printf("  MIG capable: %v\n", d.MIGCapable)
+		fmt.Printf("  MIG mode:    %s\n", migMode)
+
+		if d.MIGEnabled {
+			instances, listErr := gpu.ListInstances(d.PCIAddress)
+			if listErr != nil {
+				fmt.Printf("  Slices:      (error: %v)\n", listErr)
+			} else {
+				fmt.Printf("  Slices:      %d\n", len(instances))
+				for _, inst := range instances {
+					fmt.Printf("    GI %d: %-12s  mdev: %s\n",
+						inst.GIID, inst.Profile.Name, inst.MdevPath)
+				}
+			}
+		}
+		fmt.Println()
+	}
+
+	if !anyMIG {
+		fmt.Println("No MIG-capable GPUs found. MIG requires an NVIDIA A100, A30, or H100 GPU.")
+	}
+}
+
+func runAdminGpuMigEnable(cmd *cobra.Command, _ []string) {
+	if os.Getuid() != 0 {
+		fmt.Fprintln(os.Stderr, "Error: spx admin gpu mig enable must be run as root.")
+		os.Exit(1)
+	}
+
+	profileName, _ := cmd.Flags().GetString("profile")
+	targetPCI, _ := cmd.Flags().GetString("gpu")
+
+	cfgPath := viper.GetString("config")
+	if cfgPath == "" {
+		cfgPath = DefaultConfigFile()
+	}
+	cfg, err := config.LoadConfig(cfgPath)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error loading config: %v\n", err)
+		os.Exit(1)
+	}
+
+	resp, err := gpuNodeStatus(cfg.Node)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		os.Exit(1)
+	}
+	if resp.AllocGPUs > 0 {
+		fmt.Fprintf(os.Stderr, "Error: %d GPU instance(s) are running. Terminate them first.\n", resp.AllocGPUs)
+		os.Exit(1)
+	}
+
+	devices, err := gpu.Discover()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "GPU discovery failed: %v\n", err)
+		os.Exit(1)
+	}
+
+	targets := migFilterTargets(devices, targetPCI)
+	if len(targets) == 0 {
+		if targetPCI != "" {
+			fmt.Fprintf(os.Stderr, "Error: no MIG-capable GPU found at PCI address %s\n", targetPCI)
+		} else {
+			fmt.Fprintln(os.Stderr, "Error: no MIG-capable GPUs found. MIG requires an NVIDIA A100, A30, or H100 GPU.")
+		}
+		os.Exit(1)
+	}
+
+	for _, d := range targets {
+		fmt.Printf("==> Configuring MIG on %s (%s)\n", d.PCIAddress, d.Model)
+
+		if !d.MIGEnabled {
+			fmt.Println("    Enabling MIG mode...")
+			if err := gpu.EnableMIGMode(d.PCIAddress); err != nil {
+				fmt.Fprintf(os.Stderr, "    Error: %v\n", err)
+				os.Exit(1)
+			}
+		} else {
+			fmt.Println("    MIG mode already enabled")
+		}
+
+		if profileName == "" {
+			// Dynamic mode: the daemon carves slices on demand at RunInstances time.
+			// Destroy any pre-carved slices so the daemon starts with a free GPU.
+			existing, _ := gpu.ListInstances(d.PCIAddress)
+			if len(existing) > 0 {
+				fmt.Printf("    Destroying %d pre-carved instance(s) so daemon can manage slices dynamically...\n", len(existing))
+				if err := gpu.DestroyAllInstances(d.PCIAddress); err != nil {
+					fmt.Fprintf(os.Stderr, "    Error: %v\n", err)
+					os.Exit(1)
+				}
+			}
+			fmt.Println("    Dynamic slice management enabled — use 'mig.*' instance types at RunInstances time.")
+			continue
+		}
+
+		// Static pre-carve: create a fixed set of slices for the given profile.
+		profiles, err := gpu.ListProfiles(d.PCIAddress)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "    Error listing profiles: %v\n", err)
+			os.Exit(1)
+		}
+
+		var selected *gpu.MIGProfile
+		for i, p := range profiles {
+			if p.Name == profileName {
+				selected = &profiles[i]
+				break
+			}
+		}
+		if selected == nil {
+			names := make([]string, len(profiles))
+			for i, p := range profiles {
+				names[i] = p.Name
+			}
+			fmt.Fprintf(os.Stderr, "    Error: profile %q not found. Available: %s\n",
+				profileName, strings.Join(names, ", "))
+			os.Exit(1)
+		}
+
+		existing, _ := gpu.ListInstances(d.PCIAddress)
+		if len(existing) > 0 {
+			fmt.Printf("    Destroying %d existing instance(s)...\n", len(existing))
+			if err := gpu.DestroyAllInstances(d.PCIAddress); err != nil {
+				fmt.Fprintf(os.Stderr, "    Error: %v\n", err)
+				os.Exit(1)
+			}
+		}
+
+		fmt.Printf("    Creating MIG instances (profile: %s)...\n", selected.Name)
+		instances, err := gpu.CreateInstances(d.PCIAddress, *selected)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "    Error: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Printf("    Created %d instance(s)\n", len(instances))
+		for _, inst := range instances {
+			fmt.Printf("      GI %d: %s  mdev: %s\n", inst.GIID, inst.Profile.Name, inst.MdevPath)
+		}
+	}
+
+	migReloadDaemon()
+	fmt.Println("MIG enabled. Run 'spx admin gpu mig status' to verify.")
+}
+
+func runAdminGpuMigDisable(cmd *cobra.Command, _ []string) {
+	if os.Getuid() != 0 {
+		fmt.Fprintln(os.Stderr, "Error: spx admin gpu mig disable must be run as root.")
+		os.Exit(1)
+	}
+
+	targetPCI, _ := cmd.Flags().GetString("gpu")
+
+	cfgPath := viper.GetString("config")
+	if cfgPath == "" {
+		cfgPath = DefaultConfigFile()
+	}
+	cfg, err := config.LoadConfig(cfgPath)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error loading config: %v\n", err)
+		os.Exit(1)
+	}
+
+	resp, err := gpuNodeStatus(cfg.Node)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		os.Exit(1)
+	}
+	if resp.AllocGPUs > 0 {
+		fmt.Fprintf(os.Stderr, "Error: %d GPU instance(s) are running. Terminate them first.\n", resp.AllocGPUs)
+		os.Exit(1)
+	}
+
+	devices, err := gpu.Discover()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "GPU discovery failed: %v\n", err)
+		os.Exit(1)
+	}
+
+	targets := migFilterTargets(devices, targetPCI)
+	if len(targets) == 0 {
+		if targetPCI != "" {
+			fmt.Fprintf(os.Stderr, "Error: no MIG-capable GPU found at PCI address %s\n", targetPCI)
+		} else {
+			fmt.Fprintln(os.Stderr, "Error: no MIG-capable GPUs found on this node.")
+		}
+		os.Exit(1)
+	}
+
+	for _, d := range targets {
+		fmt.Printf("==> Disabling MIG on %s (%s)\n", d.PCIAddress, d.Model)
+		if !d.MIGEnabled {
+			fmt.Println("    MIG mode not active — skipping")
+			continue
+		}
+		fmt.Println("    Destroying all GPU instances...")
+		if err := gpu.DestroyAllInstances(d.PCIAddress); err != nil {
+			fmt.Fprintf(os.Stderr, "    Error: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Println("    Disabling MIG mode...")
+		if err := gpu.DisableMIGMode(d.PCIAddress); err != nil {
+			fmt.Fprintf(os.Stderr, "    Error: %v\n", err)
+			os.Exit(1)
+		}
+	}
+
+	fmt.Println("==> Clearing mig_profile from config...")
+	if err := admin.SetMIGProfile(cfgPath, cfg.Node, ""); err != nil {
+		fmt.Fprintf(os.Stderr, "Error writing config: %v\n", err)
+		os.Exit(1)
+	}
+
+	migReloadDaemon()
+	fmt.Println("MIG disabled. GPUs will operate in whole-GPU passthrough mode.")
+}
+
+// migFilterTargets returns all MIG-capable GPUs, optionally filtered to a
+// specific PCI address.
+func migFilterTargets(devices []gpu.GPUDevice, targetPCI string) []gpu.GPUDevice {
+	var out []gpu.GPUDevice
+	for _, d := range devices {
+		if !d.MIGCapable {
+			continue
+		}
+		if targetPCI != "" && d.PCIAddress != targetPCI {
+			continue
+		}
+		out = append(out, d)
+	}
+	return out
+}
+
+// migReloadDaemon sends SIGHUP to spinifex-daemon so it picks up the updated
+// MIG configuration without a full restart.
+func migReloadDaemon() {
+	out, err := exec.Command("systemctl", "kill", "-s", "HUP", "spinifex-daemon").CombinedOutput()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: failed to signal daemon: %v\n%s\n", err, out)
+		fmt.Fprintln(os.Stderr, "Restart the daemon manually: sudo systemctl restart spinifex-daemon")
+		return
+	}
+	fmt.Println("==> Daemon signalled (SIGHUP) — MIG pool will be rebuilt on next request.")
+}
+
 // gpuSetupCollectVFIOIDs collects vendor:device PCI ID pairs for each GPU and
 // all sibling PCI functions on the same device slot (e.g. GPU + HDMI audio).
 // All siblings must bind to vfio-pci together for IOMMU group isolation.

cmd/spinifex/cmd/admin_test.go

@@ -142,49 +142,6 @@ func TestSpinifexTomlTemplate_AdvertiseOmittedWhenEmpty(t *testing.T) {
 	assert.NotContains(t, string(data), "advertise =")
 }
 
-// detectedDhcpBindBridge must return the default-route interface name verbatim
-// when it's a bridge (Linux or OVS, br-* prefix). The old detectedWanBridge()
-// returned hardcoded "br-ext" for Linux bridges — that value broke DHCP on
-// consumer-router LANs because br-ext never sees LAN DHCP traffic (mulga-998).
-func TestDetectedDhcpBindBridge(t *testing.T) {
-	tests := []struct {
-		name     string
-		detected *admin.DetectedNetwork
-		want     string
-	}{
-		{
-			name:     "Linux bridge default route returns bridge name (not 'br-ext')",
-			detected: &admin.DetectedNetwork{WAN: &admin.DetectedInterface{Name: "br-wan"}},
-			want:     "br-wan",
-		},
-		{
-			name:     "OVS bridge default route returned verbatim",
-			detected: &admin.DetectedNetwork{WAN: &admin.DetectedInterface{Name: "br-ext"}},
-			want:     "br-ext",
-		},
-		{
-			name:     "physical NIC defaults to 'br-wan' (bridge the installer creates)",
-			detected: &admin.DetectedNetwork{WAN: &admin.DetectedInterface{Name: "enp0s3"}},
-			want:     "br-wan",
-		},
-		{
-			name:     "nil DetectedNetwork",
-			detected: nil,
-			want:     "",
-		},
-		{
-			name:     "DetectedNetwork with nil WAN",
-			detected: &admin.DetectedNetwork{},
-			want:     "",
-		},
-	}
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			assert.Equal(t, tt.want, detectedDhcpBindBridge(tt.detected))
-		})
-	}
-}
-
 // Legacy `wan_bridge` TOML key must fail-start vpcd with guidance, not silently
 // alias (per mulga-998 D3). Prevents the footgun where operators inherited the
 // old key pointing at 'br-ext' and got broken DHCP on veth-mode hosts.
@@ -206,8 +163,8 @@ wan_bridge = "br-ext"
 
 	err := checkLegacyWanBridgeKey("node1", cfgPath)
 	require.Error(t, err)
-	assert.Contains(t, err.Error(), "dhcp_bind_bridge")
 	assert.Contains(t, err.Error(), "wan_bridge")
+	assert.Contains(t, err.Error(), "Remove")
 }
 
 func TestCheckLegacyWanBridgeKey_EnvVarRejected(t *testing.T) {
@@ -218,7 +175,7 @@ func TestCheckLegacyWanBridgeKey_EnvVarRejected(t *testing.T) {
 	err := checkLegacyWanBridgeKey("node1", "/tmp/unused.toml")
 	require.Error(t, err)
 	assert.Contains(t, err.Error(), "SPINIFEX_VPCD_WAN_BRIDGE")
-	assert.Contains(t, err.Error(), "dhcp_bind_bridge")
+	assert.Contains(t, err.Error(), "Remove")
 }
 
 func TestCheckLegacyWanBridgeKey_CleanConfigPasses(t *testing.T) {
@@ -228,7 +185,7 @@ func TestCheckLegacyWanBridgeKey_CleanConfigPasses(t *testing.T) {
 node = "node1"
 [nodes.node1.vpcd]
 ovn_nb_addr = "tcp:127.0.0.1:6641"
-dhcp_bind_bridge = "br-wan"
+external_interface = "enp0s3"
 `), 0o644))
 
 	viper.Reset()
@@ -295,6 +252,74 @@ func TestWritePredastoreEncryptionKey(t *testing.T) {
 	assert.NotEqual(t, key1, key2, "per-node keys must differ")
 }
 
+// The viperblock key gates at-rest encryption for every volume on a fresh
+// install. masterkey.LoadShared rejects anything that isn't exactly 32 bytes at
+// 0640-or-stricter, so a regression in size/mode would silently fall back to
+// cleartext (empty path) or fail service startup.
+func TestWriteViperblockEncryptionKey(t *testing.T) {
+	configDir := t.TempDir()
+
+	keyPath, err := writeViperblockEncryptionKey(configDir)
+	require.NoError(t, err)
+	assert.Equal(t, filepath.Join(configDir, "viperblock", "encryption.key"), keyPath)
+
+	info, err := os.Stat(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, int64(32), info.Size(), "viperblock master key must be exactly 32 bytes")
+	assert.Equal(t, os.FileMode(0600), info.Mode().Perm(), "viperblock master key must be mode 0600")
+}
+
+// Joiners persist the leader's shared key via saveViperblockEncryptionKey
+// against a configDir with no viperblock/ subdir yet. The bytes round-trip
+// verbatim — the whole cluster shares one key, so any mutation would orphan
+// volumes sealed on other nodes.
+func TestSaveViperblockEncryptionKey_RoundTrip(t *testing.T) {
+	configDir := t.TempDir()
+	key := make([]byte, 32)
+	for i := range key {
+		key[i] = byte(i)
+	}
+
+	keyPath, err := saveViperblockEncryptionKey(configDir, key)
+	require.NoError(t, err)
+	assert.Equal(t, filepath.Join(configDir, "viperblock", "encryption.key"), keyPath)
+
+	got, err := os.ReadFile(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, key, got, "shared key must be written verbatim")
+}
+
+// A fresh install must render encryption_key_file so viperblockd enables
+// at-rest encryption; an empty field (upgrade / legacy) must omit the line
+// entirely so the volume stays in cleartext legacy mode.
+func TestSpinifexTomlTemplate_EncryptionKeyFile(t *testing.T) {
+	base := admin.ConfigSettings{
+		Node: "node1", Az: "ap-southeast-2a", Port: "4432", Region: "ap-southeast-2",
+		BindIP: "0.0.0.0", AccessKey: "AKIATEST", SecretKey: "SECRET",
+		AccountID: "123456789012", NatsToken: "token",
+		OVNNBAddr: "tcp:127.0.0.1:6641", OVNSBAddr: "tcp:127.0.0.1:6642",
+	}
+
+	dir := t.TempDir()
+	withKey := base
+	withKey.ConfigDir = dir
+	withKey.EncryptionKeyFile = "/etc/spinifex/viperblock/encryption.key"
+	path := filepath.Join(dir, "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, withKey))
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	assert.Contains(t, string(data), `encryption_key_file = "/etc/spinifex/viperblock/encryption.key"`)
+
+	dir2 := t.TempDir()
+	noKey := base
+	noKey.ConfigDir = dir2
+	path2 := filepath.Join(dir2, "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path2, spinifexTomlTemplate, noKey))
+	data2, err := os.ReadFile(path2)
+	require.NoError(t, err)
+	assert.NotContains(t, string(data2), "encryption_key_file", "blank key must omit the field")
+}
+
 // Joiners run `spx admin join` against a configDir that doesn't yet contain
 // a predastore/ subdir. The helper must create it (predastore subdir is
 // owned by spinifex-storage in prod, but the helper just needs the dir to
@@ -331,3 +356,101 @@ func TestImagesRemoveCmd_FlagSchema(t *testing.T) {
 
 // cobraRequiredAnnotation is the annotation key cobra uses to mark required flags.
 const cobraRequiredAnnotation = "cobra_annotation_bash_completion_one_required_flag"
+
+// `spx admin init` must expose --external-source and --external-bind-bridge so
+// operators can pick the DHCP path for [[network.external_pools]].
+func TestAdminInitCmd_ExternalDHCPFlagSchema(t *testing.T) {
+	sourceFlag := adminInitCmd.Flags().Lookup("external-source")
+	require.NotNil(t, sourceFlag, "--external-source must be defined")
+	assert.Equal(t, "", sourceFlag.DefValue)
+
+	bindFlag := adminInitCmd.Flags().Lookup("external-bind-bridge")
+	require.NotNil(t, bindFlag, "--external-bind-bridge must be defined")
+	assert.Equal(t, "", bindFlag.DefValue)
+
+	poolFlag := adminInitCmd.Flags().Lookup("external-pool")
+	require.NotNil(t, poolFlag)
+	assert.Equal(t, "", poolFlag.DefValue)
+}
+
+// DHCP-sourced pool must emit source + bind_bridge into spinifex.toml so the
+// daemon's validator and vpcd's DHCPManager find the bridge they DORA on.
+func TestSpinifexTomlTemplate_ExternalPoolDHCPSource(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node:           "node1",
+		Az:             "ap-southeast-2a",
+		Port:           "4432",
+		Region:         "ap-southeast-2",
+		BindIP:         "10.11.12.1",
+		ConfigDir:      dir,
+		OVNNBAddr:      "tcp:127.0.0.1:6641",
+		OVNSBAddr:      "tcp:127.0.0.1:6642",
+		ExternalMode:   "pool",
+		ExternalIface:  "eth0",
+		PoolName:       "wan",
+		PoolSource:     "dhcp",
+		PoolBindBridge: "br-wan",
+		PoolPrefixLen:  24,
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	content := string(data)
+	assert.Contains(t, content, `source      = "dhcp"`)
+	assert.Contains(t, content, `bind_bridge = "br-wan"`)
+	assert.NotContains(t, content, "range_start", "dhcp pool must not emit range_start")
+	assert.NotContains(t, content, "range_end", "dhcp pool must not emit range_end")
+	assert.NotContains(t, content, "gateway     =", "gateway omitted for dhcp pool (discovered from OFFER)")
+}
+
+// Static-sourced pool must not emit bind_bridge — the validator rejects it.
+func TestSpinifexTomlTemplate_ExternalPoolStaticSource(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node:          "node1",
+		Az:            "ap-southeast-2a",
+		Port:          "4432",
+		Region:        "ap-southeast-2",
+		BindIP:        "10.11.12.1",
+		ConfigDir:     dir,
+		OVNNBAddr:     "tcp:127.0.0.1:6641",
+		OVNSBAddr:     "tcp:127.0.0.1:6642",
+		ExternalMode:  "pool",
+		ExternalIface: "eth0",
+		PoolName:      "wan",
+		PoolSource:    "static",
+		PoolStart:     "192.168.1.150",
+		PoolEnd:       "192.168.1.250",
+		PoolGateway:   "192.168.1.1",
+		PoolPrefixLen: 24,
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	content := string(data)
+	assert.Contains(t, content, `source      = "static"`)
+	assert.NotContains(t, content, "bind_bridge", "static pool must not emit bind_bridge")
+	assert.Contains(t, content, `range_start = "192.168.1.150"`)
+	assert.Contains(t, content, `gateway     = "192.168.1.1"`)
+}
+
+// Formation joiners must receive PoolBindBridge from the init node so the
+// cluster-wide config reaches every chassis intact.
+func TestApplyNetworkConfig_PropagatesPoolBindBridge(t *testing.T) {
+	settings := &admin.ConfigSettings{}
+	nc := &formation.NetworkConfig{
+		ExternalMode:   "pool",
+		PoolName:       "wan",
+		PoolSource:     "dhcp",
+		PoolBindBridge: "br-wan",
+		PoolPrefixLen:  24,
+	}
+	applyNetworkConfig(settings, nc)
+	assert.Equal(t, "dhcp", settings.PoolSource)
+	assert.Equal(t, "br-wan", settings.PoolBindBridge)
+}

cmd/spinifex/cmd/cluster.go

@@ -8,6 +8,7 @@ import (
 	"time"
 
 	"github.com/mulgadc/spinifex/spinifex/daemon"
+	"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
 	"github.com/nats-io/nats.go"
 	"github.com/spf13/cobra"
 )
@@ -160,6 +161,79 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
 	fmt.Printf("Cluster shutdown complete (%s)\n", time.Since(start).Round(time.Millisecond))
 }
 
+// runClusterDrainDHCP asks every vpcd to DHCPRELEASE all external-pool leases
+// it currently holds, returning them to the upstream DHCP server. Intended for
+// the teardown path: an env reset otherwise strands held leases until TTL
+// because vpcd's Stop() preserves them for adopt-on-reboot. Best-effort — if
+// the cluster is already partly down it warns and exits 0 so teardown proceeds.
+func runClusterDrainDHCP(cmd *cobra.Command, args []string) {
+	timeout, _ := cmd.Flags().GetDuration("timeout")
+	if timeout <= 0 {
+		timeout = 30 * time.Second
+	}
+
+	_, nc, err := loadConfigAndConnect()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: drain-dhcp could not connect (cluster may be down): %v\n", err)
+		return
+	}
+	defer nc.Close()
+
+	total, responders := drainDHCPLeases(nc, timeout)
+	fmt.Printf("DHCP drain complete: released %d lease(s) from %d vpcd responder(s)\n", total, responders)
+}
+
+// drainDHCPLeases publishes a drain request to vpc.dhcp.drain and collects
+// replies until timeout — one responder per AZ (vpcd uses a per-AZ queue
+// group). Returns the summed released count and the number of responders.
+func drainDHCPLeases(nc *nats.Conn, timeout time.Duration) (released, responders int) {
+	reqData, err := json.Marshal(struct{}{})
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		return 0, 0
+	}
+
+	inbox := nats.NewInbox()
+	sub, err := nc.SubscribeSync(inbox)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: drain-dhcp subscribe failed: %v\n", err)
+		return 0, 0
+	}
+	defer sub.Unsubscribe()
+
+	if err := nc.PublishRequest(dhcp.TopicDrain, inbox, reqData); err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: drain-dhcp publish failed: %v\n", err)
+		return 0, 0
+	}
+	nc.Flush()
+
+	deadline := time.Now().Add(timeout)
+	for {
+		remaining := time.Until(deadline)
+		if remaining <= 0 {
+			break
+		}
+		msg, err := sub.NextMsg(remaining)
+		if err != nil {
+			break
+		}
+		var reply struct {
+			Released int    `json:"released"`
+			Error    string `json:"error,omitempty"`
+		}
+		if err := json.Unmarshal(msg.Data, &reply); err != nil {
+			continue
+		}
+		responders++
+		if reply.Error != "" {
+			fmt.Fprintf(os.Stderr, "  vpcd drain error: %s\n", reply.Error)
+			continue
+		}
+		released += reply.Released
+	}
+	return released, responders
+}
+
 // collectShutdownACKs publishes a shutdown request and collects ACKs from nodes.
 func collectShutdownACKs(nc *nats.Conn, topic string, reqData []byte, nodeCount int, timeout time.Duration) ([]daemon.ShutdownACK, error) {
 	inbox := nats.NewInbox()

cmd/spinifex/cmd/cluster_test.go

@@ -0,0 +1,58 @@
+package cmd
+
+import (
+	"encoding/json"
+	"testing"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
+	"github.com/mulgadc/spinifex/spinifex/testutil"
+	"github.com/nats-io/nats.go"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// fakeDrainResponder subscribes a stand-in vpcd to vpc.dhcp.drain that
+// replies with the given released count (or error), mirroring
+// Manager.handleDrainMsg without a real lease store.
+func fakeDrainResponder(t *testing.T, nc *nats.Conn, released int, errMsg string) {
+	t.Helper()
+	sub, err := nc.Subscribe(dhcp.TopicDrain, func(msg *nats.Msg) {
+		body, _ := json.Marshal(struct {
+			Released int    `json:"released"`
+			Error    string `json:"error,omitempty"`
+		}{Released: released, Error: errMsg})
+		_ = msg.Respond(body)
+	})
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = sub.Unsubscribe() })
+}
+
+func TestDrainDHCPLeasesSumsReleased(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+	fakeDrainResponder(t, nc, 7, "")
+
+	released, responders := drainDHCPLeases(nc, 2*time.Second)
+	assert.Equal(t, 7, released)
+	assert.Equal(t, 1, responders)
+}
+
+func TestDrainDHCPLeasesReportsResponderError(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+	fakeDrainResponder(t, nc, 0, "list leases for drain: boom")
+
+	// An erroring responder still counts as a responder but contributes
+	// zero released leases.
+	released, responders := drainDHCPLeases(nc, 2*time.Second)
+	assert.Equal(t, 0, released)
+	assert.Equal(t, 1, responders)
+}
+
+func TestDrainDHCPLeasesNoResponders(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+
+	// No vpcd subscribed: the collection window elapses with zero replies.
+	released, responders := drainDHCPLeases(nc, 200*time.Millisecond)
+	assert.Equal(t, 0, released)
+	assert.Equal(t, 0, responders)
+}

cmd/spinifex/cmd/service.go

@@ -29,7 +29,7 @@ import (
 	"github.com/mulgadc/spinifex/spinifex/services/predastore"
 	"github.com/mulgadc/spinifex/spinifex/services/spinifexui"
 	"github.com/mulgadc/spinifex/spinifex/services/viperblockd"
-	"github.com/mulgadc/spinifex/spinifex/services/vpcd"
+	"github.com/mulgadc/spinifex/spinifex/vpcd"
 	"github.com/spf13/cobra"
 	"github.com/spf13/viper"
 )
@@ -283,19 +283,25 @@ var viperblockStartCmd = &cobra.Command{
 			shardWAL = *nodeConfig.Viperblock.ShardWAL
 		}
 
+		encryptionKeyFile := nodeConfig.Viperblock.EncryptionKeyFile
+		if envKey := viper.GetString("viperblock-encryption-key-file"); envKey != "" {
+			encryptionKeyFile = envKey
+		}
+
 		service, err := service.New("viperblock", &viperblockd.Config{
-			NatsHost:   nodeConfig.NATS.Host,
-			NatsToken:  nodeConfig.NATS.ACL.Token,
-			NatsCACert: nodeConfig.NATS.CACert,
-			PluginPath: pluginPath,
-			S3Host:     nodeConfig.Predastore.Host,
-			Bucket:     nodeConfig.Predastore.Bucket,
-			Region:     nodeConfig.Predastore.Region,
-			AccessKey:  nodeConfig.Predastore.AccessKey,
-			SecretKey:  nodeConfig.Predastore.SecretKey,
-			BaseDir:    nodeConfig.Predastore.BaseDir,
-			NodeName:   clusterConfig.Node,
-			ShardWAL:   shardWAL,
+			NatsHost:          nodeConfig.NATS.Host,
+			NatsToken:         nodeConfig.NATS.ACL.Token,
+			NatsCACert:        nodeConfig.NATS.CACert,
+			PluginPath:        pluginPath,
+			S3Host:            nodeConfig.Predastore.Host,
+			Bucket:            nodeConfig.Predastore.Bucket,
+			Region:            nodeConfig.Predastore.Region,
+			AccessKey:         nodeConfig.Predastore.AccessKey,
+			SecretKey:         nodeConfig.Predastore.SecretKey,
+			BaseDir:           nodeConfig.Predastore.BaseDir,
+			NodeName:          clusterConfig.Node,
+			ShardWAL:          shardWAL,
+			EncryptionKeyFile: encryptionKeyFile,
 		})
 
 		if err != nil {
@@ -532,7 +538,7 @@ var awsgwStartCmd = &cobra.Command{
 		awsgwHost := viper.GetString("host")
 		if awsgwHost != "" {
 			fmt.Println("Overwriting awsgw host to:", awsgwHost)
-			//nodeConfig.AWSGW.Host = awsgwHost
+			nodeConfig.AWSGW.Host = awsgwHost
 		}
 
 		awsgwTlsCert := viper.GetString("tls-cert")
@@ -678,11 +684,8 @@ var spinifexUIStatusCmd = &cobra.Command{
 }
 
 // checkLegacyWanBridgeKey fails vpcd startup if the deprecated `wan_bridge`
-// TOML key or `SPINIFEX_VPCD_WAN_BRIDGE` env-var is present. The key was
-// renamed to `dhcp_bind_bridge` (see mulga-998.a) because the old name misled
-// operators into pointing at the OVN-side bridge ("br-ext"), which never sees
-// LAN DHCP traffic. Per D3: no silent alias, no auto-rewrite — operator must
-// rename the key before vpcd will start.
+// TOML key or `SPINIFEX_VPCD_WAN_BRIDGE` env-var is present. Per D3: no silent
+// alias, no auto-rewrite — operator must remove the key before vpcd will start.
 func checkLegacyWanBridgeKey(node, cfgFile string) error {
 	legacyInTOML := viper.IsSet("nodes." + node + ".vpcd.wan_bridge")
 	legacyInEnv := os.Getenv("SPINIFEX_VPCD_WAN_BRIDGE") != ""
@@ -695,9 +698,7 @@ func checkLegacyWanBridgeKey(node, cfgFile string) error {
 	}
 	return fmt.Errorf(
 		"vpcd: deprecated 'wan_bridge' key found in %s. "+
-			"Rename 'wan_bridge' to 'dhcp_bind_bridge'. The value may also be wrong — verify it is "+
-			"the Linux bridge holding your WAN NIC (veth mode) or the OVS bridge holding your WAN NIC (direct mode); "+
-			"never 'br-ext'. Typical value on a consumer-router LAN: 'br-wan'. "+
+			"Remove the key entirely; vpcd auto-detects the WAN bridge (br-wan). "+
 			"Then: sudo systemctl restart spinifex-vpcd",
 		source,
 	)
@@ -734,6 +735,7 @@ var vpcdStartCmd = &cobra.Command{
 			extPools = append(extPools, vpcd.ExternalPoolConfig{
 				Name:            p.Name,
 				Source:          p.Source,
+				BindBridge:      p.BindBridge,
 				RangeStart:      p.RangeStart,
 				RangeEnd:        p.RangeEnd,
 				Gateway:         p.Gateway,
@@ -742,7 +744,6 @@ var vpcdStartCmd = &cobra.Command{
 				DNSServers:      p.DNSServers,
 				Region:          p.Region,
 				AZ:              p.AZ,
-				DhcpBindBridge:  nodeConfig.VPCD.DhcpBindBridge,
 				GwLrpRangeStart: p.GwLrpRangeStart,
 				GwLrpRangeEnd:   p.GwLrpRangeEnd,
 			})
@@ -778,8 +779,8 @@ var vpcdStartCmd = &cobra.Command{
 			ExternalPools:     extPools,
 			Bootstrap:         bootstrap,
 			ExternalInterface: nodeConfig.VPCD.ExternalInterface,
-			DhcpBindBridge:    nodeConfig.VPCD.DhcpBindBridge,
 			BridgeMode:        nodeConfig.VPCD.BridgeMode,
+			AZ:                nodeConfig.AZ,
 		})
 		if err != nil {
 			fmt.Println("Error starting vpcd service:", err)
@@ -916,6 +917,14 @@ func init() {
 	viper.BindEnv("plugin-path", "SPINIFEX_VIPERBLOCK_PLUGIN_PATH")
 	viper.BindPFlag("plugin-path", predastoreCmd.PersistentFlags().Lookup("plugin-path"))
 
+	// Viperblock at-rest encryption master key (shared with other on-node
+	// services via group ownership; mode 0640 or stricter). Distinct viper
+	// key from predastore's encryption-key-file so the two BindPFlag calls
+	// don't collide globally.
+	viperblockCmd.PersistentFlags().String("encryption-key-file", "", "Path to the 32-byte AES-256 master key file for at-rest encryption (empty disables encryption)")
+	viper.BindEnv("viperblock-encryption-key-file", "SPINIFEX_VIPERBLOCK_ENCRYPTION_KEY_FILE")
+	viper.BindPFlag("viperblock-encryption-key-file", viperblockCmd.PersistentFlags().Lookup("encryption-key-file"))
+
 	viperblockCmd.AddCommand(viperblockStartCmd)
 	viperblockCmd.AddCommand(viperblockStopCmd)
 	viperblockCmd.AddCommand(viperblockStatusCmd)

cmd/spinifex/cmd/templates/spinifex.toml

@@ -1,5 +1,5 @@
 # Domain configuration file in TOML format.
-version = "2"
+version = "3"
 epoch = 1
 node = "{{.Node}}"
 {{- if .OperatorEmail}}
@@ -7,24 +7,27 @@ node = "{{.Node}}"
 [operator]
 email = "{{.OperatorEmail}}"
 {{- end}}
-{{- if .ExternalMode}}
 
 [network]
+ipsec_enabled = {{.IPSecEnabled}}
+{{- if .ExternalMode}}
 external_mode = "{{.ExternalMode}}"
-{{- if .ExternalDHCP}}
-external_dhcp = true
-{{- end}}
 
 [[network.external_pools]]
 name        = "{{.PoolName}}"
 {{- if .PoolSource}}
 source      = "{{.PoolSource}}"
 {{- end}}
+{{- if .PoolBindBridge}}
+bind_bridge = "{{.PoolBindBridge}}"
+{{- end}}
 {{- if .PoolStart}}
 range_start = "{{.PoolStart}}"
 range_end   = "{{.PoolEnd}}"
 {{- end}}
+{{- if .PoolGateway}}
 gateway     = "{{.PoolGateway}}"
+{{- end}}
 {{- if .PoolGatewayIP}}
 gateway_ip  = "{{.PoolGatewayIP}}"
 {{- end}}
@@ -90,6 +93,9 @@ token = "{{.NatsToken}}"
 
 [nodes.{{.Node}}.viperblock]
 shardwal = false
+{{- if .EncryptionKeyFile}}
+encryption_key_file = "{{.EncryptionKeyFile}}"
+{{- end}}
 
 [nodes.{{.Node}}.vpcd]
 ovn_nb_addr = "{{.OVNNBAddr}}"
@@ -97,15 +103,6 @@ ovn_sb_addr = "{{.OVNSBAddr}}"
 {{- if .ExternalIface}}
 external_interface = "{{.ExternalIface}}"
 {{- end}}
-{{- if .DhcpBindBridge}}
-# dhcp_bind_bridge: the bridge where vpcd's DHCP client binds its AF_PACKET
-# socket. On Linux-bridge-WAN hosts (netplan default with the WAN NIC enslaved),
-# this is the Linux bridge name (e.g. "br-wan"). On direct-OVS hosts, this is
-# the OVS bridge holding the WAN NIC.
-# NEVER set this to "br-ext" — that is the OVN-side bridge and never sees LAN
-# DHCP traffic.
-dhcp_bind_bridge = "{{.DhcpBindBridge}}"
-{{- end}}
 
 [nodes.{{.Node}}.predastore]
 host = "{{.BindIP}}:8443"

docs/administration/spinifex-admin-cli/README.md

@@ -82,6 +82,25 @@ spx admin images import --name debian-13-arm64
 
 Catalog imports verify the image against the catalog-declared SHA-256/SHA-512 digest before extraction. Use `--file` to import operator-supplied media (verification skipped — operator is responsible for integrity), or `--force` to re-download after a checksum mismatch.
 
+### EKS node image
+
+To run EKS, import the prebuilt node image from the catalog:
+
+```bash
+spx admin images import --name spinifex-eks-node
+```
+
+This pulls the Alpine + K3s node AMI from `iso.mulgadc.com`, verifies its
+checksum, and registers it tagged `spinifex:managed-by=eks`. `eks create-cluster`
+and `eks create-nodegroup` resolve the boot AMI by that tag, so no further
+configuration is needed.
+
+Operators publishing a freshly built node image to the mirror run the
+**Publish EKS node AMI** GitHub Actions workflow (manual dispatch; uses the
+`CF_R2_*` repo secrets) or, locally, `make publish-eks-node-image` (see
+`scripts/publish-system-image.sh`; requires `R2_ENDPOINT` + R2-scoped
+`AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`).
+
 
 ## Cluster Shutdown
 

docs/compute/gpu-passthrough/README.md

@@ -7,6 +7,11 @@ tags:
   - passthrough
   - vfio
   - ec2
+sections:
+  - overview
+  - prerequisites
+  - instructions
+  - troubleshooting
 resources:
   - title: "Spinifex Repository"
     url: "https://github.com/mulgadc/spinifex"
@@ -14,26 +19,6 @@ resources:
     url: "https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html"
 ---
 
-# GPU Passthrough
-
-> Configure VFIO GPU passthrough on a Spinifex node to expose NVIDIA or AMD GPUs to EC2 instances.
-
-> **Note:** GPU passthrough is supported on x86_64 hosts only.
-
-## Table of Contents
-
-- [Overview](#overview)
-- [Prerequisites](#prerequisites)
-- [Host Setup](#host-setup)
-- [Enable Passthrough](#enable-passthrough)
-- [Status and Monitoring](#status-and-monitoring)
-- [GPU Instance Types](#gpu-instance-types)
-- [Launching GPU Instances](#launching-gpu-instances)
-- [Disable Passthrough](#disable-passthrough)
-- [Troubleshooting](#troubleshooting)
-
----
-
 ## Overview
 
 Spinifex supports VFIO-based GPU passthrough, binding NVIDIA and AMD GPUs directly to guest VMs via the `vfio-pci` kernel driver. Once configured, the node exposes GPU-enabled EC2 instance types and the GPU is allocated exclusively to individual instances.
@@ -42,9 +27,9 @@ At startup, Spinifex always probes for GPU hardware and surfaces the result in t
 
 The passthrough state is reflected in the login banner:
 
-![GPU not yet enabled](../../../.github/assets/images/gpu-banner1.png)
+<img src="../../../.github/assets/images/gpu-banner1.png" alt="GPU not yet enabled">
 
----
+> **Note:** GPU passthrough is supported on x86_64 hosts only.
 
 ## Prerequisites
 
@@ -52,9 +37,9 @@ The passthrough state is reflected in the login banner:
 - An NVIDIA or AMD GPU must be physically installed.
 - The Spinifex node must be running on bare metal (not inside a VM).
 
----
+## Instructions
 
-## Host Setup
+### Host Setup
 
 Run the one-time host configuration as root. This command is idempotent — safe to re-run after a reboot.
 
@@ -79,9 +64,7 @@ Setup complete — reboot required.
   Then run: sudo spx admin gpu enable
 ```
 
----
-
-## Enable Passthrough
+### Enable Passthrough
 
 After setup and reboot, enable GPU passthrough:
 
@@ -91,14 +74,11 @@ sudo spx admin gpu enable
 
 The command writes the configuration, signals the daemon (`SIGHUP`), and waits up to 30 seconds for the daemon to confirm the new state. On success it prints the current GPU status.
 
+<img src="../../../.github/assets/images/gpu-enabled.png" alt="GPU enabled">
 
-![GPU enabled](../../../.github/assets/images/gpu-enabled.png)
+### Status and Monitoring
 
----
-
-## Status and Monitoring
-
-### Per-node status
+**Per-node status:**
 
 ```bash
 spx admin gpu status
@@ -108,7 +88,7 @@ spx admin gpu status --node <node-name>
 
 Output includes hardware detected, IOMMU state, vfio-pci binding, passthrough enabled/disabled, GPU pool allocation (`allocated/total`), and the GPU-capable EC2 instance types available on that node.
 
-### Cluster view
+**Cluster view:**
 
 `spx top nodes` includes a `GPU (used/total)` column:
 
@@ -122,30 +102,30 @@ Output includes hardware detected, IOMMU state, vfio-pci binding, passthrough en
 spx top nodes
 ```
 
----
+### GPU Instance Types
 
-## GPU Instance Types
-
-GPU instance types are derived from detected hardware.
-
-To list available GPU instance types on the current node:
+GPU instance types are derived from detected hardware. To list available GPU instance types on the current node:
 
 ```bash
 aws ec2 describe-instance-types \
   --filters Name=instance-type,Values=g*
 ```
 
----
+### Launching GPU Instances
 
-## Launching GPU Instances
-
-Import the GPU-enabled AMI before launching:
+Spinifex ships two pre-built GPU guest images. Import the one that matches your host hardware before launching:
 
 ```bash
-spx admin images import --name ubuntu-gpu-nvidia   # NVIDIA hosts
-spx admin images import --name ubuntu-gpu-amd      # AMD hosts
+spx admin images import --name ubuntu-26.04-nvidia-gpu-x86_64   # NVIDIA hosts
+spx admin images import --name ubuntu-26.04-amd-gpu-x86_64      # AMD hosts
 ```
 
+These images are built on Ubuntu 26.04 and serve as a base for GPU workloads. Both include Docker CE, Python 3, and common utilities (`git`, `curl`, `htop`, `tmux`, `ffmpeg`).
+
+**NVIDIA** (`ubuntu-26.04-nvidia-gpu-x86_64`): NVIDIA server driver with DKMS pre-built against the image kernel, `nvidia-smi`, nvidia-container-toolkit wired into Docker (`--gpus all` works on first boot). CUDA toolkit and cuDNN are not included — use an NGC container or install inside the instance.
+
+**AMD** (`ubuntu-26.04-amd-gpu-x86_64`): `linux-firmware` (amdgpu firmware blobs), ROCm 7.2 CLI tools (`rocm-smi`, `rocminfo`, `amd-smi`). Full ROCm compute libraries (`rocblas`, etc.) are not included — install inside the instance as needed.
+
 Launch a GPU instance:
 
 ```bash
@@ -165,9 +145,7 @@ nvidia-smi
 rocm-smi
 ```
 
----
-
-## Disable Passthrough
+### Disable Passthrough
 
 Passthrough cannot be disabled while GPU instances are running. Terminate all GPU instances first, then:
 
@@ -177,8 +155,6 @@ sudo spx admin gpu disable
 
 The command signals the daemon and waits for confirmation. The GPU is released back to the host kernel on the next reboot (the vfio-pci binding persists until `setup` is re-run without the blacklists).
 
----
-
 ## Troubleshooting
 
 ### IOMMU not active after reboot

docs/compute/launching-instances/README.md

@@ -164,6 +164,16 @@ spx admin images import --name debian-13-x86_64
 
 cloud-init needs time to configure the instance after boot. Wait 30-60 seconds and retry.
 
+If the connection **times out** rather than being refused, the security group is likely blocking port 22. The default security group denies all inbound traffic (matching AWS), so SSH must be explicitly allowed:
+
+```bash
+# Allow SSH from anywhere on the instance's security group
+aws ec2 authorize-security-group-ingress \
+  --group-id $SG_ID --protocol tcp --port 22 --cidr 0.0.0.0/0
+```
+
+See [VPC Networking — Security Groups](/docs/vpc-networking#security-groups) for scoping rules to a trusted CIDR.
+
 Verify the SSH key was specified correctly when launching:
 
 ```bash

docs/compute/vpc-networking/README.md

@@ -48,8 +48,9 @@ NIC. Each host runs `ovn-controller` which programs OpenFlow rules on `br-int`
 
 ## Private vs Public Subnets
 
-A subnet's behavior depends on two things: whether the VPC has an Internet
-Gateway, and whether the subnet has `MapPublicIpOnLaunch` enabled.
+A subnet's behavior depends on three things: whether the VPC has an Internet
+Gateway, whether the subnet's route table has a default route to that IGW, and
+whether the subnet has `MapPublicIpOnLaunch` enabled.
 
 ## Private Subnet (Default)
 
@@ -61,9 +62,10 @@ internet or be reached from the WAN.
   <img src="../../../.github/assets/diagrams/vpc-private-subnet-flow.svg" alt="Private subnet — instance hits router, no default route, packet dropped" width="900">
 </p>
 
-If the VPC has an IGW attached, private subnet instances CAN reach the internet
-via the VPC router's SNAT rule (outbound only — they share the gateway IP). They
-still cannot be reached from the WAN because they have no public IP.
+Private subnet instances reach the internet only if their route table has a
+default route to the IGW (shared SNAT, outbound only — they share the gateway
+IP) or to a NAT gateway. With no default route, egress is dropped. Either way
+they cannot be reached from the WAN because they have no public IP.
 
 ## Public Subnet
 
@@ -77,8 +79,18 @@ managed by OVN — the instance OS only sees its private IP.
 **Requirements for a public subnet:**
 
 1. VPC has an Internet Gateway attached
-2. Subnet has `MapPublicIpOnLaunch = true`
-3. External IP pool configured in `spinifex.toml`
+2. A route table associated with the subnet has a `0.0.0.0/0` route to the IGW
+3. Subnet has `MapPublicIpOnLaunch = true`
+4. External IP pool configured in `spinifex.toml`
+
+Spinifex follows AWS route-table semantics: a subnet is only "public" if its
+effective route table carries a default route to the IGW. A new VPC's main
+route table has the local route only — Spinifex does **not** add the IGW route
+for you. Without it, the subnet's egress is gated with a drop policy, so
+instances cannot reach the internet (and inbound connections cannot complete
+because return traffic is dropped) even with a public IP and an attached IGW.
+Add the route explicitly — either to the main route table, or to a custom route
+table associated with the subnet (see [Quick Start](#3-create-vpc-with-public-subnet)).
 
 ## Comparison
 
@@ -86,7 +98,7 @@ managed by OVN — the instance OS only sees its private IP.
 | ------------------------ | --------------------------------- | ------------------------------ |
 | Private IP               | Yes                               | Yes                            |
 | Public IP                | No                                | Auto-assigned from pool        |
-| Outbound internet        | Only if VPC has IGW (shared SNAT) | Yes (own public IP via SNAT)   |
+| Outbound internet        | Only with a default route to IGW/NAT GW | Yes (own public IP via SNAT)   |
 | Inbound from WAN         | No                                | Yes (via 1:1 NAT to public IP) |
 | Instance sees public IP? | N/A                               | No — only sees private IP      |
 | Elastic IP support       | Only if explicitly associated     | Yes                            |
@@ -190,8 +202,7 @@ use `setup-ovn.sh --dhcp` to obtain one from the router. This is the router's
 DHCP — not Spinifex's internal OVN DHCP for VMs.
 
 **Use when:** VMs only need outbound access (apt update, pulling images). Edge
-deployments behind ISP NAT. Single WAN IP available. Future: use with AWS-style
-NAT Gateway for private subnet internet access.
+deployments behind ISP NAT. Single WAN IP available.
 
 ```toml
 [network]
@@ -935,17 +946,39 @@ spx admin init
 VPC=$(aws ec2 create-vpc --cidr-block 10.200.0.0/16 \
   --query Vpc.VpcId --output text)
 
-SUBNET=$(aws ec2 create-subnet --vpc-id $VPC \
-  --cidr-block 10.200.1.0/24 \
-  --query Subnet.SubnetId --output text)
-
 IGW=$(aws ec2 create-internet-gateway \
   --query InternetGateway.InternetGatewayId --output text)
 aws ec2 attach-internet-gateway \
   --internet-gateway-id $IGW --vpc-id $VPC
 
+SUBNET=$(aws ec2 create-subnet --vpc-id $VPC \
+  --cidr-block 10.200.1.0/24 \
+  --query Subnet.SubnetId --output text)
+
+# Route table — give the subnet a default route to the IGW. Spinifex does NOT
+# add this automatically; without it the subnet's egress is dropped.
+RT=$(aws ec2 create-route-table --vpc-id $VPC \
+  --query RouteTable.RouteTableId --output text)
+
+aws ec2 create-route --route-table-id $RT \
+  --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW
+
+aws ec2 associate-route-table --route-table-id $RT --subnet-id $SUBNET
+
+# Auto-assign a public IP to instances launched into the subnet
 aws ec2 modify-subnet-attribute \
   --subnet-id $SUBNET --map-public-ip-on-launch
+
+# Allow SSH + ICMP on the VPC's default security group (blocks inbound by default)
+SG=$(aws ec2 describe-security-groups \
+  --filters Name=vpc-id,Values=$VPC \
+  --query 'SecurityGroups[0].GroupId' --output text)
+
+aws ec2 authorize-security-group-ingress --group-id $SG \
+  --protocol tcp --port 22 --cidr 0.0.0.0/0
+
+aws ec2 authorize-security-group-ingress --group-id $SG \
+  --protocol icmp --port -1 --cidr 0.0.0.0/0
 ```
 
 ## 4. Launch Instance
@@ -1135,6 +1168,37 @@ ip route show
    sudo ovn-nbctl lr-nat-list vpc-$VPC
    ```
 
+### Instance Has Public IP But No Internet
+
+The instance shows a public IP in `describe-instances` but cannot reach the
+internet, and inbound connections hang. The usual cause is a missing default
+route: the subnet's effective route table has no `0.0.0.0/0` route to the IGW,
+so Spinifex gates the subnet with a drop policy.
+
+```bash
+# Find the route table that applies to the subnet and check its routes
+aws ec2 describe-route-tables \
+  --filters Name=association.subnet-id,Values=$SUBNET \
+  --query 'RouteTables[0].Routes'
+# Expect a route with DestinationCidrBlock 0.0.0.0/0 and a GatewayId of igw-...
+```
+
+If the subnet has no explicit association it falls back to the VPC's main route
+table — check that one too, then add the route to whichever table applies:
+
+```bash
+aws ec2 create-route --route-table-id $RT \
+  --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW
+```
+
+On the host, the drop policy installed when a subnet lacks an IGW route appears
+as a `Logical_Router_Policy` on the VPC router:
+
+```bash
+sudo ovn-nbctl lr-policy-list vpc-$VPC
+# A drop policy matching the subnet CIDR means the subnet is gated.
+```
+
 ### Public IP Not Reachable from WAN
 
 Work through these checks in order — each eliminates a class of issues.

docs/getting-started/install-source/README.md

@@ -32,7 +32,7 @@ This guide builds Spinifex from source. For production deployments, the [binary
 **Requirements:**
 
 - Ubuntu 26.04 / Debian 13
-- Go 1.26.3+
+- Go 1.26.4+
 - GCC, make, pkg-config
 - QEMU/KVM, OVN/Open vSwitch, AWS CLI v2, dhcpcd-base
 

docs/getting-started/setting-up-your-cluster/README.md

@@ -34,7 +34,7 @@ resources:
 ## Overview
 
 This guide assumes Spinifex is already installed and running. If not, follow one of the installation guides first:
-- [Single-Node Install)](/docs/install)
+- [Single-Node Install](/docs/install)
 - [Multi-Node Install](/docs/install-multi-node)
 - [Source Install](/docs/install-source)
 
@@ -50,7 +50,9 @@ export AWS_PROFILE=spinifex
 
 ## 1. Import an AMI
 
-List available images and import one matching your architecture:
+### Option A: Import a bundled image
+
+List the bundled images and import one matching your architecture:
 
 ```bash
 spx admin images list
@@ -64,13 +66,11 @@ ubuntu-26.04-arm64      | ubuntu | 26.04   | arm64  | uefi
 ubuntu-26.04-x86_64     | ubuntu | 26.04   | x86_64 | uefi
 ```
 
-Import an image:
-
 ```bash
 spx admin images import --name ubuntu-26.04-x86_64
 ```
 
-Or import a local image file:
+### Option B: Import a local image file
 
 ```bash
 spx admin images import --file ~/images/ubuntu-26.04.img --distro ubuntu --version 26.04 --arch x86_64 --boot-mode uefi
@@ -79,7 +79,7 @@ spx admin images import --file ~/images/ubuntu-26.04.img --distro ubuntu --versi
 Verify the import and note the AMI ID:
 
 ```bash
-aws ec2 describe-images
+AMI_ID=$(aws ec2 describe-images --query 'Images[0].ImageId' --output text)
 ```
 
 ## 2. Create an SSH Key
@@ -115,8 +115,6 @@ aws ec2 describe-key-pairs
 ```bash
 VPC_ID=$(aws ec2 create-vpc --cidr-block 10.200.0.0/16 \
   --query 'Vpc.VpcId' --output text)
-
-echo "VPC: $VPC_ID"
 ```
 
 ### Create an Internet Gateway
@@ -130,68 +128,95 @@ IGW_ID=$(aws ec2 create-internet-gateway \
 aws ec2 attach-internet-gateway \
   --internet-gateway-id $IGW_ID \
   --vpc-id $VPC_ID
-
-echo "IGW: $IGW_ID (attached to $VPC_ID)"
 ```
 
-### Create a Public Subnet
+### Create a Subnet
 
-A public subnet auto-assigns a routable IP to each instance, making it directly reachable from your network.
+Create the subnet your instances will launch into. The routing and public-IP steps below are what make it a *public* subnet.
 
 ```bash
 SUBNET_ID=$(aws ec2 create-subnet \
   --vpc-id $VPC_ID \
   --cidr-block 10.200.1.0/24 \
   --query 'Subnet.SubnetId' --output text)
+```
 
-# Enable auto-assign public IP
+### Create a Route Table
+
+A subnet is only **public** if its route table has a default route to the Internet Gateway. Spinifex does **not** add this route automatically — a new VPC's route table only routes traffic within the VPC (matching AWS). Without a `0.0.0.0/0` route to the IGW, instances in the subnet cannot reach the internet (and inbound connections cannot complete) even with a public IP assigned.
+
+```bash
+RT_ID=$(aws ec2 create-route-table \
+  --vpc-id $VPC_ID \
+  --query 'RouteTable.RouteTableId' --output text)
+
+# Default route to the internet via the IGW
+aws ec2 create-route \
+  --route-table-id $RT_ID \
+  --destination-cidr-block 0.0.0.0/0 \
+  --gateway-id $IGW_ID
+
+# Associate the route table with the subnet
+aws ec2 associate-route-table \
+  --route-table-id $RT_ID \
+  --subnet-id $SUBNET_ID
+```
+
+### Enable Auto-Assign Public IP
+
+Give every instance launched into the subnet a routable public IP, making it directly reachable from your network.
+
+```bash
 aws ec2 modify-subnet-attribute \
   --subnet-id $SUBNET_ID \
   --map-public-ip-on-launch
-
-echo "Subnet: $SUBNET_ID (public)"
 ```
 
+### Allow SSH and ICMP
+
+Every VPC gets a default security group that **blocks inbound traffic from outside the group** (matching AWS). Instances launched without an explicit security group use this default, so you must authorize ingress before you can SSH or ping the instance.
+
+```bash
+SG_ID=$(aws ec2 describe-security-groups \
+  --filters Name=vpc-id,Values=$VPC_ID \
+  --query 'SecurityGroups[0].GroupId' --output text)
+
+# Allow SSH from anywhere
+aws ec2 authorize-security-group-ingress \
+  --group-id $SG_ID \
+  --protocol tcp --port 22 --cidr 0.0.0.0/0
+
+# Allow ICMP (ping) from anywhere
+aws ec2 authorize-security-group-ingress \
+  --group-id $SG_ID \
+  --protocol icmp --port -1 --cidr 0.0.0.0/0
+```
+
+**Note:** `0.0.0.0/0` opens these ports to every source. For anything beyond a quick evaluation, scope the `--cidr` to a trusted range. Rule changes apply immediately — no instance restart needed.
+
 ### Verify
 
 ```bash
 aws ec2 describe-vpcs --vpc-ids $VPC_ID
 aws ec2 describe-subnets --subnet-ids $SUBNET_ID
+aws ec2 describe-route-tables --route-table-ids $RT_ID
+aws ec2 describe-security-groups --group-ids $SG_ID
 ```
 
 ## 4. Launch an Instance
 
-Query available instance types for your hardware:
+Launch an instance in the public subnet on a `t3.micro` (2 vCPU / 1 GiB).
+
+> **Note:** On an **arm64** host, use `t4g.micro` instead of `t3.micro`.
 
 ```bash
-aws ec2 describe-instance-types
-```
-
-Launch an instance in the public subnet, note this will select an instance type with 2 VCPU and 1024MB of memory.
-
-> **Note:** Replace `AMI_NAME` with the previously imported image above with the `ami-` prefix.
-
-```bash
-AMI_NAME="ami-ubuntu-26.04-x86_64"
-
-AMI_ID=$(aws ec2 describe-images \
-  --filters "Name=name,Values=$AMI_NAME" \
-  --query 'Images | sort_by(@, &CreationDate) | [-1].ImageId' \
-  --output text)
-
-INSTANCE_TYPE=$(aws ec2 describe-instance-types \
-  --query "sort_by(InstanceTypes[?VCpuInfo.DefaultVCpus==\`2\` && MemoryInfo.SizeInMiB>=\`1024\`], &MemoryInfo.SizeInMiB)[0].InstanceType" \
-  --output text)
-
 INSTANCE_ID=$(aws ec2 run-instances \
   --image-id $AMI_ID \
-  --instance-type $INSTANCE_TYPE \
+  --instance-type t3.micro \
   --key-name spinifex-key \
   --subnet-id $SUBNET_ID \
   --count 1 \
   --query 'Instances[0].InstanceId' --output text)
-
-echo "Instance: $INSTANCE_ID"
 ```
 
 Wait for the instance to reach `running` state:
@@ -328,7 +353,36 @@ PRIVATE_SUBNET=$(aws ec2 create-subnet \
   --query 'Subnet.SubnetId' --output text)
 ```
 
-Instances in private subnets get a private IP only. They can still reach the internet via SNAT through the VPC router, but are not directly reachable from your network.
+Instances in a private subnet get a private IP only and are not reachable from your network. By default they also have **no internet access**: with no `0.0.0.0/0` route in the subnet's route table, Spinifex gates egress with a drop policy — matching AWS, where a private subnet has no route off the VPC. They can still reach other instances in the same VPC.
+
+To give a private subnet outbound-only internet access, deploy a NAT Gateway in a public subnet and point the private subnet's default route at it (the AWS pattern):
+
+```bash
+# Allocate an Elastic IP for the NAT Gateway
+NAT_EIP=$(aws ec2 allocate-address --query AllocationId --output text)
+
+# Create the NAT Gateway in the PUBLIC subnet (the one with the IGW route)
+NATGW_ID=$(aws ec2 create-nat-gateway \
+  --subnet-id $SUBNET_ID \
+  --allocation-id $NAT_EIP \
+  --query 'NatGateway.NatGatewayId' --output text)
+
+# Give the private subnet its own route table with a default route to the NAT Gateway
+PRIVATE_RT=$(aws ec2 create-route-table \
+  --vpc-id $VPC_ID \
+  --query 'RouteTable.RouteTableId' --output text)
+
+aws ec2 create-route \
+  --route-table-id $PRIVATE_RT \
+  --destination-cidr-block 0.0.0.0/0 \
+  --nat-gateway-id $NATGW_ID
+
+aws ec2 associate-route-table \
+  --route-table-id $PRIVATE_RT \
+  --subnet-id $PRIVATE_SUBNET
+```
+
+Instances in the private subnet now reach the internet outbound through the NAT Gateway's public IP, but remain unreachable from the WAN.
 
 ### Multiple Accounts
 
@@ -348,7 +402,7 @@ journalctl -u spinifex-daemon -f
 aws ec2 describe-images
 ```
 
-### SSH Connection Refused
+### SSH Connection Refused or Times Out
 
 cloud-init needs 30-60 seconds after boot. Check instance state:
 
@@ -356,6 +410,20 @@ cloud-init needs 30-60 seconds after boot. Check instance state:
 aws ec2 describe-instances --instance-ids $INSTANCE_ID
 ```
 
+A connection that **times out** (rather than being refused) usually means the security group is blocking port 22. Confirm the default security group allows SSH ingress:
+
+```bash
+aws ec2 describe-security-groups --group-ids $SG_ID \
+  --query 'SecurityGroups[0].IpPermissions'
+```
+
+If there are no rules for TCP 22, authorize it (see [Allow SSH and ICMP](#allow-ssh-and-icmp)):
+
+```bash
+aws ec2 authorize-security-group-ingress \
+  --group-id $SG_ID --protocol tcp --port 22 --cidr 0.0.0.0/0
+```
+
 ### No Public IP Assigned
 
 Verify the subnet has `MapPublicIpOnLaunch` enabled:
@@ -378,7 +446,23 @@ aws ec2 describe-internet-gateways
 
 ### Instance Has No Internet Access
 
-Check the VPC router's NAT rules (from the host):
+First confirm the subnet's route table has a default route to the IGW. Spinifex gates a subnet's egress with a drop policy when this route is missing, so an instance with a public IP still cannot reach the internet:
+
+```bash
+aws ec2 describe-route-tables \
+  --filters Name=association.subnet-id,Values=$SUBNET_ID \
+  --query 'RouteTables[0].Routes'
+# Expect a 0.0.0.0/0 route with a GatewayId of igw-...
+```
+
+If the route is missing, add it (see [Create a Route Table](#create-a-route-table)):
+
+```bash
+aws ec2 create-route --route-table-id $RT_ID \
+  --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW_ID
+```
+
+If the route is present, check the VPC router's NAT rules (from the host):
 
 ```bash
 sudo ovn-nbctl lr-nat-list $(sudo ovn-nbctl lr-list | awk '{print $2}' | head -1)

docs/install-vm-eval/README.md

@@ -12,8 +12,6 @@ For this tutorial download UTM.app on Mac to simplify the VM installation.
 
 * UTM - [https://mac.getutm.app](https://mac.getutm.app)
 
-
-
 ## Download Debian ISO
 
 Next, download the Debian ISO (ARM64, Apple Silicon)
@@ -28,7 +26,7 @@ Using the VM console, login using the `spinifex` user and your specified passwor
 
 For a minimal Debian install additional dependencies are required prior to the Spinifex installation:
 
-```
+```bash
 su root
 apt update
 apt install sudo curl bridge-utils
@@ -38,7 +36,7 @@ apt install sudo curl bridge-utils
 
 Enable sudo for `spinifex` user as the root user:
 
-```
+```bash
 su root
 echo "spinifex ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/spinifex
 ```

docs/reference-architectures/SMCI-H14-orchestration/README.md

@@ -0,0 +1,268 @@
+---
+title: "Multi-Tenant AI on Supermicro H14 with AMD MI350X"
+description: "Provision isolated GPU VMs for simultaneous AI workloads on a Supermicro H14 bare-metal node using Spinifex's EC2-compatible API."
+category: "Reference Architectures"
+tags:
+  - supermicro
+  - amd
+  - mi350x
+  - gpu-passthrough
+  - multi-tenant
+  - vllm
+resources:
+  - title: "GPU Passthrough"
+    url: "/docs/gpu-passthrough"
+  - title: "Launching Instances"
+    url: "/docs/launching-instances"
+  - title: "AMD Instinct MI350X"
+    url: "https://www.amd.com/en/products/accelerators/instinct/mi300/mi350x.html"
+  - title: "vLLM Documentation"
+    url: "https://docs.vllm.ai"
+sections:
+  - overview
+  - prerequisites
+  - instructions
+---
+
+## Overview
+
+Spinifex is an open-source infrastructure platform that brings core AWS services including EC2, EBS and S3 to bare-metal, edge, and on-prem environments. It exposes an EC2-compatible API, so any tooling that works against AWS (the `aws` CLI, Terraform, SDKs) works against a Spinifex node unchanged, with a single profile swap.
+
+This document serves as a tutorial for setting up multi-tenant AI workloads with Spinifex. The reference architecture for this tutorial is the Supermicro H14 platform - specifications listed below.
+
+The H14 chassis comes equipped with **8× AMD Instinct MI350X** (288 GB HBM3e each, 2.3 TB total). We used Spinifex to provision three isolated VMs, each assigned 2× MI350X via direct PCIe passthrough, and ran three computationally intensive workloads simultaneously: a YOLO11x vision model, a Qwen3-VL 235B FP8 vision language model, and a Llama 3.3 70B large language model.
+
+<p><video src="https://iso.mulgadc.com/clipped-demo.mp4" controls width="100%" style="border-radius:6px"></video></p>
+
+### Platform
+
+| Component | Specification |
+|---|---|
+| **Bare-metal host** | Supermicro H14 |
+| **Host OS** | Ubuntu 24.04 LTS or Debian 13 (minimum) |
+| **Orchestration** | Spinifex — EC2-compatible bare-metal API |
+| **Guest OS** | Ubuntu 26.04 LTS |
+| **GPUs** | 8× AMD Instinct MI350X (288 GB HBM3e each, 2.3 TB total) |
+| **GPU passthrough** | PCIe passthrough via vfio-pci |
+| **Instance type** | `g7e.12xlarge` — 2× MI350X per VM |
+| **Container runtime** | Docker (with ROCm device access) |
+| **Inference runtime** | vLLM (`rocm/vllm` image) |
+| **VM network** | Internal VPC, 192.168.10.0/24 |
+| **Block storage** | Viperblock — EBS-compatible, local NVMe-backed |
+
+Spinifex runs on the bare-metal host and exposes an EC2-compatible API to any standard `aws ec2` tooling. Guest VMs receive their GPU allocation at the hardware level — each VM's OS sees the MI350Xs as native PCIe devices with no virtualisation layer in the data path. Workloads across VMs share no GPU memory, no address space, and no network segment except through the VPC.
+
+GPU passthrough requires a host kernel and OS that supports vfio-pci. Ubuntu 24.04 LTS (kernel 6.8+) and Debian 13 (kernel 6.12+) are the tested minimum baselines for the host. Guest VMs run Ubuntu 26.04 LTS.
+
+## Prerequisites
+
+- Supermicro H14 with 8× AMD Instinct MI350X installed
+- Host OS: **Ubuntu 24.04 LTS** (kernel 6.8+) or **Debian 13** (kernel 6.12+) — minimum for vfio-pci support
+- Spinifex installed and all services running (`systemctl status spinifex.target`)
+- AMD GPU AMI registered (`ami-ubuntu-amd-gpu`) — Ubuntu 26.04 LTS with ROCm-compatible kernel
+- AWS CLI configured with `AWS_PROFILE=spinifex` pointing at the Spinifex endpoint
+- SSH key pair imported into Spinifex, VPC and security group created (see [Launching Instances](/docs/launching-instances))
+
+GPU passthrough must be configured before launching GPU instances. This is a one-time step per host.
+
+```bash
+# Detect GPUs and bind to vfio-pci (requires reboot)
+sudo spx admin gpu setup
+
+# After rebooting — confirm passthrough is active and signal the daemon
+sudo spx admin gpu enable
+```
+
+`spx admin gpu setup` blacklists the AMD driver on the host and binds each GPU to `vfio-pci`. After reboot, `spx admin gpu enable` verifies the binding and makes the GPU pool available to `RunInstances`.
+
+<img src="../../../.github/assets/images/gpu-enabled.png" alt="Spinifex console confirming PCIe passthrough is active for each GPU">
+
+To check available GPU instance types:
+
+```bash
+export AWS_PROFILE=spinifex
+aws ec2 describe-instance-types \
+    --query 'InstanceTypes[?GpuInfo].[InstanceType,GpuInfo.Gpus[0].Count,GpuInfo.Gpus[0].Name]' \
+    --output table
+```
+
+## Instructions
+
+### 1. Provision the VMs
+
+Each VM was provisioned using standard AWS EC2 CLI commands — the only change from a normal AWS workflow is `AWS_PROFILE=spinifex`, which redirects the CLI to Spinifex's local EC2-compatible endpoint instead of AWS:
+
+| VM | Instance type | GPUs | Role | Model |
+|---|---|---|---|---|
+| `vm-yolo` | g7e.12xlarge | 2× MI350X | Real-time object detection | YOLO11x |
+| `vm-vlm`  | g7e.12xlarge | 2× MI350X | Multi-modal scene analysis | Qwen3-VL 235B FP8 |
+| `vm-chat` | g7e.12xlarge | 2× MI350X | Conversational LLM | Llama 3.3 70B |
+
+Six of the eight available MI350Xs are claimed here; the remaining two stay idle on the host, available for a fourth tenant without touching the existing three.
+
+```bash
+export AWS_PROFILE=spinifex
+
+# vm-yolo — YOLO11x object detection, 200 GB disk
+YOLO_ID=$(aws ec2 run-instances \
+    --image-id ami-ubuntu-amd-gpu \
+    --instance-type g7e.12xlarge \
+    --key-name spinifex-key \
+    --subnet-id <subnet-id> \
+    --security-group-ids <sg-id> \
+    --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=200,DeleteOnTermination=true}' \
+    --count 1 \
+    --query 'Instances[0].InstanceId' --output text)
+echo "vm-yolo launched: $YOLO_ID"
+
+# vm-vlm — Qwen3-VL 235B FP8, 600 GB disk for weights
+VLM_ID=$(aws ec2 run-instances \
+    --image-id ami-ubuntu-amd-gpu \
+    --instance-type g7e.12xlarge \
+    --key-name spinifex-key \
+    --subnet-id <subnet-id> \
+    --security-group-ids <sg-id> \
+    --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=600,DeleteOnTermination=true}' \
+    --count 1 \
+    --query 'Instances[0].InstanceId' --output text)
+echo "vm-vlm launched: $VLM_ID"
+
+# vm-chat — Llama 3.3 70B, 300 GB disk
+CHAT_ID=$(aws ec2 run-instances \
+    --image-id ami-ubuntu-amd-gpu \
+    --instance-type g7e.12xlarge \
+    --key-name spinifex-key \
+    --subnet-id <subnet-id> \
+    --security-group-ids <sg-id> \
+    --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=300,DeleteOnTermination=true}' \
+    --count 1 \
+    --query 'Instances[0].InstanceId' --output text)
+echo "vm-chat launched: $CHAT_ID"
+
+# Wait for all three to reach running state
+for ID in "$YOLO_ID" "$VLM_ID" "$CHAT_ID"; do
+    aws ec2 wait instance-running --instance-ids "$ID"
+    echo "$ID is running"
+done
+
+# Retrieve IPs (Spinifex assigns addresses from the internal VPC)
+YOLO_IP=$(aws ec2 describe-instances --instance-ids "$YOLO_ID" \
+    --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)
+VLM_IP=$(aws ec2 describe-instances --instance-ids "$VLM_ID" \
+    --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)
+CHAT_IP=$(aws ec2 describe-instances --instance-ids "$CHAT_ID" \
+    --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)
+
+echo "vm-yolo: $YOLO_IP"
+echo "vm-vlm:  $VLM_IP"
+echo "vm-chat: $CHAT_IP"
+```
+
+### 2. Verify GPU access
+
+Once SSH is available, confirm both GPUs are visible inside each VM:
+
+```bash
+ssh -i ~/.ssh/spinifex-key ubuntu@$CHAT_IP 'lspci | grep -i amd'
+```
+
+Or install and run `amd-smi`:
+
+```bash
+ssh -i ~/.ssh/spinifex-key ubuntu@$CHAT_IP \
+    'curl -fsSL https://repo.radeon.com/rocm/rocm.gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg >/dev/null && \
+     echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/6.3 noble main" | sudo tee /etc/apt/sources.list.d/rocm.list && \
+     sudo apt-get update -qq && sudo apt-get install -y -q amd-smi-lib && \
+     amd-smi list'
+```
+
+<img src="../../../.github/assets/images/2gpus.png" alt="amd-smi inside vm-chat confirming two MI350Xs are directly attached, each with a unique UUID">
+
+Two MI350X entries with distinct UUIDs confirm direct PCIe passthrough is working.
+
+### 3. The orchestration layer
+
+Spinifex provides an EC2-compatible API on bare metal. The relevant capabilities for this demo:
+
+- **Multi-GPU claim per VM.** Each `g7e.12xlarge` requires two PCIe addresses claimed as a unit. Spinifex queries the available GPU pool on the host and assigns them to the guest VM at creation time, rolling back if the full allocation cannot be satisfied.
+- **PCIe passthrough via vfio-pci.** All 8 MI350Xs are bound to `vfio-pci` on the bare-metal host. Each VM gets its GPUs at the hardware level.
+- **Internal VPC (192.168.10.0/24).** VMs communicate directly without traversing the host's public network.
+
+The result: three completely independent EC2 instances, each running their own separate workloads using only the GPU resources they were allocated.
+
+### 4. Test methodology
+
+Two capture runs were made against the same workload script: **30 s baseline** (YOLO and VLM models running, no chat load) → **25 scripted Llama chat prompts** at 15 s intervals → **30 s settle tail**.
+
+The only difference between the two runs was tensor parallelism:
+
+- **Demo 1 (TP=1):** each VM uses one of its two allocated GPUs. The second sits idle.
+- **Demo 2 (TP=2):** each VM shards its model symmetrically across both GPUs using AMD Infinity Fabric (XGMI).
+
+| Metric | TP=1 baseline (Demo 1) | TP=2 (Demo 2) |
+|---|---|---|
+| Wall clock | 613 s | 600 s |
+| YOLO avg fps | 16.91 (min 15.3 / max 18.0) | 17.09 (min 15.6 / max 18.1) |
+| Qwen peak tok/s | 61.7 | 53.0 |
+| Llama peak tok/s | 36.4 | **53.1** |
+| Both GPUs active (chat VM) | no — GPU 1 idle | **yes — symmetric** |
+| VRAM per GPU (Qwen) | 259 GB on one, 0.3 GB on other | 241 GB on **both** |
+| VRAM per GPU (Llama) | 261 GB on one, 0.3 GB on other | 248 GB on **both** |
+
+### 5. Results
+
+### Inter-tenant isolation
+
+The charts below show the three independent workloads spiking activity on their associated GPUs, measured in GFX utilisation and power draw. YOLO average FPS stays at ~17 despite utilisation in the YOLO VM never exceeding 20%, indicating a bottleneck in networking or encoding rather than inference. Each chart highlights workload independence — pulses of GPU activity share no correlation across VMs.
+
+<img src="../../../.github/assets/images/smci/smci-demo1-gpu-utilisation.png" alt="GPU utilisation across all three VMs — Demo 1 TP=1">
+
+GPU utilisation per VM across the full Demo 1 capture.
+
+<img src="../../../.github/assets/images/smci/smci-demo2-gpu-power.png" alt="GPU power draw per workload — Demo 2 TP=2">
+
+Power draw per VM during Demo 2.
+
+### Single vs double GPU
+
+When TP=2 is enabled, each VM shards its model symmetrically across both GPUs. Inside the chat VM, both GPUs pulse together on every request, each holding exactly **248 GB of VRAM** (weights split + KV cache shards). With TP=1, GPU 1 sat at 0% for the entire run.
+
+<img src="../../../.github/assets/images/smci/smci-demo2-per-gpu-chat.png" alt="Llama 3.3 70B per-GPU activity — TP=2">
+
+Per-GPU utilisation inside `vm-chat` with TP=2.
+
+<img src="../../../.github/assets/images/smci/smci-demo1-per-gpu-chat.png" alt="Llama 3.3 70B per-GPU activity — TP=1">
+
+Same VM with TP=1.
+
+### vLLM generation throughput
+
+With TP=1, Qwen peaks at ~60 tok/s and Llama caps around 35 tok/s — both compute-bound on a single MI350X. Enabling TP=2 lifts Llama's peak to ~50 tok/s (+47%) as generation is distributed symmetrically across both GPUs. Qwen's throughput is slightly lower under TP=2 due to collective overhead on the shared-memory transport.
+
+<img src="../../../.github/assets/images/smci/smci-demo1-vllm-throughput.png" alt="vLLM generation throughput — TP=1">
+
+Generation throughput during Demo 1 (TP=1).
+
+<img src="../../../.github/assets/images/smci/smci-demo2-vllm-throughput.png" alt="vLLM generation throughput — TP=2">
+
+Generation throughput during Demo 2 (TP=2).
+
+### 6. Teardown
+
+```bash
+# Stop workloads on each VM
+for IP in "$YOLO_IP" "$VLM_IP" "$CHAT_IP"; do
+    ssh -i ~/.ssh/spinifex-key ubuntu@$IP 'docker stop $(docker ps -q)' || true
+done
+
+# Terminate all three instances — releases 6× MI350X back to the Spinifex pool
+aws ec2 terminate-instances --instance-ids "$YOLO_ID" "$VLM_ID" "$CHAT_ID"
+```
+
+The six MI350Xs are immediately returned to the pool once the instances terminate, ready for a new allocation without touching the host.
+
+### 7. Conclusion
+
+This tutorial demonstrates how Spinifex can turn a single bare-metal chassis into a multi-tenant AI serving platform. Spinifex utilises the flexibility of PCIe passthrough to allocate GPU resources in whichever configuration is required by the workload/s.
+
+Importantly, it does so with standard `aws ec2` CLI calls — `run-instances`, `describe-instances`, `terminate-instances` — against Spinifex's EC2-compatible endpoint. Teams already operating AWS infrastructure can point their existing tooling at a Spinifex node with a single profile change, against GPUs that sit in their own rack.

docs/reference-architectures/arcee-trinity-mi350x/README.md

@@ -0,0 +1,321 @@
+---
+title: "Arcee Trinity 400B FP8 on AMD MI350X with Spinifex"
+description: "Provision, load, and benchmark arcee-ai/Trinity-Large-Preview-FP8 across two AMD MI350X GPUs on a Supermicro H14 using Spinifex's EC2-compatible API."
+category: "Reference Architectures"
+tags:
+  - supermicro
+  - amd
+  - mi350x
+  - gpu-passthrough
+  - vllm
+  - llm
+  - arcee
+resources:
+  - title: "GPU Passthrough"
+    url: "/docs/gpu-passthrough"
+  - title: "Launching Instances"
+    url: "/docs/launching-instances"
+  - title: "Arcee AI Trinity"
+    url: "https://www.arcee.ai/trinity"
+  - title: "Trinity on HuggingFace"
+    url: "https://huggingface.co/arcee-ai/Trinity-Large-Preview-FP8"
+  - title: "AMD Instinct MI350X"
+    url: "https://www.amd.com/en/products/accelerators/instinct/mi300/mi350x.html"
+  - title: "vLLM Documentation"
+    url: "https://docs.vllm.ai"
+sections:
+  - overview
+  - prerequisites
+  - instructions
+---
+
+## Overview
+
+Spinifex is an open-source infrastructure platform that brings core AWS services including EC2, EBS and S3 to bare-metal, edge, and on-prem environments. It exposes an EC2-compatible API, so any tooling that works against AWS (the `aws` CLI, Terraform, SDKs) works against a Spinifex node unchanged, with a single profile swap.
+
+### About Trinity
+
+[Trinity](https://www.arcee.ai/trinity) is a family of open-weight language models from [Arcee AI](https://www.arcee.ai), built specifically for production agentic workloads: reliable tool calling, structured JSON output, coherent multi-turn conversations, and long-context reasoning. The family spans four sizes — Nano (6B), Mini (26B), Large Preview (400B), and Large Thinking (400B reasoning-optimised), with consistent capabilities and API surface across all of them. A workflow validated on Nano can be promoted to Large with no prompt changes.
+
+Trinity Large Preview is a **sparse Mixture-of-Experts (MoE)** model. The headline number is 400B total parameters, but only **13B parameters activate per token** — the MoE routing selects a subset of experts for each forward pass, so inference cost is closer to a much smaller dense model than the raw parameter count suggests. The result is a 400B-class model with a **512K token context window** that fits on two high-memory GPUs and generates at competitive throughput.
+
+Arcee has trained Trinity with a heavy focus on agent reliability — function selection accuracy, valid parameter generation, graceful failure recovery, and schema adherence for structured outputs. It is available as open weights on HuggingFace and is natively compatible with vLLM, SGLang, and llama.cpp.
+
+### This guide
+
+This writeup covers running Trinity Large Preview FP8 on a Supermicro H14 bare-metal node via Spinifex — our EC2-compatible orchestration layer. Two AMD Instinct MI350X GPUs (288 GB HBM3e each, 576 GB combined) provide just enough headroom for the FP8 weights plus active KV cache, provisioned and managed with standard `aws ec2` CLI commands.
+
+At that scale the FP8 weights require around 400 GB of GPU memory — more than a single MI350X can hold. This guide covers splitting the model symmetrically across two MI350Xs using vLLM tensor parallelism (TP=2), running on a `g7e.12xlarge` instance provisioned via Spinifex on the H14.
+
+The complete stack:
+
+| Layer | Detail |
+|---|---|
+| Bare-metal host | Supermicro H14, 8× AMD Instinct MI350X |
+| Orchestration | Spinifex (EC2-compatible API) |
+| Instance type | `g7e.12xlarge` — 2× MI350X via PCIe passthrough |
+| Disk | 800 GB (model weights alone are ~400 GB) |
+| Inference runtime | vLLM (`rocm/vllm` Docker image) |
+| Tensor parallelism | TP=2 — model sharded symmetrically across GPU 0 and GPU 1 |
+| Model | `arcee-ai/Trinity-Large-Preview-FP8` |
+
+### Platform
+
+| Component | Specification |
+|---|---|
+| **Bare-metal host** | Supermicro H14 |
+| **Host OS** | Ubuntu 24.04 LTS or Debian 13 (minimum) |
+| **Orchestration** | Spinifex — EC2-compatible bare-metal API |
+| **Guest OS** | Ubuntu 26.04 LTS |
+| **GPUs** | 2× AMD Instinct MI350X (288 GB HBM3e each, 576 GB combined) |
+| **GPU passthrough** | PCIe passthrough via vfio-pci |
+| **Instance type** | `g7e.12xlarge` |
+| **Container runtime** | Docker (with ROCm device access) |
+| **Inference runtime** | vLLM (`rocm/vllm` image), tensor parallelism TP=2 |
+| **Block storage** | Viperblock — EBS-compatible, 800 GB |
+| **Model** | `arcee-ai/Trinity-Large-Preview-FP8` (400B MoE, ~13B active/token) |
+
+Spinifex runs on the bare-metal host and presents an EC2-compatible API endpoint. Launching a `g7e.12xlarge` atomically claims two MI350Xs from the host's GPU pool and binds them to the guest VM via PCIe passthrough — the guest OS communicates with the hardware directly, with no software virtualisation layer in the data path. When the instance terminates, those two GPUs are immediately returned to the pool.
+
+GPU passthrough requires a host kernel and OS that supports vfio-pci. Ubuntu 24.04 LTS (kernel 6.8+) and Debian 13 (kernel 6.12+) are the tested minimum baselines for the host. Guest VMs run Ubuntu 26.04 LTS, which ships with the ROCm-compatible kernel and userspace expected by the AMD GPU AMI.
+
+## Prerequisites
+
+- Supermicro H14 with at least 2× AMD Instinct MI350X installed
+- Host OS: **Ubuntu 24.04 LTS** (kernel 6.8+) or **Debian 13** (kernel 6.12+) — minimum for vfio-pci support
+- Spinifex installed and all services running (`systemctl status spinifex.target`)
+- GPU passthrough configured — `spx admin gpu setup` (reboot required) then `spx admin gpu enable`
+- AMD GPU AMI registered (`ami-ubuntu-amd-gpu`) — Ubuntu 26.04 LTS with ROCm-compatible kernel
+- AWS CLI configured with `AWS_PROFILE=spinifex` pointing at the Spinifex endpoint
+- SSH key pair imported, VPC and security group created (see [Launching Instances](/docs/launching-instances))
+- Docker installed in the guest VM (included in the AMD GPU AMI)
+- HuggingFace account with access to `arcee-ai/Trinity-Large-Preview-FP8`
+
+GPU passthrough must be configured before launching GPU instances:
+
+```bash
+sudo spx admin gpu setup   # binds GPUs to vfio-pci — requires reboot
+# ... reboot ...
+sudo spx admin gpu enable  # confirms passthrough and makes GPU pool available
+```
+
+## Instructions
+
+### 1. Provision the instance
+
+Spinifex exposes a standard EC2 API on the bare-metal host. The only difference from a real AWS workflow is `AWS_PROFILE=spinifex`.
+
+```bash
+export AWS_PROFILE=spinifex
+
+# Check available GPU instance types
+aws ec2 describe-instance-types \
+    --query 'InstanceTypes[?GpuInfo].[InstanceType,GpuInfo.Gpus[0].Count,GpuInfo.Gpus[0].Name]' \
+    --output table
+
+# Launch a g7e.12xlarge with 800 GB disk
+# Spinifex atomically claims 2× MI350X via PCIe passthrough for this instance type
+INSTANCE_ID=$(aws ec2 run-instances \
+    --image-id ami-ubuntu-amd-gpu \
+    --instance-type g7e.12xlarge \
+    --key-name spinifex-key \
+    --subnet-id <subnet-id> \
+    --security-group-ids <sg-id> \
+    --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=800,DeleteOnTermination=true}' \
+    --count 1 \
+    --query 'Instances[0].InstanceId' --output text)
+
+echo "Launched: $INSTANCE_ID"
+
+# Wait for running state
+aws ec2 wait instance-running --instance-ids "$INSTANCE_ID"
+
+# Get the IP
+INSTANCE_IP=$(aws ec2 describe-instances \
+    --instance-ids "$INSTANCE_ID" \
+    --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)
+
+echo "IP: $INSTANCE_IP"
+```
+
+Once SSH is available, confirm both GPUs are visible:
+
+```bash
+ssh -i ~/.ssh/spinifex-key ubuntu@$INSTANCE_IP 'lspci | grep -i amd'
+```
+
+Or install and run `amd-smi` on the instance:
+
+<img src="../../../.github/assets/images/2gpus.png" alt="amd-smi inside the guest VM confirming two MI350Xs are directly attached, each with a unique UUID">
+
+Two MI350X entries with distinct UUIDs confirm direct PCIe passthrough.
+
+### 2. Pull the vLLM Docker image
+
+```bash
+ssh -i ~/.ssh/spinifex-key ubuntu@$INSTANCE_IP
+
+# On the VM:
+docker pull rocm/vllm:latest
+```
+
+> The `rocm/vllm` image is large (~20 GB). Pull it while the model is downloading in parallel if bandwidth allows.
+
+### 3. Pull the model
+
+Trinity Large Preview FP8 weighs roughly 400 GB, so expect the model download to take some time.
+
+```bash
+# On the VM:
+pip install 'huggingface-hub[hf_xet]'
+hf download 'arcee-ai/Trinity-Large-Preview-FP8' --repo-type model
+
+# Monitor progress — looking for ~400 GB total
+du -sh ~/.cache/huggingface/hub/
+```
+
+The model lands in `~/.cache/huggingface/hub/`. vLLM picks it up automatically from there on serve.
+
+### 4. Run vLLM
+
+Trinity needs TP=2 to fit across both MI350Xs:
+
+```bash
+# On the VM:
+docker run --rm -d \
+    --name trinity \
+    --device /dev/kfd \
+    --device /dev/dri \
+    --group-add video \
+    --ipc host \
+    --network host \
+    -v ~/.cache/huggingface:/root/.cache/huggingface \
+    rocm/vllm:latest \
+    vllm serve arcee-ai/Trinity-Large-Preview-FP8 \
+        --tensor-parallel-size 2 \
+        --dtype auto \
+        --max-model-len 8192 \
+        --gpu-memory-utilization 0.90 \
+        --host 0.0.0.0 \
+        --port 8003
+```
+
+Watch the startup log:
+
+```bash
+docker logs -f trinity
+```
+
+Loading a 400B FP8 model across two GPUs takes **2–4 minutes**. Wait for:
+
+```
+INFO:     Application startup complete.
+```
+
+Then verify:
+
+```bash
+curl http://localhost:8003/health
+curl http://localhost:8003/v1/models | python3 -m json.tool
+```
+
+### 5. Test it
+
+A quick sanity check via the OpenAI-compatible API before running benchmarks:
+
+```bash
+curl http://localhost:8003/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "arcee-ai/Trinity-Large-Preview-FP8",
+        "messages": [{"role": "user", "content": "Explain tensor parallelism in three sentences."}],
+        "max_tokens": 256
+    }'
+```
+
+The first request will be slowest — KV cache is cold. Subsequent requests warm up noticeably.
+
+### 6. Dashboard
+
+We created a quick dashboard for a live view of the model's telemetry — throughput, latency, GPU utilisation, VRAM, and KV cache — streamed via SSE from the vLLM metrics endpoint and the GPU stats sidecar.
+
+<p><video src="https://iso.mulgadc.com/trinity-workload.mp4" controls width="100%" style="border-radius:6px"></video></p>
+
+### 7. Benchmark results
+
+Two captures were run against the live model:
+
+- **Chat session:** A short interactive session — low concurrency, variable cadence, 56 s total. Reflects real conversational use.
+- **Scripted benchmark:** 5-minute sustained load driver — rotating batch of 15 long-form technical prompts at 1.5 s cadence. Reflects peak throughput.
+
+| Metric | Chat session | Benchmark (5 min) |
+|---|---|---|
+| Peak tok/s | 67.1 | 67.0 |
+| Avg tok/s (under load) | 18.7 | **52.9** |
+| Avg TTFT | 1668 ms | 758 ms |
+| Min TTFT | 1502 ms | 538 ms |
+| Max TTFT | 1871 ms | 1165 ms |
+| GPU 0 util (avg) | 30.2% | 88.8% |
+| GPU 1 util (avg) | 30.1% | 88.7% |
+| VRAM per GPU | 259.6 GB | 259.6 GB |
+| Avg power (both GPUs) | 664 W | 788 W |
+| Peak power | 828 W | 839 W |
+| GPU temp | 65 °C | 65 °C |
+
+The lower avg utilisation in the chat session reflects idle gaps between conversational turns; peak throughput is identical at ~67 tok/s in both runs, indicating the hardware ceiling rather than a software one.
+
+### Generation throughput
+
+<img src="../../../.github/assets/images/trinity/trinity-throughput.png" alt="Trinity generation throughput — scripted benchmark">
+
+Generation and prompt-ingest tok/s over the 5-minute benchmark. The model sustains ~53 tok/s average at continuous load, peaking at 67 tok/s.
+
+### Per-GPU compute utilisation
+
+<img src="../../../.github/assets/images/trinity/trinity-per-gpu-util.png" alt="Trinity per-GPU utilisation — scripted benchmark">
+
+GPU 0 and GPU 1 track each other closely throughout the benchmark — TP=2 distributes attention and FFN layers symmetrically across both MI350Xs.
+
+### Per-GPU VRAM
+
+<img src="../../../.github/assets/images/trinity/trinity-per-gpu-vram.png" alt="Trinity per-GPU VRAM — scripted benchmark">
+
+Both GPUs hold 259.6 GB — model weights sharded evenly, plus KV cache. 90.2% of HBM3e is occupied at rest, leaving ~26 GB headroom per GPU for active KV cache during generation.
+
+### Per-GPU power
+
+<img src="../../../.github/assets/images/trinity/trinity-per-gpu-power.png" alt="Trinity per-GPU power — scripted benchmark">
+
+Power draw during active generation peaks at ~420 W per GPU, for a combined chassis draw well under the MI350X's 750 W TDP per card. Idle between prompts returns to ~300 W.
+
+### Latency
+
+<img src="../../../.github/assets/images/trinity/trinity-latency.png" alt="Trinity latency — scripted benchmark">
+
+Cumulative average TTFT and E2E latency over the benchmark. TTFT settles to ~758 ms avg once the KV cache is warm; E2E tracks request length as expected.
+
+### KV cache utilisation
+
+<img src="../../../.github/assets/images/trinity/trinity-kv-cache.png" alt="Trinity KV cache utilisation — scripted benchmark">
+
+KV cache occupancy rises as requests pile up during the sustained load phase. The queue depth stays near zero — the model keeps pace with the 1.5 s prompt cadence.
+
+### 8. Teardown
+
+```bash
+# Stop the container
+ssh -i ~/.ssh/spinifex-key ubuntu@$INSTANCE_IP 'docker stop trinity'
+
+# Terminate the instance — releases the 2× MI350X back to the Spinifex pool
+aws ec2 terminate-instances --instance-ids "$INSTANCE_ID"
+```
+
+The two MI350Xs are immediately available to Spinifex for a new workload once the instance terminates.
+
+### 9. Conclusion
+
+Trinity Large Preview demonstrates that frontier-class open-weight models are now within reach of a two-GPU bare-metal setup. The MoE architecture is crucial: 400B total parameters with only 13B active per token means the model generates at ~53 tok/s sustained and peaks at 67 tok/s on a pair of MI350Xs — throughput that would be difficult to match with a dense model of comparable quality at that memory footprint.
+
+Combining this with Spinifex as the infrastructure layer allows teams to own the entire workflow. Provisioning a `g7e.12xlarge` instance, pulling a 400 GB model, and serving it through vLLM took a handful of standard `aws ec2` commands and a single `docker run`. The same workflow runs identically on any Spinifex-managed bare-metal node, with the GPUs returned to the pool when instances terminate.
+
+For teams evaluating Trinity for production agentic workloads — tool calling, structured outputs, long-context reasoning — this represents a credible self-hosted deployment path: open weights, Apache 2.0 license, owned hardware and familiar tooling.

docs/service-interfaces.yaml

@@ -30,10 +30,12 @@ services:
     publishes:
       - "ec2.*"
       - "elbv2.*"
+      - "acm.*"
       - "iam.*"
     depends_on:
       - "ec2.*"
       - "elbv2.*"
+      - "acm.*"
       - "iam.*"
 
   spinifex_daemon:
@@ -50,6 +52,7 @@ services:
       - spinifex/daemon
       - spinifex/handlers/ec2
       - spinifex/handlers/elbv2
+      - spinifex/handlers/acm
       - spinifex/handlers/iam
     subscribes:
       # EC2 instance lifecycle.
@@ -81,6 +84,12 @@ services:
       - "spinifex.node.vms"
       - "spinifex.nodes.discover"
       - "spinifex.cluster.shutdown.progress"
+      # Coordinated cluster-shutdown phases (fan-out, no queue group).
+      - "spinifex.cluster.shutdown.gate"
+      - "spinifex.cluster.shutdown.drain"
+      - "spinifex.cluster.shutdown.storage"
+      - "spinifex.cluster.shutdown.persist"
+      - "spinifex.cluster.shutdown.infra"
       - "spinifex.storage.config"
       # ELBv2 control plane (handlers/elbv2/service_nats.go topic map).
       - "elbv2.CreateLoadBalancer"
@@ -95,6 +104,15 @@ services:
       - "elbv2.CreateListener"
       - "elbv2.DeleteListener"
       - "elbv2.DescribeListeners"
+      # ACM-alike certificate store (handlers/acm/service_nats.go topic map).
+      - "acm.ImportCertificate"
+      - "acm.DescribeCertificate"
+      - "acm.ListCertificates"
+      - "acm.DeleteCertificate"
+      - "elbv2.AddListenerCertificates"
+      - "elbv2.RemoveListenerCertificates"
+      - "elbv2.DescribeListenerCertificates"
+      - "elbv2.DescribeSSLPolicies"
     publishes:
       # → vpcd
       - "vpc.create"
@@ -121,6 +139,8 @@ services:
       - "ebs.delete"
       - "ebs.sync"
       - "ebs.snapshot.*"
+      - "ebs.config"
+      - "ebs.config.*"
       # → predastore (S3 / AMI / keypair store)
       - "s3.*"
       # IAM lifecycle event
@@ -131,7 +151,7 @@ services:
       - "s3.*"
 
   vpcd:
-    path: spinifex/services/vpcd
+    path: spinifex/vpcd
     # OVN-backed VPC / subnet / SG / NAT / IGW data plane.
     subscribes:
       - "vpc.create"
@@ -169,6 +189,8 @@ services:
       - "ebs.delete"
       - "ebs.sync"
       - "ebs.snapshot.*"
+      - "ebs.config"
+      - "ebs.config.*"
     publishes:
       - "ebs.mount.response"
       - "ebs.unmount.response"

docs/terraform-workbooks/eks-addons-access/README.md

@@ -0,0 +1,181 @@
+---
+title: "EKS with Addons + Access-Entry RBAC"
+description: "Extend a single-node EKS cluster with a managed CoreDNS addon, access-entry RBAC, and a browsable load-balanced demo app, using Terraform on Spinifex."
+category: "Terraform Workbooks"
+tags:
+  - terraform
+  - eks
+  - kubernetes
+  - addons
+  - rbac
+  - iam
+  - workbook
+resources:
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "Terraform Kubernetes Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "OpenTofu"
+    url: "https://opentofu.org/"
+---
+
+# Terraform: EKS with Addons + Access-Entry RBAC
+
+> Builds on the quickstart by installing a managed addon (CoreDNS) and granting a second IAM principal read-only cluster access — then deploys the same browsable, load-balanced demo app so you can see the result.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+This workbook takes the same single-node cluster (and the same browsable demo app) from [EKS Quickstart](../eks-quickstart) and adds the two things you reach for on almost any real cluster:
+
+1. **A managed addon.** CoreDNS is installed through the EKS API (`aws_eks_addon`) rather than `kubectl apply`, so its lifecycle is managed by the cluster. The demo app is deployed *after* CoreDNS, so a healthy page also confirms in-cluster DNS is working.
+2. **Access-entry RBAC.** A second IAM role is granted read-only access via an `aws_eks_access_entry` plus an `aws_eks_access_policy_association` bound to the AWS-managed `AmazonEKSViewPolicy`. This is the modern access-entry path (`authentication_mode = "API"`) — no `aws-auth` ConfigMap.
+
+When `apply` finishes, open the **`demo_url`** output and refresh — the `nginxdemos/hello` page reports which pod served each request.
+
+**What you'll learn:**
+
+- Installing a managed EKS addon with `aws_eks_addon`
+- Registering an IAM principal with `aws_eks_access_entry` and scoping it with `aws_eks_access_policy_association`
+- The difference between the cluster creator's admin access and a scoped viewer
+- Deploying a workload with the Kubernetes provider once the addon is healthy
+
+**What gets created**
+
+| Resource | Name | Purpose |
+|---|---|---|
+| VPC | `eks-addons-vpc` | Isolated network (10.31.0.0/16) |
+| Subnets | `eks-addons-subnet-a/-b` | Public subnets for the cluster and worker |
+| Internet Gateway | `eks-addons-igw` | Egress so the worker can pull the demo image |
+| IAM Roles | `eks-addons-cluster-role`, `eks-addons-node-role` | Control-plane and worker roles |
+| IAM Role | `eks-addons-viewer` | Stand-in principal granted read-only access |
+| EKS Cluster | `eks-addons` | Public endpoint, API auth mode |
+| Node Group | `default` | One `t3.medium` worker |
+| Addon | `coredns` | Cluster DNS, managed via the EKS API |
+| Access Entry + Assoc. | `eks-addons-viewer` → `AmazonEKSViewPolicy` | Read-only, cluster scope |
+| SG Ingress Rule | `eks-addons-demo-nodeport` | Opens the demo NodePort on the worker SG |
+| K8s Deployment + Service | `hello` | `nginxdemos/hello`, 2 replicas, NodePort |
+
+**Spinifex specifics**
+
+- Addon catalog is limited. Spinifex currently serves `coredns`, `aws-load-balancer-controller` and `spinifex-noop`. There is no `vpc-cni`, `kube-proxy`, `ebs-csi` or `metrics-server`.
+- **Leave `addon_version` unset.** Spinifex defaults an unset version to its catalog default (`coredns` 1.11.1). Pinning is a dead end: the AWS provider rejects a bare `1.11.1` (it requires the `v`-prefixed `v1.11.1-eksbuild.N` form), while Spinifex's catalog only accepts the bare `1.11.1` — so no single string satisfies both.
+- Cluster access policies are the four AWS-managed `cluster-access-policy/Amazon EKS{ClusterAdmin,Admin,Edit,View}Policy` ARNs.
+- The worker SG is auto-managed and the worker AMI is always the `eks-node` image; the NodePort is exposed by adding one rule to the looked-up worker SG.
+
+**Prerequisites:**
+
+- Spinifex installed and running
+- The Spinifex `eks-node` image available on the cluster
+- OpenTofu or Terraform, plus `kubectl` and the AWS CLI
+
+## Instructions
+
+### Step 1. Get the Template
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-addons-access
+```
+
+Or create a `main.tf` file and paste the full configuration below.
+
+<!-- INCLUDE: main.tf lang:hcl -->
+
+### Step 2. Deploy
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+```
+
+Allow a few minutes for the cluster to reach `ACTIVE` and the worker to register. CoreDNS installs onto the worker, then Terraform deploys the demo app.
+
+> Keep `AWS_PROFILE=spinifex` exported for the whole `apply` — the Kubernetes provider runs `aws eks get-token` against the Spinifex STS endpoint.
+
+### Step 3. Open the Demo
+
+```bash
+tofu output demo_url
+```
+
+Open it in a browser and refresh — the `nginxdemos/hello` page shows the pod that answered, alternating between the two replicas.
+
+### Step 4. Verify the Addon and Access Entry
+
+```bash
+aws eks list-addons --cluster-name eks-addons
+aws eks describe-addon --cluster-name eks-addons --addon-name coredns --query 'addon.status'
+
+aws eks list-access-entries --cluster-name eks-addons
+aws eks list-associated-access-policies \
+  --cluster-name eks-addons \
+  --principal-arn "$(tofu output -raw viewer_principal_arn)"
+```
+
+The viewer principal is bound to `AmazonEKSViewPolicy` at cluster scope — it can read across the cluster but cannot mutate anything. The principal that ran `apply` keeps full cluster-admin from `bootstrap_cluster_creator_admin_permissions`.
+
+### Cleanup
+
+```bash
+tofu destroy
+```
+
+## Troubleshooting
+
+### Demo URL Doesn't Load
+
+Same chain as the quickstart — check the worker's public IP, that the demo pods are `Running`, and that the NodePort rule landed on the worker SG:
+
+```bash
+kubectl get pods -o wide
+aws ec2 describe-security-groups --filters "Name=group-name,Values=eks-cluster-eks-addons-nodegroup-sg" \
+  --query 'SecurityGroups[0].IpPermissions'
+```
+
+If the demo pods are stuck `ContainerCreating` with DNS errors, confirm CoreDNS is healthy first:
+
+```bash
+kubectl -n kube-system get deploy coredns
+```
+
+### Addon Stuck in CREATING / DEGRADED
+
+CoreDNS needs a schedulable worker. Confirm the node group is `ACTIVE` and the node is `Ready`:
+
+```bash
+aws eks describe-nodegroup --cluster-name eks-addons --nodegroup-name default --query 'nodegroup.status'
+kubectl get nodes
+```
+
+### Addon Version Rejected
+
+This workbook leaves `addon_version` unset so Spinifex picks its catalog default. If you add an explicit `addon_version`, expect a wall: the AWS provider requires the `v`-prefixed `v1.11.1-eksbuild.N` form, but Spinifex's catalog only knows the bare `1.11.1` — a value that passes one is rejected by the other. Prefer omitting the field.
+
+### Access Entry Not Taking Effect
+
+Assume the viewer role and confirm it can read but not write:
+
+```bash
+kubectl auth can-i list pods --all-namespaces   # yes
+kubectl auth can-i create deployments           # no
+```
+
+### Provider Connection Refused
+
+```bash
+sudo systemctl status spinifex.target
+curl -k https://localhost:9999/
+```

docs/terraform-workbooks/eks-addons-access/main.tf

@@ -0,0 +1,468 @@
+# Example: EKS with an Addon + Access-Entry RBAC (with a visible demo app)
+#
+# Builds on eks-quickstart: same single-worker cluster and browsable demo app,
+# but adds two things you reach for on a real cluster:
+#
+#   * a managed addon (CoreDNS) installed through the EKS API, and
+#   * a second IAM principal granted scoped, read-only access via an EKS access
+#     entry + access-policy association (authentication_mode = "API").
+#
+# After `apply`, open the demo_url output — the page reports the pod that
+# served it, so a refresh shows requests landing on different replicas. CoreDNS
+# being healthy is what lets in-cluster name resolution work.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-addons-access
+#   export AWS_PROFILE=spinifex
+#   tofu init && tofu apply
+#   # then open the demo_url output
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.20"
+    }
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Variables
+# ---------------------------------------------------------------------------
+
+variable "region" {
+  type    = string
+  default = "ap-southeast-2"
+}
+
+variable "cluster_name" {
+  type    = string
+  default = "eks-addons"
+}
+
+variable "k8s_version" {
+  type    = string
+  default = "1.32"
+}
+
+variable "instance_type" {
+  type    = string
+  default = "t3.medium"
+}
+
+variable "node_port" {
+  type    = number
+  default = 30080
+}
+
+variable "replicas" {
+  type    = number
+  default = 2
+}
+
+variable "browse_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the demo NodePort; tighten to your own IP in production"
+}
+
+variable "spinifex_endpoint" {
+  type    = string
+  default = "https://127.0.0.1:9999"
+}
+
+# ---------------------------------------------------------------------------
+# Providers
+# ---------------------------------------------------------------------------
+
+provider "aws" {
+  region = var.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+provider "kubernetes" {
+  host                   = aws_eks_cluster.this.endpoint
+  cluster_ca_certificate = base64decode(aws_eks_cluster.this.certificate_authority[0].data)
+
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    command     = "aws"
+    args        = ["eks", "get-token", "--cluster-name", aws_eks_cluster.this.name, "--region", var.region]
+  }
+}
+
+data "aws_availability_zones" "available" {
+  state = "available"
+}
+
+# ---------------------------------------------------------------------------
+# VPC + two public subnets
+# ---------------------------------------------------------------------------
+
+resource "aws_vpc" "main" {
+  cidr_block           = "10.31.0.0/16"
+  enable_dns_hostnames = true
+  enable_dns_support   = true
+
+  tags = {
+    Name = "${var.cluster_name}-vpc"
+  }
+}
+
+resource "aws_internet_gateway" "igw" {
+  vpc_id = aws_vpc.main.id
+
+  tags = {
+    Name = "${var.cluster_name}-igw"
+  }
+}
+
+resource "aws_subnet" "a" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.31.1.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-subnet-a"
+  }
+}
+
+resource "aws_subnet" "b" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.31.2.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-subnet-b"
+  }
+}
+
+resource "aws_route_table" "public" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block = "0.0.0.0/0"
+    gateway_id = aws_internet_gateway.igw.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-public-rt"
+  }
+}
+
+resource "aws_route_table_association" "a" {
+  subnet_id      = aws_subnet.a.id
+  route_table_id = aws_route_table.public.id
+}
+
+resource "aws_route_table_association" "b" {
+  subnet_id      = aws_subnet.b.id
+  route_table_id = aws_route_table.public.id
+}
+
+# ---------------------------------------------------------------------------
+# IAM — cluster role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "cluster" {
+  name = "${var.cluster_name}-cluster-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "eks.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "cluster" {
+  role       = aws_iam_role.cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — node role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "node" {
+  name = "${var.cluster_name}-node-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_worker" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_cni" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_ecr" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — a second principal to grant read-only cluster access to
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "viewer" {
+  name = "${var.cluster_name}-viewer"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+# ---------------------------------------------------------------------------
+# EKS cluster
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_cluster" "this" {
+  name     = var.cluster_name
+  role_arn = aws_iam_role.cluster.arn
+  version  = var.k8s_version
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.a.id, aws_subnet.b.id]
+    endpoint_public_access  = true
+    endpoint_private_access = false
+  }
+
+  access_config {
+    authentication_mode                         = "API"
+    bootstrap_cluster_creator_admin_permissions = true
+  }
+
+  depends_on = [aws_iam_role_policy_attachment.cluster]
+
+  tags = {
+    Name = var.cluster_name
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Node group
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_node_group" "default" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "default"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.a.id, aws_subnet.b.id]
+
+  scaling_config {
+    desired_size = 1
+    min_size     = 1
+    max_size     = 2
+  }
+
+  instance_types = [var.instance_type]
+  ami_type       = "AL2_x86_64"
+
+  depends_on = [
+    aws_iam_role_policy_attachment.node_worker,
+    aws_iam_role_policy_attachment.node_cni,
+    aws_iam_role_policy_attachment.node_ecr,
+  ]
+
+  tags = {
+    Name = "${var.cluster_name}-default"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Addon — CoreDNS
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_addon" "coredns" {
+  cluster_name                = aws_eks_cluster.this.name
+  addon_name                  = "coredns"
+  resolve_conflicts_on_create = "OVERWRITE"
+
+  # addon_version omitted on purpose. Spinifex defaults an unset version to its
+  # catalog default (coredns 1.11.1). The AWS provider rejects a bare "1.11.1"
+  # (it demands a v-prefixed form) which Spinifex's catalog would in turn
+  # reject — so pinning is a dead end here; let the server choose.
+  depends_on = [aws_eks_node_group.default]
+
+  tags = {
+    Name = "${var.cluster_name}-coredns"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Access entry — grant the viewer principal read-only access
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_access_entry" "viewer" {
+  cluster_name  = aws_eks_cluster.this.name
+  principal_arn = aws_iam_role.viewer.arn
+  type          = "STANDARD"
+}
+
+resource "aws_eks_access_policy_association" "viewer" {
+  cluster_name  = aws_eks_cluster.this.name
+  principal_arn = aws_iam_role.viewer.arn
+  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
+
+  access_scope {
+    type = "cluster"
+  }
+
+  depends_on = [aws_eks_access_entry.viewer]
+}
+
+# ---------------------------------------------------------------------------
+# Open the NodePort on the auto-managed nodegroup SG (see eks-quickstart)
+# ---------------------------------------------------------------------------
+
+data "aws_security_group" "nodegroup" {
+  filter {
+    name   = "group-name"
+    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
+  }
+
+  filter {
+    name   = "vpc-id"
+    values = [aws_vpc.main.id]
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+resource "aws_vpc_security_group_ingress_rule" "nodeport" {
+  security_group_id = data.aws_security_group.nodegroup.id
+  cidr_ipv4         = var.browse_cidr
+  from_port         = var.node_port
+  to_port           = var.node_port
+  ip_protocol       = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-demo-nodeport"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Demo workload — deployed after CoreDNS so name resolution is up
+# ---------------------------------------------------------------------------
+
+resource "kubernetes_deployment_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+    labels    = { app = "hello" }
+  }
+
+  spec {
+    replicas = var.replicas
+
+    selector {
+      match_labels = { app = "hello" }
+    }
+
+    template {
+      metadata {
+        labels = { app = "hello" }
+      }
+
+      spec {
+        container {
+          name  = "hello"
+          image = "nginxdemos/hello"
+
+          port {
+            container_port = 80
+          }
+        }
+      }
+    }
+  }
+
+  depends_on = [aws_eks_addon.coredns]
+}
+
+resource "kubernetes_service_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+  }
+
+  spec {
+    selector = { app = "hello" }
+    type     = "NodePort"
+
+    port {
+      port        = 80
+      target_port = 80
+      node_port   = var.node_port
+    }
+  }
+
+  depends_on = [kubernetes_deployment_v1.hello]
+}
+
+data "aws_instances" "workers" {
+  instance_tags = {
+    "spinifex:eks-cluster" = aws_eks_cluster.this.name
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+# ---------------------------------------------------------------------------
+# Outputs
+# ---------------------------------------------------------------------------
+
+output "cluster_name" {
+  value = aws_eks_cluster.this.name
+}
+
+output "viewer_principal_arn" {
+  value = aws_iam_role.viewer.arn
+}
+
+output "demo_url" {
+  value       = "http://${data.aws_instances.workers.public_ips[0]}:${var.node_port}"
+  description = "Open in a browser; refresh to see different pods answer"
+}
+
+output "update_kubeconfig" {
+  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
+}

docs/terraform-workbooks/eks-https-ingress/README.md

@@ -0,0 +1,227 @@
+---
+title: "3-Node EKS behind an HTTPS ALB (ACM TLS)"
+description: "Provision an HA-shaped EKS cluster with public + private API endpoints and three workers, deploy a demo app, and publish it through an internet-facing ALB that terminates TLS with an ACM-imported certificate, using Terraform on Spinifex."
+category: "Terraform Workbooks"
+tags:
+  - terraform
+  - eks
+  - kubernetes
+  - acm
+  - tls
+  - alb
+  - elbv2
+  - vpc
+  - workbook
+resources:
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "Terraform Kubernetes Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "OpenTofu"
+    url: "https://opentofu.org/"
+---
+
+# Terraform: 3-Node EKS behind an HTTPS ALB
+
+> The largest EKS workbook, and the clearest demo. An HA-shaped cluster with public + private API endpoints and three workers, running a demo app published over HTTPS through an internet-facing ALB that terminates TLS with an ACM-imported certificate.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+This workbook is sized for a **3-node Spinifex deployment** and is built to show off. It brings together the pieces from the smaller EKS examples, deploys a demo app across three workers, and fronts it with a TLS-terminating load balancer. When `apply` finishes, fetch the ALB's public IP and open `https://<ip>` — the `nginxdemos/hello` page reports which pod answered, and refreshing moves between the three replicas spread across the cluster. That single page demonstrates a multi-node Kubernetes cluster load-balancing behind HTTPS.
+
+- **Public + private API endpoints.** `endpoint_public_access` serves `kubectl` and the Kubernetes provider from your host (CIDR-restricted); `endpoint_private_access` exposes an in-VPC private endpoint the workers use to join.
+- **Private workers + NAT.** The three workers sit in private subnets. They *join* the cluster over the in-VPC private endpoint, and a NAT gateway gives them outbound access to pull the demo container image.
+- **CoreDNS** installed as a managed addon.
+- **TLS via ACM.** A self-signed certificate is generated with the `tls` provider and **imported into ACM** (Spinifex ACM supports `ImportCertificate`, not `RequestCertificate`). The ALB's HTTPS listener references the ACM ARN and forwards to the workers' NodePort.
+- **The app, deployed by Terraform.** The Kubernetes provider deploys the Deployment + NodePort Service, and one security-group rule lets the ALB reach the NodePort.
+
+<p align="center">
+  <img src="../../../.github/assets/diagrams/tf-eks-https-ingress.svg" alt="3-node EKS — public+private API endpoints, ALB in public subnets terminating TLS, three workers in private subnets behind a NAT gateway" width="900">
+</p>
+
+**What you'll learn:**
+
+- Running an EKS cluster with both public and private API endpoints
+- Placing workers in private subnets that join over the private endpoint and egress via NAT
+- Importing a certificate into ACM and terminating TLS on an ALB HTTPS listener
+- Wiring an ALB target group to managed node-group workers via a NodePort, including the SG rule that allows it
+- Deploying a load-balanced workload with the Kubernetes provider
+
+**What gets created**
+
+| Resource | Name | Purpose |
+|---|---|---|
+| VPC | `eks-https-vpc` | Isolated network (10.32.0.0/16) |
+| Public Subnets | `eks-https-public-a/-b` | Host the ALB and NAT gateway |
+| Private Subnets | `eks-https-private-a/-b` | Host the workers |
+| Internet Gateway | `eks-https-igw` | Internet for the public subnets |
+| NAT Gateway (+ EIP) | `eks-https-nat` | Outbound for the private workers (image pulls) |
+| Route Tables | `eks-https-public-rt`, `eks-https-private-rt` | Public via IGW; private via NAT |
+| IAM Roles | `eks-https-cluster-role`, `eks-https-node-role` | Control-plane and worker roles |
+| EKS Cluster | `eks-https` | Public (CIDR-restricted) + private endpoints |
+| Node Group | `workers` | Three `t3.large` workers in the private subnets |
+| Addon | `coredns` | Cluster DNS |
+| ACM Cert | `eks-https-ingress` | Self-signed cert imported into ACM |
+| ALB + HTTPS Listener | `eks-https-alb` | Internet-facing, terminates TLS on :443 |
+| Target Group | `eks-https-tg` | Instance targets on the NodePort |
+| SG Ingress Rule | `eks-https-alb-nodeport` | Lets the ALB SG reach the worker NodePort |
+| K8s Deployment + Service | `hello` | `nginxdemos/hello`, 3 replicas, NodePort |
+
+**Spinifex specifics**
+
+- ACM supports `ImportCertificate`, `DescribeCertificate`, `ListCertificates`, `DeleteCertificate` — there is no `RequestCertificate`/DNS-validation flow, hence the self-signed import.
+- The ALB HTTPS listener resolves the cert from ACM. Valid SSL policies are `ELBSecurityPolicy-2016-08` (default) and `ELBSecurityPolicy-TLS13-1-2-2021-06`.
+- The worker SG is auto-managed (`vpc_config.security_group_ids` is ignored) and admits only intra-cluster traffic. The ALB→NodePort rule is added by looking the SG up by its deterministic name and admitting the ALB SG.
+- Worker discovery uses the `spinifex:eks-cluster` tag Spinifex stamps on node-group instances; the target-group attachment count is the known `node_desired_size`.
+- Leave `addon_version` unset (see the addons workbook for why pinning it can't satisfy both the provider and the catalog).
+
+**Prerequisites:**
+
+- A 3-node Spinifex deployment, installed and running
+- The Spinifex `eks-node` image available on the cluster
+- OpenTofu or Terraform, plus `kubectl` and the AWS CLI
+
+## Instructions
+
+### Step 1. Get the Template
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-https-ingress
+```
+
+Or create a `main.tf` file and paste the full configuration below.
+
+<!-- INCLUDE: main.tf lang:hcl -->
+
+### Step 2. Deploy
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+```
+
+Expect several minutes: the control plane bootstraps, three workers launch and join over the private endpoint, the NAT gateway comes up, CoreDNS installs, the ALB is provisioned, and the Kubernetes provider rolls out the demo app.
+
+> **Keep `AWS_PROFILE=spinifex` exported** for the whole `apply` — the Kubernetes provider authenticates with `aws eks get-token` against the Spinifex STS endpoint.
+
+> **Tighten access in production.** `api_public_access_cidr` and `alb_ingress_cidr` both default to `0.0.0.0/0`. Set them to your own CIDR:
+>
+> ```bash
+> export TF_VAR_api_public_access_cidr="203.0.113.10/32"
+> export TF_VAR_alb_ingress_cidr="203.0.113.10/32"
+> ```
+
+### Step 3. Open the Demo over HTTPS
+
+The ALB DNS name (`*.elb.spinifex.local`) won't resolve from your host, so fetch its public IP:
+
+```bash
+ALB_IP=$(aws elbv2 describe-load-balancers --names eks-https-alb \
+  --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
+  --output text)
+```
+
+The certificate is self-signed, so a browser will warn (proceed anyway) and `curl` needs `-k`:
+
+```bash
+curl -k https://$ALB_IP
+curl -k https://$ALB_IP   # refresh: the "Server name" changes between pods
+```
+
+Open `https://$ALB_IP` in a browser and refresh to watch requests land on different pods across the three nodes.
+
+### Step 4. Inspect (optional)
+
+```bash
+aws eks update-kubeconfig --name eks-https --region ap-southeast-2
+kubectl get nodes
+kubectl get pods -o wide          # three replicas, one per node
+
+TG_ARN=$(aws elbv2 describe-target-groups --names eks-https-tg \
+  --query 'TargetGroups[0].TargetGroupArn' --output text)
+aws elbv2 describe-target-health --target-group-arn "$TG_ARN"
+```
+
+### Cleanup
+
+```bash
+tofu destroy
+```
+
+## Troubleshooting
+
+### Targets Unhealthy / ALB Returns 5xx
+
+The ALB health-checks `/` on the NodePort. Targets turn healthy once the demo pods are running *and* the SG rule admits the ALB. Check, in order:
+
+```bash
+kubectl get pods -o wide                                  # pods Running?
+aws ec2 describe-security-groups \
+  --filters "Name=group-name,Values=eks-cluster-eks-https-nodegroup-sg" \
+  --query 'SecurityGroups[0].IpPermissions'               # NodePort rule present?
+aws elbv2 describe-target-health --target-group-arn "$TG_ARN"
+```
+
+If pods are `ImagePullBackOff`, the workers can't reach Docker Hub — confirm the NAT gateway is `available`:
+
+```bash
+aws ec2 describe-nat-gateways --query 'NatGateways[].[NatGatewayId,State]'
+```
+
+### Workers Never Join
+
+The workers join over the private endpoint. Confirm it's enabled and the node group is progressing:
+
+```bash
+aws eks describe-cluster --name eks-https --query 'cluster.resourcesVpcConfig.endpointPrivateAccess'
+aws eks describe-nodegroup --cluster-name eks-https --nodegroup-name workers --query 'nodegroup.status'
+```
+
+### Target-Group Attachment Errors at Apply
+
+The attachment indexes the discovered worker IDs by `node_desired_size`. If `apply` errors with an index out of range, the workers weren't all tagged/running when the `aws_instances` lookup ran. Re-run `tofu apply` — the node group will be `ACTIVE` by then:
+
+```bash
+aws ec2 describe-instances --filters "Name=tag:spinifex:eks-cluster,Values=eks-https" \
+  --query 'Reservations[].Instances[].[InstanceId,State.Name]' --output text
+```
+
+### kubernetes provider: connection refused / Unauthorized
+
+The provider runs `aws eks get-token` against the Spinifex STS endpoint. If the cluster was still `CREATING` when the provider first connected, re-run `tofu apply`. Otherwise confirm:
+
+```bash
+aws sts get-caller-identity
+aws eks describe-cluster --name eks-https --query 'cluster.status'
+```
+
+### ACM Import / HTTPS Listener Errors
+
+```bash
+aws acm list-certificates
+aws acm describe-certificate --certificate-arn "$(tofu output -raw certificate_arn)" \
+  --query 'Certificate.Status'
+```
+
+A `CertificateNotFound` on the listener means the ACM ARN didn't resolve — check the cert exists and belongs to the same account/profile.
+
+### Provider Connection Refused
+
+```bash
+sudo systemctl status spinifex.target
+curl -k https://localhost:9999/
+```

docs/terraform-workbooks/eks-https-ingress/main.tf

@@ -0,0 +1,671 @@
+# Example: 3-Node EKS behind an HTTPS ALB (ACM TLS) with a load-balanced app
+#
+# The largest EKS workbook, and the one with the clearest payoff. It stands up
+# an HA-shaped cluster sized for a 3-node Spinifex deployment, deploys a demo
+# web app across the workers, and publishes it through an internet-facing ALB
+# that terminates TLS with a certificate held in ACM.
+#
+# After `apply`, fetch the ALB's public IP (see alb_public_ip_hint) and open
+# https://<ip> in a browser. The page reports which pod answered; refresh and
+# it moves between the three replicas spread across the cluster — a live picture
+# of a multi-node Kubernetes cluster load-balancing behind HTTPS.
+#
+# What it builds:
+#   * VPC with two public subnets (ALB + NAT) and two private subnets (workers).
+#   * A NAT gateway so the private workers can pull the demo container image.
+#   * EKS cluster with BOTH a public endpoint (kubectl from your host) and a
+#     private endpoint (the in-VPC path the workers use to join).
+#   * A managed node group of three workers (t3.large) in the private subnets.
+#   * CoreDNS addon.
+#   * A self-signed cert imported into ACM (Spinifex ACM supports
+#     ImportCertificate, not RequestCertificate), wired to the ALB's HTTPS
+#     listener, which forwards to the workers' NodePort.
+#   * The demo Deployment + NodePort Service, deployed by the Kubernetes
+#     provider, plus the one SG rule that lets the ALB reach the NodePort.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-https-ingress
+#   export AWS_PROFILE=spinifex
+#   tofu init && tofu apply
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    tls = {
+      source  = "hashicorp/tls"
+      version = ">= 4.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.20"
+    }
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Variables
+# ---------------------------------------------------------------------------
+
+variable "region" {
+  type    = string
+  default = "ap-southeast-2"
+}
+
+variable "cluster_name" {
+  type    = string
+  default = "eks-https"
+}
+
+variable "k8s_version" {
+  type    = string
+  default = "1.32"
+}
+
+variable "node_instance_type" {
+  type    = string
+  default = "t3.large"
+}
+
+variable "node_desired_size" {
+  type        = number
+  default     = 3
+  description = "Worker count; spread across a 3-node Spinifex deployment for HA"
+}
+
+variable "node_port" {
+  type        = number
+  default     = 30080
+  description = "NodePort the ALB forwards to and the demo Service is published on"
+}
+
+variable "cert_common_name" {
+  type    = string
+  default = "eks-https.spinifex.local"
+}
+
+variable "api_public_access_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the public Kubernetes API endpoint; tighten in production"
+}
+
+variable "alb_ingress_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the ALB HTTPS listener"
+}
+
+variable "spinifex_endpoint" {
+  type    = string
+  default = "https://127.0.0.1:9999"
+}
+
+# ---------------------------------------------------------------------------
+# Providers
+# ---------------------------------------------------------------------------
+
+provider "aws" {
+  region = var.region
+
+  endpoints {
+    ec2                    = var.spinifex_endpoint
+    iam                    = var.spinifex_endpoint
+    sts                    = var.spinifex_endpoint
+    eks                    = var.spinifex_endpoint
+    acm                    = var.spinifex_endpoint
+    elasticloadbalancingv2 = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+provider "kubernetes" {
+  host                   = aws_eks_cluster.this.endpoint
+  cluster_ca_certificate = base64decode(aws_eks_cluster.this.certificate_authority[0].data)
+
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    command     = "aws"
+    args        = ["eks", "get-token", "--cluster-name", aws_eks_cluster.this.name, "--region", var.region]
+  }
+}
+
+data "aws_availability_zones" "available" {
+  state = "available"
+}
+
+# ---------------------------------------------------------------------------
+# VPC — public subnets for the ALB + NAT, private subnets for the workers
+# ---------------------------------------------------------------------------
+
+resource "aws_vpc" "main" {
+  cidr_block           = "10.32.0.0/16"
+  enable_dns_hostnames = true
+  enable_dns_support   = true
+
+  tags = {
+    Name = "${var.cluster_name}-vpc"
+  }
+}
+
+resource "aws_internet_gateway" "igw" {
+  vpc_id = aws_vpc.main.id
+
+  tags = {
+    Name = "${var.cluster_name}-igw"
+  }
+}
+
+resource "aws_subnet" "public_a" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.32.1.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-public-a"
+  }
+}
+
+resource "aws_subnet" "public_b" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.32.2.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-public-b"
+  }
+}
+
+resource "aws_subnet" "private_a" {
+  vpc_id            = aws_vpc.main.id
+  cidr_block        = "10.32.11.0/24"
+  availability_zone = data.aws_availability_zones.available.names[0]
+
+  tags = {
+    Name = "${var.cluster_name}-private-a"
+  }
+}
+
+resource "aws_subnet" "private_b" {
+  vpc_id            = aws_vpc.main.id
+  cidr_block        = "10.32.12.0/24"
+  availability_zone = data.aws_availability_zones.available.names[0]
+
+  tags = {
+    Name = "${var.cluster_name}-private-b"
+  }
+}
+
+resource "aws_route_table" "public" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block = "0.0.0.0/0"
+    gateway_id = aws_internet_gateway.igw.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-public-rt"
+  }
+}
+
+resource "aws_route_table_association" "public_a" {
+  subnet_id      = aws_subnet.public_a.id
+  route_table_id = aws_route_table.public.id
+}
+
+resource "aws_route_table_association" "public_b" {
+  subnet_id      = aws_subnet.public_b.id
+  route_table_id = aws_route_table.public.id
+}
+
+# NAT gateway gives the private workers outbound internet to pull the demo
+# container image. The workers still *join* the cluster over the in-VPC private
+# endpoint, not through the NAT.
+resource "aws_eip" "nat" {
+  domain = "vpc"
+
+  tags = {
+    Name = "${var.cluster_name}-nat-eip"
+  }
+}
+
+resource "aws_nat_gateway" "nat" {
+  allocation_id = aws_eip.nat.id
+  subnet_id     = aws_subnet.public_a.id
+
+  depends_on = [aws_internet_gateway.igw]
+
+  tags = {
+    Name = "${var.cluster_name}-nat"
+  }
+}
+
+resource "aws_route_table" "private" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block     = "0.0.0.0/0"
+    nat_gateway_id = aws_nat_gateway.nat.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-private-rt"
+  }
+}
+
+resource "aws_route_table_association" "private_a" {
+  subnet_id      = aws_subnet.private_a.id
+  route_table_id = aws_route_table.private.id
+}
+
+resource "aws_route_table_association" "private_b" {
+  subnet_id      = aws_subnet.private_b.id
+  route_table_id = aws_route_table.private.id
+}
+
+# ---------------------------------------------------------------------------
+# IAM — cluster role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "cluster" {
+  name = "${var.cluster_name}-cluster-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "eks.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "cluster" {
+  role       = aws_iam_role.cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — node role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "node" {
+  name = "${var.cluster_name}-node-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_worker" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_cni" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_ecr" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+}
+
+# ---------------------------------------------------------------------------
+# EKS cluster — public + private endpoints
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_cluster" "this" {
+  name     = var.cluster_name
+  role_arn = aws_iam_role.cluster.arn
+  version  = var.k8s_version
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.private_a.id, aws_subnet.private_b.id]
+    endpoint_public_access  = true
+    endpoint_private_access = true
+    public_access_cidrs     = [var.api_public_access_cidr]
+  }
+
+  access_config {
+    authentication_mode                         = "API"
+    bootstrap_cluster_creator_admin_permissions = true
+  }
+
+  depends_on = [aws_iam_role_policy_attachment.cluster]
+
+  tags = {
+    Name = var.cluster_name
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Managed node group — three workers in the private subnets
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_node_group" "workers" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "workers"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.private_a.id, aws_subnet.private_b.id]
+
+  scaling_config {
+    desired_size = var.node_desired_size
+    min_size     = var.node_desired_size
+    max_size     = var.node_desired_size * 2
+  }
+
+  instance_types = [var.node_instance_type]
+  ami_type       = "AL2_x86_64"
+
+  depends_on = [
+    aws_iam_role_policy_attachment.node_worker,
+    aws_iam_role_policy_attachment.node_cni,
+    aws_iam_role_policy_attachment.node_ecr,
+  ]
+
+  tags = {
+    Name = "${var.cluster_name}-workers"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Addon — CoreDNS
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_addon" "coredns" {
+  cluster_name                = aws_eks_cluster.this.name
+  addon_name                  = "coredns"
+  resolve_conflicts_on_create = "OVERWRITE"
+
+  # addon_version omitted on purpose. Spinifex defaults an unset version to its
+  # catalog default (coredns 1.11.1). The AWS provider also rejects a bare
+  # "1.11.1" (it demands a v-prefixed form) which Spinifex's catalog would in
+  # turn reject — so pinning is a dead end here; let the server choose.
+  depends_on = [aws_eks_node_group.workers]
+
+  tags = {
+    Name = "${var.cluster_name}-coredns"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# TLS — self-signed certificate imported into ACM
+# ---------------------------------------------------------------------------
+
+resource "tls_private_key" "ingress" {
+  algorithm = "RSA"
+  rsa_bits  = 2048
+}
+
+resource "tls_self_signed_cert" "ingress" {
+  private_key_pem = tls_private_key.ingress.private_key_pem
+
+  subject {
+    common_name  = var.cert_common_name
+    organization = "Spinifex EKS Demo"
+  }
+
+  dns_names             = [var.cert_common_name]
+  validity_period_hours = 8760
+  early_renewal_hours   = 720
+
+  allowed_uses = [
+    "key_encipherment",
+    "digital_signature",
+    "server_auth",
+  ]
+}
+
+resource "aws_acm_certificate" "ingress" {
+  private_key      = tls_private_key.ingress.private_key_pem
+  certificate_body = tls_self_signed_cert.ingress.cert_pem
+
+  tags = {
+    Name = "${var.cluster_name}-ingress"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# ALB — internet-facing, HTTPS terminated with the ACM cert
+# ---------------------------------------------------------------------------
+
+resource "aws_security_group" "alb" {
+  name        = "${var.cluster_name}-alb-sg"
+  description = "ALB HTTPS inbound, NodePort to workers"
+  vpc_id      = aws_vpc.main.id
+
+  ingress {
+    description = "HTTPS"
+    from_port   = 443
+    to_port     = 443
+    protocol    = "tcp"
+    cidr_blocks = [var.alb_ingress_cidr]
+  }
+
+  egress {
+    description = "All outbound"
+    from_port   = 0
+    to_port     = 0
+    protocol    = "-1"
+    cidr_blocks = ["0.0.0.0/0"]
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-alb-sg"
+  }
+}
+
+resource "aws_lb" "ingress" {
+  name               = "${var.cluster_name}-alb"
+  internal           = false
+  load_balancer_type = "application"
+  security_groups    = [aws_security_group.alb.id]
+  subnets            = [aws_subnet.public_a.id, aws_subnet.public_b.id]
+
+  tags = {
+    Name = "${var.cluster_name}-alb"
+  }
+}
+
+resource "aws_lb_target_group" "workers" {
+  name        = "${var.cluster_name}-tg"
+  port        = var.node_port
+  protocol    = "HTTP"
+  target_type = "instance"
+  vpc_id      = aws_vpc.main.id
+
+  health_check {
+    path                = "/"
+    protocol            = "HTTP"
+    matcher             = "200-399"
+    healthy_threshold   = 2
+    unhealthy_threshold = 3
+    timeout             = 5
+    interval            = 10
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-tg"
+  }
+}
+
+# Discover the launched workers by their Spinifex EKS tag so the ALB can target
+# them. depends_on the node group: the lookup must run after the workers exist.
+data "aws_instances" "workers" {
+  instance_tags = {
+    "spinifex:eks-cluster" = aws_eks_cluster.this.name
+  }
+
+  depends_on = [aws_eks_node_group.workers]
+}
+
+# count is the known desired size, so it's resolvable at plan time even though
+# the worker IDs themselves are only known after the node group launches.
+resource "aws_lb_target_group_attachment" "workers" {
+  count            = var.node_desired_size
+  target_group_arn = aws_lb_target_group.workers.arn
+  target_id        = data.aws_instances.workers.ids[count.index]
+  port             = var.node_port
+}
+
+resource "aws_lb_listener" "https" {
+  load_balancer_arn = aws_lb.ingress.arn
+  port              = 443
+  protocol          = "HTTPS"
+  ssl_policy        = "ELBSecurityPolicy-TLS13-1-2-2021-06"
+  certificate_arn   = aws_acm_certificate.ingress.arn
+
+  default_action {
+    type             = "forward"
+    target_group_arn = aws_lb_target_group.workers.arn
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Let the ALB reach the workers' NodePort
+#
+# Spinifex auto-manages the nodegroup SG and admits only intra-cluster traffic.
+# Look it up by its deterministic name and admit the ALB SG on the NodePort.
+# ---------------------------------------------------------------------------
+
+data "aws_security_group" "nodegroup" {
+  filter {
+    name   = "group-name"
+    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
+  }
+
+  filter {
+    name   = "vpc-id"
+    values = [aws_vpc.main.id]
+  }
+
+  depends_on = [aws_eks_node_group.workers]
+}
+
+resource "aws_vpc_security_group_ingress_rule" "nodeport_from_alb" {
+  security_group_id            = data.aws_security_group.nodegroup.id
+  referenced_security_group_id = aws_security_group.alb.id
+  from_port                    = var.node_port
+  to_port                      = var.node_port
+  ip_protocol                  = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-alb-nodeport"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Demo workload — three replicas spread across the workers
+# ---------------------------------------------------------------------------
+
+resource "kubernetes_deployment_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+    labels    = { app = "hello" }
+  }
+
+  spec {
+    replicas = var.node_desired_size
+
+    selector {
+      match_labels = { app = "hello" }
+    }
+
+    template {
+      metadata {
+        labels = { app = "hello" }
+      }
+
+      spec {
+        container {
+          name  = "hello"
+          image = "nginxdemos/hello"
+
+          port {
+            container_port = 80
+          }
+        }
+
+        # Spread one replica per node so a refresh visibly hits different nodes.
+        topology_spread_constraint {
+          max_skew           = 1
+          topology_key       = "kubernetes.io/hostname"
+          when_unsatisfiable = "ScheduleAnyway"
+
+          label_selector {
+            match_labels = { app = "hello" }
+          }
+        }
+      }
+    }
+  }
+
+  depends_on = [aws_eks_addon.coredns]
+}
+
+resource "kubernetes_service_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+  }
+
+  spec {
+    selector = { app = "hello" }
+    type     = "NodePort"
+
+    port {
+      port        = 80
+      target_port = 80
+      node_port   = var.node_port
+    }
+  }
+
+  depends_on = [kubernetes_deployment_v1.hello]
+}
+
+# ---------------------------------------------------------------------------
+# Outputs
+# ---------------------------------------------------------------------------
+
+output "cluster_name" {
+  value = aws_eks_cluster.this.name
+}
+
+output "certificate_arn" {
+  value = aws_acm_certificate.ingress.arn
+}
+
+output "alb_dns_name" {
+  value = aws_lb.ingress.dns_name
+}
+
+output "alb_public_ip_hint" {
+  value = "aws elbv2 describe-load-balancers --names ${var.cluster_name}-alb --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' --output text"
+}
+
+output "demo_url_hint" {
+  value = "Fetch the ALB IP via alb_public_ip_hint, then open https://<that-ip> (self-signed cert: curl -k). Refresh to see different pods answer."
+}
+
+output "update_kubeconfig" {
+  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
+}

docs/terraform-workbooks/eks-quickstart/README.md

@@ -0,0 +1,173 @@
+---
+title: "EKS Quickstart"
+description: "Stand up a minimal managed-Kubernetes cluster and a browsable demo app — VPC, IAM roles, an EKS cluster, a single-worker node group, and a load-balanced web page — using Terraform on Spinifex."
+category: "Terraform Workbooks"
+tags:
+  - terraform
+  - eks
+  - kubernetes
+  - iam
+  - vpc
+  - workbook
+resources:
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "Terraform Kubernetes Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "OpenTofu"
+    url: "https://opentofu.org/"
+---
+
+# Terraform: EKS Quickstart
+
+> The smallest end-to-end EKS example on Spinifex: a VPC, the two IAM roles EKS needs, a single-worker cluster, and a demo web app you can open in a browser.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+Provision a managed Kubernetes cluster on Spinifex with Terraform/OpenTofu — and finish with something you can actually see. As well as the cluster, this workbook deploys a small demo app onto it and publishes it on the worker's public IP. When `apply` finishes, open the **`demo_url`** output: the page is served by `nginxdemos/hello` and prints the name of the pod that answered. Refresh it and the answering pod changes, so even a non-technical viewer can watch Kubernetes scheduling and load-balancing in real time.
+
+Under the hood it keeps things minimal: a VPC with two public subnets, an IAM role for the control plane, an IAM role for the workers with the three managed policies AWS-managed node groups expect, a public-endpoint `aws_eks_cluster`, and a single-node `aws_eks_node_group`. Terraform then uses the **Kubernetes provider** (authenticating exactly like `kubectl` does, via `aws eks get-token`) to deploy the Deployment + NodePort Service.
+
+**What you'll learn:**
+
+- Configuring the AWS provider to target Spinifex's `eks`, `ec2`, `iam` and `sts` endpoints
+- Creating the EKS cluster and worker IAM roles with faithful managed-policy attachments
+- Provisioning an `aws_eks_cluster` and a managed `aws_eks_node_group`
+- Pointing the Kubernetes provider at the new cluster and deploying a workload from the same `apply`
+- Opening one rule on the auto-managed worker security group to expose a NodePort
+
+**What gets created**
+
+| Resource | Name | Purpose |
+|---|---|---|
+| VPC | `eks-quickstart-vpc` | Isolated network (10.30.0.0/16) |
+| Subnets | `eks-quickstart-subnet-a/-b` | Public subnets for the cluster and worker |
+| Internet Gateway | `eks-quickstart-igw` | Egress so the worker can pull the demo image |
+| IAM Roles | `eks-quickstart-cluster-role`, `eks-quickstart-node-role` | Control-plane and worker roles |
+| EKS Cluster | `eks-quickstart` | Public API endpoint, API auth mode, Kubernetes 1.32 |
+| Node Group | `default` | One `t3.medium` worker |
+| SG Ingress Rule | `eks-quickstart-demo-nodeport` | Opens the demo NodePort on the auto-managed worker SG |
+| K8s Deployment | `hello` | `nginxdemos/hello`, 2 replicas |
+| K8s Service | `hello` | NodePort publishing the demo on the worker |
+
+**Spinifex specifics**
+
+- The cluster's security groups are **auto-managed** — `vpc_config.security_group_ids` is ignored. To reach a NodePort, this workbook looks the worker SG up by its deterministic name (`eks-cluster-<name>-nodegroup-sg`) and adds a single ingress rule.
+- The worker AMI is always Spinifex's `eks-node` image; `ami_type` is recorded but does **not** select the image.
+- `authentication_mode` must be `"API"` (the `API_AND_CONFIG_MAP` mode is rejected).
+- The workers need outbound internet (here via the IGW) to pull the demo container image.
+
+**Prerequisites:**
+
+- Spinifex installed and running (see [Installing Spinifex](/docs/install))
+- The Spinifex `eks-node` image available on the cluster
+- OpenTofu or Terraform, plus `kubectl` and the AWS CLI
+
+## Instructions
+
+### Step 1. Get the Template
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-quickstart
+```
+
+Or create a `main.tf` file and paste the full configuration below.
+
+<!-- INCLUDE: main.tf lang:hcl -->
+
+### Step 2. Deploy
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+```
+
+This single `apply` does the whole job: it creates the cluster (which bootstraps a control-plane VM and brings up k3s — a few minutes in `CREATING`), launches the worker, opens the NodePort, then deploys the demo app and waits for the pods to roll out.
+
+> **Same profile for the Kubernetes provider.** The Kubernetes provider authenticates by shelling out to `aws eks get-token`, which has to reach the Spinifex STS endpoint. Keep `AWS_PROFILE=spinifex` exported for the whole `apply`.
+
+### Step 3. Open the Demo
+
+```bash
+tofu output demo_url
+```
+
+Open that URL in a browser. You'll see the `nginxdemos/hello` page showing the **Server name** (the pod that handled the request). Refresh a few times — with two replicas, the pod name alternates, demonstrating that the Service is load-balancing across the cluster.
+
+### Step 4. Inspect with kubectl (optional)
+
+```bash
+aws eks update-kubeconfig --name eks-quickstart --region ap-southeast-2
+kubectl get nodes
+kubectl get pods -o wide
+```
+
+`kubectl get pods -o wide` shows the demo pods and which node each landed on.
+
+### Cleanup
+
+```bash
+tofu destroy
+```
+
+## Troubleshooting
+
+### Demo URL Doesn't Load
+
+The page is served from a NodePort on the worker's public IP. Work through the chain:
+
+```bash
+# Is the worker running with a public IP?
+aws ec2 describe-instances --filters "Name=tag:spinifex:eks-cluster,Values=eks-quickstart" \
+  --query 'Reservations[].Instances[].[InstanceId,State.Name,PublicIpAddress]' --output text
+
+# Did the demo pods roll out?
+kubectl get pods -o wide
+
+# Is the NodePort rule on the worker SG?
+aws ec2 describe-security-groups --filters "Name=group-name,Values=eks-cluster-eks-quickstart-nodegroup-sg" \
+  --query 'SecurityGroups[0].IpPermissions'
+```
+
+If the pods are `Pending`, the worker may not be `Ready` yet — give it a moment. If they're `ImagePullBackOff`, the worker can't reach Docker Hub; confirm the IGW route and the worker's public IP.
+
+### kubernetes provider: connection refused / Unauthorized
+
+The provider runs `aws eks get-token` against the Spinifex STS endpoint. Confirm the AWS CLI is still pointed at Spinifex and the cluster is `ACTIVE`:
+
+```bash
+aws sts get-caller-identity
+aws eks describe-cluster --name eks-quickstart --query 'cluster.status'
+```
+
+If the cluster was still `CREATING` when the provider first tried to connect, just re-run `tofu apply`.
+
+### Cluster Stuck in CREATING
+
+Control-plane bootstrap takes a few minutes. Confirm the underlying VM came up:
+
+```bash
+aws eks describe-cluster --name eks-quickstart --query 'cluster.status'
+aws ec2 describe-instances --profile spinifex
+```
+
+### Provider Connection Refused
+
+```bash
+sudo systemctl status spinifex.target
+curl -k https://localhost:9999/
+```

docs/terraform-workbooks/eks-quickstart/main.tf

@@ -0,0 +1,428 @@
+# Example: EKS Quickstart on Spinifex
+#
+# A minimal managed-Kubernetes cluster that ends in something you can see: a
+# VPC, the two IAM roles EKS needs, a single-worker cluster, and a demo web app
+# deployed onto it by Terraform and exposed on the worker's public IP.
+#
+# Once `apply` finishes, open the demo_url output in a browser. The page is
+# served by the nginxdemos/hello image and prints the pod name that answered —
+# refresh and it bounces between the app's replicas, showing the cluster is
+# really scheduling and load-balancing pods.
+#
+# Demonstrates: VPC + subnets, EKS cluster + node IAM roles, an aws_eks_cluster
+# and managed aws_eks_node_group, the Kubernetes provider authenticating to the
+# cluster, and a Deployment + NodePort Service reachable from your browser.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-quickstart
+#   export AWS_PROFILE=spinifex
+#   tofu init && tofu apply
+#   # then open the demo_url output
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.20"
+    }
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Variables
+# ---------------------------------------------------------------------------
+
+variable "region" {
+  type    = string
+  default = "ap-southeast-2"
+}
+
+variable "cluster_name" {
+  type    = string
+  default = "eks-quickstart"
+}
+
+variable "k8s_version" {
+  type        = string
+  default     = "1.32"
+  description = "Kubernetes minor version for the control plane and workers"
+}
+
+variable "instance_type" {
+  type    = string
+  default = "t3.medium"
+}
+
+variable "node_port" {
+  type        = number
+  default     = 30080
+  description = "NodePort the demo Service is published on"
+}
+
+variable "replicas" {
+  type        = number
+  default     = 2
+  description = "Demo app replicas; refresh the page to see requests land on different pods"
+}
+
+variable "browse_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the demo NodePort; tighten to your own IP in production"
+}
+
+variable "spinifex_endpoint" {
+  type        = string
+  default     = "https://127.0.0.1:9999"
+  description = "Spinifex AWS gateway endpoint"
+}
+
+# ---------------------------------------------------------------------------
+# Providers
+# ---------------------------------------------------------------------------
+
+provider "aws" {
+  region = var.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+# Authenticates to the cluster with the same `aws eks get-token` exec flow the
+# generated kubeconfig uses, so the Kubernetes provider can deploy the demo app.
+provider "kubernetes" {
+  host                   = aws_eks_cluster.this.endpoint
+  cluster_ca_certificate = base64decode(aws_eks_cluster.this.certificate_authority[0].data)
+
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    command     = "aws"
+    args        = ["eks", "get-token", "--cluster-name", aws_eks_cluster.this.name, "--region", var.region]
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Data sources
+# ---------------------------------------------------------------------------
+
+data "aws_availability_zones" "available" {
+  state = "available"
+}
+
+# ---------------------------------------------------------------------------
+# VPC + two public subnets (workers get public IPs to pull the demo image)
+# ---------------------------------------------------------------------------
+
+resource "aws_vpc" "main" {
+  cidr_block           = "10.30.0.0/16"
+  enable_dns_hostnames = true
+  enable_dns_support   = true
+
+  tags = {
+    Name = "${var.cluster_name}-vpc"
+  }
+}
+
+resource "aws_internet_gateway" "igw" {
+  vpc_id = aws_vpc.main.id
+
+  tags = {
+    Name = "${var.cluster_name}-igw"
+  }
+}
+
+resource "aws_subnet" "a" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.30.1.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-subnet-a"
+  }
+}
+
+resource "aws_subnet" "b" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.30.2.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-subnet-b"
+  }
+}
+
+resource "aws_route_table" "public" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block = "0.0.0.0/0"
+    gateway_id = aws_internet_gateway.igw.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-public-rt"
+  }
+}
+
+resource "aws_route_table_association" "a" {
+  subnet_id      = aws_subnet.a.id
+  route_table_id = aws_route_table.public.id
+}
+
+resource "aws_route_table_association" "b" {
+  subnet_id      = aws_subnet.b.id
+  route_table_id = aws_route_table.public.id
+}
+
+# ---------------------------------------------------------------------------
+# IAM — EKS cluster role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "cluster" {
+  name = "${var.cluster_name}-cluster-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "eks.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "cluster" {
+  role       = aws_iam_role.cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — worker node role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "node" {
+  name = "${var.cluster_name}-node-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_worker" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_cni" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_ecr" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+}
+
+# ---------------------------------------------------------------------------
+# EKS cluster — public API endpoint, API (access-entry) auth mode
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_cluster" "this" {
+  name     = var.cluster_name
+  role_arn = aws_iam_role.cluster.arn
+  version  = var.k8s_version
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.a.id, aws_subnet.b.id]
+    endpoint_public_access  = true
+    endpoint_private_access = false
+  }
+
+  access_config {
+    authentication_mode                         = "API"
+    bootstrap_cluster_creator_admin_permissions = true
+  }
+
+  depends_on = [aws_iam_role_policy_attachment.cluster]
+
+  tags = {
+    Name = var.cluster_name
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Managed node group — one worker
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_node_group" "default" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "default"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.a.id, aws_subnet.b.id]
+
+  scaling_config {
+    desired_size = 1
+    min_size     = 1
+    max_size     = 2
+  }
+
+  instance_types = [var.instance_type]
+  ami_type       = "AL2_x86_64"
+
+  depends_on = [
+    aws_iam_role_policy_attachment.node_worker,
+    aws_iam_role_policy_attachment.node_cni,
+    aws_iam_role_policy_attachment.node_ecr,
+  ]
+
+  tags = {
+    Name = "${var.cluster_name}-default"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Open the NodePort on the auto-managed nodegroup SG
+#
+# Spinifex creates the worker SG itself (vpc_config.security_group_ids is
+# ignored) and admits only intra-cluster traffic. To reach the demo NodePort
+# from a browser, look the SG up by its deterministic name and add one rule.
+# ---------------------------------------------------------------------------
+
+data "aws_security_group" "nodegroup" {
+  filter {
+    name   = "group-name"
+    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
+  }
+
+  filter {
+    name   = "vpc-id"
+    values = [aws_vpc.main.id]
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+resource "aws_vpc_security_group_ingress_rule" "nodeport" {
+  security_group_id = data.aws_security_group.nodegroup.id
+  cidr_ipv4         = var.browse_cidr
+  from_port         = var.node_port
+  to_port           = var.node_port
+  ip_protocol       = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-demo-nodeport"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Demo workload — deployed onto the cluster by Terraform
+#
+# nginxdemos/hello renders a page showing which pod served the request; with
+# multiple replicas, refreshing the demo_url alternates between them.
+# ---------------------------------------------------------------------------
+
+resource "kubernetes_deployment_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+    labels    = { app = "hello" }
+  }
+
+  spec {
+    replicas = var.replicas
+
+    selector {
+      match_labels = { app = "hello" }
+    }
+
+    template {
+      metadata {
+        labels = { app = "hello" }
+      }
+
+      spec {
+        container {
+          name  = "hello"
+          image = "nginxdemos/hello"
+
+          port {
+            container_port = 80
+          }
+        }
+      }
+    }
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+resource "kubernetes_service_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+  }
+
+  spec {
+    selector = { app = "hello" }
+    type     = "NodePort"
+
+    port {
+      port        = 80
+      target_port = 80
+      node_port   = var.node_port
+    }
+  }
+
+  depends_on = [kubernetes_deployment_v1.hello]
+}
+
+# ---------------------------------------------------------------------------
+# Discover the worker's public IP for the demo URL
+# ---------------------------------------------------------------------------
+
+data "aws_instances" "workers" {
+  instance_tags = {
+    "spinifex:eks-cluster" = aws_eks_cluster.this.name
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+# ---------------------------------------------------------------------------
+# Outputs
+# ---------------------------------------------------------------------------
+
+output "cluster_name" {
+  value = aws_eks_cluster.this.name
+}
+
+output "demo_url" {
+  value       = "http://${data.aws_instances.workers.public_ips[0]}:${var.node_port}"
+  description = "Open in a browser; refresh to see different pods answer"
+}
+
+output "update_kubeconfig" {
+  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
+}

docs/terraform-workbooks/nginx-alb/README.md

@@ -142,7 +142,6 @@ The ALB is internet-facing, but the DNS name Spinifex returns (`*.elb.spinifex.l
 ALB_IP=$(aws elbv2 describe-load-balancers --names nginx-alb \
   --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
   --output text)
-echo "ALB public IP: $ALB_IP"
 ```
 
 Then hit the ALB — successive requests should alternate between Server 1 and Server 2:
@@ -163,8 +162,6 @@ TG_ARN=$(aws elbv2 describe-target-groups \
   --query 'TargetGroups[0].TargetGroupArn' \
   --output text)
 
-echo "Target Group ARN: $TG_ARN"
-
 aws elbv2 describe-target-health --target-group-arn $TG_ARN
 ```
 

docs/terraform-workbooks/s3-webapp/README.md

@@ -9,6 +9,9 @@ tags:
   - flask
   - webapp
   - workbook
+  - imds
+  - sts
+  - instance-profile
 resources:
   - title: "Terraform AWS Provider"
     url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
@@ -36,18 +39,21 @@ resources:
 
 Deploy an EC2 instance running a Flask file-sharing web application backed by S3 (Predastore). Users can upload files through a web form and browse uploaded content — demonstrating Terraform managing both compute and object storage together.
 
+This workbook uses the idiomatic AWS credential model: the instance is launched with an **IAM instance profile** and the app pulls short-lived STS credentials from **IMDS** (`169.254.169.254`) through boto3's default credential chain. **No long-lived S3 keys are baked into the instance.** The Terraform run still uses the operator's admin credentials (to create the bucket, role, profile, and instance), but the running instance authenticates to S3 with credentials it fetches at runtime and which expire in ~1 hour.
+
 **Architecture:**
 
 <p align="center">
-  <img src="../../../.github/assets/diagrams/tf-s3-webapp.svg" alt="S3 webapp — browser to Flask EC2 instance to Predastore via S3 API" width="900">
+  <img src="../../../.github/assets/diagrams/tf-s3-webapp.svg" alt="S3 webapp — browser to Flask EC2 instance; the instance fetches short-lived STS credentials from IMDS (169.254.169.254) via its instance profile, then calls Predastore over the S3 API" width="900">
 </p>
 
 **What you'll learn:**
 
 - Configuring the AWS provider with both Spinifex and Predastore endpoints
 - Creating S3 buckets on Predastore via Terraform
-- Deploying a Python webapp with cloud-init that talks to S3
-- Passing credentials and configuration to instances via user-data
+- Defining an IAM role, least-privilege managed policy, and instance profile, and passing the role to an instance (`iam:PassRole`)
+- How the instance obtains short-lived credentials from IMDS via boto3's default credential chain — no static keys on the instance
+- Deploying a Python webapp with cloud-init that signs S3 requests with the STS session token
 
 **Prerequisites:**
 
@@ -55,6 +61,7 @@ Deploy an EC2 instance running a Flask file-sharing web application backed by S3
 - Predastore running (S3 API on port 8443)
 - An Ubuntu 26.04 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
 - OpenTofu or Terraform installed
+- The operator identity running the Terraform apply (`AWS_PROFILE=spinifex`) must be allowed to manage IAM and pass the role: `iam:CreateRole`, `iam:CreatePolicy`, `iam:AttachRolePolicy`, `iam:CreateInstanceProfile`, `iam:AddRoleToInstanceProfile`, and `iam:PassRole` on the new role. The bootstrap admin profile satisfies this.
 - The EC2 instance must be able to reach Predastore — use the host's br-wan IP, not localhost
 
 ## Instructions
@@ -141,16 +148,39 @@ sudo systemctl status s3-webapp
 sudo journalctl -u s3-webapp --no-pager -n 50
 ```
 
-### Upload Fails or Files Not Appearing
+### Upload Fails with `403 AccessDenied`
 
-Check that the S3 credentials in `/opt/webapp/.env` are correct and that the bucket exists:
+The instance reached S3 but was not authorized. The `.env` file deliberately holds **no** keys — credentials come from IMDS:
 
 ```bash
 ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
-cat /opt/webapp/.env
-/opt/webapp/venv/bin/python -c "import boto3; print('boto3 OK')"
+cat /opt/webapp/.env   # S3_ENDPOINT / S3_BUCKET / S3_REGION only — no keys
 ```
 
+Check that:
+
+- the managed policy is attached to the role (`s3:ListBucket` on the bucket, `s3:GetObject`/`s3:PutObject` on its objects), and the policy's bucket name matches `bucket_name`;
+- the assumed-role session's account matches the bucket owner's account — Predastore enforces bucket ownership. Both are created by the same operator identity, so they align by construction; a mismatch surfaces as `403`.
+
+### `NoCredentialsError` / No Credentials
+
+boto3 could not obtain credentials, which means the IMDS credential chain did not resolve. From the instance:
+
+```bash
+ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
+
+# IMDSv2: fetch a token, then the role name behind the instance profile
+TOKEN=$(curl -sX PUT http://169.254.169.254/latest/api/token \
+  -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')
+curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
+  http://169.254.169.254/latest/meta-data/iam/security-credentials/
+
+# Confirm the resolved identity — expect assumed-role/s3-webapp-role/<instance-id>
+aws sts get-caller-identity --endpoint-url https://<predastore_host>
+```
+
+If the role name is empty, the instance profile was not attached — confirm `iam_instance_profile` on the instance and that `iam:PassRole` is allowed for the operator.
+
 ### AMI Not Found
 
 Ensure you have imported an Ubuntu 26.04 image:

docs/terraform-workbooks/s3-webapp/main.tf

@@ -8,13 +8,25 @@
 # Architecture:
 #
 #   Browser ──HTTP──▶ EC2 Instance (Flask webapp, port 80)
-#                         │
-#                         ▼ S3 API (boto3)
+#                         │  ▲ IMDS: short-lived STS creds (ASIA + token)
+#                         │  └──── 169.254.169.254
+#                         ▼ S3 API (boto3, SigV4 with session token)
 #                     Predastore (port 8443)
 #
+# Credentials:
+#   The Terraform run uses the operator's admin keys (s3_access_key /
+#   s3_secret_key) to create the bucket, IAM role, instance profile, and
+#   instance. The INSTANCE receives no long-lived secret — boto3's default
+#   credential chain pulls short-lived STS credentials from IMDS
+#   (169.254.169.254), resolved via the attached instance profile -> role.
+#
 # Prerequisites:
 #   - Spinifex services running (gateway on port 9999)
 #   - Predastore running (S3 API on port 8443)
+#   - The operator identity (AWS_PROFILE=spinifex) must be allowed
+#     iam:CreateRole / iam:CreatePolicy / iam:AttachRolePolicy /
+#     iam:CreateInstanceProfile / iam:AddRoleToInstanceProfile and
+#     iam:PassRole on the new role.
 #   - The EC2 instance must be able to reach the Predastore endpoint.
 #     Set `predastore_host` to the IP reachable from inside the VPC
 #     (e.g. the host's br-wan IP, NOT localhost).
@@ -80,13 +92,13 @@ variable "predastore_host" {
 
 variable "s3_access_key" {
   type        = string
-  description = "S3 access key for Predastore"
+  description = "Operator S3 access key for the Terraform run (creates the bucket on Predastore); the instance uses IMDS, not this key"
 }
 
 variable "s3_secret_key" {
   type        = string
   sensitive   = true
-  description = "S3 secret key for Predastore"
+  description = "Operator S3 secret key for the Terraform run; the instance uses IMDS, not this key"
 }
 
 variable "bucket_name" {
@@ -154,6 +166,55 @@ resource "aws_s3_bucket" "uploads" {
   bucket = var.bucket_name
 }
 
+# ---------------------------------------------------------------------------
+# IAM Instance Role — least-privilege S3 access fetched at runtime via IMDS
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "webapp" {
+  name = "s3-webapp-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Principal = { Service = "ec2.amazonaws.com" }
+      Action    = "sts:AssumeRole"
+    }]
+  })
+}
+
+resource "aws_iam_policy" "webapp_s3" {
+  name = "s3-webapp-policy"
+
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Sid      = "ListBucket"
+        Effect   = "Allow"
+        Action   = ["s3:ListBucket"]
+        Resource = ["arn:aws:s3:::${var.bucket_name}"]
+      },
+      {
+        Sid      = "ObjectRW"
+        Effect   = "Allow"
+        Action   = ["s3:GetObject", "s3:PutObject"]
+        Resource = ["arn:aws:s3:::${var.bucket_name}/*"]
+      }
+    ]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "webapp" {
+  role       = aws_iam_role.webapp.name
+  policy_arn = aws_iam_policy.webapp_s3.arn
+}
+
+resource "aws_iam_instance_profile" "webapp" {
+  name = "s3-webapp-profile"
+  role = aws_iam_role.webapp.name
+}
+
 # ---------------------------------------------------------------------------
 # SSH Key Pair
 # ---------------------------------------------------------------------------
@@ -273,9 +334,14 @@ resource "aws_instance" "webapp" {
   subnet_id              = aws_subnet.public.id
   vpc_security_group_ids = [aws_security_group.webapp.id]
   key_name               = aws_key_pair.webapp.key_name
+  iam_instance_profile   = aws_iam_instance_profile.webapp.name
 
   associate_public_ip_address = true
 
+  # The role's permissions must exist before the instance boots and makes its
+  # first IMDS-credentialed S3 call
+  depends_on = [aws_iam_role_policy_attachment.webapp]
+
   user_data_base64 = base64encode(<<-USERDATA
     #!/bin/bash
     set -euo pipefail
@@ -293,8 +359,6 @@ resource "aws_instance" "webapp" {
     cat > /opt/webapp/.env <<'ENVFILE'
     S3_ENDPOINT=https://${var.predastore_host}
     S3_BUCKET=${var.bucket_name}
-    S3_ACCESS_KEY=${var.s3_access_key}
-    S3_SECRET_KEY=${var.s3_secret_key}
     S3_REGION=${var.region}
     ENVFILE
 
@@ -321,8 +385,6 @@ resource "aws_instance" "webapp" {
     s3 = boto3.client(
         "s3",
         endpoint_url=env["S3_ENDPOINT"],
-        aws_access_key_id=env["S3_ACCESS_KEY"],
-        aws_secret_access_key=env["S3_SECRET_KEY"],
         region_name=env["S3_REGION"],
         verify=False,
         config=Config(s3={"addressing_style": "path"}),

docs/terraform-workbooks/s3-webapp/terraform.tfvars.example

@@ -2,6 +2,11 @@
 #
 # The predastore_host must be reachable from INSIDE the VPC guest — not
 # localhost. Use the host's br-wan or LAN IP, e.g. "192.168.1.10:8443".
+#
+# s3_access_key / s3_secret_key are the operator credentials for the Terraform
+# run only — they create the bucket on Predastore and the IAM role/profile.
+# They are NOT passed to the instance; the instance pulls short-lived STS
+# credentials from IMDS via its instance profile.
 
 predastore_host = "192.168.1.10:8443"
 s3_access_key   = "AKIAIOSFODNN7EXAMPLE"

go.mod

@@ -1,6 +1,6 @@
 module github.com/mulgadc/spinifex
 
-go 1.26.3
+go 1.26.4
 
 require (
 	github.com/aws/aws-sdk-go v1.55.8
@@ -8,13 +8,14 @@ require (
 	github.com/charmbracelet/bubbletea v1.3.10
 	github.com/charmbracelet/lipgloss v1.1.0
 	github.com/go-chi/chi/v5 v5.3.0
+	github.com/golang-jwt/jwt/v5 v5.3.1
 	github.com/google/uuid v1.6.0
 	github.com/insomniacslk/dhcp v0.0.0-20260407060928-11b94ed970f2
 	github.com/kdomanski/iso9660 v0.4.0
 	github.com/klauspost/cpuid/v2 v2.3.0
-	github.com/mulgadc/predastore v1.5.0
-	github.com/mulgadc/viperblock v1.5.0
-	github.com/nats-io/nats-server/v2 v2.14.1
+	github.com/mulgadc/predastore v1.8.0
+	github.com/mulgadc/viperblock v1.8.0
+	github.com/nats-io/nats-server/v2 v2.14.2
 	github.com/nats-io/nats.go v1.52.0
 	github.com/ovn-kubernetes/libovsdb v0.8.1
 	github.com/pelletier/go-toml/v2 v2.3.1
@@ -23,9 +24,10 @@ require (
 	github.com/spf13/viper v1.21.0
 	github.com/stretchr/testify v1.11.1
 	go.uber.org/goleak v1.3.0
-	golang.org/x/crypto v0.52.0
-	golang.org/x/sync v0.20.0
-	gopkg.in/ini.v1 v1.67.2
+	golang.org/x/crypto v0.53.0
+	golang.org/x/sync v0.21.0
+	golang.org/x/sys v0.46.0
+	gopkg.in/ini.v1 v1.67.3
 	gopkg.in/yaml.v3 v3.0.1
 )
 
@@ -36,8 +38,8 @@ require (
 	github.com/antithesishq/antithesis-sdk-go v0.7.0 // indirect
 	github.com/armon/go-metrics v0.4.1 // indirect
 	github.com/atotto/clipboard v0.1.4 // indirect
-	github.com/aws/aws-sdk-go-v2 v1.41.7 // indirect
-	github.com/aws/smithy-go v1.25.1 // indirect
+	github.com/aws/aws-sdk-go-v2 v1.42.0 // indirect
+	github.com/aws/smithy-go v1.27.1 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/boltdb/bolt v1.3.1 // indirect
@@ -55,7 +57,7 @@ require (
 	github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
 	github.com/containerd/console v1.0.5 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/dgraph-io/badger/v4 v4.9.1 // indirect
+	github.com/dgraph-io/badger/v4 v4.9.2 // indirect
 	github.com/dgraph-io/ristretto/v2 v2.4.0 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
@@ -99,8 +101,8 @@ require (
 	github.com/muesli/cancelreader v0.2.2 // indirect
 	github.com/muesli/termenv v0.16.0 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
-	github.com/nats-io/jwt/v2 v2.8.1 // indirect
-	github.com/nats-io/nkeys v0.4.15 // indirect
+	github.com/nats-io/jwt/v2 v2.8.2 // indirect
+	github.com/nats-io/nkeys v0.4.16 // indirect
 	github.com/nats-io/nuid v1.0.1 // indirect
 	github.com/pierrec/lz4/v4 v4.1.14 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
@@ -108,7 +110,7 @@ require (
 	github.com/prometheus/client_model v0.6.2 // indirect
 	github.com/prometheus/common v0.67.5 // indirect
 	github.com/prometheus/procfs v0.20.1 // indirect
-	github.com/quic-go/quic-go v0.59.1 // indirect
+	github.com/quic-go/quic-go v0.60.0 // indirect
 	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/sagikazarmark/locafero v0.12.0 // indirect
 	github.com/spf13/afero v1.15.0 // indirect
@@ -124,14 +126,13 @@ require (
 	go.opentelemetry.io/otel/trace v1.42.0 // indirect
 	go.yaml.in/yaml/v2 v2.4.4 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
-	golang.org/x/mod v0.35.0 // indirect
-	golang.org/x/net v0.55.0 // indirect
-	golang.org/x/sys v0.45.0 // indirect
-	golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa // indirect
-	golang.org/x/term v0.43.0 // indirect
-	golang.org/x/text v0.37.0 // indirect
+	golang.org/x/mod v0.36.0 // indirect
+	golang.org/x/net v0.56.0 // indirect
+	golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6 // indirect
+	golang.org/x/term v0.44.0 // indirect
+	golang.org/x/text v0.38.0 // indirect
 	golang.org/x/time v0.15.0 // indirect
-	golang.org/x/tools v0.44.0 // indirect
+	golang.org/x/tools v0.45.0 // indirect
 	golang.org/x/vuln v1.1.4 // indirect
 	google.golang.org/protobuf v1.36.11 // indirect
 )

go.sum

@@ -31,30 +31,30 @@ github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z
 github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
 github.com/aws/aws-sdk-go v1.55.8 h1:JRmEUbU52aJQZ2AjX4q4Wu7t4uZjOu71uyNmaWlUkJQ=
 github.com/aws/aws-sdk-go v1.55.8/go.mod h1:ZkViS9AqA6otK+JBBNH2++sx1sgxrPKcSzPPvQkUtXk=
-github.com/aws/aws-sdk-go-v2 v1.41.7 h1:DWpAJt66FmnnaRIOT/8ASTucrvuDPZASqhhLey6tLY8=
-github.com/aws/aws-sdk-go-v2 v1.41.7/go.mod h1:4LAfZOPHNVNQEckOACQx60Y8pSRjIkNZQz1w92xpMJc=
-github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 h1:gx1AwW1Iyk9Z9dD9F4akX5gnN3QZwUB20GGKH/I+Rho=
-github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10/go.mod h1:qqY157uZoqm5OXq/amuaBJyC9hgBCBQnsaWnPe905GY=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.16 h1:r3RJBuU7X9ibt8RHbMjWE6y60QbKBiII6wSrXnapxSU=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.16/go.mod h1:6cx7zqDENJDbBIIWX6P8s0h6hqHC8Avbjh9Dseo27ug=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 h1:GpT/TrnBYuE5gan2cZbTtvP+JlHsutdmlV2YfEyNde0=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23/go.mod h1:xYWD6BS9ywC5bS3sz9Xh04whO/hzK2plt2Zkyrp4JuA=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 h1:bpd8vxhlQi2r1hiueOw02f/duEPTMK59Q4QMAoTTtTo=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23/go.mod h1:15DfR2nw+CRHIk0tqNyifu3G1YdAOy68RftkhMDDwYk=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24 h1:OQqn11BtaYv1WLUowvcA30MpzIu8Ti4pcLPIIyoKZrA=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24/go.mod h1:X5ZJyfwVrWA96GzPmUCWFQaEARPR7gCrpq2E92PJwAE=
-github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9 h1:FLudkZLt5ci0ozzgkVo8BJGwvqNaZbTWb3UcucAateA=
-github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9/go.mod h1:w7wZ/s9qK7c8g4al+UyoF1Sp/Z45UwMGcqIzLWVQHWk=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15 h1:ieLCO1JxUWuxTZ1cRd0GAaeX7O6cIxnwk7tc1LsQhC4=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15/go.mod h1:e3IzZvQ3kAWNykvE0Tr0RDZCMFInMvhku3qNpcIQXhM=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23 h1:pbrxO/kuIwgEsOPLkaHu0O+m4fNgLU8B3vxQ+72jTPw=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23/go.mod h1:/CMNUqoj46HpS3MNRDEDIwcgEnrtZlKRaHNaHxIFpNA=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23 h1:03xatSQO4+AM1lTAbnRg5OK528EUg744nW7F73U8DKw=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23/go.mod h1:M8l3mwgx5ToK7wot2sBBce/ojzgnPzZXUV445gTSyE8=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0 h1:etqBTKY581iwLL/H/S2sVgk3C9lAsTJFeXWFDsDcWOU=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0/go.mod h1:L2dcoOgS2VSgbPLvpak2NyUPsO1TBN7M45Z4H7DlRc4=
-github.com/aws/smithy-go v1.25.1 h1:J8ERsGSU7d+aCmdQur5Txg6bVoYelvQJgtZehD12GkI=
-github.com/aws/smithy-go v1.25.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
+github.com/aws/aws-sdk-go-v2 v1.42.0 h1:XvXMJTkFQtpBKIWZnmr9ZEOc2InWM2yldjXEJ/bymhA=
+github.com/aws/aws-sdk-go-v2 v1.42.0/go.mod h1:27+ACypSLljLAEKsCYOmrjKh83vuTRkuAe9Uv/3A4bg=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.13 h1:p1BBrg/Hhp6uK7zpejeI8QFXHJeC/mynzi04Sl03k9g=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.13/go.mod h1:8cIfkE9MDhkRZGpQ22aV6/lkYeYSozpz16Smrs5x4Ls=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.24 h1:2hQqYCV9yqyePQ9o6dCrZc/zO8U3TwPr9mIKlZnPu/I=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.24/go.mod h1:IDwpACtwqHLISdzfwUUNq4P9DsB/h5BLg4FwJPNfqFY=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.29 h1:f3vKqSo13fhTYb+JEcXwXefZQE26I1FB5eTSniU67ko=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.29/go.mod h1:MzoLFUArKGpGD+ukmPiTPG1X5x4o6M2kq4v2dr1FiEc=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.29 h1:RdwIf/CuUsvJX3RgJagbOyotl/cxoLY4xviKuE7p2GY=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.29/go.mod h1:71wt8W2EgswdZy9Mf9KNnzxZ3TiZlv4caKghPktDOkA=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.30 h1:VTGy885W5DKBxWRUJbym9hytNaYzsyaPkCHGRRMAOhU=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.30/go.mod h1:AS0HycUvJRFvTt613AYDOgO2jzw+00cVSMny8XB3yMY=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.12 h1:ZD2+BSw9vFsNlKYIasSNt3uDbjqqXIBcM13UJv/Lx2k=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.12/go.mod h1:Ms4zlcVBbXbiP7EVLhl+lgjvA/a7YphqQ3Ih3174EmI=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.22 h1:V51LGlOq/1VsDsHUdoklAQi7rMmx4qQubvFYAlP2254=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.22/go.mod h1:4Pzhyz8hJOm2bepgl+NjvRx8vlUFAIIvJnZ/MkcNPpU=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.29 h1:DRebniUGZ2MqiiIVmQJ04vIXr918hubdHMnarSLEWyU=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.29/go.mod h1:LfRkPCD8YHDM2E5eTkos2UpwYeZnBcVarTa8L59bJHA=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.29 h1:hiME6pBzC7OTl9LMtlyTWBuEl1f4QBcUmFDKC7MLXtc=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.29/go.mod h1:G7RP+uhagpKtKhd1BM9N6JQqjCcGEU47K5lBVZQyRQw=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.103.3 h1:JRseEu/vIDMaWis4bSw0QbXL+cvIGc1XnX076H5ZXLE=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.103.3/go.mod h1:77ZAgynvx1txMvDG8gGWoWkO1augYDxkp9JElWFgjQU=
+github.com/aws/smithy-go v1.27.1 h1:4T340VFndXtADGF52gYa1POyL7s9E4Z1OeZ1hCscIw8=
+github.com/aws/smithy-go v1.27.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
@@ -103,8 +103,8 @@ github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6N
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/dgraph-io/badger/v4 v4.9.1 h1:DocZXZkg5JJHJPtUErA0ibyHxOVUDVoXLSCV6t8NC8w=
-github.com/dgraph-io/badger/v4 v4.9.1/go.mod h1:5/MEx97uzdPUHR4KtkNt8asfI2T4JiEiQlV7kWUo8c0=
+github.com/dgraph-io/badger/v4 v4.9.2 h1:Wb5qw8gElqwV1a8msHTeQKova9b1V10heFKMIiPd80E=
+github.com/dgraph-io/badger/v4 v4.9.2/go.mod h1:nJjaJTUOSsQEBhsq209FmwCvMJzEA3e74RjZw6V2pQI=
 github.com/dgraph-io/ristretto/v2 v2.4.0 h1:I/w09yLjhdcVD2QV192UJcq8dPBaAJb9pOuMyNy0XlU=
 github.com/dgraph-io/ristretto/v2 v2.4.0/go.mod h1:0KsrXtXvnv0EqnzyowllbVJB8yBonswa2lTCK2gGo9E=
 github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da h1:aIftn67I1fkbMa512G+w+Pxci9hJPB8oMnkcP3iZF38=
@@ -147,6 +147,8 @@ github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/me
 github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro=
 github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
 github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
+github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
+github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
 github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
@@ -286,22 +288,22 @@ github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELU
 github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
 github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
 github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
-github.com/mulgadc/predastore v1.5.0 h1:FtL9EquoGKl3tcIRIoAXGhQStkN4uPDx0TOQZ89Mh+c=
-github.com/mulgadc/predastore v1.5.0/go.mod h1:fXIGz7yCKcHh68ybcw7aigdOdzIfyt19h/0diPF+JCU=
-github.com/mulgadc/viperblock v1.5.0 h1:2OprAI1oAtitUosS5A9LbenU/RCaw0oxUJvUhv8+tgU=
-github.com/mulgadc/viperblock v1.5.0/go.mod h1:lIuOaBO3gC0o/EwWhJPpCuhUEPQ3/y74xckGlcBKaos=
+github.com/mulgadc/predastore v1.8.0 h1:Di/PlAP8nIVjXMmjVPuS9UtqQkqMjR41RrC5J21vHmw=
+github.com/mulgadc/predastore v1.8.0/go.mod h1:TC6I4Z27RCe2gPqlxqUehJ2J+/jKggnH+4DyMq1H/ng=
+github.com/mulgadc/viperblock v1.8.0 h1:kVyM5RxomokVa1uA2kbhtS02kZ9aDXKVj4MvGeqsKiM=
+github.com/mulgadc/viperblock v1.8.0/go.mod h1:18zxN4szsp5h6w8nTqkZWP0SWmIrLU3NdpKXDYNanls=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
-github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
-github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
-github.com/nats-io/nats-server/v2 v2.14.1 h1:wXs/a5fw9Hzm3CvuzLxGeIwpjPulSa7gMT3eSuhGkcg=
-github.com/nats-io/nats-server/v2 v2.14.1/go.mod h1:4N17zLpuS7WMbG8T9gsE2B7z9hC9PraPyulVBfpK6nU=
+github.com/nats-io/jwt/v2 v2.8.2 h1:XXRgB60MSTnqsRwejQurVDs/hcv2dkt+86GjI+I/bMc=
+github.com/nats-io/jwt/v2 v2.8.2/go.mod h1:Ag/56sq9OblL4JgdYufDd16Egb17Kr/8WwwuO/forVc=
+github.com/nats-io/nats-server/v2 v2.14.2 h1:Q7dRhCY03Y00rETFW3KV+KGaCIajlDfWgWUVgbMxyuk=
+github.com/nats-io/nats-server/v2 v2.14.2/go.mod h1:lWpb1bSpRELZfRdlMkdz8E7lbXKKyNe8RIn0vvepIHs=
 github.com/nats-io/nats.go v1.52.0 h1:n3avV4VBsCgsdwh71TppsTwtv+QdPs7ntSKM8qJLGsc=
 github.com/nats-io/nats.go v1.52.0/go.mod h1:26HypzazeOkyO3/mqd1zZd53STJN0EjCYF9Uy2ZOBno=
-github.com/nats-io/nkeys v0.4.15 h1:JACV5jRVO9V856KOapQ7x+EY8Jo3qw1vJt/9Jpwzkk4=
-github.com/nats-io/nkeys v0.4.15/go.mod h1:CpMchTXC9fxA5zrMo4KpySxNjiDVvr8ANOSZdiNfUrs=
+github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
+github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
 github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
 github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
 github.com/ovn-kubernetes/libovsdb v0.8.1 h1:M2J8bcJt5mXCom0HqzfEtuHkT80CTSQRcYG7acT8gf4=
@@ -351,8 +353,10 @@ github.com/pterm/pterm v0.12.36/go.mod h1:NjiL09hFhT/vWjQHSj1athJpx6H8cjpHXNAK5b
 github.com/pterm/pterm v0.12.40/go.mod h1:ffwPLwlbXxP+rxT0GsgDTzS3y3rmpAO1NMjUkGTYf8s=
 github.com/pterm/pterm v0.12.83 h1:ie+YmGmA727VuhxBlyGr74Ks+7McV6kT99IB8EU80aA=
 github.com/pterm/pterm v0.12.83/go.mod h1:xlgc6bFWyJIMtmLJvGim+L7jhSReilOlOnodeIYe4Tk=
-github.com/quic-go/quic-go v0.59.1 h1:0Gmua0HW1Tv7ANR7hUYwRyD0MG5OJfgvYSZasGZzBic=
-github.com/quic-go/quic-go v0.59.1/go.mod h1:upnsH4Ju1YkqpLXC305eW3yDZ4NfnNbmQRCMWS58IKU=
+github.com/quic-go/go-ossfuzz-seeds v0.1.0 h1:APacT+iIaNF6fd8AGEiN3bT/Jtkd2jz4v4TzM7MFjy0=
+github.com/quic-go/go-ossfuzz-seeds v0.1.0/go.mod h1:3IOHRbJIc+L6YKMwfDtJAM9Vj9k0YY4muhuyUYk5tbk=
+github.com/quic-go/quic-go v0.60.0 h1:xcQioE8OM66UQLeUMHltK1CCcOu3JbVB4JAQdDQSB+0=
+github.com/quic-go/quic-go v0.60.0/go.mod h1:wpKpjmPpftl30sL6pFh7REVpjbcCVy4zt2vDyK1TuJk=
 github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
 github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
 github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
@@ -424,14 +428,14 @@ golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnf
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
-golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
-golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
+golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
+golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa h1:FRnLl4eNAQl8hwxVVC17teOw8kdjVDVAiFMtgUdTSRQ=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa/go.mod h1:zk2irFbV9DP96SEBUUAy67IdHUaZuSnrz1n472HUCLE=
 golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
 golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
-golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM=
-golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU=
+golang.org/x/mod v0.36.0 h1:JJjpVx6myfUsUdAzZuOSTTmRE0PfZeNWzzvKrP7amb4=
+golang.org/x/mod v0.36.0/go.mod h1:moc6ELqsWcOw5Ef3xVprK5ul/MvtVvkIXLziUOICjUQ=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -442,8 +446,8 @@ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81R
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
-golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
-golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
+golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
+golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -452,8 +456,8 @@ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
-golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
+golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
+golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -484,33 +488,33 @@ golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
-golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
-golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
-golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa h1:efT73AJZfAAUV7SOip6pWGkwJDzIGiKBZGVzHYa+ve4=
-golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa/go.mod h1:kHjTxDEnAu6/Nl9lDkzjWpR+bmKfxeiRuSDlsMb70gE=
+golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
+golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6 h1:HjU6IWBiAgRIdAJ9/y1rwCn+UELEmwV+VsTLzj/W4sE=
+golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6/go.mod h1:Eqhaxk/wZsWEH8CRxLwj6xzEJbz7k1EFGqx7nyCoabE=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
-golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4=
-golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=
+golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc=
+golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
-golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
-golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
+golang.org/x/text v0.38.0 h1:sXmwo9DwP3OK9EZ7PqAdaooSGozfl/3a6/xJcbzPRhE=
+golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4=
 golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
 golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
 golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
-golang.org/x/tools v0.44.0 h1:UP4ajHPIcuMjT1GqzDWRlalUEoY+uzoZKnhOjbIPD2c=
-golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI=
+golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8=
+golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0=
 golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM=
 golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY=
 golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM=
@@ -534,8 +538,8 @@ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
-gopkg.in/ini.v1 v1.67.2 h1:JtOSMb9OuaCZKr7h5D/h6iii14sK0hLbplTc6frx4Ss=
-gopkg.in/ini.v1 v1.67.2/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
+gopkg.in/ini.v1 v1.67.3 h1:iM9Lhz5MRSGhHVGGwCuzG9KO8PoirCXj/m/qTmOJJQw=
+gopkg.in/ini.v1 v1.67.3/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

internal/eksgw/client.go

@@ -0,0 +1,112 @@
+// Package eksgw is the on-VM SigV4 client used by EKS helpers (eks-gateway-publish,
+// eks-token-webhook) to POST over HTTPS to the AWS gateway instead of NATS.
+// Signs with Predastore credentials for service "eks".
+package eksgw
+
+import (
+	"bytes"
+	"crypto/sha256"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/mulgadc/predastore/auth"
+	"github.com/mulgadc/spinifex/internal/tlsconfig"
+)
+
+// Client posts SigV4-signed requests to the AWS gateway for the "eks" service.
+type Client struct {
+	baseURL    string
+	accessKey  string
+	secretKey  string
+	region     string
+	httpClient *http.Client
+}
+
+// New builds a client. caPath optionally pins the gateway TLS CA; when empty the
+// client relies on the system trust store. region defaults to us-east-1 when
+// empty (SigV4 requires a non-empty region).
+func New(baseURL, caPath, accessKey, secretKey, region string) (*Client, error) {
+	if baseURL == "" {
+		return nil, fmt.Errorf("eksgw: baseURL is required")
+	}
+	if accessKey == "" || secretKey == "" {
+		return nil, fmt.Errorf("eksgw: access key and secret key are required")
+	}
+	if region == "" {
+		region = "us-east-1"
+	}
+
+	tlsCfg := &tls.Config{
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
+	}
+	if caPath != "" {
+		pem, err := os.ReadFile(caPath)
+		if err != nil {
+			return nil, fmt.Errorf("eksgw: read gateway CA %q: %w", caPath, err)
+		}
+		pool := x509.NewCertPool()
+		if !pool.AppendCertsFromPEM(pem) {
+			return nil, fmt.Errorf("eksgw: gateway CA %q has no usable certificates", caPath)
+		}
+		tlsCfg.RootCAs = pool
+	}
+
+	return &Client{
+		baseURL:   strings.TrimRight(baseURL, "/"),
+		accessKey: accessKey,
+		secretKey: secretKey,
+		region:    region,
+		httpClient: &http.Client{
+			Timeout:   10 * time.Second,
+			Transport: &http.Transport{TLSClientConfig: tlsCfg, MaxIdleConns: 2, IdleConnTimeout: 30 * time.Second},
+		},
+	}, nil
+}
+
+// Post SigV4-signs and sends body to path, returning the response body on 2xx.
+// Errors include the gateway status and body. No retry; callers wrap as needed.
+func (c *Client) Post(path string, body []byte) ([]byte, error) {
+	return c.do(http.MethodPost, path, body)
+}
+
+// Get SigV4-signs and sends a single GET to path (e.g.
+// "/clusters/alpha/internal-addons?accountId=..."). Same contract as Post: 2xx
+// body or an error carrying the gateway status. No retry.
+func (c *Client) Get(path string) ([]byte, error) {
+	return c.do(http.MethodGet, path, nil)
+}
+
+func (c *Client) do(method, path string, body []byte) ([]byte, error) {
+	req, err := http.NewRequest(method, c.baseURL+path, bytes.NewReader(body))
+	if err != nil {
+		return nil, fmt.Errorf("create request: %w", err)
+	}
+	if body != nil {
+		req.Header.Set("Content-Type", "application/x-amz-json-1.1")
+	}
+
+	sum := sha256.Sum256(body)
+	if err := auth.SignReq(req, c.accessKey, c.secretKey, hex.EncodeToString(sum[:]), "eks", c.region); err != nil {
+		return nil, fmt.Errorf("sign request: %w", err)
+	}
+
+	resp, err := c.httpClient.Do(req)
+	if err != nil {
+		return nil, fmt.Errorf("send request: %w", err)
+	}
+	defer resp.Body.Close()
+
+	respBody, _ := io.ReadAll(resp.Body)
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		return nil, fmt.Errorf("gateway returned %d: %s", resp.StatusCode, strings.TrimSpace(string(respBody)))
+	}
+	return respBody, nil
+}

internal/fipsboot/fipsboot.go

@@ -1,12 +1,6 @@
-// Package fipsboot enforces FIPS 140-3 mode at process startup.
-//
-// Blank-import this package from every binary's main:
-//
-//	import _ "github.com/mulgadc/spinifex/internal/fipsboot"
-//
-// Builds with GOFIPS140=v1.0.0 link the Go Cryptographic Module v1.0.0
-// (NIST CMVP cert #5247). At runtime FIPS mode can still be turned off
-// via GODEBUG=fips140=off; init() panics in that case to close the hole.
+// Package fipsboot enforces FIPS 140-3 at startup via blank import in every binary.
+// Requires GOFIPS140=v1.0.0 at build time; panics if GODEBUG=fips140=off disables
+// the runtime FIPS mode.
 package fipsboot
 
 import (

internal/tlsconfig/curves.go

@@ -1,34 +1,13 @@
 // Package tlsconfig pins the cluster-wide TLS 1.3 curve preference list.
-//
-// The list is hardcoded because mixed classification on a single cluster is
-// not supported and the certified FIPS build is itself the per-deployment
-// choice. A typo in a config string would silently downgrade where a missing
-// constant cannot.
+// Hardcoded to prevent silent downgrade via misconfiguration; the FIPS build
+// is the per-deployment choice.
 package tlsconfig
 
 import "crypto/tls"
 
-// Curves is the fixed TLS 1.3 curve allowlist:
-//
-//   - X25519MLKEM768     — Cat 3 ML-KEM hybrid, wins under Go 1.26 stdlib filter
-//   - SecP384r1MLKEM1024 — Cat 5 ML-KEM hybrid, fallback for peers that prefer it
-//   - X25519             — classical TLS 1.3 fallback for SDKs without ML-KEM (aws CLI v2 today)
-//   - CurveP384          — classical TLS 1.3 fallback (Java/.NET enterprise SDKs)
-//
-// PQ-capable peers (browsers since 2024, Go aws-sdk v2 with Go 1.24+) always
-// negotiate the hybrid first; classical fallback only triggers for peers that
-// cannot do ML-KEM. Those connections still get TLS 1.3 + AES-256-GCM but
-// remain HNDL-exposed — accepted trade-off until AWS CLI v2 / bundled
-// OpenSSL ships ML-KEM (currently lagging upstream OpenSSL 3.5).
-//
-// Weak/niche entries from the Go stdlib default (SecP256r1MLKEM768, P-256,
-// P-521) are intentionally excluded so they cannot be advertised.
-//
-// Go 1.26 stdlib treats tls.Config.CurvePreferences as an allowlist filter
-// applied to the stdlib default order, not as a user-supplied order.
-//
-// Callers must not mutate this slice. Assign it directly to
-// tls.Config.CurvePreferences; do not append to it.
+// Curves is the fixed TLS 1.3 curve allowlist: two ML-KEM hybrids (PQ-capable
+// peers always negotiate these) followed by two classical fallbacks for SDKs
+// without ML-KEM. Weak Go defaults (SecP256r1MLKEM768, P-256, P-521) excluded.
 var Curves = []tls.CurveID{
 	tls.X25519MLKEM768,
 	tls.SecP384r1MLKEM1024,

internal/tlsconfig/curves_test.go

@@ -46,11 +46,9 @@ func TestCurves_ExcludesWeakPrimitives(t *testing.T) {
 	}
 }
 
-// TestIntegration_NegotiatesPQHybrid spins up an httptest TLS server
-// configured with Curves + TLS 1.3 minimum, dials it, and asserts a PQ
-// hybrid is negotiated at TLS 1.3. Per Go 1.26 stdlib filter behavior the
-// winner is X25519MLKEM768; the assertion checks for any approved hybrid
-// to remain stable across stdlib reorderings of our 4 entries.
+// TestIntegration_NegotiatesPQHybrid verifies that a Curves-configured server
+// negotiates a PQ hybrid at TLS 1.3; checks for any approved hybrid to stay
+// stable across Go stdlib reorderings.
 func TestIntegration_NegotiatesPQHybrid(t *testing.T) {
 	cert := selfSignedCert(t)
 	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
@@ -95,10 +93,8 @@ func TestIntegration_NegotiatesPQHybrid(t *testing.T) {
 	}
 }
 
-// TestIntegration_TLS12ClientRejected confirms MinVersion: tls.VersionTLS13
-// is enforced — a TLS 1.2-pinned client gets a handshake error rather than
-// silently downgrading. This is the negative test that closes the HNDL gap
-// on external surfaces.
+// TestIntegration_TLS12ClientRejected confirms MinVersion TLS 1.3 is enforced:
+// a TLS 1.2-pinned client must get a handshake error, not a silent downgrade.
 func TestIntegration_TLS12ClientRejected(t *testing.T) {
 	cert := selfSignedCert(t)
 	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {

scripts/ansible/README.md

@@ -1,96 +0,0 @@
-# Ansible Dev Lifecycle (parallel to bash scripts)
-
-Experimental Ansible-based lifecycle for a single-node Spinifex dev host.
-Lives alongside the existing `scripts/dev-*.sh` and `scripts/reset-dev-env.sh`;
-neither set is removed. Decide which to keep after parity testing.
-
-Plan: `docs/development/improvements/ansible-dev-lifecycle.md` (mulga top-level).
-Bead: `mulga-siv-9`.
-
-## Prerequisites
-
-Venv lives at sub-repo root (`spinifex/.venv/`) — one venv per sub-repo,
-shared by any future python tooling in spinifex. Bootstrap from `spinifex/`:
-
-```
-sudo apt install python3.13-venv
-python3 -m venv .venv
-source .venv/bin/activate
-pip install 'ansible>=9'
-ansible-galaxy collection install -r scripts/ansible/requirements.yml
-```
-
-Then `cd scripts/ansible` to run playbooks — `ansible.cfg` resolves the
-inventory and roles relative to CWD.
-
-## Playbooks
-
-| Playbook | Purpose | Bash parity |
-|---|---|---|
-| `playbooks/dev-preflight.yml` | Dependency + port check | `dev-setup.sh` §checks |
-| `playbooks/dev-teardown.yml` | Wipe state only (no reinstall) | `reset-dev-env.sh` §teardown |
-| `playbooks/dev-install.yml` | Build + install + init + smoketest on clean box | `dev-install.sh` |
-| `playbooks/dev-reset.yml` | Capture settings → teardown → build → install → init → smoketest | `reset-dev-env.sh` (full) |
-| `playbooks/dev-deploy.yml` | Rebuild + swap binaries/microvm artifacts + restart (no setup.sh, no smoketest) | `make deploy` |
-
-Upcoming (not yet implemented):
-
-- `dev-mode-start.yml` / `dev-mode-stop.yml` — parity with `start-dev.sh` / `stop-dev.sh`
-  (lowest priority — deprecated bash scripts, CI-only callers).
-
-## Invocation
-
-```
-ansible-playbook playbooks/dev-preflight.yml
-ansible-playbook playbooks/dev-teardown.yml
-ansible-playbook playbooks/dev-install.yml
-ansible-playbook playbooks/dev-reset.yml
-ansible-playbook playbooks/dev-deploy.yml
-```
-
-Or via `make` (from `spinifex/`):
-
-```
-make ansible-dev-preflight
-make ansible-dev-teardown
-make ansible-dev-install
-make ansible-dev-reset
-make ansible-dev-deploy
-```
-
-### When to use which
-
-- First-time / clean box → `ansible-dev-install`
-- Iterate on Go code, microvm initramfs, lb-agent → `ansible-dev-deploy` (fast)
-- Changed systemd units, helper scripts, logrotate, setup.sh → `ansible-dev-reset` (slow, full rebuild)
-- Need a clean slate without reinstall → `ansible-dev-teardown`
-
-## Variable overrides
-
-Defaults in `vars/defaults.yml`. Override per run with `-e`:
-
-```
-ansible-playbook playbooks/dev-teardown.yml -e spinifex_wipe_ssh_keys=true
-```
-
-Useful overrides:
-
-| Var | Default | What it does |
-|---|---|---|
-| `spinifex_wipe_legacy_home` | `true` | Remove `$HOME/spinifex` (legacy dev-mode path) |
-| `spinifex_wipe_ssh_keys` | `false` | Remove `~/.ssh/spinifex-key*` |
-| `spinifex_wipe_aws_creds` | `false` | Remove spinifex profile from `~/.aws/credentials` |
-| `teardown_process_wait_seconds` | `30` | Grace period for processes to exit |
-| `spinifex_region` | `ap-southeast-2` | Region passed to `spx admin init` |
-| `spinifex_az` | `{{ region }}a` | AZ passed to `spx admin init` |
-| `spinifex_external_mode` | `pool` | `pool` \| `nat` external networking |
-
-`dev-reset.yml` automatically captures `region`, `az`, `external_mode`,
-pool range, gateway, prefix-len, nat gateway_ip from the existing
-`/etc/spinifex/spinifex.toml` before teardown and replays them into
-`init`. Use `-e` to override.
-
-## Leak catalog
-
-`roles/teardown/README.md` tracks every known state source. When teardown
-misses something, add it there and in `roles/teardown/tasks/main.yml`.

scripts/ansible/ansible.cfg

@@ -1,11 +0,0 @@
-[defaults]
-inventory = inventory/localhost.yml
-roles_path = roles
-stdout_callback = ansible.builtin.default
-host_key_checking = False
-retry_files_enabled = False
-interpreter_python = auto_silent
-forks = 1
-
-[callback_default]
-result_format = yaml

scripts/ansible/inventory/localhost.yml

@@ -1,6 +0,0 @@
----
-all:
-  hosts:
-    localhost:
-      ansible_connection: local
-      ansible_python_interpreter: "{{ ansible_playbook_python }}"

scripts/ansible/playbooks/dev-deploy.yml

@@ -1,14 +0,0 @@
----
-# In-place artifact swap: rebuilds + replaces binaries/microvm artifacts on
-# an already-installed node, then restarts spinifex.target. Use when iterating
-# on code without altering systemd units, users, dirs, or running the full
-# smoketest. For first-time or unit/script changes use dev-install / dev-reset.
-- name: Dev deploy — build + swap binaries + restart (no setup.sh, no smoketest)
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-
-  roles:
-    - build
-    - deploy

scripts/ansible/playbooks/dev-install.yml

@@ -1,67 +0,0 @@
----
-- name: Dev install — build, install, init, smoketest (idempotent)
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-
-  pre_tasks:
-    - name: preflight
-      ansible.builtin.include_role:
-        name: preflight
-
-    - name: health — config file present?
-      ansible.builtin.stat:
-        path: "{{ spinifex_config_file }}"
-      register: health_config
-
-    - name: health — spinifex.target active?
-      become: true
-      ansible.builtin.command: systemctl is-active spinifex.target
-      register: health_target
-      changed_when: false
-      failed_when: false
-
-    - name: health — awsgw responds?
-      become: true
-      become_user: "{{ spinifex_invoking_user }}"
-      environment:
-        AWS_PROFILE: spinifex
-      ansible.builtin.command: aws ec2 describe-regions
-      register: health_gw
-      changed_when: false
-      failed_when: false
-
-    - name: health — spx binary installed?
-      ansible.builtin.stat:
-        path: /usr/local/bin/spx
-      register: health_spx
-
-    - name: health — set already_installed fact
-      ansible.builtin.set_fact:
-        spinifex_already_installed: >-
-          {{ health_config.stat.exists
-             and health_spx.stat.exists
-             and (health_target.stdout | trim) == 'active'
-             and health_gw.rc == 0 }}
-
-    - name: health — summary
-      ansible.builtin.debug:
-        msg: |
-          Health:
-            spx binary: {{ 'present' if health_spx.stat.exists else 'missing' }}
-            config:     {{ 'present' if health_config.stat.exists else 'missing' }}
-            target:     {{ health_target.stdout | trim }}
-            awsgw:      {{ 'ok' if health_gw.rc == 0 else 'not responding' }}
-          Already installed: {{ spinifex_already_installed }}
-          {% if spinifex_already_installed %}
-          → Skipping build/install/init. Smoketest only.
-          {% else %}
-          → Running full build → install → init → smoketest.
-          {% endif %}
-
-  roles:
-    - { role: build, when: not (spinifex_already_installed | default(false)) }
-    - { role: install, when: not (spinifex_already_installed | default(false)) }
-    - { role: init, when: not (spinifex_already_installed | default(false)) }
-    - smoketest

scripts/ansible/playbooks/dev-preflight.yml

@@ -1,8 +0,0 @@
----
-- name: Dev preflight — validate host dependencies
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-  roles:
-    - preflight

scripts/ansible/playbooks/dev-reset.yml

@@ -1,57 +0,0 @@
----
-- name: Dev reset — capture settings, wipe, rebuild, reinstall, re-init, smoketest
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-
-  pre_tasks:
-    - name: capture — check existing spinifex.toml
-      ansible.builtin.stat:
-        path: "{{ spinifex_config_file }}"
-      register: reset_config_stat
-
-    - name: capture — slurp existing config
-      ansible.builtin.slurp:
-        src: "{{ spinifex_config_file }}"
-      register: reset_config_raw
-      when: reset_config_stat.stat.exists
-
-    - name: capture — decode config
-      ansible.builtin.set_fact:
-        reset_config_txt: "{{ reset_config_raw.content | b64decode }}"
-      when: reset_config_stat.stat.exists
-
-    # List-concat-and-take-first pattern: missing regex match falls back to current default.
-    - name: capture — extract scalars
-      ansible.builtin.set_fact:
-        spinifex_region: "{{ (reset_config_txt | regex_findall('region\\s*=\\s*\"([^\"]+)\"') + [spinifex_region])[0] }}"
-        spinifex_az: "{{ (reset_config_txt | regex_findall('\\baz\\s*=\\s*\"([^\"]+)\"') + [spinifex_az])[0] }}"
-        spinifex_external_mode: "{{ (reset_config_txt | regex_findall('external_mode\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_mode])[0] }}"
-        spinifex_external_pool_start: "{{ (reset_config_txt | regex_findall('range_start\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_pool_start])[0] }}"
-        spinifex_external_pool_end: "{{ (reset_config_txt | regex_findall('range_end\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_pool_end])[0] }}"
-        spinifex_external_gateway: "{{ (reset_config_txt | regex_findall('gateway\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_gateway])[0] }}"
-        spinifex_external_prefix_len: "{{ (reset_config_txt | regex_findall('prefix_len\\s*=\\s*([0-9]+)') + [spinifex_external_prefix_len | string])[0] }}"
-        spinifex_nat_gateway_ip: "{{ (reset_config_txt | regex_findall('gateway_ip\\s*=\\s*\"([^\"]+)\"') + [spinifex_nat_gateway_ip])[0] }}"
-      when: reset_config_stat.stat.exists
-
-    - name: capture — summary
-      ansible.builtin.debug:
-        msg: |
-          {% if reset_config_stat.stat.exists %}
-          Captured settings from {{ spinifex_config_file }}:
-            region={{ spinifex_region }}, az={{ spinifex_az }}
-            external_mode={{ spinifex_external_mode }}
-            pool={{ spinifex_external_pool_start }}-{{ spinifex_external_pool_end }}
-            gateway={{ spinifex_external_gateway }}, prefix={{ spinifex_external_prefix_len }}
-            nat_gateway={{ spinifex_nat_gateway_ip }}
-          {% else %}
-          No existing config; using defaults from vars/defaults.yml (or -e overrides).
-          {% endif %}
-
-  roles:
-    - teardown
-    - build
-    - install
-    - init
-    - smoketest

scripts/ansible/playbooks/dev-teardown.yml

@@ -1,8 +0,0 @@
----
-- name: Dev teardown — wipe Spinifex state (no reinstall)
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-  roles:
-    - teardown

scripts/ansible/requirements.yml

@@ -1,6 +0,0 @@
----
-collections:
-  - name: community.general
-    version: ">=8.0.0"
-  - name: ansible.posix
-    version: ">=1.5.0"

scripts/ansible/roles/build/meta/main.yml

@@ -1,7 +0,0 @@
----
-galaxy_info:
-  role_name: build
-  description: Build spinifex + viperblock-nbdkit, stage tarball for setup.sh consumption.
-  license: BSD-3-Clause
-  min_ansible_version: "2.14"
-dependencies: []

scripts/ansible/roles/build/tasks/main.yml

@@ -1,66 +0,0 @@
----
-- name: build — spinifex (make build)
-  ansible.builtin.command:
-    cmd: make build
-    chdir: "{{ spinifex_root }}"
-  changed_when: true
-
-- name: build — viperblock nbdkit plugin (make go_build_nbd)
-  ansible.builtin.command:
-    cmd: make go_build_nbd
-    chdir: "{{ mulga_root }}/viperblock"
-  changed_when: true
-
-- name: build — detect microVM build script (present only on direct-boot branches)
-  ansible.builtin.stat:
-    path: "{{ spinifex_root }}/scripts/build-microvm-image.sh"
-  register: build_microvm_script
-
-- name: build — microVM kernel + initramfs (direct-boot LBs)
-  ansible.builtin.command:
-    cmd: make build-microvm-image
-    chdir: "{{ spinifex_root }}"
-  changed_when: true
-  when: build_microvm_script.stat.exists
-
-- name: build — verify build artifacts present
-  ansible.builtin.stat:
-    path: "{{ item }}"
-  register: build_artifacts
-  loop:
-    - "{{ spinifex_root }}/bin/spx"
-    - "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
-
-- name: build — assert artifacts exist
-  ansible.builtin.assert:
-    that: item.stat.exists
-    fail_msg: "Missing build artifact: {{ item.item }}"
-  loop: "{{ build_artifacts.results }}"
-  loop_control:
-    label: "{{ item.item }}"
-
-- name: build — stage tarball contents (mirrors dev-install.sh)
-  ansible.builtin.shell: |
-    set -e
-    STAGING=$(mktemp -d)
-    trap 'rm -rf "$STAGING"' EXIT
-    cp "{{ spinifex_root }}/bin/spx" "$STAGING/spx"
-    cp "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so" "$STAGING/"
-    cp "{{ spinifex_root }}/scripts/setup-ovn.sh" "$STAGING/"
-    mkdir -p "$STAGING/systemd"
-    cp "{{ spinifex_root }}/build/systemd/"* "$STAGING/systemd/"
-    mkdir -p "$STAGING/scripts"
-    cp "{{ spinifex_root }}/build/scripts/"* "$STAGING/scripts/"
-    cp "{{ spinifex_root }}/build/logrotate/spinifex" "$STAGING/logrotate-spinifex"
-    # microVM artifacts ship only when the branch built them.
-    if [ -f "{{ spinifex_root }}/build/microvm/vmlinuz" ] && [ -f "{{ spinifex_root }}/build/microvm/initramfs.cpio.gz" ]; then
-        mkdir -p "$STAGING/microvm"
-        cp "{{ spinifex_root }}/build/microvm/vmlinuz" "$STAGING/microvm/"
-        cp "{{ spinifex_root }}/build/microvm/initramfs.cpio.gz" "$STAGING/microvm/"
-    fi
-    tar czf "{{ build_tarball_path }}" -C "$STAGING" .
-  changed_when: true
-
-- name: build — summary
-  ansible.builtin.debug:
-    msg: "Built spx + nbdkit plugin{{ ' + microvm artifacts' if build_microvm_script.stat.exists else '' }}, tarball at {{ build_tarball_path }}"

scripts/ansible/roles/deploy/tasks/main.yml

@@ -1,148 +0,0 @@
----
-# In-place artifact swap for an already-installed spinifex node. Mirrors
-# `make deploy`: replace binaries + microvm artifacts, restart services.
-# Does NOT run setup.sh (no user/dir/systemd-unit changes) or smoketest.
-
-- name: deploy — fail fast if spinifex isn't already installed
-  ansible.builtin.stat:
-    path: "{{ spinifex_config_file }}"
-  register: deploy_installed_stat
-  failed_when: not deploy_installed_stat.stat.exists
-
-- name: deploy — assert config present
-  ansible.builtin.assert:
-    that: deploy_installed_stat.stat.exists
-    fail_msg: |
-      {{ spinifex_config_file }} missing — this playbook only updates an
-      existing install. Run ansible-dev-install or ansible-dev-reset first.
-
-- name: deploy — verify build artifacts present
-  ansible.builtin.stat:
-    path: "{{ item }}"
-  register: deploy_artifacts
-  loop:
-    - "{{ spinifex_root }}/bin/spx"
-    - "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
-
-- name: deploy — assert artifacts exist after build role
-  ansible.builtin.assert:
-    that: item.stat.exists
-    fail_msg: "Build artifact missing: {{ item.item }} — build role should have produced it"
-  loop: "{{ deploy_artifacts.results }}"
-  loop_control:
-    label: "{{ item.item }}"
-
-- name: deploy — detect microvm artifacts (direct-boot branches only)
-  ansible.builtin.stat:
-    path: "{{ spinifex_root }}/build/microvm/{{ item }}"
-  register: deploy_microvm_stat
-  loop:
-    - vmlinuz
-    - initramfs.cpio.gz
-
-- name: deploy — set microvm presence fact
-  ansible.builtin.set_fact:
-    deploy_microvm_present: "{{ deploy_microvm_stat.results | map(attribute='stat.exists') | min }}"
-
-- name: deploy — detect nbdkit plugin directory
-  ansible.builtin.shell: |
-    set -e
-    nbdkit --dump-config | awk -F= '/^plugindir=/{print $2; exit}'
-  register: deploy_plugindir_cmd
-  changed_when: false
-
-- name: deploy — set plugin path fact
-  ansible.builtin.set_fact:
-    deploy_plugindir: "{{ deploy_plugindir_cmd.stdout | trim }}"
-
-- name: deploy — assert plugindir resolved
-  ansible.builtin.assert:
-    that: deploy_plugindir | length > 0
-    fail_msg: "Could not detect nbdkit plugin directory via `nbdkit --dump-config`"
-
-- name: deploy — stop spinifex.target before binary swap
-  become: true
-  ansible.builtin.systemd:
-    name: spinifex.target
-    state: stopped
-  failed_when: false
-
-- name: deploy — install spx binary
-  become: true
-  ansible.builtin.copy:
-    src: "{{ spinifex_root }}/bin/spx"
-    dest: /usr/local/bin/spx
-    mode: "0755"
-    owner: root
-    group: root
-
-- name: deploy — install viperblock nbdkit plugin
-  become: true
-  ansible.builtin.copy:
-    src: "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
-    dest: "{{ deploy_plugindir }}/nbdkit-viperblock-plugin.so"
-    mode: "0755"
-    owner: root
-    group: root
-
-- name: deploy — ensure microvm dir
-  become: true
-  ansible.builtin.file:
-    path: /usr/share/spinifex/microvm
-    state: directory
-    mode: "0755"
-    owner: root
-    group: root
-  when: deploy_microvm_present
-
-- name: deploy — install microvm kernel
-  become: true
-  ansible.builtin.copy:
-    src: "{{ spinifex_root }}/build/microvm/vmlinuz"
-    dest: /usr/share/spinifex/microvm/vmlinuz
-    mode: "0644"
-    owner: root
-    group: root
-  when: deploy_microvm_present
-
-- name: deploy — install microvm initramfs
-  become: true
-  ansible.builtin.copy:
-    src: "{{ spinifex_root }}/build/microvm/initramfs.cpio.gz"
-    dest: /usr/share/spinifex/microvm/initramfs.cpio.gz
-    mode: "0644"
-    owner: root
-    group: root
-  when: deploy_microvm_present
-
-- name: deploy — daemon-reload (covers any unit changes shipped via setup.sh later)
-  become: true
-  ansible.builtin.systemd:
-    daemon_reload: true
-
-- name: deploy — start spinifex.target
-  become: true
-  ansible.builtin.systemd:
-    name: spinifex.target
-    state: started
-
-- name: deploy — wait for awsgw to respond
-  become: true
-  become_user: "{{ spinifex_invoking_user }}"
-  environment:
-    AWS_PROFILE: spinifex
-  ansible.builtin.command: aws ec2 describe-regions
-  register: deploy_awsgw
-  retries: 15
-  delay: 2
-  until: deploy_awsgw.rc == 0
-  changed_when: false
-
-- name: deploy — summary
-  ansible.builtin.debug:
-    msg: |
-      Deployed:
-        spx           → /usr/local/bin/spx
-        nbdkit plugin → {{ deploy_plugindir }}/nbdkit-viperblock-plugin.so
-        microvm       → {{ '/usr/share/spinifex/microvm/{vmlinuz,initramfs.cpio.gz}' if deploy_microvm_present else 'skipped (not built on this branch)' }}
-      spinifex.target restarted; awsgw responding.

scripts/ansible/roles/init/meta/main.yml

@@ -1,7 +0,0 @@
----
-galaxy_info:
-  role_name: init
-  description: Run spx admin init, install CA trust, start services, import LB image.
-  license: BSD-3-Clause
-  min_ansible_version: "2.14"
-dependencies: []

scripts/ansible/roles/init/tasks/main.yml

@@ -1,84 +0,0 @@
----
-- name: init — base admin-init flags
-  ansible.builtin.set_fact:
-    init_base_flags: >-
-      --force
-      --region {{ spinifex_region }}
-      --az {{ spinifex_az }}
-      --node {{ spinifex_node_name }}
-      --nodes {{ spinifex_node_count }}
-
-- name: init — pool-mode external flags
-  ansible.builtin.set_fact:
-    init_external_flags: >-
-      --external-mode=pool
-      --external-pool={{ spinifex_external_pool_start }}-{{ spinifex_external_pool_end }}
-      --external-gateway={{ spinifex_external_gateway }}
-      --external-prefix-len={{ spinifex_external_prefix_len }}
-  when:
-    - spinifex_external_mode == 'pool'
-    - spinifex_external_pool_start | length > 0
-    - spinifex_external_pool_end | length > 0
-
-- name: init — NAT-mode external flags
-  ansible.builtin.set_fact:
-    init_external_flags: >-
-      --external-mode=nat
-      --gateway-ip={{ spinifex_nat_gateway_ip }}
-      --external-gateway={{ spinifex_external_gateway }}
-      --external-prefix-len={{ spinifex_external_prefix_len }}
-  when:
-    - spinifex_external_mode == 'nat'
-    - spinifex_nat_gateway_ip | length > 0
-
-- name: init — assemble full flag string
-  ansible.builtin.set_fact:
-    init_all_flags: "{{ init_base_flags }} {{ init_external_flags | default('') }}"
-
-- name: init — run spx admin init
-  become: true
-  ansible.builtin.command: "spx admin init {{ init_all_flags }}"
-  changed_when: true
-
-- name: init — copy CA to system trust
-  become: true
-  ansible.builtin.copy:
-    src: "{{ spinifex_ca_src }}"
-    dest: "{{ spinifex_ca_trust_path }}"
-    mode: "0644"
-    remote_src: true
-
-- name: init — rebuild CA bundle (not --fresh — preserve other CAs)
-  become: true
-  ansible.builtin.command: update-ca-certificates
-  changed_when: true
-
-- name: init — start spinifex.target
-  become: true
-  ansible.builtin.systemd:
-    name: spinifex.target
-    state: started
-
-# systemctl start returns as soon as units activate; awsgw still wires up
-# to NATS + daemons after that. Bash start-dev.sh has internal wait loops;
-# systemd path does not. Poll a benign call until gateway is fully ready.
-- name: init — wait for AWS gateway readiness
-  become: true
-  become_user: "{{ spinifex_invoking_user }}"
-  environment:
-    AWS_PROFILE: spinifex
-  ansible.builtin.command: aws ec2 describe-regions
-  register: init_gw_ready
-  until: init_gw_ready.rc == 0
-  retries: 30
-  delay: 2
-  changed_when: false
-
-- name: init — summary
-  ansible.builtin.debug:
-    msg: |
-      Init complete:
-        region={{ spinifex_region }}, az={{ spinifex_az }}, node={{ spinifex_node_name }}
-        external_mode={{ spinifex_external_mode }}
-        spinifex.target: started
-        CA: {{ spinifex_ca_trust_path }}

scripts/ansible/roles/install/meta/main.yml

@@ -1,7 +0,0 @@
----
-galaxy_info:
-  role_name: install
-  description: Invoke setup.sh and setup-ovn.sh to install Spinifex into production paths.
-  license: BSD-3-Clause
-  min_ansible_version: "2.14"
-dependencies: []

scripts/ansible/roles/install/tasks/main.yml

@@ -1,65 +0,0 @@
----
-- name: install — assert tarball exists
-  ansible.builtin.stat:
-    path: "{{ build_tarball_path }}"
-  register: install_tarball
-
-- name: install — fail if tarball missing
-  ansible.builtin.assert:
-    that: install_tarball.stat.exists
-    fail_msg: "Tarball {{ build_tarball_path }} missing. Run the build role first."
-
-- name: install — stop spinifex.target before reinstall
-  become: true
-  ansible.builtin.systemd:
-    name: spinifex.target
-    state: stopped
-  failed_when: false
-
-- name: install — reset failed state (spinifex-* scope)
-  become: true
-  ansible.builtin.command: systemctl reset-failed 'spinifex-*'
-  changed_when: false
-  failed_when: false
-
-- name: install — scrub stale pid/log files from prior dev-mode installs
-  become: true
-  ansible.builtin.shell: |
-    for dir in {{ spinifex_data_dir }} {{ spinifex_log_dir }} {{ spinifex_etc_dir }}; do
-      if [ -d "$dir" ]; then
-        find "$dir" -name '*.pid' -delete 2>/dev/null || true
-        find "$dir" -name '*.log' -delete 2>/dev/null || true
-        find "$dir" -name '*.log.*' -delete 2>/dev/null || true
-      fi
-    done
-  changed_when: false
-
-- name: install — remove legacy /var/lib/spinifex/config
-  become: true
-  ansible.builtin.file:
-    path: "{{ spinifex_data_dir }}/config"
-    state: absent
-
-- name: install — run setup.sh (creates users, dirs, systemd units)
-  become: true
-  ansible.builtin.command:
-    cmd: "bash {{ spinifex_root }}/scripts/setup.sh"
-  environment:
-    INSTALL_SPINIFEX_TARBALL: "{{ build_tarball_path }}"
-    INSTALL_SPINIFEX_SKIP_NEWGRP: "1"
-  changed_when: true
-
-- name: install — cleanup tarball
-  ansible.builtin.file:
-    path: "{{ build_tarball_path }}"
-    state: absent
-
-- name: install — run setup-ovn.sh --management
-  become: true
-  ansible.builtin.command:
-    cmd: /usr/local/share/spinifex/setup-ovn.sh --management
-  changed_when: true
-
-- name: install — summary
-  ansible.builtin.debug:
-    msg: "Install complete — setup.sh + setup-ovn.sh ran. Services NOT yet started (init role handles that)."

scripts/ansible/roles/preflight/meta/main.yml

@@ -1,7 +0,0 @@
----
-galaxy_info:
-  role_name: preflight
-  description: Validate dev-host dependencies for Spinifex (parity with dev-setup.sh).
-  license: BSD-3-Clause
-  min_ansible_version: "2.14"
-dependencies: []

scripts/ansible/roles/preflight/tasks/main.yml

@@ -1,98 +0,0 @@
----
-- name: Check required commands
-  ansible.builtin.command: "which {{ item }}"
-  register: preflight_required
-  loop: "{{ spinifex_required_commands }}"
-  changed_when: false
-  failed_when: preflight_required.rc != 0
-  loop_control:
-    label: "{{ item }}"
-
-- name: Check optional commands
-  ansible.builtin.command: "which {{ item }}"
-  register: preflight_optional
-  loop: "{{ spinifex_optional_commands }}"
-  changed_when: false
-  failed_when: false
-  loop_control:
-    label: "{{ item }}"
-
-- name: Warn on missing optional commands
-  ansible.builtin.debug:
-    msg: "WARN: optional command '{{ item.item }}' not found"
-  loop: "{{ preflight_optional.results }}"
-  when: item.rc != 0
-  loop_control:
-    label: "{{ item.item }}"
-
-- name: Check OVN/OVS commands
-  ansible.builtin.command: "which {{ item }}"
-  register: preflight_ovn
-  loop: "{{ spinifex_ovn_commands }}"
-  changed_when: false
-  failed_when: false
-  loop_control:
-    label: "{{ item }}"
-
-- name: Warn on missing OVN/OVS tools
-  ansible.builtin.debug:
-    msg: "WARN: OVN/OVS tool '{{ item.item }}' not found (required for VPC networking). Install: sudo apt-get install ovn-central ovn-host openvswitch-switch"
-  loop: "{{ preflight_ovn.results }}"
-  when: item.rc != 0
-  loop_control:
-    label: "{{ item.item }}"
-
-- name: Record OVN tool availability
-  ansible.builtin.set_fact:
-    preflight_ovs_present: "{{ (preflight_ovn.results | selectattr('item','eq','ovs-vsctl') | selectattr('rc','eq',0) | list | length) > 0 }}"
-
-- name: Check br-int bridge
-  ansible.builtin.command: ovs-vsctl br-exists br-int
-  register: preflight_brint
-  changed_when: false
-  failed_when: false
-  when: preflight_ovs_present
-
-- name: Warn if br-int missing
-  ansible.builtin.debug:
-    msg: "WARN: br-int missing. Run scripts/setup-ovn.sh --management to initialise OVN networking."
-  when:
-    - preflight_ovs_present
-    - preflight_brint.rc | default(0) != 0
-
-- name: Check sibling component directories
-  ansible.builtin.stat:
-    path: "{{ playbook_dir }}/../../../../{{ item }}/go.mod"
-  register: preflight_components
-  loop: "{{ spinifex_components }}"
-  loop_control:
-    label: "{{ item }}"
-
-- name: Warn on missing component checkouts
-  ansible.builtin.debug:
-    msg: "WARN: component '{{ item.item }}' not found at expected sibling path (no go.mod). Run scripts/clone-deps.sh."
-  loop: "{{ preflight_components.results }}"
-  when: not item.stat.exists
-  loop_control:
-    label: "{{ item.item }}"
-
-- name: Check port availability
-  ansible.builtin.shell: "ss -tlnH 'sport = :{{ item.port }}' | head -1"
-  register: preflight_ports
-  loop: "{{ spinifex_ports }}"
-  changed_when: false
-  failed_when: false
-  loop_control:
-    label: "{{ item.service }} ({{ item.port }})"
-
-- name: Warn on occupied ports
-  ansible.builtin.debug:
-    msg: "WARN: port {{ item.item.port }} occupied (needed for {{ item.item.service }})"
-  loop: "{{ preflight_ports.results }}"
-  when: item.stdout | length > 0
-  loop_control:
-    label: "{{ item.item.service }}"
-
-- name: Preflight summary
-  ansible.builtin.debug:
-    msg: "Preflight passed. Required: {{ spinifex_required_commands | join(', ') }}. Warnings above (if any) are non-fatal."