release: update go.mod for v1.8.0

* feat: join nodegroup workers via cluster NLB endpoint

Under the Set-B managed-CP-VPC topology the control plane runs in the
system-account CP VPC while nodegroup workers launch in the customer
VPC. The agent user-data set K3S_URL to the in-VPC control-plane ENI IP
(:6443), which sits in a different, unpeered VPC and is unreachable, so
the join path was dead.

Point K3S_URL at the published cluster NLB endpoint (meta.Endpoint,
https://<frontend-ip>:443, the :443->:6443 listener kubectl already
uses). Guard CreateNodegroup on a non-empty endpoint.

mulga-siv-298

* feat: gate nodegroup ACTIVE on workers registering Ready

launchNodegroupInfra flipped the nodegroup to ACTIVE the moment
RunInstances returned, so a worker that booted but never joined left the
nodegroup falsely ACTIVE with no usable capacity.

Gate ACTIVE on the cluster's Ready-node count (meta.NodeCount, refreshed
by the cluster reconciler from the CP NATS state report — Ready nodes
only) rising by the desired worker count over the create-time baseline.
On timeout the nodegroup is marked CREATE_FAILED with its instance IDs
retained for the reclaim path. Mirrors the CP reconciler's
healthy-observe gating. Bounded by injectable timeout/poll knobs.

mulga-siv-299

* feat(eks): private endpoint access via cross-account Set A ENI

Make EndpointPrivateAccess real: in-VPC workers and kubectl reach the
control plane through a customer-VPC (Set A) resident ENI on the cluster
NLB's LB VM, removing the public-NLB hairpin and the NAT-GW egress path.

Teardown: the Set A ENI is an extra cross-account NIC on the LB VM, not
its primary, so the instance-terminate cascade never reclaims it. Detach
(store-clear) the stale attachment before deleting, so private clusters
delete cleanly instead of looping on InvalidNetworkInterface.InUse.

Refs: mulga-siv-301

* fix(vpc): honor SecurityGroupRuleIds in RevokeSecurityGroup ingress/egress

Modern Terraform (aws_vpc_security_group_ingress_rule) revokes per-rule
resources by SecurityGroupRuleId with empty IpPermissions. The handlers
only read IpPermissions, so they returned success while removing nothing;
the surviving SG-reference rule then pinned both SGs undeletable
(DependencyViolation), deadlocking destroy of mutually-referencing SGs.

Resolve SecurityGroupRuleIds to their stored rules and remove them
(malformed -> InvalidSecurityGroupRuleId.Malformed, absent -> NotFound).

Refs: mulga-siv-321

* fix(vpc): reap the main route table on DeleteVpc

CreateVpc auto-creates a main route table and DeleteRouteTable refuses to
delete a main RT (AWS-faithful), so nothing reclaimed it — every deleted
VPC leaked one orphaned main RT in the rtb KV bucket. DeleteVpc now reaps
the VPC's main route table before deleting the VPC record.

Refs: mulga-siv-322

* fix: make AttachRolePolicy and AssociateRouteTable atomic via KV CAS

Concurrent writes to a single role / route table record used a blind
read-modify-Put with no optimistic-concurrency guard, so parallel writers
racing from the same revision clobbered each other and lost updates.
Verified on env19: three parallel policy attachments to one node role kept
only one; two parallel subnet associations to one route table kept only
one (the provider then timed out waiting for state 'associated').

Route the mutating paths (AttachRolePolicy/DetachRolePolicy,
AssociateRouteTable/DisassociateRouteTable) through a revision-guarded
Update + bounded retry, matching the existing accountCounterBucket CAS
idiom. Add concurrent regression tests covering both lost updates.

* docs: add EKS Terraform workbooks (quickstart, addons-access, https-ingress)

Three OpenTofu workbooks demonstrating EKS on Spinifex, each ending in a
browsable load-balanced demo app:
- eks-quickstart: minimal single-node cluster + NodePort demo
- eks-addons-access: CoreDNS managed addon + access-entry RBAC
- eks-https-ingress: 3-node cluster behind an HTTPS ALB terminating TLS
  with an ACM-imported self-signed certificate

Each covers the Spinifex-specific endpoints, the auto-managed worker SG
NodePort pattern, and the omitted addon_version constraint.

.github/CODEOWNERS

@@ -6,16 +6,10 @@
 # Set the matching branch protection rule on `dev` and `main` to
 # "Require review from Code Owners" or this file is advisory only.
 
-# Fall-through: any change needs a team review.
-*                           @mulgadc/mulgadc-engineers
-
 # Network layer is governed by ADR-0006 invariant tests.
-# Changes here must respect those clauses — see
-# spinifex/network/invariants/ and
+# Changes here must respect those clauses
+# see spinifex/network/invariants/
 # docs/development/proposals/az-local-architecture/0006-spinifex-network-layer-contract.md
 /spinifex/network/          @joshsiv-mulga
 /spinifex/vpcd/             @joshsiv-mulga
-
-# Invariants suite is the executable ADR-0006 fence — any widening or
-# narrowing of the scan roots needs an explicit owner review.
 /spinifex/network/invariants/ @joshsiv-mulga

.github/workflows/e2e-baremetal.yml

@@ -2,6 +2,13 @@ name: E2E Bare-Metal
 
 on:
   workflow_dispatch:
+  workflow_call:
+    inputs:
+      ref:
+        description: "spinifex ref to checkout/test (default: triggering ref)"
+        required: false
+        type: string
+        default: ""
 
 permissions:
   contents: read
@@ -12,7 +19,7 @@ env:
 
 jobs:
   e2e_baremetal:
-    name: Bare-Metal Multi-Node E2E
+    name: Bare-Metal Single-Node E2E
     runs-on: [ self-hosted, ci-baremetal ]
     timeout-minutes: 120
     concurrency:
@@ -20,11 +27,17 @@ jobs:
       cancel-in-progress: false
     env:
       BM_ISO_CACHE: /var/cache/spinifex-iso-builder
+      ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-20-${{ github.run_id }}
     steps:
       - name: Clean Workspace
-        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          sudo umount -R -l "${BM_ISO_CACHE}/rootfs" 2>/dev/null || true
+          if [ ! -c /dev/null ]; then sudo rm -f /dev/null; sudo mknod -m 666 /dev/null c 1 3; fi
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
 
       - uses: actions/checkout@v6
+        with:
+          ref: ${{ inputs.ref || github.ref }}
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -91,6 +104,8 @@ jobs:
             echo "Ubuntu 26.04 cloud image cached ($(du -h "$UBUNTU_IMG" | cut -f1))"
           fi
 
+      - name: Install go-junit-report on runner
+        run: go install github.com/jstemmer/go-junit-report/v2@latest
 
       - name: Run Bare-Metal E2E
         env:
@@ -100,8 +115,34 @@ jobs:
           BM_UBUNTU_IMAGE: ${{ env.BM_ISO_CACHE }}/ubuntu-26.04-cloudimg-amd64.img
           SPX_EXTERNAL_POOL: ${{ env.SPINIFEX_CI_POOL }}
         run: |
+          mkdir -p "${ARTIFACT_DIR}"
+          set -o pipefail
           sudo -E mulga/scripts/baremetal/run-baremetal-e2e.sh \
             --vmlinuz   "${BM_ISO_CACHE}/vmlinuz" \
             --initrd    "${BM_ISO_CACHE}/initrd.img" \
             --squashfs  "${BM_ISO_CACHE}/spinifex.squashfs" \
-            --tests-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+            --tests-tar "${RUNNER_TEMP}/spinifex-tests.tar" \
+            2>&1 | tee "${ARTIFACT_DIR}/test-baremetal.log"
+
+      - name: Generate JUnit report
+        if: always()
+        run: |
+          LOG="${ARTIFACT_DIR}/test-baremetal.log"
+          [ -f "$LOG" ] || exit 0
+          go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-baremetal.xml" || true
+
+      - name: Analyze e2e failures
+        if: always()
+        uses: ./.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "E2E failure analysis (cell 20)"
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-artifacts-cell-20-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14

.github/workflows/e2e-ddil.yml

@@ -21,10 +21,6 @@ jobs:
   ddil:
     name: DDIL Real Multi-Node
     runs-on: [ self-hosted, ci-multi ]
-    # Budget: ~45m go test + tofu provision/bootstrap/teardown on ci-multi.
-    # In Phase 1 the scenarios all t.Skip, so the real-world runtime is
-    # dominated by provisioning; the 60m ceiling leaves headroom for
-    # Phase 2/3 when assertions come online.
     timeout-minutes: 60
     env:
       TOFU_LOG: /tmp/e2e-tofu-ddil-${{ github.run_id }}.log
@@ -39,10 +35,6 @@ jobs:
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
-      # Full spinifex checkout: the Go harness must run on the runner
-      # (outside the cluster) so its SSH path survives scenario partitions
-      # that drop peer-to-peer cluster traffic. Running inside the cluster
-      # would self-sever when Scenario C isolates a node.
       - uses: actions/checkout@v6
         name: Checkout Spinifex
         with:
@@ -88,7 +80,6 @@ jobs:
         working-directory: mulga/scripts/tofu-cluster
         run: |
           set -euo pipefail
-          # shellcheck source=/dev/null
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           export TF_VAR_git_branch="${GIT_BRANCH}"
           quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
@@ -140,10 +131,6 @@ jobs:
             exit 1
           fi
 
-          # Guest SSH key: bootstrap-install.sh imports spinifex-key into EC2;
-          # the matching private material was generated on the primary node at
-          # ~/.ssh/spinifex-key. Pull it to the runner so the harness can SSH
-          # the witness via its MapPublicIpOnLaunch-assigned EIP.
           GUEST_KEY="${RUNNER_TEMP}/spinifex-key"
           scp -i "${SSH_KEY}" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
             "${SSH_USER}@${PRIMARY_IP}:~/.ssh/spinifex-key" "${GUEST_KEY}"
@@ -151,11 +138,6 @@ jobs:
 
           mkdir -p "${RUNNER_TEMP}/ddil-results"
           cd "${GITHUB_WORKSPACE}/spinifex"
-          # gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout,
-          # plus machine-readable jsonfile + JUnit XML artifacts. The JUnit
-          # file feeds the junit-report composite action below so failures
-          # become summary table rows + workflow annotations.
-          # shellcheck source=/dev/null
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           banner "DDIL scenarios"
           DDIL_NODES="${NODE_IPS}" \
@@ -185,10 +167,6 @@ jobs:
         if: failure()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          # Pull spinifex-* journals + QEMU process listings from every
-          # node into the artifact bundle so failures can be triaged
-          # without re-provisioning the cluster. Best-effort — a missing
-          # node should not mask the original test failure.
           set +e
           INVENTORY=$(tofu output -json inventory)
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
@@ -197,7 +175,6 @@ jobs:
           LOG_DIR="${RUNNER_TEMP}/ddil-results/cluster-logs"
           mkdir -p "${LOG_DIR}"
 
-          # shellcheck source=/dev/null
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           mapfile -t NODE_IPS < <(echo "$INVENTORY" | jq -r '.nodes[].wan_ip')
           for ip in "${NODE_IPS[@]}"; do
@@ -221,7 +198,6 @@ jobs:
         working-directory: mulga/scripts/tofu-cluster
         run: |
           set -euo pipefail
-          # shellcheck source=/dev/null
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
           quiet_run "tofu destroy"          "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false

.github/workflows/e2e-nightly.yml

@@ -40,6 +40,7 @@ jobs:
     timeout-minutes: 35
     env:
       TOFU_LOG: /tmp/e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}.log
+      ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}
 
     steps:
       # blank or scheduled run = full matrix.
@@ -60,7 +61,10 @@ jobs:
 
       - name: Clean Workspace
         if: steps.gate.outputs.selected == 'true'
-        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          sudo umount -R -l /var/cache/spinifex-iso-builder/rootfs 2>/dev/null || true
+          if [ ! -c /dev/null ]; then sudo rm -f /dev/null; sudo mknod -m 666 /dev/null c 1 3; fi
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         if: steps.gate.outputs.selected == 'true'
@@ -78,6 +82,11 @@ jobs:
           sparse-checkout: scripts/tofu-cluster
           path: mulga
 
+      - uses: actions/checkout@v6
+        if: steps.gate.outputs.selected == 'true'
+        with:
+          path: spinifex
+
       - name: Build Install Artifacts
         if: steps.gate.outputs.selected == 'true'
         working-directory: mulga/scripts/tofu-cluster
@@ -132,8 +141,18 @@ jobs:
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
-          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
-            "cd ~/e2e-tests && ./run-e2e.sh && ./run-cert-e2e.sh && ENDPOINT=https://${WAN_IP}:9999 ./run-lb-e2e.sh"
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
+             for SUITE in single cert lb; do \
+               echo \"::group::suite \$SUITE\"; \
+               date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
+               GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+                 ARTIFACT_DIR='${ARTIFACT_DIR}' \
+                 ./\$SUITE.test -test.v -test.timeout=30m \
+                 2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log'; \
+               rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
+               if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
+             done; exit \$FAIL"
 
       - name: Run Multi Node E2E
         if: steps.gate.outputs.selected == 'true' && matrix.topo == 'real-multi'
@@ -144,32 +163,57 @@ jobs:
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
 
-          NODE_IPS=""
+          NODES_CSV=""
           NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
           for i in $(seq 0 $((NODE_COUNT - 1))); do
-            NODE_IPS="${NODE_IPS} $(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")"
+            ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
+            NODES_CSV="${NODES_CSV:+$NODES_CSV,}$ip"
           done
 
-          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
-            "cd ~/e2e-tests && ./run-multinode-e2e.sh ${NODE_IPS} && ./run-cert-e2e.sh --multinode ${NODE_IPS} && ENDPOINT=https://${NODE1_IP}:9999 ./run-lb-e2e.sh --nodes ${NODE_IPS}"
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
+             for SUITE in multinode cert lb; do \
+               echo \"::group::suite \$SUITE\"; \
+               date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
+               GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
+                 SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
+                 SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
+                 ARTIFACT_DIR='${ARTIFACT_DIR}' \
+                 ./\$SUITE.test -test.v -test.timeout=30m \
+                 2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log'; \
+               rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
+               if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
+             done; exit \$FAIL"
 
       - name: Run Tofu Workbooks
         if: steps.gate.outputs.selected == 'true' && matrix.extra == 'tofu-examples'
         working-directory: mulga/scripts/tofu-cluster
+        # The driver emits go test -v markers per workbook; teeing to
+        # test-tofu.log feeds the same go-junit-report -> e2e-analyze -> upload
+        # chain the Go suites use, so cell 17 produces an RCA bundle too.
         run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
           INVENTORY=$(tofu output -json inventory)
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          date -u +%Y-%m-%dT%H:%M:%SZ > "${ARTIFACT_DIR}/test-tofu.start"
           ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
-            "cd ~/e2e-tests && ./run-tofu-examples-e2e.sh"
+            "cd ~/e2e-tests && ./run-tofu-examples-e2e.sh" \
+            2>&1 | tee "${ARTIFACT_DIR}/test-tofu.log"
+          rc=${PIPESTATUS[0]}
+          if [ $rc -ne 0 ]; then echo "tofu: FAILED" >> "${ARTIFACT_DIR}/suite-status.txt"; fi
+          exit $rc
 
       - name: Run Reboot Resilience E2E
         if: steps.gate.outputs.selected == 'true' && matrix.extra == 'reboot-single'
         working-directory: mulga/scripts/tofu-cluster
-        # Driver runs from the runner (NOT via ssh-and-run) so it survives the
+        # reboot.test runs from the runner (NOT via ssh-and-run) so it survives the
         # systemctl-reboot-induced ssh disconnect mid-run.
         run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
           INVENTORY=$(tofu output -json inventory)
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
@@ -178,31 +222,85 @@ jobs:
           REBOOT_DIR="${RUNNER_TEMP}/reboot-e2e"
           mkdir -p "$REBOOT_DIR"
           tar xf "${RUNNER_TEMP}/spinifex-tests.tar" -C "$REBOOT_DIR"
-          chmod +x "$REBOOT_DIR/run-reboot-e2e.sh"
+          chmod +x "$REBOOT_DIR/_bin/reboot.test"
 
-          WAN_IP="$WAN_IP" SSH_KEY="$SSH_KEY" "$REBOOT_DIR/run-reboot-e2e.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i "$SSH_KEY")
+          CREDS=$(ssh "${SSH_OPTS[@]}" tf-user@${WAN_IP} 'cat ~/.aws/credentials')
+          AWS_AK=$(echo "$CREDS" | awk -F '= *' '/^aws_access_key_id/ {print $2; exit}')
+          AWS_SK=$(echo "$CREDS" | awk -F '= *' '/^aws_secret_access_key/ {print $2; exit}')
+          echo "::add-mask::${AWS_AK}"; echo "::add-mask::${AWS_SK}"
 
-      - name: Dump Cluster Journals on Failure
-        if: failure() && steps.gate.outputs.selected == 'true'
+          date -u +%Y-%m-%dT%H:%M:%SZ > "${ARTIFACT_DIR}/test-reboot.start"
+          GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+            SPINIFEX_WAN_IP="${WAN_IP}" SPINIFEX_SSH_KEY="${SSH_KEY}" \
+            SPINIFEX_AWS_ENDPOINT="https://${WAN_IP}:9999" \
+            SPINIFEX_AWS_ACCESS_KEY_ID="${AWS_AK}" SPINIFEX_AWS_SECRET_ACCESS_KEY="${AWS_SK}" \
+            ARTIFACT_DIR="${ARTIFACT_DIR}" \
+            "$REBOOT_DIR/_bin/reboot.test" -test.v -test.timeout=30m \
+            2>&1 | tee "${ARTIFACT_DIR}/test-reboot.log"
+          rc=${PIPESTATUS[0]}
+          if [ $rc -ne 0 ]; then echo "reboot: FAILED" >> "${ARTIFACT_DIR}/suite-status.txt"; fi
+          exit $rc
+
+      - name: Collect e2e artifacts
+        if: always() && steps.gate.outputs.selected == 'true'
         working-directory: mulga/scripts/tofu-cluster
         run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
           INVENTORY=$(tofu output -json inventory 2>/dev/null) || { echo "no inventory; skipping"; exit 0; }
-          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
-          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path'); SSH_KEY="${SSH_KEY/#\~/$HOME}"
           NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY")
+
+          ssh "${SSH_OPTS[@]}" tf-user@${NODE1_IP} "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+
           for i in $(seq 0 $((NODE_COUNT - 1))); do
-            IP=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
-            echo "===== node$((i+1)) ($IP) ====="
-            for svc in spinifex-nats spinifex-predastore spinifex-viperblock spinifex-daemon spinifex-awsgw spinifex-vpcd dhcpcd; do
-              echo "--- $svc ---"
-              ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" tf-user@${IP} \
-                "sudo journalctl -u $svc --no-pager -n 200 2>/dev/null || echo '(not found)'" 2>&1 || echo "(node unreachable)"
-            done
-            echo "--- DHCP client state ---"
-            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" tf-user@${IP} \
-              "systemctl is-enabled dhcpcd.service 2>/dev/null; systemctl is-active dhcpcd.service 2>/dev/null; sudo ss -lupn 'sport = 68' 2>/dev/null; pgrep -laf dhcpcd 2>/dev/null" 2>&1 || echo "(node unreachable)"
+            IP=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip"); idx=$((i + 1))
+            mkdir -p "${ARTIFACT_DIR}/vm-console-node${idx}"
+            ssh "${SSH_OPTS[@]}" tf-user@${IP} 'sudo sh -c "cd /run/spinifex 2>/dev/null && tar -cf - console-*.log 2>/dev/null"' \
+              | tar -C "${ARTIFACT_DIR}/vm-console-node${idx}" -xf - 2>/dev/null
+            ssh "${SSH_OPTS[@]}" tf-user@${IP} 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "60 min ago"' \
+              > "${ARTIFACT_DIR}/spinifex-journal-node${idx}.log" 2>/dev/null
           done
 
+          cat "${ARTIFACT_DIR}"/spinifex-journal-node*.log \
+            > "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || true
+          echo "collected: $(ls "${ARTIFACT_DIR}" 2>/dev/null | wc -l) entries"
+
+      - name: Install go-junit-report
+        if: always() && steps.gate.outputs.selected == 'true'
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Convert test logs to JUnit XML
+        if: always() && steps.gate.outputs.selected == 'true'
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log); SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Analyze e2e failures
+        if: always() && steps.gate.outputs.selected == 'true'
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "E2E failure analysis (cell ${{ matrix.cell }})"
+
+      - uses: actions/upload-artifact@v7
+        if: always() && steps.gate.outputs.selected == 'true'
+        with:
+          name: e2e-artifacts-cell-${{ matrix.cell }}-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
       - name: Teardown
         if: always()
         working-directory: mulga/scripts/tofu-cluster
@@ -216,3 +314,21 @@ jobs:
           tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
           tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1
           ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+
+  # Bare-metal single-node suite. Doesn't fit the tofu matrix shape, so it
+  # runs as a sibling job via the reusable e2e-baremetal workflow. Gated like a
+  # matrix cell ("cell 20"): every weeknight cron, or dispatch with 20 selected
+  # (blank cells = all). Tests `dev` on schedule to match GIT_BRANCH above.
+  baremetal:
+    name: cell-20 (baremetal/pxe/real-hw)
+    if: >-
+      github.event_name == 'schedule' ||
+      inputs.cells == '' ||
+      contains(format(',{0},', inputs.cells), ',20,')
+    permissions:
+      contents: read
+      statuses: write
+    uses: ./.github/workflows/e2e-baremetal.yml
+    with:
+      ref: ${{ github.event_name == 'schedule' && 'dev' || github.ref_name }}
+    secrets: inherit

.github/workflows/e2e-suites.yml

@@ -0,0 +1,799 @@
+# E2E (per-suite blocks) — EXPERIMENTAL, workflow_dispatch only.
+#
+# Restructures the looped e2e.yml into env-grouped job blocks so each suite
+# surfaces as its own node in the Actions graph (mulga-siv-283):
+#
+#   single_provision ─┬─ single_suite (matrix: single, cert, lb, eks)
+#                     └─ single_teardown (if: always)
+#   multi_provision  ─┬─ multi_suite  (matrix: multinode)
+#                     └─ multi_teardown (if: always)
+#
+# Constraint: tofu state is runner-filesystem-local, so *_provision and
+# *_teardown must execute on the same self-hosted runner (one runner per label
+# today). Teardown reuses the persisted mulga/ tofu-cluster workspace and does
+# NOT re-checkout mulga (that would wipe local state). Suite-matrix legs reach
+# the already-provisioned VM over its WAN IP using the key path the provision
+# job emits; they share the runner home dir, so the key file persists.
+#
+# This file does NOT post the gating `e2e` commit status — it runs alongside
+# e2e.yml for validation. Once green across a few manual runs, fold the
+# triggers from e2e.yml here and retire the loop.
+
+name: E2E (suite blocks)
+
+on:
+  workflow_dispatch:
+    inputs:
+      single_suites:
+        description: "Single-node suites to run as matrix legs"
+        required: false
+        default: "single cert lb eks"
+      run_multinode:
+        description: "Also run the multinode env block"
+        type: boolean
+        required: false
+        default: true
+      test_run:
+        description: "Optional -test.run regex (empty = all tests in suite)"
+        required: false
+        default: ""
+
+permissions:
+  contents: read
+
+env:
+  GIT_BRANCH: ${{ github.ref_name }}
+  TEST_RUN: ${{ inputs.test_run || '' }}
+
+jobs:
+  # ---------------------------------------------------------------------------
+  # Single-node env
+  # ---------------------------------------------------------------------------
+  single_provision:
+    name: Single env · provision
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 40
+    outputs:
+      wan_ip: ${{ steps.vm.outputs.wan_ip }}
+      ssh_key: ${{ steps.vm.outputs.ssh_key }}
+      suites: ${{ steps.suites.outputs.list }}
+      matrix: ${{ steps.suites.outputs.matrix }}
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
+    steps:
+      - name: Reap Orphan Procs
+        run: |
+          set +e
+          UID_SELF=$(id -u)
+          WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in
+                "$WS"|"$WS"/*) echo "reap $sig pid=$pid cwd=$cwd"; kill "-$sig" "$pid" 2>/dev/null ;;
+              esac
+            done
+          }
+          reap TERM; sleep 3; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          true
+
+      - name: Clean Workspace
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - uses: actions/checkout@v6
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/tofu-cluster
+          path: mulga
+
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Re-enable Go toolchain auto-download
+        run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
+
+      - name: Compute suite matrix
+        id: suites
+        run: |
+          LIST="${{ inputs.single_suites || 'single cert lb eks' }}"
+          echo "list=${LIST}" >> "$GITHUB_OUTPUT"
+          JSON=$(printf '%s\n' $LIST | jq -R . | jq -cs .)
+          echo "matrix=${JSON}" >> "$GITHUB_OUTPUT"
+
+      - name: Reclaim docker build cache
+        run: |
+          docker container prune -f || true
+          docker image prune -f || true
+          docker buildx prune -af --filter "until=24h" || true
+          docker system df || true
+
+      - name: Build Install Artifacts
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Provision VM
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          export TF_VAR_git_branch="${GIT_BRANCH}"
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "VM provisioned"
+
+      - name: Bootstrap via Binary Install
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Resolve VM SSH details
+        id: vm
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          echo "wan_ip=${WAN_IP}" >> "$GITHUB_OUTPUT"
+          echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
+
+      - name: Build eks-node AMI (libguestfs, runner)
+        if: contains(steps.suites.outputs.list, 'eks')
+        working-directory: spinifex
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "build eks-node AMI (libguestfs — no host devices)"
+          sudo apt-get update -qq
+          sudo apt-get install -y --no-install-recommends libguestfs-tools qemu-utils
+          export LIBGUESTFS_BACKEND=direct
+          sudo chmod 0644 /boot/vmlinuz-* || true
+          group_open "make build-eks-node-image"
+          make build-eks-node-image
+          group_close
+          ok "raw: $(du -h /tmp/eks-node-image-build/eks-node-alpine.raw | cut -f1)"
+
+      - name: Import eks-node AMI on VM
+        if: contains(steps.suites.outputs.list, 'eks')
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          eval "$(grep -E '^(ALPINE_VERSION|AMI_NAME|SYSTEM_TAG)=' \
+            spinifex/scripts/images/eks-node/manifest.conf)"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          banner "import eks-node AMI to VM image store"
+          scp "${SSH_OPTS[@]}" \
+            /tmp/eks-node-image-build/eks-node-alpine.raw \
+            "$VM":/tmp/eks-node-alpine.raw
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "spx admin images import --file /tmp/eks-node-alpine.raw \
+               --distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
+               --boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}" >/dev/null
+          ok "eks-node AMI imported"
+
+      - name: Install kubectl on VM
+        if: contains(steps.suites.outputs.list, 'eks')
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          banner "install kubectl on VM (eks data-plane subtest)"
+          ssh "${SSH_OPTS[@]}" "$VM" 'bash -s' <<'EOF'
+          set -euo pipefail
+          KVER=v1.32.5
+          case "$(uname -m)" in
+            x86_64)  ARCH=amd64 ;;
+            aarch64) ARCH=arm64 ;;
+            *) echo "unsupported arch $(uname -m)" >&2; exit 1 ;;
+          esac
+          cd /tmp
+          curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl"
+          curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl.sha256"
+          echo "$(cat kubectl.sha256)  kubectl" | sha256sum -c -
+          sudo install -m 0755 kubectl /usr/local/bin/kubectl
+          kubectl version --client
+          EOF
+          ok "kubectl installed on VM"
+
+      - name: Build E2E test binaries on runner
+        working-directory: spinifex/tests/e2e
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          # teardown always builds: the run-scoped sweep binary runs in the
+          # teardown job, it is never a selectable suite leg.
+          SUITES_BUILD="${{ steps.suites.outputs.list }} teardown"
+          banner "build ${SUITES_BUILD// /, }"
+          group_open "go test -c"
+          make e2e-build SUITES_BUILD="${SUITES_BUILD}"
+          group_close
+          for f in _bin/*.test; do
+            ok "$(basename "$f") $(stat -c '%s' "$f" | numfmt --to=iec)"
+          done
+
+      - name: Ship test binaries to VM
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          ssh "${SSH_OPTS[@]}" "$VM" "rm -rf \$HOME/spinifex-e2e-bin && mkdir -p \$HOME/spinifex-e2e-bin"
+          tar -C spinifex/tests/e2e/_bin -cf - . | ssh "${SSH_OPTS[@]}" "$VM" "tar -xf - -C \$HOME/spinifex-e2e-bin"
+          ok "shipped binaries to $VM"
+
+      - name: Start packet capture on VM br-wan
+        run: |
+          set +e
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          scp "${SSH_OPTS[@]}" \
+            "${GITHUB_WORKSPACE}/spinifex/.github/scripts/start-net-capture.sh" \
+            "$VM":/tmp/start-net-capture.sh
+          ssh "${SSH_OPTS[@]}" "$VM" "bash /tmp/start-net-capture.sh" || true
+
+  single_suite:
+    name: Single env · suite
+    needs: single_provision
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 45
+    strategy:
+      fail-fast: false
+      matrix:
+        suite: ${{ fromJSON(needs.single_provision.outputs.matrix) }}
+    env:
+      WAN_IP: ${{ needs.single_provision.outputs.wan_ip }}
+      SSH_KEY: ${{ needs.single_provision.outputs.ssh_key }}
+      ARTIFACT_DIR: /tmp/spinifex-e2e-single-${{ github.run_id }}-${{ matrix.suite }}
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Install go-junit-report
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Run ${{ matrix.suite }}
+        id: e2e
+        run: |
+          set +e
+          SUITE="${{ matrix.suite }}"
+          VM_HOST="tf-user@${WAN_IP}"
+          TEST_RUN_FLAG=""
+          [ -n "${TEST_RUN}" ] && TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+          cleanup_remote() {
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" "$VM_HOST" \
+              "pkill -u tf-user -TERM -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; true" \
+              2>/dev/null || true
+          }
+          trap cleanup_remote EXIT INT TERM
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" "$VM_HOST" \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/spinifex-e2e-bin; \
+             R=\$'\033[0m' B=\$'\033[1m' C=\$'\033[36m' G=\$'\033[32m' RED=\$'\033[31m'; \
+             printf '\n%s%s━━ suite %s ━━%s\n' \"\$B\" \"\$C\" \"${SUITE}\" \"\$R\"; \
+             date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-${SUITE}.start'; \
+             GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+               SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+               ARTIFACT_DIR='${ARTIFACT_DIR}' \
+               ./${SUITE}.test -test.v -test.timeout=30m ${TEST_RUN_FLAG} \
+               2>&1 | tee '${ARTIFACT_DIR}/test-${SUITE}.log' \
+                    | sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'; \
+             exit \${PIPESTATUS[0]}"
+          echo "rc=$?" >> "$GITHUB_OUTPUT"
+          exit 0
+
+      - name: Collect suite artifacts from VM
+        if: always()
+        run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${SSH_KEY}")
+          VM=tf-user@${WAN_IP}
+          ssh "${SSH_OPTS[@]}" "$VM" "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+          true
+
+      - name: Convert test log to JUnit XML
+        if: always()
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log); SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          title: "E2E ${{ matrix.suite }} results"
+
+      - name: Analyze e2e failures
+        if: always()
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "E2E ${{ matrix.suite }} failure analysis"
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-single-${{ matrix.suite }}-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
+      - name: Fail job if suite failed
+        if: always()
+        run: |
+          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
+            echo "::error::suite ${{ matrix.suite }} failed (rc=${{ steps.e2e.outputs.rc }})"
+            exit 1
+          fi
+
+  single_teardown:
+    name: Single env · teardown
+    needs: [ single_provision, single_suite ]
+    if: always()
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 20
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-single-td-${{ github.run_id }}.log
+      ARTIFACT_DIR: /tmp/spinifex-e2e-single-diag-${{ github.run_id }}
+      WAN_IP: ${{ needs.single_provision.outputs.wan_ip }}
+      SSH_KEY: ${{ needs.single_provision.outputs.ssh_key }}
+    steps:
+      # Reuse the persisted mulga/ tofu-cluster workspace from single_provision.
+      # Re-checkout only spinifex (for ci-fmt + diag scripts); checking out mulga
+      # again would wipe the local tofu state and orphan the VM.
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - name: Capture network/IPsec diagnostics on VM
+        if: ${{ needs.single_provision.result == 'success' && env.WAN_IP != '' }}
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}")
+          VM=tf-user@${WAN_IP}
+          scp "${SSH_OPTS[@]}" \
+            "${GITHUB_WORKSPACE}/spinifex/.github/scripts/capture-net-diag.sh" \
+            "$VM":/tmp/capture-net-diag.sh 2>/dev/null
+          ssh "${SSH_OPTS[@]}" "$VM" "bash /tmp/capture-net-diag.sh '${ARTIFACT_DIR}/net-diag'" 2>/dev/null
+          ssh "${SSH_OPTS[@]}" "$VM" 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "90 min ago"' \
+            > "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null
+          mkdir -p "${ARTIFACT_DIR}/lb-console"
+          ssh "${SSH_OPTS[@]}" "$VM" 'sudo sh -c "cd /run/spinifex 2>/dev/null && tar -cf - console-*.log 2>/dev/null"' \
+            | tar -C "${ARTIFACT_DIR}/lb-console" -xf - 2>/dev/null
+          true
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-single-diag-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: ignore
+          retention-days: 14
+
+      # Reclaim any resource leaked by a crashed/interrupted suite before the VM
+      # is destroyed. Scoped to this run's tag, best-effort, never fails the job.
+      - name: Sweep leaked run resources on VM
+        if: ${{ needs.single_provision.result == 'success' && env.WAN_IP != '' }}
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}")
+          VM=tf-user@${WAN_IP}
+          banner "sweep run-tagged resources (run ${{ github.run_id }})"
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "cd \$HOME/spinifex-e2e-bin 2>/dev/null && \
+             SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+             SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+             ./teardown.test -test.v -test.run TestSweepRunResources -test.timeout=10m" \
+            2>&1 || true
+          true
+
+      - name: Teardown VM
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          if [ -n "${WAN_IP}" ] && [ -n "${SSH_KEY}" ]; then
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}" \
+              tf-user@${WAN_IP} \
+              'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
+              2>/dev/null || true
+          fi
+          UID_SELF=$(id -u); WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in "$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;; esac
+            done
+          }
+          reap TERM; sleep 2; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
+          ok "teardown complete"
+
+  # ---------------------------------------------------------------------------
+  # Multinode env
+  # ---------------------------------------------------------------------------
+  multi_provision:
+    name: Multi env · provision
+    if: ${{ inputs.run_multinode }}
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 45
+    outputs:
+      node1_ip: ${{ steps.cluster.outputs.node1_ip }}
+      ssh_key: ${{ steps.cluster.outputs.ssh_key }}
+      nodes_csv: ${{ steps.cluster.outputs.nodes_csv }}
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
+    steps:
+      - name: Reap Orphan Procs
+        run: |
+          set +e
+          UID_SELF=$(id -u); WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in "$WS"|"$WS"/*) echo "reap $sig pid=$pid"; kill "-$sig" "$pid" 2>/dev/null ;; esac
+            done
+          }
+          reap TERM; sleep 3; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          true
+
+      - name: Clean Workspace
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - uses: actions/checkout@v6
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/tofu-cluster
+          path: mulga
+
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Re-enable Go toolchain auto-download
+        run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
+
+      - name: Reclaim docker build cache
+        run: |
+          docker container prune -f || true
+          docker image prune -f || true
+          docker buildx prune -af --filter "until=24h" || true
+          docker system df || true
+
+      - name: Build Install Artifacts
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Provision VMs
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          export TF_VAR_git_branch="${GIT_BRANCH}"
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "3-node cluster provisioned"
+
+      - name: Bootstrap via Binary Install
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Resolve cluster SSH details
+        id: cluster
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path'); SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          NODES_CSV=""
+          for i in $(seq 0 $((NODE_COUNT - 1))); do
+            ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
+            if [ -z "$NODES_CSV" ]; then NODES_CSV="$ip"; else NODES_CSV="$NODES_CSV,$ip"; fi
+          done
+          echo "node1_ip=${NODE1_IP}" >> "$GITHUB_OUTPUT"
+          echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
+          echo "nodes_csv=${NODES_CSV}" >> "$GITHUB_OUTPUT"
+
+      - name: Build multinode binary on runner
+        working-directory: spinifex/tests/e2e
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "build multinode, teardown"
+          group_open "go test -c"
+          make e2e-build SUITES_BUILD="multinode teardown"
+          group_close
+          ok "$(basename _bin/multinode.test) $(stat -c '%s' _bin/multinode.test | numfmt --to=iec)"
+
+      - name: Ship test binaries to node1
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.cluster.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.cluster.outputs.node1_ip }}
+          ssh "${SSH_OPTS[@]}" "$VM" "rm -rf \$HOME/spinifex-e2e-bin && mkdir -p \$HOME/spinifex-e2e-bin"
+          tar -C spinifex/tests/e2e/_bin -cf - . | ssh "${SSH_OPTS[@]}" "$VM" "tar -xf - -C \$HOME/spinifex-e2e-bin"
+          ok "shipped binaries to $VM"
+
+      - name: Start packet capture on all nodes
+        run: |
+          set +e
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.cluster.outputs.ssh_key }}")
+          IFS=',' read -ra NODE_LIST <<< "${{ steps.cluster.outputs.nodes_csv }}"
+          for ip in "${NODE_LIST[@]}"; do
+            [ -z "$ip" ] && continue
+            scp "${SSH_OPTS[@]}" \
+              "${GITHUB_WORKSPACE}/spinifex/.github/scripts/start-net-capture.sh" \
+              tf-user@"$ip":/tmp/start-net-capture.sh
+            ssh "${SSH_OPTS[@]}" tf-user@"$ip" "bash /tmp/start-net-capture.sh" || true
+          done
+
+  multi_suite:
+    name: Multi env · suite
+    needs: multi_provision
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 60
+    strategy:
+      fail-fast: false
+      matrix:
+        suite: [ multinode ]
+    env:
+      NODE1_IP: ${{ needs.multi_provision.outputs.node1_ip }}
+      SSH_KEY: ${{ needs.multi_provision.outputs.ssh_key }}
+      NODES_CSV: ${{ needs.multi_provision.outputs.nodes_csv }}
+      ARTIFACT_DIR: /tmp/spinifex-e2e-multi-${{ github.run_id }}-${{ matrix.suite }}
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Install go-junit-report
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Run ${{ matrix.suite }}
+        id: e2e
+        run: |
+          set +e
+          SUITE="${{ matrix.suite }}"
+          VM_HOST="tf-user@${NODE1_IP}"
+          TEST_RUN_FLAG=""
+          [ -n "${TEST_RUN}" ] && TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+          cleanup_remote() {
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" "$VM_HOST" \
+              "pkill -u tf-user -TERM -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; true" \
+              2>/dev/null || true
+          }
+          trap cleanup_remote EXIT INT TERM
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" "$VM_HOST" \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/spinifex-e2e-bin; \
+             printf '\n━━ suite %s ━━\n' \"${SUITE}\"; \
+             date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-${SUITE}.start'; \
+             GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
+               SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+               SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
+               SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
+               ARTIFACT_DIR='${ARTIFACT_DIR}' \
+               ./${SUITE}.test -test.v -test.timeout=30m ${TEST_RUN_FLAG} \
+               2>&1 | tee '${ARTIFACT_DIR}/test-${SUITE}.log' \
+                    | sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'; \
+             exit \${PIPESTATUS[0]}"
+          echo "rc=$?" >> "$GITHUB_OUTPUT"
+          exit 0
+
+      - name: Collect suite artifacts from node1
+        if: always()
+        run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${SSH_KEY}")
+          ssh "${SSH_OPTS[@]}" tf-user@${NODE1_IP} "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+          true
+
+      - name: Convert test log to JUnit XML
+        if: always()
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log); SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          title: "Multi-Node E2E ${{ matrix.suite }} results"
+
+      - name: Analyze e2e failures
+        if: always()
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "Multi-Node E2E ${{ matrix.suite }} failure analysis"
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-multi-${{ matrix.suite }}-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
+      - name: Fail job if suite failed
+        if: always()
+        run: |
+          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
+            echo "::error::suite ${{ matrix.suite }} failed (rc=${{ steps.e2e.outputs.rc }})"
+            exit 1
+          fi
+
+  multi_teardown:
+    name: Multi env · teardown
+    needs: [ multi_provision, multi_suite ]
+    if: always() && inputs.run_multinode
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 20
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-multi-td-${{ github.run_id }}.log
+      SSH_KEY: ${{ needs.multi_provision.outputs.ssh_key }}
+      NODES_CSV: ${{ needs.multi_provision.outputs.nodes_csv }}
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      # Reclaim leaked run-tagged resources before the cluster is destroyed.
+      - name: Sweep leaked run resources on node1
+        if: ${{ needs.multi_provision.result == 'success' && needs.multi_provision.outputs.node1_ip != '' }}
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}")
+          VM=tf-user@${{ needs.multi_provision.outputs.node1_ip }}
+          banner "sweep run-tagged resources (run ${{ github.run_id }})"
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "cd \$HOME/spinifex-e2e-bin 2>/dev/null && \
+             SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
+             SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+             ./teardown.test -test.v -test.run TestSweepRunResources -test.timeout=10m" \
+            2>&1 || true
+          true
+
+      - name: Teardown cluster
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          if [ -n "${NODES_CSV}" ] && [ -n "${SSH_KEY}" ]; then
+            IFS=',' read -ra NODE_LIST <<< "${NODES_CSV}"
+            for ip in "${NODE_LIST[@]}"; do
+              ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}" \
+                tf-user@"$ip" \
+                'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
+                2>/dev/null || true
+            done
+          fi
+          UID_SELF=$(id -u); WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in "$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;; esac
+            done
+          }
+          reap TERM; sleep 2; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
+          ok "teardown complete"
+
+  e2e_suites_status:
+    name: E2E suite blocks · summary
+    if: always()
+    needs: [ single_suite, multi_suite ]
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - name: Summarize
+        run: |
+          echo "## E2E suite-block results" >> "$GITHUB_STEP_SUMMARY"
+          echo "single_suite: ${{ needs.single_suite.result }}" >> "$GITHUB_STEP_SUMMARY"
+          echo "multi_suite:  ${{ needs.multi_suite.result }}" >> "$GITHUB_STEP_SUMMARY"
+          case "${{ needs.single_suite.result }}" in success|skipped) ;; *) exit 1 ;; esac
+          case "${{ needs.multi_suite.result }}" in success|skipped) ;; *) exit 1 ;; esac

.github/workflows/e2e.yml

@@ -18,7 +18,7 @@ on:
         description: "PR head SHA to report status on"
         required: false
       suites:
-        description: "Space-separated Make targets to run (e.g. 'e2e-single e2e-multinode e2e-cert e2e-lb')"
+        description: "Space-separated Make targets to run (e.g. 'e2e-single e2e-multinode e2e-cert e2e-lb e2e-eks')"
         required: false
         default: "e2e-single e2e-multinode e2e-cert e2e-lb"
       test_run:
@@ -30,27 +30,56 @@ permissions:
   contents: read
   statuses: write
 
-# Pushes to the same ref stack on the single `ci-single` self-hosted runner,
-# and a cancelled run can leave orphan `tofu`/`go test -c` procs behind that
-# wedge the runner host. Auto-cancel keeps at most one in-flight run per ref.
-concurrency:
-  group: e2e-${{ github.ref }}
-  cancel-in-progress: true
-
 env:
   GIT_BRANCH: ${{ github.ref_name }}
   SUITES: ${{ inputs.suites || 'e2e-single e2e-multinode e2e-cert e2e-lb' }}
   TEST_RUN: ${{ inputs.test_run || '' }}
 
 jobs:
+  # Shadow-mode suite selector. Computes which suites the diff *would* run and logs it to the job summary.
+  # Does NOT gate or override the real SUITES env yet — it runs alongside the full matrix so we
+  # can compare predicted vs actual before flipping it on.
+  select_suites:
+    name: Select Suites (shadow)
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: go.mod
+          cache: false
+      - name: Compute targeted suite selection
+        run: |
+          set -euo pipefail
+          BASE="${{ github.event.before }}"
+          HEAD="${{ github.sha }}"
+          # First push to a branch reports an all-zero before-SHA; fall back to
+          # the first parent so the diff is still meaningful.
+          if [ -z "$BASE" ] || [ "$BASE" = "0000000000000000000000000000000000000000" ]; then
+            BASE="${HEAD}~1"
+          fi
+          echo "## Targeted suite selection (shadow)" >> "$GITHUB_STEP_SUMMARY"
+          echo '```' >> "$GITHUB_STEP_SUMMARY"
+          echo "base=$BASE head=$HEAD" >> "$GITHUB_STEP_SUMMARY"
+          go run ./tests/e2e/select-suites/cmd/select-suites \
+            --manifest docs/service-interfaces.yaml \
+            --base-sha "$BASE" --head-sha "$HEAD" --repo-root . \
+            2>>"$GITHUB_STEP_SUMMARY" \
+            | tee -a "$GITHUB_STEP_SUMMARY"
+          echo '```' >> "$GITHUB_STEP_SUMMARY"
+          echo "Shadow only — actual run uses SUITES='${SUITES}'." >> "$GITHUB_STEP_SUMMARY"
+
   e2e_single:
     name: Single-Node E2E
     # Skip when the dispatch input requests no in-VM suites (e.g.
-    # suites='e2e-reboot' only). Push triggers and the default SUITES list
+    # suites='e2e-multinode' only). Push triggers and the default SUITES list
     # both include single/cert/lb so the inputs.suites null-check keeps the
     # job running on plain PR pushes.
-    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-lb')
-    runs-on: [ self-hosted, ci-single ]
+    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-lb') || contains(github.event.inputs.suites, 'e2e-eks')
+    runs-on: [self-hosted, ci-single]
     timeout-minutes: 45
     env:
       TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
@@ -77,7 +106,10 @@ jobs:
           true
 
       - name: Clean Workspace
-        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -105,10 +137,10 @@ jobs:
       - name: Re-enable Go toolchain auto-download
         run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
 
-      - name: Prune rootless docker cache
+      - name: Reclaim docker build cache
         run: |
           docker container prune -f || true
-          docker image prune -af || true
+          docker image prune -f || true
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
@@ -153,12 +185,90 @@ jobs:
           echo "wan_ip=${WAN_IP}" >> "$GITHUB_OUTPUT"
           echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
 
+      # The eks suite's CreateCluster boots a control-plane VM from the
+      # eks-node AMI, so that AMI must exist in the VM's image store before the
+      # suite runs. build-system-image.sh customizes the image with libguestfs
+      # (an isolated userspace appliance) — it touches no host block device, so
+      # it is safe on the shared runner. The build needs the spinifex source +
+      # network (runner only); the import needs predastore/NATS (VM only), so
+      # build on the runner then ship the raw and import on the VM.
+      - name: Build eks-node AMI (libguestfs, runner)
+        if: contains(env.SUITES, 'e2e-eks')
+        working-directory: spinifex
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "build eks-node AMI (libguestfs — no host devices)"
+          sudo apt-get update -qq
+          sudo apt-get install -y --no-install-recommends libguestfs-tools qemu-utils
+          # Debian/Ubuntu ship /boot/vmlinuz-* mode 0600; the libguestfs
+          # appliance needs to read a host kernel. direct backend = no libvirt.
+          export LIBGUESTFS_BACKEND=direct
+          sudo chmod 0644 /boot/vmlinuz-* || true
+          group_open "make build-eks-node-image"
+          make build-eks-node-image
+          group_close
+          ok "raw: $(du -h /tmp/eks-node-image-build/eks-node-alpine.raw | cut -f1)"
+
+      - name: Import eks-node AMI on VM
+        if: contains(env.SUITES, 'e2e-eks')
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          eval "$(grep -E '^(ALPINE_VERSION|AMI_NAME|SYSTEM_TAG)=' \
+            spinifex/scripts/images/eks-node/manifest.conf)"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          banner "import eks-node AMI to VM image store"
+          scp "${SSH_OPTS[@]}" \
+            /tmp/eks-node-image-build/eks-node-alpine.raw \
+            "$VM":/tmp/eks-node-alpine.raw
+          # Drop the import progress bar (pterm redraws to stdout — thousands of
+          # lines over the SSH pipe). Errors go to stderr, so they survive; the
+          # exit code still gates the step.
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "spx admin images import --file /tmp/eks-node-alpine.raw \
+               --distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
+               --boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}" >/dev/null
+          ok "eks-node AMI imported"
+
+      # The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
+      # `aws eks get-token` → kubectl → apiserver → token-webhook → gateway
+      # broker → STS verify path. The e2e binaries run ON the VM, so kubectl
+      # must be on the VM's PATH or harness.NewKubectl t.Skips the subtest.
+      # Pin to the k3s server version (setup.sh K3S_VERSION) — kubectl is
+      # ±1-minor skew tolerant; matching is simplest. Checksum-verified.
+      - name: Install kubectl on VM
+        if: contains(env.SUITES, 'e2e-eks')
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          banner "install kubectl on VM (eks data-plane subtest)"
+          ssh "${SSH_OPTS[@]}" "$VM" 'bash -s' <<'EOF'
+          set -euo pipefail
+          KVER=v1.32.5
+          case "$(uname -m)" in
+            x86_64)  ARCH=amd64 ;;
+            aarch64) ARCH=arm64 ;;
+            *) echo "unsupported arch $(uname -m)" >&2; exit 1 ;;
+          esac
+          cd /tmp
+          curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl"
+          curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl.sha256"
+          echo "$(cat kubectl.sha256)  kubectl" | sha256sum -c -
+          sudo install -m 0755 kubectl /usr/local/bin/kubectl
+          kubectl version --client
+          EOF
+          ok "kubectl installed on VM"
+
       - name: Build E2E test binaries on runner
         working-directory: spinifex/tests/e2e
         run: |
           set -euo pipefail
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          SUITES_BUILD=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(reboot|multinode)$' | tr '\n' ' ')
+          SUITES_BUILD=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(multinode)$' | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_BUILD}" | tr -d ' ')" ]; then
             echo "no in-VM suites requested (SUITES=${SUITES}); nothing to build"
             exit 0
@@ -200,7 +310,7 @@ jobs:
         id: e2e
         run: |
           set +e
-          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(reboot|multinode)$' | tr '\n' ' ')
+          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(multinode)$' | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
             echo "no in-VM suites requested (SUITES=${SUITES}); skipping"
             echo "rc=0" >> "$GITHUB_OUTPUT"
@@ -277,6 +387,15 @@ jobs:
           ssh "${SSH_OPTS[@]}" "$VM" 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "60 min ago"' \
             > "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null
           ok "journal: $(wc -l < "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || echo 0) lines"
+          # LB/system microVM serial consoles. The lb-agent logs its nginx (NLB)
+          # engine activation and any `nginx -t`/reload error + active-probe
+          # results to the guest serial console, which the host journal never
+          # captures. These files persist in /run/spinifex past VM teardown, so
+          # collecting them here makes an NLB "0/N healthy" timeout diagnosable.
+          mkdir -p "${ARTIFACT_DIR}/lb-console"
+          ssh "${SSH_OPTS[@]}" "$VM" 'sudo sh -c "cd /run/spinifex 2>/dev/null && tar -cf - console-*.log 2>/dev/null"' \
+            | tar -C "${ARTIFACT_DIR}/lb-console" -xf - 2>/dev/null
+          ok "lb-console: $(ls "${ARTIFACT_DIR}/lb-console" 2>/dev/null | wc -l) files"
 
       - name: Convert test logs to JUnit XML
         if: always()
@@ -356,7 +475,7 @@ jobs:
   e2e_real_multi:
     name: Real Multi-Node E2E
     if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-multinode')
-    runs-on: [ self-hosted, ci-multi ]
+    runs-on: [self-hosted, ci-multi]
     timeout-minutes: 60
     env:
       TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
@@ -383,7 +502,10 @@ jobs:
           true
 
       - name: Clean Workspace
-        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -411,10 +533,10 @@ jobs:
       - name: Re-enable Go toolchain auto-download
         run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
 
-      - name: Prune rootless docker cache
+      - name: Reclaim docker build cache
         run: |
           docker container prune -f || true
-          docker image prune -af || true
+          docker image prune -f || true
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
@@ -471,7 +593,12 @@ jobs:
         run: |
           set -euo pipefail
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          SUITES_BUILD=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(reboot|single)$' | tr '\n' ' ')
+          # eks is excluded on multinode: EKS KV is a clustered (Raft) JetStream
+          # stream and create-cluster needs an elected leader, but multinode
+          # JetStream clustering isn't solid yet (mulga-omq/z2z/5yg). EKS is
+          # validated single-node only for now; re-enable here once clustered
+          # JetStream + multinode EKS are a deliberate target.
+          SUITES_BUILD=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(single|eks)$' | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_BUILD}" | tr -d ' ')" ]; then
             echo "no multinode-eligible suites requested (SUITES=${SUITES}); nothing to build"
             exit 0
@@ -554,7 +681,7 @@ jobs:
         id: e2e
         run: |
           set +e
-          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(reboot|single)$' | tr '\n' ' ')
+          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(single|eks)$' | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
             echo "no multinode-eligible suites requested (SUITES=${SUITES}); skipping"
             echo "rc=0" >> "$GITHUB_OUTPUT"
@@ -664,6 +791,18 @@ jobs:
             ssh "${SSH_OPTS[@]}" tf-user@"$ip" 'sudo journalctl -u openvswitch-ipsec.service --no-pager --output=short-iso --since "60 min ago"' \
               > "${ARTIFACT_DIR}/openvswitch-ipsec-node${idx}.log" 2>/dev/null
           done
+          # LB/system microVM serial consoles, per node. The lb-agent logs its
+          # nginx (NLB) engine activation and any `nginx -t`/reload error +
+          # active-probe results to the guest serial console (never in the host
+          # journal). Files persist in /run/spinifex past VM teardown.
+          for i in "${!NODE_LIST[@]}"; do
+            idx=$((i + 1))
+            ip="${NODE_LIST[$i]}"
+            mkdir -p "${ARTIFACT_DIR}/lb-console/node${idx}"
+            ssh "${SSH_OPTS[@]}" tf-user@"$ip" 'sudo sh -c "cd /run/spinifex 2>/dev/null && tar -cf - console-*.log 2>/dev/null"' \
+              | tar -C "${ARTIFACT_DIR}/lb-console/node${idx}" -xf - 2>/dev/null
+            ok "lb-console node${idx}: $(ls "${ARTIFACT_DIR}/lb-console/node${idx}" 2>/dev/null | wc -l) files"
+          done
 
       - name: Convert test logs to JUnit XML
         if: always()
@@ -743,268 +882,11 @@ jobs:
             exit 1
           fi
 
-  e2e_reboot:
-    name: Reboot Resilience E2E
-    # Opt-in: only fires when the SUITES dispatch input lists e2e-reboot.
-    # The reboot test reboots the spinifex VM, so it must run on the runner
-    # (not shipped to the VM like the cert/lb/single suites). Runs in
-    # parallel with e2e_single — both target [self-hosted, ci-single]
-    # and GitHub Actions serialises on the underlying runner allocation,
-    # so no explicit needs: ordering is required.
-    if: contains(github.event.inputs.suites, 'e2e-reboot')
-    runs-on: [ self-hosted, ci-single ]
-    timeout-minutes: 60
-    env:
-      TOFU_LOG: /tmp/e2e-tofu-reboot-${{ github.run_id }}.log
-      ARTIFACT_DIR: /tmp/spinifex-e2e-reboot-artifacts-${{ github.run_id }}
-    steps:
-      - name: Reap Orphan Procs
-        run: |
-          set +e
-          UID_SELF=$(id -u)
-          WS="${GITHUB_WORKSPACE}"
-          reap() {
-            local sig="$1"
-            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
-              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
-              case "$cwd" in
-                "$WS"|"$WS"/*) echo "reap $sig pid=$pid cwd=$cwd"; kill "-$sig" "$pid" 2>/dev/null ;;
-              esac
-            done
-          }
-          reap TERM
-          sleep 3
-          reap KILL
-          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
-          true
-
-      - name: Clean Workspace
-        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
-
-      - uses: actions/create-github-app-token@v3
-        id: app-token
-        with:
-          client-id: ${{ secrets.APP_ID }}
-          private-key: ${{ secrets.APP_PRIVATE_KEY }}
-          repositories: mulga
-
-      - uses: actions/checkout@v6
-        with:
-          repository: mulgadc/mulga
-          token: ${{ steps.app-token.outputs.token }}
-          sparse-checkout: scripts/tofu-cluster
-          path: mulga
-
-      - uses: actions/checkout@v6
-        with:
-          path: spinifex
-
-      - uses: actions/setup-go@v6
-        with:
-          go-version-file: spinifex/go.mod
-          cache: false
-
-      - name: Re-enable Go toolchain auto-download
-        run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
-
-      - name: Prune rootless docker cache
-        run: |
-          docker container prune -f || true
-          docker image prune -af || true
-          docker buildx prune -af --filter "until=24h" || true
-          docker system df || true
-
-      - name: Build Install Artifacts
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          ./build-install-artifacts.sh "${GIT_BRANCH}" \
-            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
-            "${RUNNER_TEMP}/setup.sh" \
-            "${RUNNER_TEMP}/spinifex-tests.tar"
-
-      - name: Provision VM
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          set -euo pipefail
-          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          export TF_VAR_git_branch="${GIT_BRANCH}"
-          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
-          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
-          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
-          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
-          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
-          ok "VM provisioned"
-
-      - name: Bootstrap via Binary Install
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
-          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
-            "${RUNNER_TEMP}/setup.sh" \
-            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
-            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
-
-      - name: Resolve VM SSH details
-        id: vm
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          INVENTORY=$(tofu output -json inventory)
-          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
-          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
-          SSH_KEY="${SSH_KEY/#\~/$HOME}"
-          echo "wan_ip=${WAN_IP}" >> "$GITHUB_OUTPUT"
-          echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
-
-      - name: Build reboot.test on runner
-        working-directory: spinifex/tests/e2e
-        run: |
-          set -euo pipefail
-          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          banner "build reboot"
-          group_open "go test -c"
-          make e2e-build SUITES_BUILD="reboot"
-          group_close
-          ok "$(basename _bin/reboot.test) $(stat -c '%s' _bin/reboot.test | numfmt --to=iec)"
-
-      - name: Fetch AWS credentials from VM
-        id: creds
-        run: |
-          set -euo pipefail
-          SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
-                    -i "${{ steps.vm.outputs.ssh_key }}")
-          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
-          CREDS=$(ssh "${SSH_OPTS[@]}" "$VM" 'cat ~/.aws/credentials')
-          AWS_AK=$(echo "$CREDS" | awk -F '= *' '/^aws_access_key_id/ {print $2; exit}')
-          AWS_SK=$(echo "$CREDS" | awk -F '= *' '/^aws_secret_access_key/ {print $2; exit}')
-          if [ -z "$AWS_AK" ] || [ -z "$AWS_SK" ]; then
-            echo "::error::Failed to extract spinifex AWS credentials from ${{ steps.vm.outputs.wan_ip }}"
-            exit 1
-          fi
-          echo "::add-mask::${AWS_AK}"
-          echo "::add-mask::${AWS_SK}"
-          echo "ak=${AWS_AK}" >> "$GITHUB_OUTPUT"
-          echo "sk=${AWS_SK}" >> "$GITHUB_OUTPUT"
-
-      - name: Run reboot suite on runner
-        id: e2e
-        run: |
-          set +e
-          mkdir -p "${ARTIFACT_DIR}"
-          date -u +%Y-%m-%dT%H:%M:%SZ > "${ARTIFACT_DIR}/test-reboot.start"
-          R=$'\033[0m'; B=$'\033[1m'; C=$'\033[36m'
-          printf '\n%s%s━━ suite reboot ━━%s\n' "$B" "$C" "$R"
-
-          if [ -n "${TEST_RUN}" ]; then
-            TEST_RUN_FLAG="-test.run=${TEST_RUN}"
-          else
-            TEST_RUN_FLAG=""
-          fi
-          GITHUB_ACTIONS=true FORCE_COLOR=1 \
-            SPINIFEX_E2E=1 SPINIFEX_MODE=single \
-            SPINIFEX_WAN_IP="${{ steps.vm.outputs.wan_ip }}" \
-            SPINIFEX_SSH_KEY="${{ steps.vm.outputs.ssh_key }}" \
-            SPINIFEX_AWS_ENDPOINT="https://${{ steps.vm.outputs.wan_ip }}:9999" \
-            SPINIFEX_AWS_ACCESS_KEY_ID="${{ steps.creds.outputs.ak }}" \
-            SPINIFEX_AWS_SECRET_ACCESS_KEY="${{ steps.creds.outputs.sk }}" \
-            ARTIFACT_DIR="${ARTIFACT_DIR}" \
-            ./spinifex/tests/e2e/_bin/reboot.test -test.v -test.timeout=30m ${TEST_RUN_FLAG} \
-            2>&1 | tee "${ARTIFACT_DIR}/test-reboot.log" \
-                 | sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'
-          rc=${PIPESTATUS[0]}
-          echo "rc=${rc}" >> "$GITHUB_OUTPUT"
-          exit 0
-
-      - name: Collect VM journals
-        if: always()
-        run: |
-          set +e
-          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          mkdir -p "${ARTIFACT_DIR}"
-          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 \
-                    -i "${{ steps.vm.outputs.ssh_key }}")
-          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
-          ssh "${SSH_OPTS[@]}" "$VM" 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "60 min ago"' \
-            > "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || true
-          ok "journal: $(wc -l < "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || echo 0) lines"
-
-      - name: Install go-junit-report
-        if: always()
-        run: |
-          go install github.com/jstemmer/go-junit-report/v2@latest
-          echo "$HOME/go/bin" >> "$GITHUB_PATH"
-
-      - name: Convert test log to JUnit XML
-        if: always()
-        run: |
-          shopt -s nullglob
-          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
-            BASE=$(basename "$LOG" .log)
-            SUITE=${BASE#test-}
-            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
-          done
-
-      - name: Publish test summary
-        if: always()
-        uses: ./spinifex/.github/actions/junit-report
-        with:
-          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
-          title: "E2E reboot results"
-
-      - name: Analyze e2e failures
-        if: always()
-        uses: ./spinifex/.github/actions/e2e-analyze
-        with:
-          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
-          log-dir: ${{ env.ARTIFACT_DIR }}
-          title: "Reboot E2E failure analysis"
-
-      - uses: actions/upload-artifact@v7
-        if: always()
-        with:
-          name: e2e-reboot-artifacts-${{ github.run_id }}
-          path: ${{ env.ARTIFACT_DIR }}
-          if-no-files-found: warn
-          retention-days: 14
-
-      - name: Teardown
-        if: always()
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          set +e
-          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          UID_SELF=$(id -u)
-          WS="${GITHUB_WORKSPACE}"
-          reap() {
-            local sig="$1"
-            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
-              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
-              case "$cwd" in
-                "$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;;
-              esac
-            done
-          }
-          reap TERM
-          sleep 2
-          reap KILL
-          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
-          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
-          quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
-          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
-          ok "teardown complete"
-
-      - name: Fail job if reboot suite failed
-        if: always()
-        run: |
-          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
-            echo "::error::Reboot suite failed (rc=${{ steps.e2e.outputs.rc }})"
-            exit 1
-          fi
-
   e2e_status:
     name: Report E2E Status
     if: always()
-    needs: [ e2e_single, e2e_real_multi, e2e_reboot ]
-    runs-on: [ self-hosted, ci-single ]
+    needs: [e2e_single, e2e_real_multi]
+    runs-on: ubuntu-latest
     timeout-minutes: 5
     steps:
       - name: Report Status to PR
@@ -1016,16 +898,15 @@ jobs:
             echo "::error::no SHA available (inputs.sha and github.sha both empty)"
             exit 1
           fi
-          single_ok=true; multi_ok=true; reboot_ok=true
+          single_ok=true; multi_ok=true
           case "${{ needs.e2e_single.result }}" in success|skipped) ;; *) single_ok=false ;; esac
           case "${{ needs.e2e_real_multi.result }}" in success|skipped) ;; *) multi_ok=false ;; esac
-          case "${{ needs.e2e_reboot.result }}" in success|skipped) ;; *) reboot_ok=false ;; esac
-          if $single_ok && $multi_ok && $reboot_ok; then
+          if $single_ok && $multi_ok; then
             STATE="success"
             DESC="E2E suites passed"
           else
             STATE="failure"
-            DESC="E2E failed: single=${{ needs.e2e_single.result }} multi=${{ needs.e2e_real_multi.result }} reboot=${{ needs.e2e_reboot.result }}"
+            DESC="E2E failed: single=${{ needs.e2e_single.result }} multi=${{ needs.e2e_real_multi.result }}"
           fi
           gh api "repos/${{ github.repository }}/statuses/${REPORT_SHA}" \
             -f state="$STATE" \

.github/workflows/publish-eks-ami.yml

@@ -0,0 +1,96 @@
+name: Publish EKS node AMI
+
+# Builds the eks-node system AMI (libguestfs, on the runner) and publishes the
+# compressed qcow2 + .sha256 sidecar to the Cloudflare R2 bucket that backs
+# iso.mulgadc.com/system-ami/. Operators then install it on any cluster with:
+#   spx admin images import --name spinifex-eks-node
+#
+# Manual-only: publishing overwrites the mirror artifact, so it is a deliberate
+# action, not something every push should trigger.
+
+on:
+  workflow_dispatch:
+    inputs:
+      dry_run:
+        description: "Build + convert only; skip the R2 upload"
+        type: boolean
+        default: false
+
+permissions:
+  contents: read
+
+concurrency:
+  group: publish-eks-ami
+  cancel-in-progress: false
+
+jobs:
+  publish:
+    name: Build + publish eks-node AMI
+    runs-on: [ self-hosted, ci-single ]
+
+    steps:
+      - name: Checkout Spinifex
+        uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - name: Checkout Viperblock
+        uses: actions/checkout@v6
+        with:
+          repository: mulgadc/viperblock
+          ref: dev
+          path: viperblock
+
+      - name: Checkout Predastore
+        uses: actions/checkout@v6
+        with:
+          repository: mulgadc/predastore
+          ref: dev
+          path: predastore
+
+      - name: Setup Go
+        uses: actions/setup-go@v6
+        with:
+          go-version-file: "spinifex/go.mod"
+          # Self-hosted runner: the Go module + build cache already persists on
+          # local disk between runs. setup-go's cache-save ("Post Setup Go")
+          # otherwise re-uploads hundreds of MB to the Actions cache backend,
+          # taking longer than the build itself.
+          cache: false
+
+      - name: Setup Go Workspace
+        run: go work init ./spinifex ./viperblock ./predastore
+
+      - name: Install build + upload tooling
+        run: |
+          set -euo pipefail
+          sudo apt-get update -qq
+          sudo apt-get install -y --no-install-recommends libguestfs-tools qemu-utils
+          if ! command -v aws &>/dev/null; then
+            curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o /tmp/awscliv2.zip
+            unzip -q /tmp/awscliv2.zip -d /tmp/aws-install
+            sudo /tmp/aws-install/aws/install
+            rm -rf /tmp/awscliv2.zip /tmp/aws-install
+          fi
+
+      - name: Build + publish to R2
+        working-directory: spinifex
+        env:
+          # Map the repo's R2 secrets onto the names publish-system-image.sh reads.
+          AWS_ACCESS_KEY_ID: ${{ secrets.CF_R2_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.CF_R2_SECRET_ACCESS_KEY }}
+          R2_ENDPOINT: ${{ secrets.CF_R2_ENDPOINT_URL }}
+          R2_BUCKET: spinifex-iso
+          DRY_RUN: ${{ inputs.dry_run }}
+        run: |
+          set -euo pipefail
+          # libguestfs needs to read a host kernel; Ubuntu ships it mode 0600.
+          # direct backend = no libvirt on the runner.
+          export LIBGUESTFS_BACKEND=direct
+          sudo chmod 0644 /boot/vmlinuz-* || true
+          FLAGS=(--build)
+          if [[ "${DRY_RUN}" == "true" ]]; then
+            FLAGS+=(--dry-run)
+          fi
+          ./scripts/publish-system-image.sh \
+            scripts/images/eks-node/manifest.conf "${FLAGS[@]}"

.github/workflows/release.yml

@@ -34,7 +34,7 @@ jobs:
       - name: Set up Go
         uses: actions/setup-go@v6
         with:
-          go-version: '1.26.4'
+          go-version: "1.26.4"
 
       - name: Build microVM artifacts (lb-agent + kernel + initramfs)
         run: cd spinifex && make build-microvm-image
@@ -131,7 +131,10 @@ jobs:
       REPO: ${{ github.repository }}
     steps:
       - name: Clean Workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -169,8 +172,30 @@ jobs:
           TARBALL=$(find "${GITHUB_WORKSPACE}/release-assets" -name "*-linux-amd64.tar.gz" | head -1)
           ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" "$SETUP_SH" "$TARBALL"
 
+      - name: Teardown
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+
+  baremetal:
+    needs: release
+    permissions:
+      contents: read
+      statuses: write
+    uses: ./.github/workflows/e2e-baremetal.yml
+    secrets: inherit
+
+  promote:
+    needs: [ verify, baremetal ]
+    runs-on: ubuntu-24.04
+    env:
+      TAG: ${{ github.ref_name }}
+      REPO: ${{ github.repository }}
+    steps:
       - name: Promote Release
-        if: success()
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
@@ -182,41 +207,41 @@ jobs:
             echo "Release ${TAG} promoted to latest"
           fi
 
+  # Annotate the (still-draft) release if either gate failed.
+  mark-failed:
+    needs: [ verify, baremetal ]
+    if: failure()
+    runs-on: ubuntu-24.04
+    env:
+      TAG: ${{ github.ref_name }}
+      REPO: ${{ github.repository }}
+    steps:
       - name: Mark Release Failed
-        if: failure()
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
           EXISTING_BODY=$(gh release view "$TAG" --repo "$REPO" --json body -q .body)
           NOTES_FILE=$(mktemp)
-          printf 'Verification failed — see workflow run for details.\n\n%s' "$EXISTING_BODY" > "$NOTES_FILE"
+          printf 'Verification failed (tofu install or bare-metal boot) — see workflow run for details.\n\n%s' "$EXISTING_BODY" > "$NOTES_FILE"
           gh release edit "$TAG" --repo "$REPO" --notes-file "$NOTES_FILE"
           rm -f "$NOTES_FILE"
 
-      - name: Teardown
-        if: always()
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
-
   iso:
     needs: release
     runs-on: [ self-hosted, ci-single ]
-    timeout-minutes: 90
+    # ISO build (~60-90m) overlaps the parallel baremetal run (~120m); the
+    # upload gate then waits for baremetal to finish, so the job must outlast it.
+    timeout-minutes: 180
     steps:
       - name: Clean Workspace
         run: |
           ROOTFS="${GITHUB_WORKSPACE}/mulga/scripts/iso-builder/.cache/rootfs"
-          # Mirror make clean's umount sequence unconditionally — lazy unmount
-          # detaches immediately even if the mount is busy, so rm -rf is safe.
           sudo umount -l -R "${ROOTFS}" 2>/dev/null || true
           sudo umount -l "${ROOTFS}/dev/pts" 2>/dev/null || true
           sudo umount -l "${ROOTFS}/dev" 2>/dev/null || true
           sudo umount -l "${ROOTFS}/sys" 2>/dev/null || true
           sudo umount -l "${ROOTFS}/proc" 2>/dev/null || true
-          sudo rm -rf "$GITHUB_WORKSPACE"/*
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -289,24 +314,25 @@ jobs:
             SPINIFEX_BIN="${GITHUB_WORKSPACE}/spinifex/bin/spx" \
             INSTALLER_BIN="${GITHUB_WORKSPACE}/spinifex/bin/spinifex-installer"
 
-      - name: Wait for verify
+      - name: Wait for verify + baremetal
         id: wait-verify
         if: steps.build-iso.outcome == 'success'
         continue-on-error: true
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
-          echo "Waiting for verify job to complete..."
+          echo "Waiting for verify + baremetal jobs to complete..."
           while true; do
-            CONCLUSION=$(gh api \
-              "repos/${{ github.repository }}/actions/runs/${{ github.run_id }}/jobs" \
-              --jq '.jobs[] | select(.name == "verify") | .conclusion // empty')
-            if [ -n "$CONCLUSION" ]; then
-              echo "Verify conclusion: $CONCLUSION"
-              [ "$CONCLUSION" = "success" ] && exit 0
+            JOBS=$(gh api \
+              "repos/${{ github.repository }}/actions/runs/${{ github.run_id }}/jobs" --paginate)
+            VERIFY=$(echo "$JOBS" | jq -r '.jobs[] | select(.name == "verify") | .conclusion // empty')
+            BAREMETAL=$(echo "$JOBS" | jq -r '.jobs[] | select(.name | startswith("baremetal")) | .conclusion // empty')
+            if [ -n "$VERIFY" ] && [ -n "$BAREMETAL" ]; then
+              echo "verify=$VERIFY baremetal=$BAREMETAL"
+              { [ "$VERIFY" = "success" ] && [ "$BAREMETAL" = "success" ]; } && exit 0
               exit 1
             fi
-            echo "Verify still running, retrying in 30s..."
+            echo "Still waiting (verify=${VERIFY:-pending} baremetal=${BAREMETAL:-pending}), retry 30s..."
             sleep 30
           done
 
@@ -327,14 +353,13 @@ jobs:
           AWS_ACCESS_KEY_ID: ${{ secrets.CF_R2_ACCESS_KEY_ID }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.CF_R2_SECRET_ACCESS_KEY }}
           CF_R2_ENDPOINT_URL: ${{ secrets.CF_R2_ENDPOINT_URL }}
-          VERSION: ${{ github.ref_name }}
         run: |
           aws --endpoint-url="${CF_R2_ENDPOINT_URL}" \
-            s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso" \
-                  "s3://spinifex-iso/spinifex-${VERSION}.iso"
+            s3 cp "mulga/scripts/iso-builder/dist/spinifex.iso" \
+                  "s3://spinifex-iso/spinifex.iso"
           aws --endpoint-url="${CF_R2_ENDPOINT_URL}" \
-            s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso.sha256" \
-                  "s3://spinifex-iso/spinifex-${VERSION}.iso.sha256"
+            s3 cp "mulga/scripts/iso-builder/dist/spinifex.iso.sha256" \
+                  "s3://spinifex-iso/spinifex.iso.sha256"
 
       - name: Mark ISO failed
         if: always() && (steps.build-iso.outcome == 'failure' || steps.upload.outcome == 'failure')

.github/workflows/spinifex.yml

@@ -58,6 +58,9 @@ jobs:
       - name: Build Spinifex
         run: make -C spinifex build
 
+      - name: Manifest check + drift lint
+        run: make -C spinifex manifest-check manifest-lint
+
       - name: Run Tests with Coverage
         run: make -C spinifex test-cover
 

INSTALL.md

@@ -1,3 +0,0 @@
-# Spinifex Installation
-
-For documentation on using and installing Spinifex see [https://docs.mulgadc.com](https://docs.mulgadc.com/)

Makefile

@@ -6,17 +6,12 @@ ARCH := $(shell uname -m)
 ifeq ($(ARCH),x86_64)
   GO_ARCH := amd64
   AWS_ARCH := x86_64
-  # On x86, we can run ARM VMs via emulation
-  QEMU_PACKAGES := qemu-system-x86 qemu-system-arm
 else ifeq ($(ARCH),aarch64)
   GO_ARCH := arm64
   AWS_ARCH := aarch64
-  # On ARM, we can run x86 VMs via emulation
-  QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
 else ifeq ($(ARCH),arm64)
   GO_ARCH := arm64
   AWS_ARCH := aarch64
-  QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
 else
   $(error Unsupported architecture: $(ARCH). Only x86_64 and aarch64/arm64 are supported.)
 endif
@@ -73,24 +68,27 @@ build-lb-agent:
 	@echo -e "\n....Building lb-agent (static)"
 	CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
 
-build-system-image: ## Build a system image from manifest (use IMAGE=lb or IMAGE=eks-server)
+build-system-image: ## Build a system image from manifest (use IMAGE=lb or IMAGE=eks-node)
 ifndef IMAGE
 	$(error IMAGE is required. Usage: make build-system-image IMAGE=lb)
 endif
 	@if [ -f scripts/images/$(IMAGE).conf ]; then \
-		./scripts/build-system-image.sh scripts/images/$(IMAGE).conf; \
+		./scripts/build-system-image.sh scripts/images/$(IMAGE).conf $(if $(IMPORT),--import); \
 	elif [ -f scripts/images/$(IMAGE)/manifest.conf ]; then \
-		./scripts/build-system-image.sh scripts/images/$(IMAGE)/manifest.conf; \
+		./scripts/build-system-image.sh scripts/images/$(IMAGE)/manifest.conf $(if $(IMPORT),--import); \
 	else \
 		echo "ERROR: no manifest at scripts/images/$(IMAGE).conf or scripts/images/$(IMAGE)/manifest.conf"; \
 		exit 1; \
 	fi
 
-build-eks-server-image: ## Build the eks-server AMI (K3s control-plane variant)
-	$(MAKE) build-system-image IMAGE=eks-server
+build-eks-node-image: ## Build the unified eks-node AMI (K3s server+agent; role at first boot; IMPORT=1 to register)
+	$(MAKE) build-system-image IMAGE=eks-node
 
-build-eks-agent-image: ## Build the eks-agent AMI (K3s worker variant)
-	$(MAKE) build-system-image IMAGE=eks-agent
+import-eks-node-image: ## Build + register the eks-node AMI (requires a running cluster)
+	$(MAKE) build-system-image IMAGE=eks-node IMPORT=1
+
+publish-eks-node-image: ## Build + publish the eks-node AMI to Cloudflare R2 (needs R2_ENDPOINT + AWS_* env)
+	./scripts/publish-system-image.sh scripts/images/eks-node/manifest.conf --build
 
 MICROVM_OUT_DIR := build/microvm
 MICROVM_ARTIFACTS := $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz
@@ -112,14 +110,10 @@ install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
 
-go_run:
-	@echo -e "\n....Running $(GO_PROJECT_NAME)...."
-	$(GOPATH)/bin/$(GO_PROJECT_NAME)
-
 # Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
 # Use this before committing to catch CI failures locally.
 preflight:
-	@$(MAKE) --no-print-directory QUIET=1 manifest-check lint govulncheck test-cover diff-coverage test-race test-harness
+	@$(MAKE) --no-print-directory QUIET=1 manifest-check manifest-lint lint govulncheck test-cover diff-coverage test-race test-harness
 	@echo -e "\n ✅ Preflight passed — safe to commit."
 
 # E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
@@ -136,6 +130,17 @@ manifest-check:
 	@echo -e "\n....Checking service-interfaces.yaml...."
 	@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml
 
+# Drift guards (Bead 5): direct-create fixture lint + NATS subject lint,
+# ratcheted against tests/e2e/manifest-lint/baseline.txt. Fails only on NEW
+# drift beyond the baseline.
+manifest-lint:
+	@echo -e "\n....Linting manifest drift (fixtures + subjects)...."
+	@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root .
+
+# Accept current drift into the baseline. Run after an intentional change.
+manifest-lint-update:
+	@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root . -update
+
 # Run unit tests
 test:
 	@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
@@ -165,13 +170,8 @@ diff-coverage: test-cover
 
 bench:
 	@echo -e "\n....Running benchmarks for $(GO_PROJECT_NAME)...."
-	$(MAKE) easyjson
 	LOG_IGNORE=1 go test -benchmem -run=. -bench=. ./...
 
-run:
-	$(MAKE) go_build
-	$(MAKE) go_run
-
 # Fast iteration: build + install binary + restart all services.
 # Microvm artifacts are reinstalled when they already exist on disk — the rule's
 # input timestamps drive a rebuild only if anything actually changed. On a fresh
@@ -198,11 +198,10 @@ clean:
 
 install-system:
 	@echo -e "\n....Installing system dependencies for $(ARCH)...."
-	@echo "QEMU packages: $(QEMU_PACKAGES)"
 	sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
 		-o Dpkg::Options::="--force-confdef" \
 		-o Dpkg::Options::="--force-confold" \
-		nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
+		nbdkit nbdkit-plugin-dev pkg-config qemu-system-x86 qemu-system-arm qemu-utils qemu-kvm \
 		ovmf qemu-efi-aarch64 \
 		libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
 		iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
@@ -231,19 +230,15 @@ install-aws:
 quickinstall: install-system install-go install-aws
 	@echo -e "\n✅ Quickinstall complete for $(ARCH)."
 	@echo "   Please ensure /usr/local/go/bin is in your PATH."
-	@echo "   Installed: Go ($(GO_ARCH)), AWS CLI ($(AWS_ARCH)), QEMU ($(QEMU_PACKAGES))"
 
-# Lint all Go code via golangci-lint (replaces check-format, vet, gosec, staticcheck)
 lint:
 	@echo "Running golangci-lint..."
 	$(_Q)golangci-lint run ./...
 	@echo "  golangci-lint ok"
 
-# Auto-fix all linter issues that have fixers
 fix:
 	golangci-lint run --fix ./...
 
-# Govulncheck — dependency vulnerability scanning (not covered by golangci-lint)
 govulncheck:
 	@echo "Running govulncheck..."
 	$(_Q)go tool govulncheck ./...
@@ -292,55 +287,8 @@ distro-arm64:
 distro-clean:
 	rm -rf dist/
 
-# Ansible dev lifecycle (experimental, parallel to dev-*.sh / reset-dev-env.sh).
-# See scripts/ansible/README.md and docs/development/improvements/ansible-dev-lifecycle.md.
-#
-# Variable overrides: pass EXTRA_VARS="key=val key2=val2" to forward as
-# `--extra-vars` to ansible-playbook. Plain `-e` on the make command line is
-# make's own flag and does NOT reach ansible.
-#   make ansible-dev-install EXTRA_VARS="spinifex_external_pool_start=192.168.1.90 spinifex_external_pool_end=192.168.1.99"
-EXTRA_VARS ?=
-_ANSIBLE_EXTRA = $(if $(strip $(EXTRA_VARS)),--extra-vars "$(EXTRA_VARS)",)
-
-ansible-dev-preflight:
-	cd scripts/ansible && ansible-playbook playbooks/dev-preflight.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-teardown:
-	cd scripts/ansible && ansible-playbook playbooks/dev-teardown.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-install:
-	cd scripts/ansible && ansible-playbook playbooks/dev-install.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-reset:
-	cd scripts/ansible && ansible-playbook playbooks/dev-reset.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-deploy:
-	cd scripts/ansible && ansible-playbook playbooks/dev-deploy.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-status:
-	cd scripts/ansible && ansible-playbook playbooks/dev-status.yml
-
-ansible-dev-logs:
-	cd scripts/ansible && ansible-playbook playbooks/dev-logs.yml
-
-# Snapshot / restore require an explicit -e snapshot_name=<name>. The
-# wrapper passes ANSIBLE_EXTRA through so devs can run
-# `make ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=before-merge'`.
-ansible-dev-snapshot:
-	cd scripts/ansible && ansible-playbook playbooks/dev-snapshot.yml $(ANSIBLE_EXTRA)
-
-ansible-dev-restore:
-	cd scripts/ansible && ansible-playbook playbooks/dev-restore.yml $(ANSIBLE_EXTRA)
-
-ansible-dev-version:
-	cd scripts/ansible && ansible-playbook playbooks/dev-version.yml
-
-ansible-dev-vpc:
-	cd scripts/ansible && ansible-playbook playbooks/dev-vpc.yml $(ANSIBLE_EXTRA)
-
-.PHONY: build build-ui build-installer build-lb-agent build-system-image build-eks-server-image build-eks-agent-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race diff-coverage bench run test-actions test-harness manifest-check diff-coverage bench run \
+.PHONY: build build-ui build-installer build-lb-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-harness manifest-check manifest-lint manifest-lint-update \
 	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
 	lint fix govulncheck \
-	distro distro-amd64 distro-arm64 distro-clean \
-	ansible-dev-preflight ansible-dev-teardown ansible-dev-install ansible-dev-reset ansible-dev-deploy ansible-dev-status ansible-dev-logs ansible-dev-snapshot ansible-dev-restore ansible-dev-version ansible-dev-vpc
+	distro distro-amd64 distro-arm64 distro-clean

build/microvm/init.sh

@@ -6,20 +6,58 @@ mount -t devtmpfs dev /dev
 # virtio_mmio is built-in (CONFIG_VIRTIO_MMIO=y) so virtio-net devices are
 # already enumerated by the time PID 1 runs — no modprobe needed.
 # qemu_fw_cfg and virtio_net are =m in stock Alpine linux-virt, so load them.
-modprobe qemu_fw_cfg 2>/dev/null || modprobe fw_cfg_sysfs 2>/dev/null || true
-modprobe virtio_net 2>/dev/null || true
+#
+# modprobe needs a usable modules.dep; when the image is built on a host whose
+# depmod can't produce one, busybox modprobe silently no-ops and the module
+# never loads — fw_cfg then never appears and init panics. insmod by explicit
+# path needs no modules.dep, so fall back to it. Handles .ko and .ko.gz.
+load_mod() {
+    modprobe "$1" 2>/dev/null && return 0
+    for ko in $(find /lib/modules -name "$1.ko" -o -name "$1.ko.gz" 2>/dev/null); do
+        insmod "$ko" 2>/dev/null && return 0
+        case "$ko" in
+            *.gz) gunzip -c "$ko" >/tmp/mod.ko 2>/dev/null \
+                  && insmod /tmp/mod.ko 2>/dev/null && return 0 ;;
+        esac
+    done
+    return 1
+}
+load_mod qemu_fw_cfg || load_mod fw_cfg_sysfs || true
+# virtio_net depends on net_failover, which depends on failover. The load_mod
+# insmod fallback loads a single .ko with no dependency resolution, and
+# modules.dep is absent when the image is built on a kmod-less host (depmod
+# skipped), so modprobe can't resolve the chain either. Load it explicitly
+# bottom-up so the NIC comes up regardless of how the image was built —
+# without this the agent has no network and the LB hangs in provisioning.
+load_mod failover || true
+load_mod net_failover || true
+load_mod virtio_net || true
 
-for i in 1 2 3 4 5; do
-    [ -f /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw ] && break
-    sleep 0.02
+NETCFG=/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw
+i=0
+while [ "$i" -lt 25 ]; do
+    [ -f "$NETCFG" ] && break
+    sleep 0.04
+    i=$((i + 1))
 done
 
-if [ ! -f /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw ]; then
+if [ ! -f "$NETCFG" ]; then
+    # Dump host/guest fw_cfg state to the serial console before init exits so a
+    # "netcfg not available" panic is diagnosable from the captured console log
+    # (module-not-loaded vs qemu-not-exposing-by_name vs wrong machine type).
     echo "[init] FATAL: fw_cfg netcfg not available" >&2
+    echo "[init] dmi: vendor=$(cat /sys/class/dmi/id/sys_vendor 2>/dev/null) product=$(cat /sys/class/dmi/id/product_name 2>/dev/null) bios=$(cat /sys/class/dmi/id/bios_version 2>/dev/null)" >&2
+    echo "[init] qemu_fw_cfg dir present: $([ -d /sys/firmware/qemu_fw_cfg ] && echo yes || echo NO)" >&2
+    echo "[init] modules: $(lsmod 2>/dev/null | awk 'NR>1{print $1}' | tr '\n' ',')" >&2
+    echo "[init] by_name listing:" >&2
+    ls -la /sys/firmware/qemu_fw_cfg/by_name/ 2>&1 | sed 's/^/[init]   /' >&2
+    ls -la /sys/firmware/qemu_fw_cfg/by_name/opt/ 2>&1 | sed 's/^/[init]   /' >&2
+    echo "[init] dmesg fw_cfg:" >&2
+    dmesg 2>/dev/null | grep -i fw_cfg | sed 's/^/[init]   /' >&2
     exit 1
 fi
 
-. /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw
+. "$NETCFG"
 
 for n in 0 1 2 3 4 5; do
     eval mac=\$NIC${n}_MAC

build/systemd/spinifex-awsgw.service

@@ -6,6 +6,7 @@ Wants=spinifex-nats.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-gw
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh

build/systemd/spinifex-daemon.service

@@ -2,66 +2,29 @@
 Description=Spinifex Daemon (VM Orchestration)
 PartOf=spinifex.target
 After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.service ovn-controller.service openvswitch-ipsec.service
-# Wants=openvswitch-ipsec.service so systemd brings the IPsec userspace
-# daemon up before spinifex flips other_config:ipsec_encapsulation=true.
-# Without that daemon, OVS holds the cert pointers unused and strongSwan
-# never gets IKE config — tunnel traffic is encrypted with no SAs and
-# silently dropped. Wants= (not Requires=) keeps daemon bootable on
-# single-node / lab hosts where the package is intentionally absent.
+# Wants (not Requires): stay bootable when openvswitch-ipsec is absent
 Wants=openvswitch-ipsec.service
-# No Wants=spinifex-nats.service: Wants= reactivates a stopped unit during
-# `systemctl restart spinifex-daemon`, which would mask the standalone-startup
-# path the Tier 1 DDIL Scenario B exercises. spinifex.target Wants= covers
-# cold boot.
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-daemon
 Group=spinifex
-# DDIL Tier 1 (siv-17): no ExecStartPre wait-for-nats. The daemon starts in
-# standalone mode and reconnects to NATS in the background — process-exiting
-# on a missing NATS would kill the local VM management plane.
-#
-# Opt-in strict-startup (§1d-strict, siv-44): set SPINIFEX_REQUIRE_NATS=1 to
-# restore pre-DDIL fail-fast UX. First NATS connect uses a ~30s bounded retry
-# and the daemon aborts on timeout. Intended for dev/test/single-node deploys
-# where silent NATS misconfig is more confusing than helpful. Do NOT set this
-# in the shipped unit — it re-introduces the SPOF that Tier 1 removed.
 ExecStart=/usr/local/bin/spx service spinifex start
 Restart=on-failure
 RestartSec=5
-# DDIL Tier 1: KillMode=process so systemd only signals the
-# daemon's main PID, not the cgroup. QEMU children stay alive across
-# `systemctl restart spinifex-daemon` and the new daemon reattaches to them
-# via the persisted local state file + pidfile + QMP socket. With the
-# default control-group mode, restart would terminate every guest VM —
-# defeating Scenario B's "daemon restart preserves running instances".
+# KillMode=process: guest QEMU survives daemon restart; the new daemon reattaches (DDIL)
 KillMode=process
-# 15s upper bound on the SIGTERM grace window. The new SIGTERM handler
-# skips VM teardown (it now just persists state and exits), so 15s is
-# plenty of slack for the cluster-manager + NATS unsubscribe path.
 TimeoutStopSec=15
+# RG-4: infra tier — reaped only after every guest
 OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
 Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex/
 Environment=SPINIFEX_WAL_DIR=/var/lib/spinifex/spinifex/
-# DDIL Tier 1: QEMU pidfiles, QMP sockets, and the
-# shared qemu.log all resolve via utils.pidPath() — which falls back
-# to os.TempDir() (=/tmp) when $HOME/spinifex doesn't exist. With
-# PrivateTmp=yes, each daemon start gets a fresh /tmp namespace, so
-# pidfiles written by QEMU (running in the OLD daemon's namespace)
-# are invisible to the new daemon. The /local/instances handler then
-# reports PID=0 and Scenario B fails its "live QEMU PID" assertion.
-# Pin XDG_RUNTIME_DIR to the host-wide tmpfs at /run/spinifex (created
-# by tmpfiles.d, already in ReadWritePaths) so pidPath() returns a
-# stable, namespace-independent path that both daemon incarnations and
-# QEMU children resolve identically.
+# Pin to host tmpfs: QEMU pidfiles/QMP sockets must survive PrivateTmp namespace churn (DDIL)
 Environment=XDG_RUNTIME_DIR=/run/spinifex
 
-# Filesystem access — NoNewPrivileges omitted (daemon uses sudo for ip, ovs-vsctl, dhcpcd)
-# CAP_SYS_ADMIN: required for GPU passthrough sysfs operations (vfio group management).
-# CAP_DAC_OVERRIDE: sysfs driver bind/unbind paths (/sys/bus/pci/drivers/vfio-pci/bind,
-# unbind) are root:root 0200; non-root processes need CAP_DAC_OVERRIDE to open them.
+# RG-9: daemon tier — privileged by necessity (GPU vfio). NoNewPrivileges omitted (RG-10: sudo ip/ovs-vsctl/dhcpcd)
 AmbientCapabilities=CAP_SYS_ADMIN CAP_DAC_OVERRIDE
 ProtectSystem=full
 PrivateDevices=no
@@ -73,7 +36,7 @@ DeviceAllow=/dev/vfio/vfio rw
 ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
 ReadWritePaths=/var/lib/spinifex/spinifex /var/log/spinifex /run/spinifex /run/spinifex/nbd
 
-# Hardening
+# RG-9: hardening baseline
 ProtectHome=yes
 PrivateTmp=yes
 RestrictSUIDSGID=yes
@@ -88,8 +51,6 @@ ProtectControlGroups=yes
 ProtectProc=invisible
 SystemCallArchitectures=native
 MemoryDenyWriteExecute=yes
-# DHCP lease acquisition moved to spinifex-vpcd (see dhcp-allocate-error.md);
-# the daemon no longer needs AF_PACKET or the softened kernel protections.
 RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
 ProtectHostname=yes
 KeyringMode=private
@@ -97,8 +58,7 @@ RemoveIPC=yes
 LimitCORE=0
 LimitNOFILE=65536
 LimitNPROC=4096
-# VFIO GPU passthrough requires locking all guest RAM into physical memory.
-# Without this, QEMU fails with RLIMIT_MEMLOCK exceeded when pinning guest pages.
+# LimitMEMLOCK=infinity: VFIO pins all guest RAM
 LimitMEMLOCK=infinity
 
 [Install]

build/systemd/spinifex-guests.slice

@@ -0,0 +1,8 @@
+[Unit]
+Description=Spinifex guest VM tier
+Before=slices.target
+
+[Slice]
+# RG-3: child of spinifex.slice (spinifex-guests -> spinifex), sibling of
+# spinifex-system. Parent for the per-instance spinifex-instance-<id>.slice
+# leaves (0002 RG-7). Inherits the spinifex.slice ceiling; no extra limit here.

build/systemd/spinifex-nats.service

@@ -5,6 +5,7 @@ Before=spinifex-predastore.service spinifex-daemon.service spinifex-awsgw.servic
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-nats
 Group=spinifex
 ExecStart=/usr/local/bin/spx service nats start

build/systemd/spinifex-predastore.service

@@ -6,6 +6,7 @@ Wants=spinifex-nats.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-storage
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh

build/systemd/spinifex-system.slice

@@ -0,0 +1,9 @@
+[Unit]
+Description=Spinifex control/infra tier (NATS, predastore, viperblock, daemon, awsgw, vpcd, ui)
+Before=slices.target
+
+[Slice]
+# RG-3: child of spinifex.slice (spinifex-system -> spinifex), inherits the
+# parent ceiling, no extra limit. Holds the 7 control/infra services. Guest QEMU
+# moves to the sibling spinifex-guests.slice (per-instance placement lands in
+# 0002 RG-7); until then the daemon-forked guests stay under spinifex-daemon.service.

build/systemd/spinifex-ui.service

@@ -5,6 +5,7 @@ After=spinifex-awsgw.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-ui
 Group=spinifex
 ExecStart=/usr/local/bin/spx service spinifex-ui start

build/systemd/spinifex-viperblock.service

@@ -6,6 +6,7 @@ Wants=spinifex-nats.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-viperblock
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh

build/systemd/spinifex-vpcd.service

@@ -6,6 +6,7 @@ Wants=spinifex-nats.service ovn-controller.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-vpcd
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
@@ -16,37 +17,25 @@ OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
 Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/vpcd/
 
-# Filesystem access — NoNewPrivileges omitted (vpcd uses sudo for ovs-vsctl, ovn-nbctl, systemctl)
-ProtectSystem=full
+# RG-9: network tier. vpcd MUST share the host mount namespace — IMDS enters
+# per-VPC netns via setns(), and any filesystem-sandbox directive (ProtectSystem/
+# Home, PrivateTmp, ProtectKernel*, ProtectControlGroups/Proc, Read*Paths) forks a
+# private mount ns that traps the /run/netns bind-mount and breaks setns(2). Those,
+# plus MemoryDenyWriteExecute, are omitted by design — not regressions.
 PrivateDevices=no
-ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/ca.pem
-ReadWritePaths=/var/lib/spinifex/vpcd /var/log/spinifex
-
-# Hardening
-ProtectHome=yes
-PrivateTmp=yes
 RestrictSUIDSGID=no
 LockPersonality=yes
 RestrictRealtime=yes
-RestrictNamespaces=yes
 ProtectClock=yes
-ProtectKernelTunables=yes
-ProtectKernelModules=yes
-ProtectKernelLogs=yes
-ProtectControlGroups=yes
-ProtectProc=invisible
+# mnt alongside net: in-netns `ip` unshares a mount ns for its own /sys
+RestrictNamespaces=net mnt
 SystemCallArchitectures=native
 MemoryDenyWriteExecute=no
-# AF_PACKET: required by the in-process DHCP client (github.com/insomniacslk/dhcp
-# / nclient4) for raw L2 BOOTP on br-wan before an IP is configured on the
-# interface. See docs/development/bugs/dhcp-allocate-error.md.
+# AF_PACKET: raw L2 BOOTP for the in-process DHCP client before the iface has an IP
 RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK AF_PACKET
-# CAP_NET_RAW: open AF_PACKET sockets.
-# CAP_NET_BIND_SERVICE: covers unicast RENEW to UDP :68.
-# CAP_NET_ADMIN: already used for ovs-vsctl / ip link wiring on the bridge.
-# Note, disabled for now, sudo issue with ovs
-AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID
-CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID
+# Caps: NET_ADMIN (ovs/ip), NET_RAW (AF_PACKET), NET_BIND_SERVICE (:68), SETUID/SETGID, SYS_ADMIN (setns into VPC netns)
+AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID CAP_SYS_ADMIN
+CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID CAP_SYS_ADMIN
 NoNewPrivileges=no
 ProtectHostname=yes
 KeyringMode=private

build/systemd/spinifex.slice

@@ -0,0 +1,13 @@
+[Unit]
+Description=Spinifex resource-bounded slice (host ceiling)
+Before=slices.target
+
+[Slice]
+# RG-1: no CPUWeight — slice roots are siblings; capping below the default would
+# let user.slice out-prioritize the VM fleet. Operator CPU priority lives on
+# system.slice (system.slice.d/spinifex-reserve.conf).
+# RG-1/RG-5: throttle+reclaim at High, hard wall at Max (% of physical RAM).
+MemoryHigh=80%
+MemoryMax=90%
+# RG-5: spill to the full swap device before the MemoryMax wall; matched to swap size.
+MemorySwapMax=8G

cmd/eks-gateway-fetch/main.go

@@ -0,0 +1,139 @@
+// eks-gateway-fetch runs inside the EKS K3s control-plane VM and fetches a
+// control-plane resource from the host through the AWS gateway, instead of
+// dialing core NATS directly. It is the read-side companion to
+// eks-gateway-publish.
+//
+// It SigV4-signs (service "eks") an HTTPS GET to the gateway, retrying with
+// backoff until the gateway returns 2xx or the attempt budget is exhausted (so
+// a degraded link surfaces as a non-zero exit), then emits the staged add-ons
+// as tab-separated lines the on-VM addon-sync shell agent consumes without a
+// JSON parser:
+//
+//	<addonName>\t<addonVersion>\t<serviceAccountRoleArn>\t<base64(configurationValues)>
+//
+// The configuration values are base64-encoded so embedded tabs/newlines cannot
+// break the line framing.
+//
+// Usage:
+//
+//	eks-gateway-fetch -resource addons   # GET /clusters/{cluster}/internal-addons/{accountId}
+//
+// Flags default to environment variables seeded by cloud-init:
+// EKS_GATEWAY_URL, EKS_GATEWAY_CA, EKS_ACCESS_KEY, EKS_SECRET_KEY, EKS_REGION,
+// EKS_ACCOUNT_ID, EKS_CLUSTER_NAME.
+package main
+
+import (
+	"bufio"
+	"encoding/base64"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"log/slog"
+	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/eksgw"
+
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
+)
+
+// stagedAddon mirrors the gateway's StagedAddonManifest wire shape; duplicated
+// here to keep this tiny VM binary free of the handler package's dependency
+// graph.
+type stagedAddon struct {
+	AddonName             string `json:"addonName"`
+	AddonVersion          string `json:"addonVersion"`
+	ServiceAccountRoleArn string `json:"serviceAccountRoleArn"`
+	ConfigurationValues   string `json:"configurationValues"`
+}
+
+type internalAddonsResponse struct {
+	Addons []stagedAddon `json:"addons"`
+}
+
+const (
+	maxAttempts = 30
+	retryDelay  = 5 * time.Second
+)
+
+func main() {
+	var (
+		gatewayURL  string
+		gatewayCA   string
+		accessKey   string
+		secretKey   string
+		region      string
+		accountID   string
+		clusterName string
+		resource    string
+	)
+
+	flag.StringVar(&gatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
+	flag.StringVar(&gatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
+	flag.StringVar(&accessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
+	flag.StringVar(&secretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
+	flag.StringVar(&region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
+	flag.StringVar(&accountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
+	flag.StringVar(&clusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
+	flag.StringVar(&resource, "resource", "addons", "Resource to fetch: addons")
+	flag.Parse()
+
+	switch {
+	case accountID == "":
+		fatal("--account-id is required (or set EKS_ACCOUNT_ID)")
+	case clusterName == "":
+		fatal("--cluster is required (or set EKS_CLUSTER_NAME)")
+	case resource != "addons":
+		fatal("--resource must be addons")
+	}
+
+	client, err := eksgw.New(gatewayURL, gatewayCA, accessKey, secretKey, region)
+	if err != nil {
+		fatal(fmt.Sprintf("build gateway client: %v", err))
+	}
+	path := "/clusters/" + clusterName + "/internal-addons/" + accountID
+
+	var lastErr error
+	for attempt := 1; attempt <= maxAttempts; attempt++ {
+		body, err := client.Get(path)
+		if err != nil {
+			lastErr = err
+			slog.Warn("eks-gateway-fetch: attempt failed", "resource", resource, "attempt", attempt, "err", err)
+			if attempt < maxAttempts {
+				time.Sleep(retryDelay)
+			}
+			continue
+		}
+		if werr := emitAddonsTSV(os.Stdout, body); werr != nil {
+			fatal(fmt.Sprintf("render response: %v", werr))
+		}
+		return
+	}
+	fatal(fmt.Sprintf("fetch failed after %d attempts: %v", maxAttempts, lastErr))
+}
+
+// emitAddonsTSV decodes the internal-addons response and writes one TSV line per
+// staged add-on to w. An empty add-on set writes nothing (the agent then GCs
+// every locally-rendered manifest), which is the correct steady state for a
+// cluster with no managed add-ons.
+func emitAddonsTSV(out io.Writer, body []byte) error {
+	var resp internalAddonsResponse
+	if err := json.Unmarshal(body, &resp); err != nil {
+		return fmt.Errorf("unmarshal internal-addons response: %w", err)
+	}
+	w := bufio.NewWriter(out)
+	for _, a := range resp.Addons {
+		cfg := base64.StdEncoding.EncodeToString([]byte(a.ConfigurationValues))
+		if _, err := fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", a.AddonName, a.AddonVersion, a.ServiceAccountRoleArn, cfg); err != nil {
+			return err
+		}
+	}
+	return w.Flush()
+}
+
+func fatal(msg string) {
+	slog.Error("eks-gateway-fetch: " + msg)
+	os.Exit(1)
+}

cmd/eks-gateway-fetch/main_test.go

@@ -0,0 +1,44 @@
+package main
+
+import (
+	"bytes"
+	"encoding/base64"
+	"testing"
+)
+
+func TestEmitAddonsTSV(t *testing.T) {
+	body := []byte(`{"addons":[
+		{"addonName":"spinifex-noop","addonVersion":"0.1.0"},
+		{"addonName":"aws-load-balancer-controller","addonVersion":"2.8.1",
+		 "serviceAccountRoleArn":"arn:aws:iam::111122223333:role/alb",
+		 "configurationValues":"{\"replicaCount\":2}"}
+	]}`)
+
+	var buf bytes.Buffer
+	if err := emitAddonsTSV(&buf, body); err != nil {
+		t.Fatalf("emitAddonsTSV: %v", err)
+	}
+	cfg := base64.StdEncoding.EncodeToString([]byte(`{"replicaCount":2}`))
+	want := "spinifex-noop\t0.1.0\t\t\n" +
+		"aws-load-balancer-controller\t2.8.1\tarn:aws:iam::111122223333:role/alb\t" + cfg + "\n"
+	if buf.String() != want {
+		t.Fatalf("TSV mismatch:\n got: %q\nwant: %q", buf.String(), want)
+	}
+}
+
+func TestEmitAddonsTSV_Empty(t *testing.T) {
+	var buf bytes.Buffer
+	if err := emitAddonsTSV(&buf, []byte(`{"addons":[]}`)); err != nil {
+		t.Fatalf("emitAddonsTSV: %v", err)
+	}
+	if buf.Len() != 0 {
+		t.Fatalf("empty addon set must emit nothing, got %q", buf.String())
+	}
+}
+
+func TestEmitAddonsTSV_BadJSON(t *testing.T) {
+	var buf bytes.Buffer
+	if err := emitAddonsTSV(&buf, []byte(`{not-json`)); err == nil {
+		t.Fatal("expected error on malformed JSON")
+	}
+}

cmd/eks-gateway-publish/main.go

@@ -0,0 +1,134 @@
+// eks-gateway-publish runs inside the EKS K3s control-plane VM and relays a
+// single publication to the host through the AWS gateway, instead of dialing
+// core NATS directly.
+//
+// It reads a JSON payload on stdin, wraps it as
+// {accountId, channel, kind, payload}, SigV4-signs (service "eks") an HTTPS
+// POST to {gateway}/clusters/{cluster}/internal-publish, and retries with
+// backoff until the gateway returns 2xx or the attempt budget is exhausted —
+// so a degraded link surfaces as a non-zero exit rather than a silently
+// dropped message (the failure mode of fire-and-forget `nats pub`).
+//
+// Usage:
+//
+//	echo '{"token":"..."}' | eks-gateway-publish -channel bootstrap -kind k3s-bootstrap-token
+//	kubectl ... | eks-gateway-publish -channel state
+//
+// Flags default to environment variables seeded by cloud-init:
+// EKS_GATEWAY_URL, EKS_GATEWAY_CA, EKS_ACCESS_KEY, EKS_SECRET_KEY, EKS_REGION,
+// EKS_ACCOUNT_ID, EKS_CLUSTER_NAME.
+package main
+
+import (
+	"bytes"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"log/slog"
+	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/eksgw"
+
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
+)
+
+const (
+	// Retry budget: the control plane reaches readiness before this runs, so a
+	// failing POST means a degraded link, not a cold start. Bounded so a stuck
+	// boot still terminates the OpenRC service.
+	maxAttempts = 30
+	retryDelay  = 5 * time.Second
+)
+
+type publishBody struct {
+	AccountID string          `json:"accountId"`
+	Channel   string          `json:"channel"`
+	Kind      string          `json:"kind"`
+	Payload   json.RawMessage `json:"payload"`
+}
+
+func main() {
+	var (
+		gatewayURL  string
+		gatewayCA   string
+		accessKey   string
+		secretKey   string
+		region      string
+		accountID   string
+		clusterName string
+		channel     string
+		kind        string
+	)
+
+	flag.StringVar(&gatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
+	flag.StringVar(&gatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
+	flag.StringVar(&accessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
+	flag.StringVar(&secretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
+	flag.StringVar(&region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
+	flag.StringVar(&accountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
+	flag.StringVar(&clusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
+	flag.StringVar(&channel, "channel", "", "Publish channel: bootstrap|state|addon")
+	flag.StringVar(&kind, "kind", "", "Bootstrap subject kind (bootstrap channel only)")
+	flag.Parse()
+
+	switch {
+	case accountID == "":
+		fatal("--account-id is required (or set EKS_ACCOUNT_ID)")
+	case clusterName == "":
+		fatal("--cluster is required (or set EKS_CLUSTER_NAME)")
+	case channel != "bootstrap" && channel != "state" && channel != "addon":
+		fatal("--channel must be bootstrap, state or addon")
+	case channel == "bootstrap" && kind == "":
+		fatal("--kind is required for the bootstrap channel")
+	}
+
+	payload, err := io.ReadAll(os.Stdin)
+	if err != nil {
+		fatal(fmt.Sprintf("read stdin payload: %v", err))
+	}
+	if len(bytes.TrimSpace(payload)) == 0 {
+		fatal("empty stdin payload")
+	}
+	if !json.Valid(payload) {
+		fatal("stdin payload is not valid JSON")
+	}
+
+	body, err := json.Marshal(publishBody{
+		AccountID: accountID,
+		Channel:   channel,
+		Kind:      kind,
+		Payload:   json.RawMessage(payload),
+	})
+	if err != nil {
+		fatal(fmt.Sprintf("marshal request body: %v", err))
+	}
+
+	client, err := eksgw.New(gatewayURL, gatewayCA, accessKey, secretKey, region)
+	if err != nil {
+		fatal(fmt.Sprintf("build gateway client: %v", err))
+	}
+	path := "/clusters/" + clusterName + "/internal-publish"
+
+	var lastErr error
+	for attempt := 1; attempt <= maxAttempts; attempt++ {
+		if _, err := client.Post(path, body); err != nil {
+			lastErr = err
+			slog.Warn("eks-gateway-publish: attempt failed",
+				"channel", channel, "kind", kind, "attempt", attempt, "err", err)
+			if attempt < maxAttempts {
+				time.Sleep(retryDelay)
+			}
+			continue
+		}
+		slog.Info("eks-gateway-publish: published", "channel", channel, "kind", kind)
+		return
+	}
+	fatal(fmt.Sprintf("publish failed after %d attempts: %v", maxAttempts, lastErr))
+}
+
+func fatal(msg string) {
+	slog.Error("eks-gateway-publish: " + msg)
+	os.Exit(1)
+}

cmd/eks-token-webhook/cert.go

@@ -0,0 +1,137 @@
+package main
+
+import (
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/tls"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/base64"
+	"encoding/pem"
+	"fmt"
+	"math/big"
+	"net"
+	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/tlsconfig"
+)
+
+// tlsCertificate is the webhook's loopback serving certificate.
+type tlsCertificate struct {
+	cert tls.Certificate
+}
+
+func (c tlsCertificate) serverTLSConfig() *tls.Config {
+	return &tls.Config{
+		Certificates:     []tls.Certificate{c.cert},
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
+	}
+}
+
+// ensureServingCert loads the persisted self-signed serving cert/key, or mints
+// a fresh ECDSA-P256 pair (SAN 127.0.0.1 / localhost) and persists it. Reusing
+// across restarts keeps the CA the apiserver loaded into its webhook kubeconfig
+// valid even if the webhook process is restarted. Returns the parsed cert and
+// the certificate PEM (for embedding as the kubeconfig CA bundle).
+func ensureServingCert(certPath, keyPath string) (tlsCertificate, []byte, error) {
+	if certPEM, keyPEM, ok := readCertPair(certPath, keyPath); ok {
+		cert, err := tls.X509KeyPair(certPEM, keyPEM)
+		if err == nil {
+			return tlsCertificate{cert: cert}, certPEM, nil
+		}
+		// Fall through to regenerate on a corrupt/incompatible persisted pair.
+	}
+
+	certPEM, keyPEM, err := generateSelfSigned()
+	if err != nil {
+		return tlsCertificate{}, nil, err
+	}
+	if err := os.WriteFile(certPath, certPEM, 0o644); err != nil {
+		return tlsCertificate{}, nil, fmt.Errorf("write cert %s: %w", certPath, err)
+	}
+	if err := os.WriteFile(keyPath, keyPEM, 0o600); err != nil {
+		return tlsCertificate{}, nil, fmt.Errorf("write key %s: %w", keyPath, err)
+	}
+	cert, err := tls.X509KeyPair(certPEM, keyPEM)
+	if err != nil {
+		return tlsCertificate{}, nil, fmt.Errorf("parse minted keypair: %w", err)
+	}
+	return tlsCertificate{cert: cert}, certPEM, nil
+}
+
+func readCertPair(certPath, keyPath string) (certPEM, keyPEM []byte, ok bool) {
+	c, err := os.ReadFile(certPath)
+	if err != nil {
+		return nil, nil, false
+	}
+	k, err := os.ReadFile(keyPath)
+	if err != nil {
+		return nil, nil, false
+	}
+	return c, k, true
+}
+
+func generateSelfSigned() (certPEM, keyPEM []byte, err error) {
+	priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		return nil, nil, fmt.Errorf("generate key: %w", err)
+	}
+	serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
+	if err != nil {
+		return nil, nil, fmt.Errorf("serial: %w", err)
+	}
+	now := time.Now().UTC()
+	tmpl := x509.Certificate{
+		SerialNumber:          serial,
+		Subject:               pkix.Name{CommonName: "eks-token-webhook"},
+		NotBefore:             now.Add(-time.Hour),
+		NotAfter:              now.AddDate(10, 0, 0),
+		KeyUsage:              x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment | x509.KeyUsageCertSign,
+		ExtKeyUsage:           []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+		BasicConstraintsValid: true,
+		IsCA:                  true,
+		IPAddresses:           []net.IP{net.IPv4(127, 0, 0, 1), net.IPv6loopback},
+		DNSNames:              []string{"localhost"},
+	}
+	der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &priv.PublicKey, priv)
+	if err != nil {
+		return nil, nil, fmt.Errorf("create cert: %w", err)
+	}
+	keyDER, err := x509.MarshalECPrivateKey(priv)
+	if err != nil {
+		return nil, nil, fmt.Errorf("marshal key: %w", err)
+	}
+	certPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
+	keyPEM = pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
+	return certPEM, keyPEM, nil
+}
+
+// writeAPIServerKubeconfig writes the kubeconfig kube-apiserver loads via
+// --authentication-token-webhook-config-file. It points the apiserver at the
+// loopback webhook and trusts its self-signed serving cert as the CA bundle.
+func writeAPIServerKubeconfig(path, addr string, certPEM []byte) error {
+	caB64 := base64.StdEncoding.EncodeToString(certPEM)
+	kubeconfig := fmt.Sprintf(`apiVersion: v1
+kind: Config
+clusters:
+- name: mulga-eks-token-webhook
+  cluster:
+    certificate-authority-data: %s
+    server: https://%s/authenticate
+users:
+- name: kube-apiserver
+contexts:
+- name: webhook
+  context:
+    cluster: mulga-eks-token-webhook
+    user: kube-apiserver
+current-context: webhook
+`, caB64, addr)
+	if err := os.WriteFile(path, []byte(kubeconfig), 0o600); err != nil {
+		return fmt.Errorf("write %s: %w", path, err)
+	}
+	return nil
+}

cmd/eks-token-webhook/main.go

@@ -1,38 +1,227 @@
 // eks-token-webhook is the K8s TokenReview webhook authenticator baked into
-// the eks-server AMI. It decodes the SigV4-presigned GetCallerIdentity URL
-// produced by `aws eks get-token`, enforces the X-K8s-Aws-Id cross-cluster
-// pin, calls Mulga STS over NATS, and resolves the principal to a TokenReview
-// response via the cluster's AccessEntry KV.
+// the eks-server AMI. Per kubectl call the kube-apiserver POSTs the bearer
+// token (produced by `aws eks get-token`); the webhook relays it through the
+// AWS gateway broker (SigV4 HTTPS POST to /clusters/{name}/token-review), which
+// host-side runs the STS verify (cross-cluster x-k8s-aws-id pin) + AccessEntry
+// KV lookup and returns the resolved identity. The webhook never speaks core
+// NATS — same gateway-broker model as ELBv2's lb-agent and eks-gateway-publish.
 //
-// This is a placeholder build that returns 503 Service Unavailable on every
-// path. It exists so the eks-server AMI build succeeds (INSTALL_BINARIES is
-// satisfied) and the K3s server VM boots cleanly. Real implementation lands
-// with bead mulga-cs-eks-6c.
+// It binds 127.0.0.1 only; the kube-apiserver is the sole client (loopback).
+// On startup it writes the apiserver webhook kubeconfig (with its self-signed
+// serving CA) so k3s can be pointed at it via
+// --authentication-token-webhook-config-file.
 package main
 
 import (
+	"encoding/json"
+	"errors"
 	"flag"
+	"fmt"
 	"log/slog"
 	"net/http"
 	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/eksgw"
+	handlers_eks "github.com/mulgadc/spinifex/spinifex/handlers/eks"
 )
 
+// config is the webhook's runtime configuration, sourced from the first-boot
+// env (/etc/spinifex-eks/first-boot.env) cloud-init seeds onto the VM.
+type config struct {
+	addr        string
+	gatewayURL  string
+	gatewayCA   string
+	accessKey   string
+	secretKey   string
+	region      string
+	accountID   string
+	clusterName string
+	certPath    string
+	keyPath     string
+	kubeconfig  string
+}
+
+func loadConfig() (config, error) {
+	c := config{
+		addr:        flagAddr,
+		gatewayURL:  os.Getenv("EKS_GATEWAY_URL"),
+		gatewayCA:   os.Getenv("EKS_GATEWAY_CA"),
+		accessKey:   os.Getenv("EKS_ACCESS_KEY"),
+		secretKey:   os.Getenv("EKS_SECRET_KEY"),
+		region:      os.Getenv("EKS_REGION"),
+		accountID:   os.Getenv("EKS_ACCOUNT_ID"),
+		clusterName: os.Getenv("EKS_CLUSTER_NAME"),
+		certPath:    envOr("EKS_WEBHOOK_CERT", "/etc/spinifex-eks/token-webhook.crt"),
+		keyPath:     envOr("EKS_WEBHOOK_KEY", "/etc/spinifex-eks/token-webhook.key"),
+		kubeconfig:  envOr("EKS_WEBHOOK_KUBECONFIG", "/etc/spinifex-eks/token-webhook.kubeconfig"),
+	}
+	switch {
+	case c.gatewayURL == "":
+		return c, errors.New("EKS_GATEWAY_URL not set")
+	case c.accessKey == "" || c.secretKey == "":
+		return c, errors.New("EKS_ACCESS_KEY / EKS_SECRET_KEY not set")
+	case c.accountID == "":
+		return c, errors.New("EKS_ACCOUNT_ID not set")
+	case c.clusterName == "":
+		return c, errors.New("EKS_CLUSTER_NAME not set")
+	}
+	return c, nil
+}
+
+func envOr(key, def string) string {
+	if v := os.Getenv(key); v != "" {
+		return v
+	}
+	return def
+}
+
+var flagAddr string
+
 func main() {
-	addr := flag.String("addr", "127.0.0.1:8443", "listen address")
+	flag.StringVar(&flagAddr, "addr", "127.0.0.1:8443", "listen address (loopback only)")
 	flag.Parse()
 
 	slog.SetDefault(slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelInfo})))
 
-	mux := http.NewServeMux()
-	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
-		w.Header().Set("Retry-After", "30")
-		w.Header().Set("Content-Type", "application/json")
-		http.Error(w, `{"error":"eks-token-webhook stub: real implementation not yet deployed"}`, http.StatusServiceUnavailable)
-	})
+	cfg, err := loadConfig()
+	if err != nil {
+		slog.Error("eks-token-webhook config error", "err", err)
+		os.Exit(1)
+	}
 
-	slog.Info("eks-token-webhook stub starting", "addr", *addr)
-	if err := http.ListenAndServe(*addr, mux); err != nil {
-		slog.Error("listen failed", "err", err)
+	if err := run(cfg); err != nil {
+		slog.Error("eks-token-webhook fatal", "err", err)
 		os.Exit(1)
 	}
 }
+
+func run(cfg config) error {
+	// Serving cert + apiserver kubeconfig. Persisted and reused across restarts
+	// so a webhook crash does not invalidate the CA the apiserver already loaded.
+	tlsCert, certPEM, err := ensureServingCert(cfg.certPath, cfg.keyPath)
+	if err != nil {
+		return fmt.Errorf("serving cert: %w", err)
+	}
+	if err := writeAPIServerKubeconfig(cfg.kubeconfig, cfg.addr, certPEM); err != nil {
+		return fmt.Errorf("write apiserver kubeconfig: %w", err)
+	}
+
+	client, err := eksgw.New(cfg.gatewayURL, cfg.gatewayCA, cfg.accessKey, cfg.secretKey, cfg.region)
+	if err != nil {
+		return fmt.Errorf("build gateway client: %w", err)
+	}
+
+	authr := &authenticator{
+		accountID:   cfg.accountID,
+		clusterName: cfg.clusterName,
+		review:      gatewayReviewer(client, cfg.clusterName, cfg.accountID),
+	}
+
+	mux := http.NewServeMux()
+	mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("ok"))
+	})
+	mux.HandleFunc("/authenticate", authr.handle)
+
+	server := newServer(cfg.addr, mux, tlsCert)
+	slog.Info("eks-token-webhook listening", "addr", cfg.addr, "cluster", cfg.clusterName)
+	if err := server.ListenAndServeTLS("", ""); err != nil && !errors.Is(err, http.ErrServerClosed) {
+		return fmt.Errorf("serve: %w", err)
+	}
+	return nil
+}
+
+// tokenReviewRequest is the body POSTed to /clusters/{name}/token-review. The
+// webhook holds system SigV4 creds (system account, not the cluster account),
+// so accountId names the cluster account explicitly — same as PublishInternal.
+type tokenReviewRequest struct {
+	AccountID string `json:"accountId"`
+	Token     string `json:"token"`
+}
+
+// gatewayReviewer returns the per-call relay: SigV4-POST {accountId,token} to
+// the gateway and decode the resolved identity. A non-2xx / transport error is
+// surfaced (handle() turns it into a 5xx so the apiserver retries) rather than a
+// silent deny, distinguishing "webhook broker down" from "token rejected".
+func gatewayReviewer(client *eksgw.Client, clusterName, accountID string) func(string) (handlers_eks.WebhookTokenReviewResult, error) {
+	path := "/clusters/" + clusterName + "/token-review"
+	return func(token string) (handlers_eks.WebhookTokenReviewResult, error) {
+		body, err := json.Marshal(tokenReviewRequest{AccountID: accountID, Token: token})
+		if err != nil {
+			return handlers_eks.WebhookTokenReviewResult{}, fmt.Errorf("marshal request: %w", err)
+		}
+		respBody, err := client.Post(path, body)
+		if err != nil {
+			return handlers_eks.WebhookTokenReviewResult{}, err
+		}
+		var res handlers_eks.WebhookTokenReviewResult
+		if err := json.Unmarshal(respBody, &res); err != nil {
+			return handlers_eks.WebhookTokenReviewResult{}, fmt.Errorf("decode response: %w", err)
+		}
+		return res, nil
+	}
+}
+
+// authenticator resolves a TokenReview by relaying to the gateway broker.
+type authenticator struct {
+	accountID   string
+	clusterName string
+	review      func(token string) (handlers_eks.WebhookTokenReviewResult, error)
+}
+
+func (a *authenticator) handle(w http.ResponseWriter, r *http.Request) {
+	if r.Method != http.MethodPost {
+		http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
+		return
+	}
+	var review tokenReview
+	if err := json.NewDecoder(r.Body).Decode(&review); err != nil {
+		http.Error(w, "bad request", http.StatusBadRequest)
+		return
+	}
+
+	res, err := a.review(review.Spec.Token)
+	if err != nil {
+		// Broker fault (gateway down, transport error): a 5xx tells the apiserver
+		// to retry rather than treating a transient outage as a hard deny.
+		slog.Error("TokenReview relay failed", "cluster", a.clusterName, "err", err)
+		http.Error(w, "token review unavailable", http.StatusServiceUnavailable)
+		return
+	}
+
+	status := tokenReviewStatus{Authenticated: res.Authenticated}
+	if res.Authenticated {
+		status.User = userInfo{
+			Username: res.Username,
+			UID:      res.UID,
+			Groups:   res.Groups,
+		}
+		// Confirm the token for the audiences the apiserver asked about; an empty
+		// status.audiences is rejected when --api-audiences is configured.
+		status.Audiences = review.Spec.Audiences
+	}
+	slog.Info("TokenReview decision", "authenticated", status.Authenticated, "username", status.User.Username, "audiences", status.Audiences)
+
+	resp := tokenReview{
+		APIVersion: "authentication.k8s.io/v1",
+		Kind:       "TokenReview",
+		Status:     status,
+	}
+	w.Header().Set("Content-Type", "application/json")
+	if err := json.NewEncoder(w).Encode(resp); err != nil {
+		slog.Error("encode TokenReview response", "err", err)
+	}
+}
+
+// newServer builds the loopback TLS server. Split out so tests can construct it
+// without binding a port.
+func newServer(addr string, handler http.Handler, cert tlsCertificate) *http.Server {
+	return &http.Server{
+		Addr:              addr,
+		Handler:           handler,
+		ReadHeaderTimeout: 5 * time.Second,
+		TLSConfig:         cert.serverTLSConfig(),
+	}
+}

cmd/eks-token-webhook/main_test.go

@@ -0,0 +1,123 @@
+package main
+
+import (
+	"encoding/base64"
+	"encoding/json"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	handlers_eks "github.com/mulgadc/spinifex/spinifex/handlers/eks"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// postReview drives authenticator.handle with a TokenReview carrying token and
+// audiences, returning the decoded response status and HTTP code.
+func postReview(t *testing.T, a *authenticator, token string, audiences []string) (tokenReviewStatus, int) {
+	t.Helper()
+	body, err := json.Marshal(tokenReview{
+		APIVersion: "authentication.k8s.io/v1",
+		Kind:       "TokenReview",
+		Spec:       tokenReviewSpec{Token: token, Audiences: audiences},
+	})
+	require.NoError(t, err)
+
+	req := httptest.NewRequest(http.MethodPost, "/authenticate", strings.NewReader(string(body)))
+	w := httptest.NewRecorder()
+	a.handle(w, req)
+
+	if w.Code != http.StatusOK {
+		return tokenReviewStatus{}, w.Code
+	}
+	var resp tokenReview
+	require.NoError(t, json.Unmarshal(w.Body.Bytes(), &resp))
+	return resp.Status, w.Code
+}
+
+func TestHandle_GrantsAndStampsAudiences(t *testing.T) {
+	a := &authenticator{
+		accountID:   "111122223333",
+		clusterName: "alpha",
+		review: func(string) (handlers_eks.WebhookTokenReviewResult, error) {
+			return handlers_eks.WebhookTokenReviewResult{
+				Authenticated: true,
+				Username:      "arn:aws:iam::111122223333:role/admin",
+				UID:           "AROAEXAMPLE:session",
+				Groups:        []string{"system:masters"},
+			}, nil
+		},
+	}
+
+	status, code := postReview(t, a, "tok", []string{"https://kubernetes.default.svc"})
+	require.Equal(t, http.StatusOK, code)
+	require.True(t, status.Authenticated)
+	assert.Equal(t, "arn:aws:iam::111122223333:role/admin", status.User.Username)
+	assert.Equal(t, "AROAEXAMPLE:session", status.User.UID)
+	assert.Equal(t, []string{"system:masters"}, status.User.Groups)
+	assert.Equal(t, []string{"https://kubernetes.default.svc"}, status.Audiences)
+}
+
+func TestHandle_DeniesWithoutStampingAudiences(t *testing.T) {
+	a := &authenticator{
+		clusterName: "alpha",
+		review: func(string) (handlers_eks.WebhookTokenReviewResult, error) {
+			return handlers_eks.WebhookTokenReviewResult{Authenticated: false}, nil
+		},
+	}
+
+	status, code := postReview(t, a, "tok", []string{"https://kubernetes.default.svc"})
+	require.Equal(t, http.StatusOK, code)
+	assert.False(t, status.Authenticated)
+	assert.Empty(t, status.User.Username)
+	assert.Empty(t, status.Audiences, "unauthenticated review must not stamp audiences")
+}
+
+// A broker fault is a 5xx (retryable), not a silent deny — a transient gateway
+// outage must not look identical to a rejected token.
+func TestHandle_BrokerFaultReturns5xx(t *testing.T) {
+	a := &authenticator{
+		clusterName: "alpha",
+		review: func(string) (handlers_eks.WebhookTokenReviewResult, error) {
+			return handlers_eks.WebhookTokenReviewResult{}, errors.New("gateway returned 503")
+		},
+	}
+
+	_, code := postReview(t, a, "tok", nil)
+	assert.Equal(t, http.StatusServiceUnavailable, code)
+}
+
+func TestEnsureServingCert_PersistsAndReuses(t *testing.T) {
+	dir := t.TempDir()
+	certPath := filepath.Join(dir, "wh.crt")
+	keyPath := filepath.Join(dir, "wh.key")
+
+	c1, pem1, err := ensureServingCert(certPath, keyPath)
+	require.NoError(t, err)
+	require.NotEmpty(t, pem1)
+	require.NotNil(t, c1.serverTLSConfig())
+
+	// Second call reuses the persisted pair (same cert bytes).
+	_, pem2, err := ensureServingCert(certPath, keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, pem1, pem2)
+}
+
+func TestWriteAPIServerKubeconfig(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "wh.kubeconfig")
+	certPEM := []byte("-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----\n")
+
+	require.NoError(t, writeAPIServerKubeconfig(path, "127.0.0.1:8443", certPEM))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	body := string(data)
+	assert.Contains(t, body, "server: https://127.0.0.1:8443/authenticate")
+	assert.Contains(t, body, base64.StdEncoding.EncodeToString(certPEM))
+	assert.Contains(t, body, "current-context: webhook")
+}

cmd/eks-token-webhook/tokenreview.go

@@ -0,0 +1,34 @@
+package main
+
+// Minimal authentication.k8s.io/v1 TokenReview wire types. Hand-rolled to avoid
+// pulling k8s.io/api into the spinifex module for these few fields; the shapes
+// match what kube-apiserver sends and expects on the webhook contract.
+
+type tokenReview struct {
+	APIVersion string            `json:"apiVersion"`
+	Kind       string            `json:"kind"`
+	Spec       tokenReviewSpec   `json:"spec,omitzero"`
+	Status     tokenReviewStatus `json:"status"`
+}
+
+type tokenReviewSpec struct {
+	Token     string   `json:"token"`
+	Audiences []string `json:"audiences,omitempty"`
+}
+
+type tokenReviewStatus struct {
+	Authenticated bool     `json:"authenticated"`
+	User          userInfo `json:"user,omitzero"`
+	// Audiences echoes the request's spec.audiences on success. When the
+	// apiserver runs with --api-audiences, it sends spec.audiences and discards
+	// an authenticated=true response that does not confirm a matching audience —
+	// so an empty Audiences here surfaces as a 401 despite a valid principal.
+	Audiences []string `json:"audiences,omitempty"`
+	Error     string   `json:"error,omitempty"`
+}
+
+type userInfo struct {
+	Username string   `json:"username,omitempty"`
+	UID      string   `json:"uid,omitempty"`
+	Groups   []string `json:"groups,omitempty"`
+}

cmd/spinifex/cmd/admin.go

@@ -115,6 +115,16 @@ Each phase waits for all nodes to ACK before proceeding to the next.`,
 	Run: runClusterShutdown,
 }
 
+var clusterDrainDHCPCmd = &cobra.Command{
+	Use:   "drain-dhcp",
+	Short: "Release all upstream DHCP leases held by vpcd",
+	Long: `Ask each vpcd to DHCPRELEASE every external-pool DHCP lease it currently
+holds, returning them to the upstream DHCP server. Run this on teardown before
+stopping services: an env reset otherwise strands held leases on the upstream
+server until their TTL expires, eventually exhausting the upstream scope.`,
+	Run: runClusterDrainDHCP,
+}
+
 var adminInitCmd = &cobra.Command{
 	Use:   "init",
 	Short: "Initialize Spinifex platform configuration",
@@ -245,6 +255,9 @@ func init() {
 	clusterShutdownCmd.Flags().Duration("timeout", 120*time.Second, "Maximum time to wait per phase")
 	clusterShutdownCmd.Flags().Bool("dry-run", false, "Print phase plan without executing")
 
+	clusterCmd.AddCommand(clusterDrainDHCPCmd)
+	clusterDrainDHCPCmd.Flags().Duration("timeout", 30*time.Second, "Reply-collection window for vpcd drain responders")
+
 	adminCmd.AddCommand(imagesCmd)
 	imagesCmd.AddCommand(imagesImportCmd)
 	imagesCmd.AddCommand(imagesListCmd)
@@ -325,6 +338,7 @@ func init() {
 	imagesImportCmd.Flags().String("tmp-dir", os.TempDir(), "Temporary directory for image import processing")
 
 	imagesImportCmd.Flags().String("name", "", "Import specified image by name")
+	imagesImportCmd.Flags().String("ami-name", "", "Override the registered AMI name (DescribeImages name). Defaults to ami-{distro}-{version}-{arch}. Use for locally-built appliances (e.g. --ami-name spinifex-eks-node).")
 	imagesImportCmd.Flags().String("file", "", "Import file from specified path (raw, qcow2, compressed)")
 	imagesImportCmd.Flags().String("distro", "", "Specified distro name (e.g debian)")
 	imagesImportCmd.Flags().String("version", "", "Specified distro version (e.g 12)")
@@ -380,6 +394,7 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	// and the file is used directly (no download). When only --file is set,
 	// metadata comes from flags and no catalog tags are applied.
 	imageName, _ := cmd.Flags().GetString("name")
+	amiNameOverride, _ := cmd.Flags().GetString("ami-name")
 	localFile, _ := cmd.Flags().GetString("file")
 
 	if imageName == "" && localFile == "" {
@@ -535,6 +550,9 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	// Calculate the size
 
 	manifest.AMIMetadata.Name = fmt.Sprintf("ami-%s-%s-%s", image.Distro, image.Version, image.Arch)
+	if amiNameOverride != "" {
+		manifest.AMIMetadata.Name = amiNameOverride
+	}
 	volumeId := utils.GenerateResourceID("ami")
 	manifest.AMIMetadata.ImageID = volumeId
 
@@ -1109,6 +1127,17 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	fmt.Println("\n🔐 Generated predastore encryption key (per-node, never transmitted)")
 	fmt.Printf("   Key: %s\n", predastoreKeyPath)
 
+	// Viperblock at-rest encryption key is cluster-wide; on a single-node init
+	// there are no joiners, so generate it locally and enable encryption by
+	// default for all volumes created on this install.
+	viperblockKeyPath, err := writeViperblockEncryptionKey(configDir)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error generating viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	fmt.Println("\n🔐 Generated viperblock at-rest encryption key")
+	fmt.Printf("   Key: %s\n", viperblockKeyPath)
+
 	fmt.Printf("\n🔑 Generated admin credentials (save these — they won't be shown again):\n")
 	fmt.Printf("   Access Key:  %s\n", bootstrapResult.AdminAccessKey)
 	fmt.Printf("   Secret Key:  %s\n", bootstrapResult.AdminSecretKey)
@@ -1296,6 +1325,8 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 
 		GPUPassthrough: gpuPassthrough,
 		IPSecEnabled:   ipsecEnabled,
+
+		EncryptionKeyFile: viperblockKeyPath,
 	}
 
 	// Print external networking summary
@@ -1404,6 +1435,22 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	fmt.Println("\n🔐 Generated predastore encryption key (per-node, never transmitted)")
 	fmt.Printf("   Key: %s\n", predastoreKeyPath)
 
+	// Viperblock at-rest encryption key is cluster-wide and shared: the leader
+	// generates it once and distributes it to joiners via the formation server
+	// so a volume sealed on any node can be opened on any other.
+	viperblockKey, err := handlers_iam.GenerateMasterKey()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "❌ Error generating viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	viperblockKeyPath, err := saveViperblockEncryptionKey(configDir, viperblockKey)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "❌ Error saving viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	fmt.Println("\n🔐 Generated viperblock at-rest encryption key (cluster-wide, shared with joiners)")
+	fmt.Printf("   Key: %s\n", viperblockKeyPath)
+
 	// Read CA cert/key for distribution to joining nodes
 	caCertData, err := os.ReadFile(filepath.Join(configDir, "ca.pem"))
 	if err != nil {
@@ -1432,6 +1479,9 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	// Include master key in formation server for distribution to joining nodes
 	fs.SetMasterKey(base64.StdEncoding.EncodeToString(masterKey))
 
+	// Distribute the cluster-wide viperblock encryption key to joiners
+	fs.SetViperblockKey(base64.StdEncoding.EncodeToString(viperblockKey))
+
 	// Register self (init node) as the first node
 	selfNode := formation.NodeInfo{
 		Name:        node,
@@ -1539,6 +1589,8 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 
 		OperatorEmail: email,
 
+		EncryptionKeyFile: viperblockKeyPath,
+
 		// Init node runs ovn-central locally
 		OVNNBAddr: "tcp:127.0.0.1:6641",
 		OVNSBAddr: "tcp:127.0.0.1:6642",
@@ -1580,7 +1632,7 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	}
 	fmt.Println("\n📋 Next steps:")
 	fmt.Println("   1. Start services on ALL nodes:")
-	fmt.Println("      ./scripts/start-dev.sh")
+	fmt.Println("      sudo systemctl start spinifex.target")
 	fmt.Println()
 }
 
@@ -1884,6 +1936,28 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 	}
 	fmt.Printf("✅ Predastore encryption key generated: %s\n", predastoreKeyPath)
 
+	// Viperblock at-rest encryption key is cluster-wide: receive it from the
+	// leader rather than generating one, so this node can open volumes sealed
+	// elsewhere. Lenient on absence — an older leader that predates key
+	// distribution returns nothing; fall back to cleartext rather than fail
+	// the join.
+	var viperblockKeyPath string
+	if statusResp.ViperblockKey == "" {
+		fmt.Println("⚠️  Leader did not provide a viperblock encryption key; at-rest encryption disabled on this node")
+	} else {
+		viperblockKeyBytes, err := base64.StdEncoding.DecodeString(statusResp.ViperblockKey)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "❌ Error decoding viperblock encryption key: %v\n", err)
+			os.Exit(1)
+		}
+		viperblockKeyPath, err = saveViperblockEncryptionKey(configDir, viperblockKeyBytes)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "❌ Error saving viperblock encryption key: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Printf("✅ Viperblock encryption key received from leader: %s\n", viperblockKeyPath)
+	}
+
 	// Generate server cert signed by CA with this node's bind IP
 	if err := admin.GenerateServerCertOnly(configDir, bindIP); err != nil {
 		fmt.Fprintf(os.Stderr, "Error generating server certificate: %v\n", err)
@@ -1975,6 +2049,8 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 
 		OperatorEmail: email,
 
+		EncryptionKeyFile: viperblockKeyPath,
+
 		// Joining nodes connect to the init node's OVN NB/SB DB
 		OVNNBAddr: fmt.Sprintf("tcp:%s:6641", leaderIP),
 		OVNSBAddr: fmt.Sprintf("tcp:%s:6642", leaderIP),
@@ -2260,6 +2336,7 @@ func runCertRenew(cmd *cobra.Command, _ []string) {
 type configDirs struct {
 	AWSGW      string
 	Predastore string
+	Viperblock string
 	NATS       string
 	Spinifex   string
 }
@@ -2269,10 +2346,11 @@ func createConfigSubdirs(configDir string) (configDirs, error) {
 	dirs := configDirs{
 		AWSGW:      filepath.Join(configDir, "awsgw"),
 		Predastore: filepath.Join(configDir, "predastore"),
+		Viperblock: filepath.Join(configDir, "viperblock"),
 		NATS:       filepath.Join(configDir, "nats"),
 		Spinifex:   filepath.Join(configDir, "spinifex"),
 	}
-	for _, dir := range []string{dirs.AWSGW, dirs.Predastore, dirs.NATS, dirs.Spinifex} {
+	for _, dir := range []string{dirs.AWSGW, dirs.Predastore, dirs.Viperblock, dirs.NATS, dirs.Spinifex} {
 		if err := os.MkdirAll(dir, 0700); err != nil {
 			return configDirs{}, fmt.Errorf("create directory %s: %w", dir, err)
 		}
@@ -2394,6 +2472,43 @@ func writePredastoreEncryptionKey(configDir string) (string, error) {
 	return keyPath, nil
 }
 
+// writeViperblockEncryptionKey generates a fresh 32-byte AES-256 master key for
+// viperblock at-rest encryption and writes it to
+// <configDir>/viperblock/encryption.key with mode 0600. Unlike the predastore
+// key, this key is cluster-wide and shared: the leader generates it once at init
+// and distributes it to joiners via the formation server, so a volume sealed on
+// any node can be opened on any other. Loaded at startup via masterkey.LoadShared.
+func writeViperblockEncryptionKey(configDir string) (string, error) {
+	viperblockDir := filepath.Join(configDir, "viperblock")
+	if err := os.MkdirAll(viperblockDir, 0750); err != nil {
+		return "", fmt.Errorf("create viperblock config dir: %w", err)
+	}
+	key, err := handlers_iam.GenerateMasterKey()
+	if err != nil {
+		return "", fmt.Errorf("generate viperblock encryption key: %w", err)
+	}
+	keyPath := filepath.Join(viperblockDir, "encryption.key")
+	if err := handlers_iam.SaveMasterKey(keyPath, key); err != nil {
+		return "", fmt.Errorf("save viperblock encryption key: %w", err)
+	}
+	return keyPath, nil
+}
+
+// saveViperblockEncryptionKey writes an already-generated 32-byte viperblock
+// master key (received from the formation leader) to
+// <configDir>/viperblock/encryption.key with mode 0600.
+func saveViperblockEncryptionKey(configDir string, key []byte) (string, error) {
+	viperblockDir := filepath.Join(configDir, "viperblock")
+	if err := os.MkdirAll(viperblockDir, 0750); err != nil {
+		return "", fmt.Errorf("create viperblock config dir: %w", err)
+	}
+	keyPath := filepath.Join(viperblockDir, "encryption.key")
+	if err := handlers_iam.SaveMasterKey(keyPath, key); err != nil {
+		return "", fmt.Errorf("save viperblock encryption key: %w", err)
+	}
+	return keyPath, nil
+}
+
 // writeSystemCredentials writes the system access key to a plaintext JSON file.
 // The daemon reads this at startup to inject credentials into ALB VM cloud-init
 // for SigV4-authenticated communication with the AWS gateway.

cmd/spinifex/cmd/admin_test.go

@@ -252,6 +252,74 @@ func TestWritePredastoreEncryptionKey(t *testing.T) {
 	assert.NotEqual(t, key1, key2, "per-node keys must differ")
 }
 
+// The viperblock key gates at-rest encryption for every volume on a fresh
+// install. masterkey.LoadShared rejects anything that isn't exactly 32 bytes at
+// 0640-or-stricter, so a regression in size/mode would silently fall back to
+// cleartext (empty path) or fail service startup.
+func TestWriteViperblockEncryptionKey(t *testing.T) {
+	configDir := t.TempDir()
+
+	keyPath, err := writeViperblockEncryptionKey(configDir)
+	require.NoError(t, err)
+	assert.Equal(t, filepath.Join(configDir, "viperblock", "encryption.key"), keyPath)
+
+	info, err := os.Stat(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, int64(32), info.Size(), "viperblock master key must be exactly 32 bytes")
+	assert.Equal(t, os.FileMode(0600), info.Mode().Perm(), "viperblock master key must be mode 0600")
+}
+
+// Joiners persist the leader's shared key via saveViperblockEncryptionKey
+// against a configDir with no viperblock/ subdir yet. The bytes round-trip
+// verbatim — the whole cluster shares one key, so any mutation would orphan
+// volumes sealed on other nodes.
+func TestSaveViperblockEncryptionKey_RoundTrip(t *testing.T) {
+	configDir := t.TempDir()
+	key := make([]byte, 32)
+	for i := range key {
+		key[i] = byte(i)
+	}
+
+	keyPath, err := saveViperblockEncryptionKey(configDir, key)
+	require.NoError(t, err)
+	assert.Equal(t, filepath.Join(configDir, "viperblock", "encryption.key"), keyPath)
+
+	got, err := os.ReadFile(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, key, got, "shared key must be written verbatim")
+}
+
+// A fresh install must render encryption_key_file so viperblockd enables
+// at-rest encryption; an empty field (upgrade / legacy) must omit the line
+// entirely so the volume stays in cleartext legacy mode.
+func TestSpinifexTomlTemplate_EncryptionKeyFile(t *testing.T) {
+	base := admin.ConfigSettings{
+		Node: "node1", Az: "ap-southeast-2a", Port: "4432", Region: "ap-southeast-2",
+		BindIP: "0.0.0.0", AccessKey: "AKIATEST", SecretKey: "SECRET",
+		AccountID: "123456789012", NatsToken: "token",
+		OVNNBAddr: "tcp:127.0.0.1:6641", OVNSBAddr: "tcp:127.0.0.1:6642",
+	}
+
+	dir := t.TempDir()
+	withKey := base
+	withKey.ConfigDir = dir
+	withKey.EncryptionKeyFile = "/etc/spinifex/viperblock/encryption.key"
+	path := filepath.Join(dir, "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, withKey))
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	assert.Contains(t, string(data), `encryption_key_file = "/etc/spinifex/viperblock/encryption.key"`)
+
+	dir2 := t.TempDir()
+	noKey := base
+	noKey.ConfigDir = dir2
+	path2 := filepath.Join(dir2, "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path2, spinifexTomlTemplate, noKey))
+	data2, err := os.ReadFile(path2)
+	require.NoError(t, err)
+	assert.NotContains(t, string(data2), "encryption_key_file", "blank key must omit the field")
+}
+
 // Joiners run `spx admin join` against a configDir that doesn't yet contain
 // a predastore/ subdir. The helper must create it (predastore subdir is
 // owned by spinifex-storage in prod, but the helper just needs the dir to

cmd/spinifex/cmd/cluster.go

@@ -8,6 +8,7 @@ import (
 	"time"
 
 	"github.com/mulgadc/spinifex/spinifex/daemon"
+	"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
 	"github.com/nats-io/nats.go"
 	"github.com/spf13/cobra"
 )
@@ -160,6 +161,79 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
 	fmt.Printf("Cluster shutdown complete (%s)\n", time.Since(start).Round(time.Millisecond))
 }
 
+// runClusterDrainDHCP asks every vpcd to DHCPRELEASE all external-pool leases
+// it currently holds, returning them to the upstream DHCP server. Intended for
+// the teardown path: an env reset otherwise strands held leases until TTL
+// because vpcd's Stop() preserves them for adopt-on-reboot. Best-effort — if
+// the cluster is already partly down it warns and exits 0 so teardown proceeds.
+func runClusterDrainDHCP(cmd *cobra.Command, args []string) {
+	timeout, _ := cmd.Flags().GetDuration("timeout")
+	if timeout <= 0 {
+		timeout = 30 * time.Second
+	}
+
+	_, nc, err := loadConfigAndConnect()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: drain-dhcp could not connect (cluster may be down): %v\n", err)
+		return
+	}
+	defer nc.Close()
+
+	total, responders := drainDHCPLeases(nc, timeout)
+	fmt.Printf("DHCP drain complete: released %d lease(s) from %d vpcd responder(s)\n", total, responders)
+}
+
+// drainDHCPLeases publishes a drain request to vpc.dhcp.drain and collects
+// replies until timeout — one responder per AZ (vpcd uses a per-AZ queue
+// group). Returns the summed released count and the number of responders.
+func drainDHCPLeases(nc *nats.Conn, timeout time.Duration) (released, responders int) {
+	reqData, err := json.Marshal(struct{}{})
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		return 0, 0
+	}
+
+	inbox := nats.NewInbox()
+	sub, err := nc.SubscribeSync(inbox)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: drain-dhcp subscribe failed: %v\n", err)
+		return 0, 0
+	}
+	defer sub.Unsubscribe()
+
+	if err := nc.PublishRequest(dhcp.TopicDrain, inbox, reqData); err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: drain-dhcp publish failed: %v\n", err)
+		return 0, 0
+	}
+	nc.Flush()
+
+	deadline := time.Now().Add(timeout)
+	for {
+		remaining := time.Until(deadline)
+		if remaining <= 0 {
+			break
+		}
+		msg, err := sub.NextMsg(remaining)
+		if err != nil {
+			break
+		}
+		var reply struct {
+			Released int    `json:"released"`
+			Error    string `json:"error,omitempty"`
+		}
+		if err := json.Unmarshal(msg.Data, &reply); err != nil {
+			continue
+		}
+		responders++
+		if reply.Error != "" {
+			fmt.Fprintf(os.Stderr, "  vpcd drain error: %s\n", reply.Error)
+			continue
+		}
+		released += reply.Released
+	}
+	return released, responders
+}
+
 // collectShutdownACKs publishes a shutdown request and collects ACKs from nodes.
 func collectShutdownACKs(nc *nats.Conn, topic string, reqData []byte, nodeCount int, timeout time.Duration) ([]daemon.ShutdownACK, error) {
 	inbox := nats.NewInbox()

cmd/spinifex/cmd/cluster_test.go

@@ -0,0 +1,58 @@
+package cmd
+
+import (
+	"encoding/json"
+	"testing"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
+	"github.com/mulgadc/spinifex/spinifex/testutil"
+	"github.com/nats-io/nats.go"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// fakeDrainResponder subscribes a stand-in vpcd to vpc.dhcp.drain that
+// replies with the given released count (or error), mirroring
+// Manager.handleDrainMsg without a real lease store.
+func fakeDrainResponder(t *testing.T, nc *nats.Conn, released int, errMsg string) {
+	t.Helper()
+	sub, err := nc.Subscribe(dhcp.TopicDrain, func(msg *nats.Msg) {
+		body, _ := json.Marshal(struct {
+			Released int    `json:"released"`
+			Error    string `json:"error,omitempty"`
+		}{Released: released, Error: errMsg})
+		_ = msg.Respond(body)
+	})
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = sub.Unsubscribe() })
+}
+
+func TestDrainDHCPLeasesSumsReleased(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+	fakeDrainResponder(t, nc, 7, "")
+
+	released, responders := drainDHCPLeases(nc, 2*time.Second)
+	assert.Equal(t, 7, released)
+	assert.Equal(t, 1, responders)
+}
+
+func TestDrainDHCPLeasesReportsResponderError(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+	fakeDrainResponder(t, nc, 0, "list leases for drain: boom")
+
+	// An erroring responder still counts as a responder but contributes
+	// zero released leases.
+	released, responders := drainDHCPLeases(nc, 2*time.Second)
+	assert.Equal(t, 0, released)
+	assert.Equal(t, 1, responders)
+}
+
+func TestDrainDHCPLeasesNoResponders(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+
+	// No vpcd subscribed: the collection window elapses with zero replies.
+	released, responders := drainDHCPLeases(nc, 200*time.Millisecond)
+	assert.Equal(t, 0, released)
+	assert.Equal(t, 0, responders)
+}

cmd/spinifex/cmd/service.go

@@ -538,7 +538,7 @@ var awsgwStartCmd = &cobra.Command{
 		awsgwHost := viper.GetString("host")
 		if awsgwHost != "" {
 			fmt.Println("Overwriting awsgw host to:", awsgwHost)
-			//nodeConfig.AWSGW.Host = awsgwHost
+			nodeConfig.AWSGW.Host = awsgwHost
 		}
 
 		awsgwTlsCert := viper.GetString("tls-cert")

cmd/spinifex/cmd/templates/spinifex.toml

@@ -93,6 +93,9 @@ token = "{{.NatsToken}}"
 
 [nodes.{{.Node}}.viperblock]
 shardwal = false
+{{- if .EncryptionKeyFile}}
+encryption_key_file = "{{.EncryptionKeyFile}}"
+{{- end}}
 
 [nodes.{{.Node}}.vpcd]
 ovn_nb_addr = "{{.OVNNBAddr}}"

docs/COMMANDS.md

@@ -8,162 +8,103 @@ Platform management commands not exposed via the AWS gateway API. These are CLI-
 
 Service lifecycle commands for starting, stopping, and checking status of all Spinifex cluster services. Each service subcommand supports `start`, `stop`, and `status` operations.
 
-| Command | Flags | Prerequisites | Basic Logic | Status |
-|---------|-------|---------------|-------------|--------|
-| `spx service predastore start` | `--port` (default: 8443), `--host` (default: 0.0.0.0), `--base-path`, `--config-path`, `--debug`, `--tls-cert`, `--tls-key`, `--backend` (distributed/filesystem, default: distributed), `--node-id` (default: 0), `--pprof`, `--pprof-output` | TLS cert/key, base-path, config-path required | Creates predastore service instance with S3-compatible storage backend → starts service | **DONE** |
-| `spx service predastore stop` | — | Predastore must be running | Stops the predastore service | **DONE** |
-| `spx service predastore status` | — | None | Reports predastore service status | **DONE** |
-| `spx service viperblock start` | `--s3-host` (default: 0.0.0.0:8443), `--s3-bucket` (default: predastore), `--s3-region` (default: ap-southeast-2), `--plugin-path` (auto-detected via `nbdkit --dump-config plugindir`; typically `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` on amd64, overridable via `SPINIFEX_VIPERBLOCK_PLUGIN_PATH` in `/etc/spinifex/systemd.env`) | `--config` required, plugin-path must exist on disk | Loads cluster config → connects to NATS and Predastore → starts viperblock block storage service with NBD plugin | **DONE** |
-| `spx service viperblock stop` | — | Viperblock must be running | Stops the viperblock service | **DONE** |
-| `spx service viperblock status` | — | None | Reports viperblock service status | **DONE** |
-| `spx service nats start` | `--port` (default: 4222), `--host` (default: 0.0.0.0), `--debug`, `--data-dir`, `--jetstream` | None | Starts embedded NATS server with optional JetStream | **DONE** |
-| `spx service nats stop` | — | NATS must be running | Stops the NATS service | **DONE** |
-| `spx service nats status` | — | None | Reports NATS service status | **DONE** |
-| `spx service spinifex start` | `--wal-dir` | `--config` required | Loads cluster config → starts spinifex daemon (VM orchestration, NATS subscriptions, health endpoint) | **DONE** |
-| `spx service spinifex stop` | — | Spinifex daemon must be running | Stops the spinifex daemon service | **DONE** |
-| `spx service spinifex status` | — | None | Reports spinifex daemon service status | **DONE** |
-| `spx service awsgw start` | `--host` (default: 0.0.0.0:9999), `--tls-cert`, `--tls-key`, `--debug` | `--config` required | Loads cluster config → starts AWS-compatible gateway with SigV4 auth, IAM policy enforcement, TLS | **DONE** |
-| `spx service awsgw stop` | — | AWS gateway must be running | Stops the AWS gateway service | **DONE** |
-| `spx service awsgw status` | — | None | Reports AWS gateway service status | **DONE** |
-| `spx service spinifex-ui start` | `--port` (default: 3000), `--host` (default: 0.0.0.0), `--tls-cert`, `--tls-key` | None | Starts embedded web UI server serving the React frontend. Aliases: `ui`, `spinifexui` | **DONE** |
-| `spx service spinifex-ui stop` | — | spinifex-ui must be running | Stops the spinifex-ui service | **DONE** |
-| `spx service spinifex-ui status` | — | None | Reports spinifex-ui service status | **DONE** |
-| `spx service vpcd start` | — | `--config` required, OVN/OVS installed | Loads cluster config → starts VPC daemon (subscribes to `vpc.*` NATS events, translates to OVN logical switches/ports/routers) | **DONE** |
-| `spx service vpcd stop` | — | vpcd must be running | Stops the vpcd service | **DONE** |
-| `spx service vpcd status` | — | None | Reports vpcd service status | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx service predastore start` | `--port` (default: 8443), `--host` (default: 0.0.0.0), `--base-path`, `--config-path`, `--debug`, `--tls-cert`, `--tls-key`, `--backend` (distributed/filesystem, default: distributed), `--node-id` (default: 0), `--pprof`, `--pprof-output` | Creates predastore service instance with S3-compatible storage backend → starts service |
+| `spx service predastore stop` | — | Stops the predastore service |
+| `spx service predastore status` | — | Reports predastore service status |
+| `spx service viperblock start` | `--s3-host` (default: 0.0.0.0:8443), `--s3-bucket` (default: predastore), `--s3-region` (default: ap-southeast-2), `--plugin-path` (auto-detected via `nbdkit --dump-config plugindir`; typically `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` on amd64, overridable via `SPINIFEX_VIPERBLOCK_PLUGIN_PATH` in `/etc/spinifex/systemd.env`) | Loads cluster config → connects to NATS and Predastore → starts viperblock block storage service with NBD plugin |
+| `spx service viperblock stop` | — | Stops the viperblock service |
+| `spx service viperblock status` | — | Reports viperblock service status |
+| `spx service nats start` | `--port` (default: 4222), `--host` (default: 0.0.0.0), `--debug`, `--data-dir`, `--jetstream` | Starts embedded NATS server with optional JetStream |
+| `spx service nats stop` | — | Stops the NATS service |
+| `spx service nats status` | — | Reports NATS service status |
+| `spx service spinifex start` | `--wal-dir` | Loads cluster config → starts spinifex daemon (VM orchestration, NATS subscriptions, health endpoint) |
+| `spx service spinifex stop` | — | Stops the spinifex daemon service |
+| `spx service spinifex status` | — | Reports spinifex daemon service status |
+| `spx service awsgw start` | `--host` (default: 0.0.0.0:9999), `--tls-cert`, `--tls-key`, `--debug` | Loads cluster config → starts AWS-compatible gateway with SigV4 auth, IAM policy enforcement, TLS |
+| `spx service awsgw stop` | — | Stops the AWS gateway service |
+| `spx service awsgw status` | — | Reports AWS gateway service status |
+| `spx service spinifex-ui start` | `--port` (default: 3000), `--host` (default: 0.0.0.0), `--tls-cert`, `--tls-key` | Starts embedded web UI server serving the React frontend. Aliases: `ui`, `spinifexui` |
+| `spx service spinifex-ui stop` | — | Stops the spinifex-ui service |
+| `spx service spinifex-ui status` | — | Reports spinifex-ui service status |
+| `spx service vpcd start` | — | Loads cluster config → starts VPC daemon (subscribes to `vpc.*` NATS events, translates to OVN logical switches/ports/routers) |
+| `spx service vpcd stop` | — | Stops the vpcd service |
+| `spx service vpcd status` | — | Reports vpcd service status |
 
 ### Cluster Inspection
 
 Operational commands for inspecting cluster state. These fan out NATS requests to all nodes and aggregate responses.
 
-| Command | Flags | Prerequisites | Basic Logic | Output Columns | Status |
-|---------|-------|---------------|-------------|----------------|--------|
-| `spx get nodes` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Loads config → publishes to `spinifex.node.status` fan-out topic → collects responses within timeout → merges config-known nodes with NATS responders → nodes that don't respond shown as `NotReady` | NAME, STATUS, IP, REGION, AZ, UPTIME, VMs, SERVICES | **DONE** |
-| `spx get vms` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Publishes to `spinifex.node.vms` fan-out topic → collects VM info from all nodes → sorts by node then instance ID → prints table. Alias: `spx get instances` | INSTANCE, STATUS, TYPE, VCPU, MEM, NODE, IP, AGE | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx get nodes` | `--timeout` (default: 3s) | Loads config → publishes to `spinifex.node.status` fan-out topic → collects responses within timeout → merges config-known nodes with NATS responders → nodes that don't respond shown as `NotReady` |
+| `spx get vms` | `--timeout` (default: 3s) | Publishes to `spinifex.node.vms` fan-out topic → collects VM info from all nodes → sorts by node then instance ID → prints table. Alias: `spx get instances` |
 
 ### Resource Monitoring
 
-| Command | Flags | Prerequisites | Basic Logic | Output | Status |
-|---------|-------|---------------|-------------|--------|--------|
-| `spx top nodes` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Publishes to `spinifex.node.status` fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability | Table 1: NAME, CPU (used/total), MEM (used/total), GPU (used/total), VMs — GPU column shows `allocated/total` when passthrough is enabled, `0/count*` when the node is GPU-capable but passthrough is disabled, or `-` when no GPU is present. Table 2: INSTANCE TYPE, AVAILABLE, VCPU, MEMORY | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx top nodes` | `--timeout` (default: 3s) | Publishes to `spinifex.node.status` fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability |
 
 ### Cluster Initialization
 
-| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
-|---------|-------|---------------|-------------|------------|--------|
-| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--token-ttl`, `--force` | None (first-time setup) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` | 1. Init creates all config files<br>2. Root credentials printed once<br>3. master.key is 32 bytes, mode 0600<br>4. bootstrap.json consumed on first start<br>5. `--force` re-initializes existing config<br>6. AWS CLI profile `spx` auto-configured<br>7. Multi-node init generates join token and writes to `<config-dir>/join-token` | **DONE** |
-| `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services` | Leader node must be running | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations | 1. Join existing cluster<br>2. Missing host (error)<br>3. Missing node name (error)<br>4. Missing token (error)<br>5. Invalid token (401)<br>6. Expired token (401) | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--token-ttl`, `--force` | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` |
+| `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services` | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations |
 
 ### Version
 
-| Command | Flags | Prerequisites | Basic Logic | Status |
-|---------|-------|---------------|-------------|--------|
-| `spx version` | — | None | Prints Spinifex version, commit hash, OS, and architecture (populated via build-time ldflags) | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx version` | — | Prints Spinifex version, commit hash, OS, and architecture (populated via build-time ldflags) |
 
 ### Cluster Operations
 
-| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
-|---------|-------|---------------|-------------|------------|--------|
-| `spx admin cluster shutdown` | `--force` (shutdown even if nodes don't respond), `--timeout` (max wait per phase, default 120s), `--dry-run` (print phase plan without executing) | Cluster must be running | Performs coordinated, phased shutdown of entire cluster. Phases execute in order: GATE (stop API/UI) → DRAIN (stop VMs) → STORAGE (stop viperblock) → PERSIST (stop predastore) → INFRA (stop NATS/daemon). Each phase waits for all nodes to ACK before proceeding. Uses JetStream state tracking. | 1. Shutdown running cluster<br>2. All nodes stop cleanly<br>3. Force shutdown with unresponsive nodes<br>4. Dry-run prints plan | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin cluster shutdown` | `--force` (shutdown even if nodes don't respond), `--timeout` (max wait per phase, default 120s), `--dry-run` (print phase plan without executing) | Performs coordinated, phased shutdown of entire cluster. Phases execute in order: GATE (stop API/UI) → DRAIN (stop VMs) → STORAGE (stop viperblock) → PERSIST (stop predastore) → INFRA (stop NATS/daemon). Each phase waits for all nodes to ACK before proceeding. Uses JetStream state tracking. |
+| `spx admin cluster drain-dhcp` | `--timeout` (reply-collection window, default 30s) | Asks each vpcd to DHCPRELEASE every external-pool DHCP lease it currently holds, returning them to the upstream DHCP server. Run on teardown before stopping services — an env reset otherwise strands held leases upstream until their TTL expires, eventually exhausting the upstream scope. Best-effort: warns and exits 0 if the cluster is already down. |
 
 ### Certificate Management
 
-| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
-|---------|-------|---------------|-------------|------------|--------|
-| `spx admin cert renew` | `--extra-ip` (additional IPs for SANs), `--extra-dns` (additional DNS names for SANs) | Existing CA from `spx admin init` | Reads existing CA → regenerates server certificate with all current network interface IPs and machine hostname in SANs → writes new cert. Use after adding a new network interface or changing IP addresses. | 1. Renew with auto-detected IPs<br>2. Renew with extra IPs<br>3. Renew with extra DNS names | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin cert renew` | `--extra-ip` (additional IPs for SANs), `--extra-dns` (additional DNS names for SANs) | Reads existing CA → regenerates server certificate with all current network interface IPs and machine hostname in SANs → writes new cert. Use after adding a new network interface or changing IP addresses. |
 
 ### Upgrade Management
 
-| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
-|---------|-------|---------------|-------------|------------|--------|
-| `spx admin upgrade` | `--yes` (apply without prompting), `--config-dir` (persistent, default: `~/spinifex/config`), `--spinifex-dir` (persistent, default: `~/spinifex`) | Existing installation (`spinifex.toml` must exist in config dir) | Reads current config versions from registry → computes pending config migrations (from→to per target) → prints version summary and pending list → prompts for confirmation unless `--yes` → runs `migrate.DefaultRegistry.RunAllConfig()` to apply migrations to config files. Intended for upgrades between Spinifex versions. Operators can skip migrations during install by setting `INSTALL_SPINIFEX_SKIP_MIGRATE=1`, then run `spx admin upgrade` manually to review and apply. After completion, services must be restarted with `sudo systemctl restart spinifex.target`. Invoked non-interactively by `scripts/setup.sh` with `--yes`. | 1. No installation present (error, suggests `spx admin init`)<br>2. No pending migrations (prints "No pending config migrations")<br>3. Pending migrations with interactive confirmation (y/yes applies, anything else aborts)<br>4. Pending migrations with `--yes` (applies without prompt)<br>5. Migration failure surfaces error and exits non-zero | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin upgrade` | `--yes` (apply without prompting), `--config-dir` (persistent, default: `~/spinifex/config`), `--spinifex-dir` (persistent, default: `~/spinifex`) | Reads current config versions from registry → computes pending config migrations (from→to per target) → prints version summary and pending list → prompts for confirmation unless `--yes` → runs `migrate.DefaultRegistry.RunAllConfig()` to apply migrations to config files. Intended for upgrades between Spinifex versions. Operators can skip migrations during install by setting `INSTALL_SPINIFEX_SKIP_MIGRATE=1`, then run `spx admin upgrade` manually to review and apply. After completion, services must be restarted with `sudo systemctl restart spinifex.target`. Invoked non-interactively by `scripts/setup.sh` with `--yes`. |
 
 ### Account Management
 
-| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
-|---------|-------|---------------|-------------|------------|--------|
-| `spx admin account create` | `--name` | Cluster must be running (NATS), master key must exist | Connects to NATS → CAS loop on `spinifex-account-counter:next_id` for sequential 12-digit ID → creates Account record in `spinifex-accounts` KV → creates `admin` user in new account → creates access key for admin → creates AdministratorAccess policy (Action:*, Resource:*) → attaches policy → prints credentials | 1. Create first account (ID 000000000001)<br>2. Sequential IDs (000000000002, etc.)<br>3. Admin user has full access<br>4. Credentials printed once | **DONE** |
-| `spx admin account list` | — | Cluster must be running (NATS), master key must exist | Connects to NATS → IAMService.ListAccounts() → prints table with Account ID, Name, Status, Created | 1. List all accounts<br>2. Empty list (only global account) | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin account create` | `--name` | Connects to NATS → CAS loop on `spinifex-account-counter:next_id` for sequential 12-digit ID → creates Account record in `spinifex-accounts` KV → creates `admin` user in new account → creates access key for admin → creates AdministratorAccess policy (Action:*, Resource:*) → attaches policy → prints credentials |
+| `spx admin account list` | — | Connects to NATS → IAMService.ListAccounts() → prints table with Account ID, Name, Status, Created |
 
 ### Image Management
 
-| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
-|---------|-------|---------------|-------------|------------|--------|
-| `spx admin images import` | `--name`, `--file`, `--distro`, `--version`, `--arch`, `--platform`, `--boot-mode` (bios/uefi/uefi-preferred), `--tag`, `--force`, `--skip-verify` | Cluster must be running; either `--name` (catalog download) or `--file` (operator-supplied media). `--file` mode additionally requires `--distro`, `--version`, `--arch`, `--boot-mode`. | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, verify the SHA-256/SHA-512 digest, and inherit `BootMode` from the catalog entry. `--boot-mode` overrides the catalog value when set. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports skip checksum verification (operator-supplied media is outside Spinifex's trust boundary, the skip is logged at INFO for audit) and require an explicit `--boot-mode` because there is no catalog metadata to inherit from. `--skip-verify` bypasses verification for catalog imports and emits a WARN slog + stderr notice; use only for debugging or when upstream mirrors are confirmed-broken. | 1. Import valid catalog image (verifies checksum)<br>2. Tampered cache hit fails with `ErrChecksumMismatch`<br>3. `--force` recovers after a mismatch<br>4. `--file` import skips verification<br>5. `--skip-verify` bypasses checksum with WARN<br>6. `--file` without `--boot-mode` rejected<br>7. `--boot-mode=invalid` rejected | **DONE** |
-| `spx admin images list` | — | None | Lists available OS images that can be imported or downloaded | 1. List available images | **DONE** |
-| `spx admin images remove` | `--image-id` (required), `--force`, `--yes` | Cluster must be running; predastore reachable. Targets AMIs imported via `spx admin images import` (non-account `ImageOwnerAlias`, e.g. `"system"`). | Loads `ami-<id>/config.json`, walks transitive dependents — copied snapshots whose `VolumeID == imageID`, volumes whose `SnapshotID` references the internal `snap-ami-<id>` or any derived snap, and account AMIs created via `CopyImage` whose `SnapshotID` is a derived snap — then prompts (skipped with `--yes`) before deleting `ami-<id>/config.json` (the DescribeImages barrier) followed by the rest of `ami-<id>/` and `snap-ami-<id>/`. Account-owned AMIs are refused with a hint pointing at `aws ec2 deregister-image` + `aws ec2 delete-snapshot`. `--force` bypasses the dependency, ownership and config-corrupt checks for salvage of orphaned blocks. | 1. Happy path: system AMI, no deps → deleted<br>2. Account-owned → refused with AWS-flow hint<br>3. Missing/corrupt config → `InvalidAMIID.NotFound` (salvageable with `--force`)<br>4. Dependent volume (direct or via copied snap) → refused<br>5. Dependent account AMI → refused<br>6. `--force` overrides dependents<br>7. Idempotent re-run after partial delete | **DONE** |
-
-#### Image integrity verification (CMMC SI.L1-3.14.2)
-
-Catalog imports (`spx admin images import --name <name>`) verify the image
-against the catalog-declared SHA-256/SHA-512 digest before extraction. The sums
-file is fetched from the catalog `Checksum` URL over HTTPS only (cross-scheme
-redirects refused), and verification runs on both fresh downloads and cache
-hits so a poisoned cache is caught on the next import.
-
-On mismatch the import exits non-zero, the cached file is left on disk for
-inspection, and the printed guidance is `spx admin images import --name <name>
---force` to re-download.
-
-`--file` imports skip verification by design: operator-supplied media is
-outside Spinifex's trust boundary and the operator is responsible for
-integrity (e.g. `sha256sum` against a trusted upstream digest before import).
-The skip is recorded as an INFO `slog` event with `reason=local-file-import`
-so a CMMC assessor can audit the decision from journald.
-
-`--skip-verify` bypasses the checksum step for catalog imports. The command
-still downloads via the catalog URL but does not compare the image digest
-against the sums file. Intended for narrow cases such as debugging upstream
-mirror issues or running against a transiently-broken `latest/` path; the
-skip is logged at WARN with `reason=skip-verify-flag` and printed to stderr
-so operators and assessors see it. Prefer `--file` with an out-of-band
-verified image over `--skip-verify` whenever possible.
-
-**Limitation:** verification confirms the image matches the digest the mirror
-served. A mirror compromise that swaps both image and sums file is not
-detected; closing that gap requires GPG signature verification of the sums
-file, deferred to a later phase.
-
-#### `spx admin images remove` caveats
-
-Admin-imported AMIs (`ImageOwnerAlias = "system"`) live
-under the `ami-<id>/` S3 prefix and use a viperblock-internal snap checkpoint
-at `snap-ami-<id>/` — there is no `snap-<id>/metadata.json`. The AWS handlers
-(`DeregisterImage`, `DeleteSnapshot`) reject system owners with
-`UnauthorizedOperation`, which is the right behaviour for tenant API callers
-but leaves no AWS-flow path to reclaim space. `spx admin images remove` is
-the admin-trust-boundary counterpart that performs the dependency walk and
-hard-deletes the blocks directly against predastore.
-
-`CopyImage` of a system AMI is metadata-only: it writes a fresh
-`snap-<acct>/metadata.json` whose `VolumeID` points at `ami-<sys>` and a new
-`ami-<acct>/config.json` referencing that snap. Volumes launched from the
-copied AMI read transitively from `ami-<sys>/chunks/...`. The remove command
-walks this transitive set and refuses if anything references the target.
-
-**TOCTOU window:** between the safety scan and the `config.json` delete a
-concurrent `RunInstances` against the AMI could create a new dependent
-volume. The window is sub-second on a healthy cluster. The admin running
-this command is expected to know the fleet's operational state; if the race
-fires the result is a `vol-<id>` with deleted backing blocks, recovered by
-terminating the orphaned instance.
-
-**`--force` bypasses every safety check** (dependents, ownership, missing /
-corrupt `config.json`). Use only for salvage of orphaned blocks. Running it
-against a live system AMI corrupts every dependent volume on the next disk
-read.
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin images import` | `--name`, `--file`, `--distro`, `--version`, `--arch`, `--platform`, `--boot-mode` (bios/uefi/uefi-preferred), `--tag`, `--force`, `--skip-verify` | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, verify the SHA-256/SHA-512 digest, and inherit `BootMode` from the catalog entry. `--boot-mode` overrides the catalog value when set. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports skip checksum verification (operator-supplied media is outside Spinifex's trust boundary, the skip is logged at INFO for audit) and require an explicit `--boot-mode` because there is no catalog metadata to inherit from. `--skip-verify` bypasses verification for catalog imports and emits a WARN slog + stderr notice; use only for debugging or when upstream mirrors are confirmed-broken. |
+| `spx admin images list` | — | Lists available OS images that can be imported or downloaded |
+| `spx admin images remove` | `--image-id` (required), `--force`, `--yes` | Loads `ami-<id>/config.json`, walks transitive dependents — copied snapshots whose `VolumeID == imageID`, volumes whose `SnapshotID` references the internal `snap-ami-<id>` or any derived snap, and account AMIs created via `CopyImage` whose `SnapshotID` is a derived snap — then prompts (skipped with `--yes`) before deleting `ami-<id>/config.json` (the DescribeImages barrier) followed by the rest of `ami-<id>/` and `snap-ami-<id>/`. Account-owned AMIs are refused with a hint pointing at `aws ec2 deregister-image` + `aws ec2 delete-snapshot`. `--force` bypasses the dependency, ownership and config-corrupt checks for salvage of orphaned blocks. |
 
 ### GPU Management
 
-| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
-|---------|-------|---------------|-------------|------------|--------|
-| `spx admin gpu status` | `--node` (default: local node) | Cluster must be running (NATS) | Queries `spinifex.node.status` fan-out → finds the target node response → prints Node, GPU hardware (model list or "none detected"), IOMMU state, vfio-pci state, passthrough enabled/disabled, and GPU pool allocation (`allocated/total`). Also lists GPU-capable instance types when passthrough is active. | 1. Local node with passthrough enabled (shows pool and instance types)<br>2. Local node capable but passthrough disabled (shows "run enable")<br>3. Node without GPU hardware (shows "none detected")<br>4. `--node` targets a remote node<br>5. Node not found or not responding (error) | **DONE** |
-| `spx admin gpu enable` | — | Cluster must be running (NATS); node must be GPU-capable (IOMMU + vfio-pci configured) | Checks current passthrough state via NATS → errors if already enabled or prerequisites not met (directs to `setup`) → writes `gpu_passthrough = true` to `spinifex.toml` via `admin.SetGPUPassthrough` → sends SIGHUP to `spinifex-daemon` via `systemctl kill -s HUP` → polls node status for up to 30 s until daemon confirms new state → prints final `gpu status` output. Must be run directly on the target host. | 1. Enable on capable node<br>2. Already enabled (no-op)<br>3. Prerequisites not met (error, suggests `setup`)<br>4. Daemon does not confirm within 30 s (timeout error) | **DONE** |
-| `spx admin gpu disable` | — | Cluster must be running (NATS); no GPU instances may be running | Checks current passthrough state via NATS → errors if already disabled or if `AllocGPUs > 0` (must terminate GPU instances first) → writes `gpu_passthrough = false` → sends SIGHUP to `spinifex-daemon` → polls for up to 30 s → prints final `gpu status` output. Must be run directly on the target host. | 1. Disable with no GPU instances running<br>2. Already disabled (no-op)<br>3. GPU instances running (error with count)<br>4. Daemon does not confirm within 30 s (timeout error) | **DONE** |
-| `spx admin gpu setup` | — | Must be run as root; NVIDIA or AMD GPU must be present | Idempotent host configuration for GPU passthrough. Steps (skipped if already applied): detect GPUs via `gpu.Discover` → collect PCI IDs for all IOMMU-group siblings → check/enable IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`) → write vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`) → blacklist nouveau (`/etc/modprobe.d/blacklist-nouveau.conf`) → blacklist amdgpu if AMD GPU present (`/etc/modprobe.d/blacklist-amdgpu.conf`) → write vfio-pci early binding config (`/etc/modprobe.d/vfio-pci.conf`) → add vfio modules to initramfs. If any change requires a reboot: runs `update-initramfs -u` and exits with reboot instructions. After reboot: verifies `vfio_pci` module is loaded → verifies each GPU is bound to `vfio-pci` (binds explicitly via `driver_override` if unbound) → calls `gpu enable` to activate passthrough. | 1. First run on unconfigured host (GRUB updated, reboot required)<br>2. Re-run after reboot (verifies bindings, enables passthrough)<br>3. Already fully configured (all steps skipped, enables passthrough)<br>4. AMD GPU present (blacklists amdgpu in addition to nouveau)<br>5. Not run as root (error)<br>6. No GPU hardware detected (error) | **DONE** |
-| `spx admin gpu mig status` | — (must be run on target host) | NVIDIA GPU with MIG support present | Runs `gpu.Discover()` locally → for each GPU prints: PCI address, model, MIG capability, MIG mode (enabled/disabled/N/A); for GPUs with MIG enabled lists active MIG slices with GI ID, profile name, and mdev path. | 1. Node with MIG enabled (shows slices)<br>2. MIG capable but not enabled (shows disabled)<br>3. No MIG-capable GPU (shows N/A for MIG fields) | **DONE** |
-| `spx admin gpu mig enable` | `--profile <name>` (required, e.g. `1g.10gb`), `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Must be run as root; MIG-capable GPU present; no GPU instances running | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → enables MIG mode on each target (`gpu.EnableMIGMode`) → lists available profiles (`gpu.ListProfiles`) and validates requested profile name → destroys any existing instances (`gpu.DestroyAllInstances`) → creates new instances filling GPU capacity (`gpu.CreateInstances`) → writes `mig_profile` to `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. | 1. Enable MIG with valid profile<br>2. Invalid profile name (error with available profile list)<br>3. No MIG-capable GPU (error)<br>4. GPU instances running (error)<br>5. `--gpu` targets specific GPU<br>6. Not run as root (error) | **DONE** |
-| `spx admin gpu mig disable` | `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Must be run as root; no GPU instances running | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → destroys all GPU instances (`gpu.DestroyAllInstances`) → disables MIG mode (`gpu.DisableMIGMode`) → clears `mig_profile` in `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. | 1. Disable with no running GPU instances<br>2. GPU instances running (error)<br>3. MIG not enabled on GPU (skips that GPU)<br>4. `--gpu` targets specific GPU<br>5. Not run as root (error) | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin gpu status` | `--node` (default: local node) | Queries `spinifex.node.status` fan-out → finds the target node response → prints Node, GPU hardware (model list or "none detected"), IOMMU state, vfio-pci state, passthrough enabled/disabled, and GPU pool allocation (`allocated/total`). Also lists GPU-capable instance types when passthrough is active. |
+| `spx admin gpu enable` | — | Checks current passthrough state via NATS → errors if already enabled or prerequisites not met (directs to `setup`) → writes `gpu_passthrough = true` to `spinifex.toml` via `admin.SetGPUPassthrough` → sends SIGHUP to `spinifex-daemon` via `systemctl kill -s HUP` → polls node status for up to 30 s until daemon confirms new state → prints final `gpu status` output. Must be run directly on the target host. |
+| `spx admin gpu disable` | — | Checks current passthrough state via NATS → errors if already disabled or if `AllocGPUs > 0` (must terminate GPU instances first) → writes `gpu_passthrough = false` → sends SIGHUP to `spinifex-daemon` → polls for up to 30 s → prints final `gpu status` output. Must be run directly on the target host. |
+| `spx admin gpu setup` | — | Idempotent host configuration for GPU passthrough. Steps (skipped if already applied): detect GPUs via `gpu.Discover` → collect PCI IDs for all IOMMU-group siblings → check/enable IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`) → write vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`) → blacklist nouveau (`/etc/modprobe.d/blacklist-nouveau.conf`) → blacklist amdgpu if AMD GPU present (`/etc/modprobe.d/blacklist-amdgpu.conf`) → write vfio-pci early binding config (`/etc/modprobe.d/vfio-pci.conf`) → add vfio modules to initramfs. If any change requires a reboot: runs `update-initramfs -u` and exits with reboot instructions. After reboot: verifies `vfio_pci` module is loaded → verifies each GPU is bound to `vfio-pci` (binds explicitly via `driver_override` if unbound) → calls `gpu enable` to activate passthrough. |
+| `spx admin gpu mig status` | — (must be run on target host) | Runs `gpu.Discover()` locally → for each GPU prints: PCI address, model, MIG capability, MIG mode (enabled/disabled/N/A); for GPUs with MIG enabled lists active MIG slices with GI ID, profile name, and mdev path. |
+| `spx admin gpu mig enable` | `--profile <name>` (required, e.g. `1g.10gb`), `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → enables MIG mode on each target (`gpu.EnableMIGMode`) → lists available profiles (`gpu.ListProfiles`) and validates requested profile name → destroys any existing instances (`gpu.DestroyAllInstances`) → creates new instances filling GPU capacity (`gpu.CreateInstances`) → writes `mig_profile` to `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. |
+| `spx admin gpu mig disable` | `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → destroys all GPU instances (`gpu.DestroyAllInstances`) → disables MIG mode (`gpu.DisableMIGMode`) → clears `mig_profile` in `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. |
 
 
 ## AWS-Compatible API
@@ -243,7 +184,7 @@ input but treat it as `uefi` at launch.
 | `attach-volume` | `--volume-id`, `--instance-id`, `--device` (auto-assigns `/dev/sd[f-p]`) | `--dry-run` | **DONE** |
 | `detach-volume` | `--volume-id`, `--instance-id` (optional), `--device`, `--force` | `--dry-run` | **DONE** |
 | `describe-volume-status` | `--volume-ids`, `--filters` (volume-id, volume-status.status, availability-zone) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
-| `describe-volumes-modifications` | — | `--volume-ids`, `--filters`, `--max-results` | **NOT STARTED** |
+| `describe-volumes-modifications` | `--volume-ids`, `--filters` (modification-state, original-iops, original-size, original-volume-type, start-time, target-iops, target-size, target-volume-type, volume-id) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
 
 ### EC2 — Snapshots
 
@@ -295,9 +236,6 @@ by downstream services.
 
 ### EC2 — VPC Core
 
-VPC/Subnet/ENI/SG CRUD stores metadata in NATS KV and publishes events to
-vpcd for OVN translation. Single AZ for Spinifex v1.
-
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
 | `create-vpc` | `--cidr-block`, `--tag-specifications` | `--instance-tenancy`, `--dry-run` | **DONE** |
@@ -393,9 +331,6 @@ EIP commands are only registered when an external IPAM pool is configured.
 
 ### EC2 — NAT Gateway
 
-Deleted gateways move to a separate KV bucket with 1-hour TTL for Terraform
-polling.
-
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
 | `create-nat-gateway` | `--subnet-id`, `--allocation-id` | `--connectivity-type`, `--tag-specifications`, `--dry-run` | **DONE** |
@@ -406,9 +341,6 @@ polling.
 
 ### EC2 — Placement Groups
 
-Strategies: `spread` (1 instance per node, strict) and `cluster` (all pinned
-to single node). `partition` rejected.
-
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
 | `create-placement-group` | `--group-name`, `--strategy` (spread/cluster) | `--partition-count`, `--spread-level`, `--tag-specifications`, `--dry-run` | **DONE** |
@@ -543,9 +475,6 @@ bypasses policy evaluation entirely.
 
 ### IAM — Access Keys
 
-Max 2 keys per user. Secrets encrypted with AES-256-GCM using master key,
-returned plaintext only at creation.
-
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
 | `create-access-key` | `--user-name` | — | **DONE** |
@@ -555,34 +484,18 @@ returned plaintext only at creation.
 
 ### IAM — Policies
 
-Policy documents require `Version: "2012-10-17"` and a valid Statement array.
-Wildcard action matching supported (`ec2:*`, `ec2:Describe*`, `*`). Evaluation
-order: explicit Deny > explicit Allow > implicit Deny.
-
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
 | `create-policy` | `--policy-name`, `--policy-document`, `--path`, `--description` | `--tags` | **DONE** |
 | `get-policy` | `--policy-arn` | — | **DONE** |
 | `get-policy-version` | `--policy-arn`, `--version-id` | — | **DONE** |
+| `list-policy-versions` | `--policy-arn` | `--max-items`, `--marker` | **DONE** |
 | `list-policies` | — | `--scope`, `--only-attached`, `--path-prefix`, `--max-items`, `--marker` | **DONE** |
 | `delete-policy` | `--policy-arn` | — | **DONE** |
 | `attach-user-policy` | `--user-name`, `--policy-arn` | — | **DONE** |
 | `detach-user-policy` | `--user-name`, `--policy-arn` | — | **DONE** |
 | `list-attached-user-policies` | `--user-name` | `--path-prefix`, `--max-items`, `--marker` | **DONE** |
 
-### IAM — Policy Evaluation (internal — not an AWS API)
-
-| Feature | Status |
-|---------|--------|
-| Root user bypass | **DONE** |
-| Default deny | **DONE** |
-| Explicit allow | **DONE** |
-| Explicit deny | **DONE** |
-| Wildcard matching (`*`, `ec2:*`, `ec2:Describe*`) | **DONE** |
-| Account-scoped evaluation | **DONE** |
-| EC2 action mapping | **DONE** |
-| IAM action mapping (16 actions) | **DONE** |
-
 ### IAM — Roles
 
 | Command | Implemented Flags | Missing Flags | Status |
@@ -596,12 +509,11 @@ order: explicit Deny > explicit Allow > implicit Deny.
 | `attach-role-policy` | `--role-name`, `--policy-arn` | — | **DONE** |
 | `detach-role-policy` | `--role-name`, `--policy-arn` | — | **DONE** |
 | `list-attached-role-policies` | `--role-name`, `--path-prefix` | `--max-items`, `--marker` | **DONE** |
+| `list-role-policies` | `--role-name` | `--max-items`, `--marker` | **DONE** (gateway stub — inline policies unsupported, always returns empty) |
+| `get-role-policy` / `put-role-policy` / `delete-role-policy` | — | inline role policies | **NOT STARTED** |
 
 ### IAM — Instance Profiles
 
-Containers for IAM roles that allow EC2 instances to assume a role. Required
-for `run-instances --iam-instance-profile`. Max 1 role per profile.
-
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
 | `create-instance-profile` | `--instance-profile-name`, `--path`, `--tags` | — | **DONE** |
@@ -612,6 +524,19 @@ for `run-instances --iam-instance-profile`. Max 1 role per profile.
 | `add-role-to-instance-profile` | `--instance-profile-name`, `--role-name` | — | **DONE** |
 | `remove-role-from-instance-profile` | `--instance-profile-name`, `--role-name` | — | **DONE** |
 
+### IAM — OIDC Providers
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `create-open-id-connect-provider` | `--url`, `--client-id-list`, `--thumbprint-list`, `--tags` | — | **DONE** |
+| `get-open-id-connect-provider` | `--open-id-connect-provider-arn` | — | **DONE** |
+| `list-open-id-connect-providers` | — | — | **DONE** |
+| `delete-open-id-connect-provider` | `--open-id-connect-provider-arn` | — | **DONE** |
+| `add-client-id-to-open-id-connect-provider` | — | `--open-id-connect-provider-arn`, `--client-id` | **NOT STARTED** |
+| `remove-client-id-from-open-id-connect-provider` | — | `--open-id-connect-provider-arn`, `--client-id` | **NOT STARTED** |
+| `update-open-id-connect-provider-thumbprint` | — | `--open-id-connect-provider-arn`, `--thumbprint-list` | **NOT STARTED** |
+| `tag-open-id-connect-provider` / `untag-open-id-connect-provider` | — | `--open-id-connect-provider-arn`, `--tags`/`--tag-keys` | **NOT STARTED** |
+
 ### IAM — Groups
 
 | Command | Implemented Flags | Missing Flags | Status |
@@ -631,27 +556,78 @@ for `run-instances --iam-instance-profile`. Max 1 role per profile.
 
 ## STS
 
-| Command | Implemented Flags | Missing Flags (rejected if supplied) | Status |
+| Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|--------------------------------------|--------|
 | `get-caller-identity` | — | — | **DONE** |
 | `assume-role` | `--role-arn`, `--role-session-name`, `--duration-seconds` (900–min(role MaxSessionDuration, 43200)) | `--policy`, `--policy-arns` (→ `PackedPolicyTooLarge`); `--tags`, `--transitive-tag-keys` (→ `InvalidParameterValue`); `--serial-number`, `--token-code` (→ `InvalidParameterValue`); `--external-id`, `--source-identity` (accepted and logged, **not enforced** — no Condition evaluator in v1) | **DONE** |
-| `get-session-token` | — | `--duration-seconds`, `--serial-number`, `--token-code` | **NOT STARTED** |
-| `assume-role-with-web-identity` | — | `--role-arn`, `--role-session-name`, `--web-identity-token`, `--provider-id`, `--policy`, `--policy-arns`, `--duration-seconds` | **NOT STARTED** |
+| `get-session-token` | `--duration-seconds` (900–129600, default 43200 = 12h; clamped, not rejected) | `--serial-number`, `--token-code` (MFA → `InvalidParameterValue`) | **DONE** |
+| `assume-role-with-web-identity` | `--role-arn`, `--role-session-name`, `--web-identity-token`, `--duration-seconds` (900–43200, default 3600) | `--provider-id`; `--policy`, `--policy-arns` (→ `PackedPolicyTooLarge`) | **DONE** |
 | `assume-role-with-saml` | — | `--role-arn`, `--principal-arn`, `--saml-assertion`, `--policy`, `--policy-arns`, `--duration-seconds` | **NOT STARTED** |
 | `get-access-key-info` | — | `--access-key-id` | **NOT STARTED** |
 | `get-federation-token` | — | `--name`, `--policy`, `--policy-arns`, `--duration-seconds`, `--tags` | **NOT STARTED** |
 | `decode-authorization-message` | — | `--encoded-message` | **NOT STARTED** |
 
-Trust policies stored on roles (`AssumeRolePolicyDocument`) reject `Condition`,
-`NotPrincipal`, `NotAction`, empty-string `Action` elements, and empty
-`Principal` blocks at write time (`MalformedPolicyDocument`) — v1 has no
-Condition evaluator so accepting them would silently allow.
+Trust policies stored on roles (`AssumeRolePolicyDocument`) reject `Condition`, `NotPrincipal`, `NotAction`, empty-string `Action` elements, and empty `Principal` blocks at write time (`MalformedPolicyDocument`) — v1 has no Condition evaluator so accepting them would silently allow.
+
+---
+
+## IMDS (Instance Metadata Service)
+
+Available at `169.254.169.254` from inside every running guest VM, matching AWS. The endpoint is reached from within a guest over plain HTTP, exactly as on EC2, with no in-VM agent to install. DHCP and fully static guests reach it identically, with no in-guest route configuration.
+
+**IMDSv2-only.** Every read requires a session token. A tokenless (v1-style) `GET` returns `401 Unauthorized` with an empty body. Obtain a token with a `PUT /latest/api/token` carrying `X-aws-ec2-metadata-token-ttl-seconds` (1–21600), then send it back in `X-aws-ec2-metadata-token` on every read.
+
+```bash
+# Inside the guest VM:
+TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
+    -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
+curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
+    http://169.254.169.254/latest/meta-data/instance-id
+
+# Tokenless GET → 401 Unauthorized (empty body)
+curl -i http://169.254.169.254/latest/meta-data/instance-id
+```
+
+### IMDS — Supported Paths
+
+| Path | Method | Source | Status |
+|------|--------|--------|--------|
+| `/latest/api/token` | PUT | Issues an ENI-bound IMDSv2 token; `X-aws-ec2-metadata-token-ttl-seconds` ∈ [1, 21600] required | **DONE** |
+| `/latest/meta-data/` | GET | Directory listing of supported children | **DONE** |
+| `/latest/meta-data/instance-id` | GET | `vm.ID` | **DONE** |
+| `/latest/meta-data/instance-type` | GET | `vm.InstanceType` | **DONE** |
+| `/latest/meta-data/ami-id` | GET | launch `ImageId` | **DONE** |
+| `/latest/meta-data/local-ipv4` | GET | `ENIRecord.PrivateIpAddress` (== request source IP) | **DONE** |
+| `/latest/meta-data/public-ipv4` | GET | EIP, else instance public IP; empty body if none | **DONE** |
+| `/latest/meta-data/mac` | GET | `ENIRecord.MacAddress` | **DONE** |
+| `/latest/meta-data/security-groups` | GET | `ENIRecord.SecurityGroupIds`, newline-separated | **DONE** |
+| `/latest/meta-data/hostname`, `/local-hostname` | GET | Synthesised `ip-<dashed-ip>.<region>.compute.internal` | **DONE** |
+| `/latest/meta-data/placement/availability-zone` | GET | `ENIRecord.AvailabilityZone` | **DONE** |
+| `/latest/meta-data/placement/region` | GET | Derived from AZ (trailing letter stripped) | **DONE** |
+| `/latest/meta-data/iam/info` | GET | `{InstanceProfileArn, InstanceProfileId}`; 404 if no profile | **DONE** |
+| `/latest/meta-data/iam/security-credentials/` | GET | Role name(s) under the profile, one per line; empty body if none | **DONE** |
+| `/latest/meta-data/iam/security-credentials/<role>` | GET | STS `AssumeRoleForInstance` → ASIA-prefixed temporary credential JSON | **DONE** |
+| `/latest/meta-data/public-keys/` | GET | `0=<keyName>` from the launch key pair; 404 if none | **DONE** |
+| `/latest/meta-data/public-keys/0/` | GET | `openssh-key` (format list for index 0) | **DONE** |
+| `/latest/meta-data/public-keys/0/openssh-key` | GET | Launch SSH public key, live-fetched from the key store; 404 if the key was deleted, 500 on backend fault | **DONE** |
+| `/latest/user-data` | GET | `vm.UserData`; 404 if none | **DONE** |
+| `/latest/dynamic/instance-identity/document` (+ `signature`, `pkcs7`, `rsa2048`) | GET | Needs a per-cluster signing key | **NOT STARTED** (404; lands with EKS IRSA) |
+| `/latest/meta-data/network/interfaces/macs/<mac>/...` | GET | Multi-ENI rendering (subnet-id, vpc-id, ipv4s, security-group-ids, …) | **NOT STARTED** (404) |
+| `/latest/meta-data/tags/instance/<key>` | GET | Instance-tag metadata; gated on `InstanceMetadataTags` enablement | **NOT STARTED** (404) |
+| `/latest/meta-data/public-hostname` | GET | No public DNS today; would mirror `public-ipv4` | **NOT STARTED** (404) |
+| `/latest/meta-data/reservation-id` | GET | Available on the stored `Reservation`; cheap, not yet surfaced | **NOT STARTED** (404) |
+| `/latest/meta-data/ami-launch-index` | GET | `0` for single-instance launches; trivial once multi-count is exercised | **NOT STARTED** (404) |
+| `/latest/meta-data/block-device-mapping/...` | GET | `ami`/`root`/`ebsN`/`ephemeralN` device map | **NOT STARTED** (404) |
+| `/latest/meta-data/placement/{group-name,partition-number,availability-zone-id,host-id}` | GET | Placement extras beyond `availability-zone`/`region` | **NOT STARTED** (404) |
+| `/latest/meta-data/instance-life-cycle`, `/instance-action` | GET | `on-demand`/`spot`; `none` unless interruptible instances ship | **NOT STARTED** (404) |
+| `/latest/meta-data/services/{domain,partition}` | GET | Static per-deployment values | **NOT STARTED** (404) |
+| `/latest/meta-data/spot/{instance-action,termination-time}` | GET | Only meaningful once Spot is modelled | **NOT STARTED** (404) |
 
 ---
 
 ## ELBv2 (Application & Network Load Balancer)
 
-The data plane uses a system-managed LB VM running HAProxy, launched automatically during `create-load-balancer`. HAProxy config is pushed via NATS on listener/target changes.
+The data plane uses a system-managed LB VM, launched automatically during `create-load-balancer`. Application Load Balancers run **HAProxy** (L7: rules, fixed-response, redirect, HTTP/HTTPS). Network Load Balancers run **nginx `stream`** (L4: TCP, UDP, TLS, TCP_UDP) — HAProxy cannot load-balance UDP. The agent selects the engine from the `Engine` field on the config-delivery response.
 
 ### ELBv2 — Load Balancers
 
@@ -681,10 +657,54 @@ The data plane uses a system-managed LB VM running HAProxy, launched automatical
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
-| `create-listener` | `--load-balancer-arn`, `--default-actions` (Type=forward, TargetGroupArn), `--protocol` (HTTP), `--port` | `--ssl-policy`, `--certificates`, `--alpn-policy`, `--mutual-authentication`, `--dry-run` | **DONE** |
+| `create-listener` | `--load-balancer-arn`, `--default-actions` (Type=forward, TargetGroupArn), `--protocol` (HTTP/HTTPS/TLS), `--port`, `--certificates` (HTTPS/TLS), `--ssl-policy` | `--alpn-policy`, `--mutual-authentication`, `--dry-run` | **DONE** |
+| `modify-listener` | `--listener-arn`, `--protocol`, `--port`, `--default-actions`, `--certificates`, `--ssl-policy` | `--alpn-policy`, `--mutual-authentication`, `--dry-run` | **DONE** |
 | `delete-listener` | `--listener-arn` | `--dry-run` | **DONE** |
 | `describe-listeners` | `--load-balancer-arn`, `--listener-arns` | `--page-size`, `--marker`, `--dry-run` | **DONE** |
 
+### ELBv2 — Listener Rules
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `create-rule` | `--listener-arn`, `--priority` (1–50000), `--conditions` (host-header, path-pattern, http-header, http-request-method, query-string, source-ip), `--actions` (forward, redirect, fixed-response), `--tags` | `--dry-run` | **DONE** |
+| `modify-rule` | `--rule-arn`, `--conditions`, `--actions` | `--dry-run` | **DONE** |
+| `delete-rule` | `--rule-arn` | `--dry-run` | **DONE** |
+| `describe-rules` | `--listener-arn`, `--rule-arns` | `--page-size`, `--marker` (parsed, not enforced) | **DONE** |
+| `set-rule-priorities` | `--rule-priorities` (RuleArn, Priority) | — | **DONE** |
+
+A synthetic `default` rule is derived from the listener's `DefaultActions`.
+
+### ELBv2 — Listener Certificates & SSL Policies
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `add-listener-certificates` | `--listener-arn`, `--certificates` | `--dry-run` | **DONE** |
+| `remove-listener-certificates` | `--listener-arn`, `--certificates` | `--dry-run` | **DONE** |
+| `describe-listener-certificates` | `--listener-arn` | `--page-size`, `--marker` | **DONE** |
+| `describe-ssl-policies` | `--names` | `--load-balancer-type`, `--page-size`, `--marker` | **DONE** (static catalog: `ELBSecurityPolicy-FS-1-2-Res-2019-08`, `ELBSecurityPolicy-TLS13-1-2-2021-06` — metadata only, no in-platform TLS termination) |
+
+The default certificate cannot be added/removed via these calls — set it on the listener.
+
+### ELBv2 — Attributes & Modify
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `describe-load-balancer-attributes` | `--load-balancer-arn` | — | **DONE** (stored values over per-type defaults) |
+| `modify-load-balancer-attributes` | `--load-balancer-arn`, `--attributes` | — | **DONE** (unknown keys → `ValidationError`) |
+| `describe-target-group-attributes` | `--target-group-arn` | — | **DONE** |
+| `modify-target-group-attributes` | `--target-group-arn`, `--attributes` (incl. `deregistration_delay.timeout_seconds`, `stickiness.*`) | — | **DONE** |
+| `modify-target-group` | `--target-group-arn`, `--health-check-*`, `--matcher` | `--target-type`/`--protocol`/`--vpc-id` (immutable) | **DONE** |
+| `describe-listener-attributes` | `--listener-arn` | — | **DONE** (stub — returns empty; not persisted) |
+| `modify-listener-attributes` | `--listener-arn`, `--attributes` | — | **DONE** (stub — echoes input; not persisted) |
+
+### ELBv2 — Network & Security
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `set-security-groups` | `--load-balancer-arn`, `--security-groups` | `--enforce-security-group-inbound-rules-on-private-link-traffic` | **DONE** (ALB only — NLB → `InvalidConfigurationRequest`) |
+| `set-subnets` | `--load-balancer-arn`, `--subnets`, `--subnet-mappings` | `--ip-address-type` | **DONE** (live ENI add/remove with rollback) |
+| `set-ip-address-type` | `--load-balancer-arn`, `--ip-address-type` (ipv4) | dualstack/IPv6 (rejected) | **DONE** |
+
 ### ELBv2 — Tags
 
 | Command | Implemented Flags | Missing Flags | Status |
@@ -695,18 +715,32 @@ The data plane uses a system-managed LB VM running HAProxy, launched automatical
 
 | Feature | Priority | Status |
 |---------|----------|--------|
-| Listener rules (CreateRule, DeleteRule, DescribeRules, ModifyRule) | High | **NOT STARTED** |
-| HTTPS/TLS termination (SSL certificates, policies, ALPN) | High | **NOT STARTED** |
-| Modify operations (`ModifyLoadBalancerAttributes`, `ModifyTargetGroup`, `ModifyTargetGroupAttributes`, `ModifyListener`) | Medium | **NOT STARTED** |
-| Connection draining (deregistration delay) | Medium | **NOT STARTED** |
-| Stickiness / session affinity | Medium | **NOT STARTED** |
-| Active health checking (API-driven, vs. HAProxy-only today) | Medium | **NOT STARTED** |
+| In-platform HTTPS/TLS termination (cert + SSL-policy APIs exist; data-plane TLS not terminated) | High | **NOT STARTED** |
+| ALPN policy, mutual TLS (mTLS) | Medium | **NOT STARTED** |
+| Listener attribute persistence (`Describe/ModifyListenerAttributes` are stubs) | Medium | **NOT STARTED** |
+| Active health checking (API-driven, vs. HAProxy/nginx-only today) | Medium | **NOT STARTED** |
 | IP and Lambda target types | Low | **NOT STARTED** |
 | S3 access log delivery | Low | **NOT STARTED** |
 | WAF integration | Low | **NOT STARTED** |
 
 ---
 
+## ACM (AWS Certificate Manager)
+
+Import-only — Spinifex stores externally-issued certificates for ELBv2 listener references; it does not issue certificates or validate domains (`RequestCertificate`). Certs are account-scoped; `describe`/`delete` enforce ownership.
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `import-certificate` | `--certificate`, `--private-key`, `--certificate-chain`, `--certificate-arn` (re-import) | `--tags` | **DONE** |
+| `describe-certificate` | `--certificate-arn` | — | **DONE** |
+| `list-certificates` | — | `--certificate-statuses`, `--includes`, `--max-items`, `--next-token` | **DONE** |
+| `delete-certificate` | `--certificate-arn` | — | **DONE** |
+| `request-certificate` | — | `--domain-name`, `--validation-method`, `--subject-alternative-names`, `--tags` | **NOT STARTED** |
+| `add-tags-to-certificate` / `list-tags-for-certificate` / `remove-tags-from-certificate` | — | `--certificate-arn`, `--tags`/`--tag-keys` | **NOT STARTED** |
+| `export-certificate` | — | `--certificate-arn`, `--passphrase` | **NOT STARTED** |
+
+---
+
 ## CloudWatch (Basic Monitoring)
 
 | Command | Implemented Flags | Missing Flags | Status |

docs/administration/spinifex-admin-cli/README.md

@@ -82,6 +82,25 @@ spx admin images import --name debian-13-arm64
 
 Catalog imports verify the image against the catalog-declared SHA-256/SHA-512 digest before extraction. Use `--file` to import operator-supplied media (verification skipped — operator is responsible for integrity), or `--force` to re-download after a checksum mismatch.
 
+### EKS node image
+
+To run EKS, import the prebuilt node image from the catalog:
+
+```bash
+spx admin images import --name spinifex-eks-node
+```
+
+This pulls the Alpine + K3s node AMI from `iso.mulgadc.com`, verifies its
+checksum, and registers it tagged `spinifex:managed-by=eks`. `eks create-cluster`
+and `eks create-nodegroup` resolve the boot AMI by that tag, so no further
+configuration is needed.
+
+Operators publishing a freshly built node image to the mirror run the
+**Publish EKS node AMI** GitHub Actions workflow (manual dispatch; uses the
+`CF_R2_*` repo secrets) or, locally, `make publish-eks-node-image` (see
+`scripts/publish-system-image.sh`; requires `R2_ENDPOINT` + R2-scoped
+`AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`).
+
 
 ## Cluster Shutdown
 

docs/compute/launching-instances/README.md

@@ -164,6 +164,16 @@ spx admin images import --name debian-13-x86_64
 
 cloud-init needs time to configure the instance after boot. Wait 30-60 seconds and retry.
 
+If the connection **times out** rather than being refused, the security group is likely blocking port 22. The default security group denies all inbound traffic (matching AWS), so SSH must be explicitly allowed:
+
+```bash
+# Allow SSH from anywhere on the instance's security group
+aws ec2 authorize-security-group-ingress \
+  --group-id $SG_ID --protocol tcp --port 22 --cidr 0.0.0.0/0
+```
+
+See [VPC Networking — Security Groups](/docs/vpc-networking#security-groups) for scoping rules to a trusted CIDR.
+
 Verify the SSH key was specified correctly when launching:
 
 ```bash

docs/compute/vpc-networking/README.md

@@ -48,8 +48,9 @@ NIC. Each host runs `ovn-controller` which programs OpenFlow rules on `br-int`
 
 ## Private vs Public Subnets
 
-A subnet's behavior depends on two things: whether the VPC has an Internet
-Gateway, and whether the subnet has `MapPublicIpOnLaunch` enabled.
+A subnet's behavior depends on three things: whether the VPC has an Internet
+Gateway, whether the subnet's route table has a default route to that IGW, and
+whether the subnet has `MapPublicIpOnLaunch` enabled.
 
 ## Private Subnet (Default)
 
@@ -61,9 +62,10 @@ internet or be reached from the WAN.
   <img src="../../../.github/assets/diagrams/vpc-private-subnet-flow.svg" alt="Private subnet — instance hits router, no default route, packet dropped" width="900">
 </p>
 
-If the VPC has an IGW attached, private subnet instances CAN reach the internet
-via the VPC router's SNAT rule (outbound only — they share the gateway IP). They
-still cannot be reached from the WAN because they have no public IP.
+Private subnet instances reach the internet only if their route table has a
+default route to the IGW (shared SNAT, outbound only — they share the gateway
+IP) or to a NAT gateway. With no default route, egress is dropped. Either way
+they cannot be reached from the WAN because they have no public IP.
 
 ## Public Subnet
 
@@ -77,8 +79,18 @@ managed by OVN — the instance OS only sees its private IP.
 **Requirements for a public subnet:**
 
 1. VPC has an Internet Gateway attached
-2. Subnet has `MapPublicIpOnLaunch = true`
-3. External IP pool configured in `spinifex.toml`
+2. A route table associated with the subnet has a `0.0.0.0/0` route to the IGW
+3. Subnet has `MapPublicIpOnLaunch = true`
+4. External IP pool configured in `spinifex.toml`
+
+Spinifex follows AWS route-table semantics: a subnet is only "public" if its
+effective route table carries a default route to the IGW. A new VPC's main
+route table has the local route only — Spinifex does **not** add the IGW route
+for you. Without it, the subnet's egress is gated with a drop policy, so
+instances cannot reach the internet (and inbound connections cannot complete
+because return traffic is dropped) even with a public IP and an attached IGW.
+Add the route explicitly — either to the main route table, or to a custom route
+table associated with the subnet (see [Quick Start](#3-create-vpc-with-public-subnet)).
 
 ## Comparison
 
@@ -86,7 +98,7 @@ managed by OVN — the instance OS only sees its private IP.
 | ------------------------ | --------------------------------- | ------------------------------ |
 | Private IP               | Yes                               | Yes                            |
 | Public IP                | No                                | Auto-assigned from pool        |
-| Outbound internet        | Only if VPC has IGW (shared SNAT) | Yes (own public IP via SNAT)   |
+| Outbound internet        | Only with a default route to IGW/NAT GW | Yes (own public IP via SNAT)   |
 | Inbound from WAN         | No                                | Yes (via 1:1 NAT to public IP) |
 | Instance sees public IP? | N/A                               | No — only sees private IP      |
 | Elastic IP support       | Only if explicitly associated     | Yes                            |
@@ -190,8 +202,7 @@ use `setup-ovn.sh --dhcp` to obtain one from the router. This is the router's
 DHCP — not Spinifex's internal OVN DHCP for VMs.
 
 **Use when:** VMs only need outbound access (apt update, pulling images). Edge
-deployments behind ISP NAT. Single WAN IP available. Future: use with AWS-style
-NAT Gateway for private subnet internet access.
+deployments behind ISP NAT. Single WAN IP available.
 
 ```toml
 [network]
@@ -935,17 +946,39 @@ spx admin init
 VPC=$(aws ec2 create-vpc --cidr-block 10.200.0.0/16 \
   --query Vpc.VpcId --output text)
 
-SUBNET=$(aws ec2 create-subnet --vpc-id $VPC \
-  --cidr-block 10.200.1.0/24 \
-  --query Subnet.SubnetId --output text)
-
 IGW=$(aws ec2 create-internet-gateway \
   --query InternetGateway.InternetGatewayId --output text)
 aws ec2 attach-internet-gateway \
   --internet-gateway-id $IGW --vpc-id $VPC
 
+SUBNET=$(aws ec2 create-subnet --vpc-id $VPC \
+  --cidr-block 10.200.1.0/24 \
+  --query Subnet.SubnetId --output text)
+
+# Route table — give the subnet a default route to the IGW. Spinifex does NOT
+# add this automatically; without it the subnet's egress is dropped.
+RT=$(aws ec2 create-route-table --vpc-id $VPC \
+  --query RouteTable.RouteTableId --output text)
+
+aws ec2 create-route --route-table-id $RT \
+  --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW
+
+aws ec2 associate-route-table --route-table-id $RT --subnet-id $SUBNET
+
+# Auto-assign a public IP to instances launched into the subnet
 aws ec2 modify-subnet-attribute \
   --subnet-id $SUBNET --map-public-ip-on-launch
+
+# Allow SSH + ICMP on the VPC's default security group (blocks inbound by default)
+SG=$(aws ec2 describe-security-groups \
+  --filters Name=vpc-id,Values=$VPC \
+  --query 'SecurityGroups[0].GroupId' --output text)
+
+aws ec2 authorize-security-group-ingress --group-id $SG \
+  --protocol tcp --port 22 --cidr 0.0.0.0/0
+
+aws ec2 authorize-security-group-ingress --group-id $SG \
+  --protocol icmp --port -1 --cidr 0.0.0.0/0
 ```
 
 ## 4. Launch Instance
@@ -1135,6 +1168,37 @@ ip route show
    sudo ovn-nbctl lr-nat-list vpc-$VPC
    ```
 
+### Instance Has Public IP But No Internet
+
+The instance shows a public IP in `describe-instances` but cannot reach the
+internet, and inbound connections hang. The usual cause is a missing default
+route: the subnet's effective route table has no `0.0.0.0/0` route to the IGW,
+so Spinifex gates the subnet with a drop policy.
+
+```bash
+# Find the route table that applies to the subnet and check its routes
+aws ec2 describe-route-tables \
+  --filters Name=association.subnet-id,Values=$SUBNET \
+  --query 'RouteTables[0].Routes'
+# Expect a route with DestinationCidrBlock 0.0.0.0/0 and a GatewayId of igw-...
+```
+
+If the subnet has no explicit association it falls back to the VPC's main route
+table — check that one too, then add the route to whichever table applies:
+
+```bash
+aws ec2 create-route --route-table-id $RT \
+  --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW
+```
+
+On the host, the drop policy installed when a subnet lacks an IGW route appears
+as a `Logical_Router_Policy` on the VPC router:
+
+```bash
+sudo ovn-nbctl lr-policy-list vpc-$VPC
+# A drop policy matching the subnet CIDR means the subnet is gated.
+```
+
 ### Public IP Not Reachable from WAN
 
 Work through these checks in order — each eliminates a class of issues.

docs/getting-started/setting-up-your-cluster/README.md

@@ -34,7 +34,7 @@ resources:
 ## Overview
 
 This guide assumes Spinifex is already installed and running. If not, follow one of the installation guides first:
-- [Single-Node Install)](/docs/install)
+- [Single-Node Install](/docs/install)
 - [Multi-Node Install](/docs/install-multi-node)
 - [Source Install](/docs/install-source)
 
@@ -50,7 +50,9 @@ export AWS_PROFILE=spinifex
 
 ## 1. Import an AMI
 
-List available images and import one matching your architecture:
+### Option A: Import a bundled image
+
+List the bundled images and import one matching your architecture:
 
 ```bash
 spx admin images list
@@ -64,13 +66,11 @@ ubuntu-26.04-arm64      | ubuntu | 26.04   | arm64  | uefi
 ubuntu-26.04-x86_64     | ubuntu | 26.04   | x86_64 | uefi
 ```
 
-Import an image:
-
 ```bash
 spx admin images import --name ubuntu-26.04-x86_64
 ```
 
-Or import a local image file:
+### Option B: Import a local image file
 
 ```bash
 spx admin images import --file ~/images/ubuntu-26.04.img --distro ubuntu --version 26.04 --arch x86_64 --boot-mode uefi
@@ -79,7 +79,7 @@ spx admin images import --file ~/images/ubuntu-26.04.img --distro ubuntu --versi
 Verify the import and note the AMI ID:
 
 ```bash
-aws ec2 describe-images
+AMI_ID=$(aws ec2 describe-images --query 'Images[0].ImageId' --output text)
 ```
 
 ## 2. Create an SSH Key
@@ -115,8 +115,6 @@ aws ec2 describe-key-pairs
 ```bash
 VPC_ID=$(aws ec2 create-vpc --cidr-block 10.200.0.0/16 \
   --query 'Vpc.VpcId' --output text)
-
-echo "VPC: $VPC_ID"
 ```
 
 ### Create an Internet Gateway
@@ -130,68 +128,95 @@ IGW_ID=$(aws ec2 create-internet-gateway \
 aws ec2 attach-internet-gateway \
   --internet-gateway-id $IGW_ID \
   --vpc-id $VPC_ID
-
-echo "IGW: $IGW_ID (attached to $VPC_ID)"
 ```
 
-### Create a Public Subnet
+### Create a Subnet
 
-A public subnet auto-assigns a routable IP to each instance, making it directly reachable from your network.
+Create the subnet your instances will launch into. The routing and public-IP steps below are what make it a *public* subnet.
 
 ```bash
 SUBNET_ID=$(aws ec2 create-subnet \
   --vpc-id $VPC_ID \
   --cidr-block 10.200.1.0/24 \
   --query 'Subnet.SubnetId' --output text)
+```
 
-# Enable auto-assign public IP
+### Create a Route Table
+
+A subnet is only **public** if its route table has a default route to the Internet Gateway. Spinifex does **not** add this route automatically — a new VPC's route table only routes traffic within the VPC (matching AWS). Without a `0.0.0.0/0` route to the IGW, instances in the subnet cannot reach the internet (and inbound connections cannot complete) even with a public IP assigned.
+
+```bash
+RT_ID=$(aws ec2 create-route-table \
+  --vpc-id $VPC_ID \
+  --query 'RouteTable.RouteTableId' --output text)
+
+# Default route to the internet via the IGW
+aws ec2 create-route \
+  --route-table-id $RT_ID \
+  --destination-cidr-block 0.0.0.0/0 \
+  --gateway-id $IGW_ID
+
+# Associate the route table with the subnet
+aws ec2 associate-route-table \
+  --route-table-id $RT_ID \
+  --subnet-id $SUBNET_ID
+```
+
+### Enable Auto-Assign Public IP
+
+Give every instance launched into the subnet a routable public IP, making it directly reachable from your network.
+
+```bash
 aws ec2 modify-subnet-attribute \
   --subnet-id $SUBNET_ID \
   --map-public-ip-on-launch
-
-echo "Subnet: $SUBNET_ID (public)"
 ```
 
+### Allow SSH and ICMP
+
+Every VPC gets a default security group that **blocks inbound traffic from outside the group** (matching AWS). Instances launched without an explicit security group use this default, so you must authorize ingress before you can SSH or ping the instance.
+
+```bash
+SG_ID=$(aws ec2 describe-security-groups \
+  --filters Name=vpc-id,Values=$VPC_ID \
+  --query 'SecurityGroups[0].GroupId' --output text)
+
+# Allow SSH from anywhere
+aws ec2 authorize-security-group-ingress \
+  --group-id $SG_ID \
+  --protocol tcp --port 22 --cidr 0.0.0.0/0
+
+# Allow ICMP (ping) from anywhere
+aws ec2 authorize-security-group-ingress \
+  --group-id $SG_ID \
+  --protocol icmp --port -1 --cidr 0.0.0.0/0
+```
+
+**Note:** `0.0.0.0/0` opens these ports to every source. For anything beyond a quick evaluation, scope the `--cidr` to a trusted range. Rule changes apply immediately — no instance restart needed.
+
 ### Verify
 
 ```bash
 aws ec2 describe-vpcs --vpc-ids $VPC_ID
 aws ec2 describe-subnets --subnet-ids $SUBNET_ID
+aws ec2 describe-route-tables --route-table-ids $RT_ID
+aws ec2 describe-security-groups --group-ids $SG_ID
 ```
 
 ## 4. Launch an Instance
 
-Query available instance types for your hardware:
+Launch an instance in the public subnet on a `t3.micro` (2 vCPU / 1 GiB).
+
+> **Note:** On an **arm64** host, use `t4g.micro` instead of `t3.micro`.
 
 ```bash
-aws ec2 describe-instance-types
-```
-
-Launch an instance in the public subnet, note this will select an instance type with 2 VCPU and 1024MB of memory.
-
-> **Note:** Replace `AMI_NAME` with the previously imported image above with the `ami-` prefix.
-
-```bash
-AMI_NAME="ami-ubuntu-26.04-x86_64"
-
-AMI_ID=$(aws ec2 describe-images \
-  --filters "Name=name,Values=$AMI_NAME" \
-  --query 'Images | sort_by(@, &CreationDate) | [-1].ImageId' \
-  --output text)
-
-INSTANCE_TYPE=$(aws ec2 describe-instance-types \
-  --query "sort_by(InstanceTypes[?VCpuInfo.DefaultVCpus==\`2\` && MemoryInfo.SizeInMiB>=\`1024\`], &MemoryInfo.SizeInMiB)[0].InstanceType" \
-  --output text)
-
 INSTANCE_ID=$(aws ec2 run-instances \
   --image-id $AMI_ID \
-  --instance-type $INSTANCE_TYPE \
+  --instance-type t3.micro \
   --key-name spinifex-key \
   --subnet-id $SUBNET_ID \
   --count 1 \
   --query 'Instances[0].InstanceId' --output text)
-
-echo "Instance: $INSTANCE_ID"
 ```
 
 Wait for the instance to reach `running` state:
@@ -328,7 +353,36 @@ PRIVATE_SUBNET=$(aws ec2 create-subnet \
   --query 'Subnet.SubnetId' --output text)
 ```
 
-Instances in private subnets get a private IP only. They can still reach the internet via SNAT through the VPC router, but are not directly reachable from your network.
+Instances in a private subnet get a private IP only and are not reachable from your network. By default they also have **no internet access**: with no `0.0.0.0/0` route in the subnet's route table, Spinifex gates egress with a drop policy — matching AWS, where a private subnet has no route off the VPC. They can still reach other instances in the same VPC.
+
+To give a private subnet outbound-only internet access, deploy a NAT Gateway in a public subnet and point the private subnet's default route at it (the AWS pattern):
+
+```bash
+# Allocate an Elastic IP for the NAT Gateway
+NAT_EIP=$(aws ec2 allocate-address --query AllocationId --output text)
+
+# Create the NAT Gateway in the PUBLIC subnet (the one with the IGW route)
+NATGW_ID=$(aws ec2 create-nat-gateway \
+  --subnet-id $SUBNET_ID \
+  --allocation-id $NAT_EIP \
+  --query 'NatGateway.NatGatewayId' --output text)
+
+# Give the private subnet its own route table with a default route to the NAT Gateway
+PRIVATE_RT=$(aws ec2 create-route-table \
+  --vpc-id $VPC_ID \
+  --query 'RouteTable.RouteTableId' --output text)
+
+aws ec2 create-route \
+  --route-table-id $PRIVATE_RT \
+  --destination-cidr-block 0.0.0.0/0 \
+  --nat-gateway-id $NATGW_ID
+
+aws ec2 associate-route-table \
+  --route-table-id $PRIVATE_RT \
+  --subnet-id $PRIVATE_SUBNET
+```
+
+Instances in the private subnet now reach the internet outbound through the NAT Gateway's public IP, but remain unreachable from the WAN.
 
 ### Multiple Accounts
 
@@ -348,7 +402,7 @@ journalctl -u spinifex-daemon -f
 aws ec2 describe-images
 ```
 
-### SSH Connection Refused
+### SSH Connection Refused or Times Out
 
 cloud-init needs 30-60 seconds after boot. Check instance state:
 
@@ -356,6 +410,20 @@ cloud-init needs 30-60 seconds after boot. Check instance state:
 aws ec2 describe-instances --instance-ids $INSTANCE_ID
 ```
 
+A connection that **times out** (rather than being refused) usually means the security group is blocking port 22. Confirm the default security group allows SSH ingress:
+
+```bash
+aws ec2 describe-security-groups --group-ids $SG_ID \
+  --query 'SecurityGroups[0].IpPermissions'
+```
+
+If there are no rules for TCP 22, authorize it (see [Allow SSH and ICMP](#allow-ssh-and-icmp)):
+
+```bash
+aws ec2 authorize-security-group-ingress \
+  --group-id $SG_ID --protocol tcp --port 22 --cidr 0.0.0.0/0
+```
+
 ### No Public IP Assigned
 
 Verify the subnet has `MapPublicIpOnLaunch` enabled:
@@ -378,7 +446,23 @@ aws ec2 describe-internet-gateways
 
 ### Instance Has No Internet Access
 
-Check the VPC router's NAT rules (from the host):
+First confirm the subnet's route table has a default route to the IGW. Spinifex gates a subnet's egress with a drop policy when this route is missing, so an instance with a public IP still cannot reach the internet:
+
+```bash
+aws ec2 describe-route-tables \
+  --filters Name=association.subnet-id,Values=$SUBNET_ID \
+  --query 'RouteTables[0].Routes'
+# Expect a 0.0.0.0/0 route with a GatewayId of igw-...
+```
+
+If the route is missing, add it (see [Create a Route Table](#create-a-route-table)):
+
+```bash
+aws ec2 create-route --route-table-id $RT_ID \
+  --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW_ID
+```
+
+If the route is present, check the VPC router's NAT rules (from the host):
 
 ```bash
 sudo ovn-nbctl lr-nat-list $(sudo ovn-nbctl lr-list | awk '{print $2}' | head -1)

docs/install-vm-eval/README.md

@@ -12,8 +12,6 @@ For this tutorial download UTM.app on Mac to simplify the VM installation.
 
 * UTM - [https://mac.getutm.app](https://mac.getutm.app)
 
-
-
 ## Download Debian ISO
 
 Next, download the Debian ISO (ARM64, Apple Silicon)
@@ -28,7 +26,7 @@ Using the VM console, login using the `spinifex` user and your specified passwor
 
 For a minimal Debian install additional dependencies are required prior to the Spinifex installation:
 
-```
+```bash
 su root
 apt update
 apt install sudo curl bridge-utils
@@ -38,7 +36,7 @@ apt install sudo curl bridge-utils
 
 Enable sudo for `spinifex` user as the root user:
 
-```
+```bash
 su root
 echo "spinifex ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/spinifex
 ```

docs/reference-architectures/SMCI-H14-orchestration/README.md

@@ -26,6 +26,8 @@ sections:
 
 ## Overview
 
+Spinifex is an open-source infrastructure platform that brings core AWS services including EC2, EBS and S3 to bare-metal, edge, and on-prem environments. It exposes an EC2-compatible API, so any tooling that works against AWS (the `aws` CLI, Terraform, SDKs) works against a Spinifex node unchanged, with a single profile swap.
+
 This document serves as a tutorial for setting up multi-tenant AI workloads with Spinifex. The reference architecture for this tutorial is the Supermicro H14 platform - specifications listed below.
 
 The H14 chassis comes equipped with **8× AMD Instinct MI350X** (288 GB HBM3e each, 2.3 TB total). We used Spinifex to provision three isolated VMs, each assigned 2× MI350X via direct PCIe passthrough, and ran three computationally intensive workloads simultaneously: a YOLO11x vision model, a Qwen3-VL 235B FP8 vision language model, and a Llama 3.3 70B large language model.

docs/reference-architectures/arcee-trinity-mi350x/README.md

@@ -31,6 +31,8 @@ sections:
 
 ## Overview
 
+Spinifex is an open-source infrastructure platform that brings core AWS services including EC2, EBS and S3 to bare-metal, edge, and on-prem environments. It exposes an EC2-compatible API, so any tooling that works against AWS (the `aws` CLI, Terraform, SDKs) works against a Spinifex node unchanged, with a single profile swap.
+
 ### About Trinity
 
 [Trinity](https://www.arcee.ai/trinity) is a family of open-weight language models from [Arcee AI](https://www.arcee.ai), built specifically for production agentic workloads: reliable tool calling, structured JSON output, coherent multi-turn conversations, and long-context reasoning. The family spans four sizes — Nano (6B), Mini (26B), Large Preview (400B), and Large Thinking (400B reasoning-optimised), with consistent capabilities and API surface across all of them. A workflow validated on Nano can be promoted to Large with no prompt changes.

docs/service-interfaces.yaml

@@ -30,10 +30,12 @@ services:
     publishes:
       - "ec2.*"
       - "elbv2.*"
+      - "acm.*"
       - "iam.*"
     depends_on:
       - "ec2.*"
       - "elbv2.*"
+      - "acm.*"
       - "iam.*"
 
   spinifex_daemon:
@@ -50,6 +52,7 @@ services:
       - spinifex/daemon
       - spinifex/handlers/ec2
       - spinifex/handlers/elbv2
+      - spinifex/handlers/acm
       - spinifex/handlers/iam
     subscribes:
       # EC2 instance lifecycle.
@@ -81,6 +84,12 @@ services:
       - "spinifex.node.vms"
       - "spinifex.nodes.discover"
       - "spinifex.cluster.shutdown.progress"
+      # Coordinated cluster-shutdown phases (fan-out, no queue group).
+      - "spinifex.cluster.shutdown.gate"
+      - "spinifex.cluster.shutdown.drain"
+      - "spinifex.cluster.shutdown.storage"
+      - "spinifex.cluster.shutdown.persist"
+      - "spinifex.cluster.shutdown.infra"
       - "spinifex.storage.config"
       # ELBv2 control plane (handlers/elbv2/service_nats.go topic map).
       - "elbv2.CreateLoadBalancer"
@@ -95,6 +104,15 @@ services:
       - "elbv2.CreateListener"
       - "elbv2.DeleteListener"
       - "elbv2.DescribeListeners"
+      # ACM-alike certificate store (handlers/acm/service_nats.go topic map).
+      - "acm.ImportCertificate"
+      - "acm.DescribeCertificate"
+      - "acm.ListCertificates"
+      - "acm.DeleteCertificate"
+      - "elbv2.AddListenerCertificates"
+      - "elbv2.RemoveListenerCertificates"
+      - "elbv2.DescribeListenerCertificates"
+      - "elbv2.DescribeSSLPolicies"
     publishes:
       # → vpcd
       - "vpc.create"
@@ -121,6 +139,8 @@ services:
       - "ebs.delete"
       - "ebs.sync"
       - "ebs.snapshot.*"
+      - "ebs.config"
+      - "ebs.config.*"
       # → predastore (S3 / AMI / keypair store)
       - "s3.*"
       # IAM lifecycle event
@@ -169,6 +189,8 @@ services:
       - "ebs.delete"
       - "ebs.sync"
       - "ebs.snapshot.*"
+      - "ebs.config"
+      - "ebs.config.*"
     publishes:
       - "ebs.mount.response"
       - "ebs.unmount.response"

docs/terraform-workbooks/eks-addons-access/README.md

@@ -0,0 +1,181 @@
+---
+title: "EKS with Addons + Access-Entry RBAC"
+description: "Extend a single-node EKS cluster with a managed CoreDNS addon, access-entry RBAC, and a browsable load-balanced demo app, using Terraform on Spinifex."
+category: "Terraform Workbooks"
+tags:
+  - terraform
+  - eks
+  - kubernetes
+  - addons
+  - rbac
+  - iam
+  - workbook
+resources:
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "Terraform Kubernetes Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "OpenTofu"
+    url: "https://opentofu.org/"
+---
+
+# Terraform: EKS with Addons + Access-Entry RBAC
+
+> Builds on the quickstart by installing a managed addon (CoreDNS) and granting a second IAM principal read-only cluster access — then deploys the same browsable, load-balanced demo app so you can see the result.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+This workbook takes the same single-node cluster (and the same browsable demo app) from [EKS Quickstart](../eks-quickstart) and adds the two things you reach for on almost any real cluster:
+
+1. **A managed addon.** CoreDNS is installed through the EKS API (`aws_eks_addon`) rather than `kubectl apply`, so its lifecycle is managed by the cluster. The demo app is deployed *after* CoreDNS, so a healthy page also confirms in-cluster DNS is working.
+2. **Access-entry RBAC.** A second IAM role is granted read-only access via an `aws_eks_access_entry` plus an `aws_eks_access_policy_association` bound to the AWS-managed `AmazonEKSViewPolicy`. This is the modern access-entry path (`authentication_mode = "API"`) — no `aws-auth` ConfigMap.
+
+When `apply` finishes, open the **`demo_url`** output and refresh — the `nginxdemos/hello` page reports which pod served each request.
+
+**What you'll learn:**
+
+- Installing a managed EKS addon with `aws_eks_addon`
+- Registering an IAM principal with `aws_eks_access_entry` and scoping it with `aws_eks_access_policy_association`
+- The difference between the cluster creator's admin access and a scoped viewer
+- Deploying a workload with the Kubernetes provider once the addon is healthy
+
+**What gets created**
+
+| Resource | Name | Purpose |
+|---|---|---|
+| VPC | `eks-addons-vpc` | Isolated network (10.31.0.0/16) |
+| Subnets | `eks-addons-subnet-a/-b` | Public subnets for the cluster and worker |
+| Internet Gateway | `eks-addons-igw` | Egress so the worker can pull the demo image |
+| IAM Roles | `eks-addons-cluster-role`, `eks-addons-node-role` | Control-plane and worker roles |
+| IAM Role | `eks-addons-viewer` | Stand-in principal granted read-only access |
+| EKS Cluster | `eks-addons` | Public endpoint, API auth mode |
+| Node Group | `default` | One `t3.medium` worker |
+| Addon | `coredns` | Cluster DNS, managed via the EKS API |
+| Access Entry + Assoc. | `eks-addons-viewer` → `AmazonEKSViewPolicy` | Read-only, cluster scope |
+| SG Ingress Rule | `eks-addons-demo-nodeport` | Opens the demo NodePort on the worker SG |
+| K8s Deployment + Service | `hello` | `nginxdemos/hello`, 2 replicas, NodePort |
+
+**Spinifex specifics**
+
+- Addon catalog is limited. Spinifex currently serves `coredns`, `aws-load-balancer-controller` and `spinifex-noop`. There is no `vpc-cni`, `kube-proxy`, `ebs-csi` or `metrics-server`.
+- **Leave `addon_version` unset.** Spinifex defaults an unset version to its catalog default (`coredns` 1.11.1). Pinning is a dead end: the AWS provider rejects a bare `1.11.1` (it requires the `v`-prefixed `v1.11.1-eksbuild.N` form), while Spinifex's catalog only accepts the bare `1.11.1` — so no single string satisfies both.
+- Cluster access policies are the four AWS-managed `cluster-access-policy/Amazon EKS{ClusterAdmin,Admin,Edit,View}Policy` ARNs.
+- The worker SG is auto-managed and the worker AMI is always the `eks-node` image; the NodePort is exposed by adding one rule to the looked-up worker SG.
+
+**Prerequisites:**
+
+- Spinifex installed and running
+- The Spinifex `eks-node` image available on the cluster
+- OpenTofu or Terraform, plus `kubectl` and the AWS CLI
+
+## Instructions
+
+### Step 1. Get the Template
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-addons-access
+```
+
+Or create a `main.tf` file and paste the full configuration below.
+
+<!-- INCLUDE: main.tf lang:hcl -->
+
+### Step 2. Deploy
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+```
+
+Allow a few minutes for the cluster to reach `ACTIVE` and the worker to register. CoreDNS installs onto the worker, then Terraform deploys the demo app.
+
+> Keep `AWS_PROFILE=spinifex` exported for the whole `apply` — the Kubernetes provider runs `aws eks get-token` against the Spinifex STS endpoint.
+
+### Step 3. Open the Demo
+
+```bash
+tofu output demo_url
+```
+
+Open it in a browser and refresh — the `nginxdemos/hello` page shows the pod that answered, alternating between the two replicas.
+
+### Step 4. Verify the Addon and Access Entry
+
+```bash
+aws eks list-addons --cluster-name eks-addons
+aws eks describe-addon --cluster-name eks-addons --addon-name coredns --query 'addon.status'
+
+aws eks list-access-entries --cluster-name eks-addons
+aws eks list-associated-access-policies \
+  --cluster-name eks-addons \
+  --principal-arn "$(tofu output -raw viewer_principal_arn)"
+```
+
+The viewer principal is bound to `AmazonEKSViewPolicy` at cluster scope — it can read across the cluster but cannot mutate anything. The principal that ran `apply` keeps full cluster-admin from `bootstrap_cluster_creator_admin_permissions`.
+
+### Cleanup
+
+```bash
+tofu destroy
+```
+
+## Troubleshooting
+
+### Demo URL Doesn't Load
+
+Same chain as the quickstart — check the worker's public IP, that the demo pods are `Running`, and that the NodePort rule landed on the worker SG:
+
+```bash
+kubectl get pods -o wide
+aws ec2 describe-security-groups --filters "Name=group-name,Values=eks-cluster-eks-addons-nodegroup-sg" \
+  --query 'SecurityGroups[0].IpPermissions'
+```
+
+If the demo pods are stuck `ContainerCreating` with DNS errors, confirm CoreDNS is healthy first:
+
+```bash
+kubectl -n kube-system get deploy coredns
+```
+
+### Addon Stuck in CREATING / DEGRADED
+
+CoreDNS needs a schedulable worker. Confirm the node group is `ACTIVE` and the node is `Ready`:
+
+```bash
+aws eks describe-nodegroup --cluster-name eks-addons --nodegroup-name default --query 'nodegroup.status'
+kubectl get nodes
+```
+
+### Addon Version Rejected
+
+This workbook leaves `addon_version` unset so Spinifex picks its catalog default. If you add an explicit `addon_version`, expect a wall: the AWS provider requires the `v`-prefixed `v1.11.1-eksbuild.N` form, but Spinifex's catalog only knows the bare `1.11.1` — a value that passes one is rejected by the other. Prefer omitting the field.
+
+### Access Entry Not Taking Effect
+
+Assume the viewer role and confirm it can read but not write:
+
+```bash
+kubectl auth can-i list pods --all-namespaces   # yes
+kubectl auth can-i create deployments           # no
+```
+
+### Provider Connection Refused
+
+```bash
+sudo systemctl status spinifex.target
+curl -k https://localhost:9999/
+```

docs/terraform-workbooks/eks-addons-access/main.tf

@@ -0,0 +1,468 @@
+# Example: EKS with an Addon + Access-Entry RBAC (with a visible demo app)
+#
+# Builds on eks-quickstart: same single-worker cluster and browsable demo app,
+# but adds two things you reach for on a real cluster:
+#
+#   * a managed addon (CoreDNS) installed through the EKS API, and
+#   * a second IAM principal granted scoped, read-only access via an EKS access
+#     entry + access-policy association (authentication_mode = "API").
+#
+# After `apply`, open the demo_url output — the page reports the pod that
+# served it, so a refresh shows requests landing on different replicas. CoreDNS
+# being healthy is what lets in-cluster name resolution work.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-addons-access
+#   export AWS_PROFILE=spinifex
+#   tofu init && tofu apply
+#   # then open the demo_url output
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.20"
+    }
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Variables
+# ---------------------------------------------------------------------------
+
+variable "region" {
+  type    = string
+  default = "ap-southeast-2"
+}
+
+variable "cluster_name" {
+  type    = string
+  default = "eks-addons"
+}
+
+variable "k8s_version" {
+  type    = string
+  default = "1.32"
+}
+
+variable "instance_type" {
+  type    = string
+  default = "t3.medium"
+}
+
+variable "node_port" {
+  type    = number
+  default = 30080
+}
+
+variable "replicas" {
+  type    = number
+  default = 2
+}
+
+variable "browse_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the demo NodePort; tighten to your own IP in production"
+}
+
+variable "spinifex_endpoint" {
+  type    = string
+  default = "https://127.0.0.1:9999"
+}
+
+# ---------------------------------------------------------------------------
+# Providers
+# ---------------------------------------------------------------------------
+
+provider "aws" {
+  region = var.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+provider "kubernetes" {
+  host                   = aws_eks_cluster.this.endpoint
+  cluster_ca_certificate = base64decode(aws_eks_cluster.this.certificate_authority[0].data)
+
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    command     = "aws"
+    args        = ["eks", "get-token", "--cluster-name", aws_eks_cluster.this.name, "--region", var.region]
+  }
+}
+
+data "aws_availability_zones" "available" {
+  state = "available"
+}
+
+# ---------------------------------------------------------------------------
+# VPC + two public subnets
+# ---------------------------------------------------------------------------
+
+resource "aws_vpc" "main" {
+  cidr_block           = "10.31.0.0/16"
+  enable_dns_hostnames = true
+  enable_dns_support   = true
+
+  tags = {
+    Name = "${var.cluster_name}-vpc"
+  }
+}
+
+resource "aws_internet_gateway" "igw" {
+  vpc_id = aws_vpc.main.id
+
+  tags = {
+    Name = "${var.cluster_name}-igw"
+  }
+}
+
+resource "aws_subnet" "a" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.31.1.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-subnet-a"
+  }
+}
+
+resource "aws_subnet" "b" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.31.2.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-subnet-b"
+  }
+}
+
+resource "aws_route_table" "public" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block = "0.0.0.0/0"
+    gateway_id = aws_internet_gateway.igw.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-public-rt"
+  }
+}
+
+resource "aws_route_table_association" "a" {
+  subnet_id      = aws_subnet.a.id
+  route_table_id = aws_route_table.public.id
+}
+
+resource "aws_route_table_association" "b" {
+  subnet_id      = aws_subnet.b.id
+  route_table_id = aws_route_table.public.id
+}
+
+# ---------------------------------------------------------------------------
+# IAM — cluster role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "cluster" {
+  name = "${var.cluster_name}-cluster-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "eks.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "cluster" {
+  role       = aws_iam_role.cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — node role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "node" {
+  name = "${var.cluster_name}-node-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_worker" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_cni" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_ecr" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — a second principal to grant read-only cluster access to
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "viewer" {
+  name = "${var.cluster_name}-viewer"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+# ---------------------------------------------------------------------------
+# EKS cluster
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_cluster" "this" {
+  name     = var.cluster_name
+  role_arn = aws_iam_role.cluster.arn
+  version  = var.k8s_version
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.a.id, aws_subnet.b.id]
+    endpoint_public_access  = true
+    endpoint_private_access = false
+  }
+
+  access_config {
+    authentication_mode                         = "API"
+    bootstrap_cluster_creator_admin_permissions = true
+  }
+
+  depends_on = [aws_iam_role_policy_attachment.cluster]
+
+  tags = {
+    Name = var.cluster_name
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Node group
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_node_group" "default" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "default"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.a.id, aws_subnet.b.id]
+
+  scaling_config {
+    desired_size = 1
+    min_size     = 1
+    max_size     = 2
+  }
+
+  instance_types = [var.instance_type]
+  ami_type       = "AL2_x86_64"
+
+  depends_on = [
+    aws_iam_role_policy_attachment.node_worker,
+    aws_iam_role_policy_attachment.node_cni,
+    aws_iam_role_policy_attachment.node_ecr,
+  ]
+
+  tags = {
+    Name = "${var.cluster_name}-default"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Addon — CoreDNS
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_addon" "coredns" {
+  cluster_name                = aws_eks_cluster.this.name
+  addon_name                  = "coredns"
+  resolve_conflicts_on_create = "OVERWRITE"
+
+  # addon_version omitted on purpose. Spinifex defaults an unset version to its
+  # catalog default (coredns 1.11.1). The AWS provider rejects a bare "1.11.1"
+  # (it demands a v-prefixed form) which Spinifex's catalog would in turn
+  # reject — so pinning is a dead end here; let the server choose.
+  depends_on = [aws_eks_node_group.default]
+
+  tags = {
+    Name = "${var.cluster_name}-coredns"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Access entry — grant the viewer principal read-only access
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_access_entry" "viewer" {
+  cluster_name  = aws_eks_cluster.this.name
+  principal_arn = aws_iam_role.viewer.arn
+  type          = "STANDARD"
+}
+
+resource "aws_eks_access_policy_association" "viewer" {
+  cluster_name  = aws_eks_cluster.this.name
+  principal_arn = aws_iam_role.viewer.arn
+  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
+
+  access_scope {
+    type = "cluster"
+  }
+
+  depends_on = [aws_eks_access_entry.viewer]
+}
+
+# ---------------------------------------------------------------------------
+# Open the NodePort on the auto-managed nodegroup SG (see eks-quickstart)
+# ---------------------------------------------------------------------------
+
+data "aws_security_group" "nodegroup" {
+  filter {
+    name   = "group-name"
+    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
+  }
+
+  filter {
+    name   = "vpc-id"
+    values = [aws_vpc.main.id]
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+resource "aws_vpc_security_group_ingress_rule" "nodeport" {
+  security_group_id = data.aws_security_group.nodegroup.id
+  cidr_ipv4         = var.browse_cidr
+  from_port         = var.node_port
+  to_port           = var.node_port
+  ip_protocol       = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-demo-nodeport"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Demo workload — deployed after CoreDNS so name resolution is up
+# ---------------------------------------------------------------------------
+
+resource "kubernetes_deployment_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+    labels    = { app = "hello" }
+  }
+
+  spec {
+    replicas = var.replicas
+
+    selector {
+      match_labels = { app = "hello" }
+    }
+
+    template {
+      metadata {
+        labels = { app = "hello" }
+      }
+
+      spec {
+        container {
+          name  = "hello"
+          image = "nginxdemos/hello"
+
+          port {
+            container_port = 80
+          }
+        }
+      }
+    }
+  }
+
+  depends_on = [aws_eks_addon.coredns]
+}
+
+resource "kubernetes_service_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+  }
+
+  spec {
+    selector = { app = "hello" }
+    type     = "NodePort"
+
+    port {
+      port        = 80
+      target_port = 80
+      node_port   = var.node_port
+    }
+  }
+
+  depends_on = [kubernetes_deployment_v1.hello]
+}
+
+data "aws_instances" "workers" {
+  instance_tags = {
+    "spinifex:eks-cluster" = aws_eks_cluster.this.name
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+# ---------------------------------------------------------------------------
+# Outputs
+# ---------------------------------------------------------------------------
+
+output "cluster_name" {
+  value = aws_eks_cluster.this.name
+}
+
+output "viewer_principal_arn" {
+  value = aws_iam_role.viewer.arn
+}
+
+output "demo_url" {
+  value       = "http://${data.aws_instances.workers.public_ips[0]}:${var.node_port}"
+  description = "Open in a browser; refresh to see different pods answer"
+}
+
+output "update_kubeconfig" {
+  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
+}

docs/terraform-workbooks/eks-https-ingress/README.md

@@ -0,0 +1,227 @@
+---
+title: "3-Node EKS behind an HTTPS ALB (ACM TLS)"
+description: "Provision an HA-shaped EKS cluster with public + private API endpoints and three workers, deploy a demo app, and publish it through an internet-facing ALB that terminates TLS with an ACM-imported certificate, using Terraform on Spinifex."
+category: "Terraform Workbooks"
+tags:
+  - terraform
+  - eks
+  - kubernetes
+  - acm
+  - tls
+  - alb
+  - elbv2
+  - vpc
+  - workbook
+resources:
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "Terraform Kubernetes Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "OpenTofu"
+    url: "https://opentofu.org/"
+---
+
+# Terraform: 3-Node EKS behind an HTTPS ALB
+
+> The largest EKS workbook, and the clearest demo. An HA-shaped cluster with public + private API endpoints and three workers, running a demo app published over HTTPS through an internet-facing ALB that terminates TLS with an ACM-imported certificate.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+This workbook is sized for a **3-node Spinifex deployment** and is built to show off. It brings together the pieces from the smaller EKS examples, deploys a demo app across three workers, and fronts it with a TLS-terminating load balancer. When `apply` finishes, fetch the ALB's public IP and open `https://<ip>` — the `nginxdemos/hello` page reports which pod answered, and refreshing moves between the three replicas spread across the cluster. That single page demonstrates a multi-node Kubernetes cluster load-balancing behind HTTPS.
+
+- **Public + private API endpoints.** `endpoint_public_access` serves `kubectl` and the Kubernetes provider from your host (CIDR-restricted); `endpoint_private_access` exposes an in-VPC private endpoint the workers use to join.
+- **Private workers + NAT.** The three workers sit in private subnets. They *join* the cluster over the in-VPC private endpoint, and a NAT gateway gives them outbound access to pull the demo container image.
+- **CoreDNS** installed as a managed addon.
+- **TLS via ACM.** A self-signed certificate is generated with the `tls` provider and **imported into ACM** (Spinifex ACM supports `ImportCertificate`, not `RequestCertificate`). The ALB's HTTPS listener references the ACM ARN and forwards to the workers' NodePort.
+- **The app, deployed by Terraform.** The Kubernetes provider deploys the Deployment + NodePort Service, and one security-group rule lets the ALB reach the NodePort.
+
+<p align="center">
+  <img src="../../../.github/assets/diagrams/tf-eks-https-ingress.svg" alt="3-node EKS — public+private API endpoints, ALB in public subnets terminating TLS, three workers in private subnets behind a NAT gateway" width="900">
+</p>
+
+**What you'll learn:**
+
+- Running an EKS cluster with both public and private API endpoints
+- Placing workers in private subnets that join over the private endpoint and egress via NAT
+- Importing a certificate into ACM and terminating TLS on an ALB HTTPS listener
+- Wiring an ALB target group to managed node-group workers via a NodePort, including the SG rule that allows it
+- Deploying a load-balanced workload with the Kubernetes provider
+
+**What gets created**
+
+| Resource | Name | Purpose |
+|---|---|---|
+| VPC | `eks-https-vpc` | Isolated network (10.32.0.0/16) |
+| Public Subnets | `eks-https-public-a/-b` | Host the ALB and NAT gateway |
+| Private Subnets | `eks-https-private-a/-b` | Host the workers |
+| Internet Gateway | `eks-https-igw` | Internet for the public subnets |
+| NAT Gateway (+ EIP) | `eks-https-nat` | Outbound for the private workers (image pulls) |
+| Route Tables | `eks-https-public-rt`, `eks-https-private-rt` | Public via IGW; private via NAT |
+| IAM Roles | `eks-https-cluster-role`, `eks-https-node-role` | Control-plane and worker roles |
+| EKS Cluster | `eks-https` | Public (CIDR-restricted) + private endpoints |
+| Node Group | `workers` | Three `t3.large` workers in the private subnets |
+| Addon | `coredns` | Cluster DNS |
+| ACM Cert | `eks-https-ingress` | Self-signed cert imported into ACM |
+| ALB + HTTPS Listener | `eks-https-alb` | Internet-facing, terminates TLS on :443 |
+| Target Group | `eks-https-tg` | Instance targets on the NodePort |
+| SG Ingress Rule | `eks-https-alb-nodeport` | Lets the ALB SG reach the worker NodePort |
+| K8s Deployment + Service | `hello` | `nginxdemos/hello`, 3 replicas, NodePort |
+
+**Spinifex specifics**
+
+- ACM supports `ImportCertificate`, `DescribeCertificate`, `ListCertificates`, `DeleteCertificate` — there is no `RequestCertificate`/DNS-validation flow, hence the self-signed import.
+- The ALB HTTPS listener resolves the cert from ACM. Valid SSL policies are `ELBSecurityPolicy-2016-08` (default) and `ELBSecurityPolicy-TLS13-1-2-2021-06`.
+- The worker SG is auto-managed (`vpc_config.security_group_ids` is ignored) and admits only intra-cluster traffic. The ALB→NodePort rule is added by looking the SG up by its deterministic name and admitting the ALB SG.
+- Worker discovery uses the `spinifex:eks-cluster` tag Spinifex stamps on node-group instances; the target-group attachment count is the known `node_desired_size`.
+- Leave `addon_version` unset (see the addons workbook for why pinning it can't satisfy both the provider and the catalog).
+
+**Prerequisites:**
+
+- A 3-node Spinifex deployment, installed and running
+- The Spinifex `eks-node` image available on the cluster
+- OpenTofu or Terraform, plus `kubectl` and the AWS CLI
+
+## Instructions
+
+### Step 1. Get the Template
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-https-ingress
+```
+
+Or create a `main.tf` file and paste the full configuration below.
+
+<!-- INCLUDE: main.tf lang:hcl -->
+
+### Step 2. Deploy
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+```
+
+Expect several minutes: the control plane bootstraps, three workers launch and join over the private endpoint, the NAT gateway comes up, CoreDNS installs, the ALB is provisioned, and the Kubernetes provider rolls out the demo app.
+
+> **Keep `AWS_PROFILE=spinifex` exported** for the whole `apply` — the Kubernetes provider authenticates with `aws eks get-token` against the Spinifex STS endpoint.
+
+> **Tighten access in production.** `api_public_access_cidr` and `alb_ingress_cidr` both default to `0.0.0.0/0`. Set them to your own CIDR:
+>
+> ```bash
+> export TF_VAR_api_public_access_cidr="203.0.113.10/32"
+> export TF_VAR_alb_ingress_cidr="203.0.113.10/32"
+> ```
+
+### Step 3. Open the Demo over HTTPS
+
+The ALB DNS name (`*.elb.spinifex.local`) won't resolve from your host, so fetch its public IP:
+
+```bash
+ALB_IP=$(aws elbv2 describe-load-balancers --names eks-https-alb \
+  --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
+  --output text)
+```
+
+The certificate is self-signed, so a browser will warn (proceed anyway) and `curl` needs `-k`:
+
+```bash
+curl -k https://$ALB_IP
+curl -k https://$ALB_IP   # refresh: the "Server name" changes between pods
+```
+
+Open `https://$ALB_IP` in a browser and refresh to watch requests land on different pods across the three nodes.
+
+### Step 4. Inspect (optional)
+
+```bash
+aws eks update-kubeconfig --name eks-https --region ap-southeast-2
+kubectl get nodes
+kubectl get pods -o wide          # three replicas, one per node
+
+TG_ARN=$(aws elbv2 describe-target-groups --names eks-https-tg \
+  --query 'TargetGroups[0].TargetGroupArn' --output text)
+aws elbv2 describe-target-health --target-group-arn "$TG_ARN"
+```
+
+### Cleanup
+
+```bash
+tofu destroy
+```
+
+## Troubleshooting
+
+### Targets Unhealthy / ALB Returns 5xx
+
+The ALB health-checks `/` on the NodePort. Targets turn healthy once the demo pods are running *and* the SG rule admits the ALB. Check, in order:
+
+```bash
+kubectl get pods -o wide                                  # pods Running?
+aws ec2 describe-security-groups \
+  --filters "Name=group-name,Values=eks-cluster-eks-https-nodegroup-sg" \
+  --query 'SecurityGroups[0].IpPermissions'               # NodePort rule present?
+aws elbv2 describe-target-health --target-group-arn "$TG_ARN"
+```
+
+If pods are `ImagePullBackOff`, the workers can't reach Docker Hub — confirm the NAT gateway is `available`:
+
+```bash
+aws ec2 describe-nat-gateways --query 'NatGateways[].[NatGatewayId,State]'
+```
+
+### Workers Never Join
+
+The workers join over the private endpoint. Confirm it's enabled and the node group is progressing:
+
+```bash
+aws eks describe-cluster --name eks-https --query 'cluster.resourcesVpcConfig.endpointPrivateAccess'
+aws eks describe-nodegroup --cluster-name eks-https --nodegroup-name workers --query 'nodegroup.status'
+```
+
+### Target-Group Attachment Errors at Apply
+
+The attachment indexes the discovered worker IDs by `node_desired_size`. If `apply` errors with an index out of range, the workers weren't all tagged/running when the `aws_instances` lookup ran. Re-run `tofu apply` — the node group will be `ACTIVE` by then:
+
+```bash
+aws ec2 describe-instances --filters "Name=tag:spinifex:eks-cluster,Values=eks-https" \
+  --query 'Reservations[].Instances[].[InstanceId,State.Name]' --output text
+```
+
+### kubernetes provider: connection refused / Unauthorized
+
+The provider runs `aws eks get-token` against the Spinifex STS endpoint. If the cluster was still `CREATING` when the provider first connected, re-run `tofu apply`. Otherwise confirm:
+
+```bash
+aws sts get-caller-identity
+aws eks describe-cluster --name eks-https --query 'cluster.status'
+```
+
+### ACM Import / HTTPS Listener Errors
+
+```bash
+aws acm list-certificates
+aws acm describe-certificate --certificate-arn "$(tofu output -raw certificate_arn)" \
+  --query 'Certificate.Status'
+```
+
+A `CertificateNotFound` on the listener means the ACM ARN didn't resolve — check the cert exists and belongs to the same account/profile.
+
+### Provider Connection Refused
+
+```bash
+sudo systemctl status spinifex.target
+curl -k https://localhost:9999/
+```

docs/terraform-workbooks/eks-https-ingress/main.tf

@@ -0,0 +1,671 @@
+# Example: 3-Node EKS behind an HTTPS ALB (ACM TLS) with a load-balanced app
+#
+# The largest EKS workbook, and the one with the clearest payoff. It stands up
+# an HA-shaped cluster sized for a 3-node Spinifex deployment, deploys a demo
+# web app across the workers, and publishes it through an internet-facing ALB
+# that terminates TLS with a certificate held in ACM.
+#
+# After `apply`, fetch the ALB's public IP (see alb_public_ip_hint) and open
+# https://<ip> in a browser. The page reports which pod answered; refresh and
+# it moves between the three replicas spread across the cluster — a live picture
+# of a multi-node Kubernetes cluster load-balancing behind HTTPS.
+#
+# What it builds:
+#   * VPC with two public subnets (ALB + NAT) and two private subnets (workers).
+#   * A NAT gateway so the private workers can pull the demo container image.
+#   * EKS cluster with BOTH a public endpoint (kubectl from your host) and a
+#     private endpoint (the in-VPC path the workers use to join).
+#   * A managed node group of three workers (t3.large) in the private subnets.
+#   * CoreDNS addon.
+#   * A self-signed cert imported into ACM (Spinifex ACM supports
+#     ImportCertificate, not RequestCertificate), wired to the ALB's HTTPS
+#     listener, which forwards to the workers' NodePort.
+#   * The demo Deployment + NodePort Service, deployed by the Kubernetes
+#     provider, plus the one SG rule that lets the ALB reach the NodePort.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-https-ingress
+#   export AWS_PROFILE=spinifex
+#   tofu init && tofu apply
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    tls = {
+      source  = "hashicorp/tls"
+      version = ">= 4.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.20"
+    }
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Variables
+# ---------------------------------------------------------------------------
+
+variable "region" {
+  type    = string
+  default = "ap-southeast-2"
+}
+
+variable "cluster_name" {
+  type    = string
+  default = "eks-https"
+}
+
+variable "k8s_version" {
+  type    = string
+  default = "1.32"
+}
+
+variable "node_instance_type" {
+  type    = string
+  default = "t3.large"
+}
+
+variable "node_desired_size" {
+  type        = number
+  default     = 3
+  description = "Worker count; spread across a 3-node Spinifex deployment for HA"
+}
+
+variable "node_port" {
+  type        = number
+  default     = 30080
+  description = "NodePort the ALB forwards to and the demo Service is published on"
+}
+
+variable "cert_common_name" {
+  type    = string
+  default = "eks-https.spinifex.local"
+}
+
+variable "api_public_access_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the public Kubernetes API endpoint; tighten in production"
+}
+
+variable "alb_ingress_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the ALB HTTPS listener"
+}
+
+variable "spinifex_endpoint" {
+  type    = string
+  default = "https://127.0.0.1:9999"
+}
+
+# ---------------------------------------------------------------------------
+# Providers
+# ---------------------------------------------------------------------------
+
+provider "aws" {
+  region = var.region
+
+  endpoints {
+    ec2                    = var.spinifex_endpoint
+    iam                    = var.spinifex_endpoint
+    sts                    = var.spinifex_endpoint
+    eks                    = var.spinifex_endpoint
+    acm                    = var.spinifex_endpoint
+    elasticloadbalancingv2 = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+provider "kubernetes" {
+  host                   = aws_eks_cluster.this.endpoint
+  cluster_ca_certificate = base64decode(aws_eks_cluster.this.certificate_authority[0].data)
+
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    command     = "aws"
+    args        = ["eks", "get-token", "--cluster-name", aws_eks_cluster.this.name, "--region", var.region]
+  }
+}
+
+data "aws_availability_zones" "available" {
+  state = "available"
+}
+
+# ---------------------------------------------------------------------------
+# VPC — public subnets for the ALB + NAT, private subnets for the workers
+# ---------------------------------------------------------------------------
+
+resource "aws_vpc" "main" {
+  cidr_block           = "10.32.0.0/16"
+  enable_dns_hostnames = true
+  enable_dns_support   = true
+
+  tags = {
+    Name = "${var.cluster_name}-vpc"
+  }
+}
+
+resource "aws_internet_gateway" "igw" {
+  vpc_id = aws_vpc.main.id
+
+  tags = {
+    Name = "${var.cluster_name}-igw"
+  }
+}
+
+resource "aws_subnet" "public_a" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.32.1.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-public-a"
+  }
+}
+
+resource "aws_subnet" "public_b" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.32.2.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-public-b"
+  }
+}
+
+resource "aws_subnet" "private_a" {
+  vpc_id            = aws_vpc.main.id
+  cidr_block        = "10.32.11.0/24"
+  availability_zone = data.aws_availability_zones.available.names[0]
+
+  tags = {
+    Name = "${var.cluster_name}-private-a"
+  }
+}
+
+resource "aws_subnet" "private_b" {
+  vpc_id            = aws_vpc.main.id
+  cidr_block        = "10.32.12.0/24"
+  availability_zone = data.aws_availability_zones.available.names[0]
+
+  tags = {
+    Name = "${var.cluster_name}-private-b"
+  }
+}
+
+resource "aws_route_table" "public" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block = "0.0.0.0/0"
+    gateway_id = aws_internet_gateway.igw.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-public-rt"
+  }
+}
+
+resource "aws_route_table_association" "public_a" {
+  subnet_id      = aws_subnet.public_a.id
+  route_table_id = aws_route_table.public.id
+}
+
+resource "aws_route_table_association" "public_b" {
+  subnet_id      = aws_subnet.public_b.id
+  route_table_id = aws_route_table.public.id
+}
+
+# NAT gateway gives the private workers outbound internet to pull the demo
+# container image. The workers still *join* the cluster over the in-VPC private
+# endpoint, not through the NAT.
+resource "aws_eip" "nat" {
+  domain = "vpc"
+
+  tags = {
+    Name = "${var.cluster_name}-nat-eip"
+  }
+}
+
+resource "aws_nat_gateway" "nat" {
+  allocation_id = aws_eip.nat.id
+  subnet_id     = aws_subnet.public_a.id
+
+  depends_on = [aws_internet_gateway.igw]
+
+  tags = {
+    Name = "${var.cluster_name}-nat"
+  }
+}
+
+resource "aws_route_table" "private" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block     = "0.0.0.0/0"
+    nat_gateway_id = aws_nat_gateway.nat.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-private-rt"
+  }
+}
+
+resource "aws_route_table_association" "private_a" {
+  subnet_id      = aws_subnet.private_a.id
+  route_table_id = aws_route_table.private.id
+}
+
+resource "aws_route_table_association" "private_b" {
+  subnet_id      = aws_subnet.private_b.id
+  route_table_id = aws_route_table.private.id
+}
+
+# ---------------------------------------------------------------------------
+# IAM — cluster role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "cluster" {
+  name = "${var.cluster_name}-cluster-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "eks.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "cluster" {
+  role       = aws_iam_role.cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — node role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "node" {
+  name = "${var.cluster_name}-node-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_worker" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_cni" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_ecr" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+}
+
+# ---------------------------------------------------------------------------
+# EKS cluster — public + private endpoints
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_cluster" "this" {
+  name     = var.cluster_name
+  role_arn = aws_iam_role.cluster.arn
+  version  = var.k8s_version
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.private_a.id, aws_subnet.private_b.id]
+    endpoint_public_access  = true
+    endpoint_private_access = true
+    public_access_cidrs     = [var.api_public_access_cidr]
+  }
+
+  access_config {
+    authentication_mode                         = "API"
+    bootstrap_cluster_creator_admin_permissions = true
+  }
+
+  depends_on = [aws_iam_role_policy_attachment.cluster]
+
+  tags = {
+    Name = var.cluster_name
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Managed node group — three workers in the private subnets
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_node_group" "workers" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "workers"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.private_a.id, aws_subnet.private_b.id]
+
+  scaling_config {
+    desired_size = var.node_desired_size
+    min_size     = var.node_desired_size
+    max_size     = var.node_desired_size * 2
+  }
+
+  instance_types = [var.node_instance_type]
+  ami_type       = "AL2_x86_64"
+
+  depends_on = [
+    aws_iam_role_policy_attachment.node_worker,
+    aws_iam_role_policy_attachment.node_cni,
+    aws_iam_role_policy_attachment.node_ecr,
+  ]
+
+  tags = {
+    Name = "${var.cluster_name}-workers"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Addon — CoreDNS
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_addon" "coredns" {
+  cluster_name                = aws_eks_cluster.this.name
+  addon_name                  = "coredns"
+  resolve_conflicts_on_create = "OVERWRITE"
+
+  # addon_version omitted on purpose. Spinifex defaults an unset version to its
+  # catalog default (coredns 1.11.1). The AWS provider also rejects a bare
+  # "1.11.1" (it demands a v-prefixed form) which Spinifex's catalog would in
+  # turn reject — so pinning is a dead end here; let the server choose.
+  depends_on = [aws_eks_node_group.workers]
+
+  tags = {
+    Name = "${var.cluster_name}-coredns"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# TLS — self-signed certificate imported into ACM
+# ---------------------------------------------------------------------------
+
+resource "tls_private_key" "ingress" {
+  algorithm = "RSA"
+  rsa_bits  = 2048
+}
+
+resource "tls_self_signed_cert" "ingress" {
+  private_key_pem = tls_private_key.ingress.private_key_pem
+
+  subject {
+    common_name  = var.cert_common_name
+    organization = "Spinifex EKS Demo"
+  }
+
+  dns_names             = [var.cert_common_name]
+  validity_period_hours = 8760
+  early_renewal_hours   = 720
+
+  allowed_uses = [
+    "key_encipherment",
+    "digital_signature",
+    "server_auth",
+  ]
+}
+
+resource "aws_acm_certificate" "ingress" {
+  private_key      = tls_private_key.ingress.private_key_pem
+  certificate_body = tls_self_signed_cert.ingress.cert_pem
+
+  tags = {
+    Name = "${var.cluster_name}-ingress"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# ALB — internet-facing, HTTPS terminated with the ACM cert
+# ---------------------------------------------------------------------------
+
+resource "aws_security_group" "alb" {
+  name        = "${var.cluster_name}-alb-sg"
+  description = "ALB HTTPS inbound, NodePort to workers"
+  vpc_id      = aws_vpc.main.id
+
+  ingress {
+    description = "HTTPS"
+    from_port   = 443
+    to_port     = 443
+    protocol    = "tcp"
+    cidr_blocks = [var.alb_ingress_cidr]
+  }
+
+  egress {
+    description = "All outbound"
+    from_port   = 0
+    to_port     = 0
+    protocol    = "-1"
+    cidr_blocks = ["0.0.0.0/0"]
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-alb-sg"
+  }
+}
+
+resource "aws_lb" "ingress" {
+  name               = "${var.cluster_name}-alb"
+  internal           = false
+  load_balancer_type = "application"
+  security_groups    = [aws_security_group.alb.id]
+  subnets            = [aws_subnet.public_a.id, aws_subnet.public_b.id]
+
+  tags = {
+    Name = "${var.cluster_name}-alb"
+  }
+}
+
+resource "aws_lb_target_group" "workers" {
+  name        = "${var.cluster_name}-tg"
+  port        = var.node_port
+  protocol    = "HTTP"
+  target_type = "instance"
+  vpc_id      = aws_vpc.main.id
+
+  health_check {
+    path                = "/"
+    protocol            = "HTTP"
+    matcher             = "200-399"
+    healthy_threshold   = 2
+    unhealthy_threshold = 3
+    timeout             = 5
+    interval            = 10
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-tg"
+  }
+}
+
+# Discover the launched workers by their Spinifex EKS tag so the ALB can target
+# them. depends_on the node group: the lookup must run after the workers exist.
+data "aws_instances" "workers" {
+  instance_tags = {
+    "spinifex:eks-cluster" = aws_eks_cluster.this.name
+  }
+
+  depends_on = [aws_eks_node_group.workers]
+}
+
+# count is the known desired size, so it's resolvable at plan time even though
+# the worker IDs themselves are only known after the node group launches.
+resource "aws_lb_target_group_attachment" "workers" {
+  count            = var.node_desired_size
+  target_group_arn = aws_lb_target_group.workers.arn
+  target_id        = data.aws_instances.workers.ids[count.index]
+  port             = var.node_port
+}
+
+resource "aws_lb_listener" "https" {
+  load_balancer_arn = aws_lb.ingress.arn
+  port              = 443
+  protocol          = "HTTPS"
+  ssl_policy        = "ELBSecurityPolicy-TLS13-1-2-2021-06"
+  certificate_arn   = aws_acm_certificate.ingress.arn
+
+  default_action {
+    type             = "forward"
+    target_group_arn = aws_lb_target_group.workers.arn
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Let the ALB reach the workers' NodePort
+#
+# Spinifex auto-manages the nodegroup SG and admits only intra-cluster traffic.
+# Look it up by its deterministic name and admit the ALB SG on the NodePort.
+# ---------------------------------------------------------------------------
+
+data "aws_security_group" "nodegroup" {
+  filter {
+    name   = "group-name"
+    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
+  }
+
+  filter {
+    name   = "vpc-id"
+    values = [aws_vpc.main.id]
+  }
+
+  depends_on = [aws_eks_node_group.workers]
+}
+
+resource "aws_vpc_security_group_ingress_rule" "nodeport_from_alb" {
+  security_group_id            = data.aws_security_group.nodegroup.id
+  referenced_security_group_id = aws_security_group.alb.id
+  from_port                    = var.node_port
+  to_port                      = var.node_port
+  ip_protocol                  = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-alb-nodeport"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Demo workload — three replicas spread across the workers
+# ---------------------------------------------------------------------------
+
+resource "kubernetes_deployment_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+    labels    = { app = "hello" }
+  }
+
+  spec {
+    replicas = var.node_desired_size
+
+    selector {
+      match_labels = { app = "hello" }
+    }
+
+    template {
+      metadata {
+        labels = { app = "hello" }
+      }
+
+      spec {
+        container {
+          name  = "hello"
+          image = "nginxdemos/hello"
+
+          port {
+            container_port = 80
+          }
+        }
+
+        # Spread one replica per node so a refresh visibly hits different nodes.
+        topology_spread_constraint {
+          max_skew           = 1
+          topology_key       = "kubernetes.io/hostname"
+          when_unsatisfiable = "ScheduleAnyway"
+
+          label_selector {
+            match_labels = { app = "hello" }
+          }
+        }
+      }
+    }
+  }
+
+  depends_on = [aws_eks_addon.coredns]
+}
+
+resource "kubernetes_service_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+  }
+
+  spec {
+    selector = { app = "hello" }
+    type     = "NodePort"
+
+    port {
+      port        = 80
+      target_port = 80
+      node_port   = var.node_port
+    }
+  }
+
+  depends_on = [kubernetes_deployment_v1.hello]
+}
+
+# ---------------------------------------------------------------------------
+# Outputs
+# ---------------------------------------------------------------------------
+
+output "cluster_name" {
+  value = aws_eks_cluster.this.name
+}
+
+output "certificate_arn" {
+  value = aws_acm_certificate.ingress.arn
+}
+
+output "alb_dns_name" {
+  value = aws_lb.ingress.dns_name
+}
+
+output "alb_public_ip_hint" {
+  value = "aws elbv2 describe-load-balancers --names ${var.cluster_name}-alb --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' --output text"
+}
+
+output "demo_url_hint" {
+  value = "Fetch the ALB IP via alb_public_ip_hint, then open https://<that-ip> (self-signed cert: curl -k). Refresh to see different pods answer."
+}
+
+output "update_kubeconfig" {
+  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
+}

docs/terraform-workbooks/eks-quickstart/README.md

@@ -0,0 +1,173 @@
+---
+title: "EKS Quickstart"
+description: "Stand up a minimal managed-Kubernetes cluster and a browsable demo app — VPC, IAM roles, an EKS cluster, a single-worker node group, and a load-balanced web page — using Terraform on Spinifex."
+category: "Terraform Workbooks"
+tags:
+  - terraform
+  - eks
+  - kubernetes
+  - iam
+  - vpc
+  - workbook
+resources:
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "Terraform Kubernetes Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "OpenTofu"
+    url: "https://opentofu.org/"
+---
+
+# Terraform: EKS Quickstart
+
+> The smallest end-to-end EKS example on Spinifex: a VPC, the two IAM roles EKS needs, a single-worker cluster, and a demo web app you can open in a browser.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+Provision a managed Kubernetes cluster on Spinifex with Terraform/OpenTofu — and finish with something you can actually see. As well as the cluster, this workbook deploys a small demo app onto it and publishes it on the worker's public IP. When `apply` finishes, open the **`demo_url`** output: the page is served by `nginxdemos/hello` and prints the name of the pod that answered. Refresh it and the answering pod changes, so even a non-technical viewer can watch Kubernetes scheduling and load-balancing in real time.
+
+Under the hood it keeps things minimal: a VPC with two public subnets, an IAM role for the control plane, an IAM role for the workers with the three managed policies AWS-managed node groups expect, a public-endpoint `aws_eks_cluster`, and a single-node `aws_eks_node_group`. Terraform then uses the **Kubernetes provider** (authenticating exactly like `kubectl` does, via `aws eks get-token`) to deploy the Deployment + NodePort Service.
+
+**What you'll learn:**
+
+- Configuring the AWS provider to target Spinifex's `eks`, `ec2`, `iam` and `sts` endpoints
+- Creating the EKS cluster and worker IAM roles with faithful managed-policy attachments
+- Provisioning an `aws_eks_cluster` and a managed `aws_eks_node_group`
+- Pointing the Kubernetes provider at the new cluster and deploying a workload from the same `apply`
+- Opening one rule on the auto-managed worker security group to expose a NodePort
+
+**What gets created**
+
+| Resource | Name | Purpose |
+|---|---|---|
+| VPC | `eks-quickstart-vpc` | Isolated network (10.30.0.0/16) |
+| Subnets | `eks-quickstart-subnet-a/-b` | Public subnets for the cluster and worker |
+| Internet Gateway | `eks-quickstart-igw` | Egress so the worker can pull the demo image |
+| IAM Roles | `eks-quickstart-cluster-role`, `eks-quickstart-node-role` | Control-plane and worker roles |
+| EKS Cluster | `eks-quickstart` | Public API endpoint, API auth mode, Kubernetes 1.32 |
+| Node Group | `default` | One `t3.medium` worker |
+| SG Ingress Rule | `eks-quickstart-demo-nodeport` | Opens the demo NodePort on the auto-managed worker SG |
+| K8s Deployment | `hello` | `nginxdemos/hello`, 2 replicas |
+| K8s Service | `hello` | NodePort publishing the demo on the worker |
+
+**Spinifex specifics**
+
+- The cluster's security groups are **auto-managed** — `vpc_config.security_group_ids` is ignored. To reach a NodePort, this workbook looks the worker SG up by its deterministic name (`eks-cluster-<name>-nodegroup-sg`) and adds a single ingress rule.
+- The worker AMI is always Spinifex's `eks-node` image; `ami_type` is recorded but does **not** select the image.
+- `authentication_mode` must be `"API"` (the `API_AND_CONFIG_MAP` mode is rejected).
+- The workers need outbound internet (here via the IGW) to pull the demo container image.
+
+**Prerequisites:**
+
+- Spinifex installed and running (see [Installing Spinifex](/docs/install))
+- The Spinifex `eks-node` image available on the cluster
+- OpenTofu or Terraform, plus `kubectl` and the AWS CLI
+
+## Instructions
+
+### Step 1. Get the Template
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-quickstart
+```
+
+Or create a `main.tf` file and paste the full configuration below.
+
+<!-- INCLUDE: main.tf lang:hcl -->
+
+### Step 2. Deploy
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+```
+
+This single `apply` does the whole job: it creates the cluster (which bootstraps a control-plane VM and brings up k3s — a few minutes in `CREATING`), launches the worker, opens the NodePort, then deploys the demo app and waits for the pods to roll out.
+
+> **Same profile for the Kubernetes provider.** The Kubernetes provider authenticates by shelling out to `aws eks get-token`, which has to reach the Spinifex STS endpoint. Keep `AWS_PROFILE=spinifex` exported for the whole `apply`.
+
+### Step 3. Open the Demo
+
+```bash
+tofu output demo_url
+```
+
+Open that URL in a browser. You'll see the `nginxdemos/hello` page showing the **Server name** (the pod that handled the request). Refresh a few times — with two replicas, the pod name alternates, demonstrating that the Service is load-balancing across the cluster.
+
+### Step 4. Inspect with kubectl (optional)
+
+```bash
+aws eks update-kubeconfig --name eks-quickstart --region ap-southeast-2
+kubectl get nodes
+kubectl get pods -o wide
+```
+
+`kubectl get pods -o wide` shows the demo pods and which node each landed on.
+
+### Cleanup
+
+```bash
+tofu destroy
+```
+
+## Troubleshooting
+
+### Demo URL Doesn't Load
+
+The page is served from a NodePort on the worker's public IP. Work through the chain:
+
+```bash
+# Is the worker running with a public IP?
+aws ec2 describe-instances --filters "Name=tag:spinifex:eks-cluster,Values=eks-quickstart" \
+  --query 'Reservations[].Instances[].[InstanceId,State.Name,PublicIpAddress]' --output text
+
+# Did the demo pods roll out?
+kubectl get pods -o wide
+
+# Is the NodePort rule on the worker SG?
+aws ec2 describe-security-groups --filters "Name=group-name,Values=eks-cluster-eks-quickstart-nodegroup-sg" \
+  --query 'SecurityGroups[0].IpPermissions'
+```
+
+If the pods are `Pending`, the worker may not be `Ready` yet — give it a moment. If they're `ImagePullBackOff`, the worker can't reach Docker Hub; confirm the IGW route and the worker's public IP.
+
+### kubernetes provider: connection refused / Unauthorized
+
+The provider runs `aws eks get-token` against the Spinifex STS endpoint. Confirm the AWS CLI is still pointed at Spinifex and the cluster is `ACTIVE`:
+
+```bash
+aws sts get-caller-identity
+aws eks describe-cluster --name eks-quickstart --query 'cluster.status'
+```
+
+If the cluster was still `CREATING` when the provider first tried to connect, just re-run `tofu apply`.
+
+### Cluster Stuck in CREATING
+
+Control-plane bootstrap takes a few minutes. Confirm the underlying VM came up:
+
+```bash
+aws eks describe-cluster --name eks-quickstart --query 'cluster.status'
+aws ec2 describe-instances --profile spinifex
+```
+
+### Provider Connection Refused
+
+```bash
+sudo systemctl status spinifex.target
+curl -k https://localhost:9999/
+```

docs/terraform-workbooks/eks-quickstart/main.tf

@@ -0,0 +1,428 @@
+# Example: EKS Quickstart on Spinifex
+#
+# A minimal managed-Kubernetes cluster that ends in something you can see: a
+# VPC, the two IAM roles EKS needs, a single-worker cluster, and a demo web app
+# deployed onto it by Terraform and exposed on the worker's public IP.
+#
+# Once `apply` finishes, open the demo_url output in a browser. The page is
+# served by the nginxdemos/hello image and prints the pod name that answered —
+# refresh and it bounces between the app's replicas, showing the cluster is
+# really scheduling and load-balancing pods.
+#
+# Demonstrates: VPC + subnets, EKS cluster + node IAM roles, an aws_eks_cluster
+# and managed aws_eks_node_group, the Kubernetes provider authenticating to the
+# cluster, and a Deployment + NodePort Service reachable from your browser.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-quickstart
+#   export AWS_PROFILE=spinifex
+#   tofu init && tofu apply
+#   # then open the demo_url output
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.20"
+    }
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Variables
+# ---------------------------------------------------------------------------
+
+variable "region" {
+  type    = string
+  default = "ap-southeast-2"
+}
+
+variable "cluster_name" {
+  type    = string
+  default = "eks-quickstart"
+}
+
+variable "k8s_version" {
+  type        = string
+  default     = "1.32"
+  description = "Kubernetes minor version for the control plane and workers"
+}
+
+variable "instance_type" {
+  type    = string
+  default = "t3.medium"
+}
+
+variable "node_port" {
+  type        = number
+  default     = 30080
+  description = "NodePort the demo Service is published on"
+}
+
+variable "replicas" {
+  type        = number
+  default     = 2
+  description = "Demo app replicas; refresh the page to see requests land on different pods"
+}
+
+variable "browse_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the demo NodePort; tighten to your own IP in production"
+}
+
+variable "spinifex_endpoint" {
+  type        = string
+  default     = "https://127.0.0.1:9999"
+  description = "Spinifex AWS gateway endpoint"
+}
+
+# ---------------------------------------------------------------------------
+# Providers
+# ---------------------------------------------------------------------------
+
+provider "aws" {
+  region = var.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+# Authenticates to the cluster with the same `aws eks get-token` exec flow the
+# generated kubeconfig uses, so the Kubernetes provider can deploy the demo app.
+provider "kubernetes" {
+  host                   = aws_eks_cluster.this.endpoint
+  cluster_ca_certificate = base64decode(aws_eks_cluster.this.certificate_authority[0].data)
+
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    command     = "aws"
+    args        = ["eks", "get-token", "--cluster-name", aws_eks_cluster.this.name, "--region", var.region]
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Data sources
+# ---------------------------------------------------------------------------
+
+data "aws_availability_zones" "available" {
+  state = "available"
+}
+
+# ---------------------------------------------------------------------------
+# VPC + two public subnets (workers get public IPs to pull the demo image)
+# ---------------------------------------------------------------------------
+
+resource "aws_vpc" "main" {
+  cidr_block           = "10.30.0.0/16"
+  enable_dns_hostnames = true
+  enable_dns_support   = true
+
+  tags = {
+    Name = "${var.cluster_name}-vpc"
+  }
+}
+
+resource "aws_internet_gateway" "igw" {
+  vpc_id = aws_vpc.main.id
+
+  tags = {
+    Name = "${var.cluster_name}-igw"
+  }
+}
+
+resource "aws_subnet" "a" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.30.1.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-subnet-a"
+  }
+}
+
+resource "aws_subnet" "b" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.30.2.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-subnet-b"
+  }
+}
+
+resource "aws_route_table" "public" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block = "0.0.0.0/0"
+    gateway_id = aws_internet_gateway.igw.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-public-rt"
+  }
+}
+
+resource "aws_route_table_association" "a" {
+  subnet_id      = aws_subnet.a.id
+  route_table_id = aws_route_table.public.id
+}
+
+resource "aws_route_table_association" "b" {
+  subnet_id      = aws_subnet.b.id
+  route_table_id = aws_route_table.public.id
+}
+
+# ---------------------------------------------------------------------------
+# IAM — EKS cluster role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "cluster" {
+  name = "${var.cluster_name}-cluster-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "eks.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "cluster" {
+  role       = aws_iam_role.cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — worker node role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "node" {
+  name = "${var.cluster_name}-node-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_worker" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_cni" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_ecr" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+}
+
+# ---------------------------------------------------------------------------
+# EKS cluster — public API endpoint, API (access-entry) auth mode
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_cluster" "this" {
+  name     = var.cluster_name
+  role_arn = aws_iam_role.cluster.arn
+  version  = var.k8s_version
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.a.id, aws_subnet.b.id]
+    endpoint_public_access  = true
+    endpoint_private_access = false
+  }
+
+  access_config {
+    authentication_mode                         = "API"
+    bootstrap_cluster_creator_admin_permissions = true
+  }
+
+  depends_on = [aws_iam_role_policy_attachment.cluster]
+
+  tags = {
+    Name = var.cluster_name
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Managed node group — one worker
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_node_group" "default" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "default"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.a.id, aws_subnet.b.id]
+
+  scaling_config {
+    desired_size = 1
+    min_size     = 1
+    max_size     = 2
+  }
+
+  instance_types = [var.instance_type]
+  ami_type       = "AL2_x86_64"
+
+  depends_on = [
+    aws_iam_role_policy_attachment.node_worker,
+    aws_iam_role_policy_attachment.node_cni,
+    aws_iam_role_policy_attachment.node_ecr,
+  ]
+
+  tags = {
+    Name = "${var.cluster_name}-default"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Open the NodePort on the auto-managed nodegroup SG
+#
+# Spinifex creates the worker SG itself (vpc_config.security_group_ids is
+# ignored) and admits only intra-cluster traffic. To reach the demo NodePort
+# from a browser, look the SG up by its deterministic name and add one rule.
+# ---------------------------------------------------------------------------
+
+data "aws_security_group" "nodegroup" {
+  filter {
+    name   = "group-name"
+    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
+  }
+
+  filter {
+    name   = "vpc-id"
+    values = [aws_vpc.main.id]
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+resource "aws_vpc_security_group_ingress_rule" "nodeport" {
+  security_group_id = data.aws_security_group.nodegroup.id
+  cidr_ipv4         = var.browse_cidr
+  from_port         = var.node_port
+  to_port           = var.node_port
+  ip_protocol       = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-demo-nodeport"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Demo workload — deployed onto the cluster by Terraform
+#
+# nginxdemos/hello renders a page showing which pod served the request; with
+# multiple replicas, refreshing the demo_url alternates between them.
+# ---------------------------------------------------------------------------
+
+resource "kubernetes_deployment_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+    labels    = { app = "hello" }
+  }
+
+  spec {
+    replicas = var.replicas
+
+    selector {
+      match_labels = { app = "hello" }
+    }
+
+    template {
+      metadata {
+        labels = { app = "hello" }
+      }
+
+      spec {
+        container {
+          name  = "hello"
+          image = "nginxdemos/hello"
+
+          port {
+            container_port = 80
+          }
+        }
+      }
+    }
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+resource "kubernetes_service_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+  }
+
+  spec {
+    selector = { app = "hello" }
+    type     = "NodePort"
+
+    port {
+      port        = 80
+      target_port = 80
+      node_port   = var.node_port
+    }
+  }
+
+  depends_on = [kubernetes_deployment_v1.hello]
+}
+
+# ---------------------------------------------------------------------------
+# Discover the worker's public IP for the demo URL
+# ---------------------------------------------------------------------------
+
+data "aws_instances" "workers" {
+  instance_tags = {
+    "spinifex:eks-cluster" = aws_eks_cluster.this.name
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+# ---------------------------------------------------------------------------
+# Outputs
+# ---------------------------------------------------------------------------
+
+output "cluster_name" {
+  value = aws_eks_cluster.this.name
+}
+
+output "demo_url" {
+  value       = "http://${data.aws_instances.workers.public_ips[0]}:${var.node_port}"
+  description = "Open in a browser; refresh to see different pods answer"
+}
+
+output "update_kubeconfig" {
+  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
+}

docs/terraform-workbooks/nginx-alb/README.md

@@ -142,7 +142,6 @@ The ALB is internet-facing, but the DNS name Spinifex returns (`*.elb.spinifex.l
 ALB_IP=$(aws elbv2 describe-load-balancers --names nginx-alb \
   --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
   --output text)
-echo "ALB public IP: $ALB_IP"
 ```
 
 Then hit the ALB — successive requests should alternate between Server 1 and Server 2:
@@ -163,8 +162,6 @@ TG_ARN=$(aws elbv2 describe-target-groups \
   --query 'TargetGroups[0].TargetGroupArn' \
   --output text)
 
-echo "Target Group ARN: $TG_ARN"
-
 aws elbv2 describe-target-health --target-group-arn $TG_ARN
 ```
 

docs/terraform-workbooks/s3-webapp/README.md

@@ -9,6 +9,9 @@ tags:
   - flask
   - webapp
   - workbook
+  - imds
+  - sts
+  - instance-profile
 resources:
   - title: "Terraform AWS Provider"
     url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
@@ -36,18 +39,21 @@ resources:
 
 Deploy an EC2 instance running a Flask file-sharing web application backed by S3 (Predastore). Users can upload files through a web form and browse uploaded content — demonstrating Terraform managing both compute and object storage together.
 
+This workbook uses the idiomatic AWS credential model: the instance is launched with an **IAM instance profile** and the app pulls short-lived STS credentials from **IMDS** (`169.254.169.254`) through boto3's default credential chain. **No long-lived S3 keys are baked into the instance.** The Terraform run still uses the operator's admin credentials (to create the bucket, role, profile, and instance), but the running instance authenticates to S3 with credentials it fetches at runtime and which expire in ~1 hour.
+
 **Architecture:**
 
 <p align="center">
-  <img src="../../../.github/assets/diagrams/tf-s3-webapp.svg" alt="S3 webapp — browser to Flask EC2 instance to Predastore via S3 API" width="900">
+  <img src="../../../.github/assets/diagrams/tf-s3-webapp.svg" alt="S3 webapp — browser to Flask EC2 instance; the instance fetches short-lived STS credentials from IMDS (169.254.169.254) via its instance profile, then calls Predastore over the S3 API" width="900">
 </p>
 
 **What you'll learn:**
 
 - Configuring the AWS provider with both Spinifex and Predastore endpoints
 - Creating S3 buckets on Predastore via Terraform
-- Deploying a Python webapp with cloud-init that talks to S3
-- Passing credentials and configuration to instances via user-data
+- Defining an IAM role, least-privilege managed policy, and instance profile, and passing the role to an instance (`iam:PassRole`)
+- How the instance obtains short-lived credentials from IMDS via boto3's default credential chain — no static keys on the instance
+- Deploying a Python webapp with cloud-init that signs S3 requests with the STS session token
 
 **Prerequisites:**
 
@@ -55,6 +61,7 @@ Deploy an EC2 instance running a Flask file-sharing web application backed by S3
 - Predastore running (S3 API on port 8443)
 - An Ubuntu 26.04 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
 - OpenTofu or Terraform installed
+- The operator identity running the Terraform apply (`AWS_PROFILE=spinifex`) must be allowed to manage IAM and pass the role: `iam:CreateRole`, `iam:CreatePolicy`, `iam:AttachRolePolicy`, `iam:CreateInstanceProfile`, `iam:AddRoleToInstanceProfile`, and `iam:PassRole` on the new role. The bootstrap admin profile satisfies this.
 - The EC2 instance must be able to reach Predastore — use the host's br-wan IP, not localhost
 
 ## Instructions
@@ -141,16 +148,39 @@ sudo systemctl status s3-webapp
 sudo journalctl -u s3-webapp --no-pager -n 50
 ```
 
-### Upload Fails or Files Not Appearing
+### Upload Fails with `403 AccessDenied`
 
-Check that the S3 credentials in `/opt/webapp/.env` are correct and that the bucket exists:
+The instance reached S3 but was not authorized. The `.env` file deliberately holds **no** keys — credentials come from IMDS:
 
 ```bash
 ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
-cat /opt/webapp/.env
-/opt/webapp/venv/bin/python -c "import boto3; print('boto3 OK')"
+cat /opt/webapp/.env   # S3_ENDPOINT / S3_BUCKET / S3_REGION only — no keys
 ```
 
+Check that:
+
+- the managed policy is attached to the role (`s3:ListBucket` on the bucket, `s3:GetObject`/`s3:PutObject` on its objects), and the policy's bucket name matches `bucket_name`;
+- the assumed-role session's account matches the bucket owner's account — Predastore enforces bucket ownership. Both are created by the same operator identity, so they align by construction; a mismatch surfaces as `403`.
+
+### `NoCredentialsError` / No Credentials
+
+boto3 could not obtain credentials, which means the IMDS credential chain did not resolve. From the instance:
+
+```bash
+ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
+
+# IMDSv2: fetch a token, then the role name behind the instance profile
+TOKEN=$(curl -sX PUT http://169.254.169.254/latest/api/token \
+  -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')
+curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
+  http://169.254.169.254/latest/meta-data/iam/security-credentials/
+
+# Confirm the resolved identity — expect assumed-role/s3-webapp-role/<instance-id>
+aws sts get-caller-identity --endpoint-url https://<predastore_host>
+```
+
+If the role name is empty, the instance profile was not attached — confirm `iam_instance_profile` on the instance and that `iam:PassRole` is allowed for the operator.
+
 ### AMI Not Found
 
 Ensure you have imported an Ubuntu 26.04 image:

docs/terraform-workbooks/s3-webapp/main.tf

@@ -8,13 +8,25 @@
 # Architecture:
 #
 #   Browser ──HTTP──▶ EC2 Instance (Flask webapp, port 80)
-#                         │
-#                         ▼ S3 API (boto3)
+#                         │  ▲ IMDS: short-lived STS creds (ASIA + token)
+#                         │  └──── 169.254.169.254
+#                         ▼ S3 API (boto3, SigV4 with session token)
 #                     Predastore (port 8443)
 #
+# Credentials:
+#   The Terraform run uses the operator's admin keys (s3_access_key /
+#   s3_secret_key) to create the bucket, IAM role, instance profile, and
+#   instance. The INSTANCE receives no long-lived secret — boto3's default
+#   credential chain pulls short-lived STS credentials from IMDS
+#   (169.254.169.254), resolved via the attached instance profile -> role.
+#
 # Prerequisites:
 #   - Spinifex services running (gateway on port 9999)
 #   - Predastore running (S3 API on port 8443)
+#   - The operator identity (AWS_PROFILE=spinifex) must be allowed
+#     iam:CreateRole / iam:CreatePolicy / iam:AttachRolePolicy /
+#     iam:CreateInstanceProfile / iam:AddRoleToInstanceProfile and
+#     iam:PassRole on the new role.
 #   - The EC2 instance must be able to reach the Predastore endpoint.
 #     Set `predastore_host` to the IP reachable from inside the VPC
 #     (e.g. the host's br-wan IP, NOT localhost).
@@ -80,13 +92,13 @@ variable "predastore_host" {
 
 variable "s3_access_key" {
   type        = string
-  description = "S3 access key for Predastore"
+  description = "Operator S3 access key for the Terraform run (creates the bucket on Predastore); the instance uses IMDS, not this key"
 }
 
 variable "s3_secret_key" {
   type        = string
   sensitive   = true
-  description = "S3 secret key for Predastore"
+  description = "Operator S3 secret key for the Terraform run; the instance uses IMDS, not this key"
 }
 
 variable "bucket_name" {
@@ -154,6 +166,55 @@ resource "aws_s3_bucket" "uploads" {
   bucket = var.bucket_name
 }
 
+# ---------------------------------------------------------------------------
+# IAM Instance Role — least-privilege S3 access fetched at runtime via IMDS
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "webapp" {
+  name = "s3-webapp-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Principal = { Service = "ec2.amazonaws.com" }
+      Action    = "sts:AssumeRole"
+    }]
+  })
+}
+
+resource "aws_iam_policy" "webapp_s3" {
+  name = "s3-webapp-policy"
+
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Sid      = "ListBucket"
+        Effect   = "Allow"
+        Action   = ["s3:ListBucket"]
+        Resource = ["arn:aws:s3:::${var.bucket_name}"]
+      },
+      {
+        Sid      = "ObjectRW"
+        Effect   = "Allow"
+        Action   = ["s3:GetObject", "s3:PutObject"]
+        Resource = ["arn:aws:s3:::${var.bucket_name}/*"]
+      }
+    ]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "webapp" {
+  role       = aws_iam_role.webapp.name
+  policy_arn = aws_iam_policy.webapp_s3.arn
+}
+
+resource "aws_iam_instance_profile" "webapp" {
+  name = "s3-webapp-profile"
+  role = aws_iam_role.webapp.name
+}
+
 # ---------------------------------------------------------------------------
 # SSH Key Pair
 # ---------------------------------------------------------------------------
@@ -273,9 +334,14 @@ resource "aws_instance" "webapp" {
   subnet_id              = aws_subnet.public.id
   vpc_security_group_ids = [aws_security_group.webapp.id]
   key_name               = aws_key_pair.webapp.key_name
+  iam_instance_profile   = aws_iam_instance_profile.webapp.name
 
   associate_public_ip_address = true
 
+  # The role's permissions must exist before the instance boots and makes its
+  # first IMDS-credentialed S3 call
+  depends_on = [aws_iam_role_policy_attachment.webapp]
+
   user_data_base64 = base64encode(<<-USERDATA
     #!/bin/bash
     set -euo pipefail
@@ -293,8 +359,6 @@ resource "aws_instance" "webapp" {
     cat > /opt/webapp/.env <<'ENVFILE'
     S3_ENDPOINT=https://${var.predastore_host}
     S3_BUCKET=${var.bucket_name}
-    S3_ACCESS_KEY=${var.s3_access_key}
-    S3_SECRET_KEY=${var.s3_secret_key}
     S3_REGION=${var.region}
     ENVFILE
 
@@ -321,8 +385,6 @@ resource "aws_instance" "webapp" {
     s3 = boto3.client(
         "s3",
         endpoint_url=env["S3_ENDPOINT"],
-        aws_access_key_id=env["S3_ACCESS_KEY"],
-        aws_secret_access_key=env["S3_SECRET_KEY"],
         region_name=env["S3_REGION"],
         verify=False,
         config=Config(s3={"addressing_style": "path"}),

docs/terraform-workbooks/s3-webapp/terraform.tfvars.example

@@ -2,6 +2,11 @@
 #
 # The predastore_host must be reachable from INSIDE the VPC guest — not
 # localhost. Use the host's br-wan or LAN IP, e.g. "192.168.1.10:8443".
+#
+# s3_access_key / s3_secret_key are the operator credentials for the Terraform
+# run only — they create the bucket on Predastore and the IAM role/profile.
+# They are NOT passed to the instance; the instance pulls short-lived STS
+# credentials from IMDS via its instance profile.
 
 predastore_host = "192.168.1.10:8443"
 s3_access_key   = "AKIAIOSFODNN7EXAMPLE"

go.mod

@@ -13,9 +13,9 @@ require (
 	github.com/insomniacslk/dhcp v0.0.0-20260407060928-11b94ed970f2
 	github.com/kdomanski/iso9660 v0.4.0
 	github.com/klauspost/cpuid/v2 v2.3.0
-	github.com/mulgadc/predastore v1.6.0
-	github.com/mulgadc/viperblock v1.6.0
-	github.com/nats-io/nats-server/v2 v2.14.1
+	github.com/mulgadc/predastore v1.8.0
+	github.com/mulgadc/viperblock v1.8.0
+	github.com/nats-io/nats-server/v2 v2.14.2
 	github.com/nats-io/nats.go v1.52.0
 	github.com/ovn-kubernetes/libovsdb v0.8.1
 	github.com/pelletier/go-toml/v2 v2.3.1
@@ -24,10 +24,10 @@ require (
 	github.com/spf13/viper v1.21.0
 	github.com/stretchr/testify v1.11.1
 	go.uber.org/goleak v1.3.0
-	golang.org/x/crypto v0.52.0
-	golang.org/x/sync v0.20.0
-	golang.org/x/sys v0.45.0
-	gopkg.in/ini.v1 v1.67.2
+	golang.org/x/crypto v0.53.0
+	golang.org/x/sync v0.21.0
+	golang.org/x/sys v0.46.0
+	gopkg.in/ini.v1 v1.67.3
 	gopkg.in/yaml.v3 v3.0.1
 )
 
@@ -38,8 +38,8 @@ require (
 	github.com/antithesishq/antithesis-sdk-go v0.7.0 // indirect
 	github.com/armon/go-metrics v0.4.1 // indirect
 	github.com/atotto/clipboard v0.1.4 // indirect
-	github.com/aws/aws-sdk-go-v2 v1.41.9 // indirect
-	github.com/aws/smithy-go v1.26.0 // indirect
+	github.com/aws/aws-sdk-go-v2 v1.42.0 // indirect
+	github.com/aws/smithy-go v1.27.1 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/boltdb/bolt v1.3.1 // indirect
@@ -57,7 +57,7 @@ require (
 	github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
 	github.com/containerd/console v1.0.5 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/dgraph-io/badger/v4 v4.9.1 // indirect
+	github.com/dgraph-io/badger/v4 v4.9.2 // indirect
 	github.com/dgraph-io/ristretto/v2 v2.4.0 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
@@ -101,8 +101,8 @@ require (
 	github.com/muesli/cancelreader v0.2.2 // indirect
 	github.com/muesli/termenv v0.16.0 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
-	github.com/nats-io/jwt/v2 v2.8.1 // indirect
-	github.com/nats-io/nkeys v0.4.15 // indirect
+	github.com/nats-io/jwt/v2 v2.8.2 // indirect
+	github.com/nats-io/nkeys v0.4.16 // indirect
 	github.com/nats-io/nuid v1.0.1 // indirect
 	github.com/pierrec/lz4/v4 v4.1.14 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
@@ -110,7 +110,7 @@ require (
 	github.com/prometheus/client_model v0.6.2 // indirect
 	github.com/prometheus/common v0.67.5 // indirect
 	github.com/prometheus/procfs v0.20.1 // indirect
-	github.com/quic-go/quic-go v0.59.1 // indirect
+	github.com/quic-go/quic-go v0.60.0 // indirect
 	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/sagikazarmark/locafero v0.12.0 // indirect
 	github.com/spf13/afero v1.15.0 // indirect
@@ -126,13 +126,13 @@ require (
 	go.opentelemetry.io/otel/trace v1.42.0 // indirect
 	go.yaml.in/yaml/v2 v2.4.4 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
-	golang.org/x/mod v0.35.0 // indirect
-	golang.org/x/net v0.55.0 // indirect
-	golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa // indirect
-	golang.org/x/term v0.43.0 // indirect
-	golang.org/x/text v0.37.0 // indirect
+	golang.org/x/mod v0.36.0 // indirect
+	golang.org/x/net v0.56.0 // indirect
+	golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6 // indirect
+	golang.org/x/term v0.44.0 // indirect
+	golang.org/x/text v0.38.0 // indirect
 	golang.org/x/time v0.15.0 // indirect
-	golang.org/x/tools v0.44.0 // indirect
+	golang.org/x/tools v0.45.0 // indirect
 	golang.org/x/vuln v1.1.4 // indirect
 	google.golang.org/protobuf v1.36.11 // indirect
 )

go.sum

@@ -31,30 +31,30 @@ github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z
 github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
 github.com/aws/aws-sdk-go v1.55.8 h1:JRmEUbU52aJQZ2AjX4q4Wu7t4uZjOu71uyNmaWlUkJQ=
 github.com/aws/aws-sdk-go v1.55.8/go.mod h1:ZkViS9AqA6otK+JBBNH2++sx1sgxrPKcSzPPvQkUtXk=
-github.com/aws/aws-sdk-go-v2 v1.41.9 h1:/rYeyO2+HrMztAmxAq9++XJtFMqSIpSsNA0yDGALYq4=
-github.com/aws/aws-sdk-go-v2 v1.41.9/go.mod h1:+HsoOEX80qAVUitj1A2DhCNTjmb3edVyuDypb6LNEeo=
-github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11 h1:h5+3VT69KUBK24grGuuA5saDJTj2IIjLb9au668Fo5I=
-github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11/go.mod h1:dnakxebH6UwFvcvujL0LVggYQ8nEvBGjU4G/V79Nv94=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.19 h1:yuFzSV1U0aRNYCQGVaTY2zW2M/L93pYHnXnrJUphYhU=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.19/go.mod h1:7y63L1kGzeoDlJaQ3Z578KrnmfBut96JjvJUzGwR+YE=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.25 h1:Uii3frf9ztec/ABM2/FSH9/z7PLzxfpG8h4RpkUFflQ=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.25/go.mod h1:G6kntsA2GorAxDPbap6xgB2F+amSLUF8GJTi7PUoX44=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.25 h1:r1+/l6m+WaUJF9HISEsNOLHSNj5EXYQxK8VX6Cz9NlA=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.25/go.mod h1:cKf+D+NMDK1LndD7BowHbBZPgR9V0/5HubH0PFWvA+c=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.26 h1:A1PmWU2zfkIm9EyFlJncFXL4W4phML+h8KjltUsCvNQ=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.26/go.mod h1:dY4MRzXEizrD4hqtpKvWVGPX7QleSGGVY+EBolo1RmM=
-github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.10 h1:d5/908OJ4bXg8lyjeMPvXetEKqoDoLi5Owy1zNue3yg=
-github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.10/go.mod h1:a57l7Hwh+FWI+we50g5NPJHYUKeJKfXbc4w8SyXu8Ig=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.18 h1:W/EyPFl9A5rXrtoilfwHYEvzHER+K4SpBPtMXi24Mos=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.18/go.mod h1:UG50K+pvd/uy6xExbobg0rjqFBFZe6I3l75EPDZw4tg=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.25 h1:dD3dhHNglpd98gs72my22Ndqi1hqQGllFFg1F+twfxg=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.25/go.mod h1:0yAbjPfd64gG7mj85RW+fMEYdfBgCRZw8g/oWcL1pjc=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.25 h1:2pQEbwf+/6EDbiit/GcBE2K4IUpMZymaA0kOz3xK978=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.25/go.mod h1:KvT6NCcQ0EZ+ZkVRrlBMt04Po3ok23YELEp7WimhLhM=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.102.2 h1:ie4ElCmUKS26pzrZcIk/lmt4yWjAqLLcawstyQCh298=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.102.2/go.mod h1:zjsomFeX5duj+4PlMB+o4JoWTIx+G0XMyzjYrUbQkN0=
-github.com/aws/smithy-go v1.26.0 h1:9ouqbi+NyKP7fV3Te7UElCwdAb6Y8uk7LGwPE5tVe/s=
-github.com/aws/smithy-go v1.26.0/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
+github.com/aws/aws-sdk-go-v2 v1.42.0 h1:XvXMJTkFQtpBKIWZnmr9ZEOc2InWM2yldjXEJ/bymhA=
+github.com/aws/aws-sdk-go-v2 v1.42.0/go.mod h1:27+ACypSLljLAEKsCYOmrjKh83vuTRkuAe9Uv/3A4bg=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.13 h1:p1BBrg/Hhp6uK7zpejeI8QFXHJeC/mynzi04Sl03k9g=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.13/go.mod h1:8cIfkE9MDhkRZGpQ22aV6/lkYeYSozpz16Smrs5x4Ls=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.24 h1:2hQqYCV9yqyePQ9o6dCrZc/zO8U3TwPr9mIKlZnPu/I=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.24/go.mod h1:IDwpACtwqHLISdzfwUUNq4P9DsB/h5BLg4FwJPNfqFY=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.29 h1:f3vKqSo13fhTYb+JEcXwXefZQE26I1FB5eTSniU67ko=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.29/go.mod h1:MzoLFUArKGpGD+ukmPiTPG1X5x4o6M2kq4v2dr1FiEc=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.29 h1:RdwIf/CuUsvJX3RgJagbOyotl/cxoLY4xviKuE7p2GY=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.29/go.mod h1:71wt8W2EgswdZy9Mf9KNnzxZ3TiZlv4caKghPktDOkA=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.30 h1:VTGy885W5DKBxWRUJbym9hytNaYzsyaPkCHGRRMAOhU=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.30/go.mod h1:AS0HycUvJRFvTt613AYDOgO2jzw+00cVSMny8XB3yMY=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.12 h1:ZD2+BSw9vFsNlKYIasSNt3uDbjqqXIBcM13UJv/Lx2k=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.12/go.mod h1:Ms4zlcVBbXbiP7EVLhl+lgjvA/a7YphqQ3Ih3174EmI=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.22 h1:V51LGlOq/1VsDsHUdoklAQi7rMmx4qQubvFYAlP2254=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.22/go.mod h1:4Pzhyz8hJOm2bepgl+NjvRx8vlUFAIIvJnZ/MkcNPpU=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.29 h1:DRebniUGZ2MqiiIVmQJ04vIXr918hubdHMnarSLEWyU=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.29/go.mod h1:LfRkPCD8YHDM2E5eTkos2UpwYeZnBcVarTa8L59bJHA=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.29 h1:hiME6pBzC7OTl9LMtlyTWBuEl1f4QBcUmFDKC7MLXtc=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.29/go.mod h1:G7RP+uhagpKtKhd1BM9N6JQqjCcGEU47K5lBVZQyRQw=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.103.3 h1:JRseEu/vIDMaWis4bSw0QbXL+cvIGc1XnX076H5ZXLE=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.103.3/go.mod h1:77ZAgynvx1txMvDG8gGWoWkO1augYDxkp9JElWFgjQU=
+github.com/aws/smithy-go v1.27.1 h1:4T340VFndXtADGF52gYa1POyL7s9E4Z1OeZ1hCscIw8=
+github.com/aws/smithy-go v1.27.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
@@ -103,8 +103,8 @@ github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6N
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/dgraph-io/badger/v4 v4.9.1 h1:DocZXZkg5JJHJPtUErA0ibyHxOVUDVoXLSCV6t8NC8w=
-github.com/dgraph-io/badger/v4 v4.9.1/go.mod h1:5/MEx97uzdPUHR4KtkNt8asfI2T4JiEiQlV7kWUo8c0=
+github.com/dgraph-io/badger/v4 v4.9.2 h1:Wb5qw8gElqwV1a8msHTeQKova9b1V10heFKMIiPd80E=
+github.com/dgraph-io/badger/v4 v4.9.2/go.mod h1:nJjaJTUOSsQEBhsq209FmwCvMJzEA3e74RjZw6V2pQI=
 github.com/dgraph-io/ristretto/v2 v2.4.0 h1:I/w09yLjhdcVD2QV192UJcq8dPBaAJb9pOuMyNy0XlU=
 github.com/dgraph-io/ristretto/v2 v2.4.0/go.mod h1:0KsrXtXvnv0EqnzyowllbVJB8yBonswa2lTCK2gGo9E=
 github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da h1:aIftn67I1fkbMa512G+w+Pxci9hJPB8oMnkcP3iZF38=
@@ -288,22 +288,22 @@ github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELU
 github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
 github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
 github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
-github.com/mulgadc/predastore v1.6.0 h1:a0xJtrmAvmLlc19B5xUR9upbkl7VlDntpAKnxbkM07E=
-github.com/mulgadc/predastore v1.6.0/go.mod h1:RJh0V0TG27s0q6n0o4CIQFu7CjjPYv4yGnKpIiP9tLs=
-github.com/mulgadc/viperblock v1.6.0 h1:mFx0eVOJoT3WL+X/GPrZOI0ySqerpiCtUW4vPpUxbtI=
-github.com/mulgadc/viperblock v1.6.0/go.mod h1:cwE2FVrvxUD+QaMWnAZeKVnxF8VtTG9zxiUaP1m5oL0=
+github.com/mulgadc/predastore v1.8.0 h1:Di/PlAP8nIVjXMmjVPuS9UtqQkqMjR41RrC5J21vHmw=
+github.com/mulgadc/predastore v1.8.0/go.mod h1:TC6I4Z27RCe2gPqlxqUehJ2J+/jKggnH+4DyMq1H/ng=
+github.com/mulgadc/viperblock v1.8.0 h1:kVyM5RxomokVa1uA2kbhtS02kZ9aDXKVj4MvGeqsKiM=
+github.com/mulgadc/viperblock v1.8.0/go.mod h1:18zxN4szsp5h6w8nTqkZWP0SWmIrLU3NdpKXDYNanls=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
-github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
-github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
-github.com/nats-io/nats-server/v2 v2.14.1 h1:wXs/a5fw9Hzm3CvuzLxGeIwpjPulSa7gMT3eSuhGkcg=
-github.com/nats-io/nats-server/v2 v2.14.1/go.mod h1:4N17zLpuS7WMbG8T9gsE2B7z9hC9PraPyulVBfpK6nU=
+github.com/nats-io/jwt/v2 v2.8.2 h1:XXRgB60MSTnqsRwejQurVDs/hcv2dkt+86GjI+I/bMc=
+github.com/nats-io/jwt/v2 v2.8.2/go.mod h1:Ag/56sq9OblL4JgdYufDd16Egb17Kr/8WwwuO/forVc=
+github.com/nats-io/nats-server/v2 v2.14.2 h1:Q7dRhCY03Y00rETFW3KV+KGaCIajlDfWgWUVgbMxyuk=
+github.com/nats-io/nats-server/v2 v2.14.2/go.mod h1:lWpb1bSpRELZfRdlMkdz8E7lbXKKyNe8RIn0vvepIHs=
 github.com/nats-io/nats.go v1.52.0 h1:n3avV4VBsCgsdwh71TppsTwtv+QdPs7ntSKM8qJLGsc=
 github.com/nats-io/nats.go v1.52.0/go.mod h1:26HypzazeOkyO3/mqd1zZd53STJN0EjCYF9Uy2ZOBno=
-github.com/nats-io/nkeys v0.4.15 h1:JACV5jRVO9V856KOapQ7x+EY8Jo3qw1vJt/9Jpwzkk4=
-github.com/nats-io/nkeys v0.4.15/go.mod h1:CpMchTXC9fxA5zrMo4KpySxNjiDVvr8ANOSZdiNfUrs=
+github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
+github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
 github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
 github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
 github.com/ovn-kubernetes/libovsdb v0.8.1 h1:M2J8bcJt5mXCom0HqzfEtuHkT80CTSQRcYG7acT8gf4=
@@ -353,8 +353,10 @@ github.com/pterm/pterm v0.12.36/go.mod h1:NjiL09hFhT/vWjQHSj1athJpx6H8cjpHXNAK5b
 github.com/pterm/pterm v0.12.40/go.mod h1:ffwPLwlbXxP+rxT0GsgDTzS3y3rmpAO1NMjUkGTYf8s=
 github.com/pterm/pterm v0.12.83 h1:ie+YmGmA727VuhxBlyGr74Ks+7McV6kT99IB8EU80aA=
 github.com/pterm/pterm v0.12.83/go.mod h1:xlgc6bFWyJIMtmLJvGim+L7jhSReilOlOnodeIYe4Tk=
-github.com/quic-go/quic-go v0.59.1 h1:0Gmua0HW1Tv7ANR7hUYwRyD0MG5OJfgvYSZasGZzBic=
-github.com/quic-go/quic-go v0.59.1/go.mod h1:upnsH4Ju1YkqpLXC305eW3yDZ4NfnNbmQRCMWS58IKU=
+github.com/quic-go/go-ossfuzz-seeds v0.1.0 h1:APacT+iIaNF6fd8AGEiN3bT/Jtkd2jz4v4TzM7MFjy0=
+github.com/quic-go/go-ossfuzz-seeds v0.1.0/go.mod h1:3IOHRbJIc+L6YKMwfDtJAM9Vj9k0YY4muhuyUYk5tbk=
+github.com/quic-go/quic-go v0.60.0 h1:xcQioE8OM66UQLeUMHltK1CCcOu3JbVB4JAQdDQSB+0=
+github.com/quic-go/quic-go v0.60.0/go.mod h1:wpKpjmPpftl30sL6pFh7REVpjbcCVy4zt2vDyK1TuJk=
 github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
 github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
 github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
@@ -426,14 +428,14 @@ golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnf
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
-golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
-golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
+golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
+golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa h1:FRnLl4eNAQl8hwxVVC17teOw8kdjVDVAiFMtgUdTSRQ=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa/go.mod h1:zk2irFbV9DP96SEBUUAy67IdHUaZuSnrz1n472HUCLE=
 golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
 golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
-golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM=
-golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU=
+golang.org/x/mod v0.36.0 h1:JJjpVx6myfUsUdAzZuOSTTmRE0PfZeNWzzvKrP7amb4=
+golang.org/x/mod v0.36.0/go.mod h1:moc6ELqsWcOw5Ef3xVprK5ul/MvtVvkIXLziUOICjUQ=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -444,8 +446,8 @@ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81R
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
-golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
-golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
+golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
+golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -454,8 +456,8 @@ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
-golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
+golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
+golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -486,33 +488,33 @@ golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
-golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
-golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
-golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa h1:efT73AJZfAAUV7SOip6pWGkwJDzIGiKBZGVzHYa+ve4=
-golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa/go.mod h1:kHjTxDEnAu6/Nl9lDkzjWpR+bmKfxeiRuSDlsMb70gE=
+golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
+golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6 h1:HjU6IWBiAgRIdAJ9/y1rwCn+UELEmwV+VsTLzj/W4sE=
+golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6/go.mod h1:Eqhaxk/wZsWEH8CRxLwj6xzEJbz7k1EFGqx7nyCoabE=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
-golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4=
-golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=
+golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc=
+golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
-golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
-golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
+golang.org/x/text v0.38.0 h1:sXmwo9DwP3OK9EZ7PqAdaooSGozfl/3a6/xJcbzPRhE=
+golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4=
 golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
 golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
 golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
-golang.org/x/tools v0.44.0 h1:UP4ajHPIcuMjT1GqzDWRlalUEoY+uzoZKnhOjbIPD2c=
-golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI=
+golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8=
+golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0=
 golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM=
 golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY=
 golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM=
@@ -536,8 +538,8 @@ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
-gopkg.in/ini.v1 v1.67.2 h1:JtOSMb9OuaCZKr7h5D/h6iii14sK0hLbplTc6frx4Ss=
-gopkg.in/ini.v1 v1.67.2/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
+gopkg.in/ini.v1 v1.67.3 h1:iM9Lhz5MRSGhHVGGwCuzG9KO8PoirCXj/m/qTmOJJQw=
+gopkg.in/ini.v1 v1.67.3/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

internal/eksgw/client.go

@@ -0,0 +1,112 @@
+// Package eksgw is the on-VM SigV4 client used by EKS helpers (eks-gateway-publish,
+// eks-token-webhook) to POST over HTTPS to the AWS gateway instead of NATS.
+// Signs with Predastore credentials for service "eks".
+package eksgw
+
+import (
+	"bytes"
+	"crypto/sha256"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/mulgadc/predastore/auth"
+	"github.com/mulgadc/spinifex/internal/tlsconfig"
+)
+
+// Client posts SigV4-signed requests to the AWS gateway for the "eks" service.
+type Client struct {
+	baseURL    string
+	accessKey  string
+	secretKey  string
+	region     string
+	httpClient *http.Client
+}
+
+// New builds a client. caPath optionally pins the gateway TLS CA; when empty the
+// client relies on the system trust store. region defaults to us-east-1 when
+// empty (SigV4 requires a non-empty region).
+func New(baseURL, caPath, accessKey, secretKey, region string) (*Client, error) {
+	if baseURL == "" {
+		return nil, fmt.Errorf("eksgw: baseURL is required")
+	}
+	if accessKey == "" || secretKey == "" {
+		return nil, fmt.Errorf("eksgw: access key and secret key are required")
+	}
+	if region == "" {
+		region = "us-east-1"
+	}
+
+	tlsCfg := &tls.Config{
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
+	}
+	if caPath != "" {
+		pem, err := os.ReadFile(caPath)
+		if err != nil {
+			return nil, fmt.Errorf("eksgw: read gateway CA %q: %w", caPath, err)
+		}
+		pool := x509.NewCertPool()
+		if !pool.AppendCertsFromPEM(pem) {
+			return nil, fmt.Errorf("eksgw: gateway CA %q has no usable certificates", caPath)
+		}
+		tlsCfg.RootCAs = pool
+	}
+
+	return &Client{
+		baseURL:   strings.TrimRight(baseURL, "/"),
+		accessKey: accessKey,
+		secretKey: secretKey,
+		region:    region,
+		httpClient: &http.Client{
+			Timeout:   10 * time.Second,
+			Transport: &http.Transport{TLSClientConfig: tlsCfg, MaxIdleConns: 2, IdleConnTimeout: 30 * time.Second},
+		},
+	}, nil
+}
+
+// Post SigV4-signs and sends body to path, returning the response body on 2xx.
+// Errors include the gateway status and body. No retry; callers wrap as needed.
+func (c *Client) Post(path string, body []byte) ([]byte, error) {
+	return c.do(http.MethodPost, path, body)
+}
+
+// Get SigV4-signs and sends a single GET to path (e.g.
+// "/clusters/alpha/internal-addons?accountId=..."). Same contract as Post: 2xx
+// body or an error carrying the gateway status. No retry.
+func (c *Client) Get(path string) ([]byte, error) {
+	return c.do(http.MethodGet, path, nil)
+}
+
+func (c *Client) do(method, path string, body []byte) ([]byte, error) {
+	req, err := http.NewRequest(method, c.baseURL+path, bytes.NewReader(body))
+	if err != nil {
+		return nil, fmt.Errorf("create request: %w", err)
+	}
+	if body != nil {
+		req.Header.Set("Content-Type", "application/x-amz-json-1.1")
+	}
+
+	sum := sha256.Sum256(body)
+	if err := auth.SignReq(req, c.accessKey, c.secretKey, hex.EncodeToString(sum[:]), "eks", c.region); err != nil {
+		return nil, fmt.Errorf("sign request: %w", err)
+	}
+
+	resp, err := c.httpClient.Do(req)
+	if err != nil {
+		return nil, fmt.Errorf("send request: %w", err)
+	}
+	defer resp.Body.Close()
+
+	respBody, _ := io.ReadAll(resp.Body)
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		return nil, fmt.Errorf("gateway returned %d: %s", resp.StatusCode, strings.TrimSpace(string(respBody)))
+	}
+	return respBody, nil
+}

internal/fipsboot/fipsboot.go

@@ -1,12 +1,6 @@
-// Package fipsboot enforces FIPS 140-3 mode at process startup.
-//
-// Blank-import this package from every binary's main:
-//
-//	import _ "github.com/mulgadc/spinifex/internal/fipsboot"
-//
-// Builds with GOFIPS140=v1.0.0 link the Go Cryptographic Module v1.0.0
-// (NIST CMVP cert #5247). At runtime FIPS mode can still be turned off
-// via GODEBUG=fips140=off; init() panics in that case to close the hole.
+// Package fipsboot enforces FIPS 140-3 at startup via blank import in every binary.
+// Requires GOFIPS140=v1.0.0 at build time; panics if GODEBUG=fips140=off disables
+// the runtime FIPS mode.
 package fipsboot
 
 import (

internal/tlsconfig/curves.go

@@ -1,34 +1,13 @@
 // Package tlsconfig pins the cluster-wide TLS 1.3 curve preference list.
-//
-// The list is hardcoded because mixed classification on a single cluster is
-// not supported and the certified FIPS build is itself the per-deployment
-// choice. A typo in a config string would silently downgrade where a missing
-// constant cannot.
+// Hardcoded to prevent silent downgrade via misconfiguration; the FIPS build
+// is the per-deployment choice.
 package tlsconfig
 
 import "crypto/tls"
 
-// Curves is the fixed TLS 1.3 curve allowlist:
-//
-//   - X25519MLKEM768     — Cat 3 ML-KEM hybrid, wins under Go 1.26 stdlib filter
-//   - SecP384r1MLKEM1024 — Cat 5 ML-KEM hybrid, fallback for peers that prefer it
-//   - X25519             — classical TLS 1.3 fallback for SDKs without ML-KEM (aws CLI v2 today)
-//   - CurveP384          — classical TLS 1.3 fallback (Java/.NET enterprise SDKs)
-//
-// PQ-capable peers (browsers since 2024, Go aws-sdk v2 with Go 1.24+) always
-// negotiate the hybrid first; classical fallback only triggers for peers that
-// cannot do ML-KEM. Those connections still get TLS 1.3 + AES-256-GCM but
-// remain HNDL-exposed — accepted trade-off until AWS CLI v2 / bundled
-// OpenSSL ships ML-KEM (currently lagging upstream OpenSSL 3.5).
-//
-// Weak/niche entries from the Go stdlib default (SecP256r1MLKEM768, P-256,
-// P-521) are intentionally excluded so they cannot be advertised.
-//
-// Go 1.26 stdlib treats tls.Config.CurvePreferences as an allowlist filter
-// applied to the stdlib default order, not as a user-supplied order.
-//
-// Callers must not mutate this slice. Assign it directly to
-// tls.Config.CurvePreferences; do not append to it.
+// Curves is the fixed TLS 1.3 curve allowlist: two ML-KEM hybrids (PQ-capable
+// peers always negotiate these) followed by two classical fallbacks for SDKs
+// without ML-KEM. Weak Go defaults (SecP256r1MLKEM768, P-256, P-521) excluded.
 var Curves = []tls.CurveID{
 	tls.X25519MLKEM768,
 	tls.SecP384r1MLKEM1024,

internal/tlsconfig/curves_test.go

@@ -46,11 +46,9 @@ func TestCurves_ExcludesWeakPrimitives(t *testing.T) {
 	}
 }
 
-// TestIntegration_NegotiatesPQHybrid spins up an httptest TLS server
-// configured with Curves + TLS 1.3 minimum, dials it, and asserts a PQ
-// hybrid is negotiated at TLS 1.3. Per Go 1.26 stdlib filter behavior the
-// winner is X25519MLKEM768; the assertion checks for any approved hybrid
-// to remain stable across stdlib reorderings of our 4 entries.
+// TestIntegration_NegotiatesPQHybrid verifies that a Curves-configured server
+// negotiates a PQ hybrid at TLS 1.3; checks for any approved hybrid to stay
+// stable across Go stdlib reorderings.
 func TestIntegration_NegotiatesPQHybrid(t *testing.T) {
 	cert := selfSignedCert(t)
 	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
@@ -95,10 +93,8 @@ func TestIntegration_NegotiatesPQHybrid(t *testing.T) {
 	}
 }
 
-// TestIntegration_TLS12ClientRejected confirms MinVersion: tls.VersionTLS13
-// is enforced — a TLS 1.2-pinned client gets a handshake error rather than
-// silently downgrading. This is the negative test that closes the HNDL gap
-// on external surfaces.
+// TestIntegration_TLS12ClientRejected confirms MinVersion TLS 1.3 is enforced:
+// a TLS 1.2-pinned client must get a handshake error, not a silent downgrade.
 func TestIntegration_TLS12ClientRejected(t *testing.T) {
 	cert := selfSignedCert(t)
 	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {

scripts/ansible/.gitignore

@@ -1,2 +0,0 @@
-.creds/
-*.retry

scripts/ansible/README.md

@@ -1,222 +0,0 @@
-# Ansible Dev Lifecycle (parallel to bash scripts)
-
-Experimental Ansible-based lifecycle for a single-node Spinifex dev host.
-Lives alongside the existing `scripts/dev-*.sh` and `scripts/reset-dev-env.sh`;
-neither set is removed. Decide which to keep after parity testing.
-
-Plan: `docs/development/improvements/ansible-dev-lifecycle.md` (mulga top-level).
-
-## Prerequisites
-
-Venv lives at sub-repo root (`spinifex/.venv/`) — one venv per sub-repo,
-shared by any future python tooling in spinifex. Bootstrap from `spinifex/`:
-
-```
-sudo apt install python3.13-venv
-python3 -m venv .venv
-source .venv/bin/activate
-pip install 'ansible>=9'
-ansible-galaxy collection install -r scripts/ansible/requirements.yml
-```
-
-Then `cd scripts/ansible` to run playbooks — `ansible.cfg` resolves the
-inventory and roles relative to CWD.
-
-## Playbooks
-
-| Playbook | Purpose | Bash parity |
-|---|---|---|
-| `playbooks/dev-preflight.yml` | Dependency + port check | `dev-setup.sh` §checks |
-| `playbooks/dev-teardown.yml` | Wipe state only (no reinstall) | `reset-dev-env.sh` §teardown |
-| `playbooks/dev-install.yml` | Build + install + init + smoketest on clean box | `dev-install.sh` |
-| `playbooks/dev-reset.yml` | Capture settings → teardown → build → install → init → smoketest | `reset-dev-env.sh` (full) |
-| `playbooks/dev-deploy.yml` | Rebuild + swap binaries/microvm artifacts + restart (no setup.sh, no smoketest) | `make deploy` |
-| `playbooks/dev-status.yml` | Read-only health snapshot (units, ports, OVN/OVS, gateway, counts, drift) | none |
-| `playbooks/dev-logs.yml` | Dump journald + `/var/log/spinifex/*` + status/networking/versions into a tarball | none |
-| `playbooks/dev-snapshot.yml` | Quiesce spinifex+OVN+OVS, tar state paths to a named bundle, restart | none |
-| `playbooks/dev-restore.yml` | Stop services, wipe + untar a named bundle, refresh CA, restart | none |
-| `playbooks/dev-version.yml` | Build-vs-install drift report (spx, plugin, microvm) + sub-repo HEADs | none |
-| `playbooks/dev-vpc.yml` | Idempotent dev VPC fixture (VPC + public subnet + IGW + default route); `-e dev_vpc_state=absent` tears down | partial overlap with `scripts/iac/examples/*` |
-
-Upcoming (not yet implemented):
-
-- `dev-mode-start.yml` / `dev-mode-stop.yml` — parity with `start-dev.sh` / `stop-dev.sh`
-  (lowest priority — deprecated bash scripts, CI-only callers).
-
-## Invocation
-
-```
-ansible-playbook playbooks/dev-preflight.yml
-ansible-playbook playbooks/dev-teardown.yml
-ansible-playbook playbooks/dev-install.yml
-ansible-playbook playbooks/dev-reset.yml
-ansible-playbook playbooks/dev-deploy.yml
-ansible-playbook playbooks/dev-status.yml
-ansible-playbook playbooks/dev-logs.yml
-ansible-playbook playbooks/dev-snapshot.yml -e snapshot_name=before-merge
-ansible-playbook playbooks/dev-restore.yml  -e snapshot_name=before-merge
-ansible-playbook playbooks/dev-version.yml
-ansible-playbook playbooks/dev-vpc.yml
-```
-
-Or via `make` (from `spinifex/`):
-
-```
-make ansible-dev-preflight
-make ansible-dev-teardown
-make ansible-dev-install
-make ansible-dev-reset
-make ansible-dev-deploy
-make ansible-dev-status
-make ansible-dev-logs
-make ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=before-merge'
-make ansible-dev-restore  ANSIBLE_EXTRA='-e snapshot_name=before-merge'
-make ansible-dev-version
-make ansible-dev-vpc
-make ansible-dev-vpc ANSIBLE_EXTRA='-e dev_vpc_state=absent'
-```
-
-### When to use which
-
-- First-time / clean box → `ansible-dev-install`
-- Iterate on Go code, microvm initramfs, lb-agent → `ansible-dev-deploy` (fast)
-- Changed systemd units, helper scripts, logrotate, setup.sh → `ansible-dev-reset` (slow, full rebuild)
-- Need a clean slate without reinstall → `ansible-dev-teardown`
-- "Is my dev box healthy?" → `ansible-dev-status` (read-only, never mutates)
-- Filing a bug / capturing the state of a broken box → `ansible-dev-logs` (writes `/tmp/spinifex-logs-<host>-<ts>.tar.gz`; override `-e logs_since=10min` or `-e logs_include_dmesg=false`)
-- Checkpoint before a risky op (branch switch, viperblock surgery, ovn restart cycle) → `ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=<name>'`; roll back via `ansible-dev-restore` instead of a full `dev-reset`. Bundles live in `/var/lib/spinifex-snapshots/<name>.tar.gz` (root-owned, survives `dev-teardown`).
-- "Do I need to `ansible-dev-deploy`?" → `ansible-dev-version` (drift-only check; faster than `dev-status`)
-- Want a known-good VPC/subnet for manual EC2 testing without writing tofu → `ansible-dev-vpc` (smoketest will then find and use it automatically). Tear down with `ANSIBLE_EXTRA='-e dev_vpc_state=absent'`.
-
-## Variable overrides
-
-Defaults in `vars/defaults.yml`. Override per run with `-e`:
-
-```
-ansible-playbook playbooks/dev-teardown.yml -e spinifex_wipe_ssh_keys=true
-```
-
-Useful overrides:
-
-| Var | Default | What it does |
-|---|---|---|
-| `spinifex_wipe_legacy_home` | `true` | Remove `$HOME/spinifex` (legacy dev-mode path) |
-| `spinifex_wipe_ssh_keys` | `false` | Remove `~/.ssh/spinifex-key*` |
-| `spinifex_wipe_aws_creds` | `false` | Remove spinifex profile from `~/.aws/credentials` |
-| `teardown_process_wait_seconds` | `30` | Grace period for processes to exit |
-| `spinifex_region` | `ap-southeast-2` | Region passed to `spx admin init` |
-| `spinifex_az` | `{{ region }}a` | AZ passed to `spx admin init` |
-| `spinifex_external_mode` | `pool` | External networking mode — `pool` or empty (NAT removed) |
-| `spinifex_external_no_external` | `false` | `true` → emit `--no-external` and skip external networking |
-| `spinifex_external_pool_start` | `""` | **Required** for `pool` mode — first IP of the static WAN pool |
-| `spinifex_external_pool_end` | `""` | **Required** for `pool` mode — last IP of the static WAN pool |
-| `spinifex_external_gateway` | `""` | WAN gateway (auto-detected when blank) |
-| `spinifex_external_prefix_len` | `""` | WAN prefix length (auto-detected when blank) |
-| `spinifex_external_gateway_ip` | `""` | OVN router external-IP override (`--gateway-ip`) |
-
-`pool` mode is mandatory unless you set `spinifex_external_no_external=true`.
-`spx admin init` rejects the run without a `--external-pool` range, so the
-init role asserts `spinifex_external_pool_start/_end` upfront. Either persist
-them in `vars/defaults.yml`, or pass them at invocation:
-
-Direct `ansible-playbook` (use `-e` here — this is ansible's flag):
-
-```
-ansible-playbook playbooks/dev-install.yml \
-  -e spinifex_external_pool_start=192.168.1.150 \
-  -e spinifex_external_pool_end=192.168.1.250
-```
-
-Via `make` (use `EXTRA_VARS=` — make eats its own `-e`):
-
-```
-make ansible-dev-install \
-  EXTRA_VARS="spinifex_external_pool_start=192.168.1.150 spinifex_external_pool_end=192.168.1.250"
-```
-
-`dev-reset.yml` automatically captures `region`, `az`, `external_mode`,
-pool range, gateway, prefix-len and `gateway_ip` from the existing
-`/etc/spinifex/spinifex.toml` before teardown and replays them into
-`init`. Legacy `external_mode = "nat"` is coerced to `"pool"` with a warn.
-Use `-e` to override.
-
-## Helper scripts
-
-`scripts/aws-as-spinifex.sh` and `scripts/spx-as-spinifex.sh` wrap `aws`
-and `spx` invocations in `sg spinifex -c '...'` so they run with the
-`spinifex` supplementary group active. Both helpers preset
-`AWS_PROFILE=spinifex`. All AWS-calling tasks in the roles use these
-via `spinifex_aws_cli` / `spinifex_spx_cli` from `vars/defaults.yml`.
-
-Background: `/etc/spinifex/ca.pem` lives under `drwxr-x--- root:spinifex`,
-so the AWS CLI needs `spinifex` group membership to read the CA bundle.
-Ansible's `become_user` is a no-op when the target user equals the
-calling user (no `sudo -u` invoked → no `initgroups(3)` → supplementary
-groups inherited from the operator's shell). On a fresh login the
-operator's shell would have `spinifex` (added by `setup.sh`); on shells
-that predate the group add it does not. The helpers sidestep this
-entirely.
-
-## Leak catalog
-
-`roles/teardown/README.md` tracks every known state source. When teardown
-misses something, add it there and in `roles/teardown/tasks/main.yml`.
-
-## Multi-node cluster bootstrap
-
-`playbooks/cluster-bootstrap.yml` provisions a 2+ node Spinifex cluster
-across a long-running tofu env (`env1`, `env2`, `bryn`, etc). Inventory
-is generated dynamically from `scripts/tofu-cluster/envs/<env>.tfvars`
-via `inventory/tofu.py` — first node in `nodes = [...]` is the init host,
-the rest are joins.
-
-```bash
-make ansible-cluster-bootstrap ENV=env1 POOL=192.168.1.150-192.168.1.159
-```
-
-`POOL` is optional; omit for DHCP fallback. Equivalent without make:
-
-```bash
-cd scripts/ansible
-CLUSTER_ENV=env1 ansible-playbook -i inventory/tofu.py \
-    playbooks/cluster-bootstrap.yml \
-    -e cluster_env=env1 \
-    -e cluster_external_pool=192.168.1.150-192.168.1.159
-```
-
-Plays sequence:
-
-1. **runner (localhost)** — `build` role produces `/tmp/spinifex-local.tar.gz`.
-2. **all env hosts (parallel)** — copies tarball, stops any dev-layout
-   services (`~/spinifex/`), runs `preflight` + `install` roles. `setup.sh`
-   creates `/etc/spinifex`, the `spinifex` user/group, systemd units, and
-   installs `/usr/local/bin/spx`.
-3. **init host only** — `cluster-init` role runs `spx admin init` in the
-   background, waits for the formation server, captures the join token.
-   CA is then copied into `/usr/local/share/ca-certificates/spinifex-ca.crt`
-   **before** `spinifex.target` starts (required — predastore enforces
-   strict TLS verify against the system trust store).
-4. **join hosts (parallel)** — `cluster-join` role runs `spx admin join`
-   against the formation server, then installs CA, then starts
-   `spinifex.target`.
-5. **all hosts** — polls `https://<ip>:4432/health` until `awsgw=ok`.
-6. **init host only** — `cluster-post` role imports the SSH key + Ubuntu
-   AMI (both idempotent: skip if already present).
-
-Re-running the playbook against an already-bootstrapped env is safe —
-`setup.sh`, `spx admin init --force`, CA copy, key/AMI import are all
-idempotent.
-
-### Migration from `bootstrap.sh`
-
-`scripts/tofu-cluster/bootstrap.sh` ran services under `~/spinifex/` as
-`tf-user` via `start-dev.sh` with no systemd supervision, and installed
-the cluster CA *after* services started — predastore would crash with
-`x509: certificate signed by unknown authority` on every first boot.
-`cluster-bootstrap.yml` replaces that flow with production-layout
-(`/etc/spinifex/` + systemd) and correct CA-before-services ordering.
-The first run on an env previously bootstrapped by `bootstrap.sh` will
-stop the dev-layout services, install the prod layout, and re-init the
-cluster from scratch via `spx admin init --force`.
-
-`bootstrap.sh` stays in-tree for CI and pseudo-multinode envs.

scripts/ansible/ansible.cfg

@@ -1,17 +0,0 @@
-[defaults]
-inventory = inventory/localhost.yml
-roles_path = roles
-stdout_callback = ansible.builtin.default
-host_key_checking = False
-retry_files_enabled = False
-interpreter_python = auto_silent
-forks = 10
-
-[inventory]
-# inventory/tofu.py is an executable script inventory used by
-# cluster-bootstrap.yml — opted into via `-i inventory/tofu.py` on the
-# command line; not default to keep dev-* playbooks pointing at localhost.
-enable_plugins = host_list, script, auto, yaml, ini, toml
-
-[callback_default]
-result_format = yaml

scripts/ansible/inventory/localhost.yml

@@ -1,6 +0,0 @@
----
-all:
-  hosts:
-    localhost:
-      ansible_connection: local
-      ansible_python_interpreter: "{{ ansible_playbook_python }}"

scripts/ansible/inventory/tofu.py

@@ -1,170 +0,0 @@
-#!/usr/bin/env python3
-"""Dynamic ansible inventory generator from tofu-cluster tfvars.
-
-Reads scripts/tofu-cluster/envs/<env>.tfvars (HCL subset) and emits ansible
-JSON inventory. First node in the tfvars `nodes = [...]` list is the init
-host (forms the cluster); the rest are join hosts.
-
-Usage:
-    inventory/tofu.py --list --env env1
-    inventory/tofu.py --host <ip> --env env1
-
-Selecting the env:
-    CLUSTER_ENV=env1 inventory/tofu.py --list
-    inventory/tofu.py --list --env env1
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import os
-import re
-import sys
-from pathlib import Path
-
-
-def _repo_root() -> Path:
-    here = Path(__file__).resolve()
-    for parent in here.parents:
-        if (parent / "go.work").exists() or (parent / ".gitmodules").exists():
-            return parent
-    raise SystemExit(f"could not locate mulga repo root from {here}")
-
-
-def _tfvars_path(env: str) -> Path:
-    root = _repo_root()
-    candidate = root / "scripts" / "tofu-cluster" / "envs" / f"{env}.tfvars"
-    if not candidate.exists():
-        raise SystemExit(f"tfvars file not found: {candidate}")
-    return candidate
-
-
-_STRING_RE = re.compile(r'^\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*=\s*"([^"]*)"\s*$')
-_NUMBER_RE = re.compile(r'^\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*=\s*([0-9]+)\s*$')
-_NODE_FIELD_RE = re.compile(r'(\w+)\s*=\s*"([^"]*)"')
-
-
-def _parse_tfvars(path: Path) -> dict:
-    """Minimal tfvars parser — handles the subset used by envs/*.tfvars.
-
-    Supports:
-      key = "string"
-      key = 12345
-      nodes = [
-        { name = "x", wan_ip = "1.2.3.4", ... },
-        ...
-      ]
-    """
-    out: dict = {"nodes": []}
-    text = path.read_text()
-
-    # Strip line comments
-    text = re.sub(r"#.*", "", text)
-
-    # Pull the nodes block out first, then process remaining lines.
-    nodes_match = re.search(r"nodes\s*=\s*\[(.+?)\]\s*$", text, re.DOTALL | re.MULTILINE)
-    if nodes_match:
-        nodes_body = nodes_match.group(1)
-        for entry in re.finditer(r"\{([^}]+)\}", nodes_body):
-            fields = dict(_NODE_FIELD_RE.findall(entry.group(1)))
-            if "name" in fields and "wan_ip" in fields:
-                out["nodes"].append(fields)
-        text = text.replace(nodes_match.group(0), "")
-
-    for line in text.splitlines():
-        if m := _STRING_RE.match(line):
-            out[m.group(1)] = m.group(2)
-        elif m := _NUMBER_RE.match(line):
-            out[m.group(1)] = int(m.group(2))
-
-    return out
-
-
-def _resolve_ssh_key(value: str | None) -> str:
-    if not value:
-        return os.path.expanduser("~/.ssh/tf-user-ap-southeast-2")
-    return os.path.expanduser(value)
-
-
-def _build_inventory(env: str) -> dict:
-    data = _parse_tfvars(_tfvars_path(env))
-    nodes = data.get("nodes") or []
-    if not nodes:
-        raise SystemExit(f"{env}.tfvars has no nodes")
-
-    ssh_user = data.get("ssh_user", "tf-user")
-    ssh_key = _resolve_ssh_key(data.get("ssh_public_key_path", "").replace(".pub", ""))
-
-    hostvars: dict[str, dict] = {}
-    all_hosts: list[str] = []
-    init_hosts: list[str] = []
-    join_hosts: list[str] = []
-
-    for idx, node in enumerate(nodes):
-        ip = node["wan_ip"]
-        role = "init" if idx == 0 else "join"
-        hostvars[ip] = {
-            "ansible_host": ip,
-            "ansible_user": ssh_user,
-            "ansible_ssh_private_key_file": ssh_key,
-            "ansible_ssh_common_args": "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR",
-            "spinifex_node_name": node["name"],
-            "spinifex_node_role": role,
-            "spinifex_node_index": idx,
-            "spinifex_wan_ip": ip,
-        }
-        all_hosts.append(ip)
-        (init_hosts if role == "init" else join_hosts).append(ip)
-
-    # Group-level vars derived from tfvars (operator can override with -e).
-    cluster_init_host = init_hosts[0]
-    group_vars = {
-        "cluster_env": env,
-        "cluster_init_host": cluster_init_host,
-        "cluster_init_ip": cluster_init_host,
-        "spinifex_node_count": len(nodes),
-        "spinifex_region": data.get("region", "ap-southeast-2"),
-        "spinifex_az": data.get("az", "ap-southeast-2a"),
-    }
-    if cidr := data.get("env_cidr"):
-        group_vars["cluster_env_cidr"] = cidr
-
-    inventory = {
-        "_meta": {"hostvars": hostvars},
-        env: {"hosts": all_hosts, "vars": group_vars},
-        "cluster": {"children": [env]},
-        "init": {"hosts": init_hosts},
-        "join": {"hosts": join_hosts},
-    }
-    return inventory
-
-
-def main() -> int:
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("--list", action="store_true", help="emit full inventory")
-    parser.add_argument("--host", help="emit hostvars for a single host (ignored — _meta carries them)")
-    parser.add_argument(
-        "--env",
-        default=os.environ.get("CLUSTER_ENV"),
-        help="tofu env name (matches scripts/tofu-cluster/envs/<env>.tfvars); also via CLUSTER_ENV env var",
-    )
-    args = parser.parse_args()
-
-    if not args.env:
-        sys.stderr.write("error: --env required (or CLUSTER_ENV)\n")
-        return 64
-
-    if args.host:
-        json.dump({}, sys.stdout)
-        sys.stdout.write("\n")
-        return 0
-
-    inventory = _build_inventory(args.env)
-    json.dump(inventory, sys.stdout, indent=2)
-    sys.stdout.write("\n")
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

scripts/ansible/playbooks/cluster-bootstrap.yml

@@ -1,330 +0,0 @@
----
-# cluster-bootstrap.yml — multi-node Spinifex cluster bootstrap against a
-# tofu-provisioned env. Inventory comes from inventory/tofu.py which parses
-# scripts/tofu-cluster/envs/<env>.tfvars.
-#
-# Usage:
-#   cd spinifex/scripts/ansible
-#   ansible-playbook -i inventory/tofu.py playbooks/cluster-bootstrap.yml \
-#     -e cluster_env=env1 -e cluster_external_pool=192.168.1.150-192.168.1.159
-#
-# Or via the make wrapper:
-#   make ansible-cluster-bootstrap ENV=env1 POOL=192.168.1.150-192.168.1.159
-#
-# Behaviour:
-#   - all hosts: preflight + install (setup.sh) — production layout
-#   - init host: spx admin init (backgrounded) + CA install + spinifex.target
-#   - join hosts: spx admin join + CA install + spinifex.target (parallel)
-#   - init host: import ssh key + ubuntu AMI (idempotent)
-
-- name: Cluster bootstrap — build artifact once on runner
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: assert cluster_env supplied
-      ansible.builtin.assert:
-        that: cluster_env | default('') | length > 0
-        fail_msg: "Pass -e cluster_env=<env> (matches scripts/tofu-cluster/envs/<env>.tfvars)"
-  roles:
-    - build
-
-- name: Cluster bootstrap — install (all nodes, parallel)
-  hosts: "{{ cluster_env }}"
-  gather_facts: true
-  serial: 100%
-  vars:
-    install_run_ovn: false   # we drive setup-ovn ourselves with role-aware args
-    cluster_bootstrap_purge: true   # wipe stale single-node predastore.toml + raft data
-  vars_files:
-    - ../vars/defaults.yml
-  pre_tasks:
-    - name: copy build tarball to each node
-      ansible.builtin.copy:
-        src: "{{ build_tarball_path }}"
-        dest: "{{ build_tarball_path }}"
-        mode: "0644"
-
-    - name: stop dev-layout services if running (migration from ~/spinifex/)
-      become: true
-      # Bracket trick on the first letter prevents the pattern literal in the
-      # shell command line from matching the running shell itself (pkill -f
-      # against literal patterns is otherwise self-killing — rc=-9).
-      # `spx [a]dmin` catches lingering formation servers from prior bootstrap
-      # attempts that own port 4432 and steal joins from the new init.
-      ansible.builtin.shell:
-        cmd: |
-          pkill -9 -f "[b]in/spx service" 2>/dev/null || true
-          pkill -9 -f "spx [a]dmin" 2>/dev/null || true
-          pkill -9 -f "spinifex-[u]i" 2>/dev/null || true
-          pkill -9 -f "[n]ats-server" 2>/dev/null || true
-          pkill -9 -f "[p]redastore" 2>/dev/null || true
-          pkill -9 -f "[v]iperblock" 2>/dev/null || true
-          sleep 2
-          rm -rf "{{ ansible_env.HOME }}/spinifex" 2>/dev/null || true
-          exit 0
-      changed_when: false
-  roles:
-    - install
-
-# Wipe OVN/OVS state before setup-ovn runs. Without this, re-bootstrapping a
-# previously-built cluster leaves stale chassis rows in the SB DB, stale
-# external_ids on Open_vSwitch, stale br-int flows + ports, stale
-# systemd-networkd unit files, and stale ovs-monitor-ipsec strongSwan certs.
-# Symptom: ovn-controller flows reference dead encap endpoints; IPsec SAs
-# fail AUTHENTICATION_FAILED; new joins shadow ghosts under the same
-# node-name.
-- name: Cluster bootstrap — OVN/OVS state reset (all nodes, parallel)
-  hosts: "{{ cluster_env }}"
-  gather_facts: false
-  serial: 100%
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: reset — stop OVN/OVS services
-      become: true
-      ansible.builtin.systemd:
-        name: "{{ item }}"
-        state: stopped
-      loop:
-        - ovn-controller
-        - ovn-central
-        - openvswitch-ipsec
-        - openvswitch-switch
-      failed_when: false
-
-    - name: reset — delete OVN NB/SB databases
-      become: true
-      ansible.builtin.file:
-        path: "{{ item }}"
-        state: absent
-      loop:
-        - "{{ ovn_nb_db }}"
-        - "{{ ovn_sb_db }}"
-
-    - name: reset — delete OVS conf.db + system-id (force fresh ovsdb)
-      become: true
-      ansible.builtin.file:
-        path: "{{ item }}"
-        state: absent
-      loop:
-        - /etc/openvswitch/conf.db
-        - /etc/openvswitch/.conf.db.~lock~
-        - /etc/openvswitch/system-id.conf
-
-    - name: reset — drop spinifex-managed systemd-networkd unit files
-      become: true
-      ansible.builtin.shell: |
-        rm -f /etc/systemd/network/14-spinifex-*.netdev \
-              /etc/systemd/network/15-spinifex-*.netdev \
-              /etc/systemd/network/15-spinifex-*.network \
-              /etc/systemd/network/16-spinifex-*.network
-        networkctl reload 2>/dev/null || true
-      changed_when: true
-
-    - name: reset — enumerate stale spinifex veth/tap netdevs
-      ansible.builtin.shell: |
-        ip -o link show | awk -F': ' '{print $2}' | cut -d'@' -f1 | \
-          grep -E '^(veth-wan-br|veth-wan-ovs|spx-ext-|spx-tap-|spx-)' || true
-      register: reset_links
-      changed_when: false
-
-    - name: reset — delete stale netdevs
-      become: true
-      ansible.builtin.command: "ip link delete {{ item }}"
-      loop: "{{ reset_links.stdout_lines }}"
-      changed_when: true
-      failed_when: false
-
-    - name: reset — clear ovs-monitor-ipsec strongSwan state
-      become: true
-      ansible.builtin.shell: |
-        rm -rf /var/lib/openvswitch/pki 2>/dev/null || true
-        rm -f /etc/ipsec.d/certs/ovs-* /etc/ipsec.d/private/ovs-* 2>/dev/null || true
-        rm -f /etc/ipsec.conf.d/ovs-* 2>/dev/null || true
-      changed_when: true
-
-    - name: reset — start openvswitch-switch with fresh conf.db
-      become: true
-      ansible.builtin.systemd:
-        name: openvswitch-switch
-        state: started
-        daemon_reload: true
-
-- name: Cluster bootstrap — OVN central (init host only)
-  hosts: init
-  gather_facts: false
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: setup-ovn --management on init host
-      become: true
-      ansible.builtin.command:
-        cmd: >-
-          /usr/local/share/spinifex/setup-ovn.sh
-          --management
-          --encap-ip={{ spinifex_wan_ip }}
-          --node-name={{ hostvars[inventory_hostname]['spinifex_node_name'] }}
-      changed_when: true
-
-- name: Cluster bootstrap — OVN chassis (join hosts, parallel)
-  hosts: join
-  gather_facts: false
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: setup-ovn --ovn-remote on join hosts
-      become: true
-      ansible.builtin.command:
-        cmd: >-
-          /usr/local/share/spinifex/setup-ovn.sh
-          --ovn-remote=tcp:{{ cluster_init_host }}:6642
-          --encap-ip={{ spinifex_wan_ip }}
-          --node-name={{ hostvars[inventory_hostname]['spinifex_node_name'] }}
-      changed_when: true
-
-- name: Cluster bootstrap — init host (start formation server)
-  hosts: init
-  gather_facts: false
-  vars_files:
-    - ../vars/defaults.yml
-  roles:
-    - cluster-init
-
-- name: Cluster bootstrap — join hosts run spx admin join (parallel)
-  hosts: join
-  gather_facts: false
-  serial: 100%
-  vars_files:
-    - ../vars/defaults.yml
-  roles:
-    - cluster-join
-
-- name: Cluster bootstrap — wait for init to finish writing multi-node configs
-  hosts: init
-  gather_facts: false
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: wait for spx admin init async job to complete
-      become: true
-      ansible.builtin.async_status:
-        jid: "{{ cluster_init_async.ansible_job_id }}"
-      register: cluster_init_done
-      until: cluster_init_done.finished
-      retries: 60
-      delay: 5
-
-    - name: dump init log on failure
-      ansible.builtin.command: tail -100 /tmp/spx-admin-init.log
-      register: cluster_init_log
-      changed_when: false
-      when: cluster_init_done.rc | default(1) != 0
-
-    - name: assert init exited cleanly
-      ansible.builtin.assert:
-        that: cluster_init_done.rc == 0
-        fail_msg: |
-          spx admin init failed (rc={{ cluster_init_done.rc }})
-          {{ cluster_init_log.stdout | default('(log not captured)') }}
-
-    - name: confirm multi-node predastore.toml written
-      become: true
-      ansible.builtin.shell: |
-        if grep -q '127\.0\.0\.1' {{ spinifex_etc_dir }}/predastore/predastore.toml 2>/dev/null; then
-          echo "ERROR: predastore.toml still references loopback — multi-node config not written"
-          exit 1
-        fi
-        grep -c '^\[\[nodes\]\]' {{ spinifex_etc_dir }}/predastore/predastore.toml 2>/dev/null || echo 0
-      register: predastore_node_count
-      changed_when: false
-
-- name: Cluster bootstrap — install CA on all nodes (before service start)
-  hosts: "{{ cluster_env }}"
-  gather_facts: false
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: install CA into system trust
-      become: true
-      ansible.builtin.copy:
-        src: "{{ spinifex_ca_src }}"
-        dest: "{{ spinifex_ca_trust_path }}"
-        mode: "0644"
-        remote_src: true
-
-    - name: rebuild CA bundle
-      become: true
-      ansible.builtin.command: update-ca-certificates
-      changed_when: true
-
-    # spinifex-ui reverse-proxies localhost:9999 (awsgw) and localhost:8443
-    # (predastore) — hardcoded in spinifex/services/spinifexui/spinifexui.go.
-    # spx admin init binds those services to BindIP only, so the loopback
-    # connection fails (BadGateway in UI). Patch to wildcard bind; cert
-    # already includes 127.0.0.1 in IP SANs.
-    - name: patch awsgw bind to 0.0.0.0 (UI proxies via localhost)
-      become: true
-      ansible.builtin.replace:
-        path: "{{ spinifex_config_file }}"
-        regexp: '^(\s*host\s*=\s*")[^"]+(:9999")'
-        replace: '\g<1>0.0.0.0\g<2>'
-
-    - name: patch predastore bind to 0.0.0.0 (UI proxies via localhost)
-      become: true
-      ansible.builtin.replace:
-        path: "{{ spinifex_config_file }}"
-        regexp: '^(\s*host\s*=\s*")[^"]+(:8443")'
-        replace: '\g<1>0.0.0.0\g<2>'
-
-- name: Cluster bootstrap — start services (init first, then joins)
-  hosts: init
-  gather_facts: false
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: start spinifex.target on init
-      become: true
-      ansible.builtin.systemd:
-        name: spinifex.target
-        state: started
-
-- name: Cluster bootstrap — start services on join hosts
-  hosts: join
-  gather_facts: false
-  serial: 100%
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: start spinifex.target on joiners
-      become: true
-      ansible.builtin.systemd:
-        name: spinifex.target
-        state: started
-
-- name: Cluster bootstrap — readiness check (all nodes)
-  hosts: "{{ cluster_env }}"
-  gather_facts: false
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: wait for awsgw responsiveness
-      ansible.builtin.uri:
-        url: "https://{{ spinifex_wan_ip }}:{{ cluster_formation_port }}/health"
-        validate_certs: false
-        return_content: true
-      register: cluster_health
-      until:
-        - cluster_health.status == 200
-        - cluster_health.json.service_health.awsgw == "ok"
-      retries: 30
-      delay: 5
-
-- name: Cluster bootstrap — post-bootstrap fixtures (init host only)
-  hosts: init
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-  roles:
-    - cluster-post

scripts/ansible/playbooks/dev-deploy.yml

@@ -1,14 +0,0 @@
----
-# In-place artifact swap: rebuilds + replaces binaries/microvm artifacts on
-# an already-installed node, then restarts spinifex.target. Use when iterating
-# on code without altering systemd units, users, dirs, or running the full
-# smoketest. For first-time or unit/script changes use dev-install / dev-reset.
-- name: Dev deploy — build + swap binaries + restart (no setup.sh, no smoketest)
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-
-  roles:
-    - build
-    - deploy

scripts/ansible/playbooks/dev-install.yml

@@ -1,67 +0,0 @@
----
-- name: Dev install — build, install, init, smoketest (idempotent)
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-
-  pre_tasks:
-    - name: preflight
-      ansible.builtin.include_role:
-        name: preflight
-
-    - name: health — config file present?
-      ansible.builtin.stat:
-        path: "{{ spinifex_config_file }}"
-      register: health_config
-
-    - name: health — spinifex.target active?
-      become: true
-      ansible.builtin.command: systemctl is-active spinifex.target
-      register: health_target
-      changed_when: false
-      failed_when: false
-
-    - name: health — awsgw responds?
-      become: true
-      become_user: "{{ spinifex_invoking_user }}"
-      environment:
-        AWS_PROFILE: spinifex
-      ansible.builtin.command: aws ec2 describe-regions
-      register: health_gw
-      changed_when: false
-      failed_when: false
-
-    - name: health — spx binary installed?
-      ansible.builtin.stat:
-        path: /usr/local/bin/spx
-      register: health_spx
-
-    - name: health — set already_installed fact
-      ansible.builtin.set_fact:
-        spinifex_already_installed: >-
-          {{ health_config.stat.exists
-             and health_spx.stat.exists
-             and (health_target.stdout | trim) == 'active'
-             and health_gw.rc == 0 }}
-
-    - name: health — summary
-      ansible.builtin.debug:
-        msg: |
-          Health:
-            spx binary: {{ 'present' if health_spx.stat.exists else 'missing' }}
-            config:     {{ 'present' if health_config.stat.exists else 'missing' }}
-            target:     {{ health_target.stdout | trim }}
-            awsgw:      {{ 'ok' if health_gw.rc == 0 else 'not responding' }}
-          Already installed: {{ spinifex_already_installed }}
-          {% if spinifex_already_installed %}
-          → Skipping build/install/init. Smoketest only.
-          {% else %}
-          → Running full build → install → init → smoketest.
-          {% endif %}
-
-  roles:
-    - { role: build, when: not (spinifex_already_installed | default(false)) }
-    - { role: install, when: not (spinifex_already_installed | default(false)) }
-    - { role: init, when: not (spinifex_already_installed | default(false)) }
-    - smoketest

scripts/ansible/playbooks/dev-logs.yml

@@ -1,17 +0,0 @@
----
-# Capture spinifex + ovn/ovs + qemu/nbdkit logs from journald and
-# /var/log/spinifex/ into a single tarball for bug reports.
-# Tail mode is intentionally not provided — use `journalctl -u 'spinifex-*' -f`
-# directly; ansible is the wrong tool for follow.
-- name: Dev logs — dump journald + varlog + status into a bundle
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-  vars:
-    logs_since: "1h"
-    logs_output_dir: "/tmp"
-    logs_include_dmesg: true
-
-  roles:
-    - logs

scripts/ansible/playbooks/dev-preflight.yml

@@ -1,8 +0,0 @@
----
-- name: Dev preflight — validate host dependencies
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-  roles:
-    - preflight

scripts/ansible/playbooks/dev-reset.yml

@@ -1,72 +0,0 @@
----
-- name: Dev reset — capture settings, wipe, rebuild, reinstall, re-init, smoketest
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-
-  pre_tasks:
-    - name: capture — check existing spinifex.toml
-      ansible.builtin.stat:
-        path: "{{ spinifex_config_file }}"
-      register: reset_config_stat
-
-    - name: capture — slurp existing config
-      ansible.builtin.slurp:
-        src: "{{ spinifex_config_file }}"
-      register: reset_config_raw
-      when: reset_config_stat.stat.exists
-
-    - name: capture — decode config
-      ansible.builtin.set_fact:
-        reset_config_txt: "{{ reset_config_raw.content | b64decode }}"
-      when: reset_config_stat.stat.exists
-
-    # List-concat-and-take-first pattern: missing regex match falls back to current default.
-    # `gateway\s*=` deliberately does not match `gateway_ip` / `gateway_lrp_*`.
-    - name: capture — extract scalars
-      ansible.builtin.set_fact:
-        spinifex_region: "{{ (reset_config_txt | regex_findall('region\\s*=\\s*\"([^\"]+)\"') + [spinifex_region])[0] }}"
-        spinifex_az: "{{ (reset_config_txt | regex_findall('\\baz\\s*=\\s*\"([^\"]+)\"') + [spinifex_az])[0] }}"
-        spinifex_external_mode: "{{ (reset_config_txt | regex_findall('external_mode\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_mode])[0] }}"
-        spinifex_external_pool_start: "{{ (reset_config_txt | regex_findall('range_start\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_pool_start])[0] }}"
-        spinifex_external_pool_end: "{{ (reset_config_txt | regex_findall('range_end\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_pool_end])[0] }}"
-        spinifex_external_gateway: "{{ (reset_config_txt | regex_findall('\\bgateway\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_gateway])[0] }}"
-        spinifex_external_prefix_len: "{{ (reset_config_txt | regex_findall('prefix_len\\s*=\\s*([0-9]+)') + [spinifex_external_prefix_len | string])[0] }}"
-        spinifex_external_gateway_ip: "{{ (reset_config_txt | regex_findall('gateway_ip\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_gateway_ip])[0] }}"
-      when: reset_config_stat.stat.exists
-
-    # Legacy: external_mode = "nat" was removed from spx. Coerce to "pool"
-    # so old configs still reset cleanly; warn so the operator notices.
-    - name: capture — coerce legacy external_mode=nat to pool
-      ansible.builtin.set_fact:
-        spinifex_external_mode: "pool"
-      when: spinifex_external_mode == 'nat'
-
-    - name: capture — warn on legacy nat coercion
-      ansible.builtin.debug:
-        msg: "⚠️  Captured external_mode=nat from legacy config; coerced to 'pool' (NAT mode removed from spx admin init)."
-      when:
-        - reset_config_stat.stat.exists
-        - (reset_config_txt | regex_findall('external_mode\\s*=\\s*\"nat\"')) | length > 0
-
-    - name: capture — summary
-      ansible.builtin.debug:
-        msg: |
-          {% if reset_config_stat.stat.exists %}
-          Captured settings from {{ spinifex_config_file }}:
-            region={{ spinifex_region }}, az={{ spinifex_az }}
-            external_mode={{ spinifex_external_mode }}
-            pool={{ spinifex_external_pool_start }}-{{ spinifex_external_pool_end }}
-            gateway={{ spinifex_external_gateway }}, prefix={{ spinifex_external_prefix_len }}
-            gateway_ip={{ spinifex_external_gateway_ip }}
-          {% else %}
-          No existing config; using defaults from vars/defaults.yml (or -e overrides).
-          {% endif %}
-
-  roles:
-    - teardown
-    - build
-    - install
-    - init
-    - smoketest

scripts/ansible/playbooks/dev-restore.yml

@@ -1,14 +0,0 @@
----
-# Restore a dev-snapshot bundle: stop services, wipe the snapshot path
-# set, untar the bundle, refresh CA trust, restart.
-#
-# Required: -e snapshot_name=<name>
-# Optional: -e snapshot_dir=/some/path
-- name: Dev restore — wipe + untar snapshot + restart
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-
-  roles:
-    - restore

scripts/ansible/playbooks/dev-snapshot.yml

@@ -1,17 +0,0 @@
----
-# Quiesce spinifex + ovn + ovs, tar /etc/spinifex + /var/lib/spinifex +
-# /var/lib/ovn + /etc/openvswitch/conf.db into a named bundle, restart.
-# Use to checkpoint a working state before a risky operation (branch
-# switch, viperblock surgery, ovn-controller restart cycle). Roll back
-# via dev-restore without a full dev-reset.
-#
-# Required: -e snapshot_name=<name>
-# Optional: -e snapshot_dir=/some/path
-- name: Dev snapshot — quiesce + tar state + restart
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-
-  roles:
-    - snapshot

scripts/ansible/playbooks/dev-status.yml

@@ -1,13 +0,0 @@
----
-# Read-only health snapshot for a Spinifex dev node. Never mutates state.
-# Run any time the operator wants to know "is my box healthy?" — pairs with
-# dev-preflight (run before install) as the post-install / running-system
-# diagnostic.
-- name: Dev status — health snapshot (read-only)
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-
-  roles:
-    - status

scripts/ansible/playbooks/dev-teardown.yml

@@ -1,8 +0,0 @@
----
-- name: Dev teardown — wipe Spinifex state (no reinstall)
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-  roles:
-    - teardown

scripts/ansible/playbooks/dev-version.yml

@@ -1,11 +0,0 @@
----
-# Detailed build-vs-install drift report. Faster than dev-status (no AWS,
-# OVN, or port probing). Answers "do I need to ansible-dev-deploy?".
-- name: Dev version — build vs install drift (read-only)
-  hosts: localhost
-  gather_facts: false
-  vars_files:
-    - ../vars/defaults.yml
-
-  roles:
-    - version

scripts/ansible/playbooks/dev-vpc.yml

@@ -1,14 +0,0 @@
----
-# Idempotent dev-mode VPC fixture: spinifex-dev-vpc + public subnet +
-# IGW + default route. Hooks into smoketest's launch path (which finds
-# the public subnet automatically). Re-runs are no-ops once converged.
-#
-# Default state: present. Tear down with -e dev_vpc_state=absent.
-- name: Dev VPC — fixture VPC/subnet/IGW/route (idempotent)
-  hosts: localhost
-  gather_facts: false
-  vars_files:
-    - ../vars/defaults.yml
-
-  roles:
-    - vpc

scripts/ansible/requirements.yml

@@ -1,6 +0,0 @@
----
-collections:
-  - name: community.general
-    version: ">=8.0.0"
-  - name: ansible.posix
-    version: ">=1.5.0"

scripts/ansible/roles/build/meta/main.yml

@@ -1,7 +0,0 @@
----
-galaxy_info:
-  role_name: build
-  description: Build spinifex + viperblock-nbdkit, stage tarball for setup.sh consumption.
-  license: BSD-3-Clause
-  min_ansible_version: "2.14"
-dependencies: []

scripts/ansible/roles/build/tasks/main.yml

@@ -1,66 +0,0 @@
----
-- name: build — spinifex (make build)
-  ansible.builtin.command:
-    cmd: make build
-    chdir: "{{ spinifex_root }}"
-  changed_when: true
-
-- name: build — viperblock nbdkit plugin (make go_build_nbd)
-  ansible.builtin.command:
-    cmd: make go_build_nbd
-    chdir: "{{ mulga_root }}/viperblock"
-  changed_when: true
-
-- name: build — detect microVM build script (present only on direct-boot branches)
-  ansible.builtin.stat:
-    path: "{{ spinifex_root }}/scripts/build-microvm-image.sh"
-  register: build_microvm_script
-
-- name: build — microVM kernel + initramfs (direct-boot LBs)
-  ansible.builtin.command:
-    cmd: make build-microvm-image
-    chdir: "{{ spinifex_root }}"
-  changed_when: true
-  when: build_microvm_script.stat.exists
-
-- name: build — verify build artifacts present
-  ansible.builtin.stat:
-    path: "{{ item }}"
-  register: build_artifacts
-  loop:
-    - "{{ spinifex_root }}/bin/spx"
-    - "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
-
-- name: build — assert artifacts exist
-  ansible.builtin.assert:
-    that: item.stat.exists
-    fail_msg: "Missing build artifact: {{ item.item }}"
-  loop: "{{ build_artifacts.results }}"
-  loop_control:
-    label: "{{ item.item }}"
-
-- name: build — stage tarball contents (mirrors dev-install.sh)
-  ansible.builtin.shell: |
-    set -e
-    STAGING=$(mktemp -d)
-    trap 'rm -rf "$STAGING"' EXIT
-    cp "{{ spinifex_root }}/bin/spx" "$STAGING/spx"
-    cp "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so" "$STAGING/"
-    cp "{{ spinifex_root }}/scripts/setup-ovn.sh" "$STAGING/"
-    mkdir -p "$STAGING/systemd"
-    cp "{{ spinifex_root }}/build/systemd/"* "$STAGING/systemd/"
-    mkdir -p "$STAGING/scripts"
-    cp "{{ spinifex_root }}/build/scripts/"* "$STAGING/scripts/"
-    cp "{{ spinifex_root }}/build/logrotate/spinifex" "$STAGING/logrotate-spinifex"
-    # microVM artifacts ship only when the branch built them.
-    if [ -f "{{ spinifex_root }}/build/microvm/vmlinuz" ] && [ -f "{{ spinifex_root }}/build/microvm/initramfs.cpio.gz" ]; then
-        mkdir -p "$STAGING/microvm"
-        cp "{{ spinifex_root }}/build/microvm/vmlinuz" "$STAGING/microvm/"
-        cp "{{ spinifex_root }}/build/microvm/initramfs.cpio.gz" "$STAGING/microvm/"
-    fi
-    tar czf "{{ build_tarball_path }}" -C "$STAGING" .
-  changed_when: true
-
-- name: build — summary
-  ansible.builtin.debug:
-    msg: "Built spx + nbdkit plugin{{ ' + microvm artifacts' if build_microvm_script.stat.exists else '' }}, tarball at {{ build_tarball_path }}"

scripts/ansible/roles/cluster-init/meta/main.yml

@@ -1,9 +0,0 @@
----
-galaxy_info:
-  role_name: cluster-init
-  description: |
-    Run `spx admin init` on the cluster's init host, backgrounded so the
-    formation server accepts joins from the cluster-join role. Installs
-    cluster CA into the system trust store BEFORE starting spinifex.target
-    so predastore's strict TLS verify is satisfied at first start.
-dependencies: []

scripts/ansible/roles/cluster-init/tasks/main.yml

@@ -1,77 +0,0 @@
----
-- name: cluster-init — assert this is the init host
-  ansible.builtin.assert:
-    that: spinifex_node_role == "init"
-    fail_msg: "cluster-init role must only run against hosts in the [init] group"
-
-# vars/defaults.yml sets spinifex_node_{name,count} for single-node dev use
-# (precedence 11). Inventory hostvars (precedence 4) lose that fight, so
-# we promote the inventory values via set_fact (precedence 15) before any
-# task templates `{{ spinifex_node_count }}`.
-- name: cluster-init — promote inventory node identity over defaults.yml
-  ansible.builtin.set_fact:
-    spinifex_node_name: "{{ hostvars[inventory_hostname]['spinifex_node_name'] }}"
-    spinifex_node_count: "{{ groups[cluster_env] | length }}"
-
-- name: cluster-init — assemble external network flags
-  ansible.builtin.set_fact:
-    cluster_external_flags: >-
-      {% if cluster_external_pool | length > 0 %}--external-mode=pool --external-pool={{ cluster_external_pool }}{% endif %}
-
-- name: cluster-init — run spx admin init (backgrounded, formation server waits for joins)
-  become: true
-  ansible.builtin.shell: |
-    set -o pipefail
-    spx admin init --force \
-      --node {{ spinifex_node_name }} \
-      --nodes {{ spinifex_node_count }} \
-      --bind {{ spinifex_wan_ip }} \
-      --cluster-bind {{ spinifex_wan_ip }} \
-      --port {{ cluster_formation_port }} \
-      --region {{ spinifex_region }} \
-      --az {{ spinifex_az }} \
-      {{ cluster_external_flags }} \
-      > /tmp/spx-admin-init.log 2>&1
-  async: 600
-  poll: 0
-  register: cluster_init_async
-
-- name: cluster-init — brief pause so init can fail fast if misconfigured
-  ansible.builtin.pause:
-    seconds: 5
-
-- name: cluster-init — check if init exited prematurely (sanity)
-  ansible.builtin.shell: |
-    if pgrep -f 'spx admin init' >/dev/null; then
-      echo "running"
-    else
-      tail -50 /tmp/spx-admin-init.log 2>/dev/null
-      exit 1
-    fi
-  register: cluster_init_alive
-  changed_when: false
-  failed_when: cluster_init_alive.rc != 0
-
-- name: cluster-init — wait for formation server health
-  ansible.builtin.uri:
-    url: "https://{{ spinifex_wan_ip }}:{{ cluster_formation_port }}/formation/health"
-    validate_certs: false
-    status_code: 200
-  register: cluster_init_health
-  until: cluster_init_health.status == 200
-  retries: "{{ cluster_formation_timeout // 2 }}"
-  delay: 2
-
-- name: cluster-init — fetch join token (delegated read for joiners)
-  become: true
-  ansible.builtin.slurp:
-    src: "{{ cluster_join_token_path }}"
-  register: cluster_join_token_raw
-
-- name: cluster-init — register join token as host fact
-  ansible.builtin.set_fact:
-    cluster_join_token: "{{ cluster_join_token_raw.content | b64decode | trim }}"
-
-- name: cluster-init — join token captured; waiting for joins handled by later play
-  ansible.builtin.debug:
-    msg: "Formation server up on {{ spinifex_wan_ip }}:{{ cluster_formation_port }}, {{ spinifex_node_count }} nodes expected"

scripts/ansible/roles/cluster-join/meta/main.yml

@@ -1,8 +0,0 @@
----
-galaxy_info:
-  role_name: cluster-join
-  description: |
-    Run `spx admin join` against the cluster init host's formation server.
-    Installs cluster CA + starts spinifex.target after join completes
-    (CA-before-services ordering avoids predastore TLS-verify crashes).
-dependencies: []

scripts/ansible/roles/cluster-join/tasks/main.yml

@@ -1,39 +0,0 @@
----
-- name: cluster-join — assert this is a join host
-  ansible.builtin.assert:
-    that: spinifex_node_role == "join"
-    fail_msg: "cluster-join role must only run against hosts in the [join] group"
-
-# Promote inventory hostvars over vars/defaults.yml (see cluster-init).
-- name: cluster-join — promote inventory node identity over defaults.yml
-  ansible.builtin.set_fact:
-    spinifex_node_name: "{{ hostvars[inventory_hostname]['spinifex_node_name'] }}"
-
-- name: cluster-join — pull join token from init host
-  ansible.builtin.set_fact:
-    cluster_join_token: "{{ hostvars[cluster_init_host]['cluster_join_token'] }}"
-
-- name: cluster-join — assert token present
-  ansible.builtin.assert:
-    that: cluster_join_token | length > 0
-    fail_msg: "join token missing — init host did not export cluster_join_token"
-
-- name: cluster-join — run spx admin join (retry up to 5 times)
-  become: true
-  ansible.builtin.command:
-    cmd: >-
-      spx admin join
-      --node {{ spinifex_node_name }}
-      --bind {{ spinifex_wan_ip }}
-      --cluster-bind {{ spinifex_wan_ip }}
-      --host {{ cluster_init_host }}:{{ cluster_formation_port }}
-      --token {{ cluster_join_token }}
-      --region {{ spinifex_region }}
-      --az {{ spinifex_az }}
-  register: cluster_join_result
-  retries: 5
-  delay: 3
-  until: cluster_join_result.rc == 0
-
-# CA install + service start handled by cluster-bootstrap.yml plays that
-# run after all joins complete and init has written multi-node configs.

scripts/ansible/roles/cluster-post/meta/main.yml

@@ -1,8 +0,0 @@
----
-galaxy_info:
-  role_name: cluster-post
-  description: |
-    Post-bootstrap idempotent fixtures on the init host: import SSH key,
-    import Ubuntu AMI (dual-version detect 24.04/26.04). Skip when
-    artifacts already exist.
-dependencies: []

scripts/ansible/roles/cluster-post/tasks/main.yml

@@ -1,205 +0,0 @@
----
-- name: cluster-post — locate ssh public key for import
-  ansible.builtin.find:
-    paths: "{{ ansible_env.HOME }}/.ssh"
-    patterns:
-      - spinifex-key.pub
-      - tf-user-*.pub
-    file_type: file
-  register: cluster_post_ssh_pubkey
-
-- name: cluster-post — fail when no public key found
-  ansible.builtin.assert:
-    that: cluster_post_ssh_pubkey.files | length > 0
-    fail_msg: "no public key in ~/.ssh — needed for spinifex-key import"
-
-- name: cluster-post — set pubkey path fact
-  ansible.builtin.set_fact:
-    cluster_post_pubkey_path: "{{ cluster_post_ssh_pubkey.files[0].path }}"
-
-- name: cluster-post — check whether spinifex-key already imported
-  ansible.builtin.command:
-    cmd: "{{ spinifex_aws_cli }} ec2 describe-key-pairs --key-names spinifex-key"
-  register: cluster_post_key_check
-  changed_when: false
-  failed_when: false
-
-- name: cluster-post — import spinifex-key (if absent)
-  ansible.builtin.command:
-    cmd: >-
-      {{ spinifex_aws_cli }} ec2 import-key-pair
-      --key-name spinifex-key
-      --public-key-material fileb://{{ cluster_post_pubkey_path }}
-  register: cluster_post_key_import
-  retries: 10
-  delay: 6
-  until: cluster_post_key_import.rc == 0
-  when: cluster_post_key_check.rc != 0
-
-- name: cluster-post — locate ubuntu cloud image (26.04 preferred, 24.04 fallback)
-  ansible.builtin.stat:
-    path: "{{ item }}"
-  register: cluster_post_ubuntu_stat
-  loop:
-    - "{{ ansible_env.HOME }}/images/ubuntu-26.04.img"
-    - "{{ ansible_env.HOME }}/images/ubuntu-24.04.img"
-
-- name: cluster-post — pick first available image
-  ansible.builtin.set_fact:
-    cluster_post_ubuntu_image: "{{ (cluster_post_ubuntu_stat.results | selectattr('stat.exists') | map(attribute='item') | list | first) | default('') }}"
-
-- name: cluster-post — fail when no ubuntu image present
-  ansible.builtin.assert:
-    that: cluster_post_ubuntu_image | length > 0
-    fail_msg: "no ubuntu-{24,26}.04.img in ~/images/ — gold image must stage it"
-
-- name: cluster-post — derive ubuntu version from image filename
-  ansible.builtin.set_fact:
-    cluster_post_ubuntu_version: "{{ cluster_post_ubuntu_image | regex_search('ubuntu-([0-9]+\\.[0-9]+)\\.img', '\\1') | first }}"
-
-- name: cluster-post — check whether ubuntu AMI already imported
-  ansible.builtin.command:
-    cmd: >-
-      {{ spinifex_aws_cli }} ec2 describe-images
-      --owners self
-      --filters Name=name,Values=ubuntu-{{ cluster_post_ubuntu_version }}
-  register: cluster_post_ami_check
-  changed_when: false
-  failed_when: false
-
-- name: cluster-post — import ubuntu AMI (if absent)
-  ansible.builtin.command:
-    cmd: >-
-      {{ spinifex_spx_cli }} admin images import
-      --file {{ cluster_post_ubuntu_image }}
-      --arch x86_64
-      --distro ubuntu
-      --version {{ cluster_post_ubuntu_version }}
-      --boot-mode uefi
-  when:
-    - cluster_post_ami_check.rc != 0
-       or "ubuntu-" + cluster_post_ubuntu_version not in cluster_post_ami_check.stdout
-
-- name: cluster-post — slurp final admin creds from ~/.aws/credentials
-  # Source of truth: SetupAWSCredentials writes the FINAL admin keys here.
-  # Init log shows obsolete keys (multi-node path silently regenerates them
-  # after the initial print — see cmd/spinifex/cmd/admin.go writeBootstrapFiles
-  # called twice in the multi-node flow).
-  ansible.builtin.slurp:
-    src: "{{ ansible_env.HOME }}/.aws/credentials"
-  register: cluster_post_aws_creds_raw
-
-- name: cluster-post — extract admin access/secret keys from spinifex profile
-  ansible.builtin.set_fact:
-    cluster_post_admin_access_key: >-
-      {{ (cluster_post_aws_creds_raw.content | b64decode
-          | regex_search('\[spinifex\][^\[]*aws_access_key_id\s*=\s*(\S+)', '\1')
-          | first) | default('') }}
-    cluster_post_admin_secret_key: >-
-      {{ (cluster_post_aws_creds_raw.content | b64decode
-          | regex_search('\[spinifex\][^\[]*aws_secret_access_key\s*=\s*(\S+)', '\1')
-          | first) | default('') }}
-
-- name: cluster-post — assert creds parsed
-  ansible.builtin.assert:
-    that:
-      - cluster_post_admin_access_key | length > 0
-      - cluster_post_admin_secret_key | length > 0
-    fail_msg: "could not parse admin keys from {{ ansible_env.HOME }}/.aws/credentials [spinifex] profile"
-
-- name: cluster-post — ensure local .creds dir exists
-  delegate_to: localhost
-  become: false
-  ansible.builtin.file:
-    path: "{{ playbook_dir }}/../.creds"
-    state: directory
-    mode: "0700"
-
-- name: cluster-post — write creds to operator workstation
-  delegate_to: localhost
-  become: false
-  ansible.builtin.copy:
-    dest: "{{ playbook_dir }}/../.creds/{{ cluster_env }}.txt"
-    mode: "0600"
-    content: |
-      # Spinifex cluster admin credentials — {{ cluster_env }}
-      # Generated: {{ ansible_facts['date_time']['iso8601'] }}
-      # Init host: {{ cluster_init_host }}
-      #
-      # UI:        https://{{ cluster_init_host }}:3000
-      # AWS API:   https://{{ cluster_init_host }}:4432
-      # Region:    {{ spinifex_region }}
-      #
-      # Login (admin user / AWS profile spinifex):
-      AWS_ACCESS_KEY_ID={{ cluster_post_admin_access_key }}
-      AWS_SECRET_ACCESS_KEY={{ cluster_post_admin_secret_key }}
-
-# Per-env AWS profile in operator's ~/.aws/credentials so tofu / aws CLI can
-# target this cluster via AWS_PROFILE=spinifex-{{ cluster_env }}. blockinfile
-# marker keeps re-runs idempotent — replaces the block in place, never
-# duplicates. Does not touch the user's existing [spinifex] profile (single-
-# node dev). Tofu side: provider "aws" { profile = "spinifex-env1" ... }.
-- name: cluster-post — ensure local ~/.aws dir exists
-  delegate_to: localhost
-  become: false
-  ansible.builtin.file:
-    path: "{{ lookup('env', 'HOME') }}/.aws"
-    state: directory
-    mode: "0700"
-
-- name: cluster-post — ensure local ~/.aws/credentials exists
-  delegate_to: localhost
-  become: false
-  ansible.builtin.copy:
-    dest: "{{ lookup('env', 'HOME') }}/.aws/credentials"
-    content: ""
-    force: false
-    mode: "0600"
-
-- name: cluster-post — upsert spinifex-{{ cluster_env }} profile in local ~/.aws/credentials
-  delegate_to: localhost
-  become: false
-  ansible.builtin.blockinfile:
-    path: "{{ lookup('env', 'HOME') }}/.aws/credentials"
-    marker: "# {mark} SPINIFEX-{{ cluster_env | upper }} MANAGED BLOCK"
-    block: |
-      [spinifex-{{ cluster_env }}]
-      aws_access_key_id = {{ cluster_post_admin_access_key }}
-      aws_secret_access_key = {{ cluster_post_admin_secret_key }}
-      region = {{ spinifex_region }}
-      ca_bundle = {{ playbook_dir }}/../.creds/{{ cluster_env }}-ca.pem
-      endpoint_url = https://{{ cluster_init_host }}:4432
-
-- name: cluster-post — slurp cluster CA from init host
-  ansible.builtin.slurp:
-    src: "{{ spinifex_ca_src }}"
-  register: cluster_post_ca_raw
-  become: true
-
-- name: cluster-post — write CA pem to operator workstation
-  delegate_to: localhost
-  become: false
-  ansible.builtin.copy:
-    dest: "{{ playbook_dir }}/../.creds/{{ cluster_env }}-ca.pem"
-    mode: "0644"
-    content: "{{ cluster_post_ca_raw.content | b64decode }}"
-
-- name: cluster-post — resolve absolute paths for summary
-  delegate_to: localhost
-  become: false
-  ansible.builtin.command: "readlink -f {{ playbook_dir }}/../.creds"
-  register: cluster_post_creds_abs
-  changed_when: false
-
-- name: cluster-post — summary
-  ansible.builtin.debug:
-    msg: |
-      Post-bootstrap done:
-        spinifex-key:    {{ 'imported now' if (cluster_post_key_import.changed | default(false)) else 'already present' }}
-        ubuntu AMI:      ubuntu-{{ cluster_post_ubuntu_version }}
-        admin creds:     {{ cluster_post_creds_abs.stdout }}/{{ cluster_env }}.txt
-        CA cert:         {{ cluster_post_creds_abs.stdout }}/{{ cluster_env }}-ca.pem
-        AWS profile:     spinifex-{{ cluster_env }} (in ~/.aws/credentials)
-        UI URL:          https://{{ cluster_init_host }}:3000
-        Access Key:      {{ cluster_post_admin_access_key }}
-        Tofu usage:      AWS_PROFILE=spinifex-{{ cluster_env }} tofu plan

scripts/ansible/roles/deploy/meta/main.yml

@@ -1,7 +0,0 @@
----
-galaxy_info:
-  role_name: deploy
-  description: In-place binary + microvm artifact swap for a running Spinifex node (parity with `make deploy`).
-  license: BSD-3-Clause
-  min_ansible_version: "2.14"
-dependencies: []

scripts/ansible/roles/deploy/tasks/main.yml

@@ -1,144 +0,0 @@
----
-# In-place artifact swap for an already-installed spinifex node. Mirrors
-# `make deploy`: replace binaries + microvm artifacts, restart services.
-# Does NOT run setup.sh (no user/dir/systemd-unit changes) or smoketest.
-
-- name: deploy — fail fast if spinifex isn't already installed
-  ansible.builtin.stat:
-    path: "{{ spinifex_config_file }}"
-  register: deploy_installed_stat
-  failed_when: not deploy_installed_stat.stat.exists
-
-- name: deploy — assert config present
-  ansible.builtin.assert:
-    that: deploy_installed_stat.stat.exists
-    fail_msg: |
-      {{ spinifex_config_file }} missing — this playbook only updates an
-      existing install. Run ansible-dev-install or ansible-dev-reset first.
-
-- name: deploy — verify build artifacts present
-  ansible.builtin.stat:
-    path: "{{ item }}"
-  register: deploy_artifacts
-  loop:
-    - "{{ spinifex_root }}/bin/spx"
-    - "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
-
-- name: deploy — assert artifacts exist after build role
-  ansible.builtin.assert:
-    that: item.stat.exists
-    fail_msg: "Build artifact missing: {{ item.item }} — build role should have produced it"
-  loop: "{{ deploy_artifacts.results }}"
-  loop_control:
-    label: "{{ item.item }}"
-
-- name: deploy — detect microvm artifacts (direct-boot branches only)
-  ansible.builtin.stat:
-    path: "{{ spinifex_root }}/build/microvm/{{ item }}"
-  register: deploy_microvm_stat
-  loop:
-    - vmlinuz
-    - initramfs.cpio.gz
-
-- name: deploy — set microvm presence fact
-  ansible.builtin.set_fact:
-    deploy_microvm_present: "{{ deploy_microvm_stat.results | map(attribute='stat.exists') | min }}"
-
-- name: deploy — detect nbdkit plugin directory
-  ansible.builtin.shell: |
-    set -e
-    nbdkit --dump-config | awk -F= '/^plugindir=/{print $2; exit}'
-  register: deploy_plugindir_cmd
-  changed_when: false
-
-- name: deploy — set plugin path fact
-  ansible.builtin.set_fact:
-    deploy_plugindir: "{{ deploy_plugindir_cmd.stdout | trim }}"
-
-- name: deploy — assert plugindir resolved
-  ansible.builtin.assert:
-    that: deploy_plugindir | length > 0
-    fail_msg: "Could not detect nbdkit plugin directory via `nbdkit --dump-config`"
-
-- name: deploy — stop spinifex.target before binary swap
-  become: true
-  ansible.builtin.systemd:
-    name: spinifex.target
-    state: stopped
-  failed_when: false
-
-- name: deploy — install spx binary
-  become: true
-  ansible.builtin.copy:
-    src: "{{ spinifex_root }}/bin/spx"
-    dest: /usr/local/bin/spx
-    mode: "0755"
-    owner: root
-    group: root
-
-- name: deploy — install viperblock nbdkit plugin
-  become: true
-  ansible.builtin.copy:
-    src: "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
-    dest: "{{ deploy_plugindir }}/nbdkit-viperblock-plugin.so"
-    mode: "0755"
-    owner: root
-    group: root
-
-- name: deploy — ensure microvm dir
-  become: true
-  ansible.builtin.file:
-    path: /usr/share/spinifex/microvm
-    state: directory
-    mode: "0755"
-    owner: root
-    group: root
-  when: deploy_microvm_present
-
-- name: deploy — install microvm kernel
-  become: true
-  ansible.builtin.copy:
-    src: "{{ spinifex_root }}/build/microvm/vmlinuz"
-    dest: /usr/share/spinifex/microvm/vmlinuz
-    mode: "0644"
-    owner: root
-    group: root
-  when: deploy_microvm_present
-
-- name: deploy — install microvm initramfs
-  become: true
-  ansible.builtin.copy:
-    src: "{{ spinifex_root }}/build/microvm/initramfs.cpio.gz"
-    dest: /usr/share/spinifex/microvm/initramfs.cpio.gz
-    mode: "0644"
-    owner: root
-    group: root
-  when: deploy_microvm_present
-
-- name: deploy — daemon-reload (covers any unit changes shipped via setup.sh later)
-  become: true
-  ansible.builtin.systemd:
-    daemon_reload: true
-
-- name: deploy — start spinifex.target
-  become: true
-  ansible.builtin.systemd:
-    name: spinifex.target
-    state: started
-
-- name: deploy — wait for awsgw to respond
-  ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-regions"
-  register: deploy_awsgw
-  retries: 15
-  delay: 2
-  until: deploy_awsgw.rc == 0
-  changed_when: false
-
-- name: deploy — summary
-  ansible.builtin.debug:
-    msg: |
-      Deployed:
-        spx           → /usr/local/bin/spx
-        nbdkit plugin → {{ deploy_plugindir }}/nbdkit-viperblock-plugin.so
-        microvm       → {{ '/usr/share/spinifex/microvm/{vmlinuz,initramfs.cpio.gz}' if deploy_microvm_present else 'skipped (not built on this branch)' }}
-      spinifex.target restarted; awsgw responding.

scripts/ansible/roles/init/meta/main.yml

@@ -1,7 +0,0 @@
----
-galaxy_info:
-  role_name: init
-  description: Run spx admin init, install CA trust, start services, import LB image.
-  license: BSD-3-Clause
-  min_ansible_version: "2.14"
-dependencies: []

scripts/ansible/roles/init/tasks/main.yml

@@ -1,90 +0,0 @@
----
-- name: init — base admin-init flags
-  ansible.builtin.set_fact:
-    init_base_flags: >-
-      --force
-      --region {{ spinifex_region }}
-      --az {{ spinifex_az }}
-      --node {{ spinifex_node_name }}
-      --nodes {{ spinifex_node_count }}
-
-- name: init — no-external opt-out
-  ansible.builtin.set_fact:
-    init_external_flags: "--no-external"
-  when: spinifex_external_no_external | bool
-
-- name: init — assert pool range supplied when mode=pool
-  ansible.builtin.assert:
-    that:
-      - spinifex_external_pool_start | length > 0
-      - spinifex_external_pool_end | length > 0
-    fail_msg: |
-      spx admin init requires --external-pool when --external-mode=pool.
-      Set spinifex_external_pool_start / spinifex_external_pool_end in
-      scripts/ansible/vars/defaults.yml, or pass:
-        -e spinifex_external_pool_start=192.168.1.150
-        -e spinifex_external_pool_end=192.168.1.250
-      Pass -e spinifex_external_no_external=true to skip external networking.
-  when:
-    - not (spinifex_external_no_external | bool)
-    - spinifex_external_mode == 'pool'
-
-- name: init — pool-mode external flags
-  ansible.builtin.set_fact:
-    init_external_flags: >-
-      --external-mode=pool
-      --external-pool={{ spinifex_external_pool_start }}-{{ spinifex_external_pool_end }}
-      {% if spinifex_external_gateway | length > 0 %}--external-gateway={{ spinifex_external_gateway }}{% endif %}
-      {% if spinifex_external_prefix_len | string | length > 0 %}--external-prefix-len={{ spinifex_external_prefix_len }}{% endif %}
-      {% if spinifex_external_gateway_ip | length > 0 %}--gateway-ip={{ spinifex_external_gateway_ip }}{% endif %}
-  when:
-    - not (spinifex_external_no_external | bool)
-    - spinifex_external_mode == 'pool'
-
-- name: init — assemble full flag string
-  ansible.builtin.set_fact:
-    init_all_flags: "{{ init_base_flags }} {{ init_external_flags | default('') }}"
-
-- name: init — run spx admin init
-  become: true
-  ansible.builtin.command: "spx admin init {{ init_all_flags }}"
-  changed_when: true
-
-- name: init — copy CA to system trust
-  become: true
-  ansible.builtin.copy:
-    src: "{{ spinifex_ca_src }}"
-    dest: "{{ spinifex_ca_trust_path }}"
-    mode: "0644"
-    remote_src: true
-
-- name: init — rebuild CA bundle (not --fresh — preserve other CAs)
-  become: true
-  ansible.builtin.command: update-ca-certificates
-  changed_when: true
-
-- name: init — start spinifex.target
-  become: true
-  ansible.builtin.systemd:
-    name: spinifex.target
-    state: started
-
-# systemctl start returns as soon as units activate; awsgw still wires up
-# to NATS + daemons after that. Bash start-dev.sh has internal wait loops;
-# systemd path does not. Poll a benign call until gateway is fully ready.
-- name: init — wait for AWS gateway readiness
-  ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-regions"
-  register: init_gw_ready
-  until: init_gw_ready.rc == 0
-  retries: 30
-  delay: 2
-  changed_when: false
-
-- name: init — summary
-  ansible.builtin.debug:
-    msg: |
-      Init complete:
-        region={{ spinifex_region }}, az={{ spinifex_az }}, node={{ spinifex_node_name }}
-        external_mode={{ 'disabled' if (spinifex_external_no_external | bool) else spinifex_external_mode }}
-        spinifex.target: started
-        CA: {{ spinifex_ca_trust_path }}

scripts/ansible/roles/install/meta/main.yml

@@ -1,7 +0,0 @@
----
-galaxy_info:
-  role_name: install
-  description: Invoke setup.sh and setup-ovn.sh to install Spinifex into production paths.
-  license: BSD-3-Clause
-  min_ansible_version: "2.14"
-dependencies: []

scripts/ansible/roles/install/tasks/main.yml

@@ -1,79 +0,0 @@
----
-- name: install — assert tarball exists
-  ansible.builtin.stat:
-    path: "{{ build_tarball_path }}"
-  register: install_tarball
-
-- name: install — fail if tarball missing
-  ansible.builtin.assert:
-    that: install_tarball.stat.exists
-    fail_msg: "Tarball {{ build_tarball_path }} missing. Run the build role first."
-
-- name: install — stop spinifex.target before reinstall
-  become: true
-  ansible.builtin.systemd:
-    name: spinifex.target
-    state: stopped
-  failed_when: false
-
-- name: install — reset failed state (spinifex-* scope)
-  become: true
-  ansible.builtin.command: systemctl reset-failed 'spinifex-*'
-  changed_when: false
-  failed_when: false
-
-- name: install — scrub stale pid/log files from prior dev-mode installs
-  become: true
-  ansible.builtin.shell: |
-    for dir in {{ spinifex_data_dir }} {{ spinifex_log_dir }} {{ spinifex_etc_dir }}; do
-      if [ -d "$dir" ]; then
-        find "$dir" -name '*.pid' -delete 2>/dev/null || true
-        find "$dir" -name '*.log' -delete 2>/dev/null || true
-        find "$dir" -name '*.log.*' -delete 2>/dev/null || true
-      fi
-    done
-  changed_when: false
-
-- name: install — remove legacy /var/lib/spinifex/config
-  become: true
-  ansible.builtin.file:
-    path: "{{ spinifex_data_dir }}/config"
-    state: absent
-
-- name: install — purge stale cluster state (configs + raft + NATS KV) on bootstrap
-  become: true
-  ansible.builtin.file:
-    path: "{{ item }}"
-    state: absent
-  loop:
-    - "{{ spinifex_etc_dir }}/predastore/predastore.toml"
-    - "{{ spinifex_etc_dir }}/spinifex.toml"
-    - "{{ spinifex_data_dir }}/predastore"
-    - "{{ spinifex_data_dir }}/nats"
-    - "{{ spinifex_data_dir }}/awsgw"
-  when: cluster_bootstrap_purge | default(false) | bool
-
-- name: install — run setup.sh (creates users, dirs, systemd units)
-  become: true
-  ansible.builtin.command:
-    cmd: "bash {{ spinifex_root }}/scripts/setup.sh"
-  environment:
-    INSTALL_SPINIFEX_TARBALL: "{{ build_tarball_path }}"
-    INSTALL_SPINIFEX_SKIP_NEWGRP: "1"
-  changed_when: true
-
-- name: install — cleanup tarball
-  ansible.builtin.file:
-    path: "{{ build_tarball_path }}"
-    state: absent
-
-- name: install — run setup-ovn.sh --management
-  become: true
-  ansible.builtin.command:
-    cmd: "/usr/local/share/spinifex/setup-ovn.sh --management --node-name={{ spinifex_node_name }}"
-  changed_when: true
-  when: install_run_ovn | default(true) | bool
-
-- name: install — summary
-  ansible.builtin.debug:
-    msg: "Install complete — setup.sh + setup-ovn.sh ran. Services NOT yet started (init role handles that)."

scripts/ansible/roles/logs/meta/main.yml

@@ -1,7 +0,0 @@
----
-galaxy_info:
-  role_name: logs
-  description: Dump spinifex + ovn + system logs into a tarball for bug reports.
-  license: BSD-3-Clause
-  min_ansible_version: "2.14"
-dependencies: []

scripts/ansible/roles/logs/tasks/main.yml

@@ -1,153 +0,0 @@
----
-# Collect a self-contained log bundle for bug reports. NEVER mutates running
-# state — only reads journalctl + /var/log/spinifex/* + status output, then
-# tars into logs_output_dir. Errors on missing inputs are non-fatal so a
-# half-broken node still produces a bundle.
-
-- name: logs — resolve staging directory
-  ansible.builtin.set_fact:
-    logs_stage_dir: "/tmp/spinifex-logs-{{ ansible_facts['hostname'] }}-{{ ansible_facts['date_time']['iso8601_basic_short'] }}"
-    logs_bundle_path: "{{ logs_output_dir }}/spinifex-logs-{{ ansible_facts['hostname'] }}-{{ ansible_facts['date_time']['iso8601_basic_short'] }}.tar.gz"
-
-- name: logs — create staging tree
-  ansible.builtin.file:
-    path: "{{ logs_stage_dir }}/{{ item }}"
-    state: directory
-    mode: "0755"
-  loop:
-    - journal
-    - varlog
-    - status
-    - kernel
-
-- name: logs — capture journalctl for spinifex-* units
-  become: true
-  ansible.builtin.shell: |
-    journalctl --no-pager --output=short-iso \
-      -u 'spinifex-*' --since {{ logs_since | quote }} \
-      > {{ logs_stage_dir }}/journal/spinifex.log 2>&1 || true
-  changed_when: true
-
-- name: logs — capture journalctl for ovn-* + openvswitch-switch
-  become: true
-  ansible.builtin.shell: |
-    journalctl --no-pager --output=short-iso \
-      -u 'ovn-*' -u openvswitch-switch \
-      --since {{ logs_since | quote }} \
-      > {{ logs_stage_dir }}/journal/ovn-ovs.log 2>&1 || true
-  changed_when: true
-
-- name: logs — capture journalctl for nbdkit + qemu (best-effort match)
-  become: true
-  ansible.builtin.shell: |
-    journalctl --no-pager --output=short-iso \
-      --since {{ logs_since | quote }} \
-      _COMM=qemu-system-x86_64 + _COMM=qemu-system-aarch64 + _COMM=nbdkit \
-      > {{ logs_stage_dir }}/journal/qemu-nbdkit.log 2>&1 || true
-  changed_when: true
-
-- name: logs — copy /var/log/spinifex/*
-  become: true
-  ansible.builtin.shell: |
-    if [ -d {{ spinifex_log_dir | quote }} ]; then
-      cp -a {{ spinifex_log_dir | quote }}/. {{ logs_stage_dir | quote }}/varlog/ 2>/dev/null || true
-    fi
-  changed_when: true
-
-- name: logs — systemctl status snapshot (spinifex-*)
-  become: true
-  ansible.builtin.shell: |
-    systemctl status 'spinifex-*' --no-pager --all \
-      > {{ logs_stage_dir }}/status/spinifex-units.txt 2>&1 || true
-    systemctl status ovn-controller ovn-central openvswitch-switch \
-      --no-pager > {{ logs_stage_dir }}/status/ovn-ovs-units.txt 2>&1 || true
-    systemctl list-units 'spinifex-*' --all --no-legend --plain \
-      > {{ logs_stage_dir }}/status/spinifex-list.txt 2>&1 || true
-  changed_when: true
-
-- name: logs — networking snapshot (ip, ss, ovs)
-  become: true
-  ansible.builtin.shell: |
-    ip -d -s link show > {{ logs_stage_dir }}/status/ip-link.txt 2>&1 || true
-    ip addr show       > {{ logs_stage_dir }}/status/ip-addr.txt 2>&1 || true
-    ip route show table all > {{ logs_stage_dir }}/status/ip-route.txt 2>&1 || true
-    ss -tlnp           > {{ logs_stage_dir }}/status/ss-tlnp.txt 2>&1 || true
-    if command -v ovs-vsctl >/dev/null 2>&1; then
-      ovs-vsctl show     > {{ logs_stage_dir }}/status/ovs-show.txt 2>&1 || true
-      ovs-vsctl list-br  > {{ logs_stage_dir }}/status/ovs-br.txt 2>&1 || true
-    fi
-    if command -v ovn-nbctl >/dev/null 2>&1; then
-      ovn-nbctl show   > {{ logs_stage_dir }}/status/ovn-nb.txt 2>&1 || true
-      ovn-sbctl show   > {{ logs_stage_dir }}/status/ovn-sb.txt 2>&1 || true
-    fi
-  changed_when: true
-
-- name: logs — kernel ring buffer
-  become: true
-  ansible.builtin.shell: |
-    dmesg --time-format=iso > {{ logs_stage_dir }}/kernel/dmesg.txt 2>&1 || true
-  changed_when: true
-  when: logs_include_dmesg | bool
-
-- name: logs — capture spx + plugin versions
-  become: true
-  ansible.builtin.shell: |
-    {
-      echo '## spx version'
-      /usr/local/bin/spx version 2>&1 | grep -vE '^[0-9]{4}/' || true
-      echo
-      echo '## nbdkit plugin sha256'
-      pdir=$(nbdkit --dump-config 2>/dev/null | awk -F= '/^plugindir=/{print $2; exit}')
-      [ -n "$pdir" ] && sha256sum "$pdir/nbdkit-viperblock-plugin.so" 2>/dev/null || true
-      echo
-      echo '## microvm artifacts'
-      sha256sum /usr/share/spinifex/microvm/* 2>/dev/null || true
-      echo
-      echo '## kernel + os'
-      uname -a
-      cat /etc/os-release 2>/dev/null || true
-    } > {{ logs_stage_dir }}/status/versions.txt 2>&1 || true
-  changed_when: true
-
-- name: logs — create bundle tarball
-  become: true
-  ansible.builtin.shell: |
-    tar -C {{ logs_stage_dir | dirname | quote }} \
-      -czf {{ logs_bundle_path | quote }} \
-      {{ logs_stage_dir | basename | quote }}
-  changed_when: true
-
-- name: logs — make bundle owned by invoking user
-  become: true
-  ansible.builtin.file:
-    path: "{{ logs_bundle_path }}"
-    owner: "{{ spinifex_invoking_user }}"
-    group: "{{ spinifex_invoking_user }}"
-    mode: "0640"
-
-- name: logs — remove staging tree
-  become: true
-  ansible.builtin.file:
-    path: "{{ logs_stage_dir }}"
-    state: absent
-
-- name: logs — bundle size
-  ansible.builtin.stat:
-    path: "{{ logs_bundle_path }}"
-  register: logs_bundle_stat
-
-- name: logs — summary
-  ansible.builtin.debug:
-    msg: |
-      Log bundle written:
-        path  : {{ logs_bundle_path }}
-        size  : {{ (logs_bundle_stat.stat.size | int / 1024) | round(1) }} KiB
-        since : {{ logs_since }}
-      Contents:
-        journal/    spinifex-*, ovn-*+openvswitch-switch, qemu/nbdkit
-        varlog/     copy of {{ spinifex_log_dir }}
-        status/     systemctl status, ip/ss/ovs/ovn show, versions
-        kernel/     dmesg{{ '' if logs_include_dmesg else ' (skipped via -e logs_include_dmesg=false)' }}
-
-      Attach to bug report. For live tail use:
-        journalctl -u 'spinifex-*' -f

scripts/ansible/roles/preflight/meta/main.yml

@@ -1,7 +0,0 @@
----
-galaxy_info:
-  role_name: preflight
-  description: Validate dev-host dependencies for Spinifex (parity with dev-setup.sh).
-  license: BSD-3-Clause
-  min_ansible_version: "2.14"
-dependencies: []

scripts/ansible/roles/preflight/tasks/main.yml

@@ -1,98 +0,0 @@
----
-- name: Check required commands
-  ansible.builtin.command: "which {{ item }}"
-  register: preflight_required
-  loop: "{{ spinifex_required_commands }}"
-  changed_when: false
-  failed_when: preflight_required.rc != 0
-  loop_control:
-    label: "{{ item }}"
-
-- name: Check optional commands
-  ansible.builtin.command: "which {{ item }}"
-  register: preflight_optional
-  loop: "{{ spinifex_optional_commands }}"
-  changed_when: false
-  failed_when: false
-  loop_control:
-    label: "{{ item }}"
-
-- name: Warn on missing optional commands
-  ansible.builtin.debug:
-    msg: "WARN: optional command '{{ item.item }}' not found"
-  loop: "{{ preflight_optional.results }}"
-  when: item.rc != 0
-  loop_control:
-    label: "{{ item.item }}"
-
-- name: Check OVN/OVS commands
-  ansible.builtin.command: "which {{ item }}"
-  register: preflight_ovn
-  loop: "{{ spinifex_ovn_commands }}"
-  changed_when: false
-  failed_when: false
-  loop_control:
-    label: "{{ item }}"
-
-- name: Warn on missing OVN/OVS tools
-  ansible.builtin.debug:
-    msg: "WARN: OVN/OVS tool '{{ item.item }}' not found (required for VPC networking). Install: sudo apt-get install ovn-central ovn-host openvswitch-switch"
-  loop: "{{ preflight_ovn.results }}"
-  when: item.rc != 0
-  loop_control:
-    label: "{{ item.item }}"
-
-- name: Record OVN tool availability
-  ansible.builtin.set_fact:
-    preflight_ovs_present: "{{ (preflight_ovn.results | selectattr('item','eq','ovs-vsctl') | selectattr('rc','eq',0) | list | length) > 0 }}"
-
-- name: Check br-int bridge
-  ansible.builtin.command: ovs-vsctl br-exists br-int
-  register: preflight_brint
-  changed_when: false
-  failed_when: false
-  when: preflight_ovs_present
-
-- name: Warn if br-int missing
-  ansible.builtin.debug:
-    msg: "WARN: br-int missing. Run scripts/setup-ovn.sh --management to initialise OVN networking."
-  when:
-    - preflight_ovs_present
-    - preflight_brint.rc | default(0) != 0
-
-- name: Check sibling component directories
-  ansible.builtin.stat:
-    path: "{{ playbook_dir }}/../../../../{{ item }}/go.mod"
-  register: preflight_components
-  loop: "{{ spinifex_components }}"
-  loop_control:
-    label: "{{ item }}"
-
-- name: Warn on missing component checkouts
-  ansible.builtin.debug:
-    msg: "WARN: component '{{ item.item }}' not found at expected sibling path (no go.mod). Run scripts/clone-deps.sh."
-  loop: "{{ preflight_components.results }}"
-  when: not item.stat.exists
-  loop_control:
-    label: "{{ item.item }}"
-
-- name: Check port availability
-  ansible.builtin.shell: "ss -tlnH 'sport = :{{ item.port }}' | head -1"
-  register: preflight_ports
-  loop: "{{ spinifex_ports }}"
-  changed_when: false
-  failed_when: false
-  loop_control:
-    label: "{{ item.service }} ({{ item.port }})"
-
-- name: Warn on occupied ports
-  ansible.builtin.debug:
-    msg: "WARN: port {{ item.item.port }} occupied (needed for {{ item.item.service }})"
-  loop: "{{ preflight_ports.results }}"
-  when: item.stdout | length > 0
-  loop_control:
-    label: "{{ item.item.service }}"
-
-- name: Preflight summary
-  ansible.builtin.debug:
-    msg: "Preflight passed. Required: {{ spinifex_required_commands | join(', ') }}. Warnings above (if any) are non-fatal."

scripts/ansible/roles/restore/meta/main.yml

@@ -1,7 +0,0 @@
----
-galaxy_info:
-  role_name: restore
-  description: Restore a spinifex+ovn+ovs snapshot bundle, refresh CA trust, restart.
-  license: BSD-3-Clause
-  min_ansible_version: "2.14"
-dependencies: []

scripts/ansible/roles/restore/tasks/main.yml

@@ -1,140 +0,0 @@
----
-# Inverse of the snapshot role: stop services, wipe the same path set,
-# untar the named bundle, refresh CA trust (because /etc/spinifex/ca.pem
-# came from the snapshot but the system trust store didn't), restart.
-
-- name: restore — require snapshot_name
-  ansible.builtin.assert:
-    that:
-      - snapshot_name is defined
-      - snapshot_name | length > 0
-      - snapshot_name is match('^[A-Za-z0-9._-]+$')
-    fail_msg: "snapshot_name required (alphanumeric/._-): -e snapshot_name=<name>"
-
-- name: restore — resolve bundle path
-  ansible.builtin.set_fact:
-    snapshot_bundle_path: "{{ snapshot_dir }}/{{ snapshot_name }}.tar.gz"
-
-- name: restore — assert bundle exists
-  become: true
-  ansible.builtin.stat:
-    path: "{{ snapshot_bundle_path }}"
-  register: restore_bundle_stat
-
-- name: restore — fail if missing
-  ansible.builtin.assert:
-    that: restore_bundle_stat.stat.exists
-    fail_msg: |
-      Snapshot bundle missing: {{ snapshot_bundle_path }}.
-      Available bundles:
-        ls -lh {{ snapshot_dir }}/*.tar.gz 2>/dev/null
-
-- name: restore — stop services in dependency order
-  become: true
-  ansible.builtin.systemd:
-    name: "{{ item }}"
-    state: stopped
-  loop: "{{ snapshot_services_stop_order }}"
-  failed_when: false
-
-- name: restore — kill lingering QEMU VMs
-  become: true
-  ansible.builtin.command: "pkill -x {{ item }}"
-  loop: "{{ spinifex_qemu_processes }}"
-  changed_when: false
-  failed_when: false
-
-- name: restore — wait for QEMU exit
-  become: true
-  ansible.builtin.shell: |
-    set -e
-    for i in $(seq 1 {{ teardown_process_wait_seconds }}); do
-      alive=0
-      for proc in {{ spinifex_qemu_processes | join(' ') }}; do
-        if pgrep -x "$proc" >/dev/null 2>&1; then alive=1; break; fi
-      done
-      if [ "$alive" = "0" ]; then exit 0; fi
-      sleep 1
-    done
-    exit 1
-  changed_when: false
-
-- name: restore — wipe target paths (only those the snapshot will replace)
-  become: true
-  ansible.builtin.file:
-    path: "{{ item }}"
-    state: absent
-  loop: "{{ snapshot_paths }}"
-
-- name: restore — recreate parent dirs that tar will refill
-  become: true
-  ansible.builtin.file:
-    path: "{{ item }}"
-    state: directory
-    owner: root
-    group: root
-    mode: "0755"
-  loop:
-    - "{{ ovn_db_dir }}"
-    - /etc/openvswitch
-
-- name: restore — extract bundle (preserve permissions/ACLs)
-  become: true
-  ansible.builtin.command:
-    cmd: >-
-      tar --acls --xattrs -xzpf {{ snapshot_bundle_path | quote }}
-      -C /
-  changed_when: true
-
-- name: restore — refresh spinifex CA in system trust
-  become: true
-  ansible.builtin.stat:
-    path: "{{ spinifex_ca_src }}"
-  register: restore_ca_stat
-
-- name: restore — copy CA back to trust path
-  become: true
-  ansible.builtin.copy:
-    src: "{{ spinifex_ca_src }}"
-    dest: "{{ spinifex_ca_trust_path }}"
-    mode: "0644"
-    remote_src: true
-  when: restore_ca_stat.stat.exists
-
-- name: restore — rebuild CA bundle
-  become: true
-  ansible.builtin.command: update-ca-certificates
-  changed_when: true
-  when: restore_ca_stat.stat.exists
-
-- name: restore — daemon-reload (covers any unit file delta in the snapshot)
-  become: true
-  ansible.builtin.systemd:
-    daemon_reload: true
-
-- name: restore — restart services in reverse order
-  become: true
-  ansible.builtin.systemd:
-    name: "{{ item }}"
-    state: started
-  loop: "{{ snapshot_services_stop_order | reverse | list }}"
-  failed_when: false
-
-- name: restore — wait for AWS gateway readiness
-  ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-regions"
-  register: restore_gw_ready
-  retries: 30
-  delay: 2
-  until: restore_gw_ready.rc == 0
-  changed_when: false
-  failed_when: false
-
-- name: restore — summary
-  ansible.builtin.debug:
-    msg: |
-      Restore complete:
-        snapshot : {{ snapshot_name }}
-        bundle   : {{ snapshot_bundle_path }}
-        paths    : {{ snapshot_paths | join(', ') }}
-        CA       : {{ 'refreshed' if restore_ca_stat.stat.exists else 'not in bundle — system trust unchanged' }}
-        gateway  : {{ 'ready' if restore_gw_ready.rc == 0 else 'NOT ready after restart — investigate' }}

scripts/ansible/roles/smoketest/meta/main.yml

@@ -1,7 +0,0 @@
----
-galaxy_info:
-  role_name: smoketest
-  description: Import SSH key, import Ubuntu AMI, launch one smoke-test instance.
-  license: BSD-3-Clause
-  min_ansible_version: "2.14"
-dependencies: []

scripts/ansible/roles/smoketest/tasks/main.yml

@@ -1,197 +0,0 @@
----
-- name: smoketest — resolve invoking user's home
-  ansible.builtin.getent:
-    database: passwd
-    key: "{{ spinifex_invoking_user }}"
-
-- name: smoketest — set user home fact
-  ansible.builtin.set_fact:
-    smoketest_user_home: "{{ ansible_facts['getent_passwd'][spinifex_invoking_user][4] }}"
-
-- name: smoketest — generate SSH key if absent
-  become: true
-  become_user: "{{ spinifex_invoking_user }}"
-  ansible.builtin.command:
-    cmd: ssh-keygen -t ed25519 -f {{ smoketest_user_home }}/.ssh/spinifex-key -N ""
-    creates: "{{ smoketest_user_home }}/.ssh/spinifex-key"
-
-- name: smoketest — import SSH key to spinifex
-  ansible.builtin.command: >-
-    {{ spinifex_aws_cli }} ec2 import-key-pair
-    --key-name spinifex-key
-    --public-key-material fileb://{{ smoketest_user_home }}/.ssh/spinifex-key.pub
-  register: smoketest_keypair
-  until: >-
-    smoketest_keypair.rc == 0 or
-    'InvalidKeyPair.Duplicate' in (smoketest_keypair.stderr | default(''))
-  retries: 10
-  delay: 3
-  changed_when: "'InvalidKeyPair.Duplicate' not in (smoketest_keypair.stderr | default(''))"
-  failed_when:
-    - smoketest_keypair.rc != 0
-    - "'InvalidKeyPair.Duplicate' not in (smoketest_keypair.stderr | default(''))"
-
-- name: smoketest — detect architecture
-  ansible.builtin.set_fact:
-    smoketest_arch: "{{ 'arm64' if ansible_facts['architecture'] in ['aarch64','arm64'] else 'x86_64' }}"
-
-- name: smoketest — resolve local image path
-  ansible.builtin.set_fact:
-    smoketest_local_image_path: "{{ smoketest_local_image | expanduser }}"
-  vars:
-    smoketest_local_image: "{{ spinifex_smoketest_local_image | replace('~', smoketest_user_home) }}"
-
-- name: smoketest — check for local image
-  ansible.builtin.stat:
-    path: "{{ smoketest_local_image_path }}"
-  register: smoketest_local_img
-
-- name: smoketest — import AMI from local file
-  ansible.builtin.command: >-
-    {{ spinifex_spx_cli }} admin images import
-    --file {{ smoketest_local_image_path }}
-    --distro {{ spinifex_smoketest_distro }}
-    --version {{ spinifex_smoketest_version }}
-    --arch {{ smoketest_arch }}
-    --boot-mode uefi
-  register: smoketest_import_local
-  until: >-
-    smoketest_import_local.rc == 0 or
-    'already exists' in (smoketest_import_local.stderr | default(''))
-  retries: 5
-  delay: 3
-  changed_when: smoketest_import_local.rc == 0
-  failed_when:
-    - smoketest_import_local.rc != 0
-    - "'already exists' not in (smoketest_import_local.stderr | default(''))"
-  when: smoketest_local_img.stat.exists
-
-- name: smoketest — import AMI by name (fallback)
-  ansible.builtin.command: >-
-    {{ spinifex_spx_cli }} admin images import
-    --name {{ spinifex_smoketest_distro }}-{{ spinifex_smoketest_version }}-{{ smoketest_arch }}
-  register: smoketest_import_remote
-  until: >-
-    smoketest_import_remote.rc == 0 or
-    'already exists' in (smoketest_import_remote.stderr | default(''))
-  retries: 5
-  delay: 3
-  changed_when: smoketest_import_remote.rc == 0
-  failed_when:
-    - smoketest_import_remote.rc != 0
-    - "'already exists' not in (smoketest_import_remote.stderr | default(''))"
-  when: not smoketest_local_img.stat.exists
-
-- name: smoketest — detect CPU vendor for instance type choice
-  ansible.builtin.command: grep -q AuthenticAMD /proc/cpuinfo
-  register: smoketest_amd
-  changed_when: false
-  failed_when: false
-
-- name: smoketest — pick instance type
-  ansible.builtin.set_fact:
-    smoketest_instance_type: "{{ 't3a.small' if smoketest_amd.rc == 0 else 't3.small' }}"
-
-- name: smoketest — find AMI ID
-  ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-images --query Images[0].ImageId --output text"
-  register: smoketest_ami
-  until: smoketest_ami.rc == 0
-  retries: 10
-  delay: 3
-  changed_when: false
-
-- name: smoketest — find public subnet (best-effort)
-  ansible.builtin.command: >-
-    {{ spinifex_aws_cli }} ec2 describe-subnets
-    --filters Name=map-public-ip-on-launch,Values=true
-    --query Subnets[0].SubnetId
-    --output text
-  register: smoketest_subnet_public
-  changed_when: false
-  failed_when: false
-
-- name: smoketest — fallback to first subnet (best-effort)
-  ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-subnets --query Subnets[0].SubnetId --output text"
-  register: smoketest_subnet_any
-  changed_when: false
-  failed_when: false
-  when:
-    - smoketest_subnet_public.rc != 0 or smoketest_subnet_public.stdout in ['', 'None']
-
-- name: smoketest — resolve subnet id (empty string if none)
-  ansible.builtin.set_fact:
-    smoketest_subnet_id: >-
-      {%- if smoketest_subnet_public.rc == 0 and smoketest_subnet_public.stdout not in ['', 'None'] -%}
-      {{ smoketest_subnet_public.stdout }}
-      {%- elif (smoketest_subnet_any.rc | default(1)) == 0 and (smoketest_subnet_any.stdout | default('')) not in ['', 'None'] -%}
-      {{ smoketest_subnet_any.stdout }}
-      {%- else -%}{%- endif -%}
-
-- name: smoketest — skip launch if no subnet (fresh install with no user VPC)
-  ansible.builtin.debug:
-    msg: |
-      No subnet found — skipping instance launch.
-      Services are up, CA is trusted, keypair imported, AMI imported.
-      Create a VPC+subnet via tofu examples (or `aws ec2 create-vpc` /
-      `create-subnet`) then re-run if you want a live instance.
-  when: smoketest_subnet_id | trim == ''
-
-- name: smoketest — launch instance
-  ansible.builtin.command: >-
-    {{ spinifex_aws_cli }} ec2 run-instances
-    --image-id {{ smoketest_ami.stdout }}
-    --instance-type {{ smoketest_instance_type }}
-    --key-name spinifex-key
-    --subnet-id {{ smoketest_subnet_id | trim }}
-    --count 1
-    --query Instances[0].InstanceId
-    --output text
-  register: smoketest_instance
-  until: smoketest_instance.rc == 0
-  retries: 5
-  delay: 3
-  changed_when: true
-  when: smoketest_subnet_id | trim != ''
-
-- name: smoketest — wait for instance running (proves launch path works)
-  ansible.builtin.command: >-
-    {{ spinifex_aws_cli }} ec2 describe-instances
-    --instance-ids {{ smoketest_instance.stdout }}
-    --query Reservations[0].Instances[0].State.Name
-    --output text
-  register: smoketest_instance_state
-  until: smoketest_instance_state.stdout | trim == 'running'
-  retries: 30
-  delay: 2
-  changed_when: false
-  failed_when: false
-  when:
-    - smoketest_subnet_id | trim != ''
-    - smoketest_instance.stdout is defined
-    - smoketest_instance.stdout | trim != ''
-
-- name: smoketest — terminate instance (cleanup)
-  ansible.builtin.command: >-
-    {{ spinifex_aws_cli }} ec2 terminate-instances
-    --instance-ids {{ smoketest_instance.stdout }}
-  register: smoketest_terminate
-  until: smoketest_terminate.rc == 0
-  retries: 5
-  delay: 3
-  changed_when: true
-  failed_when: false
-  when:
-    - smoketest_subnet_id | trim != ''
-    - smoketest_instance.stdout is defined
-    - smoketest_instance.stdout | trim != ''
-
-- name: smoketest — summary
-  ansible.builtin.debug:
-    msg: |
-      Smoketest complete:
-        AMI:           {{ smoketest_ami.stdout }}
-        Subnet:        {{ smoketest_subnet_id | trim | default('(none — launch skipped)', true) }}
-        Instance type: {{ smoketest_instance_type }}
-        Instance ID:   {{ smoketest_instance.stdout | default('(skipped)', true) }}
-        Final state:   {{ smoketest_instance_state.stdout | default('(skipped)', true) | trim }}
-        Terminated:    {{ 'yes' if smoketest_terminate.rc | default(1) == 0 else 'no (launch skipped or terminate failed)' }}