release: update go.mod for v1.9.0

feat: predastore compaction E2E (#407 )
* feat: add predastore compaction E2E Inject a short compaction interval into test-cluster predastore config via a new --predastore-compaction-interval flag on admin init/join, gated so production output stays byte-identical when unset. Add a harness helper that drives write-then-delete churn against predastore S3, brackets it with du-over-SSH usage, and asserts end-of-churn usage settles below 70% of the churn peak; wire it into the multinode suite as TestPredastoreCompaction. * fix: compaction E2E gate writes churn to its own bucket The reclaim gate signed PutObject against the config-defined system 'predastore' bucket, which belongs to account 000000000000 with only a static [[auth]] grant. The harness admin identity is a different IAM account, so predastore returned 403 AccessDenied cross-account. Create a per-run bucket the admin identity owns (AdministratorAccess) and churn into it. Objects still erasure-code into distributed/nodes/ node-* so the du-based reclaim measurement is unchanged. The gateway (:9999) does not proxy S3, so the client stays predastore-direct. * test: run predastore compaction gate before destructive node tests The compaction reclaim gate was declared last in the multinode suite, so it measured shard-store usage after NodeFailure/NodeRecovery had torn down and restarted node2. NodeRecovery only gates on NATS/gateway/EC2 readiness, not predastore, so node2's shard store was still settling and the du baseline came back empty. Move the test before the destructive node tests so it runs against a pristine, fully-converged cluster primed by the volume tests. * fix: race-free SSH output capture + empty-tolerant du probe The compaction reclaim gate crashed with ParseInt("") when node1's du probe returned empty combined output. Harden the probe and the SSH layer: - ssh.Run used one bytes.Buffer for both session.Stdout and session.Stderr, which x/crypto/ssh copies in two concurrent goroutines — a data race on a non-concurrent-safe type. Switch to session.CombinedOutput, which guards the combined stream with a mutex-wrapped singleWriter. - shardStoreUsageBytes now terminates the remote command with a sentinel echo. A missing sentinel is reported as output lost in SSH retrieval; a present sentinel with no number is treated as a 0 measurement (anomalous, logged) instead of crashing the gate. The sentinel lets the next run distinguish a transport-layer drop from a genuinely empty remote measurement. * test(e2e): replace predastore du-reclaim gate with object lifecycle contract The multinode compaction gate measured shard-store disk usage over SSH (du) and asserted reclaim after churn. It could never pass: segments only seal at the 4 GiB roll threshold, so test-scale churn produces no compactable segment, and the du probe read 0 bytes before and after. Reclaim correctness is already covered deterministically by predastore's store-level unit tests. Delete AssertPredastoreCompaction and its du/churn/poll/sentinel machinery, the harness unit tests that exercised them, and the multinode TestPredastore Compaction entry. Add a single-node TestPredastoreObjectLifecycle that checks the user-visible contract: an object is readable after PutObject, gone after DeleteObject (GET returns NoSuchKey and ListObjectsV2 is empty), and the store keeps serving fresh writes. newPredastoreS3 is retained, refactored to take a host so it works outside the multinode cluster type. * test(e2e): drop orphaned multinode TestPredastoreCompaction Complete the du-reclaim -> object-lifecycle refactor: remove the multinode TestPredastoreCompaction caller (and its now-unused context and harness imports) that still referenced the deleted harness.AssertPredastoreCompaction, which broke e2e-build. Wire the single-node TestPredastoreObjectLifecycle entry that replaces it.
2026-10-24 21:23:51 +00:00 · 2026-06-23 16:22:08 +10:00 · 2026-06-23 16:19:10 +10:00 · 2026-06-23 15:29:14 +10:00 · 2026-06-23 09:59:54 +10:00 · 2026-06-23 09:18:59 +10:00
1263 changed files with 133376 additions and 36047 deletions
@@ -6,16 +6,10 @@
 # Set the matching branch protection rule on `dev` and `main` to
 # "Require review from Code Owners" or this file is advisory only.

-# Fall-through: any change needs a team review.
-*                           @mulgadc/mulgadc-engineers
-
 # Network layer is governed by ADR-0006 invariant tests.
-# Changes here must respect those clauses — see
-# spinifex/network/invariants/ and
+# Changes here must respect those clauses
+# see spinifex/network/invariants/
 # docs/development/proposals/az-local-architecture/0006-spinifex-network-layer-contract.md
 /spinifex/network/          @joshsiv-mulga
 /spinifex/vpcd/             @joshsiv-mulga
-
-# Invariants suite is the executable ADR-0006 fence — any widening or
-# narrowing of the scan roots needs an explicit owner review.
 /spinifex/network/invariants/ @joshsiv-mulga
@@ -0,0 +1,7 @@
+## Summary
+
+<!-- What does this PR do, and why? -->
+
+## Changes
+
+<!-- The notable changes, as bullets. -->
@@ -2,6 +2,13 @@ name: E2E Bare-Metal

 on:
  workflow_dispatch:
+  workflow_call:
+    inputs:
+      ref:
+        description: "spinifex ref to checkout/test (default: triggering ref)"
+        required: false
+        type: string
+        default: ""

 permissions:
  contents: read
@@ -12,7 +19,7 @@ env:

 jobs:
  e2e_baremetal:
-    name: Bare-Metal Multi-Node E2E
+    name: Bare-Metal Single-Node E2E
    runs-on: [ self-hosted, ci-baremetal ]
    timeout-minutes: 120
    concurrency:
@@ -20,11 +27,17 @@ jobs:
      cancel-in-progress: false
    env:
      BM_ISO_CACHE: /var/cache/spinifex-iso-builder
+      ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-20-${{ github.run_id }}
    steps:
      - name: Clean Workspace
-        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          sudo umount -R -l "${BM_ISO_CACHE}/rootfs" 2>/dev/null || true
+          if [ ! -c /dev/null ]; then sudo rm -f /dev/null; sudo mknod -m 666 /dev/null c 1 3; fi
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*

-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
+        with:
+          ref: ${{ inputs.ref || github.ref }}

      - uses: actions/create-github-app-token@v3
        id: app-token
@@ -33,7 +46,7 @@ jobs:
          private-key: ${{ secrets.APP_PRIVATE_KEY }}
          repositories: mulga

-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
        with:
          repository: mulgadc/mulga
          token: ${{ steps.app-token.outputs.token }}
@@ -91,6 +104,8 @@ jobs:
            echo "Ubuntu 26.04 cloud image cached ($(du -h "$UBUNTU_IMG" | cut -f1))"
          fi

+      - name: Install go-junit-report on runner
+        run: go install github.com/jstemmer/go-junit-report/v2@latest

      - name: Run Bare-Metal E2E
        env:
@@ -100,8 +115,34 @@ jobs:
          BM_UBUNTU_IMAGE: ${{ env.BM_ISO_CACHE }}/ubuntu-26.04-cloudimg-amd64.img
          SPX_EXTERNAL_POOL: ${{ env.SPINIFEX_CI_POOL }}
        run: |
+          mkdir -p "${ARTIFACT_DIR}"
+          set -o pipefail
          sudo -E mulga/scripts/baremetal/run-baremetal-e2e.sh \
            --vmlinuz   "${BM_ISO_CACHE}/vmlinuz" \
            --initrd    "${BM_ISO_CACHE}/initrd.img" \
            --squashfs  "${BM_ISO_CACHE}/spinifex.squashfs" \
-            --tests-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+            --tests-tar "${RUNNER_TEMP}/spinifex-tests.tar" \
+            2>&1 | tee "${ARTIFACT_DIR}/test-baremetal.log"
+
+      - name: Generate JUnit report
+        if: always()
+        run: |
+          LOG="${ARTIFACT_DIR}/test-baremetal.log"
+          [ -f "$LOG" ] || exit 0
+          go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-baremetal.xml" || true
+
+      - name: Analyze e2e failures
+        if: always()
+        uses: ./.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "E2E failure analysis (cell 20)"
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-artifacts-cell-20-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
@@ -21,10 +21,6 @@ jobs:
  ddil:
    name: DDIL Real Multi-Node
    runs-on: [ self-hosted, ci-multi ]
-    # Budget: ~45m go test + tofu provision/bootstrap/teardown on ci-multi.
-    # In Phase 1 the scenarios all t.Skip, so the real-world runtime is
-    # dominated by provisioning; the 60m ceiling leaves headroom for
-    # Phase 2/3 when assertions come online.
    timeout-minutes: 60
    env:
      TOFU_LOG: /tmp/e2e-tofu-ddil-${{ github.run_id }}.log
@@ -39,16 +35,12 @@ jobs:
          private-key: ${{ secrets.APP_PRIVATE_KEY }}
          repositories: mulga

-      # Full spinifex checkout: the Go harness must run on the runner
-      # (outside the cluster) so its SSH path survives scenario partitions
-      # that drop peer-to-peer cluster traffic. Running inside the cluster
-      # would self-sever when Scenario C isolates a node.
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
        name: Checkout Spinifex
        with:
          path: spinifex

-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
        name: Checkout Mulga (tofu-cluster)
        with:
          repository: mulgadc/mulga
@@ -88,7 +80,6 @@ jobs:
        working-directory: mulga/scripts/tofu-cluster
        run: |
          set -euo pipefail
-          # shellcheck source=/dev/null
          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
          export TF_VAR_git_branch="${GIT_BRANCH}"
          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
@@ -140,10 +131,6 @@ jobs:
            exit 1
          fi

-          # Guest SSH key: bootstrap-install.sh imports spinifex-key into EC2;
-          # the matching private material was generated on the primary node at
-          # ~/.ssh/spinifex-key. Pull it to the runner so the harness can SSH
-          # the witness via its MapPublicIpOnLaunch-assigned EIP.
          GUEST_KEY="${RUNNER_TEMP}/spinifex-key"
          scp -i "${SSH_KEY}" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
            "${SSH_USER}@${PRIMARY_IP}:~/.ssh/spinifex-key" "${GUEST_KEY}"
@@ -151,11 +138,6 @@ jobs:

          mkdir -p "${RUNNER_TEMP}/ddil-results"
          cd "${GITHUB_WORKSPACE}/spinifex"
-          # gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout,
-          # plus machine-readable jsonfile + JUnit XML artifacts. The JUnit
-          # file feeds the junit-report composite action below so failures
-          # become summary table rows + workflow annotations.
-          # shellcheck source=/dev/null
          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
          banner "DDIL scenarios"
          DDIL_NODES="${NODE_IPS}" \
@@ -185,10 +167,6 @@ jobs:
        if: failure()
        working-directory: mulga/scripts/tofu-cluster
        run: |
-          # Pull spinifex-* journals + QEMU process listings from every
-          # node into the artifact bundle so failures can be triaged
-          # without re-provisioning the cluster. Best-effort — a missing
-          # node should not mask the original test failure.
          set +e
          INVENTORY=$(tofu output -json inventory)
          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
@@ -197,7 +175,6 @@ jobs:
          LOG_DIR="${RUNNER_TEMP}/ddil-results/cluster-logs"
          mkdir -p "${LOG_DIR}"

-          # shellcheck source=/dev/null
          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
          mapfile -t NODE_IPS < <(echo "$INVENTORY" | jq -r '.nodes[].wan_ip')
          for ip in "${NODE_IPS[@]}"; do
@@ -221,7 +198,6 @@ jobs:
        working-directory: mulga/scripts/tofu-cluster
        run: |
          set -euo pipefail
-          # shellcheck source=/dev/null
          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
          quiet_run "tofu destroy"          "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
@@ -40,6 +40,7 @@ jobs:
    timeout-minutes: 35
    env:
      TOFU_LOG: /tmp/e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}.log
+      ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}

    steps:
      # blank or scheduled run = full matrix.
@@ -60,7 +61,10 @@ jobs:

      - name: Clean Workspace
        if: steps.gate.outputs.selected == 'true'
-        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          sudo umount -R -l /var/cache/spinifex-iso-builder/rootfs 2>/dev/null || true
+          if [ ! -c /dev/null ]; then sudo rm -f /dev/null; sudo mknod -m 666 /dev/null c 1 3; fi
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*

      - uses: actions/create-github-app-token@v3
        if: steps.gate.outputs.selected == 'true'
@@ -70,7 +74,7 @@ jobs:
          private-key: ${{ secrets.APP_PRIVATE_KEY }}
          repositories: mulga

-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
        if: steps.gate.outputs.selected == 'true'
        with:
          repository: mulgadc/mulga
@@ -78,6 +82,11 @@ jobs:
          sparse-checkout: scripts/tofu-cluster
          path: mulga

+      - uses: actions/checkout@v7
+        if: steps.gate.outputs.selected == 'true'
+        with:
+          path: spinifex
+
      - name: Build Install Artifacts
        if: steps.gate.outputs.selected == 'true'
        working-directory: mulga/scripts/tofu-cluster
@@ -132,8 +141,18 @@ jobs:
          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
          SSH_KEY="${SSH_KEY/#\~/$HOME}"
-          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
-            "cd ~/e2e-tests && ./run-e2e.sh && ./run-cert-e2e.sh && ENDPOINT=https://${WAN_IP}:9999 ./run-lb-e2e.sh"
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
+             for SUITE in single cert lb; do \
+               echo \"::group::suite \$SUITE\"; \
+               date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
+               GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+                 ARTIFACT_DIR='${ARTIFACT_DIR}' \
+                 ./\$SUITE.test -test.v -test.timeout=30m \
+                 2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log'; \
+               rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
+               if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
+             done; exit \$FAIL"

      - name: Run Multi Node E2E
        if: steps.gate.outputs.selected == 'true' && matrix.topo == 'real-multi'
@@ -144,32 +163,57 @@ jobs:
          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
          SSH_KEY="${SSH_KEY/#\~/$HOME}"

-          NODE_IPS=""
+          NODES_CSV=""
          NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
          for i in $(seq 0 $((NODE_COUNT - 1))); do
-            NODE_IPS="${NODE_IPS} $(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")"
+            ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
+            NODES_CSV="${NODES_CSV:+$NODES_CSV,}$ip"
          done

-          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
-            "cd ~/e2e-tests && ./run-multinode-e2e.sh ${NODE_IPS} && ./run-cert-e2e.sh --multinode ${NODE_IPS} && ENDPOINT=https://${NODE1_IP}:9999 ./run-lb-e2e.sh --nodes ${NODE_IPS}"
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
+             for SUITE in multinode cert lb; do \
+               echo \"::group::suite \$SUITE\"; \
+               date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
+               GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
+                 SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
+                 SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
+                 ARTIFACT_DIR='${ARTIFACT_DIR}' \
+                 ./\$SUITE.test -test.v -test.timeout=30m \
+                 2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log'; \
+               rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
+               if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
+             done; exit \$FAIL"

      - name: Run Tofu Workbooks
        if: steps.gate.outputs.selected == 'true' && matrix.extra == 'tofu-examples'
        working-directory: mulga/scripts/tofu-cluster
+        # The driver emits go test -v markers per workbook; teeing to
+        # test-tofu.log feeds the same go-junit-report -> e2e-analyze -> upload
+        # chain the Go suites use, so cell 17 produces an RCA bundle too.
        run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
          INVENTORY=$(tofu output -json inventory)
          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          date -u +%Y-%m-%dT%H:%M:%SZ > "${ARTIFACT_DIR}/test-tofu.start"
          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
-            "cd ~/e2e-tests && ./run-tofu-examples-e2e.sh"
+            "cd ~/e2e-tests && ./run-tofu-examples-e2e.sh" \
+            2>&1 | tee "${ARTIFACT_DIR}/test-tofu.log"
+          rc=${PIPESTATUS[0]}
+          if [ $rc -ne 0 ]; then echo "tofu: FAILED" >> "${ARTIFACT_DIR}/suite-status.txt"; fi
+          exit $rc

      - name: Run Reboot Resilience E2E
        if: steps.gate.outputs.selected == 'true' && matrix.extra == 'reboot-single'
        working-directory: mulga/scripts/tofu-cluster
-        # Driver runs from the runner (NOT via ssh-and-run) so it survives the
+        # reboot.test runs from the runner (NOT via ssh-and-run) so it survives the
        # systemctl-reboot-induced ssh disconnect mid-run.
        run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
          INVENTORY=$(tofu output -json inventory)
          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
@@ -178,31 +222,85 @@ jobs:
          REBOOT_DIR="${RUNNER_TEMP}/reboot-e2e"
          mkdir -p "$REBOOT_DIR"
          tar xf "${RUNNER_TEMP}/spinifex-tests.tar" -C "$REBOOT_DIR"
-          chmod +x "$REBOOT_DIR/run-reboot-e2e.sh"
+          chmod +x "$REBOOT_DIR/_bin/reboot.test"

-          WAN_IP="$WAN_IP" SSH_KEY="$SSH_KEY" "$REBOOT_DIR/run-reboot-e2e.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i "$SSH_KEY")
+          CREDS=$(ssh "${SSH_OPTS[@]}" tf-user@${WAN_IP} 'cat ~/.aws/credentials')
+          AWS_AK=$(echo "$CREDS" | awk -F '= *' '/^aws_access_key_id/ {print $2; exit}')
+          AWS_SK=$(echo "$CREDS" | awk -F '= *' '/^aws_secret_access_key/ {print $2; exit}')
+          echo "::add-mask::${AWS_AK}"; echo "::add-mask::${AWS_SK}"

-      - name: Dump Cluster Journals on Failure
-        if: failure() && steps.gate.outputs.selected == 'true'
+          date -u +%Y-%m-%dT%H:%M:%SZ > "${ARTIFACT_DIR}/test-reboot.start"
+          GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+            SPINIFEX_WAN_IP="${WAN_IP}" SPINIFEX_SSH_KEY="${SSH_KEY}" \
+            SPINIFEX_AWS_ENDPOINT="https://${WAN_IP}:9999" \
+            SPINIFEX_AWS_ACCESS_KEY_ID="${AWS_AK}" SPINIFEX_AWS_SECRET_ACCESS_KEY="${AWS_SK}" \
+            ARTIFACT_DIR="${ARTIFACT_DIR}" \
+            "$REBOOT_DIR/_bin/reboot.test" -test.v -test.timeout=30m \
+            2>&1 | tee "${ARTIFACT_DIR}/test-reboot.log"
+          rc=${PIPESTATUS[0]}
+          if [ $rc -ne 0 ]; then echo "reboot: FAILED" >> "${ARTIFACT_DIR}/suite-status.txt"; fi
+          exit $rc
+
+      - name: Collect e2e artifacts
+        if: always() && steps.gate.outputs.selected == 'true'
        working-directory: mulga/scripts/tofu-cluster
        run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
          INVENTORY=$(tofu output -json inventory 2>/dev/null) || { echo "no inventory; skipping"; exit 0; }
-          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
-          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path'); SSH_KEY="${SSH_KEY/#\~/$HOME}"
          NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY")
+
+          ssh "${SSH_OPTS[@]}" tf-user@${NODE1_IP} "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+
          for i in $(seq 0 $((NODE_COUNT - 1))); do
-            IP=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
-            echo "===== node$((i+1)) ($IP) ====="
-            for svc in spinifex-nats spinifex-predastore spinifex-viperblock spinifex-daemon spinifex-awsgw spinifex-vpcd dhcpcd; do
-              echo "--- $svc ---"
-              ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" tf-user@${IP} \
-                "sudo journalctl -u $svc --no-pager -n 200 2>/dev/null || echo '(not found)'" 2>&1 || echo "(node unreachable)"
-            done
-            echo "--- DHCP client state ---"
-            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" tf-user@${IP} \
-              "systemctl is-enabled dhcpcd.service 2>/dev/null; systemctl is-active dhcpcd.service 2>/dev/null; sudo ss -lupn 'sport = 68' 2>/dev/null; pgrep -laf dhcpcd 2>/dev/null" 2>&1 || echo "(node unreachable)"
+            IP=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip"); idx=$((i + 1))
+            mkdir -p "${ARTIFACT_DIR}/vm-console-node${idx}"
+            ssh "${SSH_OPTS[@]}" tf-user@${IP} 'sudo sh -c "cd /run/spinifex 2>/dev/null && tar -cf - console-*.log 2>/dev/null"' \
+              | tar -C "${ARTIFACT_DIR}/vm-console-node${idx}" -xf - 2>/dev/null
+            ssh "${SSH_OPTS[@]}" tf-user@${IP} 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "60 min ago"' \
+              > "${ARTIFACT_DIR}/spinifex-journal-node${idx}.log" 2>/dev/null
          done

+          cat "${ARTIFACT_DIR}"/spinifex-journal-node*.log \
+            > "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || true
+          echo "collected: $(ls "${ARTIFACT_DIR}" 2>/dev/null | wc -l) entries"
+
+      - name: Install go-junit-report
+        if: always() && steps.gate.outputs.selected == 'true'
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Convert test logs to JUnit XML
+        if: always() && steps.gate.outputs.selected == 'true'
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log); SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Analyze e2e failures
+        if: always() && steps.gate.outputs.selected == 'true'
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "E2E failure analysis (cell ${{ matrix.cell }})"
+
+      - uses: actions/upload-artifact@v7
+        if: always() && steps.gate.outputs.selected == 'true'
+        with:
+          name: e2e-artifacts-cell-${{ matrix.cell }}-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
      - name: Teardown
        if: always()
        working-directory: mulga/scripts/tofu-cluster
@@ -216,3 +314,21 @@ jobs:
          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1
          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+
+  # Bare-metal single-node suite. Doesn't fit the tofu matrix shape, so it
+  # runs as a sibling job via the reusable e2e-baremetal workflow. Gated like a
+  # matrix cell ("cell 20"): every weeknight cron, or dispatch with 20 selected
+  # (blank cells = all). Tests `dev` on schedule to match GIT_BRANCH above.
+  baremetal:
+    name: cell-20 (baremetal/pxe/real-hw)
+    if: >-
+      github.event_name == 'schedule' ||
+      inputs.cells == '' ||
+      contains(format(',{0},', inputs.cells), ',20,')
+    permissions:
+      contents: read
+      statuses: write
+    uses: ./.github/workflows/e2e-baremetal.yml
+    with:
+      ref: ${{ github.event_name == 'schedule' && 'dev' || github.ref_name }}
+    secrets: inherit
@@ -0,0 +1,799 @@
+# E2E (per-suite blocks) — EXPERIMENTAL, workflow_dispatch only.
+#
+# Restructures the looped e2e.yml into env-grouped job blocks so each suite
+# surfaces as its own node in the Actions graph (mulga-siv-283):
+#
+#   single_provision ─┬─ single_suite (matrix: single, cert, lb, eks, iam)
+#                     └─ single_teardown (if: always)
+#   multi_provision  ─┬─ multi_suite  (matrix: multinode)
+#                     └─ multi_teardown (if: always)
+#
+# Constraint: tofu state is runner-filesystem-local, so *_provision and
+# *_teardown must execute on the same self-hosted runner (one runner per label
+# today). Teardown reuses the persisted mulga/ tofu-cluster workspace and does
+# NOT re-checkout mulga (that would wipe local state). Suite-matrix legs reach
+# the already-provisioned VM over its WAN IP using the key path the provision
+# job emits; they share the runner home dir, so the key file persists.
+#
+# This file does NOT post the gating `e2e` commit status — it runs alongside
+# e2e.yml for validation. Once green across a few manual runs, fold the
+# triggers from e2e.yml here and retire the loop.
+
+name: E2E (suite blocks)
+
+on:
+  workflow_dispatch:
+    inputs:
+      single_suites:
+        description: "Single-node suites to run as matrix legs"
+        required: false
+        default: "single cert lb eks iam"
+      run_multinode:
+        description: "Also run the multinode env block"
+        type: boolean
+        required: false
+        default: true
+      test_run:
+        description: "Optional -test.run regex (empty = all tests in suite)"
+        required: false
+        default: ""
+
+permissions:
+  contents: read
+
+env:
+  GIT_BRANCH: ${{ github.ref_name }}
+  TEST_RUN: ${{ inputs.test_run || '' }}
+
+jobs:
+  # ---------------------------------------------------------------------------
+  # Single-node env
+  # ---------------------------------------------------------------------------
+  single_provision:
+    name: Single env · provision
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 40
+    outputs:
+      wan_ip: ${{ steps.vm.outputs.wan_ip }}
+      ssh_key: ${{ steps.vm.outputs.ssh_key }}
+      suites: ${{ steps.suites.outputs.list }}
+      matrix: ${{ steps.suites.outputs.matrix }}
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
+    steps:
+      - name: Reap Orphan Procs
+        run: |
+          set +e
+          UID_SELF=$(id -u)
+          WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in
+                "$WS"|"$WS"/*) echo "reap $sig pid=$pid cwd=$cwd"; kill "-$sig" "$pid" 2>/dev/null ;;
+              esac
+            done
+          }
+          reap TERM; sleep 3; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          true
+
+      - name: Clean Workspace
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - uses: actions/checkout@v7
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/tofu-cluster
+          path: mulga
+
+      - uses: actions/checkout@v7
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Re-enable Go toolchain auto-download
+        run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
+
+      - name: Compute suite matrix
+        id: suites
+        run: |
+          LIST="${{ inputs.single_suites || 'single cert lb eks iam' }}"
+          echo "list=${LIST}" >> "$GITHUB_OUTPUT"
+          JSON=$(printf '%s\n' $LIST | jq -R . | jq -cs .)
+          echo "matrix=${JSON}" >> "$GITHUB_OUTPUT"
+
+      - name: Reclaim docker build cache
+        run: |
+          docker container prune -f || true
+          docker image prune -f || true
+          docker buildx prune -af --filter "until=24h" || true
+          docker system df || true
+
+      - name: Build Install Artifacts
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Provision VM
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          export TF_VAR_git_branch="${GIT_BRANCH}"
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "VM provisioned"
+
+      - name: Bootstrap via Binary Install
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Resolve VM SSH details
+        id: vm
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          echo "wan_ip=${WAN_IP}" >> "$GITHUB_OUTPUT"
+          echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
+
+      - name: Build eks-node AMI (libguestfs, runner)
+        if: contains(steps.suites.outputs.list, 'eks')
+        working-directory: spinifex
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "build eks-node AMI (libguestfs — no host devices)"
+          sudo apt-get update -qq
+          sudo apt-get install -y --no-install-recommends libguestfs-tools qemu-utils
+          export LIBGUESTFS_BACKEND=direct
+          sudo chmod 0644 /boot/vmlinuz-* || true
+          group_open "make build-eks-node-image"
+          make build-eks-node-image
+          group_close
+          ok "raw: $(du -h /tmp/eks-node-image-build/eks-node-alpine.raw | cut -f1)"
+
+      - name: Import eks-node AMI on VM
+        if: contains(steps.suites.outputs.list, 'eks')
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          eval "$(grep -E '^(ALPINE_VERSION|AMI_NAME|SYSTEM_TAG)=' \
+            spinifex/scripts/images/eks-node/manifest.conf)"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          banner "import eks-node AMI to VM image store"
+          scp "${SSH_OPTS[@]}" \
+            /tmp/eks-node-image-build/eks-node-alpine.raw \
+            "$VM":/tmp/eks-node-alpine.raw
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "spx admin images import --file /tmp/eks-node-alpine.raw \
+               --distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
+               --boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}" >/dev/null
+          ok "eks-node AMI imported"
+
+      - name: Install kubectl on VM
+        if: contains(steps.suites.outputs.list, 'eks')
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          banner "install kubectl on VM (eks data-plane subtest)"
+          ssh "${SSH_OPTS[@]}" "$VM" 'bash -s' <<'EOF'
+          set -euo pipefail
+          KVER=v1.32.5
+          case "$(uname -m)" in
+            x86_64)  ARCH=amd64 ;;
+            aarch64) ARCH=arm64 ;;
+            *) echo "unsupported arch $(uname -m)" >&2; exit 1 ;;
+          esac
+          cd /tmp
+          curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl"
+          curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl.sha256"
+          echo "$(cat kubectl.sha256)  kubectl" | sha256sum -c -
+          sudo install -m 0755 kubectl /usr/local/bin/kubectl
+          kubectl version --client
+          EOF
+          ok "kubectl installed on VM"
+
+      - name: Build E2E test binaries on runner
+        working-directory: spinifex/tests/e2e
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          # teardown always builds: the run-scoped sweep binary runs in the
+          # teardown job, it is never a selectable suite leg.
+          SUITES_BUILD="${{ steps.suites.outputs.list }} teardown"
+          banner "build ${SUITES_BUILD// /, }"
+          group_open "go test -c"
+          make e2e-build SUITES_BUILD="${SUITES_BUILD}"
+          group_close
+          for f in _bin/*.test; do
+            ok "$(basename "$f") $(stat -c '%s' "$f" | numfmt --to=iec)"
+          done
+
+      - name: Ship test binaries to VM
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          ssh "${SSH_OPTS[@]}" "$VM" "rm -rf \$HOME/spinifex-e2e-bin && mkdir -p \$HOME/spinifex-e2e-bin"
+          tar -C spinifex/tests/e2e/_bin -cf - . | ssh "${SSH_OPTS[@]}" "$VM" "tar -xf - -C \$HOME/spinifex-e2e-bin"
+          ok "shipped binaries to $VM"
+
+      - name: Start packet capture on VM br-wan
+        run: |
+          set +e
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          scp "${SSH_OPTS[@]}" \
+            "${GITHUB_WORKSPACE}/spinifex/.github/scripts/start-net-capture.sh" \
+            "$VM":/tmp/start-net-capture.sh
+          ssh "${SSH_OPTS[@]}" "$VM" "bash /tmp/start-net-capture.sh" || true
+
+  single_suite:
+    name: Single env · suite
+    needs: single_provision
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 45
+    strategy:
+      fail-fast: false
+      matrix:
+        suite: ${{ fromJSON(needs.single_provision.outputs.matrix) }}
+    env:
+      WAN_IP: ${{ needs.single_provision.outputs.wan_ip }}
+      SSH_KEY: ${{ needs.single_provision.outputs.ssh_key }}
+      ARTIFACT_DIR: /tmp/spinifex-e2e-single-${{ github.run_id }}-${{ matrix.suite }}
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Install go-junit-report
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Run ${{ matrix.suite }}
+        id: e2e
+        run: |
+          set +e
+          SUITE="${{ matrix.suite }}"
+          VM_HOST="tf-user@${WAN_IP}"
+          TEST_RUN_FLAG=""
+          [ -n "${TEST_RUN}" ] && TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+          cleanup_remote() {
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" "$VM_HOST" \
+              "pkill -u tf-user -TERM -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; true" \
+              2>/dev/null || true
+          }
+          trap cleanup_remote EXIT INT TERM
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" "$VM_HOST" \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/spinifex-e2e-bin; \
+             R=\$'\033[0m' B=\$'\033[1m' C=\$'\033[36m' G=\$'\033[32m' RED=\$'\033[31m'; \
+             printf '\n%s%s━━ suite %s ━━%s\n' \"\$B\" \"\$C\" \"${SUITE}\" \"\$R\"; \
+             date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-${SUITE}.start'; \
+             GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+               SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+               ARTIFACT_DIR='${ARTIFACT_DIR}' \
+               ./${SUITE}.test -test.v -test.timeout=30m ${TEST_RUN_FLAG} \
+               2>&1 | tee '${ARTIFACT_DIR}/test-${SUITE}.log' \
+                    | sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'; \
+             exit \${PIPESTATUS[0]}"
+          echo "rc=$?" >> "$GITHUB_OUTPUT"
+          exit 0
+
+      - name: Collect suite artifacts from VM
+        if: always()
+        run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${SSH_KEY}")
+          VM=tf-user@${WAN_IP}
+          ssh "${SSH_OPTS[@]}" "$VM" "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+          true
+
+      - name: Convert test log to JUnit XML
+        if: always()
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log); SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          title: "E2E ${{ matrix.suite }} results"
+
+      - name: Analyze e2e failures
+        if: always()
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "E2E ${{ matrix.suite }} failure analysis"
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-single-${{ matrix.suite }}-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
+      - name: Fail job if suite failed
+        if: always()
+        run: |
+          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
+            echo "::error::suite ${{ matrix.suite }} failed (rc=${{ steps.e2e.outputs.rc }})"
+            exit 1
+          fi
+
+  single_teardown:
+    name: Single env · teardown
+    needs: [ single_provision, single_suite ]
+    if: always()
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 20
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-single-td-${{ github.run_id }}.log
+      ARTIFACT_DIR: /tmp/spinifex-e2e-single-diag-${{ github.run_id }}
+      WAN_IP: ${{ needs.single_provision.outputs.wan_ip }}
+      SSH_KEY: ${{ needs.single_provision.outputs.ssh_key }}
+    steps:
+      # Reuse the persisted mulga/ tofu-cluster workspace from single_provision.
+      # Re-checkout only spinifex (for ci-fmt + diag scripts); checking out mulga
+      # again would wipe the local tofu state and orphan the VM.
+      - uses: actions/checkout@v7
+        with:
+          path: spinifex
+
+      - name: Capture network/IPsec diagnostics on VM
+        if: ${{ needs.single_provision.result == 'success' && env.WAN_IP != '' }}
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}")
+          VM=tf-user@${WAN_IP}
+          scp "${SSH_OPTS[@]}" \
+            "${GITHUB_WORKSPACE}/spinifex/.github/scripts/capture-net-diag.sh" \
+            "$VM":/tmp/capture-net-diag.sh 2>/dev/null
+          ssh "${SSH_OPTS[@]}" "$VM" "bash /tmp/capture-net-diag.sh '${ARTIFACT_DIR}/net-diag'" 2>/dev/null
+          ssh "${SSH_OPTS[@]}" "$VM" 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "90 min ago"' \
+            > "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null
+          mkdir -p "${ARTIFACT_DIR}/lb-console"
+          ssh "${SSH_OPTS[@]}" "$VM" 'sudo sh -c "cd /run/spinifex 2>/dev/null && tar -cf - console-*.log 2>/dev/null"' \
+            | tar -C "${ARTIFACT_DIR}/lb-console" -xf - 2>/dev/null
+          true
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-single-diag-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: ignore
+          retention-days: 14
+
+      # Reclaim any resource leaked by a crashed/interrupted suite before the VM
+      # is destroyed. Scoped to this run's tag, best-effort, never fails the job.
+      - name: Sweep leaked run resources on VM
+        if: ${{ needs.single_provision.result == 'success' && env.WAN_IP != '' }}
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}")
+          VM=tf-user@${WAN_IP}
+          banner "sweep run-tagged resources (run ${{ github.run_id }})"
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "cd \$HOME/spinifex-e2e-bin 2>/dev/null && \
+             SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+             SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+             ./teardown.test -test.v -test.run TestSweepRunResources -test.timeout=10m" \
+            2>&1 || true
+          true
+
+      - name: Teardown VM
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          if [ -n "${WAN_IP}" ] && [ -n "${SSH_KEY}" ]; then
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}" \
+              tf-user@${WAN_IP} \
+              'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
+              2>/dev/null || true
+          fi
+          UID_SELF=$(id -u); WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in "$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;; esac
+            done
+          }
+          reap TERM; sleep 2; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
+          ok "teardown complete"
+
+  # ---------------------------------------------------------------------------
+  # Multinode env
+  # ---------------------------------------------------------------------------
+  multi_provision:
+    name: Multi env · provision
+    if: ${{ inputs.run_multinode }}
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 45
+    outputs:
+      node1_ip: ${{ steps.cluster.outputs.node1_ip }}
+      ssh_key: ${{ steps.cluster.outputs.ssh_key }}
+      nodes_csv: ${{ steps.cluster.outputs.nodes_csv }}
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
+    steps:
+      - name: Reap Orphan Procs
+        run: |
+          set +e
+          UID_SELF=$(id -u); WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in "$WS"|"$WS"/*) echo "reap $sig pid=$pid"; kill "-$sig" "$pid" 2>/dev/null ;; esac
+            done
+          }
+          reap TERM; sleep 3; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          true
+
+      - name: Clean Workspace
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - uses: actions/checkout@v7
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/tofu-cluster
+          path: mulga
+
+      - uses: actions/checkout@v7
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Re-enable Go toolchain auto-download
+        run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
+
+      - name: Reclaim docker build cache
+        run: |
+          docker container prune -f || true
+          docker image prune -f || true
+          docker buildx prune -af --filter "until=24h" || true
+          docker system df || true
+
+      - name: Build Install Artifacts
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Provision VMs
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          export TF_VAR_git_branch="${GIT_BRANCH}"
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "3-node cluster provisioned"
+
+      - name: Bootstrap via Binary Install
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Resolve cluster SSH details
+        id: cluster
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path'); SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          NODES_CSV=""
+          for i in $(seq 0 $((NODE_COUNT - 1))); do
+            ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
+            if [ -z "$NODES_CSV" ]; then NODES_CSV="$ip"; else NODES_CSV="$NODES_CSV,$ip"; fi
+          done
+          echo "node1_ip=${NODE1_IP}" >> "$GITHUB_OUTPUT"
+          echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
+          echo "nodes_csv=${NODES_CSV}" >> "$GITHUB_OUTPUT"
+
+      - name: Build multinode binary on runner
+        working-directory: spinifex/tests/e2e
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "build multinode, teardown"
+          group_open "go test -c"
+          make e2e-build SUITES_BUILD="multinode teardown"
+          group_close
+          ok "$(basename _bin/multinode.test) $(stat -c '%s' _bin/multinode.test | numfmt --to=iec)"
+
+      - name: Ship test binaries to node1
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.cluster.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.cluster.outputs.node1_ip }}
+          ssh "${SSH_OPTS[@]}" "$VM" "rm -rf \$HOME/spinifex-e2e-bin && mkdir -p \$HOME/spinifex-e2e-bin"
+          tar -C spinifex/tests/e2e/_bin -cf - . | ssh "${SSH_OPTS[@]}" "$VM" "tar -xf - -C \$HOME/spinifex-e2e-bin"
+          ok "shipped binaries to $VM"
+
+      - name: Start packet capture on all nodes
+        run: |
+          set +e
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.cluster.outputs.ssh_key }}")
+          IFS=',' read -ra NODE_LIST <<< "${{ steps.cluster.outputs.nodes_csv }}"
+          for ip in "${NODE_LIST[@]}"; do
+            [ -z "$ip" ] && continue
+            scp "${SSH_OPTS[@]}" \
+              "${GITHUB_WORKSPACE}/spinifex/.github/scripts/start-net-capture.sh" \
+              tf-user@"$ip":/tmp/start-net-capture.sh
+            ssh "${SSH_OPTS[@]}" tf-user@"$ip" "bash /tmp/start-net-capture.sh" || true
+          done
+
+  multi_suite:
+    name: Multi env · suite
+    needs: multi_provision
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 60
+    strategy:
+      fail-fast: false
+      matrix:
+        suite: [ multinode ]
+    env:
+      NODE1_IP: ${{ needs.multi_provision.outputs.node1_ip }}
+      SSH_KEY: ${{ needs.multi_provision.outputs.ssh_key }}
+      NODES_CSV: ${{ needs.multi_provision.outputs.nodes_csv }}
+      ARTIFACT_DIR: /tmp/spinifex-e2e-multi-${{ github.run_id }}-${{ matrix.suite }}
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Install go-junit-report
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Run ${{ matrix.suite }}
+        id: e2e
+        run: |
+          set +e
+          SUITE="${{ matrix.suite }}"
+          VM_HOST="tf-user@${NODE1_IP}"
+          TEST_RUN_FLAG=""
+          [ -n "${TEST_RUN}" ] && TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+          cleanup_remote() {
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" "$VM_HOST" \
+              "pkill -u tf-user -TERM -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; true" \
+              2>/dev/null || true
+          }
+          trap cleanup_remote EXIT INT TERM
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" "$VM_HOST" \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/spinifex-e2e-bin; \
+             printf '\n━━ suite %s ━━\n' \"${SUITE}\"; \
+             date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-${SUITE}.start'; \
+             GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
+               SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+               SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
+               SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
+               ARTIFACT_DIR='${ARTIFACT_DIR}' \
+               ./${SUITE}.test -test.v -test.timeout=30m ${TEST_RUN_FLAG} \
+               2>&1 | tee '${ARTIFACT_DIR}/test-${SUITE}.log' \
+                    | sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'; \
+             exit \${PIPESTATUS[0]}"
+          echo "rc=$?" >> "$GITHUB_OUTPUT"
+          exit 0
+
+      - name: Collect suite artifacts from node1
+        if: always()
+        run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${SSH_KEY}")
+          ssh "${SSH_OPTS[@]}" tf-user@${NODE1_IP} "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+          true
+
+      - name: Convert test log to JUnit XML
+        if: always()
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log); SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          title: "Multi-Node E2E ${{ matrix.suite }} results"
+
+      - name: Analyze e2e failures
+        if: always()
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "Multi-Node E2E ${{ matrix.suite }} failure analysis"
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-multi-${{ matrix.suite }}-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
+      - name: Fail job if suite failed
+        if: always()
+        run: |
+          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
+            echo "::error::suite ${{ matrix.suite }} failed (rc=${{ steps.e2e.outputs.rc }})"
+            exit 1
+          fi
+
+  multi_teardown:
+    name: Multi env · teardown
+    needs: [ multi_provision, multi_suite ]
+    if: always() && inputs.run_multinode
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 20
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-multi-td-${{ github.run_id }}.log
+      SSH_KEY: ${{ needs.multi_provision.outputs.ssh_key }}
+      NODES_CSV: ${{ needs.multi_provision.outputs.nodes_csv }}
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          path: spinifex
+
+      # Reclaim leaked run-tagged resources before the cluster is destroyed.
+      - name: Sweep leaked run resources on node1
+        if: ${{ needs.multi_provision.result == 'success' && needs.multi_provision.outputs.node1_ip != '' }}
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}")
+          VM=tf-user@${{ needs.multi_provision.outputs.node1_ip }}
+          banner "sweep run-tagged resources (run ${{ github.run_id }})"
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "cd \$HOME/spinifex-e2e-bin 2>/dev/null && \
+             SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
+             SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+             ./teardown.test -test.v -test.run TestSweepRunResources -test.timeout=10m" \
+            2>&1 || true
+          true
+
+      - name: Teardown cluster
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          if [ -n "${NODES_CSV}" ] && [ -n "${SSH_KEY}" ]; then
+            IFS=',' read -ra NODE_LIST <<< "${NODES_CSV}"
+            for ip in "${NODE_LIST[@]}"; do
+              ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}" \
+                tf-user@"$ip" \
+                'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
+                2>/dev/null || true
+            done
+          fi
+          UID_SELF=$(id -u); WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in "$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;; esac
+            done
+          }
+          reap TERM; sleep 2; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
+          ok "teardown complete"
+
+  e2e_suites_status:
+    name: E2E suite blocks · summary
+    if: always()
+    needs: [ single_suite, multi_suite ]
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - name: Summarize
+        run: |
+          echo "## E2E suite-block results" >> "$GITHUB_STEP_SUMMARY"
+          echo "single_suite: ${{ needs.single_suite.result }}" >> "$GITHUB_STEP_SUMMARY"
+          echo "multi_suite:  ${{ needs.multi_suite.result }}" >> "$GITHUB_STEP_SUMMARY"
+          case "${{ needs.single_suite.result }}" in success|skipped) ;; *) exit 1 ;; esac
+          case "${{ needs.multi_suite.result }}" in success|skipped) ;; *) exit 1 ;; esac
@@ -0,0 +1,96 @@
+name: Publish EKS node AMI
+
+# Builds the eks-node system AMI (libguestfs, on the runner) and publishes the
+# compressed qcow2 + .sha256 sidecar to the Cloudflare R2 bucket that backs
+# iso.mulgadc.com/system-ami/. Operators then install it on any cluster with:
+#   spx admin images import --name spinifex-eks-node
+#
+# Manual-only: publishing overwrites the mirror artifact, so it is a deliberate
+# action, not something every push should trigger.
+
+on:
+  workflow_dispatch:
+    inputs:
+      dry_run:
+        description: "Build + convert only; skip the R2 upload"
+        type: boolean
+        default: false
+
+permissions:
+  contents: read
+
+concurrency:
+  group: publish-eks-ami
+  cancel-in-progress: false
+
+jobs:
+  publish:
+    name: Build + publish eks-node AMI
+    runs-on: [ self-hosted, ci-single ]
+
+    steps:
+      - name: Checkout Spinifex
+        uses: actions/checkout@v7
+        with:
+          path: spinifex
+
+      - name: Checkout Viperblock
+        uses: actions/checkout@v7
+        with:
+          repository: mulgadc/viperblock
+          ref: dev
+          path: viperblock
+
+      - name: Checkout Predastore
+        uses: actions/checkout@v7
+        with:
+          repository: mulgadc/predastore
+          ref: dev
+          path: predastore
+
+      - name: Setup Go
+        uses: actions/setup-go@v6
+        with:
+          go-version-file: "spinifex/go.mod"
+          # Self-hosted runner: the Go module + build cache already persists on
+          # local disk between runs. setup-go's cache-save ("Post Setup Go")
+          # otherwise re-uploads hundreds of MB to the Actions cache backend,
+          # taking longer than the build itself.
+          cache: false
+
+      - name: Setup Go Workspace
+        run: go work init ./spinifex ./viperblock ./predastore
+
+      - name: Install build + upload tooling
+        run: |
+          set -euo pipefail
+          sudo apt-get update -qq
+          sudo apt-get install -y --no-install-recommends libguestfs-tools qemu-utils
+          if ! command -v aws &>/dev/null; then
+            curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o /tmp/awscliv2.zip
+            unzip -q /tmp/awscliv2.zip -d /tmp/aws-install
+            sudo /tmp/aws-install/aws/install
+            rm -rf /tmp/awscliv2.zip /tmp/aws-install
+          fi
+
+      - name: Build + publish to R2
+        working-directory: spinifex
+        env:
+          # Map the repo's R2 secrets onto the names publish-system-image.sh reads.
+          AWS_ACCESS_KEY_ID: ${{ secrets.CF_R2_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.CF_R2_SECRET_ACCESS_KEY }}
+          R2_ENDPOINT: ${{ secrets.CF_R2_ENDPOINT_URL }}
+          R2_BUCKET: spinifex-iso
+          DRY_RUN: ${{ inputs.dry_run }}
+        run: |
+          set -euo pipefail
+          # libguestfs needs to read a host kernel; Ubuntu ships it mode 0600.
+          # direct backend = no libvirt on the runner.
+          export LIBGUESTFS_BACKEND=direct
+          sudo chmod 0644 /boot/vmlinuz-* || true
+          FLAGS=(--build)
+          if [[ "${DRY_RUN}" == "true" ]]; then
+            FLAGS+=(--dry-run)
+          fi
+          ./scripts/publish-system-image.sh \
+            scripts/images/eks-node/manifest.conf "${FLAGS[@]}"
@@ -9,21 +9,21 @@ permissions:

 jobs:
  distro-amd64:
-    runs-on: ubuntu-24.04
+    runs-on: ubuntu-26.04
    steps:
      - name: Checkout Spinifex
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          path: spinifex

      - name: Checkout Predastore
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          repository: mulgadc/predastore
          path: predastore

      - name: Checkout Viperblock
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          repository: mulgadc/viperblock
          path: viperblock
@@ -34,7 +34,7 @@ jobs:
      - name: Set up Go
        uses: actions/setup-go@v6
        with:
-          go-version: '1.26.4'
+          go-version: "1.26.4"

      - name: Build microVM artifacts (lb-agent + kernel + initramfs)
        run: cd spinifex && make build-microvm-image
@@ -54,21 +54,21 @@ jobs:
            spinifex/scripts/setup.sh

  distro-arm64:
-    runs-on: ubuntu-24.04-arm
+    runs-on: ubuntu-26.04-arm
    steps:
      - name: Checkout Spinifex
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          path: spinifex

      - name: Checkout Predastore
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          repository: mulgadc/predastore
          path: predastore

      - name: Checkout Viperblock
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          repository: mulgadc/viperblock
          path: viperblock
@@ -91,7 +91,7 @@ jobs:

  release:
    needs: [ distro-amd64, distro-arm64 ]
-    runs-on: ubuntu-24.04
+    runs-on: ubuntu-26.04
    steps:
      - name: Download amd64 Artifacts
        uses: actions/download-artifact@v8
@@ -131,7 +131,10 @@ jobs:
      REPO: ${{ github.repository }}
    steps:
      - name: Clean Workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*

      - uses: actions/create-github-app-token@v3
        id: app-token
@@ -140,7 +143,7 @@ jobs:
          private-key: ${{ secrets.APP_PRIVATE_KEY }}
          repositories: mulga

-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
        with:
          repository: mulgadc/mulga
          token: ${{ steps.app-token.outputs.token }}
@@ -169,8 +172,30 @@ jobs:
          TARBALL=$(find "${GITHUB_WORKSPACE}/release-assets" -name "*-linux-amd64.tar.gz" | head -1)
          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" "$SETUP_SH" "$TARBALL"

+      - name: Teardown
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+
+  baremetal:
+    needs: release
+    permissions:
+      contents: read
+      statuses: write
+    uses: ./.github/workflows/e2e-baremetal.yml
+    secrets: inherit
+
+  promote:
+    needs: [ verify, baremetal ]
+    runs-on: ubuntu-26.04
+    env:
+      TAG: ${{ github.ref_name }}
+      REPO: ${{ github.repository }}
+    steps:
      - name: Promote Release
-        if: success()
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
@@ -182,41 +207,41 @@ jobs:
            echo "Release ${TAG} promoted to latest"
          fi

+  # Annotate the (still-draft) release if either gate failed.
+  mark-failed:
+    needs: [ verify, baremetal ]
+    if: failure()
+    runs-on: ubuntu-26.04
+    env:
+      TAG: ${{ github.ref_name }}
+      REPO: ${{ github.repository }}
+    steps:
      - name: Mark Release Failed
-        if: failure()
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          EXISTING_BODY=$(gh release view "$TAG" --repo "$REPO" --json body -q .body)
          NOTES_FILE=$(mktemp)
-          printf 'Verification failed — see workflow run for details.\n\n%s' "$EXISTING_BODY" > "$NOTES_FILE"
+          printf 'Verification failed (tofu install or bare-metal boot) — see workflow run for details.\n\n%s' "$EXISTING_BODY" > "$NOTES_FILE"
          gh release edit "$TAG" --repo "$REPO" --notes-file "$NOTES_FILE"
          rm -f "$NOTES_FILE"

-      - name: Teardown
-        if: always()
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
-
  iso:
    needs: release
    runs-on: [ self-hosted, ci-single ]
-    timeout-minutes: 90
+    # ISO build (~60-90m) overlaps the parallel baremetal run (~120m); the
+    # upload gate then waits for baremetal to finish, so the job must outlast it.
+    timeout-minutes: 180
    steps:
      - name: Clean Workspace
        run: |
          ROOTFS="${GITHUB_WORKSPACE}/mulga/scripts/iso-builder/.cache/rootfs"
-          # Mirror make clean's umount sequence unconditionally — lazy unmount
-          # detaches immediately even if the mount is busy, so rm -rf is safe.
          sudo umount -l -R "${ROOTFS}" 2>/dev/null || true
          sudo umount -l "${ROOTFS}/dev/pts" 2>/dev/null || true
          sudo umount -l "${ROOTFS}/dev" 2>/dev/null || true
          sudo umount -l "${ROOTFS}/sys" 2>/dev/null || true
          sudo umount -l "${ROOTFS}/proc" 2>/dev/null || true
-          sudo rm -rf "$GITHUB_WORKSPACE"/*
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*

      - uses: actions/create-github-app-token@v3
        id: app-token
@@ -226,25 +251,25 @@ jobs:
          repositories: mulga

      - name: Checkout Spinifex
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          path: spinifex
          ref: ${{ github.ref }}

      - name: Checkout Predastore
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          repository: mulgadc/predastore
          path: predastore

      - name: Checkout Viperblock
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          repository: mulgadc/viperblock
          path: viperblock

      - name: Checkout Mulga iso-builder
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          repository: mulgadc/mulga
          token: ${{ steps.app-token.outputs.token }}
@@ -289,24 +314,25 @@ jobs:
            SPINIFEX_BIN="${GITHUB_WORKSPACE}/spinifex/bin/spx" \
            INSTALLER_BIN="${GITHUB_WORKSPACE}/spinifex/bin/spinifex-installer"

-      - name: Wait for verify
+      - name: Wait for verify + baremetal
        id: wait-verify
        if: steps.build-iso.outcome == 'success'
        continue-on-error: true
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
-          echo "Waiting for verify job to complete..."
+          echo "Waiting for verify + baremetal jobs to complete..."
          while true; do
-            CONCLUSION=$(gh api \
-              "repos/${{ github.repository }}/actions/runs/${{ github.run_id }}/jobs" \
-              --jq '.jobs[] | select(.name == "verify") | .conclusion // empty')
-            if [ -n "$CONCLUSION" ]; then
-              echo "Verify conclusion: $CONCLUSION"
-              [ "$CONCLUSION" = "success" ] && exit 0
+            JOBS=$(gh api \
+              "repos/${{ github.repository }}/actions/runs/${{ github.run_id }}/jobs" --paginate)
+            VERIFY=$(echo "$JOBS" | jq -r '.jobs[] | select(.name == "verify") | .conclusion // empty')
+            BAREMETAL=$(echo "$JOBS" | jq -r '.jobs[] | select(.name | startswith("baremetal")) | .conclusion // empty')
+            if [ -n "$VERIFY" ] && [ -n "$BAREMETAL" ]; then
+              echo "verify=$VERIFY baremetal=$BAREMETAL"
+              { [ "$VERIFY" = "success" ] && [ "$BAREMETAL" = "success" ]; } && exit 0
              exit 1
            fi
-            echo "Verify still running, retrying in 30s..."
+            echo "Still waiting (verify=${VERIFY:-pending} baremetal=${BAREMETAL:-pending}), retry 30s..."
            sleep 30
          done

@@ -327,14 +353,13 @@ jobs:
          AWS_ACCESS_KEY_ID: ${{ secrets.CF_R2_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.CF_R2_SECRET_ACCESS_KEY }}
          CF_R2_ENDPOINT_URL: ${{ secrets.CF_R2_ENDPOINT_URL }}
-          VERSION: ${{ github.ref_name }}
        run: |
          aws --endpoint-url="${CF_R2_ENDPOINT_URL}" \
-            s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso" \
-                  "s3://spinifex-iso/spinifex-${VERSION}.iso"
+            s3 cp "mulga/scripts/iso-builder/dist/spinifex.iso" \
+                  "s3://spinifex-iso/spinifex.iso"
          aws --endpoint-url="${CF_R2_ENDPOINT_URL}" \
-            s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso.sha256" \
-                  "s3://spinifex-iso/spinifex-${VERSION}.iso.sha256"
+            s3 cp "mulga/scripts/iso-builder/dist/spinifex.iso.sha256" \
+                  "s3://spinifex-iso/spinifex.iso.sha256"

      - name: Mark ISO failed
        if: always() && (steps.build-iso.outcome == 'failure' || steps.upload.outcome == 'failure')
@@ -23,24 +23,24 @@ permissions:
 jobs:
  build_and_test:
    name: Build and Test
-    runs-on: ubuntu-24.04
+    runs-on: ubuntu-26.04

    steps:
      - name: Checkout Spinifex
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          path: spinifex
          fetch-depth: 2

      - name: Checkout Viperblock
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          repository: mulgadc/viperblock
          ref: dev
          path: viperblock

      - name: Checkout Predastore
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          repository: mulgadc/predastore
          ref: dev
@@ -58,6 +58,9 @@ jobs:
      - name: Build Spinifex
        run: make -C spinifex build

+      - name: Manifest check + drift lint
+        run: make -C spinifex manifest-check manifest-lint
+
      - name: Run Tests with Coverage
        run: make -C spinifex test-cover

@@ -69,23 +72,23 @@ jobs:

  race_detection:
    name: Race Detection
-    runs-on: ubuntu-24.04
+    runs-on: ubuntu-26.04

    steps:
      - name: Checkout Spinifex
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          path: spinifex

      - name: Checkout Viperblock
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          repository: mulgadc/viperblock
          ref: dev
          path: viperblock

      - name: Checkout Predastore
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          repository: mulgadc/predastore
          ref: dev
@@ -105,23 +108,23 @@ jobs:

  lint_and_security:
    name: Lint and Security
-    runs-on: ubuntu-24.04
+    runs-on: ubuntu-26.04

    steps:
      - name: Checkout Spinifex
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          path: spinifex

      - name: Checkout Viperblock
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          repository: mulgadc/viperblock
          ref: dev
          path: viperblock

      - name: Checkout Predastore
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          repository: mulgadc/predastore
          ref: dev
@@ -23,14 +23,14 @@ permissions:
 jobs:
  ui_build_and_test:
    name: UI Build and Test
-    runs-on: ubuntu-24.04
+    runs-on: ubuntu-26.04
    defaults:
      run:
        working-directory: spinifex/services/spinifexui/frontend

    steps:
      - name: Checkout Code
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
        with:
          sparse-checkout: spinifex/services/spinifexui/frontend

@@ -37,13 +37,3 @@ Thumbs.db

 # Test reports
 *-report.txt
-
-# Internal docs (moved to private mulga repo)
-docs/INTERNAL.md
-docs/PRIVATE.md
-docs/DEVNOTES.md
-docs/PLAN.md
-docs/TODO.md
-docs/development/
-docs/internal/
-scripts/iac/aws/spinifex-test.pem
@@ -1,3 +0,0 @@
-# Spinifex Installation
-
-For documentation on using and installing Spinifex see [https://docs.mulgadc.com](https://docs.mulgadc.com/)
@@ -6,17 +6,12 @@ ARCH := $(shell uname -m)
 ifeq ($(ARCH),x86_64)
  GO_ARCH := amd64
  AWS_ARCH := x86_64
-  # On x86, we can run ARM VMs via emulation
-  QEMU_PACKAGES := qemu-system-x86 qemu-system-arm
 else ifeq ($(ARCH),aarch64)
  GO_ARCH := arm64
  AWS_ARCH := aarch64
-  # On ARM, we can run x86 VMs via emulation
-  QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
 else ifeq ($(ARCH),arm64)
  GO_ARCH := arm64
  AWS_ARCH := aarch64
-  QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
 else
  $(error Unsupported architecture: $(ARCH). Only x86_64 and aarch64/arm64 are supported.)
 endif
@@ -73,24 +68,27 @@ build-lb-agent:
 	@echo -e "\n....Building lb-agent (static)"
 	CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go

-build-system-image: ## Build a system image from manifest (use IMAGE=lb or IMAGE=eks-server)
+build-system-image: ## Build a system image from manifest (use IMAGE=lb or IMAGE=eks-node)
 ifndef IMAGE
 	$(error IMAGE is required. Usage: make build-system-image IMAGE=lb)
 endif
 	@if [ -f scripts/images/$(IMAGE).conf ]; then \
-		./scripts/build-system-image.sh scripts/images/$(IMAGE).conf; \
+		./scripts/build-system-image.sh scripts/images/$(IMAGE).conf $(if $(IMPORT),--import); \
 	elif [ -f scripts/images/$(IMAGE)/manifest.conf ]; then \
-		./scripts/build-system-image.sh scripts/images/$(IMAGE)/manifest.conf; \
+		./scripts/build-system-image.sh scripts/images/$(IMAGE)/manifest.conf $(if $(IMPORT),--import); \
 	else \
 		echo "ERROR: no manifest at scripts/images/$(IMAGE).conf or scripts/images/$(IMAGE)/manifest.conf"; \
 		exit 1; \
 	fi

-build-eks-server-image: ## Build the eks-server AMI (K3s control-plane variant)
-	$(MAKE) build-system-image IMAGE=eks-server
+build-eks-node-image: ## Build the unified eks-node AMI (K3s server+agent; role at first boot; IMPORT=1 to register)
+	$(MAKE) build-system-image IMAGE=eks-node

-build-eks-agent-image: ## Build the eks-agent AMI (K3s worker variant)
-	$(MAKE) build-system-image IMAGE=eks-agent
+import-eks-node-image: ## Build + register the eks-node AMI (requires a running cluster)
+	$(MAKE) build-system-image IMAGE=eks-node IMPORT=1
+
+publish-eks-node-image: ## Build + publish the eks-node AMI to Cloudflare R2 (needs R2_ENDPOINT + AWS_* env)
+	./scripts/publish-system-image.sh scripts/images/eks-node/manifest.conf --build

 MICROVM_OUT_DIR := build/microvm
 MICROVM_ARTIFACTS := $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz
@@ -112,14 +110,10 @@ install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz

-go_run:
-	@echo -e "\n....Running $(GO_PROJECT_NAME)...."
-	$(GOPATH)/bin/$(GO_PROJECT_NAME)
-
 # Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
 # Use this before committing to catch CI failures locally.
 preflight:
-	@$(MAKE) --no-print-directory QUIET=1 manifest-check lint govulncheck test-cover diff-coverage test-race test-harness
+	@$(MAKE) --no-print-directory QUIET=1 manifest-check manifest-lint lint govulncheck test-cover diff-coverage test-race test-harness
 	@echo -e "\n ✅ Preflight passed — safe to commit."

 # E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
@@ -136,6 +130,17 @@ manifest-check:
 	@echo -e "\n....Checking service-interfaces.yaml...."
 	@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml

+# Drift guards (Bead 5): direct-create fixture lint + NATS subject lint,
+# ratcheted against tests/e2e/manifest-lint/baseline.txt. Fails only on NEW
+# drift beyond the baseline.
+manifest-lint:
+	@echo -e "\n....Linting manifest drift (fixtures + subjects)...."
+	@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root .
+
+# Accept current drift into the baseline. Run after an intentional change.
+manifest-lint-update:
+	@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root . -update
+
 # Run unit tests
 test:
 	@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
@@ -165,13 +170,8 @@ diff-coverage: test-cover

 bench:
 	@echo -e "\n....Running benchmarks for $(GO_PROJECT_NAME)...."
-	$(MAKE) easyjson
 	LOG_IGNORE=1 go test -benchmem -run=. -bench=. ./...

-run:
-	$(MAKE) go_build
-	$(MAKE) go_run
-
 # Fast iteration: build + install binary + restart all services.
 # Microvm artifacts are reinstalled when they already exist on disk — the rule's
 # input timestamps drive a rebuild only if anything actually changed. On a fresh
@@ -198,11 +198,10 @@ clean:

 install-system:
 	@echo -e "\n....Installing system dependencies for $(ARCH)...."
-	@echo "QEMU packages: $(QEMU_PACKAGES)"
 	sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
 		-o Dpkg::Options::="--force-confdef" \
 		-o Dpkg::Options::="--force-confold" \
-		nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
+		nbdkit nbdkit-plugin-dev pkg-config qemu-system-x86 qemu-system-arm qemu-utils qemu-kvm \
 		ovmf qemu-efi-aarch64 \
 		libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
 		iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
@@ -231,19 +230,15 @@ install-aws:
 quickinstall: install-system install-go install-aws
 	@echo -e "\n✅ Quickinstall complete for $(ARCH)."
 	@echo "   Please ensure /usr/local/go/bin is in your PATH."
-	@echo "   Installed: Go ($(GO_ARCH)), AWS CLI ($(AWS_ARCH)), QEMU ($(QEMU_PACKAGES))"

-# Lint all Go code via golangci-lint (replaces check-format, vet, gosec, staticcheck)
 lint:
 	@echo "Running golangci-lint..."
 	$(_Q)golangci-lint run ./...
 	@echo "  golangci-lint ok"

-# Auto-fix all linter issues that have fixers
 fix:
 	golangci-lint run --fix ./...

-# Govulncheck — dependency vulnerability scanning (not covered by golangci-lint)
 govulncheck:
 	@echo "Running govulncheck..."
 	$(_Q)go tool govulncheck ./...
@@ -292,55 +287,8 @@ distro-arm64:
 distro-clean:
 	rm -rf dist/

-# Ansible dev lifecycle (experimental, parallel to dev-*.sh / reset-dev-env.sh).
-# See scripts/ansible/README.md and docs/development/improvements/ansible-dev-lifecycle.md.
-#
-# Variable overrides: pass EXTRA_VARS="key=val key2=val2" to forward as
-# `--extra-vars` to ansible-playbook. Plain `-e` on the make command line is
-# make's own flag and does NOT reach ansible.
-#   make ansible-dev-install EXTRA_VARS="spinifex_external_pool_start=192.168.1.90 spinifex_external_pool_end=192.168.1.99"
-EXTRA_VARS ?=
-_ANSIBLE_EXTRA = $(if $(strip $(EXTRA_VARS)),--extra-vars "$(EXTRA_VARS)",)
-
-ansible-dev-preflight:
-	cd scripts/ansible && ansible-playbook playbooks/dev-preflight.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-teardown:
-	cd scripts/ansible && ansible-playbook playbooks/dev-teardown.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-install:
-	cd scripts/ansible && ansible-playbook playbooks/dev-install.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-reset:
-	cd scripts/ansible && ansible-playbook playbooks/dev-reset.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-deploy:
-	cd scripts/ansible && ansible-playbook playbooks/dev-deploy.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-status:
-	cd scripts/ansible && ansible-playbook playbooks/dev-status.yml
-
-ansible-dev-logs:
-	cd scripts/ansible && ansible-playbook playbooks/dev-logs.yml
-
-# Snapshot / restore require an explicit -e snapshot_name=<name>. The
-# wrapper passes ANSIBLE_EXTRA through so devs can run
-# `make ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=before-merge'`.
-ansible-dev-snapshot:
-	cd scripts/ansible && ansible-playbook playbooks/dev-snapshot.yml $(ANSIBLE_EXTRA)
-
-ansible-dev-restore:
-	cd scripts/ansible && ansible-playbook playbooks/dev-restore.yml $(ANSIBLE_EXTRA)
-
-ansible-dev-version:
-	cd scripts/ansible && ansible-playbook playbooks/dev-version.yml
-
-ansible-dev-vpc:
-	cd scripts/ansible && ansible-playbook playbooks/dev-vpc.yml $(ANSIBLE_EXTRA)
-
-.PHONY: build build-ui build-installer build-lb-agent build-system-image build-eks-server-image build-eks-agent-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race diff-coverage bench run test-actions test-harness manifest-check diff-coverage bench run \
+.PHONY: build build-ui build-installer build-lb-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-harness manifest-check manifest-lint manifest-lint-update \
 	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
 	lint fix govulncheck \
-	distro distro-amd64 distro-arm64 distro-clean \
-	ansible-dev-preflight ansible-dev-teardown ansible-dev-install ansible-dev-reset ansible-dev-deploy ansible-dev-status ansible-dev-logs ansible-dev-snapshot ansible-dev-restore ansible-dev-version ansible-dev-vpc
+	distro distro-amd64 distro-arm64 distro-clean
@@ -6,20 +6,58 @@ mount -t devtmpfs dev /dev
 # virtio_mmio is built-in (CONFIG_VIRTIO_MMIO=y) so virtio-net devices are
 # already enumerated by the time PID 1 runs — no modprobe needed.
 # qemu_fw_cfg and virtio_net are =m in stock Alpine linux-virt, so load them.
-modprobe qemu_fw_cfg 2>/dev/null || modprobe fw_cfg_sysfs 2>/dev/null || true
-modprobe virtio_net 2>/dev/null || true
+#
+# modprobe needs a usable modules.dep; when the image is built on a host whose
+# depmod can't produce one, busybox modprobe silently no-ops and the module
+# never loads — fw_cfg then never appears and init panics. insmod by explicit
+# path needs no modules.dep, so fall back to it. Handles .ko and .ko.gz.
+load_mod() {
+    modprobe "$1" 2>/dev/null && return 0
+    for ko in $(find /lib/modules -name "$1.ko" -o -name "$1.ko.gz" 2>/dev/null); do
+        insmod "$ko" 2>/dev/null && return 0
+        case "$ko" in
+            *.gz) gunzip -c "$ko" >/tmp/mod.ko 2>/dev/null \
+                  && insmod /tmp/mod.ko 2>/dev/null && return 0 ;;
+        esac
+    done
+    return 1
+}
+load_mod qemu_fw_cfg || load_mod fw_cfg_sysfs || true
+# virtio_net depends on net_failover, which depends on failover. The load_mod
+# insmod fallback loads a single .ko with no dependency resolution, and
+# modules.dep is absent when the image is built on a kmod-less host (depmod
+# skipped), so modprobe can't resolve the chain either. Load it explicitly
+# bottom-up so the NIC comes up regardless of how the image was built —
+# without this the agent has no network and the LB hangs in provisioning.
+load_mod failover || true
+load_mod net_failover || true
+load_mod virtio_net || true

-for i in 1 2 3 4 5; do
-    [ -f /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw ] && break
-    sleep 0.02
+NETCFG=/sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw
+i=0
+while [ "$i" -lt 25 ]; do
+    [ -f "$NETCFG" ] && break
+    sleep 0.04
+    i=$((i + 1))
 done

-if [ ! -f /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw ]; then
+if [ ! -f "$NETCFG" ]; then
+    # Dump host/guest fw_cfg state to the serial console before init exits so a
+    # "netcfg not available" panic is diagnosable from the captured console log
+    # (module-not-loaded vs qemu-not-exposing-by_name vs wrong machine type).
    echo "[init] FATAL: fw_cfg netcfg not available" >&2
+    echo "[init] dmi: vendor=$(cat /sys/class/dmi/id/sys_vendor 2>/dev/null) product=$(cat /sys/class/dmi/id/product_name 2>/dev/null) bios=$(cat /sys/class/dmi/id/bios_version 2>/dev/null)" >&2
+    echo "[init] qemu_fw_cfg dir present: $([ -d /sys/firmware/qemu_fw_cfg ] && echo yes || echo NO)" >&2
+    echo "[init] modules: $(lsmod 2>/dev/null | awk 'NR>1{print $1}' | tr '\n' ',')" >&2
+    echo "[init] by_name listing:" >&2
+    ls -la /sys/firmware/qemu_fw_cfg/by_name/ 2>&1 | sed 's/^/[init]   /' >&2
+    ls -la /sys/firmware/qemu_fw_cfg/by_name/opt/ 2>&1 | sed 's/^/[init]   /' >&2
+    echo "[init] dmesg fw_cfg:" >&2
+    dmesg 2>/dev/null | grep -i fw_cfg | sed 's/^/[init]   /' >&2
    exit 1
 fi

-. /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw
+. "$NETCFG"

 for n in 0 1 2 3 4 5; do
    eval mac=\$NIC${n}_MAC
@@ -6,6 +6,7 @@ Wants=spinifex-nats.service

 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-gw
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
@@ -2,66 +2,29 @@
 Description=Spinifex Daemon (VM Orchestration)
 PartOf=spinifex.target
 After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.service ovn-controller.service openvswitch-ipsec.service
-# Wants=openvswitch-ipsec.service so systemd brings the IPsec userspace
-# daemon up before spinifex flips other_config:ipsec_encapsulation=true.
-# Without that daemon, OVS holds the cert pointers unused and strongSwan
-# never gets IKE config — tunnel traffic is encrypted with no SAs and
-# silently dropped. Wants= (not Requires=) keeps daemon bootable on
-# single-node / lab hosts where the package is intentionally absent.
+# Wants (not Requires): stay bootable when openvswitch-ipsec is absent
 Wants=openvswitch-ipsec.service
-# No Wants=spinifex-nats.service: Wants= reactivates a stopped unit during
-# `systemctl restart spinifex-daemon`, which would mask the standalone-startup
-# path the Tier 1 DDIL Scenario B exercises. spinifex.target Wants= covers
-# cold boot.

 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-daemon
 Group=spinifex
-# DDIL Tier 1 (siv-17): no ExecStartPre wait-for-nats. The daemon starts in
-# standalone mode and reconnects to NATS in the background — process-exiting
-# on a missing NATS would kill the local VM management plane.
-#
-# Opt-in strict-startup (§1d-strict, siv-44): set SPINIFEX_REQUIRE_NATS=1 to
-# restore pre-DDIL fail-fast UX. First NATS connect uses a ~30s bounded retry
-# and the daemon aborts on timeout. Intended for dev/test/single-node deploys
-# where silent NATS misconfig is more confusing than helpful. Do NOT set this
-# in the shipped unit — it re-introduces the SPOF that Tier 1 removed.
 ExecStart=/usr/local/bin/spx service spinifex start
 Restart=on-failure
 RestartSec=5
-# DDIL Tier 1: KillMode=process so systemd only signals the
-# daemon's main PID, not the cgroup. QEMU children stay alive across
-# `systemctl restart spinifex-daemon` and the new daemon reattaches to them
-# via the persisted local state file + pidfile + QMP socket. With the
-# default control-group mode, restart would terminate every guest VM —
-# defeating Scenario B's "daemon restart preserves running instances".
+# KillMode=process: guest QEMU survives daemon restart; the new daemon reattaches (DDIL)
 KillMode=process
-# 15s upper bound on the SIGTERM grace window. The new SIGTERM handler
-# skips VM teardown (it now just persists state and exits), so 15s is
-# plenty of slack for the cluster-manager + NATS unsubscribe path.
 TimeoutStopSec=15
+# RG-4: infra tier — reaped only after every guest
 OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
 Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex/
 Environment=SPINIFEX_WAL_DIR=/var/lib/spinifex/spinifex/
-# DDIL Tier 1: QEMU pidfiles, QMP sockets, and the
-# shared qemu.log all resolve via utils.pidPath() — which falls back
-# to os.TempDir() (=/tmp) when $HOME/spinifex doesn't exist. With
-# PrivateTmp=yes, each daemon start gets a fresh /tmp namespace, so
-# pidfiles written by QEMU (running in the OLD daemon's namespace)
-# are invisible to the new daemon. The /local/instances handler then
-# reports PID=0 and Scenario B fails its "live QEMU PID" assertion.
-# Pin XDG_RUNTIME_DIR to the host-wide tmpfs at /run/spinifex (created
-# by tmpfiles.d, already in ReadWritePaths) so pidPath() returns a
-# stable, namespace-independent path that both daemon incarnations and
-# QEMU children resolve identically.
+# Pin to host tmpfs: QEMU pidfiles/QMP sockets must survive PrivateTmp namespace churn (DDIL)
 Environment=XDG_RUNTIME_DIR=/run/spinifex

-# Filesystem access — NoNewPrivileges omitted (daemon uses sudo for ip, ovs-vsctl, dhcpcd)
-# CAP_SYS_ADMIN: required for GPU passthrough sysfs operations (vfio group management).
-# CAP_DAC_OVERRIDE: sysfs driver bind/unbind paths (/sys/bus/pci/drivers/vfio-pci/bind,
-# unbind) are root:root 0200; non-root processes need CAP_DAC_OVERRIDE to open them.
+# RG-9: daemon tier — privileged by necessity (GPU vfio). NoNewPrivileges omitted (RG-10: sudo ip/ovs-vsctl/dhcpcd)
 AmbientCapabilities=CAP_SYS_ADMIN CAP_DAC_OVERRIDE
 ProtectSystem=full
 PrivateDevices=no
@@ -73,7 +36,7 @@ DeviceAllow=/dev/vfio/vfio rw
 ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
 ReadWritePaths=/var/lib/spinifex/spinifex /var/log/spinifex /run/spinifex /run/spinifex/nbd

-# Hardening
+# RG-9: hardening baseline
 ProtectHome=yes
 PrivateTmp=yes
 RestrictSUIDSGID=yes
@@ -88,8 +51,6 @@ ProtectControlGroups=yes
 ProtectProc=invisible
 SystemCallArchitectures=native
 MemoryDenyWriteExecute=yes
-# DHCP lease acquisition moved to spinifex-vpcd (see dhcp-allocate-error.md);
-# the daemon no longer needs AF_PACKET or the softened kernel protections.
 RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
 ProtectHostname=yes
 KeyringMode=private
@@ -97,8 +58,7 @@ RemoveIPC=yes
 LimitCORE=0
 LimitNOFILE=65536
 LimitNPROC=4096
-# VFIO GPU passthrough requires locking all guest RAM into physical memory.
-# Without this, QEMU fails with RLIMIT_MEMLOCK exceeded when pinning guest pages.
+# LimitMEMLOCK=infinity: VFIO pins all guest RAM
 LimitMEMLOCK=infinity

 [Install]
@@ -0,0 +1,8 @@
+[Unit]
+Description=Spinifex guest VM tier
+Before=slices.target
+
+[Slice]
+# RG-3: child of spinifex.slice (spinifex-guests -> spinifex), sibling of
+# spinifex-system. Parent for the per-instance spinifex-instance-<id>.slice
+# leaves (0002 RG-7). Inherits the spinifex.slice ceiling; no extra limit here.
@@ -5,6 +5,7 @@ Before=spinifex-predastore.service spinifex-daemon.service spinifex-awsgw.servic

 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-nats
 Group=spinifex
 ExecStart=/usr/local/bin/spx service nats start
@@ -6,6 +6,7 @@ Wants=spinifex-nats.service

 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-storage
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
@@ -0,0 +1,9 @@
+[Unit]
+Description=Spinifex control/infra tier (NATS, predastore, viperblock, daemon, awsgw, vpcd, ui)
+Before=slices.target
+
+[Slice]
+# RG-3: child of spinifex.slice (spinifex-system -> spinifex), inherits the
+# parent ceiling, no extra limit. Holds the 7 control/infra services. Guest QEMU
+# moves to the sibling spinifex-guests.slice (per-instance placement lands in
+# 0002 RG-7); until then the daemon-forked guests stay under spinifex-daemon.service.
@@ -5,6 +5,7 @@ After=spinifex-awsgw.service

 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-ui
 Group=spinifex
 ExecStart=/usr/local/bin/spx service spinifex-ui start
@@ -6,6 +6,7 @@ Wants=spinifex-nats.service

 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-viperblock
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
@@ -6,6 +6,7 @@ Wants=spinifex-nats.service ovn-controller.service

 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-vpcd
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
@@ -16,37 +17,25 @@ OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
 Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/vpcd/

-# Filesystem access — NoNewPrivileges omitted (vpcd uses sudo for ovs-vsctl, ovn-nbctl, systemctl)
-ProtectSystem=full
+# RG-9: network tier. vpcd MUST share the host mount namespace — IMDS enters
+# per-VPC netns via setns(), and any filesystem-sandbox directive (ProtectSystem/
+# Home, PrivateTmp, ProtectKernel*, ProtectControlGroups/Proc, Read*Paths) forks a
+# private mount ns that traps the /run/netns bind-mount and breaks setns(2). Those,
+# plus MemoryDenyWriteExecute, are omitted by design — not regressions.
 PrivateDevices=no
-ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/ca.pem
-ReadWritePaths=/var/lib/spinifex/vpcd /var/log/spinifex
-
-# Hardening
-ProtectHome=yes
-PrivateTmp=yes
 RestrictSUIDSGID=no
 LockPersonality=yes
 RestrictRealtime=yes
-RestrictNamespaces=yes
 ProtectClock=yes
-ProtectKernelTunables=yes
-ProtectKernelModules=yes
-ProtectKernelLogs=yes
-ProtectControlGroups=yes
-ProtectProc=invisible
+# mnt alongside net: in-netns `ip` unshares a mount ns for its own /sys
+RestrictNamespaces=net mnt
 SystemCallArchitectures=native
 MemoryDenyWriteExecute=no
-# AF_PACKET: required by the in-process DHCP client (github.com/insomniacslk/dhcp
-# / nclient4) for raw L2 BOOTP on br-wan before an IP is configured on the
-# interface. See docs/development/bugs/dhcp-allocate-error.md.
+# AF_PACKET: raw L2 BOOTP for the in-process DHCP client before the iface has an IP
 RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK AF_PACKET
-# CAP_NET_RAW: open AF_PACKET sockets.
-# CAP_NET_BIND_SERVICE: covers unicast RENEW to UDP :68.
-# CAP_NET_ADMIN: already used for ovs-vsctl / ip link wiring on the bridge.
-# Note, disabled for now, sudo issue with ovs
-AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID
-CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID
+# Caps: NET_ADMIN (ovs/ip), NET_RAW (AF_PACKET), NET_BIND_SERVICE (:68), SETUID/SETGID, SYS_ADMIN (setns into VPC netns)
+AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID CAP_SYS_ADMIN
+CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID CAP_SYS_ADMIN
 NoNewPrivileges=no
 ProtectHostname=yes
 KeyringMode=private
@@ -0,0 +1,13 @@
+[Unit]
+Description=Spinifex resource-bounded slice (host ceiling)
+Before=slices.target
+
+[Slice]
+# RG-1: no CPUWeight — slice roots are siblings; capping below the default would
+# let user.slice out-prioritize the VM fleet. Operator CPU priority lives on
+# system.slice (system.slice.d/spinifex-reserve.conf).
+# RG-1/RG-5: throttle+reclaim at High, hard wall at Max (% of physical RAM).
+MemoryHigh=80%
+MemoryMax=90%
+# RG-5: spill to the full swap device before the MemoryMax wall; matched to swap size.
+MemorySwapMax=8G
@@ -0,0 +1,368 @@
+// ecr-credential-provider is a kubelet exec credential provider that lets a
+// Spinifex EKS worker node pull images from the internal Spinifex ECR registry.
+//
+// kubelet invokes it per-pull, feeding a CredentialProviderRequest on STDIN and
+// reading a CredentialProviderResponse on STDOUT. The provider fetches the
+// node IAM-role credentials from IMDS, SigV4-signs GetAuthorizationToken against
+// the AWS gateway's ECR endpoint, and returns the minted AWS:<jwt> as Basic auth
+// for the requested registry host.
+//
+// Static node config (gateway URL, CA, region) is read from the cloud-init env
+// file /etc/spinifex-eks/agent.env (KEY=value); real env vars override it.
+package main
+
+import (
+	"bufio"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/base64"
+	"encoding/json"
+	"fmt"
+	"io"
+	"log/slog"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/aws/credentials"
+	"github.com/aws/aws-sdk-go/aws/session"
+	"github.com/aws/aws-sdk-go/service/ecr"
+	"github.com/mulgadc/spinifex/internal/tlsconfig"
+
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
+)
+
+const (
+	credProviderAPIVersion = "credentialprovider.kubelet.k8s.io/v1"
+	defaultCacheDuration   = "10h"
+	// expiryMargin trims the token's real TTL so kubelet re-invokes the provider
+	// before the JWT actually expires mid-run.
+	expiryMargin = 30 * time.Minute
+
+	defaultEnvFile   = "/etc/spinifex-eks/agent.env"
+	defaultGatewayCA = "/etc/spinifex-eks/gateway-ca.pem"
+	defaultIMDSBase  = "http://169.254.169.254/latest"
+	imdsTokenTTL     = "21600"
+)
+
+// CredentialProviderRequest is the kubelet exec-provider request on STDIN.
+type CredentialProviderRequest struct {
+	APIVersion string `json:"apiVersion"`
+	Kind       string `json:"kind"`
+	Image      string `json:"image"`
+}
+
+// AuthConfig is a single registry credential entry.
+type AuthConfig struct {
+	Username string `json:"username"`
+	Password string `json:"password"`
+}
+
+// CredentialProviderResponse is the kubelet exec-provider response on STDOUT.
+type CredentialProviderResponse struct {
+	APIVersion    string                `json:"apiVersion"`
+	Kind          string                `json:"kind"`
+	CacheKeyType  string                `json:"cacheKeyType"`
+	CacheDuration string                `json:"cacheDuration"`
+	Auth          map[string]AuthConfig `json:"auth"`
+}
+
+// config holds the static node settings the provider needs.
+type config struct {
+	GatewayURL string
+	GatewayCA  string
+	Region     string
+	IMDSBase   string
+}
+
+func main() {
+	cfg := loadConfig(defaultEnvFile)
+
+	httpClient, err := gatewayHTTPClient(cfg.GatewayCA)
+	if err != nil {
+		slog.Error("ecr-credential-provider: build gateway client", "err", err)
+		emitEmpty(os.Stdout)
+		os.Exit(1)
+	}
+
+	if err := run(os.Stdin, os.Stdout, cfg, httpClient); err != nil {
+		slog.Error("ecr-credential-provider: run failed", "err", err)
+		// Still emit a valid empty-auth response so kubelet has a parseable reply.
+		emitEmpty(os.Stdout)
+		os.Exit(1)
+	}
+}
+
+// run is the testable core: it reads a CredentialProviderRequest from in,
+// resolves credentials via IMDS + the gateway, and writes a
+// CredentialProviderResponse to out. The httpClient must trust the gateway CA.
+func run(in io.Reader, out io.Writer, cfg config, httpClient *http.Client) error {
+	var req CredentialProviderRequest
+	if err := json.NewDecoder(in).Decode(&req); err != nil {
+		return fmt.Errorf("decode request: %w", err)
+	}
+	if req.Image == "" {
+		return fmt.Errorf("request image is empty")
+	}
+	imageHost := hostFromImage(req.Image)
+
+	creds, err := fetchIMDSCredentials(httpClient, cfg.IMDSBase)
+	if err != nil {
+		return fmt.Errorf("fetch IMDS credentials: %w", err)
+	}
+
+	username, password, proxyHost, expiresAt, err := getAuthToken(cfg, httpClient, creds)
+	if err != nil {
+		return fmt.Errorf("get authorization token: %w", err)
+	}
+
+	auth := AuthConfig{Username: username, Password: password}
+	authMap := map[string]AuthConfig{imageHost: auth}
+	if proxyHost != "" && proxyHost != imageHost {
+		authMap[proxyHost] = auth
+	}
+
+	resp := CredentialProviderResponse{
+		APIVersion:    credProviderAPIVersion,
+		Kind:          "CredentialProviderResponse",
+		CacheKeyType:  "Registry",
+		CacheDuration: cacheDurationFrom(expiresAt),
+		Auth:          authMap,
+	}
+	return json.NewEncoder(out).Encode(&resp)
+}
+
+// imdsCreds mirrors the IMDS security-credentials JSON shape.
+type imdsCreds struct {
+	Code            string `json:"Code"`
+	AccessKeyID     string `json:"AccessKeyId"`
+	SecretAccessKey string `json:"SecretAccessKey"`
+	Token           string `json:"Token"`
+	Expiration      string `json:"Expiration"`
+}
+
+// fetchIMDSCredentials resolves the node role name then its credentials via
+// IMDSv2 (PUT token, then GET with the token header).
+func fetchIMDSCredentials(client *http.Client, base string) (imdsCreds, error) {
+	base = strings.TrimRight(base, "/")
+	token := imdsV2Token(client, base)
+
+	roleName, err := imdsGet(client, base+"/meta-data/iam/security-credentials/", token)
+	if err != nil {
+		return imdsCreds{}, fmt.Errorf("fetch role name: %w", err)
+	}
+	roleName = strings.TrimSpace(roleName)
+	if roleName == "" {
+		return imdsCreds{}, fmt.Errorf("IMDS returned empty role name")
+	}
+
+	body, err := imdsGet(client, base+"/meta-data/iam/security-credentials/"+roleName, token)
+	if err != nil {
+		return imdsCreds{}, fmt.Errorf("fetch role credentials: %w", err)
+	}
+	var creds imdsCreds
+	if err := json.Unmarshal([]byte(body), &creds); err != nil {
+		return imdsCreds{}, fmt.Errorf("decode role credentials: %w", err)
+	}
+	if creds.AccessKeyID == "" || creds.SecretAccessKey == "" {
+		return imdsCreds{}, fmt.Errorf("IMDS credentials missing access/secret key")
+	}
+	return creds, nil
+}
+
+// imdsV2Token requests an IMDSv2 session token; returns "" if the service does
+// not enforce v2 (tokenless v1 then applies).
+func imdsV2Token(client *http.Client, base string) string {
+	req, err := http.NewRequest(http.MethodPut, base+"/api/token", nil)
+	if err != nil {
+		return ""
+	}
+	req.Header.Set("X-Aws-Ec2-Metadata-Token-Ttl-Seconds", imdsTokenTTL)
+	resp, err := client.Do(req)
+	if err != nil {
+		return ""
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return ""
+	}
+	body, _ := io.ReadAll(resp.Body)
+	return strings.TrimSpace(string(body))
+}
+
+func imdsGet(client *http.Client, url, token string) (string, error) {
+	req, err := http.NewRequest(http.MethodGet, url, nil)
+	if err != nil {
+		return "", err
+	}
+	if token != "" {
+		req.Header.Set("X-Aws-Ec2-Metadata-Token", token)
+	}
+	resp, err := client.Do(req)
+	if err != nil {
+		return "", err
+	}
+	defer resp.Body.Close()
+	body, _ := io.ReadAll(resp.Body)
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		return "", fmt.Errorf("IMDS %s returned %d: %s", url, resp.StatusCode, strings.TrimSpace(string(body)))
+	}
+	return string(body), nil
+}
+
+// getAuthToken builds an aws-sdk ECR client pointed at the gateway and calls
+// GetAuthorizationToken, decoding the AWS:<jwt> token into Basic-auth parts. It
+// returns the username, password, the ProxyEndpoint host, and the token expiry.
+func getAuthToken(cfg config, httpClient *http.Client, creds imdsCreds) (username, password, proxyHost string, expiresAt time.Time, err error) {
+	region := cfg.Region
+	if region == "" {
+		region = "us-east-1"
+	}
+	sess, serr := session.NewSession(&aws.Config{
+		Region:      aws.String(region),
+		Endpoint:    aws.String(cfg.GatewayURL),
+		Credentials: credentials.NewStaticCredentials(creds.AccessKeyID, creds.SecretAccessKey, creds.Token),
+		HTTPClient:  httpClient,
+		DisableSSL:  aws.Bool(false),
+	})
+	if serr != nil {
+		return "", "", "", time.Time{}, fmt.Errorf("new session: %w", serr)
+	}
+
+	out, gerr := ecr.New(sess).GetAuthorizationToken(&ecr.GetAuthorizationTokenInput{})
+	if gerr != nil {
+		return "", "", "", time.Time{}, fmt.Errorf("GetAuthorizationToken: %w", gerr)
+	}
+	if len(out.AuthorizationData) == 0 || out.AuthorizationData[0].AuthorizationToken == nil {
+		return "", "", "", time.Time{}, fmt.Errorf("GetAuthorizationToken returned no authorization data")
+	}
+	data := out.AuthorizationData[0]
+
+	raw, derr := base64.StdEncoding.DecodeString(aws.StringValue(data.AuthorizationToken))
+	if derr != nil {
+		return "", "", "", time.Time{}, fmt.Errorf("decode authorization token: %w", derr)
+	}
+	user, pass, ok := strings.Cut(string(raw), ":")
+	if !ok {
+		return "", "", "", time.Time{}, fmt.Errorf("authorization token not in user:password form")
+	}
+
+	if data.ExpiresAt != nil {
+		expiresAt = aws.TimeValue(data.ExpiresAt)
+	}
+	return user, pass, hostFromImage(aws.StringValue(data.ProxyEndpoint)), expiresAt, nil
+}
+
+// cacheDurationFrom derives a Go-duration string from the token expiry minus a
+// safety margin, falling back to the default when expiry is unknown or too near.
+func cacheDurationFrom(expiresAt time.Time) string {
+	if expiresAt.IsZero() {
+		return defaultCacheDuration
+	}
+	d := time.Until(expiresAt) - expiryMargin
+	if d <= 0 {
+		return defaultCacheDuration
+	}
+	return d.Round(time.Second).String()
+}
+
+// gatewayHTTPClient builds an HTTP client trusting the gateway CA at caPath.
+// When caPath is empty it relies on the system trust store.
+func gatewayHTTPClient(caPath string) (*http.Client, error) {
+	tlsCfg := &tls.Config{
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
+	}
+	if caPath != "" {
+		pem, err := os.ReadFile(caPath)
+		if err != nil {
+			return nil, fmt.Errorf("read gateway CA %q: %w", caPath, err)
+		}
+		pool := x509.NewCertPool()
+		if !pool.AppendCertsFromPEM(pem) {
+			return nil, fmt.Errorf("gateway CA %q has no usable certificates", caPath)
+		}
+		tlsCfg.RootCAs = pool
+	}
+	return &http.Client{
+		Timeout:   15 * time.Second,
+		Transport: &http.Transport{TLSClientConfig: tlsCfg, MaxIdleConns: 2, IdleConnTimeout: 30 * time.Second},
+	}, nil
+}
+
+// loadConfig reads the cloud-init env file then lets real env vars override. The
+// IMDS base is fixed (not in the env file) but overridable for tests.
+func loadConfig(envFile string) config {
+	env := parseEnvFile(envFile)
+	get := func(key string) string {
+		if v := os.Getenv(key); v != "" {
+			return v
+		}
+		return env[key]
+	}
+
+	cfg := config{
+		GatewayURL: get("EKS_GATEWAY_URL"),
+		GatewayCA:  get("EKS_GATEWAY_CA"),
+		Region:     get("EKS_REGION"),
+		IMDSBase:   get("EKS_IMDS_BASE"),
+	}
+	if cfg.GatewayCA == "" {
+		cfg.GatewayCA = defaultGatewayCA
+	}
+	if cfg.IMDSBase == "" {
+		cfg.IMDSBase = defaultIMDSBase
+	}
+	return cfg
+}
+
+// parseEnvFile reads a simple KEY=value file; missing files yield an empty map.
+func parseEnvFile(path string) map[string]string {
+	out := map[string]string{}
+	f, err := os.Open(path)
+	if err != nil {
+		return out
+	}
+	defer f.Close()
+	scanner := bufio.NewScanner(f)
+	for scanner.Scan() {
+		line := strings.TrimSpace(scanner.Text())
+		if line == "" || strings.HasPrefix(line, "#") {
+			continue
+		}
+		key, val, ok := strings.Cut(line, "=")
+		if !ok {
+			continue
+		}
+		out[strings.TrimSpace(key)] = strings.TrimSpace(val)
+	}
+	return out
+}
+
+// hostFromImage returns the host[:port] portion of an image ref or URL. It strips
+// any scheme and trims at the first path separator.
+func hostFromImage(image string) string {
+	if i := strings.Index(image, "://"); i >= 0 {
+		image = image[i+3:]
+	}
+	if i := strings.IndexByte(image, '/'); i >= 0 {
+		image = image[:i]
+	}
+	return image
+}
+
+// emitEmpty writes a valid CredentialProviderResponse with no auth entries.
+func emitEmpty(out io.Writer) {
+	resp := CredentialProviderResponse{
+		APIVersion:    credProviderAPIVersion,
+		Kind:          "CredentialProviderResponse",
+		CacheKeyType:  "Registry",
+		CacheDuration: defaultCacheDuration,
+		Auth:          map[string]AuthConfig{},
+	}
+	if err := json.NewEncoder(out).Encode(&resp); err != nil {
+		slog.Error("ecr-credential-provider: encode empty response", "err", err)
+	}
+}
@@ -0,0 +1,138 @@
+package main
+
+import (
+	"bytes"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/base64"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+// fakeBackend serves both the IMDS role/credentials endpoints and the gateway
+// ECR GetAuthorizationToken on one TLS server, routed by request path.
+func fakeBackend(t *testing.T) *httptest.Server {
+	t.Helper()
+	mux := http.NewServeMux()
+
+	// IMDSv2 token.
+	mux.HandleFunc("/latest/api/token", func(w http.ResponseWriter, r *http.Request) {
+		if r.Method != http.MethodPut {
+			w.WriteHeader(http.StatusMethodNotAllowed)
+			return
+		}
+		_, _ = w.Write([]byte("imds-v2-token"))
+	})
+	// IMDS role name.
+	mux.HandleFunc("/latest/meta-data/iam/security-credentials/", func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path == "/latest/meta-data/iam/security-credentials/" {
+			if r.Header.Get("X-Aws-Ec2-Metadata-Token") != "imds-v2-token" {
+				w.WriteHeader(http.StatusUnauthorized)
+				return
+			}
+			_, _ = w.Write([]byte("node-role"))
+			return
+		}
+		// Role credentials.
+		_ = json.NewEncoder(w).Encode(map[string]string{
+			"Code":            "Success",
+			"Type":            "AWS-HMAC",
+			"AccessKeyId":     "AKIATEST",
+			"SecretAccessKey": "secrettest",
+			"Token":           "sessiontoken",
+			"AccountId":       "123456789012",
+		})
+	})
+	// Gateway ECR GetAuthorizationToken (X-Amz-Target dispatch).
+	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+		if !strings.Contains(r.Header.Get("X-Amz-Target"), "GetAuthorizationToken") {
+			w.WriteHeader(http.StatusBadRequest)
+			return
+		}
+		authToken := base64.StdEncoding.EncodeToString([]byte("AWS:testjwt"))
+		w.Header().Set("Content-Type", "application/x-amz-json-1.1")
+		_ = json.NewEncoder(w).Encode(map[string]any{
+			"authorizationData": []map[string]any{{
+				"authorizationToken": authToken,
+				"proxyEndpoint":      "https://123456789012.dkr.ecr.us-east-1.spinifex.internal",
+			}},
+		})
+	})
+
+	srv := httptest.NewTLSServer(mux)
+	t.Cleanup(srv.Close)
+	return srv
+}
+
+func clientTrusting(srv *httptest.Server) *http.Client {
+	pool := x509.NewCertPool()
+	pool.AddCert(srv.Certificate())
+	return &http.Client{Transport: &http.Transport{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12}}}
+}
+
+func TestRun_ReturnsBasicAuth(t *testing.T) {
+	srv := fakeBackend(t)
+	client := clientTrusting(srv)
+
+	cfg := config{
+		GatewayURL: srv.URL,
+		Region:     "us-east-1",
+		IMDSBase:   srv.URL + "/latest",
+	}
+
+	const image = "123456789012.dkr.ecr.us-east-1.spinifex.internal/spinifex-demo:latest"
+	reqBody, err := json.Marshal(CredentialProviderRequest{
+		APIVersion: credProviderAPIVersion,
+		Kind:       "CredentialProviderRequest",
+		Image:      image,
+	})
+	if err != nil {
+		t.Fatalf("marshal request: %v", err)
+	}
+
+	var out bytes.Buffer
+	if err := run(bytes.NewReader(reqBody), &out, cfg, client); err != nil {
+		t.Fatalf("run: %v", err)
+	}
+
+	var resp CredentialProviderResponse
+	if err := json.Unmarshal(out.Bytes(), &resp); err != nil {
+		t.Fatalf("unmarshal response: %v\n%s", err, out.String())
+	}
+
+	if resp.Kind != "CredentialProviderResponse" {
+		t.Errorf("Kind = %q, want CredentialProviderResponse", resp.Kind)
+	}
+	if resp.CacheKeyType != "Registry" {
+		t.Errorf("CacheKeyType = %q, want Registry", resp.CacheKeyType)
+	}
+
+	host := "123456789012.dkr.ecr.us-east-1.spinifex.internal"
+	auth, ok := resp.Auth[host]
+	if !ok {
+		t.Fatalf("auth map missing host %q; got %v", host, resp.Auth)
+	}
+	if auth.Username != "AWS" {
+		t.Errorf("username = %q, want AWS", auth.Username)
+	}
+	if auth.Password != "testjwt" {
+		t.Errorf("password = %q, want testjwt", auth.Password)
+	}
+}
+
+func TestHostFromImage(t *testing.T) {
+	cases := map[string]string{
+		"acct.dkr.ecr.us-east-1.spinifex.internal/demo:latest": "acct.dkr.ecr.us-east-1.spinifex.internal",
+		"https://acct.dkr.ecr.us-east-1.spinifex.internal":     "acct.dkr.ecr.us-east-1.spinifex.internal",
+		"host:9999/repo:tag": "host:9999",
+		"plainhost":          "plainhost",
+	}
+	for in, want := range cases {
+		if got := hostFromImage(in); got != want {
+			t.Errorf("hostFromImage(%q) = %q, want %q", in, got, want)
+		}
+	}
+}
@@ -0,0 +1,139 @@
+// eks-gateway-fetch runs inside the EKS K3s control-plane VM and fetches a
+// control-plane resource from the host through the AWS gateway, instead of
+// dialing core NATS directly. It is the read-side companion to
+// eks-gateway-publish.
+//
+// It SigV4-signs (service "eks") an HTTPS GET to the gateway, retrying with
+// backoff until the gateway returns 2xx or the attempt budget is exhausted (so
+// a degraded link surfaces as a non-zero exit), then emits the staged add-ons
+// as tab-separated lines the on-VM addon-sync shell agent consumes without a
+// JSON parser:
+//
+//	<addonName>\t<addonVersion>\t<serviceAccountRoleArn>\t<base64(configurationValues)>
+//
+// The configuration values are base64-encoded so embedded tabs/newlines cannot
+// break the line framing.
+//
+// Usage:
+//
+//	eks-gateway-fetch -resource addons   # GET /clusters/{cluster}/internal-addons/{accountId}
+//
+// Flags default to environment variables seeded by cloud-init:
+// EKS_GATEWAY_URL, EKS_GATEWAY_CA, EKS_ACCESS_KEY, EKS_SECRET_KEY, EKS_REGION,
+// EKS_ACCOUNT_ID, EKS_CLUSTER_NAME.
+package main
+
+import (
+	"bufio"
+	"encoding/base64"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"log/slog"
+	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/eksgw"
+
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
+)
+
+// stagedAddon mirrors the gateway's StagedAddonManifest wire shape; duplicated
+// here to keep this tiny VM binary free of the handler package's dependency
+// graph.
+type stagedAddon struct {
+	AddonName             string `json:"addonName"`
+	AddonVersion          string `json:"addonVersion"`
+	ServiceAccountRoleArn string `json:"serviceAccountRoleArn"`
+	ConfigurationValues   string `json:"configurationValues"`
+}
+
+type internalAddonsResponse struct {
+	Addons []stagedAddon `json:"addons"`
+}
+
+const (
+	maxAttempts = 30
+	retryDelay  = 5 * time.Second
+)
+
+func main() {
+	var (
+		gatewayURL  string
+		gatewayCA   string
+		accessKey   string
+		secretKey   string
+		region      string
+		accountID   string
+		clusterName string
+		resource    string
+	)
+
+	flag.StringVar(&gatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
+	flag.StringVar(&gatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
+	flag.StringVar(&accessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
+	flag.StringVar(&secretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
+	flag.StringVar(&region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
+	flag.StringVar(&accountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
+	flag.StringVar(&clusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
+	flag.StringVar(&resource, "resource", "addons", "Resource to fetch: addons")
+	flag.Parse()
+
+	switch {
+	case accountID == "":
+		fatal("--account-id is required (or set EKS_ACCOUNT_ID)")
+	case clusterName == "":
+		fatal("--cluster is required (or set EKS_CLUSTER_NAME)")
+	case resource != "addons":
+		fatal("--resource must be addons")
+	}
+
+	client, err := eksgw.New(gatewayURL, gatewayCA, accessKey, secretKey, region)
+	if err != nil {
+		fatal(fmt.Sprintf("build gateway client: %v", err))
+	}
+	path := "/clusters/" + clusterName + "/internal-addons/" + accountID
+
+	var lastErr error
+	for attempt := 1; attempt <= maxAttempts; attempt++ {
+		body, err := client.Get(path)
+		if err != nil {
+			lastErr = err
+			slog.Warn("eks-gateway-fetch: attempt failed", "resource", resource, "attempt", attempt, "err", err)
+			if attempt < maxAttempts {
+				time.Sleep(retryDelay)
+			}
+			continue
+		}
+		if werr := emitAddonsTSV(os.Stdout, body); werr != nil {
+			fatal(fmt.Sprintf("render response: %v", werr))
+		}
+		return
+	}
+	fatal(fmt.Sprintf("fetch failed after %d attempts: %v", maxAttempts, lastErr))
+}
+
+// emitAddonsTSV decodes the internal-addons response and writes one TSV line per
+// staged add-on to w. An empty add-on set writes nothing (the agent then GCs
+// every locally-rendered manifest), which is the correct steady state for a
+// cluster with no managed add-ons.
+func emitAddonsTSV(out io.Writer, body []byte) error {
+	var resp internalAddonsResponse
+	if err := json.Unmarshal(body, &resp); err != nil {
+		return fmt.Errorf("unmarshal internal-addons response: %w", err)
+	}
+	w := bufio.NewWriter(out)
+	for _, a := range resp.Addons {
+		cfg := base64.StdEncoding.EncodeToString([]byte(a.ConfigurationValues))
+		if _, err := fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", a.AddonName, a.AddonVersion, a.ServiceAccountRoleArn, cfg); err != nil {
+			return err
+		}
+	}
+	return w.Flush()
+}
+
+func fatal(msg string) {
+	slog.Error("eks-gateway-fetch: " + msg)
+	os.Exit(1)
+}
@@ -0,0 +1,44 @@
+package main
+
+import (
+	"bytes"
+	"encoding/base64"
+	"testing"
+)
+
+func TestEmitAddonsTSV(t *testing.T) {
+	body := []byte(`{"addons":[
+		{"addonName":"spinifex-noop","addonVersion":"0.1.0"},
+		{"addonName":"aws-load-balancer-controller","addonVersion":"2.8.1",
+		 "serviceAccountRoleArn":"arn:aws:iam::111122223333:role/alb",
+		 "configurationValues":"{\"replicaCount\":2}"}
+	]}`)
+
+	var buf bytes.Buffer
+	if err := emitAddonsTSV(&buf, body); err != nil {
+		t.Fatalf("emitAddonsTSV: %v", err)
+	}
+	cfg := base64.StdEncoding.EncodeToString([]byte(`{"replicaCount":2}`))
+	want := "spinifex-noop\t0.1.0\t\t\n" +
+		"aws-load-balancer-controller\t2.8.1\tarn:aws:iam::111122223333:role/alb\t" + cfg + "\n"
+	if buf.String() != want {
+		t.Fatalf("TSV mismatch:\n got: %q\nwant: %q", buf.String(), want)
+	}
+}
+
+func TestEmitAddonsTSV_Empty(t *testing.T) {
+	var buf bytes.Buffer
+	if err := emitAddonsTSV(&buf, []byte(`{"addons":[]}`)); err != nil {
+		t.Fatalf("emitAddonsTSV: %v", err)
+	}
+	if buf.Len() != 0 {
+		t.Fatalf("empty addon set must emit nothing, got %q", buf.String())
+	}
+}
+
+func TestEmitAddonsTSV_BadJSON(t *testing.T) {
+	var buf bytes.Buffer
+	if err := emitAddonsTSV(&buf, []byte(`{not-json`)); err == nil {
+		t.Fatal("expected error on malformed JSON")
+	}
+}
@@ -0,0 +1,134 @@
+// eks-gateway-publish runs inside the EKS K3s control-plane VM and relays a
+// single publication to the host through the AWS gateway, instead of dialing
+// core NATS directly.
+//
+// It reads a JSON payload on stdin, wraps it as
+// {accountId, channel, kind, payload}, SigV4-signs (service "eks") an HTTPS
+// POST to {gateway}/clusters/{cluster}/internal-publish, and retries with
+// backoff until the gateway returns 2xx or the attempt budget is exhausted —
+// so a degraded link surfaces as a non-zero exit rather than a silently
+// dropped message (the failure mode of fire-and-forget `nats pub`).
+//
+// Usage:
+//
+//	echo '{"token":"..."}' | eks-gateway-publish -channel bootstrap -kind k3s-bootstrap-token
+//	kubectl ... | eks-gateway-publish -channel state
+//
+// Flags default to environment variables seeded by cloud-init:
+// EKS_GATEWAY_URL, EKS_GATEWAY_CA, EKS_ACCESS_KEY, EKS_SECRET_KEY, EKS_REGION,
+// EKS_ACCOUNT_ID, EKS_CLUSTER_NAME.
+package main
+
+import (
+	"bytes"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"log/slog"
+	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/eksgw"
+
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
+)
+
+const (
+	// Retry budget: the control plane reaches readiness before this runs, so a
+	// failing POST means a degraded link, not a cold start. Bounded so a stuck
+	// boot still terminates the OpenRC service.
+	maxAttempts = 30
+	retryDelay  = 5 * time.Second
+)
+
+type publishBody struct {
+	AccountID string          `json:"accountId"`
+	Channel   string          `json:"channel"`
+	Kind      string          `json:"kind"`
+	Payload   json.RawMessage `json:"payload"`
+}
+
+func main() {
+	var (
+		gatewayURL  string
+		gatewayCA   string
+		accessKey   string
+		secretKey   string
+		region      string
+		accountID   string
+		clusterName string
+		channel     string
+		kind        string
+	)
+
+	flag.StringVar(&gatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
+	flag.StringVar(&gatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
+	flag.StringVar(&accessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
+	flag.StringVar(&secretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
+	flag.StringVar(&region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
+	flag.StringVar(&accountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
+	flag.StringVar(&clusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
+	flag.StringVar(&channel, "channel", "", "Publish channel: bootstrap|state|addon")
+	flag.StringVar(&kind, "kind", "", "Bootstrap subject kind (bootstrap channel only)")
+	flag.Parse()
+
+	switch {
+	case accountID == "":
+		fatal("--account-id is required (or set EKS_ACCOUNT_ID)")
+	case clusterName == "":
+		fatal("--cluster is required (or set EKS_CLUSTER_NAME)")
+	case channel != "bootstrap" && channel != "state" && channel != "addon":
+		fatal("--channel must be bootstrap, state or addon")
+	case channel == "bootstrap" && kind == "":
+		fatal("--kind is required for the bootstrap channel")
+	}
+
+	payload, err := io.ReadAll(os.Stdin)
+	if err != nil {
+		fatal(fmt.Sprintf("read stdin payload: %v", err))
+	}
+	if len(bytes.TrimSpace(payload)) == 0 {
+		fatal("empty stdin payload")
+	}
+	if !json.Valid(payload) {
+		fatal("stdin payload is not valid JSON")
+	}
+
+	body, err := json.Marshal(publishBody{
+		AccountID: accountID,
+		Channel:   channel,
+		Kind:      kind,
+		Payload:   json.RawMessage(payload),
+	})
+	if err != nil {
+		fatal(fmt.Sprintf("marshal request body: %v", err))
+	}
+
+	client, err := eksgw.New(gatewayURL, gatewayCA, accessKey, secretKey, region)
+	if err != nil {
+		fatal(fmt.Sprintf("build gateway client: %v", err))
+	}
+	path := "/clusters/" + clusterName + "/internal-publish"
+
+	var lastErr error
+	for attempt := 1; attempt <= maxAttempts; attempt++ {
+		if _, err := client.Post(path, body); err != nil {
+			lastErr = err
+			slog.Warn("eks-gateway-publish: attempt failed",
+				"channel", channel, "kind", kind, "attempt", attempt, "err", err)
+			if attempt < maxAttempts {
+				time.Sleep(retryDelay)
+			}
+			continue
+		}
+		slog.Info("eks-gateway-publish: published", "channel", channel, "kind", kind)
+		return
+	}
+	fatal(fmt.Sprintf("publish failed after %d attempts: %v", maxAttempts, lastErr))
+}
+
+func fatal(msg string) {
+	slog.Error("eks-gateway-publish: " + msg)
+	os.Exit(1)
+}
@@ -0,0 +1,137 @@
+package main
+
+import (
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/tls"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/base64"
+	"encoding/pem"
+	"fmt"
+	"math/big"
+	"net"
+	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/tlsconfig"
+)
+
+// tlsCertificate is the webhook's loopback serving certificate.
+type tlsCertificate struct {
+	cert tls.Certificate
+}
+
+func (c tlsCertificate) serverTLSConfig() *tls.Config {
+	return &tls.Config{
+		Certificates:     []tls.Certificate{c.cert},
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
+	}
+}
+
+// ensureServingCert loads the persisted self-signed serving cert/key, or mints
+// a fresh ECDSA-P256 pair (SAN 127.0.0.1 / localhost) and persists it. Reusing
+// across restarts keeps the CA the apiserver loaded into its webhook kubeconfig
+// valid even if the webhook process is restarted. Returns the parsed cert and
+// the certificate PEM (for embedding as the kubeconfig CA bundle).
+func ensureServingCert(certPath, keyPath string) (tlsCertificate, []byte, error) {
+	if certPEM, keyPEM, ok := readCertPair(certPath, keyPath); ok {
+		cert, err := tls.X509KeyPair(certPEM, keyPEM)
+		if err == nil {
+			return tlsCertificate{cert: cert}, certPEM, nil
+		}
+		// Fall through to regenerate on a corrupt/incompatible persisted pair.
+	}
+
+	certPEM, keyPEM, err := generateSelfSigned()
+	if err != nil {
+		return tlsCertificate{}, nil, err
+	}
+	if err := os.WriteFile(certPath, certPEM, 0o644); err != nil {
+		return tlsCertificate{}, nil, fmt.Errorf("write cert %s: %w", certPath, err)
+	}
+	if err := os.WriteFile(keyPath, keyPEM, 0o600); err != nil {
+		return tlsCertificate{}, nil, fmt.Errorf("write key %s: %w", keyPath, err)
+	}
+	cert, err := tls.X509KeyPair(certPEM, keyPEM)
+	if err != nil {
+		return tlsCertificate{}, nil, fmt.Errorf("parse minted keypair: %w", err)
+	}
+	return tlsCertificate{cert: cert}, certPEM, nil
+}
+
+func readCertPair(certPath, keyPath string) (certPEM, keyPEM []byte, ok bool) {
+	c, err := os.ReadFile(certPath)
+	if err != nil {
+		return nil, nil, false
+	}
+	k, err := os.ReadFile(keyPath)
+	if err != nil {
+		return nil, nil, false
+	}
+	return c, k, true
+}
+
+func generateSelfSigned() (certPEM, keyPEM []byte, err error) {
+	priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		return nil, nil, fmt.Errorf("generate key: %w", err)
+	}
+	serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
+	if err != nil {
+		return nil, nil, fmt.Errorf("serial: %w", err)
+	}
+	now := time.Now().UTC()
+	tmpl := x509.Certificate{
+		SerialNumber:          serial,
+		Subject:               pkix.Name{CommonName: "eks-token-webhook"},
+		NotBefore:             now.Add(-time.Hour),
+		NotAfter:              now.AddDate(10, 0, 0),
+		KeyUsage:              x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment | x509.KeyUsageCertSign,
+		ExtKeyUsage:           []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+		BasicConstraintsValid: true,
+		IsCA:                  true,
+		IPAddresses:           []net.IP{net.IPv4(127, 0, 0, 1), net.IPv6loopback},
+		DNSNames:              []string{"localhost"},
+	}
+	der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &priv.PublicKey, priv)
+	if err != nil {
+		return nil, nil, fmt.Errorf("create cert: %w", err)
+	}
+	keyDER, err := x509.MarshalECPrivateKey(priv)
+	if err != nil {
+		return nil, nil, fmt.Errorf("marshal key: %w", err)
+	}
+	certPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
+	keyPEM = pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
+	return certPEM, keyPEM, nil
+}
+
+// writeAPIServerKubeconfig writes the kubeconfig kube-apiserver loads via
+// --authentication-token-webhook-config-file. It points the apiserver at the
+// loopback webhook and trusts its self-signed serving cert as the CA bundle.
+func writeAPIServerKubeconfig(path, addr string, certPEM []byte) error {
+	caB64 := base64.StdEncoding.EncodeToString(certPEM)
+	kubeconfig := fmt.Sprintf(`apiVersion: v1
+kind: Config
+clusters:
+- name: mulga-eks-token-webhook
+  cluster:
+    certificate-authority-data: %s
+    server: https://%s/authenticate
+users:
+- name: kube-apiserver
+contexts:
+- name: webhook
+  context:
+    cluster: mulga-eks-token-webhook
+    user: kube-apiserver
+current-context: webhook
+`, caB64, addr)
+	if err := os.WriteFile(path, []byte(kubeconfig), 0o600); err != nil {
+		return fmt.Errorf("write %s: %w", path, err)
+	}
+	return nil
+}
@@ -1,38 +1,227 @@
 // eks-token-webhook is the K8s TokenReview webhook authenticator baked into
-// the eks-server AMI. It decodes the SigV4-presigned GetCallerIdentity URL
-// produced by `aws eks get-token`, enforces the X-K8s-Aws-Id cross-cluster
-// pin, calls Mulga STS over NATS, and resolves the principal to a TokenReview
-// response via the cluster's AccessEntry KV.
+// the eks-server AMI. Per kubectl call the kube-apiserver POSTs the bearer
+// token (produced by `aws eks get-token`); the webhook relays it through the
+// AWS gateway broker (SigV4 HTTPS POST to /clusters/{name}/token-review), which
+// host-side runs the STS verify (cross-cluster x-k8s-aws-id pin) + AccessEntry
+// KV lookup and returns the resolved identity. The webhook never speaks core
+// NATS — same gateway-broker model as ELBv2's lb-agent and eks-gateway-publish.
 //
-// This is a placeholder build that returns 503 Service Unavailable on every
-// path. It exists so the eks-server AMI build succeeds (INSTALL_BINARIES is
-// satisfied) and the K3s server VM boots cleanly. Real implementation lands
-// with bead mulga-cs-eks-6c.
+// It binds 127.0.0.1 only; the kube-apiserver is the sole client (loopback).
+// On startup it writes the apiserver webhook kubeconfig (with its self-signed
+// serving CA) so k3s can be pointed at it via
+// --authentication-token-webhook-config-file.
 package main

 import (
+	"encoding/json"
+	"errors"
 	"flag"
+	"fmt"
 	"log/slog"
 	"net/http"
 	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/eksgw"
+	handlers_eks "github.com/mulgadc/spinifex/spinifex/handlers/eks"
 )

+// config is the webhook's runtime configuration, sourced from the first-boot
+// env (/etc/spinifex-eks/first-boot.env) cloud-init seeds onto the VM.
+type config struct {
+	addr        string
+	gatewayURL  string
+	gatewayCA   string
+	accessKey   string
+	secretKey   string
+	region      string
+	accountID   string
+	clusterName string
+	certPath    string
+	keyPath     string
+	kubeconfig  string
+}
+
+func loadConfig() (config, error) {
+	c := config{
+		addr:        flagAddr,
+		gatewayURL:  os.Getenv("EKS_GATEWAY_URL"),
+		gatewayCA:   os.Getenv("EKS_GATEWAY_CA"),
+		accessKey:   os.Getenv("EKS_ACCESS_KEY"),
+		secretKey:   os.Getenv("EKS_SECRET_KEY"),
+		region:      os.Getenv("EKS_REGION"),
+		accountID:   os.Getenv("EKS_ACCOUNT_ID"),
+		clusterName: os.Getenv("EKS_CLUSTER_NAME"),
+		certPath:    envOr("EKS_WEBHOOK_CERT", "/etc/spinifex-eks/token-webhook.crt"),
+		keyPath:     envOr("EKS_WEBHOOK_KEY", "/etc/spinifex-eks/token-webhook.key"),
+		kubeconfig:  envOr("EKS_WEBHOOK_KUBECONFIG", "/etc/spinifex-eks/token-webhook.kubeconfig"),
+	}
+	switch {
+	case c.gatewayURL == "":
+		return c, errors.New("EKS_GATEWAY_URL not set")
+	case c.accessKey == "" || c.secretKey == "":
+		return c, errors.New("EKS_ACCESS_KEY / EKS_SECRET_KEY not set")
+	case c.accountID == "":
+		return c, errors.New("EKS_ACCOUNT_ID not set")
+	case c.clusterName == "":
+		return c, errors.New("EKS_CLUSTER_NAME not set")
+	}
+	return c, nil
+}
+
+func envOr(key, def string) string {
+	if v := os.Getenv(key); v != "" {
+		return v
+	}
+	return def
+}
+
+var flagAddr string
+
 func main() {
-	addr := flag.String("addr", "127.0.0.1:8443", "listen address")
+	flag.StringVar(&flagAddr, "addr", "127.0.0.1:8443", "listen address (loopback only)")
 	flag.Parse()

 	slog.SetDefault(slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelInfo})))

-	mux := http.NewServeMux()
-	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
-		w.Header().Set("Retry-After", "30")
-		w.Header().Set("Content-Type", "application/json")
-		http.Error(w, `{"error":"eks-token-webhook stub: real implementation not yet deployed"}`, http.StatusServiceUnavailable)
-	})
+	cfg, err := loadConfig()
+	if err != nil {
+		slog.Error("eks-token-webhook config error", "err", err)
+		os.Exit(1)
+	}

-	slog.Info("eks-token-webhook stub starting", "addr", *addr)
-	if err := http.ListenAndServe(*addr, mux); err != nil {
-		slog.Error("listen failed", "err", err)
+	if err := run(cfg); err != nil {
+		slog.Error("eks-token-webhook fatal", "err", err)
 		os.Exit(1)
 	}
 }
+
+func run(cfg config) error {
+	// Serving cert + apiserver kubeconfig. Persisted and reused across restarts
+	// so a webhook crash does not invalidate the CA the apiserver already loaded.
+	tlsCert, certPEM, err := ensureServingCert(cfg.certPath, cfg.keyPath)
+	if err != nil {
+		return fmt.Errorf("serving cert: %w", err)
+	}
+	if err := writeAPIServerKubeconfig(cfg.kubeconfig, cfg.addr, certPEM); err != nil {
+		return fmt.Errorf("write apiserver kubeconfig: %w", err)
+	}
+
+	client, err := eksgw.New(cfg.gatewayURL, cfg.gatewayCA, cfg.accessKey, cfg.secretKey, cfg.region)
+	if err != nil {
+		return fmt.Errorf("build gateway client: %w", err)
+	}
+
+	authr := &authenticator{
+		accountID:   cfg.accountID,
+		clusterName: cfg.clusterName,
+		review:      gatewayReviewer(client, cfg.clusterName, cfg.accountID),
+	}
+
+	mux := http.NewServeMux()
+	mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("ok"))
+	})
+	mux.HandleFunc("/authenticate", authr.handle)
+
+	server := newServer(cfg.addr, mux, tlsCert)
+	slog.Info("eks-token-webhook listening", "addr", cfg.addr, "cluster", cfg.clusterName)
+	if err := server.ListenAndServeTLS("", ""); err != nil && !errors.Is(err, http.ErrServerClosed) {
+		return fmt.Errorf("serve: %w", err)
+	}
+	return nil
+}
+
+// tokenReviewRequest is the body POSTed to /clusters/{name}/token-review. The
+// webhook holds system SigV4 creds (system account, not the cluster account),
+// so accountId names the cluster account explicitly — same as PublishInternal.
+type tokenReviewRequest struct {
+	AccountID string `json:"accountId"`
+	Token     string `json:"token"`
+}
+
+// gatewayReviewer returns the per-call relay: SigV4-POST {accountId,token} to
+// the gateway and decode the resolved identity. A non-2xx / transport error is
+// surfaced (handle() turns it into a 5xx so the apiserver retries) rather than a
+// silent deny, distinguishing "webhook broker down" from "token rejected".
+func gatewayReviewer(client *eksgw.Client, clusterName, accountID string) func(string) (handlers_eks.WebhookTokenReviewResult, error) {
+	path := "/clusters/" + clusterName + "/token-review"
+	return func(token string) (handlers_eks.WebhookTokenReviewResult, error) {
+		body, err := json.Marshal(tokenReviewRequest{AccountID: accountID, Token: token})
+		if err != nil {
+			return handlers_eks.WebhookTokenReviewResult{}, fmt.Errorf("marshal request: %w", err)
+		}
+		respBody, err := client.Post(path, body)
+		if err != nil {
+			return handlers_eks.WebhookTokenReviewResult{}, err
+		}
+		var res handlers_eks.WebhookTokenReviewResult
+		if err := json.Unmarshal(respBody, &res); err != nil {
+			return handlers_eks.WebhookTokenReviewResult{}, fmt.Errorf("decode response: %w", err)
+		}
+		return res, nil
+	}
+}
+
+// authenticator resolves a TokenReview by relaying to the gateway broker.
+type authenticator struct {
+	accountID   string
+	clusterName string
+	review      func(token string) (handlers_eks.WebhookTokenReviewResult, error)
+}
+
+func (a *authenticator) handle(w http.ResponseWriter, r *http.Request) {
+	if r.Method != http.MethodPost {
+		http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
+		return
+	}
+	var review tokenReview
+	if err := json.NewDecoder(r.Body).Decode(&review); err != nil {
+		http.Error(w, "bad request", http.StatusBadRequest)
+		return
+	}
+
+	res, err := a.review(review.Spec.Token)
+	if err != nil {
+		// Broker fault (gateway down, transport error): a 5xx tells the apiserver
+		// to retry rather than treating a transient outage as a hard deny.
+		slog.Error("TokenReview relay failed", "cluster", a.clusterName, "err", err)
+		http.Error(w, "token review unavailable", http.StatusServiceUnavailable)
+		return
+	}
+
+	status := tokenReviewStatus{Authenticated: res.Authenticated}
+	if res.Authenticated {
+		status.User = userInfo{
+			Username: res.Username,
+			UID:      res.UID,
+			Groups:   res.Groups,
+		}
+		// Confirm the token for the audiences the apiserver asked about; an empty
+		// status.audiences is rejected when --api-audiences is configured.
+		status.Audiences = review.Spec.Audiences
+	}
+	slog.Info("TokenReview decision", "authenticated", status.Authenticated, "username", status.User.Username, "audiences", status.Audiences)
+
+	resp := tokenReview{
+		APIVersion: "authentication.k8s.io/v1",
+		Kind:       "TokenReview",
+		Status:     status,
+	}
+	w.Header().Set("Content-Type", "application/json")
+	if err := json.NewEncoder(w).Encode(resp); err != nil {
+		slog.Error("encode TokenReview response", "err", err)
+	}
+}
+
+// newServer builds the loopback TLS server. Split out so tests can construct it
+// without binding a port.
+func newServer(addr string, handler http.Handler, cert tlsCertificate) *http.Server {
+	return &http.Server{
+		Addr:              addr,
+		Handler:           handler,
+		ReadHeaderTimeout: 5 * time.Second,
+		TLSConfig:         cert.serverTLSConfig(),
+	}
+}
@@ -0,0 +1,123 @@
+package main
+
+import (
+	"encoding/base64"
+	"encoding/json"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	handlers_eks "github.com/mulgadc/spinifex/spinifex/handlers/eks"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// postReview drives authenticator.handle with a TokenReview carrying token and
+// audiences, returning the decoded response status and HTTP code.
+func postReview(t *testing.T, a *authenticator, token string, audiences []string) (tokenReviewStatus, int) {
+	t.Helper()
+	body, err := json.Marshal(tokenReview{
+		APIVersion: "authentication.k8s.io/v1",
+		Kind:       "TokenReview",
+		Spec:       tokenReviewSpec{Token: token, Audiences: audiences},
+	})
+	require.NoError(t, err)
+
+	req := httptest.NewRequest(http.MethodPost, "/authenticate", strings.NewReader(string(body)))
+	w := httptest.NewRecorder()
+	a.handle(w, req)
+
+	if w.Code != http.StatusOK {
+		return tokenReviewStatus{}, w.Code
+	}
+	var resp tokenReview
+	require.NoError(t, json.Unmarshal(w.Body.Bytes(), &resp))
+	return resp.Status, w.Code
+}
+
+func TestHandle_GrantsAndStampsAudiences(t *testing.T) {
+	a := &authenticator{
+		accountID:   "111122223333",
+		clusterName: "alpha",
+		review: func(string) (handlers_eks.WebhookTokenReviewResult, error) {
+			return handlers_eks.WebhookTokenReviewResult{
+				Authenticated: true,
+				Username:      "arn:aws:iam::111122223333:role/admin",
+				UID:           "AROAEXAMPLE:session",
+				Groups:        []string{"system:masters"},
+			}, nil
+		},
+	}
+
+	status, code := postReview(t, a, "tok", []string{"https://kubernetes.default.svc"})
+	require.Equal(t, http.StatusOK, code)
+	require.True(t, status.Authenticated)
+	assert.Equal(t, "arn:aws:iam::111122223333:role/admin", status.User.Username)
+	assert.Equal(t, "AROAEXAMPLE:session", status.User.UID)
+	assert.Equal(t, []string{"system:masters"}, status.User.Groups)
+	assert.Equal(t, []string{"https://kubernetes.default.svc"}, status.Audiences)
+}
+
+func TestHandle_DeniesWithoutStampingAudiences(t *testing.T) {
+	a := &authenticator{
+		clusterName: "alpha",
+		review: func(string) (handlers_eks.WebhookTokenReviewResult, error) {
+			return handlers_eks.WebhookTokenReviewResult{Authenticated: false}, nil
+		},
+	}
+
+	status, code := postReview(t, a, "tok", []string{"https://kubernetes.default.svc"})
+	require.Equal(t, http.StatusOK, code)
+	assert.False(t, status.Authenticated)
+	assert.Empty(t, status.User.Username)
+	assert.Empty(t, status.Audiences, "unauthenticated review must not stamp audiences")
+}
+
+// A broker fault is a 5xx (retryable), not a silent deny — a transient gateway
+// outage must not look identical to a rejected token.
+func TestHandle_BrokerFaultReturns5xx(t *testing.T) {
+	a := &authenticator{
+		clusterName: "alpha",
+		review: func(string) (handlers_eks.WebhookTokenReviewResult, error) {
+			return handlers_eks.WebhookTokenReviewResult{}, errors.New("gateway returned 503")
+		},
+	}
+
+	_, code := postReview(t, a, "tok", nil)
+	assert.Equal(t, http.StatusServiceUnavailable, code)
+}
+
+func TestEnsureServingCert_PersistsAndReuses(t *testing.T) {
+	dir := t.TempDir()
+	certPath := filepath.Join(dir, "wh.crt")
+	keyPath := filepath.Join(dir, "wh.key")
+
+	c1, pem1, err := ensureServingCert(certPath, keyPath)
+	require.NoError(t, err)
+	require.NotEmpty(t, pem1)
+	require.NotNil(t, c1.serverTLSConfig())
+
+	// Second call reuses the persisted pair (same cert bytes).
+	_, pem2, err := ensureServingCert(certPath, keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, pem1, pem2)
+}
+
+func TestWriteAPIServerKubeconfig(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "wh.kubeconfig")
+	certPEM := []byte("-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----\n")
+
+	require.NoError(t, writeAPIServerKubeconfig(path, "127.0.0.1:8443", certPEM))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	body := string(data)
+	assert.Contains(t, body, "server: https://127.0.0.1:8443/authenticate")
+	assert.Contains(t, body, base64.StdEncoding.EncodeToString(certPEM))
+	assert.Contains(t, body, "current-context: webhook")
+}
@@ -0,0 +1,34 @@
+package main
+
+// Minimal authentication.k8s.io/v1 TokenReview wire types. Hand-rolled to avoid
+// pulling k8s.io/api into the spinifex module for these few fields; the shapes
+// match what kube-apiserver sends and expects on the webhook contract.
+
+type tokenReview struct {
+	APIVersion string            `json:"apiVersion"`
+	Kind       string            `json:"kind"`
+	Spec       tokenReviewSpec   `json:"spec,omitzero"`
+	Status     tokenReviewStatus `json:"status"`
+}
+
+type tokenReviewSpec struct {
+	Token     string   `json:"token"`
+	Audiences []string `json:"audiences,omitempty"`
+}
+
+type tokenReviewStatus struct {
+	Authenticated bool     `json:"authenticated"`
+	User          userInfo `json:"user,omitzero"`
+	// Audiences echoes the request's spec.audiences on success. When the
+	// apiserver runs with --api-audiences, it sends spec.audiences and discards
+	// an authenticated=true response that does not confirm a matching audience —
+	// so an empty Audiences here surfaces as a 401 despite a valid principal.
+	Audiences []string `json:"audiences,omitempty"`
+	Error     string   `json:"error,omitempty"`
+}
+
+type userInfo struct {
+	Username string   `json:"username,omitempty"`
+	UID      string   `json:"uid,omitempty"`
+	Groups   []string `json:"groups,omitempty"`
+}
@@ -0,0 +1,147 @@
+// Command eks-webhook-cert mints, then reuses, a self-signed ECDSA-P256 serving
+// certificate for an in-cluster admission webhook and prints its base64 PEM so
+// the addon renderer can embed it as the webhook Secret + each caBundle.
+//
+// The eks-node AMI carries no openssl, so this is the Go equivalent of the
+// upstream Helm chart's install-time genSelfSignedCert: one cert per cluster,
+// cached on disk under -dir, stable across addon-sync ticks. The single cert is
+// its own CA (IsCA + ServerAuth), so the same PEM serves as both the TLS leaf
+// and the caBundle the apiserver trusts.
+//
+// Usage:
+//
+//	eks-webhook-cert -dir /var/lib/spinifex-eks/webhook-certs/<addon> \
+//	    -cn <svc>.<ns>.svc -dns <svc>.<ns>.svc,<svc>.<ns>.svc.cluster.local
+//
+// Output (one line): <ca_b64>\t<tls_crt_b64>\t<tls_key_b64>
+package main
+
+import (
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/base64"
+	"encoding/pem"
+	"flag"
+	"fmt"
+	"log/slog"
+	"math/big"
+	"os"
+	"path/filepath"
+	"strings"
+	"time"
+)
+
+func main() {
+	dir := flag.String("dir", "", "directory to cache tls.crt/tls.key in")
+	cn := flag.String("cn", "", "certificate common name")
+	dnsCSV := flag.String("dns", "", "comma-separated DNS SANs")
+	flag.Parse()
+
+	if *dir == "" || *cn == "" || *dnsCSV == "" {
+		fatal("eks-webhook-cert: -dir, -cn and -dns are required")
+	}
+	dnsNames := splitCSV(*dnsCSV)
+	if len(dnsNames) == 0 {
+		fatal("eks-webhook-cert: -dns lists no names")
+	}
+
+	certPEM, keyPEM, err := ensureCert(*dir, *cn, dnsNames)
+	if err != nil {
+		fatal("eks-webhook-cert: " + err.Error())
+	}
+	b := base64.StdEncoding.EncodeToString
+	fmt.Printf("%s\t%s\t%s\n", b(certPEM), b(certPEM), b(keyPEM))
+}
+
+// ensureCert returns the cached cert/key under dir, minting and persisting a
+// fresh self-signed pair (for cn + dnsNames) when none is present or readable.
+func ensureCert(dir, cn string, dnsNames []string) (certPEM, keyPEM []byte, err error) {
+	certPath := filepath.Join(dir, "tls.crt")
+	keyPath := filepath.Join(dir, "tls.key")
+
+	if c, cerr := os.ReadFile(certPath); cerr == nil {
+		if k, kerr := os.ReadFile(keyPath); kerr == nil {
+			if _, perr := x509.ParseCertificate(decodeFirst(c)); perr == nil && len(k) > 0 {
+				return c, k, nil
+			}
+		}
+	}
+
+	certPEM, keyPEM, err = generateSelfSigned(cn, dnsNames)
+	if err != nil {
+		return nil, nil, err
+	}
+	if err = os.MkdirAll(dir, 0o700); err != nil {
+		return nil, nil, fmt.Errorf("mkdir %s: %w", dir, err)
+	}
+	if err = os.WriteFile(certPath, certPEM, 0o644); err != nil {
+		return nil, nil, fmt.Errorf("write %s: %w", certPath, err)
+	}
+	if err = os.WriteFile(keyPath, keyPEM, 0o600); err != nil {
+		return nil, nil, fmt.Errorf("write %s: %w", keyPath, err)
+	}
+	return certPEM, keyPEM, nil
+}
+
+// generateSelfSigned mints an ECDSA-P256 self-signed cert valid as both a CA and
+// a server cert, with the given common name and DNS SANs.
+func generateSelfSigned(cn string, dnsNames []string) (certPEM, keyPEM []byte, err error) {
+	priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		return nil, nil, fmt.Errorf("generate key: %w", err)
+	}
+	serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
+	if err != nil {
+		return nil, nil, fmt.Errorf("serial: %w", err)
+	}
+	now := time.Now().UTC()
+	tmpl := x509.Certificate{
+		SerialNumber:          serial,
+		Subject:               pkix.Name{CommonName: cn},
+		NotBefore:             now.Add(-time.Hour),
+		NotAfter:              now.AddDate(10, 0, 0),
+		KeyUsage:              x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment | x509.KeyUsageCertSign,
+		ExtKeyUsage:           []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+		BasicConstraintsValid: true,
+		IsCA:                  true,
+		DNSNames:              dnsNames,
+	}
+	der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &priv.PublicKey, priv)
+	if err != nil {
+		return nil, nil, fmt.Errorf("create cert: %w", err)
+	}
+	keyDER, err := x509.MarshalECPrivateKey(priv)
+	if err != nil {
+		return nil, nil, fmt.Errorf("marshal key: %w", err)
+	}
+	certPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
+	keyPEM = pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
+	return certPEM, keyPEM, nil
+}
+
+// decodeFirst returns the DER bytes of the first PEM block, or nil.
+func decodeFirst(pemBytes []byte) []byte {
+	blk, _ := pem.Decode(pemBytes)
+	if blk == nil {
+		return nil
+	}
+	return blk.Bytes
+}
+
+func splitCSV(s string) []string {
+	var out []string
+	for p := range strings.SplitSeq(s, ",") {
+		if p = strings.TrimSpace(p); p != "" {
+			out = append(out, p)
+		}
+	}
+	return out
+}
+
+func fatal(msg string) {
+	slog.Error(msg)
+	os.Exit(1)
+}
@@ -0,0 +1,81 @@
+package main
+
+import (
+	"crypto/x509"
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+const (
+	testCN  = "aws-load-balancer-webhook-service.kube-system.svc"
+	testDNS = "aws-load-balancer-webhook-service.kube-system.svc.cluster.local"
+)
+
+func parseLeaf(t *testing.T, certPEM []byte) *x509.Certificate {
+	t.Helper()
+	leaf, err := x509.ParseCertificate(decodeFirst(certPEM))
+	if err != nil {
+		t.Fatalf("parse cert: %v", err)
+	}
+	return leaf
+}
+
+func TestEnsureCertMintsServingCert(t *testing.T) {
+	dir := t.TempDir()
+	certPEM, keyPEM, err := ensureCert(dir, testCN, []string{testCN, testDNS})
+	if err != nil {
+		t.Fatalf("ensureCert: %v", err)
+	}
+	if len(keyPEM) == 0 {
+		t.Fatal("empty key PEM")
+	}
+	leaf := parseLeaf(t, certPEM)
+
+	// Both webhook service DNS names must be present, or the apiserver's TLS
+	// dial to the webhook fails SAN verification.
+	for _, want := range []string{testCN, testDNS} {
+		if err := leaf.VerifyHostname(want); err != nil {
+			t.Errorf("SAN %q missing: %v", want, err)
+		}
+	}
+	if !leaf.IsCA {
+		t.Error("cert must be a CA so the same PEM serves as the webhook caBundle")
+	}
+
+	// The single self-signed cert must verify against itself as a root — this is
+	// exactly what the apiserver does with caBundle == the serving cert.
+	roots := x509.NewCertPool()
+	roots.AddCert(leaf)
+	if _, err := leaf.Verify(x509.VerifyOptions{
+		Roots:     roots,
+		DNSName:   testCN,
+		KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+	}); err != nil {
+		t.Errorf("self-signed cert does not verify as its own CA: %v", err)
+	}
+}
+
+func TestEnsureCertReusesCachedPair(t *testing.T) {
+	dir := t.TempDir()
+	cert1, key1, err := ensureCert(dir, testCN, []string{testCN})
+	if err != nil {
+		t.Fatalf("first ensureCert: %v", err)
+	}
+	cert2, key2, err := ensureCert(dir, testCN, []string{testCN})
+	if err != nil {
+		t.Fatalf("second ensureCert: %v", err)
+	}
+	if string(cert1) != string(cert2) || string(key1) != string(key2) {
+		t.Error("ensureCert regenerated instead of reusing the cached pair; a churning cert re-applies the addon every tick")
+	}
+
+	// The private key must not be world-readable on disk.
+	info, err := os.Stat(filepath.Join(dir, "tls.key"))
+	if err != nil {
+		t.Fatalf("stat key: %v", err)
+	}
+	if info.Mode().Perm()&0o077 != 0 {
+		t.Errorf("tls.key mode %o is group/other-accessible", info.Mode().Perm())
+	}
+}
@@ -115,6 +115,16 @@ Each phase waits for all nodes to ACK before proceeding to the next.`,
 	Run: runClusterShutdown,
 }

+var clusterDrainDHCPCmd = &cobra.Command{
+	Use:   "drain-dhcp",
+	Short: "Release all upstream DHCP leases held by vpcd",
+	Long: `Ask each vpcd to DHCPRELEASE every external-pool DHCP lease it currently
+holds, returning them to the upstream DHCP server. Run this on teardown before
+stopping services: an env reset otherwise strands held leases on the upstream
+server until their TTL expires, eventually exhausting the upstream scope.`,
+	Run: runClusterDrainDHCP,
+}
+
 var adminInitCmd = &cobra.Command{
 	Use:   "init",
 	Short: "Initialize Spinifex platform configuration",
@@ -245,6 +255,9 @@ func init() {
 	clusterShutdownCmd.Flags().Duration("timeout", 120*time.Second, "Maximum time to wait per phase")
 	clusterShutdownCmd.Flags().Bool("dry-run", false, "Print phase plan without executing")

+	clusterCmd.AddCommand(clusterDrainDHCPCmd)
+	clusterDrainDHCPCmd.Flags().Duration("timeout", 30*time.Second, "Reply-collection window for vpcd drain responders")
+
 	adminCmd.AddCommand(imagesCmd)
 	imagesCmd.AddCommand(imagesImportCmd)
 	imagesCmd.AddCommand(imagesListCmd)
@@ -285,6 +298,7 @@ func init() {
 	adminInitCmd.Flags().String("predastore-nodes", "", "Comma-separated IPs for multi-node Predastore cluster (e.g., 10.11.12.1,10.11.12.2,10.11.12.3). Requires >= 3 nodes.")
 	adminInitCmd.Flags().String("formation-timeout", "10m", "Timeout for cluster formation (e.g., 5m, 30s)")
 	adminInitCmd.Flags().String("token-ttl", "30m", "Join token validity duration (e.g. 30m, 1h, 2h)")
+	adminInitCmd.Flags().Int("predastore-compaction-interval", 0, "Predastore compactor interval in seconds (0 = unset, uses built-in default). Test clusters set a short interval.")
 	adminInitCmd.Flags().String("cluster-name", "spinifex", "NATS cluster name")
 	adminInitCmd.Flags().Bool("no-telemetry", false, "Disable telemetry metrics sent during init (default: enabled)")
 	adminInitCmd.Flags().String("email", "", "Operator email address (used for update and security notifications)")
@@ -318,6 +332,7 @@ func init() {
 	adminJoinCmd.Flags().StringSlice("services", nil, "Services this node runs (default: all)")
 	adminJoinCmd.Flags().Bool("no-telemetry", false, "Disable telemetry metrics sent during join (default: enabled)")
 	adminJoinCmd.Flags().String("email", "", "Operator email address (used for update and security notifications)")
+	adminJoinCmd.Flags().Int("predastore-compaction-interval", 0, "Predastore compactor interval in seconds (0 = unset, uses built-in default). Test clusters set a short interval.")
 	adminJoinCmd.MarkFlagRequired("node")
 	adminJoinCmd.MarkFlagRequired("host")
 	adminJoinCmd.MarkFlagRequired("token")
@@ -325,6 +340,7 @@ func init() {
 	imagesImportCmd.Flags().String("tmp-dir", os.TempDir(), "Temporary directory for image import processing")

 	imagesImportCmd.Flags().String("name", "", "Import specified image by name")
+	imagesImportCmd.Flags().String("ami-name", "", "Override the registered AMI name (DescribeImages name). Defaults to ami-{distro}-{version}-{arch}. Use for locally-built appliances (e.g. --ami-name spinifex-eks-node).")
 	imagesImportCmd.Flags().String("file", "", "Import file from specified path (raw, qcow2, compressed)")
 	imagesImportCmd.Flags().String("distro", "", "Specified distro name (e.g debian)")
 	imagesImportCmd.Flags().String("version", "", "Specified distro version (e.g 12)")
@@ -380,6 +396,7 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	// and the file is used directly (no download). When only --file is set,
 	// metadata comes from flags and no catalog tags are applied.
 	imageName, _ := cmd.Flags().GetString("name")
+	amiNameOverride, _ := cmd.Flags().GetString("ami-name")
 	localFile, _ := cmd.Flags().GetString("file")

 	if imageName == "" && localFile == "" {
@@ -535,6 +552,9 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	// Calculate the size

 	manifest.AMIMetadata.Name = fmt.Sprintf("ami-%s-%s-%s", image.Distro, image.Version, image.Arch)
+	if amiNameOverride != "" {
+		manifest.AMIMetadata.Name = amiNameOverride
+	}
 	volumeId := utils.GenerateResourceID("ami")
 	manifest.AMIMetadata.ImageID = volumeId

@@ -828,6 +848,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	predastoreNodesStr, _ := cmd.Flags().GetString("predastore-nodes")
 	formationTimeoutStr, _ := cmd.Flags().GetString("formation-timeout")
 	tokenTTLStr, _ := cmd.Flags().GetString("token-ttl")
+	compactionInterval, _ := cmd.Flags().GetInt("predastore-compaction-interval")
 	clusterName, _ := cmd.Flags().GetString("cluster-name")
 	services, _ := cmd.Flags().GetStringSlice("services")

@@ -1109,6 +1130,17 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	fmt.Println("\n🔐 Generated predastore encryption key (per-node, never transmitted)")
 	fmt.Printf("   Key: %s\n", predastoreKeyPath)

+	// Viperblock at-rest encryption key is cluster-wide; on a single-node init
+	// there are no joiners, so generate it locally and enable encryption by
+	// default for all volumes created on this install.
+	viperblockKeyPath, err := writeViperblockEncryptionKey(configDir)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error generating viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	fmt.Println("\n🔐 Generated viperblock at-rest encryption key")
+	fmt.Printf("   Key: %s\n", viperblockKeyPath)
+
 	fmt.Printf("\n🔑 Generated admin credentials (save these — they won't be shown again):\n")
 	fmt.Printf("   Access Key:  %s\n", bootstrapResult.AdminAccessKey)
 	fmt.Printf("   Secret Key:  %s\n", bootstrapResult.AdminSecretKey)
@@ -1116,7 +1148,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	fmt.Printf("   AWS Profile: spinifex\n")

 	// Generate SSL certificates (with bind IP in SANs for multi-node support)
-	certPath := admin.GenerateCertificatesIfNeeded(configDir, force, bindIP)
+	certPath := admin.GenerateCertificatesIfNeeded(configDir, force, bindIP, region, config.DefaultAWSInternalSuffix)

 	// Generate per-node IPsec peer cert when cluster-wide IPsec is enabled
 	// (default true). Reuses the cluster CA — no intermediate strongSwan PKI.
@@ -1227,7 +1259,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		}

 		// Generate multi-node predastore.toml
-		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP)
+		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP, compactionInterval)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Error generating multi-node predastore config: %v\n", err)
 			os.Exit(1)
@@ -1268,8 +1300,9 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		ClusterRoutes: clusterRoutes,
 		ClusterName:   clusterName,

-		PredastoreNodeID: predastoreNodeID,
-		Services:         services,
+		PredastoreNodeID:          predastoreNodeID,
+		CompactionIntervalSeconds: compactionInterval,
+		Services:                  services,

 		OVNNBAddr: "tcp:127.0.0.1:6641",
 		OVNSBAddr: "tcp:127.0.0.1:6642",
@@ -1296,6 +1329,8 @@ func runAdminInit(cmd *cobra.Command, args []string) {

 		GPUPassthrough: gpuPassthrough,
 		IPSecEnabled:   ipsecEnabled,
+
+		EncryptionKeyFile: viperblockKeyPath,
 	}

 	// Print external networking summary
@@ -1372,6 +1407,8 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 		os.Exit(1)
 	}

+	compactionInterval, _ := cmd.Flags().GetInt("predastore-compaction-interval")
+
 	joinToken, err := formation.GenerateJoinToken()
 	if err != nil {
 		fmt.Fprintf(os.Stderr, "❌ Error generating join token: %v\n", err)
@@ -1404,6 +1441,22 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	fmt.Println("\n🔐 Generated predastore encryption key (per-node, never transmitted)")
 	fmt.Printf("   Key: %s\n", predastoreKeyPath)

+	// Viperblock at-rest encryption key is cluster-wide and shared: the leader
+	// generates it once and distributes it to joiners via the formation server
+	// so a volume sealed on any node can be opened on any other.
+	viperblockKey, err := handlers_iam.GenerateMasterKey()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "❌ Error generating viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	viperblockKeyPath, err := saveViperblockEncryptionKey(configDir, viperblockKey)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "❌ Error saving viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	fmt.Println("\n🔐 Generated viperblock at-rest encryption key (cluster-wide, shared with joiners)")
+	fmt.Printf("   Key: %s\n", viperblockKeyPath)
+
 	// Read CA cert/key for distribution to joining nodes
 	caCertData, err := os.ReadFile(filepath.Join(configDir, "ca.pem"))
 	if err != nil {
@@ -1432,6 +1485,9 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	// Include master key in formation server for distribution to joining nodes
 	fs.SetMasterKey(base64.StdEncoding.EncodeToString(masterKey))

+	// Distribute the cluster-wide viperblock encryption key to joiners
+	fs.SetViperblockKey(base64.StdEncoding.EncodeToString(viperblockKey))
+
 	// Register self (init node) as the first node
 	selfNode := formation.NodeInfo{
 		Name:        node,
@@ -1496,7 +1552,7 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	var predastoreNodeID int
 	hasPredastoreConfig := len(predastoreNodes) >= 2
 	if hasPredastoreConfig {
-		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP)
+		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP, compactionInterval)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Error generating multi-node predastore config: %v\n", err)
 			os.Exit(1)
@@ -1533,12 +1589,15 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 		ClusterRoutes: clusterRoutes,
 		ClusterName:   clusterName,

-		PredastoreNodeID: predastoreNodeID,
-		Services:         services,
-		RemoteNodes:      buildRemoteNodes(allNodes, node),
+		PredastoreNodeID:          predastoreNodeID,
+		CompactionIntervalSeconds: compactionInterval,
+		Services:                  services,
+		RemoteNodes:               buildRemoteNodes(allNodes, node),

 		OperatorEmail: email,

+		EncryptionKeyFile: viperblockKeyPath,
+
 		// Init node runs ovn-central locally
 		OVNNBAddr: "tcp:127.0.0.1:6641",
 		OVNSBAddr: "tcp:127.0.0.1:6642",
@@ -1580,7 +1639,7 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	}
 	fmt.Println("\n📋 Next steps:")
 	fmt.Println("   1. Start services on ALL nodes:")
-	fmt.Println("      ./scripts/start-dev.sh")
+	fmt.Println("      sudo systemctl start spinifex.target")
 	fmt.Println()
 }

@@ -1603,6 +1662,7 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 	configDir, _ := cmd.Flags().GetString("config-dir")
 	clusterBind, _ := cmd.Flags().GetString("cluster-bind")
 	services, _ := cmd.Flags().GetStringSlice("services")
+	compactionInterval, _ := cmd.Flags().GetInt("predastore-compaction-interval")

 	email, _ := cmd.Flags().GetString("email")
 	email = strings.TrimSpace(email)
@@ -1884,8 +1944,30 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 	}
 	fmt.Printf("✅ Predastore encryption key generated: %s\n", predastoreKeyPath)

+	// Viperblock at-rest encryption key is cluster-wide: receive it from the
+	// leader rather than generating one, so this node can open volumes sealed
+	// elsewhere. Lenient on absence — an older leader that predates key
+	// distribution returns nothing; fall back to cleartext rather than fail
+	// the join.
+	var viperblockKeyPath string
+	if statusResp.ViperblockKey == "" {
+		fmt.Println("⚠️  Leader did not provide a viperblock encryption key; at-rest encryption disabled on this node")
+	} else {
+		viperblockKeyBytes, err := base64.StdEncoding.DecodeString(statusResp.ViperblockKey)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "❌ Error decoding viperblock encryption key: %v\n", err)
+			os.Exit(1)
+		}
+		viperblockKeyPath, err = saveViperblockEncryptionKey(configDir, viperblockKeyBytes)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "❌ Error saving viperblock encryption key: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Printf("✅ Viperblock encryption key received from leader: %s\n", viperblockKeyPath)
+	}
+
 	// Generate server cert signed by CA with this node's bind IP
-	if err := admin.GenerateServerCertOnly(configDir, bindIP); err != nil {
+	if err := admin.GenerateServerCertOnly(configDir, bindIP, region, config.DefaultAWSInternalSuffix); err != nil {
 		fmt.Fprintf(os.Stderr, "Error generating server certificate: %v\n", err)
 		os.Exit(1)
 	}
@@ -1928,7 +2010,7 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 	hasPredastoreConfig := len(predastoreNodes) >= 2

 	if hasPredastoreConfig {
-		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, creds.AccessKey, creds.SecretKey, creds.Region, creds.NatsToken, configDir, bindIP)
+		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, creds.AccessKey, creds.SecretKey, creds.Region, creds.NatsToken, configDir, bindIP, compactionInterval)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Error generating multi-node predastore config: %v\n", err)
 			os.Exit(1)
@@ -1969,12 +2051,15 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 		ClusterRoutes: clusterRoutes,
 		ClusterName:   creds.ClusterName,

-		PredastoreNodeID: predastoreNodeID,
-		Services:         services,
-		RemoteNodes:      buildRemoteNodes(statusResp.Nodes, node),
+		PredastoreNodeID:          predastoreNodeID,
+		CompactionIntervalSeconds: compactionInterval,
+		Services:                  services,
+		RemoteNodes:               buildRemoteNodes(statusResp.Nodes, node),

 		OperatorEmail: email,

+		EncryptionKeyFile: viperblockKeyPath,
+
 		// Joining nodes connect to the init node's OVN NB/SB DB
 		OVNNBAddr: fmt.Sprintf("tcp:%s:6641", leaderIP),
 		OVNSBAddr: fmt.Sprintf("tcp:%s:6642", leaderIP),
@@ -2260,6 +2345,7 @@ func runCertRenew(cmd *cobra.Command, _ []string) {
 type configDirs struct {
 	AWSGW      string
 	Predastore string
+	Viperblock string
 	NATS       string
 	Spinifex   string
 }
@@ -2269,10 +2355,11 @@ func createConfigSubdirs(configDir string) (configDirs, error) {
 	dirs := configDirs{
 		AWSGW:      filepath.Join(configDir, "awsgw"),
 		Predastore: filepath.Join(configDir, "predastore"),
+		Viperblock: filepath.Join(configDir, "viperblock"),
 		NATS:       filepath.Join(configDir, "nats"),
 		Spinifex:   filepath.Join(configDir, "spinifex"),
 	}
-	for _, dir := range []string{dirs.AWSGW, dirs.Predastore, dirs.NATS, dirs.Spinifex} {
+	for _, dir := range []string{dirs.AWSGW, dirs.Predastore, dirs.Viperblock, dirs.NATS, dirs.Spinifex} {
 		if err := os.MkdirAll(dir, 0700); err != nil {
 			return configDirs{}, fmt.Errorf("create directory %s: %w", dir, err)
 		}
@@ -2394,6 +2481,43 @@ func writePredastoreEncryptionKey(configDir string) (string, error) {
 	return keyPath, nil
 }

+// writeViperblockEncryptionKey generates a fresh 32-byte AES-256 master key for
+// viperblock at-rest encryption and writes it to
+// <configDir>/viperblock/encryption.key with mode 0600. Unlike the predastore
+// key, this key is cluster-wide and shared: the leader generates it once at init
+// and distributes it to joiners via the formation server, so a volume sealed on
+// any node can be opened on any other. Loaded at startup via masterkey.LoadShared.
+func writeViperblockEncryptionKey(configDir string) (string, error) {
+	viperblockDir := filepath.Join(configDir, "viperblock")
+	if err := os.MkdirAll(viperblockDir, 0750); err != nil {
+		return "", fmt.Errorf("create viperblock config dir: %w", err)
+	}
+	key, err := handlers_iam.GenerateMasterKey()
+	if err != nil {
+		return "", fmt.Errorf("generate viperblock encryption key: %w", err)
+	}
+	keyPath := filepath.Join(viperblockDir, "encryption.key")
+	if err := handlers_iam.SaveMasterKey(keyPath, key); err != nil {
+		return "", fmt.Errorf("save viperblock encryption key: %w", err)
+	}
+	return keyPath, nil
+}
+
+// saveViperblockEncryptionKey writes an already-generated 32-byte viperblock
+// master key (received from the formation leader) to
+// <configDir>/viperblock/encryption.key with mode 0600.
+func saveViperblockEncryptionKey(configDir string, key []byte) (string, error) {
+	viperblockDir := filepath.Join(configDir, "viperblock")
+	if err := os.MkdirAll(viperblockDir, 0750); err != nil {
+		return "", fmt.Errorf("create viperblock config dir: %w", err)
+	}
+	keyPath := filepath.Join(viperblockDir, "encryption.key")
+	if err := handlers_iam.SaveMasterKey(keyPath, key); err != nil {
+		return "", fmt.Errorf("save viperblock encryption key: %w", err)
+	}
+	return keyPath, nil
+}
+
 // writeSystemCredentials writes the system access key to a plaintext JSON file.
 // The daemon reads this at startup to inject credentials into ALB VM cloud-init
 // for SigV4-authenticated communication with the AWS gateway.
@@ -6,11 +6,15 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"strings"
 	"testing"

+	s3 "github.com/mulgadc/predastore/s3"
 	"github.com/mulgadc/spinifex/spinifex/admin"
 	"github.com/mulgadc/spinifex/spinifex/formation"
 	"github.com/mulgadc/spinifex/spinifex/utils"
+	toml "github.com/pelletier/go-toml/v2"
+	"github.com/spf13/cobra"
 	"github.com/spf13/viper"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -206,7 +210,7 @@ func TestPredastoreMultinodeTemplate_NatsURLLoopbackShim(t *testing.T) {
 		{ID: 3, Host: "10.0.0.3"},
 	}
 	content, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "0.0.0.0",
+		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "0.0.0.0", 0,
 	)
 	require.NoError(t, err)
 	assert.Contains(t, content, `nats_url = "nats://localhost:4222"`)
@@ -214,7 +218,7 @@ func TestPredastoreMultinodeTemplate_NatsURLLoopbackShim(t *testing.T) {

 	// Specific bind IP → stays as-is.
 	content2, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "10.11.12.1",
+		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "10.11.12.1", 0,
 	)
 	require.NoError(t, err)
 	assert.Contains(t, content2, `nats_url = "nats://10.11.12.1:4222"`)
@@ -252,6 +256,74 @@ func TestWritePredastoreEncryptionKey(t *testing.T) {
 	assert.NotEqual(t, key1, key2, "per-node keys must differ")
 }

+// The viperblock key gates at-rest encryption for every volume on a fresh
+// install. masterkey.LoadShared rejects anything that isn't exactly 32 bytes at
+// 0640-or-stricter, so a regression in size/mode would silently fall back to
+// cleartext (empty path) or fail service startup.
+func TestWriteViperblockEncryptionKey(t *testing.T) {
+	configDir := t.TempDir()
+
+	keyPath, err := writeViperblockEncryptionKey(configDir)
+	require.NoError(t, err)
+	assert.Equal(t, filepath.Join(configDir, "viperblock", "encryption.key"), keyPath)
+
+	info, err := os.Stat(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, int64(32), info.Size(), "viperblock master key must be exactly 32 bytes")
+	assert.Equal(t, os.FileMode(0600), info.Mode().Perm(), "viperblock master key must be mode 0600")
+}
+
+// Joiners persist the leader's shared key via saveViperblockEncryptionKey
+// against a configDir with no viperblock/ subdir yet. The bytes round-trip
+// verbatim — the whole cluster shares one key, so any mutation would orphan
+// volumes sealed on other nodes.
+func TestSaveViperblockEncryptionKey_RoundTrip(t *testing.T) {
+	configDir := t.TempDir()
+	key := make([]byte, 32)
+	for i := range key {
+		key[i] = byte(i)
+	}
+
+	keyPath, err := saveViperblockEncryptionKey(configDir, key)
+	require.NoError(t, err)
+	assert.Equal(t, filepath.Join(configDir, "viperblock", "encryption.key"), keyPath)
+
+	got, err := os.ReadFile(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, key, got, "shared key must be written verbatim")
+}
+
+// A fresh install must render encryption_key_file so viperblockd enables
+// at-rest encryption; an empty field (upgrade / legacy) must omit the line
+// entirely so the volume stays in cleartext legacy mode.
+func TestSpinifexTomlTemplate_EncryptionKeyFile(t *testing.T) {
+	base := admin.ConfigSettings{
+		Node: "node1", Az: "ap-southeast-2a", Port: "4432", Region: "ap-southeast-2",
+		BindIP: "0.0.0.0", AccessKey: "AKIATEST", SecretKey: "SECRET",
+		AccountID: "123456789012", NatsToken: "token",
+		OVNNBAddr: "tcp:127.0.0.1:6641", OVNSBAddr: "tcp:127.0.0.1:6642",
+	}
+
+	dir := t.TempDir()
+	withKey := base
+	withKey.ConfigDir = dir
+	withKey.EncryptionKeyFile = "/etc/spinifex/viperblock/encryption.key"
+	path := filepath.Join(dir, "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, withKey))
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	assert.Contains(t, string(data), `encryption_key_file = "/etc/spinifex/viperblock/encryption.key"`)
+
+	dir2 := t.TempDir()
+	noKey := base
+	noKey.ConfigDir = dir2
+	path2 := filepath.Join(dir2, "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path2, spinifexTomlTemplate, noKey))
+	data2, err := os.ReadFile(path2)
+	require.NoError(t, err)
+	assert.NotContains(t, string(data2), "encryption_key_file", "blank key must omit the field")
+}
+
 // Joiners run `spx admin join` against a configDir that doesn't yet contain
 // a predastore/ subdir. The helper must create it (predastore subdir is
 // owned by spinifex-storage in prod, but the helper just needs the dir to
@@ -386,3 +458,81 @@ func TestApplyNetworkConfig_PropagatesPoolBindBridge(t *testing.T) {
 	assert.Equal(t, "dhcp", settings.PoolSource)
 	assert.Equal(t, "br-wan", settings.PoolBindBridge)
 }
+
+func renderSingleNodePredastore(t *testing.T, settings admin.ConfigSettings) string {
+	t.Helper()
+	path := filepath.Join(t.TempDir(), "predastore.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, predastoreTomlTemplate, settings))
+	b, err := os.ReadFile(path)
+	require.NoError(t, err)
+	return string(b)
+}
+
+func predastoreMultinodeNodes() []admin.PredastoreNodeConfig {
+	return []admin.PredastoreNodeConfig{
+		{ID: 1, Host: "10.0.0.1"}, {ID: 2, Host: "10.0.0.2"}, {ID: 3, Host: "10.0.0.3"},
+	}
+}
+
+// Unset knob must leave production config byte-identical: no [compaction] block
+// and the original trailing bytes unchanged (single-node has no trailing
+// newline, multinode keeps one).
+func TestPredastoreTemplates_UnsetCompactionEmitsNoBlock(t *testing.T) {
+	single := renderSingleNodePredastore(t, admin.ConfigSettings{
+		Region: "ap-southeast-2", BindIP: "0.0.0.0", ConfigDir: "/cfg",
+		AccessKey: "AK", SecretKey: "SK", NatsToken: "tok",
+	})
+	assert.NotContains(t, single, "[compaction]")
+	assert.True(t, strings.HasSuffix(single, `access_keys_bucket = "spinifex-iam-access-keys"`),
+		"unset single-node tail changed: %q", single[len(single)-40:])
+
+	multi, err := admin.GenerateMultiNodePredastoreConfig(
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0,
+	)
+	require.NoError(t, err)
+	assert.NotContains(t, multi, "[compaction]")
+	assert.True(t, strings.HasSuffix(multi, "access_keys_bucket = \"spinifex-iam-access-keys\"\n"),
+		"unset multinode tail changed: %q", multi[len(multi)-40:])
+}
+
+func TestPredastoreTemplates_SetCompactionEmitsParseableBlock(t *testing.T) {
+	const interval = 30
+
+	single := renderSingleNodePredastore(t, admin.ConfigSettings{
+		Region: "ap-southeast-2", BindIP: "0.0.0.0", ConfigDir: "/cfg",
+		AccessKey: "AK", SecretKey: "SK", NatsToken: "tok", CompactionIntervalSeconds: interval,
+	})
+	assert.Contains(t, single, "[compaction]")
+	var singleCfg s3.Config
+	require.NoError(t, toml.Unmarshal([]byte(single), &singleCfg))
+	assert.Equal(t, interval, singleCfg.Compaction.IntervalSeconds)
+
+	multi, err := admin.GenerateMultiNodePredastoreConfig(
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", interval,
+	)
+	require.NoError(t, err)
+	var multiCfg s3.Config
+	require.NoError(t, toml.Unmarshal([]byte(multi), &multiCfg))
+	assert.Equal(t, interval, multiCfg.Compaction.IntervalSeconds)
+}
+
+// The init/join flag must register with an unset (0) default and flow into the
+// rendered config via ConfigSettings.
+func TestAdminInitFlag_CompactionIntervalReachesRenderedConfig(t *testing.T) {
+	for _, c := range []*cobra.Command{adminInitCmd, adminJoinCmd} {
+		f := c.Flags().Lookup("predastore-compaction-interval")
+		require.NotNil(t, f, "%s missing --predastore-compaction-interval", c.Name())
+		assert.Equal(t, "0", f.DefValue)
+	}
+
+	require.NoError(t, adminInitCmd.Flags().Set("predastore-compaction-interval", "45"))
+	defer func() { _ = adminInitCmd.Flags().Set("predastore-compaction-interval", "0") }()
+
+	v, _ := adminInitCmd.Flags().GetInt("predastore-compaction-interval")
+	require.Equal(t, 45, v)
+
+	out := renderSingleNodePredastore(t, admin.ConfigSettings{
+		Region: "ap-southeast-2", BindIP: "0.0.0.0", CompactionIntervalSeconds: v,
+	})
+	assert.Contains(t, out, "interval_seconds = 45")
+}
@@ -8,6 +8,7 @@ import (
 	"time"

 	"github.com/mulgadc/spinifex/spinifex/daemon"
+	"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
 	"github.com/nats-io/nats.go"
 	"github.com/spf13/cobra"
 )
@@ -160,6 +161,79 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
 	fmt.Printf("Cluster shutdown complete (%s)\n", time.Since(start).Round(time.Millisecond))
 }

+// runClusterDrainDHCP asks every vpcd to DHCPRELEASE all external-pool leases
+// it currently holds, returning them to the upstream DHCP server. Intended for
+// the teardown path: an env reset otherwise strands held leases until TTL
+// because vpcd's Stop() preserves them for adopt-on-reboot. Best-effort — if
+// the cluster is already partly down it warns and exits 0 so teardown proceeds.
+func runClusterDrainDHCP(cmd *cobra.Command, args []string) {
+	timeout, _ := cmd.Flags().GetDuration("timeout")
+	if timeout <= 0 {
+		timeout = 30 * time.Second
+	}
+
+	_, nc, err := loadConfigAndConnect()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: drain-dhcp could not connect (cluster may be down): %v\n", err)
+		return
+	}
+	defer nc.Close()
+
+	total, responders := drainDHCPLeases(nc, timeout)
+	fmt.Printf("DHCP drain complete: released %d lease(s) from %d vpcd responder(s)\n", total, responders)
+}
+
+// drainDHCPLeases publishes a drain request to vpc.dhcp.drain and collects
+// replies until timeout — one responder per AZ (vpcd uses a per-AZ queue
+// group). Returns the summed released count and the number of responders.
+func drainDHCPLeases(nc *nats.Conn, timeout time.Duration) (released, responders int) {
+	reqData, err := json.Marshal(struct{}{})
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		return 0, 0
+	}
+
+	inbox := nats.NewInbox()
+	sub, err := nc.SubscribeSync(inbox)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: drain-dhcp subscribe failed: %v\n", err)
+		return 0, 0
+	}
+	defer sub.Unsubscribe()
+
+	if err := nc.PublishRequest(dhcp.TopicDrain, inbox, reqData); err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: drain-dhcp publish failed: %v\n", err)
+		return 0, 0
+	}
+	nc.Flush()
+
+	deadline := time.Now().Add(timeout)
+	for {
+		remaining := time.Until(deadline)
+		if remaining <= 0 {
+			break
+		}
+		msg, err := sub.NextMsg(remaining)
+		if err != nil {
+			break
+		}
+		var reply struct {
+			Released int    `json:"released"`
+			Error    string `json:"error,omitempty"`
+		}
+		if err := json.Unmarshal(msg.Data, &reply); err != nil {
+			continue
+		}
+		responders++
+		if reply.Error != "" {
+			fmt.Fprintf(os.Stderr, "  vpcd drain error: %s\n", reply.Error)
+			continue
+		}
+		released += reply.Released
+	}
+	return released, responders
+}
+
 // collectShutdownACKs publishes a shutdown request and collects ACKs from nodes.
 func collectShutdownACKs(nc *nats.Conn, topic string, reqData []byte, nodeCount int, timeout time.Duration) ([]daemon.ShutdownACK, error) {
 	inbox := nats.NewInbox()
@@ -0,0 +1,58 @@
+package cmd
+
+import (
+	"encoding/json"
+	"testing"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
+	"github.com/mulgadc/spinifex/spinifex/testutil"
+	"github.com/nats-io/nats.go"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// fakeDrainResponder subscribes a stand-in vpcd to vpc.dhcp.drain that
+// replies with the given released count (or error), mirroring
+// Manager.handleDrainMsg without a real lease store.
+func fakeDrainResponder(t *testing.T, nc *nats.Conn, released int, errMsg string) {
+	t.Helper()
+	sub, err := nc.Subscribe(dhcp.TopicDrain, func(msg *nats.Msg) {
+		body, _ := json.Marshal(struct {
+			Released int    `json:"released"`
+			Error    string `json:"error,omitempty"`
+		}{Released: released, Error: errMsg})
+		_ = msg.Respond(body)
+	})
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = sub.Unsubscribe() })
+}
+
+func TestDrainDHCPLeasesSumsReleased(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+	fakeDrainResponder(t, nc, 7, "")
+
+	released, responders := drainDHCPLeases(nc, 2*time.Second)
+	assert.Equal(t, 7, released)
+	assert.Equal(t, 1, responders)
+}
+
+func TestDrainDHCPLeasesReportsResponderError(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+	fakeDrainResponder(t, nc, 0, "list leases for drain: boom")
+
+	// An erroring responder still counts as a responder but contributes
+	// zero released leases.
+	released, responders := drainDHCPLeases(nc, 2*time.Second)
+	assert.Equal(t, 0, released)
+	assert.Equal(t, 1, responders)
+}
+
+func TestDrainDHCPLeasesNoResponders(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+
+	// No vpcd subscribed: the collection window elapses with zero replies.
+	released, responders := drainDHCPLeases(nc, 200*time.Millisecond)
+	assert.Equal(t, 0, released)
+	assert.Equal(t, 0, responders)
+}
@@ -538,7 +538,7 @@ var awsgwStartCmd = &cobra.Command{
 		awsgwHost := viper.GetString("host")
 		if awsgwHost != "" {
 			fmt.Println("Overwriting awsgw host to:", awsgwHost)
-			//nodeConfig.AWSGW.Host = awsgwHost
+			nodeConfig.AWSGW.Host = awsgwHost
 		}

 		awsgwTlsCert := viper.GetString("tls-cert")
@@ -75,3 +75,8 @@ nats_token = "{{.NatsToken}}"
 nats_ca_cert = "{{.ConfigDir}}/ca.pem"
 master_key_path = "{{.ConfigDir}}/master.key"
 access_keys_bucket = "spinifex-iam-access-keys"
+{{- if .CompactionIntervalSeconds}}
+
+[compaction]
+interval_seconds = {{.CompactionIntervalSeconds}}
+{{- end}}
@@ -103,4 +103,9 @@ nats_url = "nats://{{if eq .BindIP "0.0.0.0"}}localhost{{else}}{{.BindIP}}{{end}
 nats_token = "{{.NatsToken}}"
 nats_ca_cert = "{{.ConfigDir}}/ca.pem"
 master_key_path = "{{.ConfigDir}}/master.key"
-access_keys_bucket = "spinifex-iam-access-keys"
+access_keys_bucket = "spinifex-iam-access-keys"
+{{- if .CompactionIntervalSeconds}}
+
+[compaction]
+interval_seconds = {{.CompactionIntervalSeconds}}
+{{- end}}
@@ -8,6 +8,10 @@ node = "{{.Node}}"
 email = "{{.OperatorEmail}}"
 {{- end}}

+[aws]
+region = "{{.Region}}"
+internal_suffix = "spinifex.internal"
+
 [network]
 ipsec_enabled = {{.IPSecEnabled}}
 {{- if .ExternalMode}}
@@ -93,6 +97,9 @@ token = "{{.NatsToken}}"

 [nodes.{{.Node}}.viperblock]
 shardwal = false
+{{- if .EncryptionKeyFile}}
+encryption_key_file = "{{.EncryptionKeyFile}}"
+{{- end}}

 [nodes.{{.Node}}.vpcd]
 ovn_nb_addr = "{{.OVNNBAddr}}"
@@ -8,162 +8,103 @@ Platform management commands not exposed via the AWS gateway API. These are CLI-

 Service lifecycle commands for starting, stopping, and checking status of all Spinifex cluster services. Each service subcommand supports `start`, `stop`, and `status` operations.

-| Command | Flags | Prerequisites | Basic Logic | Status |
-|---------|-------|---------------|-------------|--------|
-| `spx service predastore start` | `--port` (default: 8443), `--host` (default: 0.0.0.0), `--base-path`, `--config-path`, `--debug`, `--tls-cert`, `--tls-key`, `--backend` (distributed/filesystem, default: distributed), `--node-id` (default: 0), `--pprof`, `--pprof-output` | TLS cert/key, base-path, config-path required | Creates predastore service instance with S3-compatible storage backend → starts service | **DONE** |
-| `spx service predastore stop` | — | Predastore must be running | Stops the predastore service | **DONE** |
-| `spx service predastore status` | — | None | Reports predastore service status | **DONE** |
-| `spx service viperblock start` | `--s3-host` (default: 0.0.0.0:8443), `--s3-bucket` (default: predastore), `--s3-region` (default: ap-southeast-2), `--plugin-path` (auto-detected via `nbdkit --dump-config plugindir`; typically `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` on amd64, overridable via `SPINIFEX_VIPERBLOCK_PLUGIN_PATH` in `/etc/spinifex/systemd.env`) | `--config` required, plugin-path must exist on disk | Loads cluster config → connects to NATS and Predastore → starts viperblock block storage service with NBD plugin | **DONE** |
-| `spx service viperblock stop` | — | Viperblock must be running | Stops the viperblock service | **DONE** |
-| `spx service viperblock status` | — | None | Reports viperblock service status | **DONE** |
-| `spx service nats start` | `--port` (default: 4222), `--host` (default: 0.0.0.0), `--debug`, `--data-dir`, `--jetstream` | None | Starts embedded NATS server with optional JetStream | **DONE** |
-| `spx service nats stop` | — | NATS must be running | Stops the NATS service | **DONE** |
-| `spx service nats status` | — | None | Reports NATS service status | **DONE** |
-| `spx service spinifex start` | `--wal-dir` | `--config` required | Loads cluster config → starts spinifex daemon (VM orchestration, NATS subscriptions, health endpoint) | **DONE** |
-| `spx service spinifex stop` | — | Spinifex daemon must be running | Stops the spinifex daemon service | **DONE** |
-| `spx service spinifex status` | — | None | Reports spinifex daemon service status | **DONE** |
-| `spx service awsgw start` | `--host` (default: 0.0.0.0:9999), `--tls-cert`, `--tls-key`, `--debug` | `--config` required | Loads cluster config → starts AWS-compatible gateway with SigV4 auth, IAM policy enforcement, TLS | **DONE** |
-| `spx service awsgw stop` | — | AWS gateway must be running | Stops the AWS gateway service | **DONE** |
-| `spx service awsgw status` | — | None | Reports AWS gateway service status | **DONE** |
-| `spx service spinifex-ui start` | `--port` (default: 3000), `--host` (default: 0.0.0.0), `--tls-cert`, `--tls-key` | None | Starts embedded web UI server serving the React frontend. Aliases: `ui`, `spinifexui` | **DONE** |
-| `spx service spinifex-ui stop` | — | spinifex-ui must be running | Stops the spinifex-ui service | **DONE** |
-| `spx service spinifex-ui status` | — | None | Reports spinifex-ui service status | **DONE** |
-| `spx service vpcd start` | — | `--config` required, OVN/OVS installed | Loads cluster config → starts VPC daemon (subscribes to `vpc.*` NATS events, translates to OVN logical switches/ports/routers) | **DONE** |
-| `spx service vpcd stop` | — | vpcd must be running | Stops the vpcd service | **DONE** |
-| `spx service vpcd status` | — | None | Reports vpcd service status | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx service predastore start` | `--port` (default: 8443), `--host` (default: 0.0.0.0), `--base-path`, `--config-path`, `--debug`, `--tls-cert`, `--tls-key`, `--backend` (distributed/filesystem, default: distributed), `--node-id` (default: 0), `--pprof`, `--pprof-output` | Creates predastore service instance with S3-compatible storage backend → starts service |
+| `spx service predastore stop` | — | Stops the predastore service |
+| `spx service predastore status` | — | Reports predastore service status |
+| `spx service viperblock start` | `--s3-host` (default: 0.0.0.0:8443), `--s3-bucket` (default: predastore), `--s3-region` (default: ap-southeast-2), `--plugin-path` (auto-detected via `nbdkit --dump-config plugindir`; typically `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` on amd64, overridable via `SPINIFEX_VIPERBLOCK_PLUGIN_PATH` in `/etc/spinifex/systemd.env`) | Loads cluster config → connects to NATS and Predastore → starts viperblock block storage service with NBD plugin |
+| `spx service viperblock stop` | — | Stops the viperblock service |
+| `spx service viperblock status` | — | Reports viperblock service status |
+| `spx service nats start` | `--port` (default: 4222), `--host` (default: 0.0.0.0), `--debug`, `--data-dir`, `--jetstream` | Starts embedded NATS server with optional JetStream |
+| `spx service nats stop` | — | Stops the NATS service |
+| `spx service nats status` | — | Reports NATS service status |
+| `spx service spinifex start` | `--wal-dir` | Loads cluster config → starts spinifex daemon (VM orchestration, NATS subscriptions, health endpoint) |
+| `spx service spinifex stop` | — | Stops the spinifex daemon service |
+| `spx service spinifex status` | — | Reports spinifex daemon service status |
+| `spx service awsgw start` | `--host` (default: 0.0.0.0:9999), `--tls-cert`, `--tls-key`, `--debug` | Loads cluster config → starts AWS-compatible gateway with SigV4 auth, IAM policy enforcement, TLS |
+| `spx service awsgw stop` | — | Stops the AWS gateway service |
+| `spx service awsgw status` | — | Reports AWS gateway service status |
+| `spx service spinifex-ui start` | `--port` (default: 3000), `--host` (default: 0.0.0.0), `--tls-cert`, `--tls-key` | Starts embedded web UI server serving the React frontend. Aliases: `ui`, `spinifexui` |
+| `spx service spinifex-ui stop` | — | Stops the spinifex-ui service |
+| `spx service spinifex-ui status` | — | Reports spinifex-ui service status |
+| `spx service vpcd start` | — | Loads cluster config → starts VPC daemon (subscribes to `vpc.*` NATS events, translates to OVN logical switches/ports/routers) |
+| `spx service vpcd stop` | — | Stops the vpcd service |
+| `spx service vpcd status` | — | Reports vpcd service status |

 ### Cluster Inspection

 Operational commands for inspecting cluster state. These fan out NATS requests to all nodes and aggregate responses.

-| Command | Flags | Prerequisites | Basic Logic | Output Columns | Status |
-|---------|-------|---------------|-------------|----------------|--------|
-| `spx get nodes` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Loads config → publishes to `spinifex.node.status` fan-out topic → collects responses within timeout → merges config-known nodes with NATS responders → nodes that don't respond shown as `NotReady` | NAME, STATUS, IP, REGION, AZ, UPTIME, VMs, SERVICES | **DONE** |
-| `spx get vms` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Publishes to `spinifex.node.vms` fan-out topic → collects VM info from all nodes → sorts by node then instance ID → prints table. Alias: `spx get instances` | INSTANCE, STATUS, TYPE, VCPU, MEM, NODE, IP, AGE | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx get nodes` | `--timeout` (default: 3s) | Loads config → publishes to `spinifex.node.status` fan-out topic → collects responses within timeout → merges config-known nodes with NATS responders → nodes that don't respond shown as `NotReady` |
+| `spx get vms` | `--timeout` (default: 3s) | Publishes to `spinifex.node.vms` fan-out topic → collects VM info from all nodes → sorts by node then instance ID → prints table. Alias: `spx get instances` |

 ### Resource Monitoring

-| Command | Flags | Prerequisites | Basic Logic | Output | Status |
-|---------|-------|---------------|-------------|--------|--------|
-| `spx top nodes` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Publishes to `spinifex.node.status` fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability | Table 1: NAME, CPU (used/total), MEM (used/total), GPU (used/total), VMs — GPU column shows `allocated/total` when passthrough is enabled, `0/count*` when the node is GPU-capable but passthrough is disabled, or `-` when no GPU is present. Table 2: INSTANCE TYPE, AVAILABLE, VCPU, MEMORY | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx top nodes` | `--timeout` (default: 3s) | Publishes to `spinifex.node.status` fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability |

 ### Cluster Initialization

-| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
-|---------|-------|---------------|-------------|------------|--------|
-| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--token-ttl`, `--force` | None (first-time setup) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` | 1. Init creates all config files<br>2. Root credentials printed once<br>3. master.key is 32 bytes, mode 0600<br>4. bootstrap.json consumed on first start<br>5. `--force` re-initializes existing config<br>6. AWS CLI profile `spx` auto-configured<br>7. Multi-node init generates join token and writes to `<config-dir>/join-token` | **DONE** |
-| `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services` | Leader node must be running | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations | 1. Join existing cluster<br>2. Missing host (error)<br>3. Missing node name (error)<br>4. Missing token (error)<br>5. Invalid token (401)<br>6. Expired token (401) | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--token-ttl`, `--force` | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` |
+| `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services` | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations |

 ### Version

-| Command | Flags | Prerequisites | Basic Logic | Status |
-|---------|-------|---------------|-------------|--------|
-| `spx version` | — | None | Prints Spinifex version, commit hash, OS, and architecture (populated via build-time ldflags) | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx version` | — | Prints Spinifex version, commit hash, OS, and architecture (populated via build-time ldflags) |

 ### Cluster Operations

-| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
-|---------|-------|---------------|-------------|------------|--------|
-| `spx admin cluster shutdown` | `--force` (shutdown even if nodes don't respond), `--timeout` (max wait per phase, default 120s), `--dry-run` (print phase plan without executing) | Cluster must be running | Performs coordinated, phased shutdown of entire cluster. Phases execute in order: GATE (stop API/UI) → DRAIN (stop VMs) → STORAGE (stop viperblock) → PERSIST (stop predastore) → INFRA (stop NATS/daemon). Each phase waits for all nodes to ACK before proceeding. Uses JetStream state tracking. | 1. Shutdown running cluster<br>2. All nodes stop cleanly<br>3. Force shutdown with unresponsive nodes<br>4. Dry-run prints plan | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin cluster shutdown` | `--force` (shutdown even if nodes don't respond), `--timeout` (max wait per phase, default 120s), `--dry-run` (print phase plan without executing) | Performs coordinated, phased shutdown of entire cluster. Phases execute in order: GATE (stop API/UI) → DRAIN (stop VMs) → STORAGE (stop viperblock) → PERSIST (stop predastore) → INFRA (stop NATS/daemon). Each phase waits for all nodes to ACK before proceeding. Uses JetStream state tracking. |
+| `spx admin cluster drain-dhcp` | `--timeout` (reply-collection window, default 30s) | Asks each vpcd to DHCPRELEASE every external-pool DHCP lease it currently holds, returning them to the upstream DHCP server. Run on teardown before stopping services — an env reset otherwise strands held leases upstream until their TTL expires, eventually exhausting the upstream scope. Best-effort: warns and exits 0 if the cluster is already down. |

 ### Certificate Management

-| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
-|---------|-------|---------------|-------------|------------|--------|
-| `spx admin cert renew` | `--extra-ip` (additional IPs for SANs), `--extra-dns` (additional DNS names for SANs) | Existing CA from `spx admin init` | Reads existing CA → regenerates server certificate with all current network interface IPs and machine hostname in SANs → writes new cert. Use after adding a new network interface or changing IP addresses. | 1. Renew with auto-detected IPs<br>2. Renew with extra IPs<br>3. Renew with extra DNS names | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin cert renew` | `--extra-ip` (additional IPs for SANs), `--extra-dns` (additional DNS names for SANs) | Reads existing CA → regenerates server certificate with all current network interface IPs and machine hostname in SANs → writes new cert. Use after adding a new network interface or changing IP addresses. |

 ### Upgrade Management

-| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
-|---------|-------|---------------|-------------|------------|--------|
-| `spx admin upgrade` | `--yes` (apply without prompting), `--config-dir` (persistent, default: `~/spinifex/config`), `--spinifex-dir` (persistent, default: `~/spinifex`) | Existing installation (`spinifex.toml` must exist in config dir) | Reads current config versions from registry → computes pending config migrations (from→to per target) → prints version summary and pending list → prompts for confirmation unless `--yes` → runs `migrate.DefaultRegistry.RunAllConfig()` to apply migrations to config files. Intended for upgrades between Spinifex versions. Operators can skip migrations during install by setting `INSTALL_SPINIFEX_SKIP_MIGRATE=1`, then run `spx admin upgrade` manually to review and apply. After completion, services must be restarted with `sudo systemctl restart spinifex.target`. Invoked non-interactively by `scripts/setup.sh` with `--yes`. | 1. No installation present (error, suggests `spx admin init`)<br>2. No pending migrations (prints "No pending config migrations")<br>3. Pending migrations with interactive confirmation (y/yes applies, anything else aborts)<br>4. Pending migrations with `--yes` (applies without prompt)<br>5. Migration failure surfaces error and exits non-zero | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin upgrade` | `--yes` (apply without prompting), `--config-dir` (persistent, default: `~/spinifex/config`), `--spinifex-dir` (persistent, default: `~/spinifex`) | Reads current config versions from registry → computes pending config migrations (from→to per target) → prints version summary and pending list → prompts for confirmation unless `--yes` → runs `migrate.DefaultRegistry.RunAllConfig()` to apply migrations to config files. Intended for upgrades between Spinifex versions. Operators can skip migrations during install by setting `INSTALL_SPINIFEX_SKIP_MIGRATE=1`, then run `spx admin upgrade` manually to review and apply. After completion, services must be restarted with `sudo systemctl restart spinifex.target`. Invoked non-interactively by `scripts/setup.sh` with `--yes`. |

 ### Account Management

-| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
-|---------|-------|---------------|-------------|------------|--------|
-| `spx admin account create` | `--name` | Cluster must be running (NATS), master key must exist | Connects to NATS → CAS loop on `spinifex-account-counter:next_id` for sequential 12-digit ID → creates Account record in `spinifex-accounts` KV → creates `admin` user in new account → creates access key for admin → creates AdministratorAccess policy (Action:*, Resource:*) → attaches policy → prints credentials | 1. Create first account (ID 000000000001)<br>2. Sequential IDs (000000000002, etc.)<br>3. Admin user has full access<br>4. Credentials printed once | **DONE** |
-| `spx admin account list` | — | Cluster must be running (NATS), master key must exist | Connects to NATS → IAMService.ListAccounts() → prints table with Account ID, Name, Status, Created | 1. List all accounts<br>2. Empty list (only global account) | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin account create` | `--name` | Connects to NATS → CAS loop on `spinifex-account-counter:next_id` for sequential 12-digit ID → creates Account record in `spinifex-accounts` KV → creates `admin` user in new account → creates access key for admin → creates AdministratorAccess policy (Action:*, Resource:*) → attaches policy → prints credentials |
+| `spx admin account list` | — | Connects to NATS → IAMService.ListAccounts() → prints table with Account ID, Name, Status, Created |

 ### Image Management

-| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
-|---------|-------|---------------|-------------|------------|--------|
-| `spx admin images import` | `--name`, `--file`, `--distro`, `--version`, `--arch`, `--platform`, `--boot-mode` (bios/uefi/uefi-preferred), `--tag`, `--force`, `--skip-verify` | Cluster must be running; either `--name` (catalog download) or `--file` (operator-supplied media). `--file` mode additionally requires `--distro`, `--version`, `--arch`, `--boot-mode`. | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, verify the SHA-256/SHA-512 digest, and inherit `BootMode` from the catalog entry. `--boot-mode` overrides the catalog value when set. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports skip checksum verification (operator-supplied media is outside Spinifex's trust boundary, the skip is logged at INFO for audit) and require an explicit `--boot-mode` because there is no catalog metadata to inherit from. `--skip-verify` bypasses verification for catalog imports and emits a WARN slog + stderr notice; use only for debugging or when upstream mirrors are confirmed-broken. | 1. Import valid catalog image (verifies checksum)<br>2. Tampered cache hit fails with `ErrChecksumMismatch`<br>3. `--force` recovers after a mismatch<br>4. `--file` import skips verification<br>5. `--skip-verify` bypasses checksum with WARN<br>6. `--file` without `--boot-mode` rejected<br>7. `--boot-mode=invalid` rejected | **DONE** |
-| `spx admin images list` | — | None | Lists available OS images that can be imported or downloaded | 1. List available images | **DONE** |
-| `spx admin images remove` | `--image-id` (required), `--force`, `--yes` | Cluster must be running; predastore reachable. Targets AMIs imported via `spx admin images import` (non-account `ImageOwnerAlias`, e.g. `"system"`). | Loads `ami-<id>/config.json`, walks transitive dependents — copied snapshots whose `VolumeID == imageID`, volumes whose `SnapshotID` references the internal `snap-ami-<id>` or any derived snap, and account AMIs created via `CopyImage` whose `SnapshotID` is a derived snap — then prompts (skipped with `--yes`) before deleting `ami-<id>/config.json` (the DescribeImages barrier) followed by the rest of `ami-<id>/` and `snap-ami-<id>/`. Account-owned AMIs are refused with a hint pointing at `aws ec2 deregister-image` + `aws ec2 delete-snapshot`. `--force` bypasses the dependency, ownership and config-corrupt checks for salvage of orphaned blocks. | 1. Happy path: system AMI, no deps → deleted<br>2. Account-owned → refused with AWS-flow hint<br>3. Missing/corrupt config → `InvalidAMIID.NotFound` (salvageable with `--force`)<br>4. Dependent volume (direct or via copied snap) → refused<br>5. Dependent account AMI → refused<br>6. `--force` overrides dependents<br>7. Idempotent re-run after partial delete | **DONE** |
-
-#### Image integrity verification (CMMC SI.L1-3.14.2)
-
-Catalog imports (`spx admin images import --name <name>`) verify the image
-against the catalog-declared SHA-256/SHA-512 digest before extraction. The sums
-file is fetched from the catalog `Checksum` URL over HTTPS only (cross-scheme
-redirects refused), and verification runs on both fresh downloads and cache
-hits so a poisoned cache is caught on the next import.
-
-On mismatch the import exits non-zero, the cached file is left on disk for
-inspection, and the printed guidance is `spx admin images import --name <name>
--force` to re-download.
-
-`--file` imports skip verification by design: operator-supplied media is
-outside Spinifex's trust boundary and the operator is responsible for
-integrity (e.g. `sha256sum` against a trusted upstream digest before import).
-The skip is recorded as an INFO `slog` event with `reason=local-file-import`
-so a CMMC assessor can audit the decision from journald.
-
-`--skip-verify` bypasses the checksum step for catalog imports. The command
-still downloads via the catalog URL but does not compare the image digest
-against the sums file. Intended for narrow cases such as debugging upstream
-mirror issues or running against a transiently-broken `latest/` path; the
-skip is logged at WARN with `reason=skip-verify-flag` and printed to stderr
-so operators and assessors see it. Prefer `--file` with an out-of-band
-verified image over `--skip-verify` whenever possible.
-
-**Limitation:** verification confirms the image matches the digest the mirror
-served. A mirror compromise that swaps both image and sums file is not
-detected; closing that gap requires GPG signature verification of the sums
-file, deferred to a later phase.
-
-#### `spx admin images remove` caveats
-
-Admin-imported AMIs (`ImageOwnerAlias = "system"`) live
-under the `ami-<id>/` S3 prefix and use a viperblock-internal snap checkpoint
-at `snap-ami-<id>/` — there is no `snap-<id>/metadata.json`. The AWS handlers
-(`DeregisterImage`, `DeleteSnapshot`) reject system owners with
-`UnauthorizedOperation`, which is the right behaviour for tenant API callers
-but leaves no AWS-flow path to reclaim space. `spx admin images remove` is
-the admin-trust-boundary counterpart that performs the dependency walk and
-hard-deletes the blocks directly against predastore.
-
-`CopyImage` of a system AMI is metadata-only: it writes a fresh
-`snap-<acct>/metadata.json` whose `VolumeID` points at `ami-<sys>` and a new
-`ami-<acct>/config.json` referencing that snap. Volumes launched from the
-copied AMI read transitively from `ami-<sys>/chunks/...`. The remove command
-walks this transitive set and refuses if anything references the target.
-
-**TOCTOU window:** between the safety scan and the `config.json` delete a
-concurrent `RunInstances` against the AMI could create a new dependent
-volume. The window is sub-second on a healthy cluster. The admin running
-this command is expected to know the fleet's operational state; if the race
-fires the result is a `vol-<id>` with deleted backing blocks, recovered by
-terminating the orphaned instance.
-
-**`--force` bypasses every safety check** (dependents, ownership, missing /
-corrupt `config.json`). Use only for salvage of orphaned blocks. Running it
-against a live system AMI corrupts every dependent volume on the next disk
-read.
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin images import` | `--name`, `--file`, `--distro`, `--version`, `--arch`, `--platform`, `--boot-mode` (bios/uefi/uefi-preferred), `--tag`, `--force`, `--skip-verify` | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, verify the SHA-256/SHA-512 digest, and inherit `BootMode` from the catalog entry. `--boot-mode` overrides the catalog value when set. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports skip checksum verification (operator-supplied media is outside Spinifex's trust boundary, the skip is logged at INFO for audit) and require an explicit `--boot-mode` because there is no catalog metadata to inherit from. `--skip-verify` bypasses verification for catalog imports and emits a WARN slog + stderr notice; use only for debugging or when upstream mirrors are confirmed-broken. |
+| `spx admin images list` | — | Lists available OS images that can be imported or downloaded |
+| `spx admin images remove` | `--image-id` (required), `--force`, `--yes` | Loads `ami-<id>/config.json`, walks transitive dependents — copied snapshots whose `VolumeID == imageID`, volumes whose `SnapshotID` references the internal `snap-ami-<id>` or any derived snap, and account AMIs created via `CopyImage` whose `SnapshotID` is a derived snap — then prompts (skipped with `--yes`) before deleting `ami-<id>/config.json` (the DescribeImages barrier) followed by the rest of `ami-<id>/` and `snap-ami-<id>/`. Account-owned AMIs are refused with a hint pointing at `aws ec2 deregister-image` + `aws ec2 delete-snapshot`. `--force` bypasses the dependency, ownership and config-corrupt checks for salvage of orphaned blocks. |

 ### GPU Management

-| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
-|---------|-------|---------------|-------------|------------|--------|
-| `spx admin gpu status` | `--node` (default: local node) | Cluster must be running (NATS) | Queries `spinifex.node.status` fan-out → finds the target node response → prints Node, GPU hardware (model list or "none detected"), IOMMU state, vfio-pci state, passthrough enabled/disabled, and GPU pool allocation (`allocated/total`). Also lists GPU-capable instance types when passthrough is active. | 1. Local node with passthrough enabled (shows pool and instance types)<br>2. Local node capable but passthrough disabled (shows "run enable")<br>3. Node without GPU hardware (shows "none detected")<br>4. `--node` targets a remote node<br>5. Node not found or not responding (error) | **DONE** |
-| `spx admin gpu enable` | — | Cluster must be running (NATS); node must be GPU-capable (IOMMU + vfio-pci configured) | Checks current passthrough state via NATS → errors if already enabled or prerequisites not met (directs to `setup`) → writes `gpu_passthrough = true` to `spinifex.toml` via `admin.SetGPUPassthrough` → sends SIGHUP to `spinifex-daemon` via `systemctl kill -s HUP` → polls node status for up to 30 s until daemon confirms new state → prints final `gpu status` output. Must be run directly on the target host. | 1. Enable on capable node<br>2. Already enabled (no-op)<br>3. Prerequisites not met (error, suggests `setup`)<br>4. Daemon does not confirm within 30 s (timeout error) | **DONE** |
-| `spx admin gpu disable` | — | Cluster must be running (NATS); no GPU instances may be running | Checks current passthrough state via NATS → errors if already disabled or if `AllocGPUs > 0` (must terminate GPU instances first) → writes `gpu_passthrough = false` → sends SIGHUP to `spinifex-daemon` → polls for up to 30 s → prints final `gpu status` output. Must be run directly on the target host. | 1. Disable with no GPU instances running<br>2. Already disabled (no-op)<br>3. GPU instances running (error with count)<br>4. Daemon does not confirm within 30 s (timeout error) | **DONE** |
-| `spx admin gpu setup` | — | Must be run as root; NVIDIA or AMD GPU must be present | Idempotent host configuration for GPU passthrough. Steps (skipped if already applied): detect GPUs via `gpu.Discover` → collect PCI IDs for all IOMMU-group siblings → check/enable IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`) → write vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`) → blacklist nouveau (`/etc/modprobe.d/blacklist-nouveau.conf`) → blacklist amdgpu if AMD GPU present (`/etc/modprobe.d/blacklist-amdgpu.conf`) → write vfio-pci early binding config (`/etc/modprobe.d/vfio-pci.conf`) → add vfio modules to initramfs. If any change requires a reboot: runs `update-initramfs -u` and exits with reboot instructions. After reboot: verifies `vfio_pci` module is loaded → verifies each GPU is bound to `vfio-pci` (binds explicitly via `driver_override` if unbound) → calls `gpu enable` to activate passthrough. | 1. First run on unconfigured host (GRUB updated, reboot required)<br>2. Re-run after reboot (verifies bindings, enables passthrough)<br>3. Already fully configured (all steps skipped, enables passthrough)<br>4. AMD GPU present (blacklists amdgpu in addition to nouveau)<br>5. Not run as root (error)<br>6. No GPU hardware detected (error) | **DONE** |
-| `spx admin gpu mig status` | — (must be run on target host) | NVIDIA GPU with MIG support present | Runs `gpu.Discover()` locally → for each GPU prints: PCI address, model, MIG capability, MIG mode (enabled/disabled/N/A); for GPUs with MIG enabled lists active MIG slices with GI ID, profile name, and mdev path. | 1. Node with MIG enabled (shows slices)<br>2. MIG capable but not enabled (shows disabled)<br>3. No MIG-capable GPU (shows N/A for MIG fields) | **DONE** |
-| `spx admin gpu mig enable` | `--profile <name>` (required, e.g. `1g.10gb`), `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Must be run as root; MIG-capable GPU present; no GPU instances running | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → enables MIG mode on each target (`gpu.EnableMIGMode`) → lists available profiles (`gpu.ListProfiles`) and validates requested profile name → destroys any existing instances (`gpu.DestroyAllInstances`) → creates new instances filling GPU capacity (`gpu.CreateInstances`) → writes `mig_profile` to `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. | 1. Enable MIG with valid profile<br>2. Invalid profile name (error with available profile list)<br>3. No MIG-capable GPU (error)<br>4. GPU instances running (error)<br>5. `--gpu` targets specific GPU<br>6. Not run as root (error) | **DONE** |
-| `spx admin gpu mig disable` | `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Must be run as root; no GPU instances running | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → destroys all GPU instances (`gpu.DestroyAllInstances`) → disables MIG mode (`gpu.DisableMIGMode`) → clears `mig_profile` in `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. | 1. Disable with no running GPU instances<br>2. GPU instances running (error)<br>3. MIG not enabled on GPU (skips that GPU)<br>4. `--gpu` targets specific GPU<br>5. Not run as root (error) | **DONE** |
+| Command | Flags | Description |
+|---------|-------|-------------|
+| `spx admin gpu status` | `--node` (default: local node) | Queries `spinifex.node.status` fan-out → finds the target node response → prints Node, GPU hardware (model list or "none detected"), IOMMU state, vfio-pci state, passthrough enabled/disabled, and GPU pool allocation (`allocated/total`). Also lists GPU-capable instance types when passthrough is active. |
+| `spx admin gpu enable` | — | Checks current passthrough state via NATS → errors if already enabled or prerequisites not met (directs to `setup`) → writes `gpu_passthrough = true` to `spinifex.toml` via `admin.SetGPUPassthrough` → sends SIGHUP to `spinifex-daemon` via `systemctl kill -s HUP` → polls node status for up to 30 s until daemon confirms new state → prints final `gpu status` output. Must be run directly on the target host. |
+| `spx admin gpu disable` | — | Checks current passthrough state via NATS → errors if already disabled or if `AllocGPUs > 0` (must terminate GPU instances first) → writes `gpu_passthrough = false` → sends SIGHUP to `spinifex-daemon` → polls for up to 30 s → prints final `gpu status` output. Must be run directly on the target host. |
+| `spx admin gpu setup` | — | Idempotent host configuration for GPU passthrough. Steps (skipped if already applied): detect GPUs via `gpu.Discover` → collect PCI IDs for all IOMMU-group siblings → check/enable IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`) → write vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`) → blacklist nouveau (`/etc/modprobe.d/blacklist-nouveau.conf`) → blacklist amdgpu if AMD GPU present (`/etc/modprobe.d/blacklist-amdgpu.conf`) → write vfio-pci early binding config (`/etc/modprobe.d/vfio-pci.conf`) → add vfio modules to initramfs. If any change requires a reboot: runs `update-initramfs -u` and exits with reboot instructions. After reboot: verifies `vfio_pci` module is loaded → verifies each GPU is bound to `vfio-pci` (binds explicitly via `driver_override` if unbound) → calls `gpu enable` to activate passthrough. |
+| `spx admin gpu mig status` | — (must be run on target host) | Runs `gpu.Discover()` locally → for each GPU prints: PCI address, model, MIG capability, MIG mode (enabled/disabled/N/A); for GPUs with MIG enabled lists active MIG slices with GI ID, profile name, and mdev path. |
+| `spx admin gpu mig enable` | `--profile <name>` (required, e.g. `1g.10gb`), `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → enables MIG mode on each target (`gpu.EnableMIGMode`) → lists available profiles (`gpu.ListProfiles`) and validates requested profile name → destroys any existing instances (`gpu.DestroyAllInstances`) → creates new instances filling GPU capacity (`gpu.CreateInstances`) → writes `mig_profile` to `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. |
+| `spx admin gpu mig disable` | `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → destroys all GPU instances (`gpu.DestroyAllInstances`) → disables MIG mode (`gpu.DisableMIGMode`) → clears `mig_profile` in `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. |


 ## AWS-Compatible API
@@ -172,7 +113,7 @@ read.

 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
-| `run-instances` | `--image-id`, `--instance-type`, `--count`, `--key-name`, `--user-data`, `--subnet-id`, `--block-device-mappings` (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination), `--placement` (GroupName), `--iam-instance-profile` (Name/Arn) | `--security-group-ids`, `--tag-specifications`, `--dry-run`, `--client-token`, `--disable-api-termination`, `--ebs-optimized`, `--network-interfaces`, `--private-ip-address`, `--monitoring`, `--credit-specification`, `--cpu-options`, `--metadata-options`, `--launch-template`, `--hibernate-options` | **DONE** |
+| `run-instances` | `--image-id`, `--instance-type`, `--count`, `--key-name`, `--user-data`, `--subnet-id`, `--security-group-ids`, `--tag-specifications` (instance-scoped), `--block-device-mappings` (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination), `--placement` (GroupName), `--iam-instance-profile` (Name/Arn), `--capacity-reservation-specification` (CapacityReservationTarget.CapacityReservationId, targeted-by-id only) | `--dry-run`, `--client-token`, `--disable-api-termination`, `--ebs-optimized`, `--network-interfaces`, `--private-ip-address`, `--monitoring`, `--credit-specification`, `--cpu-options`, `--metadata-options`, `--launch-template`, `--hibernate-options` | **DONE** |
 | `describe-instances` | `--instance-ids`, `--filters` (instance-state-name, instance-id, instance-type, vpc-id, subnet-id, tag:*, tag-key, tag-value) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
 | `start-instances` | `--instance-ids` | `--dry-run`, `--force` | **DONE** |
 | `stop-instances` | `--instance-ids` | `--force`, `--hibernate`, `--dry-run` | **DONE** |
@@ -191,6 +132,11 @@ read.
 role ARN and rejects cross-account profile ARNs. Placement strategies: `spread`
 (1 per node, atomic CAS) and `cluster` (pin all to one node).

+`run-instances --capacity-reservation-specification` consumes a reserved slot on
+the node owning the targeted reservation (targeted-by-id only; `open`
+auto-matching is not supported). The combination with `--placement` (group) is
+rejected. `describe-instances` then reports the instance's `CapacityReservationId`.
+
 ### EC2 — IAM Instance Profile Associations

 Stored as `vm.VM.IamInstanceProfileArn` + `IamInstanceProfileAssociationId`
@@ -243,7 +189,7 @@ input but treat it as `uefi` at launch.
 | `attach-volume` | `--volume-id`, `--instance-id`, `--device` (auto-assigns `/dev/sd[f-p]`) | `--dry-run` | **DONE** |
 | `detach-volume` | `--volume-id`, `--instance-id` (optional), `--device`, `--force` | `--dry-run` | **DONE** |
 | `describe-volume-status` | `--volume-ids`, `--filters` (volume-id, volume-status.status, availability-zone) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
-| `describe-volumes-modifications` | — | `--volume-ids`, `--filters`, `--max-results` | **NOT STARTED** |
+| `describe-volumes-modifications` | `--volume-ids`, `--filters` (modification-state, original-iops, original-size, original-volume-type, start-time, target-iops, target-size, target-volume-type, volume-id) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |

 ### EC2 — Snapshots

@@ -295,9 +241,6 @@ by downstream services.

 ### EC2 — VPC Core

-VPC/Subnet/ENI/SG CRUD stores metadata in NATS KV and publishes events to
-vpcd for OVN translation. Single AZ for Spinifex v1.
-
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
 | `create-vpc` | `--cidr-block`, `--tag-specifications` | `--instance-tenancy`, `--dry-run` | **DONE** |
@@ -393,9 +336,6 @@ EIP commands are only registered when an external IPAM pool is configured.

 ### EC2 — NAT Gateway

-Deleted gateways move to a separate KV bucket with 1-hour TTL for Terraform
-polling.
-
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
 | `create-nat-gateway` | `--subnet-id`, `--allocation-id` | `--connectivity-type`, `--tag-specifications`, `--dry-run` | **DONE** |
@@ -406,9 +346,6 @@ polling.

 ### EC2 — Placement Groups

-Strategies: `spread` (1 instance per node, strict) and `cluster` (all pinned
-to single node). `partition` rejected.
-
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
 | `create-placement-group` | `--group-name`, `--strategy` (spread/cluster) | `--partition-count`, `--spread-level`, `--tag-specifications`, `--dry-run` | **DONE** |
@@ -501,9 +438,9 @@ to single node). `partition` rejected.
 | `delete-dhcp-options` | — | `--dhcp-options-id`, `--dry-run` | **NOT STARTED** |
 | `describe-dhcp-options` | — | `--dhcp-options-ids`, `--filters`, `--max-results` | **NOT STARTED** |
 | `associate-dhcp-options` | — | `--dhcp-options-id`, `--vpc-id`, `--dry-run` | **NOT STARTED** |
-| `create-capacity-reservation` | — | `--instance-type`, `--instance-platform`, `--availability-zone`, `--instance-count`, `--end-date`, `--end-date-type`, `--instance-match-criteria`, `--tag-specifications` | **NOT STARTED** |
-| `cancel-capacity-reservation` | — | `--capacity-reservation-id`, `--dry-run` | **NOT STARTED** |
-| `describe-capacity-reservations` | — | `--capacity-reservation-ids`, `--filters`, `--max-results` | **NOT STARTED** |
+| `create-capacity-reservation` | `--instance-type`, `--instance-count`, `--availability-zone`, `--instance-platform`, `--instance-match-criteria`, `--tenancy` (default only), `--dry-run` | `--end-date`, `--end-date-type` (unlimited only), `--availability-zone-id`, `--tag-specifications` | **DONE** |
+| `cancel-capacity-reservation` | `--capacity-reservation-id`, `--dry-run` | — | **DONE** |
+| `describe-capacity-reservations` | `--capacity-reservation-ids`, `--filters` | `--max-results`, `--next-token` | **DONE** |
 | `modify-capacity-reservation` | — | `--capacity-reservation-id`, `--instance-count`, `--end-date`, `--end-date-type` | **NOT STARTED** |

 ### EC2 — Misc
@@ -543,9 +480,6 @@ bypasses policy evaluation entirely.

 ### IAM — Access Keys

-Max 2 keys per user. Secrets encrypted with AES-256-GCM using master key,
-returned plaintext only at creation.
-
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
 | `create-access-key` | `--user-name` | — | **DONE** |
@@ -555,34 +489,18 @@ returned plaintext only at creation.

 ### IAM — Policies

-Policy documents require `Version: "2012-10-17"` and a valid Statement array.
-Wildcard action matching supported (`ec2:*`, `ec2:Describe*`, `*`). Evaluation
-order: explicit Deny > explicit Allow > implicit Deny.
-
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
 | `create-policy` | `--policy-name`, `--policy-document`, `--path`, `--description` | `--tags` | **DONE** |
 | `get-policy` | `--policy-arn` | — | **DONE** |
 | `get-policy-version` | `--policy-arn`, `--version-id` | — | **DONE** |
+| `list-policy-versions` | `--policy-arn` | `--max-items`, `--marker` | **DONE** |
 | `list-policies` | — | `--scope`, `--only-attached`, `--path-prefix`, `--max-items`, `--marker` | **DONE** |
 | `delete-policy` | `--policy-arn` | — | **DONE** |
 | `attach-user-policy` | `--user-name`, `--policy-arn` | — | **DONE** |
 | `detach-user-policy` | `--user-name`, `--policy-arn` | — | **DONE** |
 | `list-attached-user-policies` | `--user-name` | `--path-prefix`, `--max-items`, `--marker` | **DONE** |

-### IAM — Policy Evaluation (internal — not an AWS API)
-
-| Feature | Status |
-|---------|--------|
-| Root user bypass | **DONE** |
-| Default deny | **DONE** |
-| Explicit allow | **DONE** |
-| Explicit deny | **DONE** |
-| Wildcard matching (`*`, `ec2:*`, `ec2:Describe*`) | **DONE** |
-| Account-scoped evaluation | **DONE** |
-| EC2 action mapping | **DONE** |
-| IAM action mapping (16 actions) | **DONE** |
-
 ### IAM — Roles

 | Command | Implemented Flags | Missing Flags | Status |
@@ -596,12 +514,11 @@ order: explicit Deny > explicit Allow > implicit Deny.
 | `attach-role-policy` | `--role-name`, `--policy-arn` | — | **DONE** |
 | `detach-role-policy` | `--role-name`, `--policy-arn` | — | **DONE** |
 | `list-attached-role-policies` | `--role-name`, `--path-prefix` | `--max-items`, `--marker` | **DONE** |
+| `list-role-policies` | `--role-name` | `--max-items`, `--marker` | **DONE** (gateway stub — inline policies unsupported, always returns empty) |
+| `get-role-policy` / `put-role-policy` / `delete-role-policy` | — | inline role policies | **NOT STARTED** |

 ### IAM — Instance Profiles

-Containers for IAM roles that allow EC2 instances to assume a role. Required
-for `run-instances --iam-instance-profile`. Max 1 role per profile.
-
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
 | `create-instance-profile` | `--instance-profile-name`, `--path`, `--tags` | — | **DONE** |
@@ -612,6 +529,19 @@ for `run-instances --iam-instance-profile`. Max 1 role per profile.
 | `add-role-to-instance-profile` | `--instance-profile-name`, `--role-name` | — | **DONE** |
 | `remove-role-from-instance-profile` | `--instance-profile-name`, `--role-name` | — | **DONE** |

+### IAM — OIDC Providers
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `create-open-id-connect-provider` | `--url`, `--client-id-list`, `--thumbprint-list`, `--tags` | — | **DONE** |
+| `get-open-id-connect-provider` | `--open-id-connect-provider-arn` | — | **DONE** |
+| `list-open-id-connect-providers` | — | — | **DONE** |
+| `delete-open-id-connect-provider` | `--open-id-connect-provider-arn` | — | **DONE** |
+| `add-client-id-to-open-id-connect-provider` | — | `--open-id-connect-provider-arn`, `--client-id` | **NOT STARTED** |
+| `remove-client-id-from-open-id-connect-provider` | — | `--open-id-connect-provider-arn`, `--client-id` | **NOT STARTED** |
+| `update-open-id-connect-provider-thumbprint` | — | `--open-id-connect-provider-arn`, `--thumbprint-list` | **NOT STARTED** |
+| `tag-open-id-connect-provider` / `untag-open-id-connect-provider` | — | `--open-id-connect-provider-arn`, `--tags`/`--tag-keys` | **NOT STARTED** |
+
 ### IAM — Groups

 | Command | Implemented Flags | Missing Flags | Status |
@@ -631,27 +561,85 @@ for `run-instances --iam-instance-profile`. Max 1 role per profile.

 ## STS

-| Command | Implemented Flags | Missing Flags (rejected if supplied) | Status |
+| Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|--------------------------------------|--------|
 | `get-caller-identity` | — | — | **DONE** |
 | `assume-role` | `--role-arn`, `--role-session-name`, `--duration-seconds` (900–min(role MaxSessionDuration, 43200)) | `--policy`, `--policy-arns` (→ `PackedPolicyTooLarge`); `--tags`, `--transitive-tag-keys` (→ `InvalidParameterValue`); `--serial-number`, `--token-code` (→ `InvalidParameterValue`); `--external-id`, `--source-identity` (accepted and logged, **not enforced** — no Condition evaluator in v1) | **DONE** |
-| `get-session-token` | — | `--duration-seconds`, `--serial-number`, `--token-code` | **NOT STARTED** |
-| `assume-role-with-web-identity` | — | `--role-arn`, `--role-session-name`, `--web-identity-token`, `--provider-id`, `--policy`, `--policy-arns`, `--duration-seconds` | **NOT STARTED** |
+| `get-session-token` | `--duration-seconds` (900–129600, default 43200 = 12h; clamped, not rejected) | `--serial-number`, `--token-code` (MFA → `InvalidParameterValue`) | **DONE** |
+| `assume-role-with-web-identity` | `--role-arn`, `--role-session-name`, `--web-identity-token`, `--duration-seconds` (900–43200, default 3600) | `--provider-id`; `--policy`, `--policy-arns` (→ `PackedPolicyTooLarge`) | **DONE** |
 | `assume-role-with-saml` | — | `--role-arn`, `--principal-arn`, `--saml-assertion`, `--policy`, `--policy-arns`, `--duration-seconds` | **NOT STARTED** |
 | `get-access-key-info` | — | `--access-key-id` | **NOT STARTED** |
 | `get-federation-token` | — | `--name`, `--policy`, `--policy-arns`, `--duration-seconds`, `--tags` | **NOT STARTED** |
 | `decode-authorization-message` | — | `--encoded-message` | **NOT STARTED** |

-Trust policies stored on roles (`AssumeRolePolicyDocument`) reject `Condition`,
-`NotPrincipal`, `NotAction`, empty-string `Action` elements, and empty
-`Principal` blocks at write time (`MalformedPolicyDocument`) — v1 has no
-Condition evaluator so accepting them would silently allow.
+Trust policies stored on roles (`AssumeRolePolicyDocument`) reject `Condition`, `NotPrincipal`, `NotAction`, empty-string `Action` elements, and empty `Principal` blocks at write time (`MalformedPolicyDocument`) — v1 has no Condition evaluator so accepting them would silently allow.
+
+---
+
+## IMDS (Instance Metadata Service)
+
+Available at `169.254.169.254` from inside every running guest VM, matching AWS. The endpoint is reached from within a guest over plain HTTP, exactly as on EC2, with no in-VM agent to install. DHCP and fully static guests reach it identically, with no in-guest route configuration.
+
+**IMDSv2-only.** Every read requires a session token. A tokenless (v1-style) `GET` returns `401 Unauthorized` with an empty body. Obtain a token with a `PUT /latest/api/token` carrying `X-aws-ec2-metadata-token-ttl-seconds` (1–21600), then send it back in `X-aws-ec2-metadata-token` on every read.
+
+```bash
+# Inside the guest VM:
+TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
+    -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
+curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
+    http://169.254.169.254/latest/meta-data/instance-id
+
+# Tokenless GET → 401 Unauthorized (empty body)
+curl -i http://169.254.169.254/latest/meta-data/instance-id
+```
+
+### IMDS — Supported Paths
+
+| Path | Method | Source | Status |
+|------|--------|--------|--------|
+| `/latest/api/token` | PUT | Issues an ENI-bound IMDSv2 token; `X-aws-ec2-metadata-token-ttl-seconds` ∈ [1, 21600] required | **DONE** |
+| `/` | GET | Supported API-version list (`2021-07-15`, `latest`); token-gated | **DONE** |
+| `/latest` | GET | Top-level tree listing (`dynamic`, `meta-data`, `user-data`) | **DONE** |
+| `/<date>/...` | GET/PUT | Any dated API version aliases to `/latest` (cloud-init parity) | **DONE** |
+| `/latest/meta-data/` | GET | Directory listing of supported children | **DONE** |
+| `/latest/meta-data/instance-id` | GET | `vm.ID` | **DONE** |
+| `/latest/meta-data/instance-type` | GET | `vm.InstanceType` | **DONE** |
+| `/latest/meta-data/ami-id` | GET | launch `ImageId` | **DONE** |
+| `/latest/meta-data/ami-launch-index` | GET | Per-instance launch index (`0..n-1`), contiguous on partial failure | **DONE** |
+| `/latest/meta-data/reservation-id` | GET | `DescribeInstances` `Reservation.ReservationId` | **DONE** |
+| `/latest/meta-data/instance-life-cycle` | GET | `on-demand` (Spot not modelled) | **DONE** |
+| `/latest/meta-data/local-ipv4` | GET | `ENIRecord.PrivateIpAddress` (== request source IP) | **DONE** |
+| `/latest/meta-data/public-ipv4` | GET | EIP, else instance public IP; empty body if none | **DONE** |
+| `/latest/meta-data/public-hostname` | GET | Mirrors `public-ipv4`; 404 when no public IP | **DONE** |
+| `/latest/meta-data/mac` | GET | `ENIRecord.MacAddress` | **DONE** |
+| `/latest/meta-data/security-groups` | GET | `ENIRecord.SecurityGroupIds`, newline-separated | **DONE** |
+| `/latest/meta-data/hostname`, `/local-hostname` | GET | Synthesised `ip-<dashed-ip>.<region>.compute.internal` | **DONE** |
+| `/latest/meta-data/placement/availability-zone` | GET | `ENIRecord.AvailabilityZone` | **DONE** |
+| `/latest/meta-data/placement/region` | GET | Derived from AZ (trailing letter stripped) | **DONE** |
+| `/latest/meta-data/services/{domain,partition}` | GET | Static: `amazonaws.com` / `aws` | **DONE** |
+| `/latest/meta-data/iam/info` | GET | `{InstanceProfileArn, InstanceProfileId}`; 404 if no profile | **DONE** |
+| `/latest/meta-data/iam/security-credentials/` | GET | Role name(s) under the profile, one per line; empty body if none | **DONE** |
+| `/latest/meta-data/iam/security-credentials/<role>` | GET | STS `AssumeRoleForInstance` → ASIA-prefixed temporary credential JSON | **DONE** |
+| `/latest/meta-data/public-keys/` | GET | `0=<keyName>` from the launch key pair; 404 if none | **DONE** |
+| `/latest/meta-data/public-keys/0/` | GET | `openssh-key` (format list for index 0) | **DONE** |
+| `/latest/meta-data/public-keys/0/openssh-key` | GET | Launch SSH public key, live-fetched from the key store; 404 if the key was deleted, 500 on backend fault | **DONE** |
+| `/latest/user-data` | GET | `vm.UserData`; 404 if none | **DONE** |
+| `/latest/dynamic` | GET | Lists `instance-identity/` | **DONE** |
+| `/latest/dynamic/instance-identity` | GET | Lists `document` (signed forms listed when the signing key lands) | **DONE** |
+| `/latest/dynamic/instance-identity/document` | GET | Unsigned identity document from resolved ENI + instance facts | **DONE** |
+| `/latest/dynamic/instance-identity/{signature,pkcs7,rsa2048}` | GET | Signed forms; need a per-cluster signing key | **NOT STARTED** (404; lands with EKS IRSA) |
+| `/latest/meta-data/network/interfaces/macs/<mac>/...` | GET | Multi-ENI rendering (subnet-id, vpc-id, ipv4s, security-group-ids, …) | **NOT STARTED** (404) |
+| `/latest/meta-data/tags/instance/<key>` | GET | Instance-tag metadata; gated on `InstanceMetadataTags` enablement | **NOT STARTED** (404) |
+| `/latest/meta-data/block-device-mapping/...` | GET | `ami`/`root`/`ebsN`/`ephemeralN` device map | **NOT STARTED** (404) |
+| `/latest/meta-data/placement/{group-name,partition-number,availability-zone-id,host-id}` | GET | Placement extras beyond `availability-zone`/`region` | **NOT STARTED** (404) |
+| `/latest/meta-data/instance-action` | GET | `none` unless interruptible instances ship | **NOT STARTED** (404) |
+| `/latest/meta-data/spot/{instance-action,termination-time}` | GET | Only meaningful once Spot is modelled | **NOT STARTED** (404) |

 ---

 ## ELBv2 (Application & Network Load Balancer)

-The data plane uses a system-managed LB VM running HAProxy, launched automatically during `create-load-balancer`. HAProxy config is pushed via NATS on listener/target changes.
+The data plane uses a system-managed LB VM, launched automatically during `create-load-balancer`. Application Load Balancers run **HAProxy** (L7: rules, fixed-response, redirect, HTTP/HTTPS). Network Load Balancers run **nginx `stream`** (L4: TCP, UDP, TLS, TCP_UDP) — HAProxy cannot load-balance UDP. The agent selects the engine from the `Engine` field on the config-delivery response.

 ### ELBv2 — Load Balancers

@@ -681,10 +669,54 @@ The data plane uses a system-managed LB VM running HAProxy, launched automatical

 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
-| `create-listener` | `--load-balancer-arn`, `--default-actions` (Type=forward, TargetGroupArn), `--protocol` (HTTP), `--port` | `--ssl-policy`, `--certificates`, `--alpn-policy`, `--mutual-authentication`, `--dry-run` | **DONE** |
+| `create-listener` | `--load-balancer-arn`, `--default-actions` (Type=forward, TargetGroupArn), `--protocol` (HTTP/HTTPS/TLS), `--port`, `--certificates` (HTTPS/TLS), `--ssl-policy` | `--alpn-policy`, `--mutual-authentication`, `--dry-run` | **DONE** |
+| `modify-listener` | `--listener-arn`, `--protocol`, `--port`, `--default-actions`, `--certificates`, `--ssl-policy` | `--alpn-policy`, `--mutual-authentication`, `--dry-run` | **DONE** |
 | `delete-listener` | `--listener-arn` | `--dry-run` | **DONE** |
 | `describe-listeners` | `--load-balancer-arn`, `--listener-arns` | `--page-size`, `--marker`, `--dry-run` | **DONE** |

+### ELBv2 — Listener Rules
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `create-rule` | `--listener-arn`, `--priority` (1–50000), `--conditions` (host-header, path-pattern, http-header, http-request-method, query-string, source-ip), `--actions` (forward, redirect, fixed-response), `--tags` | `--dry-run` | **DONE** |
+| `modify-rule` | `--rule-arn`, `--conditions`, `--actions` | `--dry-run` | **DONE** |
+| `delete-rule` | `--rule-arn` | `--dry-run` | **DONE** |
+| `describe-rules` | `--listener-arn`, `--rule-arns` | `--page-size`, `--marker` (parsed, not enforced) | **DONE** |
+| `set-rule-priorities` | `--rule-priorities` (RuleArn, Priority) | — | **DONE** |
+
+A synthetic `default` rule is derived from the listener's `DefaultActions`.
+
+### ELBv2 — Listener Certificates & SSL Policies
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `add-listener-certificates` | `--listener-arn`, `--certificates` | `--dry-run` | **DONE** |
+| `remove-listener-certificates` | `--listener-arn`, `--certificates` | `--dry-run` | **DONE** |
+| `describe-listener-certificates` | `--listener-arn` | `--page-size`, `--marker` | **DONE** |
+| `describe-ssl-policies` | `--names` | `--load-balancer-type`, `--page-size`, `--marker` | **DONE** (static catalog: `ELBSecurityPolicy-FS-1-2-Res-2019-08`, `ELBSecurityPolicy-TLS13-1-2-2021-06` — metadata only, no in-platform TLS termination) |
+
+The default certificate cannot be added/removed via these calls — set it on the listener.
+
+### ELBv2 — Attributes & Modify
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `describe-load-balancer-attributes` | `--load-balancer-arn` | — | **DONE** (stored values over per-type defaults) |
+| `modify-load-balancer-attributes` | `--load-balancer-arn`, `--attributes` | — | **DONE** (unknown keys → `ValidationError`) |
+| `describe-target-group-attributes` | `--target-group-arn` | — | **DONE** |
+| `modify-target-group-attributes` | `--target-group-arn`, `--attributes` (incl. `deregistration_delay.timeout_seconds`, `stickiness.*`) | — | **DONE** |
+| `modify-target-group` | `--target-group-arn`, `--health-check-*`, `--matcher` | `--target-type`/`--protocol`/`--vpc-id` (immutable) | **DONE** |
+| `describe-listener-attributes` | `--listener-arn` | — | **DONE** (stub — returns empty; not persisted) |
+| `modify-listener-attributes` | `--listener-arn`, `--attributes` | — | **DONE** (stub — echoes input; not persisted) |
+
+### ELBv2 — Network & Security
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `set-security-groups` | `--load-balancer-arn`, `--security-groups` | `--enforce-security-group-inbound-rules-on-private-link-traffic` | **DONE** (ALB only — NLB → `InvalidConfigurationRequest`) |
+| `set-subnets` | `--load-balancer-arn`, `--subnets`, `--subnet-mappings` | `--ip-address-type` | **DONE** (live ENI add/remove with rollback) |
+| `set-ip-address-type` | `--load-balancer-arn`, `--ip-address-type` (ipv4) | dualstack/IPv6 (rejected) | **DONE** |
+
 ### ELBv2 — Tags

 | Command | Implemented Flags | Missing Flags | Status |
@@ -695,18 +727,32 @@ The data plane uses a system-managed LB VM running HAProxy, launched automatical

 | Feature | Priority | Status |
 |---------|----------|--------|
-| Listener rules (CreateRule, DeleteRule, DescribeRules, ModifyRule) | High | **NOT STARTED** |
-| HTTPS/TLS termination (SSL certificates, policies, ALPN) | High | **NOT STARTED** |
-| Modify operations (`ModifyLoadBalancerAttributes`, `ModifyTargetGroup`, `ModifyTargetGroupAttributes`, `ModifyListener`) | Medium | **NOT STARTED** |
-| Connection draining (deregistration delay) | Medium | **NOT STARTED** |
-| Stickiness / session affinity | Medium | **NOT STARTED** |
-| Active health checking (API-driven, vs. HAProxy-only today) | Medium | **NOT STARTED** |
+| In-platform HTTPS/TLS termination (cert + SSL-policy APIs exist; data-plane TLS not terminated) | High | **NOT STARTED** |
+| ALPN policy, mutual TLS (mTLS) | Medium | **NOT STARTED** |
+| Listener attribute persistence (`Describe/ModifyListenerAttributes` are stubs) | Medium | **NOT STARTED** |
+| Active health checking (API-driven, vs. HAProxy/nginx-only today) | Medium | **NOT STARTED** |
 | IP and Lambda target types | Low | **NOT STARTED** |
 | S3 access log delivery | Low | **NOT STARTED** |
 | WAF integration | Low | **NOT STARTED** |

 ---

+## ACM (AWS Certificate Manager)
+
+Import-only — Spinifex stores externally-issued certificates for ELBv2 listener references; it does not issue certificates or validate domains (`RequestCertificate`). Certs are account-scoped; `describe`/`delete` enforce ownership.
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `import-certificate` | `--certificate`, `--private-key`, `--certificate-chain`, `--certificate-arn` (re-import) | `--tags` | **DONE** |
+| `describe-certificate` | `--certificate-arn` | — | **DONE** |
+| `list-certificates` | — | `--certificate-statuses`, `--includes`, `--max-items`, `--next-token` | **DONE** |
+| `delete-certificate` | `--certificate-arn` | — | **DONE** |
+| `request-certificate` | — | `--domain-name`, `--validation-method`, `--subject-alternative-names`, `--tags` | **NOT STARTED** |
+| `add-tags-to-certificate` / `list-tags-for-certificate` / `remove-tags-from-certificate` | — | `--certificate-arn`, `--tags`/`--tag-keys` | **NOT STARTED** |
+| `export-certificate` | — | `--certificate-arn`, `--passphrase` | **NOT STARTED** |
+
+---
+
 ## CloudWatch (Basic Monitoring)

 | Command | Implemented Flags | Missing Flags | Status |
@@ -82,6 +82,18 @@ spx admin images import --name debian-13-arm64

 Catalog imports verify the image against the catalog-declared SHA-256/SHA-512 digest before extraction. Use `--file` to import operator-supplied media (verification skipped — operator is responsible for integrity), or `--force` to re-download after a checksum mismatch.

+### EKS node image
+
+To run EKS, import the prebuilt node image from the catalog:
+
+```bash
+spx admin images import --name spinifex-eks-node
+```
+
+This pulls the Alpine + K3s node AMI from `iso.mulgadc.com`, verifies its
+checksum, and registers it tagged `spinifex:managed-by=eks`. `eks create-cluster`
+and `eks create-nodegroup` resolve the boot AMI by that tag, so no further
+configuration is needed.

 ## Cluster Shutdown

@@ -164,6 +164,16 @@ spx admin images import --name debian-13-x86_64

 cloud-init needs time to configure the instance after boot. Wait 30-60 seconds and retry.

+If the connection **times out** rather than being refused, the security group is likely blocking port 22. The default security group denies all inbound traffic (matching AWS), so SSH must be explicitly allowed:
+
+```bash
+# Allow SSH from anywhere on the instance's security group
+aws ec2 authorize-security-group-ingress \
+  --group-id $SG_ID --protocol tcp --port 22 --cidr 0.0.0.0/0
+```
+
+See [VPC Networking — Security Groups](/docs/vpc-networking#security-groups) for scoping rules to a trusted CIDR.
+
 Verify the SSH key was specified correctly when launching:

 ```bash
@@ -48,8 +48,9 @@ NIC. Each host runs `ovn-controller` which programs OpenFlow rules on `br-int`

 ## Private vs Public Subnets

-A subnet's behavior depends on two things: whether the VPC has an Internet
-Gateway, and whether the subnet has `MapPublicIpOnLaunch` enabled.
+A subnet's behavior depends on three things: whether the VPC has an Internet
+Gateway, whether the subnet's route table has a default route to that IGW, and
+whether the subnet has `MapPublicIpOnLaunch` enabled.

 ## Private Subnet (Default)

@@ -61,9 +62,10 @@ internet or be reached from the WAN.
  <img src="../../../.github/assets/diagrams/vpc-private-subnet-flow.svg" alt="Private subnet — instance hits router, no default route, packet dropped" width="900">
 </p>

-If the VPC has an IGW attached, private subnet instances CAN reach the internet
-via the VPC router's SNAT rule (outbound only — they share the gateway IP). They
-still cannot be reached from the WAN because they have no public IP.
+Private subnet instances reach the internet only if their route table has a
+default route to the IGW (shared SNAT, outbound only — they share the gateway
+IP) or to a NAT gateway. With no default route, egress is dropped. Either way
+they cannot be reached from the WAN because they have no public IP.

 ## Public Subnet

@@ -77,8 +79,18 @@ managed by OVN — the instance OS only sees its private IP.
 **Requirements for a public subnet:**

 1. VPC has an Internet Gateway attached
-2. Subnet has `MapPublicIpOnLaunch = true`
-3. External IP pool configured in `spinifex.toml`
+2. A route table associated with the subnet has a `0.0.0.0/0` route to the IGW
+3. Subnet has `MapPublicIpOnLaunch = true`
+4. External IP pool configured in `spinifex.toml`
+
+Spinifex follows AWS route-table semantics: a subnet is only "public" if its
+effective route table carries a default route to the IGW. A new VPC's main
+route table has the local route only — Spinifex does **not** add the IGW route
+for you. Without it, the subnet's egress is gated with a drop policy, so
+instances cannot reach the internet (and inbound connections cannot complete
+because return traffic is dropped) even with a public IP and an attached IGW.
+Add the route explicitly — either to the main route table, or to a custom route
+table associated with the subnet (see [Quick Start](#3-create-vpc-with-public-subnet)).

 ## Comparison

@@ -86,7 +98,7 @@ managed by OVN — the instance OS only sees its private IP.
 | ------------------------ | --------------------------------- | ------------------------------ |
 | Private IP               | Yes                               | Yes                            |
 | Public IP                | No                                | Auto-assigned from pool        |
-| Outbound internet        | Only if VPC has IGW (shared SNAT) | Yes (own public IP via SNAT)   |
+| Outbound internet        | Only with a default route to IGW/NAT GW | Yes (own public IP via SNAT)   |
 | Inbound from WAN         | No                                | Yes (via 1:1 NAT to public IP) |
 | Instance sees public IP? | N/A                               | No — only sees private IP      |
 | Elastic IP support       | Only if explicitly associated     | Yes                            |
@@ -190,8 +202,7 @@ use `setup-ovn.sh --dhcp` to obtain one from the router. This is the router's
 DHCP — not Spinifex's internal OVN DHCP for VMs.

 **Use when:** VMs only need outbound access (apt update, pulling images). Edge
-deployments behind ISP NAT. Single WAN IP available. Future: use with AWS-style
-NAT Gateway for private subnet internet access.
+deployments behind ISP NAT. Single WAN IP available.

 ```toml
 [network]
@@ -935,17 +946,39 @@ spx admin init
 VPC=$(aws ec2 create-vpc --cidr-block 10.200.0.0/16 \
  --query Vpc.VpcId --output text)

-SUBNET=$(aws ec2 create-subnet --vpc-id $VPC \
-  --cidr-block 10.200.1.0/24 \
-  --query Subnet.SubnetId --output text)
-
 IGW=$(aws ec2 create-internet-gateway \
  --query InternetGateway.InternetGatewayId --output text)
 aws ec2 attach-internet-gateway \
  --internet-gateway-id $IGW --vpc-id $VPC

+SUBNET=$(aws ec2 create-subnet --vpc-id $VPC \
+  --cidr-block 10.200.1.0/24 \
+  --query Subnet.SubnetId --output text)
+
+# Route table — give the subnet a default route to the IGW. Spinifex does NOT
+# add this automatically; without it the subnet's egress is dropped.
+RT=$(aws ec2 create-route-table --vpc-id $VPC \
+  --query RouteTable.RouteTableId --output text)
+
+aws ec2 create-route --route-table-id $RT \
+  --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW
+
+aws ec2 associate-route-table --route-table-id $RT --subnet-id $SUBNET
+
+# Auto-assign a public IP to instances launched into the subnet
 aws ec2 modify-subnet-attribute \
  --subnet-id $SUBNET --map-public-ip-on-launch
+
+# Allow SSH + ICMP on the VPC's default security group (blocks inbound by default)
+SG=$(aws ec2 describe-security-groups \
+  --filters Name=vpc-id,Values=$VPC \
+  --query 'SecurityGroups[0].GroupId' --output text)
+
+aws ec2 authorize-security-group-ingress --group-id $SG \
+  --protocol tcp --port 22 --cidr 0.0.0.0/0
+
+aws ec2 authorize-security-group-ingress --group-id $SG \
+  --protocol icmp --port -1 --cidr 0.0.0.0/0
 ```

 ## 4. Launch Instance
@@ -1135,6 +1168,37 @@ ip route show
   sudo ovn-nbctl lr-nat-list vpc-$VPC
   ```

+### Instance Has Public IP But No Internet
+
+The instance shows a public IP in `describe-instances` but cannot reach the
+internet, and inbound connections hang. The usual cause is a missing default
+route: the subnet's effective route table has no `0.0.0.0/0` route to the IGW,
+so Spinifex gates the subnet with a drop policy.
+
+```bash
+# Find the route table that applies to the subnet and check its routes
+aws ec2 describe-route-tables \
+  --filters Name=association.subnet-id,Values=$SUBNET \
+  --query 'RouteTables[0].Routes'
+# Expect a route with DestinationCidrBlock 0.0.0.0/0 and a GatewayId of igw-...
+```
+
+If the subnet has no explicit association it falls back to the VPC's main route
+table — check that one too, then add the route to whichever table applies:
+
+```bash
+aws ec2 create-route --route-table-id $RT \
+  --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW
+```
+
+On the host, the drop policy installed when a subnet lacks an IGW route appears
+as a `Logical_Router_Policy` on the VPC router:
+
+```bash
+sudo ovn-nbctl lr-policy-list vpc-$VPC
+# A drop policy matching the subnet CIDR means the subnet is gated.
+```
+
 ### Public IP Not Reachable from WAN

 Work through these checks in order — each eliminates a class of issues.
@@ -0,0 +1,188 @@
+---
+title: "ECR (Container Registry)"
+description: "Store and serve container images from Spinifex's AWS-compatible Elastic Container Registry — create a repository, authenticate Docker, push and pull images, and let your EKS workers pull from it, using the AWS CLI, the Spinifex console, or Terraform."
+category: "Containers"
+tags:
+  - ecr
+  - containers
+  - docker
+  - registry
+  - oci
+sections:
+  - overview
+  - prerequisites
+  - instructions
+  - troubleshooting
+resources:
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "Terraform AWS Provider — ECR"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ecr_repository"
+  - title: "OCI Distribution Spec"
+    url: "https://github.com/opencontainers/distribution-spec"
+  - title: "Docker CLI Reference"
+    url: "https://docs.docker.com/reference/cli/docker/"
+---
+
+# ECR (Container Registry)
+
+> Spinifex's ECR service is an AWS-compatible, OCI-compliant container registry. You create repositories with `aws ecr`, authenticate Docker with a registry token, and `docker push` / `docker pull` exactly as you would on AWS — and your EKS clusters can pull from it directly.
+
+## Overview
+
+ECR on Spinifex serves a private container registry on the Spinifex gateway endpoint — the same `host:9999` you already use for the AWS API. Authentication is account-scoped by token, so you push and pull at the gateway host and never need extra DNS:
+
+```
+<spinifex-gateway-host>:9999/<repository>
+```
+
+`aws ecr describe-repositories` and `aws ecr get-login-password` return the exact host to use — always prefer that value. Where real DNS is configured, the AWS-parity per-account host `<account-id>.dkr.ecr.<region>.<your-spinifex-domain>:9999` also works.
+
+Each repository holds image manifests and their layers. Images are stored in object storage ([Predastore](https://github.com/mulgadc/predastore)) in a per-account bucket, with blobs content-addressed and de-duplicated across repositories. Authentication is token-based: `aws ecr get-login-password` mints a short-lived bearer token that `docker login` uses against the registry's `/v2/` endpoint.
+
+**What works today**
+
+- Repository lifecycle — `CreateRepository`, `DeleteRepository`, `DescribeRepositories`.
+- OCI push and pull — blobs, manifests, and tags, including cross-repository blob mounts.
+- Token auth — `GetAuthorizationToken` plus bearer/basic auth on `/v2/`.
+- Repository policies — set, get, and delete a repository policy document.
+
+**Current limitations**
+
+- **Image scanning is not supported** — scan APIs return an explicit "operation not supported" error.
+- **No automatic garbage collection yet** — deleting images does not yet reclaim underlying blobs.
+- Lifecycle-policy enforcement and tag-immutability enforcement are not active.
+
+## Prerequisites
+
+- **Spinifex running**, with the AWS CLI configured for the `spinifex` profile (see [Installing Spinifex](/docs/install)).
+- **Docker** installed locally to build, push, and pull images.
+- **Docker trusting the Spinifex CA.** The registry serves TLS signed by the Spinifex local CA. Install it into the host trust store once (Docker uses the host root pool for registry TLS, so this covers every account endpoint without per-host config):
+
+```bash
+sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-local-ca.crt
+sudo update-ca-certificates
+sudo systemctl restart docker
+```
+
+- **The registry endpoint** — the Spinifex gateway `host:9999`. `aws ecr get-login-password` and `aws ecr describe-repositories` return the exact `host:port`; copy it from there rather than constructing it by hand.
+- **For EKS pulls:** worker nodes need the `AmazonEC2ContainerRegistryReadOnly` policy on their node IAM role (the EKS prerequisites already include this) and network egress to the registry endpoint.
+
+## Instructions
+
+Create a repository and push an image using your preferred tool.
+
+:::tabs
+@tab AWS CLI
+
+### 1. Create a repository
+
+```bash
+export AWS_PROFILE=spinifex
+
+aws ecr create-repository --repository-name my-app
+aws ecr describe-repositories --repository-names my-app \
+  --query 'repositories[0].repositoryUri'
+```
+
+The `repositoryUri` is the value you tag and push to — the gateway `host:9999`
+with the repository path, port included (for example `192.0.2.10:9999/my-app`).
+Use it verbatim; don't hand-build a hostname.
+
+### 2. Authenticate Docker
+
+Take the registry host straight from the API so the `:9999` port is correct:
+
+```bash
+REGISTRY=$(aws ecr describe-repositories --repository-names my-app \
+  --query 'repositories[0].repositoryUri' --output text | cut -d/ -f1)
+
+aws ecr get-login-password --region <region> \
+  | docker login --username AWS --password-stdin "$REGISTRY"
+```
+
+### 3. Build, tag, and push
+
+```bash
+docker build -t my-app:latest .
+docker tag my-app:latest "$REGISTRY/my-app:latest"
+docker push "$REGISTRY/my-app:latest"
+```
+
+### 4. Verify and pull
+
+```bash
+aws ecr describe-images --repository-name my-app
+docker pull "$REGISTRY/my-app:latest"
+```
+
+@tab Spinifex UI
+
+From the left navigation open **ECR → Repositories**.
+
+<img src="ecr-list-repositories.png" alt="The ECR Repositories list in the Spinifex console, showing a repository with its registry URI, tag mutability, and creation date">
+
+### 1. Create a repository
+
+1. Click **Create Repository**.
+2. Enter the **repository name** and, optionally, set **tag mutability** (mutable or immutable).
+3. Submit — the repository appears in the list with its full registry URI.
+
+### 2. Copy the push commands
+
+Open the repository to see its detail page. The **Push commands** panel lists the exact `aws ecr get-login-password | docker login`, `docker build`, `docker tag`, and `docker push` commands, pre-filled with your registry host and repository URI — use the **Copy** button and run them locally.
+
+<img src="ecr-push-commands.png" alt="A repository detail page in the Spinifex console, with the Push commands panel showing the docker login, build, tag, and push commands">
+
+### 3. Manage the repository
+
+- The **Permissions** tab edits the repository's access policy.
+- The **Lifecycle** tab manages retention rules, and **Tag immutability** can be toggled from the detail page.
+- The **Scan** tab reflects that image scanning is not supported on Spinifex.
+- Delete a repository from the **Delete** action in the repositories list.
+
+@tab Terraform
+
+Repositories can be managed as code with the standard `aws_ecr_repository` resource, with the AWS provider pointed at Spinifex's `ecr` endpoint:
+
+```hcl
+provider "aws" {
+  region                      = "ap-southeast-2"
+  skip_credentials_validation = true
+  skip_requesting_account_id  = true
+
+  endpoints {
+    ecr = "https://<your-spinifex-gateway>"
+  }
+}
+
+resource "aws_ecr_repository" "my_app" {
+  name = "my-app"
+}
+
+output "repository_url" {
+  value = aws_ecr_repository.my_app.repository_url
+}
+```
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+```
+
+Then authenticate Docker and push using the `repository_url` output, following steps 2–4 in the AWS CLI tab.
+
+:::
+
+## Troubleshooting
+
+**`docker login` fails with an authorization error.** The token is account-scoped and short-lived. Re-run `aws ecr get-login-password` to mint a fresh token, and confirm the registry host in `docker login` matches your account's endpoint (`aws ecr describe-repositories` returns it).
+
+**`docker push` cannot reach the registry.** Use the exact host from `aws ecr describe-repositories` — the Spinifex gateway `host:9999`. Docker dials it directly, so the `:9999` port must be present (without it docker tries `:443` and fails). If you are using the AWS-parity `<account-id>.dkr.ecr…` hostname instead, that path needs DNS resolving it to the gateway — prefer the gateway host the API returns.
+
+**EKS workers cannot pull an image.** Confirm the node IAM role has `AmazonEC2ContainerRegistryReadOnly` and that the workers have egress to the registry endpoint (an Internet Gateway or NAT Gateway route). See the [EKS prerequisites](/docs/eks).
+
+**Image-scanning commands return an error.** Image scanning is not supported on Spinifex; the scan APIs intentionally reject these calls. Remove scan-on-push configuration from your tooling.
+
+**A pushed image still appears after deletion frees no space.** Garbage collection of unreferenced blobs is not yet automatic — deleting an image removes its manifest and tags but does not immediately reclaim the underlying layers.
@@ -0,0 +1,248 @@
+---
+title: "EKS (Managed Kubernetes)"
+description: "Run a managed, AWS-compatible Kubernetes cluster on Spinifex — provision the control plane and worker node group, wire up the VPC, IAM, and security groups it needs, and deploy your first workload via the AWS CLI, the Spinifex console, or Terraform."
+category: "Containers"
+tags:
+  - eks
+  - kubernetes
+  - containers
+  - vpc
+  - iam
+sections:
+  - overview
+  - prerequisites
+  - instructions
+  - troubleshooting
+resources:
+  - title: "EKS Quickstart Workbook"
+    url: "https://github.com/mulgadc/spinifex/tree/main/docs/terraform-workbooks/eks-quickstart"
+  - title: "EKS HTTPS Ingress Workbook (LBC + ACM)"
+    url: "https://github.com/mulgadc/spinifex/tree/main/docs/terraform-workbooks/eks-https-ingress"
+  - title: "EKS GitOps Workbook (Argo CD + EBS-CSI)"
+    url: "https://github.com/mulgadc/spinifex/tree/main/docs/terraform-workbooks/eks-gitops-argocd"
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "kubectl Documentation"
+    url: "https://kubernetes.io/docs/reference/kubectl/"
+---
+
+# EKS (Managed Kubernetes)
+
+> Spinifex's EKS service gives you a managed, AWS-compatible Kubernetes API: you create a cluster and a node group, and Spinifex provisions and runs the control plane and workers for you. Standard tools — `aws eks`, `kubectl`, the Terraform AWS provider — work unchanged against your cluster.
+
+## Overview
+
+An EKS cluster on Spinifex has two parts: a **control plane** (the managed Kubernetes API server, provisioned on a Spinifex-managed VM and fronted by a network load balancer) and one or more **managed node groups** of worker VMs that run your pods. You provide the VPC, subnets, and IAM roles; Spinifex creates and reconciles the cluster, its load balancer, and its security groups.
+
+Authentication uses **EKS access entries** (the API authentication mode) — IAM principals are mapped to Kubernetes RBAC groups, and `kubectl` authenticates with `aws eks get-token` exactly as it does on AWS.
+
+**What you'll create:**
+
+| Resource | Purpose |
+|---|---|
+| VPC + subnets | The network the cluster and workers run in |
+| Internet Gateway / NAT Gateway | Egress so workers can pull container images |
+| Cluster IAM role | Lets the control plane manage cluster resources |
+| Node IAM role | Lets workers join the cluster and pull images |
+| EKS cluster | The managed Kubernetes API endpoint |
+| Managed node group | The worker VMs that run your pods |
+| Security-group rule | Opens your app's port to reach the workers |
+
+**Spinifex specifics**
+
+- **Auth mode is API-only.** `authentication_mode` must be `API`; the legacy `CONFIG_MAP` and `API_AND_CONFIG_MAP` modes are rejected. Grant access with access entries, not an `aws-auth` ConfigMap.
+- **Security groups are auto-managed.** Spinifex creates the cluster and node-group security groups deterministically (e.g. `eks-cluster-<name>-nodegroup-sg`); `vpc_config.security_group_ids` is ignored. To expose a workload you add an ingress rule to the auto-managed node-group SG.
+- **The worker image is fixed.** Node groups always boot Spinifex's `eks-node` image. `ami_type` is recorded but does not select the image, and the image must be registered on the cluster before you create a node group.
+- **Default Kubernetes version is `1.32`.**
+
+## Prerequisites
+
+Before creating a cluster you need the supporting infrastructure in place. The Terraform workbooks build all of this for you; if you are using the CLI or console, create it first.
+
+- **Spinifex running**, with the AWS CLI configured for the `spinifex` profile (see [Installing Spinifex](/docs/install)) and `kubectl` installed.
+- **The `eks-node` image** registered on the cluster. The console blocks cluster creation until it is present.
+- **A VPC with at least one subnet.** The control plane launches in your first subnet; workers run in the subnets you pass to the node group.
+- **Egress for the workers.** Workers must reach the internet to pull container images:
+  - **Public workers** — an **Internet Gateway** with a default route (`0.0.0.0/0 → IGW`) and public IPs.
+  - **Private workers** — a **NAT Gateway** in a public subnet with a default route (`0.0.0.0/0 → NAT`) from the private subnets.
+- **Two IAM roles:**
+  - A **cluster role** trusted by `eks.amazonaws.com` with `AmazonEKSClusterPolicy` attached.
+  - A **node role** trusted by `ec2.amazonaws.com` with `AmazonEKSWorkerNodePolicy`, `AmazonEKS_CNI_Policy`, and `AmazonEC2ContainerRegistryReadOnly` attached (the last lets workers pull images from [ECR](/docs/ecr)).
+
+## Instructions
+
+The same cluster can be created three ways. Pick your tool — each path reaches the same working cluster.
+
+:::tabs
+@tab AWS CLI
+
+### 1. Create the cluster
+
+Point the cluster at your VPC subnets and the cluster role, with API authentication and a public endpoint:
+
+```bash
+export AWS_PROFILE=spinifex
+
+aws eks create-cluster \
+  --name demo \
+  --role-arn arn:aws:iam::000000000000:role/eks-cluster-role \
+  --resources-vpc-config subnetIds=subnet-aaaa,subnet-bbbb,endpointPublicAccess=true \
+  --access-config authenticationMode=API \
+  --kubernetes-version 1.32
+```
+
+Wait for it to become active:
+
+```bash
+aws eks wait cluster-active --name demo
+aws eks describe-cluster --name demo --query 'cluster.status'
+```
+
+### 2. Add a node group
+
+```bash
+aws eks create-nodegroup \
+  --cluster-name demo \
+  --nodegroup-name default \
+  --node-role-arn arn:aws:iam::000000000000:role/eks-node-role \
+  --subnets subnet-aaaa subnet-bbbb \
+  --scaling-config minSize=1,maxSize=2,desiredSize=1 \
+  --instance-types t3.medium
+
+aws eks wait nodegroup-active --cluster-name demo --nodegroup-name default
+```
+
+### 3. Connect kubectl
+
+```bash
+aws eks update-kubeconfig --name demo
+kubectl get nodes
+```
+
+### 4. Expose a workload
+
+Workers join an auto-managed security group that only allows intra-cluster traffic. To reach a NodePort service, add one ingress rule to that SG (look it up by its deterministic name):
+
+```bash
+SG=$(aws ec2 describe-security-groups \
+  --filters Name=group-name,Values=eks-cluster-demo-nodegroup-sg \
+  --query 'SecurityGroups[0].GroupId' --output text)
+
+aws ec2 authorize-security-group-ingress \
+  --group-id "$SG" --protocol tcp --port 30080 --cidr 0.0.0.0/0
+```
+
+Then deploy and publish your app:
+
+```bash
+kubectl create deployment hello --image=nginxdemos/hello --replicas=2
+kubectl expose deployment hello --type=NodePort --port=80 \
+  --overrides='{"spec":{"ports":[{"port":80,"nodePort":30080}]}}'
+```
+
+@tab Spinifex UI
+
+The Spinifex console drives the same workflow through a guided form. From the left navigation open **EKS → Clusters**.
+
+<img src="eks-list-clusters.png" alt="The EKS Clusters list in the Spinifex console, with a Create Cluster button">
+
+### 1. Create the cluster
+
+1. Click **Create Cluster**.
+2. Enter a **name** and choose the **Kubernetes version** (default `1.32`).
+3. Choose the **cluster IAM role**, or create one inline from the dialog.
+4. Select the **VPC and subnets** the cluster will run in.
+5. Set **endpoint access** (public, private, or both) and submit.
+
+<img src="eks-create-cluster.png" alt="The Create Cluster form in the Spinifex console, showing name, Kubernetes version, IAM role, VPC, subnets, and endpoint access fields">
+
+The form notes that the authentication mode is API and that the legacy aws-auth ConfigMap is not supported. The cluster appears in the **Clusters** list as `CREATING` and flips to `ACTIVE` once the control plane is healthy.
+
+### 2. Add a node group
+
+1. Open the cluster and switch to the **Node groups** tab.
+2. Click **Add node group**, choose the **node IAM role**, **subnets**, **instance type**, and **scaling** (desired / min / max).
+3. Submit and wait for the node group to reach `ACTIVE`.
+
+### 3. Manage access and addons
+
+- The **Access** tab lists IAM principals and their cluster-access policies — add an access entry to let another user or role reach the cluster.
+- The **Addons** tab installs and shows the health of managed addons.
+- The **Networking** tab shows the cluster's VPC, subnets, endpoint access, and the auto-managed security groups.
+
+To run `kubectl` against the cluster, follow the AWS CLI tab's `update-kubeconfig` step.
+
+@tab Terraform
+
+The Terraform workbooks are the fastest way to a working cluster — they build the VPC, IAM roles, cluster, node group, and a demo workload in one `apply`. Start with **eks-quickstart**:
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-quickstart
+```
+
+The cluster config points the AWS provider at Spinifex's `eks`, `ec2`, `iam`, and `sts` endpoints, then creates the cluster and node group:
+
+```hcl
+resource "aws_eks_cluster" "this" {
+  name     = "eks-quickstart"
+  role_arn = aws_iam_role.cluster.arn
+  version  = "1.32"
+
+  access_config {
+    authentication_mode = "API"
+  }
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.a.id, aws_subnet.b.id]
+    endpoint_public_access  = true
+  }
+}
+
+resource "aws_eks_node_group" "default" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "default"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.a.id, aws_subnet.b.id]
+
+  scaling_config {
+    desired_size = 1
+    min_size     = 1
+    max_size     = 2
+  }
+
+  instance_types = ["t3.medium"]
+}
+```
+
+Apply it, then connect:
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+aws eks update-kubeconfig --name eks-quickstart
+kubectl get nodes
+```
+
+The workbooks form a ladder. **eks-quickstart** is a NodePort demo; **eks-https-ingress** adds private subnets, a NAT gateway, and HTTPS via the AWS Load Balancer Controller + ACM; **eks-gitops-argocd** delivers the app from a git repo with the Argo CD addon and persists state on an EBS-CSI (Viperblock-backed) volume. See the full templates linked in Resources.
+
+:::
+
+## Troubleshooting
+
+**Cluster stuck in `CREATING`.** The control plane is still booting. Check `aws eks describe-cluster --name <name> --query 'cluster.health'` for reported issues. The control plane needs egress to bootstrap, so confirm its subnet has an Internet Gateway route.
+
+**`kubectl` cannot connect.** Re-run `aws eks update-kubeconfig --name <name>` to refresh the endpoint and CA, and confirm the cluster is `ACTIVE`. `aws sts get-caller-identity` verifies your credentials reach the Spinifex endpoint.
+
+**Workers stay `NotReady`.** Almost always an egress or IAM problem:
+- Confirm the workers can pull images — a default route to an Internet Gateway (public) or NAT Gateway (private).
+- Confirm the node role has `AmazonEKSWorkerNodePolicy`, `AmazonEKS_CNI_Policy`, and `AmazonEC2ContainerRegistryReadOnly`.
+
+**Workload unreachable from outside the cluster.** The node-group security group only allows intra-cluster traffic by default. Add an ingress rule for your NodePort to `eks-cluster-<name>-nodegroup-sg`, as shown in the CLI tab.
+
+**`create-cluster` rejected with an authentication-mode error.** `authentication_mode` must be `API`. Remove any `CONFIG_MAP` / `API_AND_CONFIG_MAP` setting and grant access with access entries instead.
+
+**Addon install fails.** Only addons bundled into the registered `eks-node` image install successfully. If you pin an addon version in Terraform, leave `addon_version` unset so Spinifex selects the catalog default.
@@ -34,7 +34,7 @@ resources:
 ## Overview

 This guide assumes Spinifex is already installed and running. If not, follow one of the installation guides first:
- [Single-Node Install)](/docs/install)
+- [Single-Node Install](/docs/install)
 - [Multi-Node Install](/docs/install-multi-node)
 - [Source Install](/docs/install-source)

@@ -50,7 +50,9 @@ export AWS_PROFILE=spinifex

 ## 1. Import an AMI

-List available images and import one matching your architecture:
+### Option A: Import a bundled image
+
+List the bundled images and import one matching your architecture:

 ```bash
 spx admin images list
@@ -64,13 +66,11 @@ ubuntu-26.04-arm64      | ubuntu | 26.04   | arm64  | uefi
 ubuntu-26.04-x86_64     | ubuntu | 26.04   | x86_64 | uefi
 ```

-Import an image:
-
 ```bash
 spx admin images import --name ubuntu-26.04-x86_64
 ```

-Or import a local image file:
+### Option B: Import a local image file

 ```bash
 spx admin images import --file ~/images/ubuntu-26.04.img --distro ubuntu --version 26.04 --arch x86_64 --boot-mode uefi
@@ -79,7 +79,7 @@ spx admin images import --file ~/images/ubuntu-26.04.img --distro ubuntu --versi
 Verify the import and note the AMI ID:

 ```bash
-aws ec2 describe-images
+AMI_ID=$(aws ec2 describe-images --query 'Images[0].ImageId' --output text)
 ```

 ## 2. Create an SSH Key
@@ -115,8 +115,6 @@ aws ec2 describe-key-pairs
 ```bash
 VPC_ID=$(aws ec2 create-vpc --cidr-block 10.200.0.0/16 \
  --query 'Vpc.VpcId' --output text)
-
-echo "VPC: $VPC_ID"
 ```

 ### Create an Internet Gateway
@@ -130,68 +128,95 @@ IGW_ID=$(aws ec2 create-internet-gateway \
 aws ec2 attach-internet-gateway \
  --internet-gateway-id $IGW_ID \
  --vpc-id $VPC_ID
-
-echo "IGW: $IGW_ID (attached to $VPC_ID)"
 ```

-### Create a Public Subnet
+### Create a Subnet

-A public subnet auto-assigns a routable IP to each instance, making it directly reachable from your network.
+Create the subnet your instances will launch into. The routing and public-IP steps below are what make it a *public* subnet.

 ```bash
 SUBNET_ID=$(aws ec2 create-subnet \
  --vpc-id $VPC_ID \
  --cidr-block 10.200.1.0/24 \
  --query 'Subnet.SubnetId' --output text)
+```

-# Enable auto-assign public IP
+### Create a Route Table
+
+A subnet is only **public** if its route table has a default route to the Internet Gateway. Spinifex does **not** add this route automatically — a new VPC's route table only routes traffic within the VPC (matching AWS). Without a `0.0.0.0/0` route to the IGW, instances in the subnet cannot reach the internet (and inbound connections cannot complete) even with a public IP assigned.
+
+```bash
+RT_ID=$(aws ec2 create-route-table \
+  --vpc-id $VPC_ID \
+  --query 'RouteTable.RouteTableId' --output text)
+
+# Default route to the internet via the IGW
+aws ec2 create-route \
+  --route-table-id $RT_ID \
+  --destination-cidr-block 0.0.0.0/0 \
+  --gateway-id $IGW_ID
+
+# Associate the route table with the subnet
+aws ec2 associate-route-table \
+  --route-table-id $RT_ID \
+  --subnet-id $SUBNET_ID
+```
+
+### Enable Auto-Assign Public IP
+
+Give every instance launched into the subnet a routable public IP, making it directly reachable from your network.
+
+```bash
 aws ec2 modify-subnet-attribute \
  --subnet-id $SUBNET_ID \
  --map-public-ip-on-launch
-
-echo "Subnet: $SUBNET_ID (public)"
 ```

+### Allow SSH and ICMP
+
+Every VPC gets a default security group that **blocks inbound traffic from outside the group** (matching AWS). Instances launched without an explicit security group use this default, so you must authorize ingress before you can SSH or ping the instance.
+
+```bash
+SG_ID=$(aws ec2 describe-security-groups \
+  --filters Name=vpc-id,Values=$VPC_ID \
+  --query 'SecurityGroups[0].GroupId' --output text)
+
+# Allow SSH from anywhere
+aws ec2 authorize-security-group-ingress \
+  --group-id $SG_ID \
+  --protocol tcp --port 22 --cidr 0.0.0.0/0
+
+# Allow ICMP (ping) from anywhere
+aws ec2 authorize-security-group-ingress \
+  --group-id $SG_ID \
+  --protocol icmp --port -1 --cidr 0.0.0.0/0
+```
+
+**Note:** `0.0.0.0/0` opens these ports to every source. For anything beyond a quick evaluation, scope the `--cidr` to a trusted range. Rule changes apply immediately — no instance restart needed.
+
 ### Verify

 ```bash
 aws ec2 describe-vpcs --vpc-ids $VPC_ID
 aws ec2 describe-subnets --subnet-ids $SUBNET_ID
+aws ec2 describe-route-tables --route-table-ids $RT_ID
+aws ec2 describe-security-groups --group-ids $SG_ID
 ```

 ## 4. Launch an Instance

-Query available instance types for your hardware:
+Launch an instance in the public subnet on a `t3.micro` (2 vCPU / 1 GiB).
+
+> **Note:** On an **arm64** host, use `t4g.micro` instead of `t3.micro`.

 ```bash
-aws ec2 describe-instance-types
-```
-
-Launch an instance in the public subnet, note this will select an instance type with 2 VCPU and 1024MB of memory.
-
-> **Note:** Replace `AMI_NAME` with the previously imported image above with the `ami-` prefix.
-
-```bash
-AMI_NAME="ami-ubuntu-26.04-x86_64"
-
-AMI_ID=$(aws ec2 describe-images \
-  --filters "Name=name,Values=$AMI_NAME" \
-  --query 'Images | sort_by(@, &CreationDate) | [-1].ImageId' \
-  --output text)
-
-INSTANCE_TYPE=$(aws ec2 describe-instance-types \
-  --query "sort_by(InstanceTypes[?VCpuInfo.DefaultVCpus==\`2\` && MemoryInfo.SizeInMiB>=\`1024\`], &MemoryInfo.SizeInMiB)[0].InstanceType" \
-  --output text)
-
 INSTANCE_ID=$(aws ec2 run-instances \
  --image-id $AMI_ID \
-  --instance-type $INSTANCE_TYPE \
+  --instance-type t3.micro \
  --key-name spinifex-key \
  --subnet-id $SUBNET_ID \
  --count 1 \
  --query 'Instances[0].InstanceId' --output text)
-
-echo "Instance: $INSTANCE_ID"
 ```

 Wait for the instance to reach `running` state:
@@ -328,7 +353,36 @@ PRIVATE_SUBNET=$(aws ec2 create-subnet \
  --query 'Subnet.SubnetId' --output text)
 ```

-Instances in private subnets get a private IP only. They can still reach the internet via SNAT through the VPC router, but are not directly reachable from your network.
+Instances in a private subnet get a private IP only and are not reachable from your network. By default they also have **no internet access**: with no `0.0.0.0/0` route in the subnet's route table, Spinifex gates egress with a drop policy — matching AWS, where a private subnet has no route off the VPC. They can still reach other instances in the same VPC.
+
+To give a private subnet outbound-only internet access, deploy a NAT Gateway in a public subnet and point the private subnet's default route at it (the AWS pattern):
+
+```bash
+# Allocate an Elastic IP for the NAT Gateway
+NAT_EIP=$(aws ec2 allocate-address --query AllocationId --output text)
+
+# Create the NAT Gateway in the PUBLIC subnet (the one with the IGW route)
+NATGW_ID=$(aws ec2 create-nat-gateway \
+  --subnet-id $SUBNET_ID \
+  --allocation-id $NAT_EIP \
+  --query 'NatGateway.NatGatewayId' --output text)
+
+# Give the private subnet its own route table with a default route to the NAT Gateway
+PRIVATE_RT=$(aws ec2 create-route-table \
+  --vpc-id $VPC_ID \
+  --query 'RouteTable.RouteTableId' --output text)
+
+aws ec2 create-route \
+  --route-table-id $PRIVATE_RT \
+  --destination-cidr-block 0.0.0.0/0 \
+  --nat-gateway-id $NATGW_ID
+
+aws ec2 associate-route-table \
+  --route-table-id $PRIVATE_RT \
+  --subnet-id $PRIVATE_SUBNET
+```
+
+Instances in the private subnet now reach the internet outbound through the NAT Gateway's public IP, but remain unreachable from the WAN.

 ### Multiple Accounts

@@ -348,7 +402,7 @@ journalctl -u spinifex-daemon -f
 aws ec2 describe-images
 ```

-### SSH Connection Refused
+### SSH Connection Refused or Times Out

 cloud-init needs 30-60 seconds after boot. Check instance state:

@@ -356,6 +410,20 @@ cloud-init needs 30-60 seconds after boot. Check instance state:
 aws ec2 describe-instances --instance-ids $INSTANCE_ID
 ```

+A connection that **times out** (rather than being refused) usually means the security group is blocking port 22. Confirm the default security group allows SSH ingress:
+
+```bash
+aws ec2 describe-security-groups --group-ids $SG_ID \
+  --query 'SecurityGroups[0].IpPermissions'
+```
+
+If there are no rules for TCP 22, authorize it (see [Allow SSH and ICMP](#allow-ssh-and-icmp)):
+
+```bash
+aws ec2 authorize-security-group-ingress \
+  --group-id $SG_ID --protocol tcp --port 22 --cidr 0.0.0.0/0
+```
+
 ### No Public IP Assigned

 Verify the subnet has `MapPublicIpOnLaunch` enabled:
@@ -378,7 +446,23 @@ aws ec2 describe-internet-gateways

 ### Instance Has No Internet Access

-Check the VPC router's NAT rules (from the host):
+First confirm the subnet's route table has a default route to the IGW. Spinifex gates a subnet's egress with a drop policy when this route is missing, so an instance with a public IP still cannot reach the internet:
+
+```bash
+aws ec2 describe-route-tables \
+  --filters Name=association.subnet-id,Values=$SUBNET_ID \
+  --query 'RouteTables[0].Routes'
+# Expect a 0.0.0.0/0 route with a GatewayId of igw-...
+```
+
+If the route is missing, add it (see [Create a Route Table](#create-a-route-table)):
+
+```bash
+aws ec2 create-route --route-table-id $RT_ID \
+  --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW_ID
+```
+
+If the route is present, check the VPC router's NAT rules (from the host):

 ```bash
 sudo ovn-nbctl lr-nat-list $(sudo ovn-nbctl lr-list | awk '{print $2}' | head -1)
@@ -12,8 +12,6 @@ For this tutorial download UTM.app on Mac to simplify the VM installation.

 * UTM - [https://mac.getutm.app](https://mac.getutm.app)

-
-
 ## Download Debian ISO

 Next, download the Debian ISO (ARM64, Apple Silicon)
@@ -28,7 +26,7 @@ Using the VM console, login using the `spinifex` user and your specified passwor

 For a minimal Debian install additional dependencies are required prior to the Spinifex installation:

-```
+```bash
 su root
 apt update
 apt install sudo curl bridge-utils
@@ -38,7 +36,7 @@ apt install sudo curl bridge-utils

 Enable sudo for `spinifex` user as the root user:

-```
+```bash
 su root
 echo "spinifex ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/spinifex
 ```
@@ -26,6 +26,8 @@ sections:

 ## Overview

+Spinifex is an open-source infrastructure platform that brings core AWS services including EC2, EBS and S3 to bare-metal, edge, and on-prem environments. It exposes an EC2-compatible API, so any tooling that works against AWS (the `aws` CLI, Terraform, SDKs) works against a Spinifex node unchanged, with a single profile swap.
+
 This document serves as a tutorial for setting up multi-tenant AI workloads with Spinifex. The reference architecture for this tutorial is the Supermicro H14 platform - specifications listed below.

 The H14 chassis comes equipped with **8× AMD Instinct MI350X** (288 GB HBM3e each, 2.3 TB total). We used Spinifex to provision three isolated VMs, each assigned 2× MI350X via direct PCIe passthrough, and ran three computationally intensive workloads simultaneously: a YOLO11x vision model, a Qwen3-VL 235B FP8 vision language model, and a Llama 3.3 70B large language model.
@@ -55,9 +57,9 @@ GPU passthrough requires a host kernel and OS that supports vfio-pci. Ubuntu 24.
 ## Prerequisites

 - Supermicro H14 with 8× AMD Instinct MI350X installed
- Host OS: **Ubuntu 24.04 LTS** (kernel 6.8+) or **Debian 13** (kernel 6.12+) — minimum for vfio-pci support
+- Host OS: **Ubuntu 26.04 LTS** (kernel 6.8+) or **Debian 13** (kernel 6.12+) — minimum for vfio-pci support
 - Spinifex installed and all services running (`systemctl status spinifex.target`)
- AMD GPU AMI registered (`ami-ubuntu-amd-gpu`) — Ubuntu 26.04 LTS with ROCm-compatible kernel
+- AMD GPU AMI registered (`ubuntu-26.04-amd-gpu-x86_64`) — Ubuntu 26.04 LTS with ROCm-compatible kernel
 - AWS CLI configured with `AWS_PROFILE=spinifex` pointing at the Spinifex endpoint
 - SSH key pair imported into Spinifex, VPC and security group created (see [Launching Instances](/docs/launching-instances))

@@ -0,0 +1,556 @@
+---
+title: "Mixed AI Workloads on a Single H200 Chassis: Guest-Managed MIG"
+description: "Install Spinifex from source, configure host-local networking, attach Predastore storage, and run four concurrent AI workloads across MIG partitions managed inside guest VMs."
+category: "Reference Architectures"
+tags:
+  - nvidia
+  - h200
+  - mig
+  - vllm
+  - yolo
+  - predastore
+  - bare-metal
+resources:
+  - title: "Install from Source"
+    url: "/docs/install-source"
+  - title: "VPC Networking"
+    url: "/docs/vpc-networking"
+  - title: "NVIDIA MIG User Guide"
+    url: "https://docs.nvidia.com/datacenter/tesla/mig-user-guide/"
+sections:
+  - overview
+  - prerequisites
+  - instructions
+---
+
+## Overview
+
+Spinifex is an open-source infrastructure platform that brings core AWS services including EC2, EBS and S3 to bare-metal, edge, and on-prem environments. It exposes an AWS compatible API, so any tooling that works against AWS (the `aws` CLI, Terraform, SDKs) works against a Spinifex node unchanged, with a single profile swap.
+
+This guide documents a full bare-metal AI deployment on a single [Supermicro X13](https://www.supermicro.com/en/products/x13) chassis using NVIDIA H200 SXM GPUs. These GPUs include the [NVIDIA Multi-Instance GPU](https://www.nvidia.com/en-au/technologies/multi-instance-gpu/) (MIG) capability, which allows a single GPU to be "sliced" into up to seven independent GPU partitions that are hardware isolated. Each partition is capable of running its own workloads with reserved resources from the "host" GPU.
+
+<p align="center"><img src="../../../.github/assets/images/h200/X13.png" alt="Supermicro X13 8U GPU System"></p>
+
+In this setup, each EC2 instance receives an entire H200 via PCIe passthrough and manages its own MIG partitions. This gives each tenant full control over how they slice their GPU — including the ability to run heterogeneous workloads at different partition sizes on the same physical card.
+
+A key consideration for this example is that we do not have access to the router upstream of the X13 host, or knowledge of potentially available IP addresses for the VMs we provision on it. The first step in the Instructions section outlines how we attach our own local IP address range to the bridge on the host, effectively creating a pseudo-airgapped environment - although in this specific case outbound connectivity is still required for remote access to the host, downloading required packages etc, with this configuration it is *not* required for host/EC2 instance communication.
+
+### EC2 workload layout
+
+| EC2 Workload | IP | MIG config | Workload | Model |
+|---|---|---|---|---|
+| `vm-llama3b` | 192.168.10.X  | 7 × 1g.18gb | Chat inference × 7 | Llama-3.2-3B-Instruct |
+| `vm-qwen32b` | 192.168.10.X  | 2 × 3g.71gb | Chat inference × 2 | Qwen2.5-32B-Instruct |
+| `vm-llama70b` | 192.168.10.X | 1 × 7g.141gb | Chat inference × 1 | Llama-3.1-70B-FP8 |
+| `vm-yolo`    | 192.168.10.X | 2 × 3g.71gb | Object detection × 2 | YOLO11x + YOLO11s |
+
+Twelve concurrent inference endpoints in total: 7 fast 3B slots, 2 mid-tier 32B slots, 1 full-GPU 70B slot, and 2 real-time vision streams running side-by-side to compare detection models.
+
+
+### Platform
+
+| Component | Specification |
+|---|---|
+| **Chassis** | Supermicro X13 8U GPU System |
+| **CPUs** | 2× Intel Xeon Platinum 8568Y+ (48 cores each, 96 cores / 192 threads total) |
+| **RAM** | 2 TB DDR5-4800 (32× 64 GB SK Hynix DIMMs) |
+| **Storage** | 4× 7.68 TB KIOXIA CD6 NVMe SSDs (30.72 TB raw) |
+| **GPUs** | 8× NVIDIA H200 SXM5 (141 GiB HBM3e per GPU, ~1.13 TB total) — 4 used in this demo |
+| **Host OS** | Ubuntu 26.04 LTS |
+| **Orchestration** | Spinifex — EC2-compatible bare-metal API |
+| **Guest OS** | Ubuntu 26.04 LTS |
+| **GPU partitioning** | NVIDIA MIG — managed inside each guest VM |
+| **Block storage** | Predastore — S3-compatible, NVMe-backed |
+| **Container runtime** | Docker |
+| **Inference runtime** | vLLM (`vllm-openai` image from local registry) |
+| **Vision runtime** | Ultralytics YOLO11 |
+
+
+## Prerequisites
+
+### 1. Verify GPU visibility on host
+Before making any changes, verify that the host can see its GPUs:
+
+<img src="../../../.github/assets/images/h200/nvidia-smi.png" alt="NVIDIA SMI">
+
+
+## Instructions
+
+
+### 1. Configure host-local VPC networking
+
+Spinifex utilises bridged networking via OVN. In this example, the remote host exposes only one public IP address, attached to a single physical NIC. Before provisioning Spinifex, we must first ensure that `br-wan` exists and enslave the physical NIC to it.
+
+Since we have no access to an upstream router or knowledge of available IP address pools, we must also attach our own `192.168.10.0/24` address range to `br-wan` which will be used by the guest VMs to communicate with the host and each other.
+
+For example:
+
+```yaml
+# /etc/netplan/…
+bridges:
+  br-wan:
+    addresses:
+      - 192.168.10.1/24      # VM pool gateway — host-local
+      - 198.51.100.10/24     # existing WAN IP — unchanged
+    routes:
+      - to: default
+        via: 198.51.100.1
+```
+
+Then apply with `sudo netplan apply`. VMs, once provisioned, are reachable from the host at
+`192.168.10.x`. For internet access through the host's WAN interface:
+
+```bash
+sysctl -w net.ipv4.ip_forward=1
+iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o br-wan -j MASQUERADE
+```
+The exact process for this is described in the [VPC Networking](/docs/vpc-networking#host-local-subnet-no-upstream-router) guide.
+
+
+
+### 2. Install Spinifex
+
+Follow the [Single Node Install](/docs/install) guide. This process will install Spinifex and start Spinifex services, however `spinifex.toml` needs to be edited to finalise the changes made to the networking in the previous section, as described in the following section.
+
+### 3. Configure spinifex.toml and restart services
+
+Edit `/etc/spinifex/spinifex.toml` to point the external pool and VPCD at the bridge created in step 1:
+
+```toml
+[network]
+external_mode = "pool"
+
+[[network.external_pools]]
+name        = "wan"
+source      = "static"         # required — no upstream DHCP for this range
+range_start = "192.168.10.2"
+range_end   = "192.168.10.100"
+gateway     = "192.168.10.1"   # second address on br-wan
+prefix_len  = 24
+dns_servers = ["8.8.8.8"]
+```
+
+Then restart all services:
+
+```bash
+sudo systemctl start spinifex.target
+sudo systemctl status spinifex.target
+```
+
+### 4. Attach Predastore storage
+
+Spinifex distributes the Predastore object storage volume across multiple nodes using Reed–Solomon encoding for data redundancy. This chassis has four NVMe drives — one for the OS, three dedicated to data — so we back each Predastore storage node with its own physical drive, giving us fault-tolerant distributed storage on a single machine — conceptually similar to RAID 5, where data and parity are spread across drives so a single drive failure is recoverable.
+
+Confirm drive assignments with `lsblk` before proceeding, as device names vary between systems.
+
+```bash
+lsblk   # identify the OS drive and the three data drives
+
+# Stop services so Predastore isn't writing while we relocate its data directories
+sudo systemctl stop spinifex.target
+
+# --- Repeat the block below for each data drive (node-1/nvme-1, node-2/nvme-2, node-3/nvme-3) ---
+
+# Mount the physical drive at a stable path
+sudo mkdir -p /mnt/nvme-1
+sudo mount /dev/nvme1n1 /mnt/nvme-1
+sudo mkdir -p /mnt/nvme-1/nodes /mnt/nvme-1/db
+
+# Move Predastore's node data and metadata off the OS drive onto the physical NVMe
+sudo mv /var/lib/spinifex/predastore/distributed/nodes/node-1 /mnt/nvme-1/nodes/node-1
+sudo mv /var/lib/spinifex/predastore/distributed/db/node-1    /mnt/nvme-1/db/node-1
+
+# Symlink the original paths back so Predastore finds its data unchanged
+sudo ln -s /mnt/nvme-1/nodes/node-1 /var/lib/spinifex/predastore/distributed/nodes/node-1
+sudo ln -s /mnt/nvme-1/db/node-1    /var/lib/spinifex/predastore/distributed/db/node-1
+
+# Persist the mount across reboots
+echo "/dev/nvme1n1  /mnt/nvme-1  auto  defaults  0  2" | sudo tee -a /etc/fstab
+
+# --- End of per-drive block ---
+
+sudo systemctl start spinifex.target
+```
+
+Verify the nodes are healthy before proceeding:
+
+```bash
+export AWS_PROFILE=spinifex
+
+aws s3 ls --endpoint-url https://localhost:8443
+# Should return without error (empty bucket list is fine)
+```
+
+### 5. Enable GPU Passthrough
+Spinifex allows GPUs to be utilised by guest VMS via PCIe-passthrough. This can be enabled via `sudo spx admin gpu setup`. Then, after a reboot, run `sudo spx admin gpu enable`. The Spinifex banner should update to reflect GPU passthrough state after every step:
+
+<p align="center"><img src="../../../.github/assets/images/h200/spinifex-banner1.png" alt="GPU Enabled"></p>
+
+
+### 6. Import the GPU AMI
+
+The demo uses the standard Spinifex NVIDIA GPU AMI (`ubuntu-26.04-nvidia-gpu-x86_64`), which includes:
+
+- Ubuntu 26.04 LTS guest image
+- NVIDIA server driver (DKMS pre-built against the pinned kernel)
+- Docker CE + nvidia-container-toolkit (`--gpus` support enabled at boot)
+- Python 3 + venv, common utilities (tmux, curl, ffmpeg, etc.)
+
+```bash
+spx admin images import --name ubuntu-26.04-nvidia-gpu-x86_64
+```
+
+Confirm and set the AMI ID:
+```bash
+AMI_ID=$(aws ec2 describe-images \
+    --filters "Name=name,Values=ubuntu-26.04-nvidia-gpu-x86_64" \
+    --query 'Images[0].ImageId' --output text)
+```
+
+### 7. Create VPC resources
+
+```bash
+# Create VPC and subnet
+VPC_ID=$(aws ec2 create-vpc --cidr-block 192.168.10.0/24 \
+    --query 'Vpc.VpcId' --output text)
+SUBNET_ID=$(aws ec2 create-subnet --vpc-id $VPC_ID \
+    --cidr-block 192.168.10.0/24 \
+    --query 'Subnet.SubnetId' --output text)
+
+# Security group — allow SSH + all vLLM/YOLO ports
+SG_ID=$(aws ec2 create-security-group \
+    --group-name demo-sg --description "H200 MIG demo" \
+    --vpc-id $VPC_ID --query 'GroupId' --output text)
+
+aws ec2 authorize-security-group-ingress --group-id $SG_ID \
+    --protocol tcp --port 22 --cidr 0.0.0.0/0
+aws ec2 authorize-security-group-ingress --group-id $SG_ID \
+    --protocol tcp --port 8000-8011 --cidr 0.0.0.0/0
+```
+
+```bash
+# Create SSH key
+aws ec2 create-key-pair --key-name spinifex-key \
+  | jq -r '.KeyMaterial | rtrimstr("\n")' > ~/.ssh/spinifex-key
+
+chmod 600 ~/.ssh/spinifex-key
+ssh-keygen -y -f ~/.ssh/spinifex-key > ~/.ssh/spinifex-key.pub
+```
+
+Verify:
+
+```bash
+aws ec2 describe-key-pairs
+```
+
+### 8. Launch the four VMs
+
+Each VM gets a whole H200 via PCIe passthrough. The `p5e.4xlarge` instance type maps one H200 per VM:
+
+```bash
+aws ec2 run-instances \
+  --image-id $AMI_ID \
+  --instance-type p5e.4xlarge \
+  --key-name spinifex-key \
+  --subnet-id $SUBNET_ID \
+  --security-group-ids $SG_ID \
+  --count 4
+```
+
+Wait until all four reach running state, then take note of the public IPs assigned to each instance:
+
+```bash
+aws ec2 describe-instances \
+    --query 'Reservations[*].Instances[*].[InstanceId,PublicIpAddress,State.Name]' \
+    --output table
+```
+
+### 9. Enable MIG inside each VM
+
+MIG is enabled inside each guest VM, not on the host. SSH into each VM and enable it:
+
+```bash
+# Repeat on each VM IP
+ssh -i ~/.ssh/spinifex-key ec2-user@<vm-ip>
+
+sudo nvidia-smi -mig 1
+nvidia-smi | grep "MIG M."
+# Should show: MIG M.  Enabled
+```
+
+### 10. Create MIG partitions inside each VM
+
+With MIG enabled, we are now able to partition each VM's assigned H200 GPU into several separate GPU instances. This process assigns each GPU slice its own UUID, so each VM goes from seeing one whole GPU to seeing a number of "MIG devices":
+
+<p align="center"><img src="../../../.github/assets/images/h200/mig-activated.png" alt="NVIDIA SMI"></p>
+
+This allows us to run several separate workloads, each assigned to its own GPU instance, on the same physical GPU, and thus utilise more of the overall GPU's resources.
+
+Run `nvidia-smi mig -lgip` to see the available partition types. MIG profiles are named Xg.Ygb, where:
+
+* X is the number of GPU slices (GPU Instances, or GIs) allocated to the partition. On an H200 there are 7 allocatable GPU slices, so the largest profile is 7g.
+* Y is the amount of HBM memory allocated to that partition.
+
+Larger profiles also receive proportionally more SMs (Streaming Multiprocessors - analagous to CPU cores), cache, copy engines, encoders/decoders, and other GPU resources.
+
+Thus we partition the GPUs assigned to our VMs as follows:
+
+**vm-llama3b — 7 × 1g.18gb (Llama 3B):**
+```bash
+sudo nvidia-smi mig -cgi 1g.18gb,1g.18gb,1g.18gb,1g.18gb,1g.18gb,1g.18gb,1g.18gb -C
+nvidia-smi -L  # Verify 7 MIG devices
+```
+
+**vm-qwen32b — 2 × 3g.71gb (Qwen 32B):**
+```bash
+sudo nvidia-smi mig -cgi 3g.71gb,3g.71gb -C
+nvidia-smi -L  # Verify 2 MIG devices
+```
+
+**vm-llama70b — 1 × 7g.141gb (Llama 70B, full GPU):**
+```bash
+sudo nvidia-smi mig -cgi 7g.141gb -C
+nvidia-smi -L  # Verify 1 MIG device
+```
+
+**vm-yolo — 2 × 3g.71gb (YOLO):**
+```bash
+sudo nvidia-smi mig -cgi 3g.71gb,3g.71gb -C
+nvidia-smi -L  # Verify 2 MIG devices
+```
+
+Each MIG device gets a UUID of the form `MIG-xxxxxxxx-...`. These UUIDs are used to pin individual containers to their slice via `--gpus "device=<UUID>"` (for Docker) or `CUDA_VISIBLE_DEVICES=<UUID>` (for direct Python processes).
+
+### 11. Configure Docker to use the local image registry
+
+The vLLM image is served from the host's local Docker registry (`192.168.10.1:5000`).
+
+Setting up the local registry on the host:
+
+```bash
+# Configure Docker to trust the local registry address
+sudo tee /etc/docker/daemon.json > /dev/null <<'EOF'
+{
+  "insecure-registries": ["192.168.10.1:5000"]
+}
+EOF
+sudo systemctl restart docker
+
+# Start the registry container, bound to the bridge interface only
+docker run -d \
+    -p 192.168.10.1:5000:5000 \
+    --name registry \
+    --restart=always \
+    registry:2
+
+# Pull the vLLM image and push it into the local registry
+docker pull vllm/vllm-openai:latest
+docker tag vllm/vllm-openai:latest 192.168.10.1:5000/vllm-openai:latest
+docker push 192.168.10.1:5000/vllm-openai:latest
+
+# Verify the image is available
+curl http://192.168.10.1:5000/v2/_catalog
+# {"repositories":["vllm-openai"]}
+```
+
+Each VM then needs to trust it as an insecure registry:
+
+```bash
+# On each VM — vm-yolo uses direct Python, not Docker for YOLO
+sudo tee /etc/docker/daemon.json > /dev/null <<'EOF'
+{
+  "insecure-registries": ["192.168.10.1:5000"]
+}
+EOF
+sudo systemctl restart docker
+```
+
+### 12. Deploy the LLM workloads
+
+Each LLM is downloaded using the Huggingface CLI onto the respective VM. First install the CLI:
+
+```bash
+pip install -U "huggingface_hub[cli]"
+```
+
+Then download the model and determine its snapshot directory. This directory contains the model’s config.json, tokenizer files, and weight files, and will be used as MODEL_DIR when launching vLLM.
+
+```bash
+#On vm-llama3b
+hf download meta-llama/Llama-3.2-3B-Instruct
+
+MODEL_DIR=$(dirname "$(find ~/.cache/huggingface/hub -path '*Llama-3.2-3B-Instruct*' -name config.json | head -1)")
+echo "$MODEL_DIR"
+
+#On vm-qwen32b
+hf download Qwen/Qwen2.5-32B-Instruct
+
+MODEL_DIR=$(dirname "$(find ~/.cache/huggingface/hub -path '*Qwen2.5-32B-Instruct*' -name config.json | head -1)")
+echo "$MODEL_DIR"
+
+#On vm-llama70b
+hf download RedHatAI/Meta-Llama-3.1-70B-Instruct-FP8
+
+MODEL_DIR=$(dirname "$(find ~/.cache/huggingface/hub -path '*Meta-Llama-3.1-70B-Instruct-FP8*' -name config.json | head -1)")
+echo "$MODEL_DIR"
+```
+
+Then we run the docker containers as follows:
+
+**vm-llama3b — 7 × Llama-3.2-3B-Instruct (one container per MIG slice):**
+
+Enumerate the MIG UUIDs and start one vLLM container per slice:
+
+
+```bash
+mapfile -t UUIDS < <(nvidia-smi -L | grep -oP 'MIG-[0-9a-f-]+')
+for i in "${!UUIDS[@]}"; do
+    docker run -d --rm \
+        --name "vllm-$i" \
+        --gpus "device=${UUIDS[$i]}" \
+        --ipc host \
+        -p $((8000 + i)):8000 \
+        -v "${MODEL_DIR}:/models:ro" \
+        192.168.10.1:5000/vllm-openai:latest \
+        vllm serve /models \
+            --served-model-name llama-3b \
+            --dtype bfloat16 \
+            --max-model-len 4096 \
+            --gpu-memory-utilization 0.90 \
+            --port 8000
+done
+```
+
+**vm-qwen32b — 2 × Qwen2.5-32B-Instruct:**
+
+```bash
+mapfile -t UUIDS < <(nvidia-smi -L | grep -oP 'MIG-[0-9a-f-]+')
+for i in "${!UUIDS[@]}"; do
+    docker run -d --rm \
+        --name "qwen32b-$i" \
+        --gpus "device=${UUIDS[$i]}" \
+        --ipc host \
+        -p $((8000 + i)):8000 \
+        -v "${MODEL_DIR}:/models:ro" \
+        192.168.10.1:5000/vllm-openai:latest \
+        vllm serve /models \
+            --served-model-name qwen2.5-32b \
+            --dtype bfloat16 \
+            --max-model-len 4096 \
+            --gpu-memory-utilization 0.90 \
+            --port 8000
+done
+```
+
+**vm-llama70b — Llama-3.1-70B-Instruct-FP8 (full GPU slice):**
+
+```bash
+UUID=$(nvidia-smi -L | grep -oP 'MIG-[0-9a-f-]+' | head -1)
+docker run -d --rm \
+    --name vllm \
+    --gpus "device=${UUID}" \
+    --ipc host \
+    -p 8000:8000 \
+    -v "${MODEL_DIR}:/models:ro" \
+    192.168.10.1:5000/vllm-openai:latest \
+    vllm serve /models \
+        --served-model-name meta-llama-3.1-70b \
+        --dtype auto \
+        --max-model-len 8192 \
+        --gpu-memory-utilization 0.90 \
+        --port 8000
+```
+
+### 13. Deploy YOLO object detection (vm-yolo)
+
+For this demo we built a simple YOLO inference server that runs detection on a looping video file and serves the annotated output as an MJPEG stream. The same script runs twice on vm-yolo — once for YOLO11x (larger, more accurate) and once for YOLO11s (smaller, faster) — each pinned to its own MIG slice and listening on a different port.
+
+The MIG pinning works through `CUDA_VISIBLE_DEVICES`: setting it to a MIG UUID before launch scopes the process to that slice, and CUDA remaps it to device index 0 inside the process. From the script's perspective it always sees one GPU at index 0 — the MIG boundary is invisible to it. `YOLO_MODEL` and `PORT` are what actually differentiate the two instances:
+
+```bash
+python3 -m venv ~/yolo-venv
+~/yolo-venv/bin/pip install ultralytics fastapi "uvicorn[standard]" opencv-python-headless httpx
+
+# Get MIG UUIDs
+nvidia-smi -L
+# GPU 0: NVIDIA H200 (UUID: GPU-...)
+#   MIG 3g.71gb  Device 0: (UUID: MIG-<uuid0>)
+#   MIG 3g.71gb  Device 1: (UUID: MIG-<uuid1>)
+
+# YOLO11x on slice 0 — larger model, port 8010
+CUDA_VISIBLE_DEVICES=MIG-<uuid0> \
+  VIDEO_PATH=<path-to-video> \
+  YOLO_MODEL=yolo11x.pt \
+  YOLO_DEVICE=0 \
+  PORT=8010 \
+  ~/yolo-venv/bin/python ~/yolo_stream.py >> ~/yolo-x.log 2>&1 &
+
+# YOLO11s on slice 1 — smaller/faster model, port 8011
+CUDA_VISIBLE_DEVICES=MIG-<uuid1> \
+  VIDEO_PATH=<path-to-video> \
+  YOLO_MODEL=yolo11s.pt \
+  YOLO_DEVICE=0 \
+  PORT=8011 \
+  ~/yolo-venv/bin/python ~/yolo_stream.py >> ~/yolo-s.log 2>&1 &
+```
+
+Each instance exposes a `/video` endpoint serving a `multipart/x-mixed-replace` MJPEG stream, consumable directly by browsers and most HTTP clients. YOLO11x (~75 MB) and YOLO11s (~9 MB) weights download automatically on first run from the Ultralytics model hub.
+
+### 14. Dashboard
+
+We also built a simple host-side dashboard — another FastAPI application that proxies all the VM streams to the browser so only one port on the host needs to be exposed. Each VM endpoint is wired in by address at startup, mapping directly to the IPs assigned in step 8:
+
+```
+http://vm-yolo-IP:8010  →  vm-yolo  YOLO11x MJPEG stream
+http://vm-yolo-IP:8011  →  vm-yolo  YOLO11s MJPEG stream
+http://vm-llama3b-IP:8000   →  vm-llama3b  vLLM endpoint 0
+http://vm-llama3b-IP:8001   →  vm-llama3b  vLLM endpoint 1
+... (one entry per endpoint across all three LLM VMs)
+```
+
+The dashboard has two proxy patterns: MJPEG passthrough for the YOLO feeds (forwarding the raw boundary stream from vm-yolo to the browser), and SSE passthrough for the LLM token streams (subscribing to each vLLM endpoint and re-emitting tokens as server-sent events). Both use a reconnect loop so the browser connection stays open if a VM is temporarily unreachable.
+
+
+
+<p><video src="https://iso.mulgadc.com/h200-demo.mp4" controls width="100%" style="border-radius:6px"></video></p>
+
+The dashboard shows:
+- GPU allocation bars for all four EC2 instancess (proportional to MIG slice size)
+- Live streaming LLM responses per endpoint, colour-coded by tier
+- Side-by-side YOLO11x vs YOLO11s video feeds with FPS and detection counts
+
+The dashboard also displays the overall GPU utilisation, derived as a percentage of maximum power usage. Importantly, it demonstrates how Spinifex combined with NVIDIA's MIG capability enables the deployment of multiple heterogeneous workloads on owned hardware.
+
+### 15. Teardown
+
+```bash
+# On vm-yolo:
+# Stop YOLO processes
+pkill -f yolo_stream.py
+
+# On each LLM instance:
+# Stop docker containers
+
+docker stop $(docker ps -q)
+
+# Disable MIG inside each VM before terminating.
+sudo nvidia-smi mig -dci
+sudo nvidia-smi mig -dgi
+sudo nvidia-smi -mig 0
+
+# On the host:
+# Terminate all four instances — releases 4× H200 back to the Spinifex pool
+aws ec2 terminate-instances --instance-ids \
+    $(aws ec2 describe-instances \
+        --filters "Name=instance-state-name,Values=running" \
+        --query 'Reservations[*].Instances[*].InstanceId' \
+        --output text)
+```
+
+The four H200s are immediately returned to the host GPU pool once the instances terminate, ready for reallocation.
+
+### 16. Conclusion
+
+This document highlights how Spinifex can turn a single bare-metal chassis into a multi-tenant AI serving platform. Spinifex utilises the flexibility of PCIe passthrough combined with NVIDIA's MIG capability to allocate GPU resources in whichever configuration is required by the workload/s. This flexibility and fine-grain control ensures maximum GPU utilisation.
+
+Importantly, it does so with standard `aws ec2` CLI calls — `run-instances`, `describe-instances`, `terminate-instances` — against Spinifex's EC2-compatible endpoint. Teams already operating AWS infrastructure can point their existing tooling at a Spinifex node with a single profile change, against GPUs that sit in their own rack.
@@ -31,6 +31,8 @@ sections:

 ## Overview

+Spinifex is an open-source infrastructure platform that brings core AWS services including EC2, EBS and S3 to bare-metal, edge, and on-prem environments. It exposes an EC2-compatible API, so any tooling that works against AWS (the `aws` CLI, Terraform, SDKs) works against a Spinifex node unchanged, with a single profile swap.
+
 ### About Trinity

 [Trinity](https://www.arcee.ai/trinity) is a family of open-weight language models from [Arcee AI](https://www.arcee.ai), built specifically for production agentic workloads: reliable tool calling, structured JSON output, coherent multi-turn conversations, and long-context reasoning. The family spans four sizes — Nano (6B), Mini (26B), Large Preview (400B), and Large Thinking (400B reasoning-optimised), with consistent capabilities and API surface across all of them. A workflow validated on Nano can be promoted to Large with no prompt changes.
@@ -30,10 +30,12 @@ services:
    publishes:
      - "ec2.*"
      - "elbv2.*"
+      - "acm.*"
      - "iam.*"
    depends_on:
      - "ec2.*"
      - "elbv2.*"
+      - "acm.*"
      - "iam.*"

  spinifex_daemon:
@@ -50,6 +52,7 @@ services:
      - spinifex/daemon
      - spinifex/handlers/ec2
      - spinifex/handlers/elbv2
+      - spinifex/handlers/acm
      - spinifex/handlers/iam
    subscribes:
      # EC2 instance lifecycle.
@@ -81,6 +84,12 @@ services:
      - "spinifex.node.vms"
      - "spinifex.nodes.discover"
      - "spinifex.cluster.shutdown.progress"
+      # Coordinated cluster-shutdown phases (fan-out, no queue group).
+      - "spinifex.cluster.shutdown.gate"
+      - "spinifex.cluster.shutdown.drain"
+      - "spinifex.cluster.shutdown.storage"
+      - "spinifex.cluster.shutdown.persist"
+      - "spinifex.cluster.shutdown.infra"
      - "spinifex.storage.config"
      # ELBv2 control plane (handlers/elbv2/service_nats.go topic map).
      - "elbv2.CreateLoadBalancer"
@@ -95,6 +104,15 @@ services:
      - "elbv2.CreateListener"
      - "elbv2.DeleteListener"
      - "elbv2.DescribeListeners"
+      # ACM-alike certificate store (handlers/acm/service_nats.go topic map).
+      - "acm.ImportCertificate"
+      - "acm.DescribeCertificate"
+      - "acm.ListCertificates"
+      - "acm.DeleteCertificate"
+      - "elbv2.AddListenerCertificates"
+      - "elbv2.RemoveListenerCertificates"
+      - "elbv2.DescribeListenerCertificates"
+      - "elbv2.DescribeSSLPolicies"
    publishes:
      # → vpcd
      - "vpc.create"
@@ -121,6 +139,8 @@ services:
      - "ebs.delete"
      - "ebs.sync"
      - "ebs.snapshot.*"
+      - "ebs.config"
+      - "ebs.config.*"
      # → predastore (S3 / AMI / keypair store)
      - "s3.*"
      # IAM lifecycle event
@@ -169,6 +189,8 @@ services:
      - "ebs.delete"
      - "ebs.sync"
      - "ebs.snapshot.*"
+      - "ebs.config"
+      - "ebs.config.*"
    publishes:
      - "ebs.mount.response"
      - "ebs.unmount.response"
@@ -262,6 +284,22 @@ suites:
    # into per-test fixture-driven coverage so the selector can pick
    # subtests, not just whole suites.

+  e2e-iam:
+    path: tests/e2e/iam
+    covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+    # "Everything IAM": the IAM/STS control plane plus the instance-identity
+    # tests (instance-profile association + IMDS), which boot guest VMs — OVN
+    # via vpcd, AMI via predastore, root volume via viperblockd, lifecycle via
+    # spinifex_daemon. Single-mode, VM-booting leg.
+
+  e2e-ecr:
+    path: tests/e2e/ecr
+    covers: [awsgw, spinifex_daemon, predastore]
+    # ECR control plane (handlers/ecr, gateway/ecr*) + OCI data plane: blobs and
+    # manifests in predastore, control + auth bridge in awsgw, KV via
+    # spinifex_daemon. docker/crane/skopeo data-plane subtests skip when the
+    # client binary or registry-host resolution is absent.
+
  e2e-ddil:
    path: tests/e2e/ddil
    covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
@@ -0,0 +1,4 @@
+README.md
+*.tfstate
+*.tfstate.backup
+.terraform
@@ -0,0 +1,17 @@
+# Spinifex EKS demo app — multi-stage build to a tiny distroless image.
+#
+# Stage 1 compiles a static binary; stage 2 ships only that binary on a
+# distroless base (no shell, no package manager) for a small, production-shaped
+# image suitable for pushing to ECR and pulling onto EKS workers.
+
+FROM golang:1.23-alpine AS build
+WORKDIR /src
+COPY go.mod ./
+COPY main.go ./
+RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/spinifex-demo .
+
+FROM gcr.io/distroless/static:nonroot
+COPY --from=build /out/spinifex-demo /spinifex-demo
+EXPOSE 8080
+USER nonroot:nonroot
+ENTRYPOINT ["/spinifex-demo"]
@@ -0,0 +1,69 @@
+# Spinifex EKS demo app
+
+A small, Spinifex-themed web app shared by the EKS Terraform workbooks. It
+replaces `nginxdemos/hello` with something production-shaped that you build
+yourself and push to **ECR**, then pull onto your EKS workers.
+
+The page reports the pod, node, cluster, and region that answered the request,
+plus a hit counter — refresh and watch requests land on different pods across a
+multi-node cluster.
+
+## What it does
+
+- `GET /` — themed HTML page.
+- `GET /api/info` — JSON of the same facts.
+- `GET /healthz` — liveness/readiness probe (`200 ok`), used by the ALB target
+  group health check.
+
+All configuration is read from the environment, so one image serves every tier:
+
+| Variable | Purpose | Source |
+|---|---|---|
+| `POD_NAME`, `NODE_NAME`, `POD_NAMESPACE` | identity shown on the page | downward API |
+| `CLUSTER_NAME`, `AWS_REGION` | cluster/region labels | deployment env |
+| `APP_TITLE` | page heading | deployment env (default `Spinifex EKS`) |
+| `PORT` | listen port | default `8080` |
+| `DATA_DIR` | when set, the hit counter is persisted here and survives pod restarts | mounted **EBS-CSI PVC** in `eks-gitops-argocd` |
+
+When `DATA_DIR` points at a writable volume the page shows a **"persisted to EBS
+volume"** badge — the gitops workbook mounts a Viperblock-backed PersistentVolume
+there to demonstrate durable storage.
+
+## Build and push to ECR
+
+The image is named `spinifex-demo`. Build it once and push it to a repository in
+your Spinifex ECR; every workbook references it by its ECR URI.
+
+```bash
+export AWS_PROFILE=spinifex
+REGION=ap-southeast-2
+
+# 1. Create the repository (the workbooks also create it via Terraform; skip if
+#    it already exists).
+aws ecr create-repository --repository-name spinifex-demo --region "$REGION"
+
+# 2. Resolve the registry host and authenticate Docker.
+REGISTRY=$(aws ecr describe-repositories --repository-names spinifex-demo \
+  --region "$REGION" --query 'repositories[0].repositoryUri' --output text)
+REGISTRY_HOST=${REGISTRY%%/*}
+aws ecr get-login-password --region "$REGION" \
+  | docker login --username AWS --password-stdin "$REGISTRY_HOST"
+
+# 3. Build, tag, and push.
+docker build -t spinifex-demo:latest .
+docker tag spinifex-demo:latest "${REGISTRY}:latest"
+docker push "${REGISTRY}:latest"
+
+echo "Image: ${REGISTRY}:latest"
+```
+
+Pass the resulting `${REGISTRY}:latest` value to a workbook as the `demo_image`
+variable (or accept its default, which resolves the URI from the ECR repository
+the workbook creates).
+
+## Run locally
+
+```bash
+go run .            # http://localhost:8080
+DATA_DIR=/tmp/demo go run .   # exercise the persisted-counter path
+```
@@ -0,0 +1,3 @@
+module github.com/mulgadc/spinifex-demo
+
+go 1.23
@@ -0,0 +1,214 @@
+// Command spinifex-demo is the demo web app served by the EKS Terraform
+// workbooks. It renders a Spinifex-themed page reporting the pod, node, cluster,
+// and region that answered the request, plus a hit counter.
+//
+// The counter is held in memory by default. When DATA_DIR points at a writable
+// directory (an EBS-CSI PersistentVolume in the gitops workbook), the counter is
+// persisted there and survives pod restarts — demonstrating Viperblock-backed
+// durable storage. All configuration comes from the environment so a single
+// image serves every workbook tier.
+package main
+
+import (
+	"encoding/json"
+	"html/template"
+	"log/slog"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strconv"
+	"sync"
+	"time"
+)
+
+// instance captures the per-pod facts surfaced on the page, populated from the
+// downward API and the deployment environment.
+type instance struct {
+	Pod       string
+	Node      string
+	Namespace string
+	Cluster   string
+	Region    string
+	Title     string
+	Persisted bool
+}
+
+func env(key, fallback string) string {
+	if v := os.Getenv(key); v != "" {
+		return v
+	}
+	return fallback
+}
+
+// counter tracks page hits, optionally persisting to dir/hits so the count
+// survives a pod restart when backed by a PersistentVolume.
+type counter struct {
+	mu   sync.Mutex
+	n    int64
+	path string
+}
+
+func newCounter(dir string) *counter {
+	c := &counter{}
+	if dir == "" {
+		return c
+	}
+	c.path = filepath.Join(dir, "hits")
+	if b, err := os.ReadFile(c.path); err == nil {
+		if n, perr := strconv.ParseInt(string(b), 10, 64); perr == nil {
+			c.n = n
+		}
+	}
+	return c
+}
+
+func (c *counter) inc() int64 {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.n++
+	if c.path != "" {
+		// Best-effort persist; a read-only or missing volume must not 500 the page.
+		if err := os.WriteFile(c.path, []byte(strconv.FormatInt(c.n, 10)), 0o644); err != nil {
+			slog.Warn("persist counter failed", "path", c.path, "err", err)
+		}
+	}
+	return c.n
+}
+
+func main() {
+	logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
+	slog.SetDefault(logger)
+
+	dataDir := os.Getenv("DATA_DIR")
+	hits := newCounter(dataDir)
+
+	base := instance{
+		Pod:       env("POD_NAME", "unknown"),
+		Node:      env("NODE_NAME", "unknown"),
+		Namespace: env("POD_NAMESPACE", "default"),
+		Cluster:   env("CLUSTER_NAME", "spinifex-eks"),
+		Region:    env("AWS_REGION", env("AWS_DEFAULT_REGION", "ap-southeast-2")),
+		Title:     env("APP_TITLE", "Spinifex EKS"),
+		Persisted: dataDir != "",
+	}
+
+	tmpl := template.Must(template.New("page").Parse(pageHTML))
+
+	mux := http.NewServeMux()
+
+	mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("ok"))
+	})
+
+	mux.HandleFunc("/api/info", func(w http.ResponseWriter, _ *http.Request) {
+		w.Header().Set("Content-Type", "application/json")
+		_ = json.NewEncoder(w).Encode(map[string]any{
+			"pod":       base.Pod,
+			"node":      base.Node,
+			"namespace": base.Namespace,
+			"cluster":   base.Cluster,
+			"region":    base.Region,
+			"hits":      hits.inc(),
+			"persisted": base.Persisted,
+		})
+	})
+
+	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path != "/" {
+			http.NotFound(w, r)
+			return
+		}
+		view := struct {
+			instance
+			Hits int64
+			Now  string
+		}{base, hits.inc(), time.Now().UTC().Format("2006-01-02 15:04:05 MST")}
+		w.Header().Set("Content-Type", "text/html; charset=utf-8")
+		if err := tmpl.Execute(w, view); err != nil {
+			slog.Error("render failed", "err", err)
+		}
+	})
+
+	addr := ":" + env("PORT", "8080")
+	slog.Info("spinifex-demo listening", "addr", addr, "cluster", base.Cluster, "persisted", base.Persisted)
+	srv := &http.Server{
+		Addr:              addr,
+		Handler:           mux,
+		ReadHeaderTimeout: 5 * time.Second,
+	}
+	if err := srv.ListenAndServe(); err != nil {
+		slog.Error("server exited", "err", err)
+		os.Exit(1)
+	}
+}
+
+const pageHTML = `<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>{{.Title}}</title>
+<style>
+  :root {
+    --bg: #0b0e14; --panel: #131823; --line: #232a39;
+    --fg: #e6edf3; --muted: #8b97a7; --accent: #f5b301; --accent2: #3ddc97;
+  }
+  * { box-sizing: border-box; }
+  body {
+    margin: 0; min-height: 100vh; background:
+      radial-gradient(1200px 600px at 80% -10%, rgba(245,179,1,.10), transparent 60%),
+      radial-gradient(900px 500px at -10% 110%, rgba(61,220,151,.08), transparent 55%), var(--bg);
+    color: var(--fg); font: 15px/1.55 ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif;
+    display: flex; align-items: center; justify-content: center; padding: 32px;
+  }
+  .card {
+    width: 100%; max-width: 640px; background: var(--panel);
+    border: 1px solid var(--line); border-radius: 16px; overflow: hidden;
+    box-shadow: 0 20px 60px rgba(0,0,0,.45);
+  }
+  .head { padding: 28px 32px; border-bottom: 1px solid var(--line); display: flex; align-items: center; gap: 16px; }
+  .logo { height: 44px; width: auto; flex: 0 0 auto; }
+  .head h1 { margin: 0; font-size: 20px; letter-spacing: .2px; }
+  .head p { margin: 2px 0 0; color: var(--muted); font-size: 13px; }
+  .body { padding: 24px 32px 8px; }
+  .grid { display: grid; grid-template-columns: 1fr 1fr; gap: 12px 20px; }
+  .kv { padding: 12px 14px; background: #0e131d; border: 1px solid var(--line); border-radius: 10px; }
+  .kv .k { color: var(--muted); font-size: 11px; text-transform: uppercase; letter-spacing: .08em; }
+  .kv .v { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 14px; word-break: break-all; margin-top: 3px; }
+  .hits { grid-column: 1 / -1; display: flex; align-items: baseline; justify-content: space-between; }
+  .hits .n { font-size: 30px; font-weight: 700; color: var(--accent); font-family: ui-monospace, monospace; }
+  .foot { padding: 16px 32px 24px; color: var(--muted); font-size: 12px; display: flex; justify-content: space-between; gap: 12px; }
+  .pill { color: var(--accent2); border: 1px solid rgba(61,220,151,.35); border-radius: 999px; padding: 2px 10px; font-size: 11px; }
+</style>
+</head>
+<body>
+  <div class="card">
+    <div class="head">
+      <svg class="logo" viewBox="0 0 364 463" fill="none" xmlns="http://www.w3.org/2000/svg" aria-hidden="true">
+        <path d="M272.523 171.205C247.773 197.595 211.323 215.815 182.103 237.425C165.663 249.575 122.633 281.165 151.093 301.825C185.533 326.835 271.333 304.985 301.013 277.585C317.193 262.645 316.693 242.485 294.703 234.075C270.913 224.975 243.483 231.925 219.353 235.755C218.543 235.885 216.823 236.805 216.993 235.345C227.053 228.885 236.103 220.925 245.933 214.135C255.403 207.585 267.053 198.225 278.643 198.015C314.693 197.345 351.803 215.135 361.203 252.215C376.833 313.885 303.333 362.255 249.833 366.635L280.913 352.235C292.563 345.305 304.353 338.105 314.653 329.215C317.213 327.005 319.073 324.145 321.243 322.035C321.753 321.535 327.053 319.265 324.003 319.125C296.253 337.335 263.793 349.415 231.323 356.335C220.173 358.715 208.323 360.995 196.963 361.285C197.243 362.845 198.643 362.525 199.663 362.845C208.823 365.795 219.313 367.545 228.843 369.005C230.163 370.005 228.733 370.415 227.683 370.555C219.063 371.665 209.923 373.305 201.293 373.745C176.843 374.985 152.203 372.435 128.573 366.425L115.293 328.755C114.153 328.865 114.073 329.895 113.783 330.725C112.613 334.015 107.893 358.025 106.313 358.365C93.6234 354.515 82.2334 345.915 72.9034 336.575L95.0434 296.915L59.7134 316.375C54.7434 305.145 53.8334 292.435 55.4934 280.355L101.213 257.645C86.9934 256.705 72.6134 258.955 58.4734 260.425C60.8734 253.495 63.8034 246.615 67.3634 240.195C68.6634 237.855 75.5634 226.305 77.3834 226.245C93.9334 229.415 111.773 230.895 128.363 227.675L92.3534 212.825L91.0334 211.045C98.3834 204.125 105.843 197.235 114.113 191.325C129.603 195.825 145.523 199.455 161.733 199.655L130.633 179.885C134.133 177.125 137.813 174.575 141.553 172.165C144.643 170.175 153.283 164.425 156.423 164.835C168.933 169.105 181.923 172.125 195.123 173.185C189.093 166.935 177.633 162.325 171.833 156.465C171.203 155.825 170.713 155.785 171.003 154.585C186.153 145.885 203.003 137.235 216.053 125.375C229.363 113.265 243.423 93.4848 233.853 75.2048C244.893 78.7148 254.393 92.6648 254.153 104.325C253.493 136.595 219.073 165.375 197.483 185.615C230.833 171.615 268.993 142.685 268.923 102.735C268.863 70.2248 239.263 57.1948 211.283 54.2348C210.103 61.3848 207.573 68.9448 203.863 75.1748C195.623 89.0148 178.803 97.6648 170.013 112.055C164.023 121.865 162.883 135.065 150.163 138.825C144.973 140.365 138.293 140.595 132.903 140.615L144.743 134.365C158.003 124.625 157.773 107.025 165.893 93.9248C164.953 92.5948 155.413 90.0748 153.183 89.5248C143.363 87.1048 126.713 84.0248 116.883 84.6948C113.843 84.9048 100.143 87.0348 98.9634 89.3748C97.2634 92.7648 105.533 107.255 107.973 110.575C108.463 111.245 109.463 111.075 109.513 111.145C109.973 111.745 109.663 113.165 108.373 112.675C107.443 112.315 101.853 108.205 100.593 107.235C90.8934 99.7648 79.8234 88.4748 71.7234 79.3648C70.4034 76.1748 68.6334 72.6148 71.7634 69.9848C88.8634 59.4048 105.793 48.3148 123.163 38.1948C174.603 8.20482 233.143 -25.6952 282.353 29.2448C323.153 74.8048 311.503 129.615 272.493 171.195L272.523 171.205ZM114.153 67.5948C127.613 65.2648 153.013 78.6248 157.193 58.8348C157.603 56.9048 156.493 55.7948 157.583 53.9148C158.403 52.5048 163.083 49.1048 164.713 47.7848C169.903 43.6048 175.823 40.3348 180.923 36.0148C161.113 39.2048 141.163 50.5648 123.853 60.5948C120.983 62.2548 117.873 63.9948 115.153 65.8548C114.453 66.3348 113.713 65.7348 114.143 67.5848L114.153 67.5948ZM21.1634 276.415C-11.8666 312.635 -4.35661 364.395 30.6734 396.135C65.5434 427.725 120.403 438.925 166.373 436.725C207.813 434.735 249.893 421.075 290.773 434.965C306.663 440.365 320.533 450.465 332.363 462.135C330.773 454.415 326.123 446.365 321.623 439.825C280.593 380.225 200.893 402.565 140.123 390.665C107.163 384.215 72.8934 369.135 52.8234 341.225C35.4734 317.105 32.7734 288.665 41.3634 260.475C39.6134 258.725 23.0334 274.355 21.1634 276.415ZM143.743 109.535C136.373 116.045 128.573 122.235 120.403 127.775L97.1634 139.135C116.843 139.885 137.193 128.635 143.753 109.545L143.743 109.535Z" fill="#f5b301"/>
+      </svg>
+      <div>
+        <h1>{{.Title}}</h1>
+        <p>Served by a pod on your Spinifex-managed Kubernetes cluster</p>
+      </div>
+    </div>
+    <div class="body">
+      <div class="grid">
+        <div class="kv"><div class="k">Pod</div><div class="v">{{.Pod}}</div></div>
+        <div class="kv"><div class="k">Node</div><div class="v">{{.Node}}</div></div>
+        <div class="kv"><div class="k">Cluster</div><div class="v">{{.Cluster}}</div></div>
+        <div class="kv"><div class="k">Region</div><div class="v">{{.Region}}</div></div>
+        <div class="kv hits">
+          <div><div class="k">Requests served</div>{{if .Persisted}}<span class="pill">persisted to EBS volume</span>{{end}}</div>
+          <div class="n">{{.Hits}}</div>
+        </div>
+      </div>
+    </div>
+    <div class="foot">
+      <span>Namespace: {{.Namespace}}</span>
+      <span>{{.Now}}</span>
+    </div>
+  </div>
+</body>
+</html>`
@@ -0,0 +1,284 @@
+---
+title: "GitOps on EKS (Argo CD + EBS-CSI)"
+description: "Deliver a Spinifex-themed app to EKS with GitOps — the Argo CD addon syncs it from a git repo, an EBS-CSI (Viperblock) PersistentVolume holds its state, and it is served over HTTPS through the AWS Load Balancer Controller + ACM, using Terraform on Spinifex."
+category: "Terraform Workbooks"
+tags:
+  - terraform
+  - eks
+  - kubernetes
+  - argocd
+  - gitops
+  - ebs-csi
+  - storage
+  - workbook
+resources:
+  - title: "Argo CD"
+    url: "https://argo-cd.readthedocs.io/"
+  - title: "AWS Load Balancer Controller"
+    url: "https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/"
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "Terraform Kubernetes Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+---
+
+# Terraform: GitOps on EKS (Argo CD + EBS-CSI)
+
+> The top rung of the EKS ladder. It extends [EKS HTTPS Ingress](../eks-https-ingress) and changes how the app is delivered: the **Argo CD** addon syncs a more elaborate Spinifex-themed app from a **git repo**, an **EBS-CSI** PersistentVolume (Viperblock-backed) holds its state, and it is still served over **HTTPS** through the AWS Load Balancer Controller + ACM.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+This workbook keeps everything from [EKS HTTPS Ingress](../eks-https-ingress) — the public/private VPC, the NAT gateway, the LBC addon, the ACM cert, the HTTPS Ingress — and adds the parts you reach for once an app is real: **GitOps delivery** and **persistent storage**.
+
+Three changes over the previous rung:
+
+1. **Argo CD delivers the app.** Instead of Terraform applying the Deployment, the **argocd** addon syncs the app from a git repository ([`mulgadc/eks-demo-app`](https://github.com/mulgadc/eks-demo-app)). Terraform only registers the repo credential and creates the Argo CD `Application`; Argo CD reconciles the manifests and self-heals drift.
+2. **State lives on an EBS volume.** The **aws-ebs-csi-driver** addon ships a default gp3 StorageClass. The app's `PersistentVolumeClaim` (in the git repo) dynamically provisions a **Viperblock-backed EBS volume**; the demo's hit counter persists to it and survives pod restarts.
+3. **Access-entry RBAC.** A second IAM principal is granted read-only cluster access via an access entry bound to `AmazonEKSViewPolicy`.
+
+Terraform manages the cluster, the addons, the ACM cert, and the HTTPS Ingress; **Argo CD** manages the app's Deployment, Service, and PVC from git. The Ingress points at the Service Argo CD creates, so the app is reachable over HTTPS the moment Argo CD finishes its first sync.
+
+**What you'll learn:**
+
+- Installing the `argocd` and `aws-ebs-csi-driver` addons through the EKS API
+- Registering a private git repo with Argo CD and driving an `Application` from Terraform
+- Dynamically provisioning a Viperblock-backed EBS volume with a `PersistentVolumeClaim`
+- Splitting ownership: Terraform owns infra + Ingress, Argo CD owns the workload
+- Granting scoped read-only access with an EKS access entry
+
+**What gets created**
+
+| Resource | Name | Purpose |
+|---|---|---|
+| VPC + subnets | `eks-gitops-*` | Public/private network (10.32.0.0/16) with a NAT gateway |
+| IAM Roles | `eks-gitops-cluster-role`, `-node-role`, `-viewer` | Control-plane, worker, and read-only viewer roles |
+| ECR Repository | `spinifex-demo` | Holds the demo image the workers pull |
+| EKS Cluster | `eks-gitops` | Public + private endpoints; `managed-ingress=false` |
+| Node Group | `workers` | `node_desired_size` `t3.large` worker(s) — 1 or 3 |
+| Addons | `aws-load-balancer-controller`, `argocd`, `aws-ebs-csi-driver` | Ingress, GitOps delivery, persistent storage |
+| ACM Certificate | `eks-gitops-ingress` | Self-signed, imported; on the HTTPS listener |
+| Access Entry + Assoc. | `eks-gitops-viewer` → `AmazonEKSViewPolicy` | Read-only, cluster scope |
+| Argo CD repo Secret | `eks-demo-app-repo` | Credential for the private git repo |
+| Argo CD Application | `spinifex-demo` | Syncs the app from git |
+| K8s Ingress | `spinifex-demo` | `ingressClassName: alb`, HTTPS via the ACM cert |
+| Git-managed (by Argo CD) | Deployment, Service, **PVC** | The app + its Viperblock-backed volume |
+
+**Spinifex specifics**
+
+- The `argocd` and `aws-ebs-csi-driver` bundles must be baked into the `eks-node` AMI — `describe-addon` returns `no baked bundle` otherwise.
+- The EBS-CSI default StorageClass uses `provisioner: ebs.csi.aws.com`; a PVC against it provisions a **Viperblock-backed EBS volume**.
+- **Verify dynamic provisioning on a live cluster.** EBS-CSI dynamic provisioning depends on the running Spinifex EBS/Viperblock data plane and may not be available on every build yet. If the PVC stays `Pending`, the app still serves (the counter falls back to in-memory) — confirm the volume binds on a stack where the driver is healthy.
+- Leave addon versions unset (the AWS provider and the catalog disagree on the version-string format).
+- The Argo CD `Application` is a `kubernetes_manifest`, so the `argoproj.io` CRDs must exist before `apply` — apply the parent and let the addon reach `ACTIVE` first.
+
+**Prerequisites:**
+
+- Spinifex running with an `eks-node` image carrying the LBC, Argo CD, and EBS-CSI bundles
+- OpenTofu or Terraform, plus `kubectl`, the AWS CLI, and Docker
+- The demo image built and pushed to ECR (see [`../demo-app/README.md`](../demo-app/README.md))
+- A git repo with the app manifests ([`mulgadc/eks-demo-app`](https://github.com/mulgadc/eks-demo-app)); a read-only PAT if it is private
+
+## Instructions
+
+### Step 1. Get the Template
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-gitops-argocd
+```
+
+<!-- INCLUDE: main.tf lang:hcl -->
+
+### Step 2. Deploy the Cluster + Addons
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply        # add -var node_desired_size=3 for an HA-shaped cluster
+```
+
+Wait for the cluster to reach `ACTIVE` and for all three addons (`aws-load-balancer-controller`, `argocd`, `aws-ebs-csi-driver`) to report healthy:
+
+```bash
+aws eks list-addons --cluster-name eks-gitops
+aws eks describe-addon --cluster-name eks-gitops --addon-name argocd --query 'addon.status'
+```
+
+### Step 3. Build and Push the Demo Image
+
+```bash
+cd ../demo-app
+REGISTRY=$(cd ../eks-gitops-argocd && tofu output -raw ecr_repository_url)
+REGISTRY_HOST=${REGISTRY%%/*}
+aws ecr get-login-password | docker login --username AWS --password-stdin "$REGISTRY_HOST"
+docker build -t "${REGISTRY}:latest" .
+docker push "${REGISTRY}:latest"
+cd ../eks-gitops-argocd
+```
+
+Point the app's manifests at this image: set the image ref in `eks-demo-app/manifests` (a kustomize `images:` override) to your `${REGISTRY}:latest`, and push that to the git repo.
+
+### Step 4. Hand Delivery to Argo CD
+
+```bash
+cd workloads
+tofu init
+tofu apply \
+  -var git_repo_url=https://github.com/mulgadc/eks-demo-app.git \
+  -var git_token=<read-only-PAT>     # omit for a public repo
+```
+
+This registers the repo credential, creates the Argo CD `Application`, the demo app's HTTPS Ingress, and an HTTPS Ingress for the Argo CD UI itself. Argo CD then syncs the Deployment, Service, and PVC from git.
+
+<!-- INCLUDE: workloads/main.tf lang:hcl -->
+
+### Step 5. Watch the Sync and Open the App
+
+The demo app and the Argo CD UI share **one ALB** (an LBC IngressGroup). On
+`:443` the app is the catch-all and `argocd.eks-gitops.spinifex.local`
+host-routes to the UI; Argo CD also gets a hostless `:8443` listener so it is
+reachable on the raw ALB IP before DNS exists. Grab the shared ALB address once:
+
+```bash
+# Point kubectl at the cluster first (writes ~/.kube/config) — without this every
+# kubectl call fails to connect:
+aws eks update-kubeconfig --name eks-gitops --region ap-southeast-2
+
+kubectl -n argocd get applications
+kubectl get pods -o wide
+kubectl get pvc                    # the EBS-CSI volume should be Bound
+
+DNSNAME=$(kubectl get ingress spinifex-demo -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
+ALB_IP=$(aws elbv2 describe-load-balancers \
+  --query "LoadBalancers[?DNSName=='${DNSNAME}'].AvailabilityZones[].LoadBalancerAddresses[].IpAddress | [0]" \
+  --output text)
+echo "$ALB_IP"
+```
+
+**Reach it by raw IP (no DNS, no host header):**
+
+| Service | URL |
+|---|---|
+| Demo app | `https://<ALB_IP>/` |
+| Argo CD UI | `https://<ALB_IP>:8443/` |
+
+```bash
+curl -k https://"$ALB_IP"/            # app — returns the Spinifex page
+curl -k https://"$ALB_IP":8443/       # Argo CD UI
+```
+
+The bare ALB IP serves the **app** on `:443`. Hitting `https://<ALB_IP>/` and
+expecting Argo CD gives the app instead — use `:8443` for the UI. Once northstar
+(or Route 53) resolves the hostnames to the ALB, `app.eks-gitops.spinifex.local`
+and `argocd.eks-gitops.spinifex.local` both work on `:443`. Self-signed cert —
+accept the browser warning.
+
+The page shows the **"persisted to EBS volume"** badge and a hit counter that keeps climbing — delete the pod (`kubectl delete pod -l app=spinifex-demo`) and the count survives, proving the volume is durable.
+
+### Step 6. Open the Argo CD UI
+
+Managing deployments through the Argo CD console is the point of this workbook, so
+the UI is exposed on the **same ALB** as the app, not behind a port-forward. The
+`workloads` module adds a `NodePort` Service in front of `argocd-server` (the addon
+ships it `ClusterIP` only) and two Ingresses in the shared group — a host-routed
+one on `:443` and a hostless one on `:8443` for raw-IP access. `argocd-server`
+serves TLS on its own port, so both use `backend-protocol: HTTPS`.
+
+**Get the admin credentials.** Argo CD generates a one-time `admin` password into
+a Secret on install; the username is always `admin`:
+
+```bash
+kubectl -n argocd get secret argocd-initial-admin-secret \
+  -o jsonpath='{.data.password}' | base64 -d ; echo
+```
+
+(`tofu output argocd_admin_password_cmd` prints this same command.)
+
+Open the UI and log in as `admin`:
+
+- By raw IP: `https://<ALB_IP>:8443/`
+- By DNS once wired: `https://argocd.eks-gitops.spinifex.local`
+
+The `spinifex-demo` Application shows the sync status, the resource tree, and live
+diffs against git — change a manifest in the repo and watch Argo CD reconcile it.
+
+If you'd rather not expose the UI at all, port-forward instead:
+
+```bash
+kubectl -n argocd port-forward svc/argocd-server 8080:443
+# then open https://localhost:8080
+```
+
+### Cleanup
+
+Destroy in reverse — the Argo CD Application + Ingress first (so Argo CD prunes the workload and the LBC tears the ALB down), then the cluster:
+
+```bash
+cd workloads
+tofu destroy
+cd ..
+tofu destroy
+```
+
+## Troubleshooting
+
+### Application Won't Sync
+
+```bash
+kubectl -n argocd get applications
+kubectl -n argocd describe application spinifex-demo     # conditions show repo/auth errors
+kubectl -n argocd logs deploy/argocd-repo-server --tail=50
+```
+
+For a private repo, confirm the `eks-demo-app-repo` Secret exists in the `argocd` namespace with the `argocd.argoproj.io/secret-type=repository` label and a valid token.
+
+### `kubernetes_manifest` Fails: no matches for kind "Application"
+
+The `argoproj.io` CRDs aren't installed yet. Apply the parent module and wait for the `argocd` addon to reach `ACTIVE` before applying `workloads/`.
+
+### PVC Stuck in Pending
+
+```bash
+kubectl get pvc
+kubectl describe pvc <name>
+kubectl get storageclass
+```
+
+Confirm the `aws-ebs-csi-driver` addon is `ACTIVE` and the default StorageClass exists. EBS-CSI dynamic provisioning depends on the live Spinifex EBS/Viperblock data plane — verify on a stack where the driver is healthy. The app still serves with an in-memory counter if the volume never binds.
+
+### The Ingress Has No Address
+
+The LBC populates `status.loadBalancer` once it provisions the ALB. Check the addon and the controller logs:
+
+```bash
+kubectl describe ingress spinifex-demo
+kubectl -n kube-system logs deploy/aws-load-balancer-controller --tail=50
+```
+
+Confirm the cluster carries `spinifex.io/managed-ingress = "false"` so the LBC owns ingress.
+
+If the controller logs show `FailedBuildModel ... DescribeAvailabilityZones ...
+403 ... AccessDenied`, the node role is missing the LBC permissions. The
+controller runs with the node instance-profile credentials, so the
+`${var.cluster_name}-node-lbc` policy (`aws_iam_policy.node_lbc`) must be
+attached to the node role — re-run `tofu apply` on the parent module if it was
+provisioned before that policy existed.
+
+### Provider Connection Refused
+
+```bash
+sudo systemctl status spinifex.target
+curl -k https://localhost:9999/
+```
@@ -0,0 +1,745 @@
+# Example: GitOps on EKS with Argo CD + a persistent EBS-CSI volume
+#
+# The top rung of the EKS ladder. It extends eks-https-ingress (LBC + ACM HTTPS)
+# and changes how the app is delivered: instead of Terraform applying the
+# workload, the Argo CD addon syncs a more elaborate Spinifex-themed app from a
+# git repository, and the app stores state on a Viperblock-backed EBS volume
+# provisioned dynamically through the EBS-CSI driver.
+#
+# What it adds over eks-https-ingress:
+#   * The argocd addon — GitOps continuous delivery, installed via the EKS API.
+#   * The aws-ebs-csi-driver addon — dynamic EBS (Viperblock) PersistentVolumes.
+#   * An access entry granting a second IAM principal read-only cluster access.
+#
+# The workloads/ module registers the (private) git repo with Argo CD, creates an
+# Argo CD Application that syncs the app from it, and keeps the HTTPS Ingress
+# (LBC + ACM) pointing at the git-managed Service. The app's manifests — including
+# the PersistentVolumeClaim — live in the git repo (see ../../../../eks-demo-app).
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-gitops-argocd
+#   export AWS_PROFILE=spinifex
+#   tofu init && tofu apply
+#   # build + push the demo image to the ECR repo this creates (see README),
+#   # then: cd workloads && tofu init && tofu apply -var git_repo_url=... -var git_token=...
+#   # finally: kubectl get ingress spinifex-demo -o wide  → open https://<address>
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    tls = {
+      source  = "hashicorp/tls"
+      version = ">= 4.0"
+    }
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Variables
+# ---------------------------------------------------------------------------
+
+variable "region" {
+  type    = string
+  default = "ap-southeast-2"
+}
+
+variable "cluster_name" {
+  type    = string
+  default = "eks-gitops"
+}
+
+variable "k8s_version" {
+  type    = string
+  default = "1.32"
+}
+
+variable "node_instance_type" {
+  type    = string
+  default = "t3.medium"
+}
+
+variable "node_desired_size" {
+  type        = number
+  default     = 1
+  description = "Worker count. Use 1 for a single-node demo, or 3 for an HA-shaped cluster."
+
+  validation {
+    condition     = var.node_desired_size == 1 || var.node_desired_size == 3
+    error_message = "node_desired_size must be 1 or 3."
+  }
+}
+
+variable "node_port" {
+  type        = number
+  default     = 30080
+  description = "NodePort the demo Service is published on and the ALB forwards to"
+}
+
+variable "argocd_node_port" {
+  type        = number
+  default     = 30081
+  description = "NodePort the Argo CD UI Service is published on and its ALB forwards to"
+}
+
+variable "cert_common_name" {
+  type    = string
+  default = "eks-gitops.spinifex.local"
+}
+
+variable "api_public_access_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the public Kubernetes API endpoint; tighten in production"
+}
+
+variable "spinifex_endpoint" {
+  type    = string
+  default = "https://127.0.0.1:9999"
+}
+
+# ---------------------------------------------------------------------------
+# Providers
+# ---------------------------------------------------------------------------
+
+provider "aws" {
+  region = var.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+    ecr = var.spinifex_endpoint
+    acm = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+data "aws_availability_zones" "available" {
+  state = "available"
+}
+
+# ---------------------------------------------------------------------------
+# VPC — public subnets for the ALB + NAT, private subnets for the workers
+# ---------------------------------------------------------------------------
+
+resource "aws_vpc" "main" {
+  cidr_block           = "10.32.0.0/16"
+  enable_dns_hostnames = true
+  enable_dns_support   = true
+
+  tags = {
+    Name = "${var.cluster_name}-vpc"
+  }
+}
+
+resource "aws_internet_gateway" "igw" {
+  vpc_id = aws_vpc.main.id
+
+  tags = {
+    Name = "${var.cluster_name}-igw"
+  }
+}
+
+resource "aws_subnet" "public_a" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.32.1.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-public-a"
+  }
+}
+
+resource "aws_subnet" "public_b" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.32.2.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-public-b"
+  }
+}
+
+resource "aws_subnet" "private_a" {
+  vpc_id            = aws_vpc.main.id
+  cidr_block        = "10.32.11.0/24"
+  availability_zone = data.aws_availability_zones.available.names[0]
+
+  tags = {
+    Name = "${var.cluster_name}-private-a"
+  }
+}
+
+resource "aws_subnet" "private_b" {
+  vpc_id            = aws_vpc.main.id
+  cidr_block        = "10.32.12.0/24"
+  availability_zone = data.aws_availability_zones.available.names[0]
+
+  tags = {
+    Name = "${var.cluster_name}-private-b"
+  }
+}
+
+resource "aws_route_table" "public" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block = "0.0.0.0/0"
+    gateway_id = aws_internet_gateway.igw.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-public-rt"
+  }
+}
+
+resource "aws_route_table_association" "public_a" {
+  subnet_id      = aws_subnet.public_a.id
+  route_table_id = aws_route_table.public.id
+}
+
+resource "aws_route_table_association" "public_b" {
+  subnet_id      = aws_subnet.public_b.id
+  route_table_id = aws_route_table.public.id
+}
+
+# NAT gateway gives the private workers outbound internet to pull the demo image
+# from ECR. The workers still join the cluster over the in-VPC private endpoint.
+resource "aws_eip" "nat" {
+  domain = "vpc"
+
+  tags = {
+    Name = "${var.cluster_name}-nat-eip"
+  }
+}
+
+resource "aws_nat_gateway" "nat" {
+  allocation_id = aws_eip.nat.id
+  subnet_id     = aws_subnet.public_a.id
+
+  depends_on = [aws_internet_gateway.igw]
+
+  tags = {
+    Name = "${var.cluster_name}-nat"
+  }
+}
+
+resource "aws_route_table" "private" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block     = "0.0.0.0/0"
+    nat_gateway_id = aws_nat_gateway.nat.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-private-rt"
+  }
+}
+
+resource "aws_route_table_association" "private_a" {
+  subnet_id      = aws_subnet.private_a.id
+  route_table_id = aws_route_table.private.id
+}
+
+resource "aws_route_table_association" "private_b" {
+  subnet_id      = aws_subnet.private_b.id
+  route_table_id = aws_route_table.private.id
+}
+
+# ---------------------------------------------------------------------------
+# IAM — cluster role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "cluster" {
+  name = "${var.cluster_name}-cluster-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "eks.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "cluster" {
+  role       = aws_iam_role.cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — node role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "node" {
+  name = "${var.cluster_name}-node-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_worker" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_cni" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_ecr" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+}
+
+# The AWS Load Balancer Controller runs with the node's instance-profile
+# credentials (Spinifex wires creds at the node level, not IRSA — see the addon
+# block below), so the permissions it needs to manage ALBs must live on the node
+# role. This mirrors the upstream AWSLoadBalancerControllerIAMPolicy, trimmed to
+# the actions an ALB Ingress exercises (Shield/WAF/Cognito are disabled on the
+# controller). Resources are "*" since Spinifex evaluates grants by action.
+# A customer-managed policy + attachment is used (Spinifex implements
+# CreatePolicy/AttachRolePolicy, not inline PutRolePolicy).
+resource "aws_iam_policy" "node_lbc" {
+  name = "${var.cluster_name}-node-lbc"
+
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Effect = "Allow"
+        Action = [
+          "ec2:DescribeAccountAttributes",
+          "ec2:DescribeAddresses",
+          "ec2:DescribeAvailabilityZones",
+          "ec2:DescribeInternetGateways",
+          "ec2:DescribeVpcs",
+          "ec2:DescribeSubnets",
+          "ec2:DescribeSecurityGroups",
+          "ec2:DescribeInstances",
+          "ec2:DescribeNetworkInterfaces",
+          "ec2:DescribeTags",
+          "ec2:GetCoipPoolUsage",
+          "ec2:DescribeCoipPools",
+          "ec2:CreateSecurityGroup",
+          "ec2:CreateTags",
+          "ec2:DeleteTags",
+          "ec2:AuthorizeSecurityGroupIngress",
+          "ec2:RevokeSecurityGroupIngress",
+          "ec2:DeleteSecurityGroup",
+          "elasticloadbalancing:*",
+          "acm:ListCertificates",
+          "acm:DescribeCertificate",
+          "acm:GetCertificate",
+          "iam:CreateServiceLinkedRole",
+          "iam:ListServerCertificates",
+          "iam:GetServerCertificate",
+          "tag:GetResources",
+          "tag:TagResources",
+          "wafv2:GetWebACLForResource",
+          "shield:GetSubscriptionState"
+        ]
+        Resource = "*"
+      }
+    ]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_lbc" {
+  role       = aws_iam_role.node.name
+  policy_arn = aws_iam_policy.node_lbc.arn
+}
+
+# The EBS CSI driver also runs with the node's instance-profile credentials, so
+# the volume-lifecycle permissions it needs must live on the node role too. This
+# mirrors the upstream AmazonEBSCSIDriverPolicy. Resources are "*" since Spinifex
+# evaluates grants by action; the CreateVolume/CreateSnapshot/CreateTags grants
+# upstream scope with request/resource tag conditions, omitted here because
+# Spinifex does not yet evaluate IAM tag conditions.
+resource "aws_iam_policy" "node_ebs_csi" {
+  name = "${var.cluster_name}-node-ebs-csi"
+
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Effect = "Allow"
+        Action = [
+          "ec2:CreateSnapshot",
+          "ec2:AttachVolume",
+          "ec2:DetachVolume",
+          "ec2:ModifyVolume",
+          "ec2:DescribeAvailabilityZones",
+          "ec2:DescribeInstances",
+          "ec2:DescribeSnapshots",
+          "ec2:DescribeTags",
+          "ec2:DescribeVolumes",
+          "ec2:DescribeVolumesModifications",
+          "ec2:CreateTags",
+          "ec2:DeleteTags",
+          "ec2:CreateVolume",
+          "ec2:DeleteVolume",
+          "ec2:DeleteSnapshot"
+        ]
+        Resource = "*"
+      }
+    ]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_ebs_csi" {
+  role       = aws_iam_role.node.name
+  policy_arn = aws_iam_policy.node_ebs_csi.arn
+}
+
+# ---------------------------------------------------------------------------
+# IAM — a second principal to grant read-only cluster access to
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "viewer" {
+  name = "${var.cluster_name}-viewer"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+# ---------------------------------------------------------------------------
+# ECR — repository the workers pull the demo image from
+# ---------------------------------------------------------------------------
+
+resource "aws_ecr_repository" "demo" {
+  name = "spinifex-demo"
+
+  # tofu destroy must remove the repo even though it still holds the pushed demo
+  # image; without this, DeleteRepository (force=false) fails RepositoryNotEmpty.
+  force_delete = true
+}
+
+# ---------------------------------------------------------------------------
+# EKS cluster — public + private endpoints, LBC-owned ingress
+#
+# The spinifex.io/managed-ingress = "false" tag disables K3s' built-in
+# traefik/servicelb so the AWS Load Balancer Controller owns ingress, matching
+# how ingress works on AWS EKS.
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_cluster" "this" {
+  name     = var.cluster_name
+  role_arn = aws_iam_role.cluster.arn
+  version  = var.k8s_version
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.private_a.id, aws_subnet.private_b.id]
+    endpoint_public_access  = true
+    endpoint_private_access = true
+    public_access_cidrs     = [var.api_public_access_cidr]
+  }
+
+  access_config {
+    authentication_mode                         = "API"
+    bootstrap_cluster_creator_admin_permissions = true
+  }
+
+  depends_on = [aws_iam_role_policy_attachment.cluster]
+
+  # Spinifex's DescribeCluster doesn't echo these back the way the AWS provider
+  # expects: access_config reads as absent (and its ForceNew
+  # bootstrap_cluster_creator_admin_permissions would replace the live cluster on
+  # re-add), and vpc_config.security_group_ids reads back as drift. Ignore both so
+  # later applies (e.g. adding the Argo CD NodePort rule) neither replace the
+  # cluster nor detach its security groups.
+  lifecycle {
+    ignore_changes = [access_config, vpc_config[0].security_group_ids]
+  }
+
+  tags = {
+    Name                          = var.cluster_name
+    "spinifex.io/managed-ingress" = "false"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Managed node group — workers in the private subnets
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_node_group" "workers" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "workers"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.private_a.id, aws_subnet.private_b.id]
+
+  scaling_config {
+    desired_size = var.node_desired_size
+    min_size     = var.node_desired_size
+    max_size     = var.node_desired_size * 2
+  }
+
+  instance_types = [var.node_instance_type]
+  ami_type       = "AL2_x86_64"
+
+  depends_on = [
+    aws_iam_role_policy_attachment.node_worker,
+    aws_iam_role_policy_attachment.node_cni,
+    aws_iam_role_policy_attachment.node_ecr,
+    aws_iam_role_policy_attachment.node_lbc,
+    aws_iam_role_policy_attachment.node_ebs_csi,
+  ]
+
+  tags = {
+    Name = "${var.cluster_name}-workers"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Addon — AWS Load Balancer Controller
+#
+# Spinifex wires the controller's AWS credentials and ELB-eligible subnets at the
+# node level, so no IRSA role or subnet tagging is needed here. addon_version is
+# omitted: the AWS provider demands a v-prefixed version that the catalog rejects,
+# so let Spinifex default to its catalog version.
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_addon" "lbc" {
+  cluster_name                = aws_eks_cluster.this.name
+  addon_name                  = "aws-load-balancer-controller"
+  resolve_conflicts_on_create = "OVERWRITE"
+
+  depends_on = [aws_eks_node_group.workers]
+
+  tags = {
+    Name = "${var.cluster_name}-lbc"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Addon — Argo CD (GitOps delivery)
+#
+# Installed through the EKS API; Spinifex stages the bundle host-side and the
+# worker renders it into the K3s auto-deploy dir. The workloads module registers
+# the git repo and creates the Argo CD Application that syncs the demo app.
+# addon_version omitted (see the LBC note).
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_addon" "argocd" {
+  cluster_name                = aws_eks_cluster.this.name
+  addon_name                  = "argocd"
+  resolve_conflicts_on_create = "OVERWRITE"
+
+  depends_on = [aws_eks_node_group.workers]
+
+  tags = {
+    Name = "${var.cluster_name}-argocd"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Addon — EBS CSI driver (Viperblock-backed PersistentVolumes)
+#
+# Ships a default gp3 StorageClass (provisioner ebs.csi.aws.com). A PVC in the
+# demo app's git manifests dynamically provisions a Viperblock-backed EBS volume
+# so the app's state survives pod restarts. addon_version omitted (see LBC note).
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_addon" "ebs_csi" {
+  cluster_name                = aws_eks_cluster.this.name
+  addon_name                  = "aws-ebs-csi-driver"
+  resolve_conflicts_on_create = "OVERWRITE"
+
+  depends_on = [aws_eks_node_group.workers]
+
+  tags = {
+    Name = "${var.cluster_name}-ebs-csi"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Access entry — grant the viewer principal read-only access (API auth mode)
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_access_entry" "viewer" {
+  cluster_name  = aws_eks_cluster.this.name
+  principal_arn = aws_iam_role.viewer.arn
+  type          = "STANDARD"
+}
+
+resource "aws_eks_access_policy_association" "viewer" {
+  cluster_name  = aws_eks_cluster.this.name
+  principal_arn = aws_iam_role.viewer.arn
+  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
+
+  access_scope {
+    type = "cluster"
+  }
+
+  depends_on = [aws_eks_access_entry.viewer]
+}
+
+# ---------------------------------------------------------------------------
+# TLS — self-signed certificate imported into ACM
+#
+# Spinifex ACM supports ImportCertificate (not RequestCertificate), so the cert
+# is generated locally by the tls provider and imported. The workloads Ingress
+# attaches it to the ALB's HTTPS listener by ARN.
+# ---------------------------------------------------------------------------
+
+resource "tls_private_key" "ingress" {
+  algorithm = "RSA"
+  rsa_bits  = 2048
+}
+
+resource "tls_self_signed_cert" "ingress" {
+  private_key_pem = tls_private_key.ingress.private_key_pem
+
+  subject {
+    common_name  = var.cert_common_name
+    organization = "Spinifex EKS Demo"
+  }
+
+  # Wildcard SAN so one ALB (shared IngressGroup) can host-route both the demo
+  # app (app.<cn>) and the Argo CD UI (argocd.<cn>) off this single cert.
+  dns_names             = [var.cert_common_name, "*.${var.cert_common_name}"]
+  validity_period_hours = 8760
+  early_renewal_hours   = 720
+
+  allowed_uses = [
+    "key_encipherment",
+    "digital_signature",
+    "server_auth",
+  ]
+}
+
+resource "aws_acm_certificate" "ingress" {
+  private_key      = tls_private_key.ingress.private_key_pem
+  certificate_body = tls_self_signed_cert.ingress.cert_pem
+
+  tags = {
+    Name = "${var.cluster_name}-ingress"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Let the ALB reach the workers' NodePort
+#
+# Spinifex auto-manages the nodegroup SG and admits only intra-cluster traffic.
+# The LBC-provisioned ALB lives in this VPC, so admit the VPC CIDR on the
+# NodePort. Look the SG up by its deterministic name and add one rule.
+# ---------------------------------------------------------------------------
+
+data "aws_security_group" "nodegroup" {
+  filter {
+    name   = "group-name"
+    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
+  }
+
+  filter {
+    name   = "vpc-id"
+    values = [aws_vpc.main.id]
+  }
+
+  depends_on = [aws_eks_node_group.workers]
+}
+
+resource "aws_vpc_security_group_ingress_rule" "nodeport_from_vpc" {
+  security_group_id = data.aws_security_group.nodegroup.id
+  cidr_ipv4         = aws_vpc.main.cidr_block
+  from_port         = var.node_port
+  to_port           = var.node_port
+  ip_protocol       = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-alb-nodeport"
+  }
+}
+
+# Same, for the Argo CD UI NodePort its ALB forwards to.
+resource "aws_vpc_security_group_ingress_rule" "argocd_nodeport_from_vpc" {
+  security_group_id = data.aws_security_group.nodegroup.id
+  cidr_ipv4         = aws_vpc.main.cidr_block
+  from_port         = var.argocd_node_port
+  to_port           = var.argocd_node_port
+  ip_protocol       = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-argocd-nodeport"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Outputs
+# ---------------------------------------------------------------------------
+
+output "cluster_name" {
+  value = aws_eks_cluster.this.name
+}
+
+output "region" {
+  value = var.region
+}
+
+output "node_port" {
+  value = var.node_port
+}
+
+output "argocd_node_port" {
+  value = var.argocd_node_port
+}
+
+output "node_desired_size" {
+  value = var.node_desired_size
+}
+
+output "certificate_arn" {
+  value = aws_acm_certificate.ingress.arn
+}
+
+output "cert_common_name" {
+  value = var.cert_common_name
+}
+
+output "viewer_principal_arn" {
+  value = aws_iam_role.viewer.arn
+}
+
+output "ecr_repository_url" {
+  value       = aws_ecr_repository.demo.repository_url
+  description = "Push the demo-app image here, then apply the workloads module"
+}
+
+output "ingress_address_hint" {
+  value = "The demo app and Argo CD UI share one ALB (LBC IngressGroup), host-routed: app.${var.cert_common_name} and argocd.${var.cert_common_name}. Get the address: kubectl get ingress spinifex-demo -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{\"\\n\"}'."
+}
+
+output "argocd_ingress_hint" {
+  value = "Same ALB as the app. Open https://argocd.${var.cert_common_name} (admin password: kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d). Resolve the host to the ALB address, or curl -k --resolve."
+}
+
+output "update_kubeconfig" {
+  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
+}
@@ -0,0 +1,370 @@
+# GitOps workload for eks-gitops-argocd — separate root module / state.
+#
+# Instead of applying the app directly, this module hands delivery to Argo CD:
+# it registers the (private) git repo as an Argo CD repository credential and
+# creates an Argo CD Application that syncs the demo app's manifests from it. The
+# app's Deployment, Service, and PersistentVolumeClaim live in the git repo (see
+# ../../../../eks-demo-app); the HTTPS Ingress (LBC + ACM) stays here and points
+# at the Service that Argo CD creates.
+#
+# The Argo CD Application is a kubernetes_manifest, so the argoproj.io CRDs must
+# already exist — apply the parent module (which installs the argocd addon) and
+# let it reach ACTIVE before applying this module.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-gitops-argocd
+#   tofu init && tofu apply                       # parent: cluster + addons + ACM
+#   cd workloads && tofu init
+#   tofu apply -var git_repo_url=https://github.com/mulgadc/eks-demo-app.git \
+#              -var git_token=<a-read-only-PAT>
+#   # teardown is the reverse — destroy here first, then the parent.
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.20"
+    }
+  }
+}
+
+variable "spinifex_endpoint" {
+  type    = string
+  default = "https://127.0.0.1:9999"
+}
+
+variable "git_repo_url" {
+  type        = string
+  default     = "https://github.com/mulgadc/eks-demo-app.git"
+  description = "Git repo Argo CD syncs the demo app from"
+}
+
+variable "git_revision" {
+  type        = string
+  default     = "main"
+  description = "Git branch, tag, or commit Argo CD tracks"
+}
+
+variable "git_path" {
+  type        = string
+  default     = "manifests"
+  description = "Path within the repo holding the app manifests"
+}
+
+variable "git_username" {
+  type        = string
+  default     = "git"
+  description = "Username for the git credential (any non-empty value for a PAT)"
+}
+
+variable "git_token" {
+  type        = string
+  default     = ""
+  sensitive   = true
+  description = "Read-only personal access token for the private repo. Leave empty for a public repo."
+}
+
+provider "aws" {
+  region = data.terraform_remote_state.infra.outputs.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+data "terraform_remote_state" "infra" {
+  backend = "local"
+
+  config = {
+    path = "../terraform.tfstate"
+  }
+}
+
+locals {
+  cluster_name     = data.terraform_remote_state.infra.outputs.cluster_name
+  region           = data.terraform_remote_state.infra.outputs.region
+  cert_arn         = data.terraform_remote_state.infra.outputs.certificate_arn
+  argocd_node_port = data.terraform_remote_state.infra.outputs.argocd_node_port
+  cert_cn          = data.terraform_remote_state.infra.outputs.cert_common_name
+  private_repo     = var.git_token != ""
+
+  # The demo app and the Argo CD UI share one ALB via an LBC IngressGroup; LBC
+  # gives each backend its own target group and host-routes between them.
+  alb_group   = local.cluster_name
+  app_host    = "app.${local.cert_cn}"
+  argocd_host = "argocd.${local.cert_cn}"
+}
+
+data "aws_eks_cluster" "this" {
+  name = local.cluster_name
+}
+
+provider "kubernetes" {
+  host                   = data.aws_eks_cluster.this.endpoint
+  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
+
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    command     = "aws"
+    args        = ["eks", "get-token", "--cluster-name", local.cluster_name, "--region", local.region]
+  }
+}
+
+# Repository credential for the private repo. Argo CD picks up Secrets in its
+# namespace labelled argocd.argoproj.io/secret-type=repository. Created only when
+# a token is supplied (a public repo needs no credential).
+resource "kubernetes_secret_v1" "repo" {
+  count = local.private_repo ? 1 : 0
+
+  metadata {
+    name      = "eks-demo-app-repo"
+    namespace = "argocd"
+    labels = {
+      "argocd.argoproj.io/secret-type" = "repository"
+    }
+  }
+
+  data = {
+    type     = "git"
+    url      = var.git_repo_url
+    username = var.git_username
+    password = var.git_token
+  }
+}
+
+# Argo CD Application: syncs the demo app from the git repo into the default
+# namespace, self-healing and pruning so the cluster tracks the repo.
+resource "kubernetes_manifest" "demo_app" {
+  manifest = {
+    apiVersion = "argoproj.io/v1alpha1"
+    kind       = "Application"
+
+    # The resources finalizer makes deleting this Application cascade-prune the
+    # synced resources (Deployment, Service, PVC). Without it, tofu destroy drops
+    # the Application CR but orphans the live PVC, whose Delete-reclaim PV then
+    # leaks an EBS volume once the cluster (and ebs-csi) is torn down.
+    metadata = {
+      name       = "spinifex-demo"
+      namespace  = "argocd"
+      finalizers = ["resources-finalizer.argocd.argoproj.io"]
+    }
+
+    spec = {
+      project = "default"
+
+      source = {
+        repoURL        = var.git_repo_url
+        targetRevision = var.git_revision
+        path           = var.git_path
+      }
+
+      destination = {
+        server    = "https://kubernetes.default.svc"
+        namespace = "default"
+      }
+
+      syncPolicy = {
+        automated = {
+          prune    = true
+          selfHeal = true
+        }
+        syncOptions = ["CreateNamespace=true"]
+      }
+    }
+  }
+
+  depends_on = [kubernetes_secret_v1.repo]
+}
+
+# HTTPS Ingress (LBC + ACM), carried over from eks-https-ingress. It points at
+# the spinifex-demo Service that Argo CD creates from the git manifests.
+resource "kubernetes_ingress_v1" "demo" {
+  metadata {
+    name      = "spinifex-demo"
+    namespace = "default"
+
+    annotations = {
+      "alb.ingress.kubernetes.io/scheme"           = "internet-facing"
+      "alb.ingress.kubernetes.io/target-type"      = "instance"
+      "alb.ingress.kubernetes.io/group.name"       = local.alb_group
+      "alb.ingress.kubernetes.io/listen-ports"     = "[{\"HTTP\":80},{\"HTTPS\":443}]"
+      "alb.ingress.kubernetes.io/certificate-arn"  = local.cert_arn
+      "alb.ingress.kubernetes.io/ssl-redirect"     = "443"
+      "alb.ingress.kubernetes.io/healthcheck-path" = "/healthz"
+    }
+  }
+
+  spec {
+    ingress_class_name = "alb"
+
+    # No host condition: the app is the ALB's catch-all on :443, so it answers on
+    # the raw ALB IP as well as app.<cn> once DNS is wired. Argo CD's host rule is
+    # more specific and still wins for argocd.<cn>.
+    rule {
+      http {
+        path {
+          path      = "/"
+          path_type = "Prefix"
+
+          backend {
+            service {
+              name = "spinifex-demo"
+              port {
+                number = 80
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+
+  depends_on = [kubernetes_manifest.demo_app]
+}
+
+# ---------------------------------------------------------------------------
+# Argo CD UI exposure — managing the cluster through the GitOps console is the
+# point of this demo, so the UI gets the same HTTPS-Ingress treatment as the app
+# rather than a port-forward. The argocd addon ships argocd-server as a ClusterIP
+# only; expose it via a NodePort the ALB targets. argocd-server serves TLS on
+# 8080 (no --insecure), so the ALB speaks HTTPS to the backend.
+#
+# Same group.name as the demo Ingress, so LBC folds both onto ONE ALB. On :443
+# the app is the catch-all and argocd.<cn> host-routes to the UI. Argo CD also
+# gets a hostless :8443 listener (argocd_ip below) so the UI is reachable on the
+# raw ALB IP before DNS exists — https://<alb-ip>:8443.
+# ---------------------------------------------------------------------------
+
+resource "kubernetes_service_v1" "argocd_server_nodeport" {
+  metadata {
+    name      = "argocd-server-nodeport"
+    namespace = "argocd"
+  }
+
+  spec {
+    type     = "NodePort"
+    selector = { "app.kubernetes.io/name" = "argocd-server" }
+
+    port {
+      port        = 443
+      target_port = 8080
+      node_port   = local.argocd_node_port
+      protocol    = "TCP"
+    }
+  }
+}
+
+resource "kubernetes_ingress_v1" "argocd" {
+  metadata {
+    name      = "argocd-server"
+    namespace = "argocd"
+
+    annotations = {
+      "alb.ingress.kubernetes.io/scheme"               = "internet-facing"
+      "alb.ingress.kubernetes.io/target-type"          = "instance"
+      "alb.ingress.kubernetes.io/group.name"           = local.alb_group
+      "alb.ingress.kubernetes.io/listen-ports"         = "[{\"HTTPS\":443}]"
+      "alb.ingress.kubernetes.io/certificate-arn"      = local.cert_arn
+      "alb.ingress.kubernetes.io/backend-protocol"     = "HTTPS"
+      "alb.ingress.kubernetes.io/healthcheck-protocol" = "HTTPS"
+      "alb.ingress.kubernetes.io/healthcheck-path"     = "/healthz"
+    }
+  }
+
+  spec {
+    ingress_class_name = "alb"
+
+    rule {
+      host = local.argocd_host
+
+      http {
+        path {
+          path      = "/"
+          path_type = "Prefix"
+
+          backend {
+            service {
+              name = kubernetes_service_v1.argocd_server_nodeport.metadata[0].name
+              port {
+                number = 443
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+}
+
+# Hostless Argo CD listener on :8443 of the same ALB, so the UI is reachable on
+# the raw ALB IP (https://<alb-ip>:8443) without DNS or a host header. argocd.<cn>
+# on :443 still works via the host-routed ingress above.
+resource "kubernetes_ingress_v1" "argocd_ip" {
+  metadata {
+    name      = "argocd-server-ip"
+    namespace = "argocd"
+
+    annotations = {
+      "alb.ingress.kubernetes.io/scheme"               = "internet-facing"
+      "alb.ingress.kubernetes.io/target-type"          = "instance"
+      "alb.ingress.kubernetes.io/group.name"           = local.alb_group
+      "alb.ingress.kubernetes.io/listen-ports"         = "[{\"HTTPS\":8443}]"
+      "alb.ingress.kubernetes.io/certificate-arn"      = local.cert_arn
+      "alb.ingress.kubernetes.io/backend-protocol"     = "HTTPS"
+      "alb.ingress.kubernetes.io/healthcheck-protocol" = "HTTPS"
+      "alb.ingress.kubernetes.io/healthcheck-path"     = "/healthz"
+    }
+  }
+
+  spec {
+    ingress_class_name = "alb"
+
+    rule {
+      http {
+        path {
+          path      = "/"
+          path_type = "Prefix"
+
+          backend {
+            service {
+              name = kubernetes_service_v1.argocd_server_nodeport.metadata[0].name
+              port {
+                number = 443
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+}
+
+output "application_name" {
+  value = kubernetes_manifest.demo_app.manifest.metadata.name
+}
+
+output "alb_address_hint" {
+  value = "One shared ALB. Get its IP: kubectl get ingress spinifex-demo -o jsonpath='{.status.loadBalancer.ingress[0].ip}{\"\\n\"}' (or [0].hostname). Reach by raw IP — app: https://<alb-ip>/  Argo CD: https://<alb-ip>:8443/ . By DNS — app: ${local.app_host}  Argo CD: ${local.argocd_host}."
+}
+
+output "argocd_admin_password_cmd" {
+  value = "kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d ; echo"
+}
+
+output "argocd_url_hint" {
+  value = "Open https://${local.argocd_host} (admin password: kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d). Resolve the host to the ALB address (northstar/CNAME), or curl -k --resolve ${local.argocd_host}:443:<alb-ip>."
+}
@@ -0,0 +1,207 @@
+---
+title: "EKS HTTPS Ingress (LBC + ACM)"
+description: "Serve a Spinifex-themed demo app over HTTPS on EKS using the AWS Load Balancer Controller addon and an ACM certificate — an internet-facing ALB provisioned from a Kubernetes Ingress, using Terraform on Spinifex."
+category: "Terraform Workbooks"
+tags:
+  - terraform
+  - eks
+  - kubernetes
+  - ingress
+  - alb
+  - acm
+  - https
+  - workbook
+resources:
+  - title: "AWS Load Balancer Controller"
+    url: "https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/"
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "Terraform Kubernetes Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+---
+
+# Terraform: EKS HTTPS Ingress (LBC + ACM)
+
+> Builds on the quickstart. Instead of a raw NodePort, it serves the Spinifex-themed demo app over **HTTPS** through an ALB that the **AWS Load Balancer Controller** provisions from a Kubernetes **Ingress**, terminating TLS with a certificate held in **ACM** — the same pattern you'd use on AWS EKS.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+This is the second rung of the EKS ladder. It takes the cluster + demo app from [EKS Quickstart](../eks-quickstart) and publishes the app the way you would in production: a Kubernetes `Ingress` reconciled by the **AWS Load Balancer Controller (LBC)** into an internet-facing **Application Load Balancer** that terminates TLS with an **ACM** certificate.
+
+To make room for an internet-facing ALB with private workers, the network grows from the quickstart's two public subnets to a public/private split: the ALB and a **NAT gateway** sit in the public subnets, the workers sit in the private subnets and reach ECR through the NAT. The cluster is tagged `spinifex.io/managed-ingress = "false"`, which disables K3s' built-in traefik/servicelb so the LBC owns ingress.
+
+The LBC is installed as a managed **addon** (`aws_eks_addon`). Spinifex wires the controller's AWS credentials and the cluster's ELB-eligible subnets at the node level and injects those subnets into the `alb` `IngressClassParams`, so the Ingress needs **no IRSA role and no subnet tags** — just `ingressClassName: alb` and a handful of annotations.
+
+**What you'll learn:**
+
+- Installing the `aws-load-balancer-controller` addon and letting it own ingress
+- Disabling the built-in K3s ingress with the `spinifex.io/managed-ingress` tag
+- Importing a self-signed certificate into ACM and attaching it to an ALB listener
+- Driving an ALB entirely from a Kubernetes `Ingress` (`ingressClassName: alb`)
+- A public/private VPC with a NAT gateway for private workers
+
+**What gets created**
+
+| Resource | Name | Purpose |
+|---|---|---|
+| VPC | `eks-https-vpc` | Isolated network (10.31.0.0/16) |
+| Subnets | `eks-https-public-a/-b`, `eks-https-private-a/-b` | Public (ALB + NAT) and private (workers) |
+| Internet + NAT Gateway | `eks-https-igw`, `eks-https-nat` | Public egress; private-worker egress to ECR |
+| IAM Roles | `eks-https-cluster-role`, `eks-https-node-role` | Control-plane and worker roles |
+| ECR Repository | `spinifex-demo` | Holds the demo image the workers pull |
+| EKS Cluster | `eks-https` | Public + private endpoints; `managed-ingress=false` |
+| Node Group | `workers` | `node_desired_size` `t3.large` worker(s) — 1 or 3 |
+| Addon | `aws-load-balancer-controller` | Provisions the ALB from the Ingress |
+| ACM Certificate | `eks-https-ingress` | Self-signed, imported; attached to the HTTPS listener |
+| SG Ingress Rule | `eks-https-alb-nodeport` | Admits the VPC CIDR to the workers' NodePort |
+| K8s Deployment + Service | `spinifex-demo` | Themed demo image, 2 replicas, NodePort |
+| K8s Ingress | `spinifex-demo` | `ingressClassName: alb`, HTTPS via the ACM cert |
+
+**Spinifex specifics**
+
+- **The LBC needs no IRSA or subnet tagging here.** Credentials and ELB-eligible subnets are injected by Spinifex at the node level and into the `alb` `IngressClassParams`. The Ingress just sets `ingressClassName: alb` plus the `alb.ingress.kubernetes.io/*` annotations.
+- **`spinifex.io/managed-ingress = "false"`** disables the K3s built-in traefik/servicelb so the LBC owns ingress, matching AWS parity. Omit the tag (or set it `true`) and the built-in ingress stays on.
+- The ALB targets the workers' **NodePort** (`target-type: instance`). The one SG rule admits the VPC CIDR on that port so the in-VPC ALB can reach the workers; the worker SG is otherwise auto-managed.
+- Spinifex ACM supports **ImportCertificate only** — the cert is generated by the `tls` provider and imported, not requested.
+- **Leave `addon_version` unset** (see the [quickstart/addons notes](../eks-quickstart)) — the AWS provider and the Spinifex catalog disagree on the version string format, so let the server choose.
+
+**Prerequisites:**
+
+- Spinifex installed and running, with the `eks-node` image (carrying the LBC bundle) available
+- OpenTofu or Terraform, plus `kubectl`, the AWS CLI, and Docker
+- The demo image built and pushed to ECR (see [`../demo-app/README.md`](../demo-app/README.md))
+
+## Instructions
+
+### Step 1. Get the Template
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-https-ingress
+```
+
+Or create a `main.tf` file and paste the full configuration below.
+
+<!-- INCLUDE: main.tf lang:hcl -->
+
+### Step 2. Deploy the Cluster
+
+The cluster and the demo app are **two root modules with separate state**. Apply the cluster first (set `node_desired_size=3` for an HA-shaped cluster):
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+```
+
+This brings up the cluster, the workers, the NAT gateway, the LBC addon, the ACM cert, and the `spinifex-demo` ECR repository. Give it a few minutes to reach `ACTIVE` and for the LBC addon to report healthy.
+
+### Step 3. Build and Push the Demo Image
+
+```bash
+cd ../demo-app
+REGISTRY=$(cd ../eks-https-ingress && tofu output -raw ecr_repository_url)
+REGISTRY_HOST=${REGISTRY%%/*}
+aws ecr get-login-password | docker login --username AWS --password-stdin "$REGISTRY_HOST"
+docker build -t "${REGISTRY}:latest" .
+docker push "${REGISTRY}:latest"
+cd ../eks-https-ingress
+```
+
+### Step 4. Deploy the App and Ingress
+
+```bash
+cd workloads
+tofu init
+tofu apply
+```
+
+> **Why two modules?** The Kubernetes provider in `workloads/` reads the cluster endpoint from a live `data "aws_eks_cluster"` source, so it's only configured while the cluster exists. Always destroy `workloads/` before the cluster, or the provider falls back to `http://localhost:80` and reports `connection refused`. Keep `AWS_PROFILE=spinifex` exported for both applies.
+
+<!-- INCLUDE: workloads/main.tf lang:hcl -->
+
+### Step 5. Open the Demo over HTTPS
+
+The LBC takes a minute or two to provision the ALB after the Ingress is created. The Ingress publishes the ALB's **DNS name**, which has no resolver yet (northstar will add one) — resolve it to the ALB's public IP with `describe-load-balancers` so you can reach it without DNS or `/etc/hosts`:
+
+```bash
+# Point kubectl at the cluster first (writes ~/.kube/config) — without this every
+# kubectl call fails to connect:
+aws eks update-kubeconfig --name eks-https --region ap-southeast-2
+
+kubectl get ingress spinifex-demo -o wide
+DNSNAME=$(kubectl get ingress spinifex-demo -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
+IP=$(aws elbv2 describe-load-balancers \
+  --query "LoadBalancers[?DNSName=='${DNSNAME}'].AvailabilityZones[].LoadBalancerAddresses[].IpAddress | [0]" \
+  --output text)
+curl -k "https://$IP"          # self-signed cert: -k skips verification
+```
+
+Open `https://$IP` in a browser (accept the self-signed certificate). The Spinifex-themed page reports the pod, node, cluster, and region that answered — refresh to watch requests land on different replicas, now over HTTPS through the ALB.
+
+### Cleanup
+
+Destroy in reverse — the app/Ingress first (so the LBC tears the ALB down while the cluster is up), then the cluster:
+
+```bash
+cd workloads
+tofu destroy
+cd ..
+tofu destroy
+```
+
+## Troubleshooting
+
+### The Ingress Has No Address
+
+The LBC populates `status.loadBalancer` once it has provisioned the ALB. If it stays empty:
+
+```bash
+kubectl describe ingress spinifex-demo            # events show LBC reconcile errors
+kubectl -n kube-system logs deploy/aws-load-balancer-controller --tail=50
+aws eks describe-addon --cluster-name eks-https --addon-name aws-load-balancer-controller --query 'addon.status'
+```
+
+Confirm the addon is `ACTIVE` and the cluster carries `spinifex.io/managed-ingress = "false"` (otherwise the built-in ingress competes).
+
+> **Verify on a live cluster.** The LBC reconcile path (ALB creation, target registration, listener wiring) exercises the running Spinifex EKS data plane end to end. Stand this up on a live stack to confirm the ALB serves the page; the manifests here are the AWS-parity shape of that flow.
+
+### HTTPS Loads but Returns 502 / 504
+
+The ALB reached the workers but the backend didn't answer. Check the demo pods and the NodePort SG rule:
+
+```bash
+kubectl get pods -o wide
+kubectl get svc spinifex-demo
+aws ec2 describe-security-groups --filters "Name=group-name,Values=eks-cluster-eks-https-nodegroup-sg" \
+  --query 'SecurityGroups[0].IpPermissions'
+```
+
+The `eks-https-alb-nodeport` rule must admit the VPC CIDR on the NodePort, and the target group health check (`/healthz`) must pass.
+
+### Addon Fails With "no baked bundle"
+
+Spinifex only installs addons whose manifests are baked into the `eks-node` AMI. If `describe-addon` reports `CREATE_FAILED` with `no baked bundle`, the running worker image predates the LBC bundle — rebuild and republish the `eks-node` image, then recreate the addon.
+
+### Pods Stuck in ImagePullBackOff
+
+The private workers pull from ECR through the NAT gateway. Confirm the image was pushed (`tofu output ecr_repository_url`), the NAT route is in place, and the node role carries `AmazonEC2ContainerRegistryReadOnly`.
+
+### Provider Connection Refused
+
+```bash
+sudo systemctl status spinifex.target
+curl -k https://localhost:9999/
+```
@@ -0,0 +1,577 @@
+# Example: EKS HTTPS Ingress via the AWS Load Balancer Controller + ACM
+#
+# Builds on eks-quickstart. Instead of poking a NodePort open on the worker, this
+# workbook serves the Spinifex-themed demo app over HTTPS through an Application
+# Load Balancer that the AWS Load Balancer Controller (LBC) provisions from a
+# Kubernetes Ingress — the same pattern you'd use on AWS EKS.
+#
+# What it adds over the quickstart:
+#   * Public subnets for the ALB + a NAT gateway, private subnets for the workers.
+#   * The aws-load-balancer-controller addon, installed through the EKS API.
+#   * The cluster tag spinifex.io/managed-ingress = "false", which disables K3s'
+#     built-in traefik/servicelb so the LBC owns ingress (AWS parity).
+#   * A self-signed certificate imported into ACM and attached to the ALB's HTTPS
+#     listener via an Ingress annotation.
+#
+# The workloads/ module then creates a Deployment, a NodePort Service, and an
+# Ingress (ingressClassName: alb). The LBC reconciles that Ingress into an
+# internet-facing ALB that terminates TLS with the ACM cert and forwards to the
+# workers' NodePort. The cluster's ELB-eligible subnets are injected by Spinifex
+# into the alb IngressClassParams, so the Ingress needs no subnet annotations.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-https-ingress
+#   export AWS_PROFILE=spinifex
+#   tofu init && tofu apply
+#   # build + push the demo image to the ECR repo this creates (see README),
+#   # then: cd workloads && tofu init && tofu apply
+#   # finally: kubectl get ingress spinifex-demo -o wide  → open https://<address>
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    tls = {
+      source  = "hashicorp/tls"
+      version = ">= 4.0"
+    }
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Variables
+# ---------------------------------------------------------------------------
+
+variable "region" {
+  type    = string
+  default = "ap-southeast-2"
+}
+
+variable "cluster_name" {
+  type    = string
+  default = "eks-https"
+}
+
+variable "k8s_version" {
+  type    = string
+  default = "1.32"
+}
+
+variable "node_instance_type" {
+  type    = string
+  default = "t3.medium"
+}
+
+variable "node_desired_size" {
+  type        = number
+  default     = 1
+  description = "Worker count. Use 1 for a single-node demo, or 3 for an HA-shaped cluster."
+
+  validation {
+    condition     = var.node_desired_size == 1 || var.node_desired_size == 3
+    error_message = "node_desired_size must be 1 or 3."
+  }
+}
+
+variable "node_port" {
+  type        = number
+  default     = 30080
+  description = "NodePort the demo Service is published on and the ALB forwards to"
+}
+
+variable "cert_common_name" {
+  type    = string
+  default = "eks-https.spinifex.local"
+}
+
+variable "api_public_access_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the public Kubernetes API endpoint; tighten in production"
+}
+
+variable "spinifex_endpoint" {
+  type    = string
+  default = "https://127.0.0.1:9999"
+}
+
+# ---------------------------------------------------------------------------
+# Providers
+# ---------------------------------------------------------------------------
+
+provider "aws" {
+  region = var.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+    ecr = var.spinifex_endpoint
+    acm = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+data "aws_availability_zones" "available" {
+  state = "available"
+}
+
+# ---------------------------------------------------------------------------
+# VPC — public subnets for the ALB + NAT, private subnets for the workers
+# ---------------------------------------------------------------------------
+
+resource "aws_vpc" "main" {
+  cidr_block           = "10.31.0.0/16"
+  enable_dns_hostnames = true
+  enable_dns_support   = true
+
+  tags = {
+    Name = "${var.cluster_name}-vpc"
+  }
+}
+
+resource "aws_internet_gateway" "igw" {
+  vpc_id = aws_vpc.main.id
+
+  tags = {
+    Name = "${var.cluster_name}-igw"
+  }
+}
+
+resource "aws_subnet" "public_a" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.31.1.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-public-a"
+  }
+}
+
+resource "aws_subnet" "public_b" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.31.2.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-public-b"
+  }
+}
+
+resource "aws_subnet" "private_a" {
+  vpc_id            = aws_vpc.main.id
+  cidr_block        = "10.31.11.0/24"
+  availability_zone = data.aws_availability_zones.available.names[0]
+
+  tags = {
+    Name = "${var.cluster_name}-private-a"
+  }
+}
+
+resource "aws_subnet" "private_b" {
+  vpc_id            = aws_vpc.main.id
+  cidr_block        = "10.31.12.0/24"
+  availability_zone = data.aws_availability_zones.available.names[0]
+
+  tags = {
+    Name = "${var.cluster_name}-private-b"
+  }
+}
+
+resource "aws_route_table" "public" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block = "0.0.0.0/0"
+    gateway_id = aws_internet_gateway.igw.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-public-rt"
+  }
+}
+
+resource "aws_route_table_association" "public_a" {
+  subnet_id      = aws_subnet.public_a.id
+  route_table_id = aws_route_table.public.id
+}
+
+resource "aws_route_table_association" "public_b" {
+  subnet_id      = aws_subnet.public_b.id
+  route_table_id = aws_route_table.public.id
+}
+
+# NAT gateway gives the private workers outbound internet to pull the demo image
+# from ECR. The workers still join the cluster over the in-VPC private endpoint.
+resource "aws_eip" "nat" {
+  domain = "vpc"
+
+  tags = {
+    Name = "${var.cluster_name}-nat-eip"
+  }
+}
+
+resource "aws_nat_gateway" "nat" {
+  allocation_id = aws_eip.nat.id
+  subnet_id     = aws_subnet.public_a.id
+
+  depends_on = [aws_internet_gateway.igw]
+
+  tags = {
+    Name = "${var.cluster_name}-nat"
+  }
+}
+
+resource "aws_route_table" "private" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block     = "0.0.0.0/0"
+    nat_gateway_id = aws_nat_gateway.nat.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-private-rt"
+  }
+}
+
+resource "aws_route_table_association" "private_a" {
+  subnet_id      = aws_subnet.private_a.id
+  route_table_id = aws_route_table.private.id
+}
+
+resource "aws_route_table_association" "private_b" {
+  subnet_id      = aws_subnet.private_b.id
+  route_table_id = aws_route_table.private.id
+}
+
+# ---------------------------------------------------------------------------
+# IAM — cluster role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "cluster" {
+  name = "${var.cluster_name}-cluster-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "eks.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "cluster" {
+  role       = aws_iam_role.cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — node role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "node" {
+  name = "${var.cluster_name}-node-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_worker" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_cni" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_ecr" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+}
+
+# The AWS Load Balancer Controller runs with the node's instance-profile
+# credentials (Spinifex wires creds at the node level, not IRSA — see the addon
+# block below), so the permissions it needs to manage ALBs must live on the node
+# role. This mirrors the upstream AWSLoadBalancerControllerIAMPolicy, trimmed to
+# the actions an ALB Ingress exercises (Shield/WAF/Cognito are disabled on the
+# controller). Resources are "*" since Spinifex evaluates grants by action.
+# A customer-managed policy + attachment is used (Spinifex implements
+# CreatePolicy/AttachRolePolicy, not inline PutRolePolicy).
+resource "aws_iam_policy" "node_lbc" {
+  name = "${var.cluster_name}-node-lbc"
+
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Effect = "Allow"
+        Action = [
+          "ec2:DescribeAccountAttributes",
+          "ec2:DescribeAddresses",
+          "ec2:DescribeAvailabilityZones",
+          "ec2:DescribeInternetGateways",
+          "ec2:DescribeVpcs",
+          "ec2:DescribeSubnets",
+          "ec2:DescribeSecurityGroups",
+          "ec2:DescribeInstances",
+          "ec2:DescribeNetworkInterfaces",
+          "ec2:DescribeTags",
+          "ec2:GetCoipPoolUsage",
+          "ec2:DescribeCoipPools",
+          "ec2:CreateSecurityGroup",
+          "ec2:CreateTags",
+          "ec2:DeleteTags",
+          "ec2:AuthorizeSecurityGroupIngress",
+          "ec2:RevokeSecurityGroupIngress",
+          "ec2:DeleteSecurityGroup",
+          "elasticloadbalancing:*",
+          "acm:ListCertificates",
+          "acm:DescribeCertificate",
+          "acm:GetCertificate",
+          "iam:CreateServiceLinkedRole",
+          "iam:ListServerCertificates",
+          "iam:GetServerCertificate",
+          "tag:GetResources",
+          "tag:TagResources",
+          "wafv2:GetWebACLForResource",
+          "shield:GetSubscriptionState"
+        ]
+        Resource = "*"
+      }
+    ]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_lbc" {
+  role       = aws_iam_role.node.name
+  policy_arn = aws_iam_policy.node_lbc.arn
+}
+
+# ---------------------------------------------------------------------------
+# ECR — repository the workers pull the demo image from
+# ---------------------------------------------------------------------------
+
+resource "aws_ecr_repository" "demo" {
+  name = "spinifex-demo"
+
+  # tofu destroy must remove the repo even though it still holds the pushed demo
+  # image; without this, DeleteRepository (force=false) fails RepositoryNotEmpty.
+  force_delete = true
+}
+
+# ---------------------------------------------------------------------------
+# EKS cluster — public + private endpoints, LBC-owned ingress
+#
+# The spinifex.io/managed-ingress = "false" tag disables K3s' built-in
+# traefik/servicelb so the AWS Load Balancer Controller owns ingress, matching
+# how ingress works on AWS EKS.
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_cluster" "this" {
+  name     = var.cluster_name
+  role_arn = aws_iam_role.cluster.arn
+  version  = var.k8s_version
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.private_a.id, aws_subnet.private_b.id]
+    endpoint_public_access  = true
+    endpoint_private_access = true
+    public_access_cidrs     = [var.api_public_access_cidr]
+  }
+
+  access_config {
+    authentication_mode                         = "API"
+    bootstrap_cluster_creator_admin_permissions = true
+  }
+
+  depends_on = [aws_iam_role_policy_attachment.cluster]
+
+  tags = {
+    Name                          = var.cluster_name
+    "spinifex.io/managed-ingress" = "false"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Managed node group — workers in the private subnets
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_node_group" "workers" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "workers"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.private_a.id, aws_subnet.private_b.id]
+
+  scaling_config {
+    desired_size = var.node_desired_size
+    min_size     = var.node_desired_size
+    max_size     = var.node_desired_size * 2
+  }
+
+  instance_types = [var.node_instance_type]
+  ami_type       = "AL2_x86_64"
+
+  depends_on = [
+    aws_iam_role_policy_attachment.node_worker,
+    aws_iam_role_policy_attachment.node_cni,
+    aws_iam_role_policy_attachment.node_ecr,
+  ]
+
+  tags = {
+    Name = "${var.cluster_name}-workers"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Addon — AWS Load Balancer Controller
+#
+# Spinifex wires the controller's AWS credentials and ELB-eligible subnets at the
+# node level, so no IRSA role or subnet tagging is needed here. addon_version is
+# omitted: the AWS provider demands a v-prefixed version that the catalog rejects,
+# so let Spinifex default to its catalog version.
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_addon" "lbc" {
+  cluster_name                = aws_eks_cluster.this.name
+  addon_name                  = "aws-load-balancer-controller"
+  resolve_conflicts_on_create = "OVERWRITE"
+
+  depends_on = [aws_eks_node_group.workers]
+
+  tags = {
+    Name = "${var.cluster_name}-lbc"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# TLS — self-signed certificate imported into ACM
+#
+# Spinifex ACM supports ImportCertificate (not RequestCertificate), so the cert
+# is generated locally by the tls provider and imported. The workloads Ingress
+# attaches it to the ALB's HTTPS listener by ARN.
+# ---------------------------------------------------------------------------
+
+resource "tls_private_key" "ingress" {
+  algorithm = "RSA"
+  rsa_bits  = 2048
+}
+
+resource "tls_self_signed_cert" "ingress" {
+  private_key_pem = tls_private_key.ingress.private_key_pem
+
+  subject {
+    common_name  = var.cert_common_name
+    organization = "Spinifex EKS Demo"
+  }
+
+  dns_names             = [var.cert_common_name]
+  validity_period_hours = 8760
+  early_renewal_hours   = 720
+
+  allowed_uses = [
+    "key_encipherment",
+    "digital_signature",
+    "server_auth",
+  ]
+}
+
+resource "aws_acm_certificate" "ingress" {
+  private_key      = tls_private_key.ingress.private_key_pem
+  certificate_body = tls_self_signed_cert.ingress.cert_pem
+
+  tags = {
+    Name = "${var.cluster_name}-ingress"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Let the ALB reach the workers' NodePort
+#
+# Spinifex auto-manages the nodegroup SG and admits only intra-cluster traffic.
+# The LBC-provisioned ALB lives in this VPC, so admit the VPC CIDR on the
+# NodePort. Look the SG up by its deterministic name and add one rule.
+# ---------------------------------------------------------------------------
+
+data "aws_security_group" "nodegroup" {
+  filter {
+    name   = "group-name"
+    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
+  }
+
+  filter {
+    name   = "vpc-id"
+    values = [aws_vpc.main.id]
+  }
+
+  depends_on = [aws_eks_node_group.workers]
+}
+
+resource "aws_vpc_security_group_ingress_rule" "nodeport_from_vpc" {
+  security_group_id = data.aws_security_group.nodegroup.id
+  cidr_ipv4         = aws_vpc.main.cidr_block
+  from_port         = var.node_port
+  to_port           = var.node_port
+  ip_protocol       = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-alb-nodeport"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Outputs
+# ---------------------------------------------------------------------------
+
+output "cluster_name" {
+  value = aws_eks_cluster.this.name
+}
+
+output "region" {
+  value = var.region
+}
+
+output "node_port" {
+  value = var.node_port
+}
+
+output "node_desired_size" {
+  value = var.node_desired_size
+}
+
+output "certificate_arn" {
+  value = aws_acm_certificate.ingress.arn
+}
+
+output "ecr_repository_url" {
+  value       = aws_ecr_repository.demo.repository_url
+  description = "Push the demo-app image here, then apply the workloads module"
+}
+
+output "ingress_address_hint" {
+  value = "After applying workloads: kubectl get ingress spinifex-demo -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{\"\\n\"}' — then open https://<that-address> (self-signed cert: curl -k)."
+}
+
+output "update_kubeconfig" {
+  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
+}
@@ -0,0 +1,258 @@
+# Demo workload for eks-https-ingress — separate root module / state.
+#
+# Kept apart from the cluster root module on purpose: the kubernetes provider
+# below reads the cluster endpoint from a *live* data source, so it is only ever
+# configured while the cluster exists. Destroy this module before the parent and
+# the provider never falls back to localhost.
+#
+# Creates the Spinifex-themed demo Deployment, a NodePort Service, and an Ingress
+# (ingressClassName: alb). The AWS Load Balancer Controller reconciles the
+# Ingress into an internet-facing ALB that terminates TLS with the imported ACM
+# certificate and forwards to the workers' NodePort.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-https-ingress
+#   tofu init && tofu apply                    # parent: cluster + LBC + ACM
+#   cd workloads && tofu init && tofu apply    # this module: demo app + Ingress
+#   # teardown is the reverse — destroy here first, then the parent.
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.20"
+    }
+  }
+}
+
+variable "spinifex_endpoint" {
+  type    = string
+  default = "https://127.0.0.1:9999"
+}
+
+variable "replicas" {
+  type    = number
+  default = 2
+}
+
+variable "demo_image" {
+  type        = string
+  default     = ""
+  description = "Demo image ref. Defaults to the parent's ECR repository URL at :latest."
+}
+
+variable "inbound_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the ALB HTTPS listener"
+}
+
+provider "aws" {
+  region = data.terraform_remote_state.infra.outputs.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+# Cluster identity comes from the parent module's state; the live endpoint/CA
+# come from a data source so the provider is only configured while the cluster
+# is up.
+data "terraform_remote_state" "infra" {
+  backend = "local"
+
+  config = {
+    path = "../terraform.tfstate"
+  }
+}
+
+locals {
+  cluster_name = data.terraform_remote_state.infra.outputs.cluster_name
+  region       = data.terraform_remote_state.infra.outputs.region
+  node_port    = data.terraform_remote_state.infra.outputs.node_port
+  cert_arn     = data.terraform_remote_state.infra.outputs.certificate_arn
+  demo_image   = var.demo_image != "" ? var.demo_image : "${data.terraform_remote_state.infra.outputs.ecr_repository_url}:latest"
+}
+
+data "aws_eks_cluster" "this" {
+  name = local.cluster_name
+}
+
+provider "kubernetes" {
+  host                   = data.aws_eks_cluster.this.endpoint
+  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
+
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    command     = "aws"
+    args        = ["eks", "get-token", "--cluster-name", local.cluster_name, "--region", local.region]
+  }
+}
+
+# The Spinifex-themed demo app reports the pod, node, cluster, and region that
+# served the request. The pod and node names come from the downward API.
+resource "kubernetes_deployment_v1" "demo" {
+  metadata {
+    name      = "spinifex-demo"
+    namespace = "default"
+    labels    = { app = "spinifex-demo" }
+  }
+
+  spec {
+    replicas = var.replicas
+
+    selector {
+      match_labels = { app = "spinifex-demo" }
+    }
+
+    template {
+      metadata {
+        labels = { app = "spinifex-demo" }
+      }
+
+      spec {
+        container {
+          name  = "spinifex-demo"
+          image = local.demo_image
+
+          port {
+            container_port = 8080
+          }
+
+          env {
+            name = "POD_NAME"
+            value_from {
+              field_ref {
+                field_path = "metadata.name"
+              }
+            }
+          }
+
+          env {
+            name = "NODE_NAME"
+            value_from {
+              field_ref {
+                field_path = "spec.nodeName"
+              }
+            }
+          }
+
+          env {
+            name = "POD_NAMESPACE"
+            value_from {
+              field_ref {
+                field_path = "metadata.namespace"
+              }
+            }
+          }
+
+          env {
+            name  = "CLUSTER_NAME"
+            value = local.cluster_name
+          }
+
+          env {
+            name  = "AWS_REGION"
+            value = local.region
+          }
+
+          env {
+            name  = "APP_TITLE"
+            value = "Spinifex EKS — HTTPS Ingress"
+          }
+
+          readiness_probe {
+            http_get {
+              path = "/healthz"
+              port = 8080
+            }
+            initial_delay_seconds = 3
+            period_seconds        = 10
+          }
+        }
+      }
+    }
+  }
+}
+
+# NodePort Service: the ALB target group registers the workers on this port.
+resource "kubernetes_service_v1" "demo" {
+  metadata {
+    name      = "spinifex-demo"
+    namespace = "default"
+  }
+
+  spec {
+    selector = { app = "spinifex-demo" }
+    type     = "NodePort"
+
+    port {
+      port        = 80
+      target_port = 8080
+      node_port   = local.node_port
+    }
+  }
+
+  depends_on = [kubernetes_deployment_v1.demo]
+}
+
+# Ingress reconciled by the AWS Load Balancer Controller into an internet-facing
+# ALB. target-type instance registers the workers' NodePort; the ACM cert ARN
+# attaches to the HTTPS:443 listener; HTTP is redirected to HTTPS. ELB subnets
+# are injected by Spinifex into the alb IngressClassParams, so none are set here.
+resource "kubernetes_ingress_v1" "demo" {
+  metadata {
+    name      = "spinifex-demo"
+    namespace = "default"
+
+    annotations = {
+      "alb.ingress.kubernetes.io/scheme"           = "internet-facing"
+      "alb.ingress.kubernetes.io/target-type"      = "instance"
+      "alb.ingress.kubernetes.io/listen-ports"     = "[{\"HTTP\":80},{\"HTTPS\":443}]"
+      "alb.ingress.kubernetes.io/certificate-arn"  = local.cert_arn
+      "alb.ingress.kubernetes.io/ssl-redirect"     = "443"
+      "alb.ingress.kubernetes.io/healthcheck-path" = "/healthz"
+      "alb.ingress.kubernetes.io/inbound-cidrs"    = var.inbound_cidr
+    }
+  }
+
+  spec {
+    ingress_class_name = "alb"
+
+    rule {
+      http {
+        path {
+          path      = "/"
+          path_type = "Prefix"
+
+          backend {
+            service {
+              name = kubernetes_service_v1.demo.metadata[0].name
+              port {
+                number = 80
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+}
+
+output "ingress_name" {
+  value = kubernetes_ingress_v1.demo.metadata[0].name
+}
@@ -0,0 +1,214 @@
+---
+title: "EKS Quickstart"
+description: "Stand up a minimal managed-Kubernetes cluster and a browsable Spinifex-themed demo app — VPC, IAM roles, an EKS cluster, a one- or three-worker node group, an ECR repository, and a load-balanced web page — using Terraform on Spinifex."
+category: "Terraform Workbooks"
+tags:
+  - terraform
+  - eks
+  - kubernetes
+  - iam
+  - vpc
+  - workbook
+resources:
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "Terraform Kubernetes Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "OpenTofu"
+    url: "https://opentofu.org/"
+---
+
+# Terraform: EKS Quickstart
+
+> The smallest end-to-end EKS example on Spinifex: a VPC, the two IAM roles EKS needs, a one- or three-worker cluster, an ECR repository, and a Spinifex-themed demo app you can open in a browser.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+Provision a managed Kubernetes cluster on Spinifex with Terraform/OpenTofu — and finish with something you can actually see. As well as the cluster, this workbook creates an **ECR repository**, then deploys the Spinifex-themed demo app onto the cluster and publishes it on the worker's public IP. When `apply` finishes, open the **`demo_url`** output: the page reports the **pod, node, cluster, and region** that answered. Refresh it and the answering pod changes, so even a non-technical viewer can watch Kubernetes scheduling and load-balancing in real time.
+
+Under the hood it keeps things minimal: a VPC with two public subnets, an IAM role for the control plane, an IAM role for the workers with the three managed policies AWS-managed node groups expect, a public-endpoint `aws_eks_cluster`, and a node group sized by **`node_desired_size`** (1 for a single-node demo, or 3 for an HA-shaped cluster). You build the demo image and push it to the ECR repository, then Terraform uses the **Kubernetes provider** (authenticating exactly like `kubectl` does, via `aws eks get-token`) to deploy the Deployment + NodePort Service.
+
+**What you'll learn:**
+
+- Configuring the AWS provider to target Spinifex's `eks`, `ec2`, `iam` and `sts` endpoints
+- Creating the EKS cluster and worker IAM roles with faithful managed-policy attachments
+- Provisioning an `aws_eks_cluster` and a managed `aws_eks_node_group`
+- Pointing the Kubernetes provider at the new cluster and deploying a workload from the same `apply`
+- Opening one rule on the auto-managed worker security group to expose a NodePort
+
+**What gets created**
+
+| Resource | Name | Purpose |
+|---|---|---|
+| VPC | `eks-quickstart-vpc` | Isolated network (10.30.0.0/16) |
+| Subnets | `eks-quickstart-subnet-a/-b` | Public subnets for the cluster and worker |
+| Internet Gateway | `eks-quickstart-igw` | Egress so the worker can pull the demo image |
+| IAM Roles | `eks-quickstart-cluster-role`, `eks-quickstart-node-role` | Control-plane and worker roles |
+| ECR Repository | `spinifex-demo` | Holds the demo image the workers pull |
+| EKS Cluster | `eks-quickstart` | Public API endpoint, API auth mode, Kubernetes 1.32 |
+| Node Group | `default` | `node_desired_size` `t3.medium` worker(s) — 1 or 3 |
+| SG Ingress Rule | `eks-quickstart-demo-nodeport` | Opens the demo NodePort on the auto-managed worker SG |
+| K8s Deployment | `spinifex-demo` | The themed demo image from ECR, 2 replicas |
+| K8s Service | `spinifex-demo` | NodePort publishing the demo on the worker |
+
+**Spinifex specifics**
+
+- The cluster's security groups are **auto-managed** — `vpc_config.security_group_ids` is ignored. To reach a NodePort, this workbook looks the worker SG up by its deterministic name (`eks-cluster-<name>-nodegroup-sg`) and adds a single ingress rule.
+- The worker AMI is always Spinifex's `eks-node` image; `ami_type` is recorded but does **not** select the image.
+- `authentication_mode` must be `"API"` (the `API_AND_CONFIG_MAP` mode is rejected).
+- The workers need outbound internet (here via the IGW) to pull the demo container image.
+
+**Prerequisites:**
+
+- Spinifex installed and running (see [Installing Spinifex](/docs/install))
+- The Spinifex `eks-node` image available on the cluster
+- OpenTofu or Terraform, plus `kubectl` and the AWS CLI
+- Docker, to build and push the demo image to ECR
+
+## Instructions
+
+### Step 1. Get the Template
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-quickstart
+```
+
+Or create a `main.tf` file and paste the full configuration below.
+
+<!-- INCLUDE: main.tf lang:hcl -->
+
+### Step 2. Deploy
+
+The cluster and the demo app are **two root modules with separate state**. Apply the cluster first:
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+```
+
+This creates the cluster (which bootstraps a control-plane VM and brings up k3s — a few minutes in `CREATING`), launches the worker(s), creates the `spinifex-demo` ECR repository, and opens the NodePort.
+
+Set the worker count with `node_desired_size` (`1` for a single node, `3` for an HA-shaped cluster):
+
+```bash
+tofu apply -var node_desired_size=3
+```
+
+Once the cluster is `ACTIVE`, **build and push the demo image** to the ECR repository this created (full commands in [`../demo-app/README.md`](../demo-app/README.md)):
+
+```bash
+cd ../demo-app
+REGISTRY=$(cd ../eks-quickstart && tofu output -raw ecr_repository_url)
+REGISTRY_HOST=${REGISTRY%%/*}
+aws ecr get-login-password | docker login --username AWS --password-stdin "$REGISTRY_HOST"
+docker build -t "${REGISTRY}:latest" .
+docker push "${REGISTRY}:latest"
+cd ../eks-quickstart
+```
+
+Then deploy the demo app from the nested `workloads/` module — it defaults `demo_image` to the parent's ECR repository at `:latest`:
+
+```bash
+cd workloads
+tofu init
+tofu apply
+```
+
+> **Why two modules?** The Kubernetes provider in `workloads/` reads the cluster endpoint from a live `data "aws_eks_cluster"` source, so it's only ever configured while the cluster exists. Keeping it out of the cluster module means `destroy` never tries to refresh a workload against a cluster that's already gone — the failure mode where the provider falls back to `http://localhost:80` and reports `connection refused`. Always destroy `workloads/` before the cluster.
+
+> **Same profile for the Kubernetes provider.** The Kubernetes provider authenticates by shelling out to `aws eks get-token`, which has to reach the Spinifex STS endpoint. Keep `AWS_PROFILE=spinifex` exported for both applies.
+
+<!-- INCLUDE: workloads/main.tf lang:hcl -->
+
+### Step 3. Open the Demo
+
+Run from the cluster module directory (`cd ..` if you're still in `workloads/`):
+
+```bash
+tofu output demo_url
+```
+
+Open that URL in a browser. You'll see the Spinifex-themed page reporting the **pod, node, cluster, and region** that handled the request. Refresh a few times — with two replicas, the pod name alternates, demonstrating that the Service is load-balancing across the cluster.
+
+### Step 4. Inspect with kubectl (optional)
+
+```bash
+aws eks update-kubeconfig --name eks-quickstart --region ap-southeast-2
+kubectl get nodes
+kubectl get pods -o wide
+```
+
+`kubectl get pods -o wide` shows the demo pods and which node each landed on.
+
+### Cleanup
+
+Destroy in reverse — the demo app first (while the cluster is still up), then the cluster:
+
+```bash
+cd workloads
+tofu destroy
+cd ..
+tofu destroy
+```
+
+## Troubleshooting
+
+### Demo URL Doesn't Load
+
+The page is served from a NodePort on the worker's public IP. Work through the chain:
+
+```bash
+# Is the worker running with a public IP?
+aws ec2 describe-instances --filters "Name=tag:spinifex:eks-cluster,Values=eks-quickstart" \
+  --query 'Reservations[].Instances[].[InstanceId,State.Name,PublicIpAddress]' --output text
+
+# Did the demo pods roll out?
+kubectl get pods -o wide
+
+# Is the NodePort rule on the worker SG?
+aws ec2 describe-security-groups --filters "Name=group-name,Values=eks-cluster-eks-quickstart-nodegroup-sg" \
+  --query 'SecurityGroups[0].IpPermissions'
+```
+
+If the pods are `Pending`, the worker may not be `Ready` yet — give it a moment. If they're `ImagePullBackOff`, the worker can't pull from ECR: confirm you built and pushed the image (`tofu output ecr_repository_url`), that the IGW route and the worker's public IP are in place, and that the node role carries `AmazonEC2ContainerRegistryReadOnly`.
+
+### kubernetes provider: connection refused / Unauthorized
+
+The provider runs `aws eks get-token` against the Spinifex STS endpoint. Confirm the AWS CLI is still pointed at Spinifex and the cluster is `ACTIVE`:
+
+```bash
+aws sts get-caller-identity
+aws eks describe-cluster --name eks-quickstart --query 'cluster.status'
+```
+
+If the cluster was still `CREATING` when the provider first tried to connect, just re-run `tofu apply`.
+
+### Cluster Stuck in CREATING
+
+Control-plane bootstrap takes a few minutes. Confirm the underlying VM came up:
+
+```bash
+aws eks describe-cluster --name eks-quickstart --query 'cluster.status'
+aws ec2 describe-instances --profile spinifex
+```
+
+### Provider Connection Refused
+
+```bash
+sudo systemctl status spinifex.target
+curl -k https://localhost:9999/
+```
@@ -0,0 +1,392 @@
+# Example: EKS Quickstart on Spinifex
+#
+# A minimal managed-Kubernetes cluster that ends in something you can see: a
+# VPC, the two IAM roles EKS needs, a one- or three-worker cluster, an ECR
+# repository, and the Spinifex-themed demo app deployed onto it by Terraform and
+# exposed on the worker's public IP.
+#
+# Once `apply` finishes, open the demo_url output in a browser. The page reports
+# the pod, node, cluster, and region that answered — refresh and it bounces
+# between the app's replicas, showing the cluster is really scheduling and
+# load-balancing pods.
+#
+# Demonstrates: VPC + subnets, EKS cluster + node IAM roles, an aws_eks_cluster
+# and managed aws_eks_node_group, an ECR repository the workers pull from, the
+# Kubernetes provider authenticating to the cluster, and a Deployment + NodePort
+# Service reachable from your browser.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-quickstart
+#   export AWS_PROFILE=spinifex
+#   tofu init && tofu apply
+#   # build + push the demo image to the ECR repo this creates (see README),
+#   # then: cd workloads && tofu init && tofu apply
+#   # finally open the demo_url output
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Variables
+# ---------------------------------------------------------------------------
+
+variable "region" {
+  type    = string
+  default = "ap-southeast-2"
+}
+
+variable "cluster_name" {
+  type    = string
+  default = "eks-quickstart"
+}
+
+variable "k8s_version" {
+  type        = string
+  default     = "1.32"
+  description = "Kubernetes minor version for the control plane and workers"
+}
+
+variable "instance_type" {
+  type    = string
+  default = "t3.medium"
+}
+
+variable "node_desired_size" {
+  type        = number
+  default     = 1
+  description = "Worker count. Use 1 for a single-node demo, or 3 for an HA-shaped cluster."
+
+  validation {
+    condition     = var.node_desired_size == 1 || var.node_desired_size == 3
+    error_message = "node_desired_size must be 1 or 3."
+  }
+}
+
+variable "node_port" {
+  type        = number
+  default     = 30080
+  description = "NodePort the demo Service is published on"
+}
+
+variable "browse_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the demo NodePort; tighten to your own IP in production"
+}
+
+variable "spinifex_endpoint" {
+  type        = string
+  default     = "https://127.0.0.1:9999"
+  description = "Spinifex AWS gateway endpoint"
+}
+
+# ---------------------------------------------------------------------------
+# Providers
+# ---------------------------------------------------------------------------
+
+provider "aws" {
+  region = var.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+    ecr = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+# ---------------------------------------------------------------------------
+# Data sources
+# ---------------------------------------------------------------------------
+
+data "aws_availability_zones" "available" {
+  state = "available"
+}
+
+# ---------------------------------------------------------------------------
+# VPC + two public subnets (workers get public IPs to pull the demo image)
+# ---------------------------------------------------------------------------
+
+resource "aws_vpc" "main" {
+  cidr_block           = "10.30.0.0/16"
+  enable_dns_hostnames = true
+  enable_dns_support   = true
+
+  tags = {
+    Name = "${var.cluster_name}-vpc"
+  }
+}
+
+resource "aws_internet_gateway" "igw" {
+  vpc_id = aws_vpc.main.id
+
+  tags = {
+    Name = "${var.cluster_name}-igw"
+  }
+}
+
+resource "aws_subnet" "a" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.30.1.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-subnet-a"
+  }
+}
+
+resource "aws_subnet" "b" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.30.2.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-subnet-b"
+  }
+}
+
+resource "aws_route_table" "public" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block = "0.0.0.0/0"
+    gateway_id = aws_internet_gateway.igw.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-public-rt"
+  }
+}
+
+resource "aws_route_table_association" "a" {
+  subnet_id      = aws_subnet.a.id
+  route_table_id = aws_route_table.public.id
+}
+
+resource "aws_route_table_association" "b" {
+  subnet_id      = aws_subnet.b.id
+  route_table_id = aws_route_table.public.id
+}
+
+# ---------------------------------------------------------------------------
+# IAM — EKS cluster role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "cluster" {
+  name = "${var.cluster_name}-cluster-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "eks.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "cluster" {
+  role       = aws_iam_role.cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — worker node role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "node" {
+  name = "${var.cluster_name}-node-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_worker" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_cni" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_ecr" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+}
+
+# ---------------------------------------------------------------------------
+# ECR — repository the workers pull the demo image from
+#
+# Build and push the demo-app image here before applying the workloads module
+# (see ../demo-app/README.md). The node role already carries
+# AmazonEC2ContainerRegistryReadOnly, so workers can pull from it.
+# ---------------------------------------------------------------------------
+
+resource "aws_ecr_repository" "demo" {
+  name = "spinifex-demo"
+
+  # tofu destroy must remove the repo even though it still holds the pushed demo
+  # image; without this, DeleteRepository (force=false) fails RepositoryNotEmpty.
+  force_delete = true
+}
+
+# ---------------------------------------------------------------------------
+# EKS cluster — public API endpoint, API (access-entry) auth mode
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_cluster" "this" {
+  name     = var.cluster_name
+  role_arn = aws_iam_role.cluster.arn
+  version  = var.k8s_version
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.a.id, aws_subnet.b.id]
+    endpoint_public_access  = true
+    endpoint_private_access = false
+  }
+
+  access_config {
+    authentication_mode                         = "API"
+    bootstrap_cluster_creator_admin_permissions = true
+  }
+
+  depends_on = [aws_iam_role_policy_attachment.cluster]
+
+  tags = {
+    Name = var.cluster_name
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Managed node group — one worker
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_node_group" "default" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "default"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.a.id, aws_subnet.b.id]
+
+  scaling_config {
+    desired_size = var.node_desired_size
+    min_size     = var.node_desired_size
+    max_size     = var.node_desired_size * 2
+  }
+
+  instance_types = [var.instance_type]
+  ami_type       = "AL2_x86_64"
+
+  depends_on = [
+    aws_iam_role_policy_attachment.node_worker,
+    aws_iam_role_policy_attachment.node_cni,
+    aws_iam_role_policy_attachment.node_ecr,
+  ]
+
+  tags = {
+    Name = "${var.cluster_name}-default"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Open the NodePort on the auto-managed nodegroup SG
+#
+# Spinifex creates the worker SG itself (vpc_config.security_group_ids is
+# ignored) and admits only intra-cluster traffic. To reach the demo NodePort
+# from a browser, look the SG up by its deterministic name and add one rule.
+# ---------------------------------------------------------------------------
+
+data "aws_security_group" "nodegroup" {
+  filter {
+    name   = "group-name"
+    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
+  }
+
+  filter {
+    name   = "vpc-id"
+    values = [aws_vpc.main.id]
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+resource "aws_vpc_security_group_ingress_rule" "nodeport" {
+  security_group_id = data.aws_security_group.nodegroup.id
+  cidr_ipv4         = var.browse_cidr
+  from_port         = var.node_port
+  to_port           = var.node_port
+  ip_protocol       = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-demo-nodeport"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Discover the worker's public IP for the demo URL
+# ---------------------------------------------------------------------------
+
+data "aws_instances" "workers" {
+  instance_tags = {
+    "spinifex:eks-cluster" = aws_eks_cluster.this.name
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+# ---------------------------------------------------------------------------
+# Outputs
+# ---------------------------------------------------------------------------
+
+output "cluster_name" {
+  value = aws_eks_cluster.this.name
+}
+
+output "region" {
+  value = var.region
+}
+
+output "node_port" {
+  value = var.node_port
+}
+
+output "node_desired_size" {
+  value = var.node_desired_size
+}
+
+output "ecr_repository_url" {
+  value       = aws_ecr_repository.demo.repository_url
+  description = "Push the demo-app image here, then apply the workloads module"
+}
+
+output "demo_url" {
+  value       = "http://${data.aws_instances.workers.public_ips[0]}:${var.node_port}"
+  description = "Open in a browser; refresh to see different pods answer"
+}
+
+output "update_kubeconfig" {
+  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
+}
@@ -0,0 +1,198 @@
+# Demo workload for eks-quickstart — separate root module / state.
+#
+# Kept apart from the cluster root module on purpose: the kubernetes provider
+# below reads the cluster endpoint from a *live* data source, so it is only ever
+# configured while the cluster exists. Destroy this module before the parent and
+# the provider never falls back to localhost.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-quickstart
+#   tofu init && tofu apply                    # parent: cluster + infra
+#   cd workloads && tofu init && tofu apply    # this module: demo app
+#   # teardown is the reverse — destroy here first, then the parent.
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.20"
+    }
+  }
+}
+
+variable "spinifex_endpoint" {
+  type        = string
+  default     = "https://127.0.0.1:9999"
+  description = "Spinifex AWS gateway endpoint"
+}
+
+variable "replicas" {
+  type        = number
+  default     = 2
+  description = "Demo app replicas; refresh the page to see requests land on different pods"
+}
+
+variable "demo_image" {
+  type        = string
+  default     = ""
+  description = "Demo image ref. Defaults to the parent's ECR repository URL at :latest."
+}
+
+provider "aws" {
+  region = data.terraform_remote_state.infra.outputs.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+# Cluster identity comes from the parent module's state; the live endpoint/CA
+# come from a data source so the provider is only configured while the cluster
+# is up.
+data "terraform_remote_state" "infra" {
+  backend = "local"
+
+  config = {
+    path = "../terraform.tfstate"
+  }
+}
+
+locals {
+  cluster_name = data.terraform_remote_state.infra.outputs.cluster_name
+  region       = data.terraform_remote_state.infra.outputs.region
+  node_port    = data.terraform_remote_state.infra.outputs.node_port
+  demo_image   = var.demo_image != "" ? var.demo_image : "${data.terraform_remote_state.infra.outputs.ecr_repository_url}:latest"
+}
+
+data "aws_eks_cluster" "this" {
+  name = local.cluster_name
+}
+
+# Authenticates with the same `aws eks get-token` exec flow the generated
+# kubeconfig uses, so the Kubernetes provider can deploy the demo app.
+provider "kubernetes" {
+  host                   = data.aws_eks_cluster.this.endpoint
+  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
+
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    command     = "aws"
+    args        = ["eks", "get-token", "--cluster-name", local.cluster_name, "--region", local.region]
+  }
+}
+
+# The Spinifex-themed demo app reports the pod, node, cluster, and region that
+# served the request. With multiple replicas, refreshing the demo_url alternates
+# between them. The pod and node names come from the downward API.
+resource "kubernetes_deployment_v1" "demo" {
+  metadata {
+    name      = "spinifex-demo"
+    namespace = "default"
+    labels    = { app = "spinifex-demo" }
+  }
+
+  spec {
+    replicas = var.replicas
+
+    selector {
+      match_labels = { app = "spinifex-demo" }
+    }
+
+    template {
+      metadata {
+        labels = { app = "spinifex-demo" }
+      }
+
+      spec {
+        container {
+          name  = "spinifex-demo"
+          image = local.demo_image
+
+          port {
+            container_port = 8080
+          }
+
+          env {
+            name = "POD_NAME"
+            value_from {
+              field_ref {
+                field_path = "metadata.name"
+              }
+            }
+          }
+
+          env {
+            name = "NODE_NAME"
+            value_from {
+              field_ref {
+                field_path = "spec.nodeName"
+              }
+            }
+          }
+
+          env {
+            name = "POD_NAMESPACE"
+            value_from {
+              field_ref {
+                field_path = "metadata.namespace"
+              }
+            }
+          }
+
+          env {
+            name  = "CLUSTER_NAME"
+            value = local.cluster_name
+          }
+
+          env {
+            name  = "AWS_REGION"
+            value = local.region
+          }
+
+          readiness_probe {
+            http_get {
+              path = "/healthz"
+              port = 8080
+            }
+            initial_delay_seconds = 3
+            period_seconds        = 10
+          }
+        }
+      }
+    }
+  }
+}
+
+resource "kubernetes_service_v1" "demo" {
+  metadata {
+    name      = "spinifex-demo"
+    namespace = "default"
+  }
+
+  spec {
+    selector = { app = "spinifex-demo" }
+    type     = "NodePort"
+
+    port {
+      port        = 80
+      target_port = 8080
+      node_port   = local.node_port
+    }
+  }
+
+  depends_on = [kubernetes_deployment_v1.demo]
+}
@@ -142,7 +142,6 @@ The ALB is internet-facing, but the DNS name Spinifex returns (`*.elb.spinifex.l
 ALB_IP=$(aws elbv2 describe-load-balancers --names nginx-alb \
  --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
  --output text)
-echo "ALB public IP: $ALB_IP"
 ```

 Then hit the ALB — successive requests should alternate between Server 1 and Server 2:
@@ -163,8 +162,6 @@ TG_ARN=$(aws elbv2 describe-target-groups \
  --query 'TargetGroups[0].TargetGroupArn' \
  --output text)

-echo "Target Group ARN: $TG_ARN"
-
 aws elbv2 describe-target-health --target-group-arn $TG_ARN
 ```

@@ -9,6 +9,9 @@ tags:
  - flask
  - webapp
  - workbook
+  - imds
+  - sts
+  - instance-profile
 resources:
  - title: "Terraform AWS Provider"
    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
@@ -36,18 +39,21 @@ resources:

 Deploy an EC2 instance running a Flask file-sharing web application backed by S3 (Predastore). Users can upload files through a web form and browse uploaded content — demonstrating Terraform managing both compute and object storage together.

+This workbook uses the idiomatic AWS credential model: the instance is launched with an **IAM instance profile** and the app pulls short-lived STS credentials from **IMDS** (`169.254.169.254`) through boto3's default credential chain. **No long-lived S3 keys are baked into the instance.** The Terraform run still uses the operator's admin credentials (to create the bucket, role, profile, and instance), but the running instance authenticates to S3 with credentials it fetches at runtime and which expire in ~1 hour.
+
 **Architecture:**

 <p align="center">
-  <img src="../../../.github/assets/diagrams/tf-s3-webapp.svg" alt="S3 webapp — browser to Flask EC2 instance to Predastore via S3 API" width="900">
+  <img src="../../../.github/assets/diagrams/tf-s3-webapp.svg" alt="S3 webapp — browser to Flask EC2 instance; the instance fetches short-lived STS credentials from IMDS (169.254.169.254) via its instance profile, then calls Predastore over the S3 API" width="900">
 </p>

 **What you'll learn:**

 - Configuring the AWS provider with both Spinifex and Predastore endpoints
 - Creating S3 buckets on Predastore via Terraform
- Deploying a Python webapp with cloud-init that talks to S3
- Passing credentials and configuration to instances via user-data
+- Defining an IAM role, least-privilege managed policy, and instance profile, and passing the role to an instance (`iam:PassRole`)
+- How the instance obtains short-lived credentials from IMDS via boto3's default credential chain — no static keys on the instance
+- Deploying a Python webapp with cloud-init that signs S3 requests with the STS session token

 **Prerequisites:**

@@ -55,6 +61,7 @@ Deploy an EC2 instance running a Flask file-sharing web application backed by S3
 - Predastore running (S3 API on port 8443)
 - An Ubuntu 26.04 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
 - OpenTofu or Terraform installed
+- The operator identity running the Terraform apply (`AWS_PROFILE=spinifex`) must be allowed to manage IAM and pass the role: `iam:CreateRole`, `iam:CreatePolicy`, `iam:AttachRolePolicy`, `iam:CreateInstanceProfile`, `iam:AddRoleToInstanceProfile`, and `iam:PassRole` on the new role. The bootstrap admin profile satisfies this.
 - The EC2 instance must be able to reach Predastore — use the host's br-wan IP, not localhost

 ## Instructions
@@ -141,16 +148,39 @@ sudo systemctl status s3-webapp
 sudo journalctl -u s3-webapp --no-pager -n 50
 ```

-### Upload Fails or Files Not Appearing
+### Upload Fails with `403 AccessDenied`

-Check that the S3 credentials in `/opt/webapp/.env` are correct and that the bucket exists:
+The instance reached S3 but was not authorized. The `.env` file deliberately holds **no** keys — credentials come from IMDS:

 ```bash
 ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
-cat /opt/webapp/.env
-/opt/webapp/venv/bin/python -c "import boto3; print('boto3 OK')"
+cat /opt/webapp/.env   # S3_ENDPOINT / S3_BUCKET / S3_REGION only — no keys
 ```

+Check that:
+
+- the managed policy is attached to the role (`s3:ListBucket` on the bucket, `s3:GetObject`/`s3:PutObject` on its objects), and the policy's bucket name matches `bucket_name`;
+- the assumed-role session's account matches the bucket owner's account — Predastore enforces bucket ownership. Both are created by the same operator identity, so they align by construction; a mismatch surfaces as `403`.
+
+### `NoCredentialsError` / No Credentials
+
+boto3 could not obtain credentials, which means the IMDS credential chain did not resolve. From the instance:
+
+```bash
+ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
+
+# IMDSv2: fetch a token, then the role name behind the instance profile
+TOKEN=$(curl -sX PUT http://169.254.169.254/latest/api/token \
+  -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')
+curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
+  http://169.254.169.254/latest/meta-data/iam/security-credentials/
+
+# Confirm the resolved identity — expect assumed-role/s3-webapp-role/<instance-id>
+aws sts get-caller-identity --endpoint-url https://<predastore_host>
+```
+
+If the role name is empty, the instance profile was not attached — confirm `iam_instance_profile` on the instance and that `iam:PassRole` is allowed for the operator.
+
 ### AMI Not Found

 Ensure you have imported an Ubuntu 26.04 image:
@@ -8,13 +8,25 @@
 # Architecture:
 #
 #   Browser ──HTTP──▶ EC2 Instance (Flask webapp, port 80)
-#                         │
-#                         ▼ S3 API (boto3)
+#                         │  ▲ IMDS: short-lived STS creds (ASIA + token)
+#                         │  └──── 169.254.169.254
+#                         ▼ S3 API (boto3, SigV4 with session token)
 #                     Predastore (port 8443)
 #
+# Credentials:
+#   The Terraform run uses the operator's admin keys (s3_access_key /
+#   s3_secret_key) to create the bucket, IAM role, instance profile, and
+#   instance. The INSTANCE receives no long-lived secret — boto3's default
+#   credential chain pulls short-lived STS credentials from IMDS
+#   (169.254.169.254), resolved via the attached instance profile -> role.
+#
 # Prerequisites:
 #   - Spinifex services running (gateway on port 9999)
 #   - Predastore running (S3 API on port 8443)
+#   - The operator identity (AWS_PROFILE=spinifex) must be allowed
+#     iam:CreateRole / iam:CreatePolicy / iam:AttachRolePolicy /
+#     iam:CreateInstanceProfile / iam:AddRoleToInstanceProfile and
+#     iam:PassRole on the new role.
 #   - The EC2 instance must be able to reach the Predastore endpoint.
 #     Set `predastore_host` to the IP reachable from inside the VPC
 #     (e.g. the host's br-wan IP, NOT localhost).
@@ -80,13 +92,13 @@ variable "predastore_host" {

 variable "s3_access_key" {
  type        = string
-  description = "S3 access key for Predastore"
+  description = "Operator S3 access key for the Terraform run (creates the bucket on Predastore); the instance uses IMDS, not this key"
 }

 variable "s3_secret_key" {
  type        = string
  sensitive   = true
-  description = "S3 secret key for Predastore"
+  description = "Operator S3 secret key for the Terraform run; the instance uses IMDS, not this key"
 }

 variable "bucket_name" {
@@ -154,6 +166,55 @@ resource "aws_s3_bucket" "uploads" {
  bucket = var.bucket_name
 }

+# ---------------------------------------------------------------------------
+# IAM Instance Role — least-privilege S3 access fetched at runtime via IMDS
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "webapp" {
+  name = "s3-webapp-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Principal = { Service = "ec2.amazonaws.com" }
+      Action    = "sts:AssumeRole"
+    }]
+  })
+}
+
+resource "aws_iam_policy" "webapp_s3" {
+  name = "s3-webapp-policy"
+
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Sid      = "ListBucket"
+        Effect   = "Allow"
+        Action   = ["s3:ListBucket"]
+        Resource = ["arn:aws:s3:::${var.bucket_name}"]
+      },
+      {
+        Sid      = "ObjectRW"
+        Effect   = "Allow"
+        Action   = ["s3:GetObject", "s3:PutObject"]
+        Resource = ["arn:aws:s3:::${var.bucket_name}/*"]
+      }
+    ]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "webapp" {
+  role       = aws_iam_role.webapp.name
+  policy_arn = aws_iam_policy.webapp_s3.arn
+}
+
+resource "aws_iam_instance_profile" "webapp" {
+  name = "s3-webapp-profile"
+  role = aws_iam_role.webapp.name
+}
+
 # ---------------------------------------------------------------------------
 # SSH Key Pair
 # ---------------------------------------------------------------------------
@@ -273,9 +334,14 @@ resource "aws_instance" "webapp" {
  subnet_id              = aws_subnet.public.id
  vpc_security_group_ids = [aws_security_group.webapp.id]
  key_name               = aws_key_pair.webapp.key_name
+  iam_instance_profile   = aws_iam_instance_profile.webapp.name

  associate_public_ip_address = true

+  # The role's permissions must exist before the instance boots and makes its
+  # first IMDS-credentialed S3 call
+  depends_on = [aws_iam_role_policy_attachment.webapp]
+
  user_data_base64 = base64encode(<<-USERDATA
    #!/bin/bash
    set -euo pipefail
@@ -293,8 +359,6 @@ resource "aws_instance" "webapp" {
    cat > /opt/webapp/.env <<'ENVFILE'
    S3_ENDPOINT=https://${var.predastore_host}
    S3_BUCKET=${var.bucket_name}
-    S3_ACCESS_KEY=${var.s3_access_key}
-    S3_SECRET_KEY=${var.s3_secret_key}
    S3_REGION=${var.region}
    ENVFILE

@@ -321,8 +385,6 @@ resource "aws_instance" "webapp" {
    s3 = boto3.client(
        "s3",
        endpoint_url=env["S3_ENDPOINT"],
-        aws_access_key_id=env["S3_ACCESS_KEY"],
-        aws_secret_access_key=env["S3_SECRET_KEY"],
        region_name=env["S3_REGION"],
        verify=False,
        config=Config(s3={"addressing_style": "path"}),
@@ -2,6 +2,11 @@
 #
 # The predastore_host must be reachable from INSIDE the VPC guest — not
 # localhost. Use the host's br-wan or LAN IP, e.g. "192.168.1.10:8443".
+#
+# s3_access_key / s3_secret_key are the operator credentials for the Terraform
+# run only — they create the bucket on Predastore and the IAM role/profile.
+# They are NOT passed to the instance; the instance pulls short-lived STS
+# credentials from IMDS via its instance profile.

 predastore_host = "192.168.1.10:8443"
 s3_access_key   = "AKIAIOSFODNN7EXAMPLE"
@@ -13,21 +13,21 @@ require (
 	github.com/insomniacslk/dhcp v0.0.0-20260407060928-11b94ed970f2
 	github.com/kdomanski/iso9660 v0.4.0
 	github.com/klauspost/cpuid/v2 v2.3.0
-	github.com/mulgadc/predastore v1.6.0
-	github.com/mulgadc/viperblock v1.6.0
-	github.com/nats-io/nats-server/v2 v2.14.1
+	github.com/mulgadc/predastore v1.9.0
+	github.com/mulgadc/viperblock v1.9.0
+	github.com/nats-io/nats-server/v2 v2.14.2
 	github.com/nats-io/nats.go v1.52.0
 	github.com/ovn-kubernetes/libovsdb v0.8.1
-	github.com/pelletier/go-toml/v2 v2.3.1
+	github.com/pelletier/go-toml/v2 v2.4.0
 	github.com/pterm/pterm v0.12.83
 	github.com/spf13/cobra v1.10.2
 	github.com/spf13/viper v1.21.0
 	github.com/stretchr/testify v1.11.1
 	go.uber.org/goleak v1.3.0
-	golang.org/x/crypto v0.52.0
-	golang.org/x/sync v0.20.0
-	golang.org/x/sys v0.45.0
-	gopkg.in/ini.v1 v1.67.2
+	golang.org/x/crypto v0.53.0
+	golang.org/x/sync v0.21.0
+	golang.org/x/sys v0.46.0
+	gopkg.in/ini.v1 v1.67.3
 	gopkg.in/yaml.v3 v3.0.1
 )

@@ -38,8 +38,8 @@ require (
 	github.com/antithesishq/antithesis-sdk-go v0.7.0 // indirect
 	github.com/armon/go-metrics v0.4.1 // indirect
 	github.com/atotto/clipboard v0.1.4 // indirect
-	github.com/aws/aws-sdk-go-v2 v1.41.9 // indirect
-	github.com/aws/smithy-go v1.26.0 // indirect
+	github.com/aws/aws-sdk-go-v2 v1.42.0 // indirect
+	github.com/aws/smithy-go v1.27.1 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/boltdb/bolt v1.3.1 // indirect
@@ -57,7 +57,7 @@ require (
 	github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
 	github.com/containerd/console v1.0.5 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/dgraph-io/badger/v4 v4.9.1 // indirect
+	github.com/dgraph-io/badger/v4 v4.9.2 // indirect
 	github.com/dgraph-io/ristretto/v2 v2.4.0 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
@@ -85,7 +85,7 @@ require (
 	github.com/jmespath/go-jmespath v0.4.0 // indirect
 	github.com/josharian/native v1.1.0 // indirect
 	github.com/klauspost/compress v1.18.6 // indirect
-	github.com/klauspost/reedsolomon v1.14.0 // indirect
+	github.com/klauspost/reedsolomon v1.14.1 // indirect
 	github.com/leodido/go-urn v1.4.0 // indirect
 	github.com/lithammer/fuzzysearch v1.1.8 // indirect
 	github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
@@ -101,8 +101,8 @@ require (
 	github.com/muesli/cancelreader v0.2.2 // indirect
 	github.com/muesli/termenv v0.16.0 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
-	github.com/nats-io/jwt/v2 v2.8.1 // indirect
-	github.com/nats-io/nkeys v0.4.15 // indirect
+	github.com/nats-io/jwt/v2 v2.8.2 // indirect
+	github.com/nats-io/nkeys v0.4.16 // indirect
 	github.com/nats-io/nuid v1.0.1 // indirect
 	github.com/pierrec/lz4/v4 v4.1.14 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
@@ -110,7 +110,7 @@ require (
 	github.com/prometheus/client_model v0.6.2 // indirect
 	github.com/prometheus/common v0.67.5 // indirect
 	github.com/prometheus/procfs v0.20.1 // indirect
-	github.com/quic-go/quic-go v0.59.1 // indirect
+	github.com/quic-go/quic-go v0.60.0 // indirect
 	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/sagikazarmark/locafero v0.12.0 // indirect
 	github.com/spf13/afero v1.15.0 // indirect
@@ -126,13 +126,13 @@ require (
 	go.opentelemetry.io/otel/trace v1.42.0 // indirect
 	go.yaml.in/yaml/v2 v2.4.4 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
-	golang.org/x/mod v0.35.0 // indirect
-	golang.org/x/net v0.55.0 // indirect
-	golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa // indirect
-	golang.org/x/term v0.43.0 // indirect
-	golang.org/x/text v0.37.0 // indirect
+	golang.org/x/mod v0.36.0 // indirect
+	golang.org/x/net v0.56.0 // indirect
+	golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6 // indirect
+	golang.org/x/term v0.44.0 // indirect
+	golang.org/x/text v0.38.0 // indirect
 	golang.org/x/time v0.15.0 // indirect
-	golang.org/x/tools v0.44.0 // indirect
+	golang.org/x/tools v0.45.0 // indirect
 	golang.org/x/vuln v1.1.4 // indirect
 	google.golang.org/protobuf v1.36.11 // indirect
 )
@@ -31,30 +31,30 @@ github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z
 github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
 github.com/aws/aws-sdk-go v1.55.8 h1:JRmEUbU52aJQZ2AjX4q4Wu7t4uZjOu71uyNmaWlUkJQ=
 github.com/aws/aws-sdk-go v1.55.8/go.mod h1:ZkViS9AqA6otK+JBBNH2++sx1sgxrPKcSzPPvQkUtXk=
-github.com/aws/aws-sdk-go-v2 v1.41.9 h1:/rYeyO2+HrMztAmxAq9++XJtFMqSIpSsNA0yDGALYq4=
-github.com/aws/aws-sdk-go-v2 v1.41.9/go.mod h1:+HsoOEX80qAVUitj1A2DhCNTjmb3edVyuDypb6LNEeo=
-github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11 h1:h5+3VT69KUBK24grGuuA5saDJTj2IIjLb9au668Fo5I=
-github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11/go.mod h1:dnakxebH6UwFvcvujL0LVggYQ8nEvBGjU4G/V79Nv94=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.19 h1:yuFzSV1U0aRNYCQGVaTY2zW2M/L93pYHnXnrJUphYhU=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.19/go.mod h1:7y63L1kGzeoDlJaQ3Z578KrnmfBut96JjvJUzGwR+YE=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.25 h1:Uii3frf9ztec/ABM2/FSH9/z7PLzxfpG8h4RpkUFflQ=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.25/go.mod h1:G6kntsA2GorAxDPbap6xgB2F+amSLUF8GJTi7PUoX44=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.25 h1:r1+/l6m+WaUJF9HISEsNOLHSNj5EXYQxK8VX6Cz9NlA=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.25/go.mod h1:cKf+D+NMDK1LndD7BowHbBZPgR9V0/5HubH0PFWvA+c=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.26 h1:A1PmWU2zfkIm9EyFlJncFXL4W4phML+h8KjltUsCvNQ=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.26/go.mod h1:dY4MRzXEizrD4hqtpKvWVGPX7QleSGGVY+EBolo1RmM=
-github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.10 h1:d5/908OJ4bXg8lyjeMPvXetEKqoDoLi5Owy1zNue3yg=
-github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.10/go.mod h1:a57l7Hwh+FWI+we50g5NPJHYUKeJKfXbc4w8SyXu8Ig=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.18 h1:W/EyPFl9A5rXrtoilfwHYEvzHER+K4SpBPtMXi24Mos=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.18/go.mod h1:UG50K+pvd/uy6xExbobg0rjqFBFZe6I3l75EPDZw4tg=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.25 h1:dD3dhHNglpd98gs72my22Ndqi1hqQGllFFg1F+twfxg=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.25/go.mod h1:0yAbjPfd64gG7mj85RW+fMEYdfBgCRZw8g/oWcL1pjc=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.25 h1:2pQEbwf+/6EDbiit/GcBE2K4IUpMZymaA0kOz3xK978=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.25/go.mod h1:KvT6NCcQ0EZ+ZkVRrlBMt04Po3ok23YELEp7WimhLhM=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.102.2 h1:ie4ElCmUKS26pzrZcIk/lmt4yWjAqLLcawstyQCh298=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.102.2/go.mod h1:zjsomFeX5duj+4PlMB+o4JoWTIx+G0XMyzjYrUbQkN0=
-github.com/aws/smithy-go v1.26.0 h1:9ouqbi+NyKP7fV3Te7UElCwdAb6Y8uk7LGwPE5tVe/s=
-github.com/aws/smithy-go v1.26.0/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
+github.com/aws/aws-sdk-go-v2 v1.42.0 h1:XvXMJTkFQtpBKIWZnmr9ZEOc2InWM2yldjXEJ/bymhA=
+github.com/aws/aws-sdk-go-v2 v1.42.0/go.mod h1:27+ACypSLljLAEKsCYOmrjKh83vuTRkuAe9Uv/3A4bg=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.13 h1:p1BBrg/Hhp6uK7zpejeI8QFXHJeC/mynzi04Sl03k9g=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.13/go.mod h1:8cIfkE9MDhkRZGpQ22aV6/lkYeYSozpz16Smrs5x4Ls=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.24 h1:2hQqYCV9yqyePQ9o6dCrZc/zO8U3TwPr9mIKlZnPu/I=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.24/go.mod h1:IDwpACtwqHLISdzfwUUNq4P9DsB/h5BLg4FwJPNfqFY=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.29 h1:f3vKqSo13fhTYb+JEcXwXefZQE26I1FB5eTSniU67ko=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.29/go.mod h1:MzoLFUArKGpGD+ukmPiTPG1X5x4o6M2kq4v2dr1FiEc=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.29 h1:RdwIf/CuUsvJX3RgJagbOyotl/cxoLY4xviKuE7p2GY=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.29/go.mod h1:71wt8W2EgswdZy9Mf9KNnzxZ3TiZlv4caKghPktDOkA=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.30 h1:VTGy885W5DKBxWRUJbym9hytNaYzsyaPkCHGRRMAOhU=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.30/go.mod h1:AS0HycUvJRFvTt613AYDOgO2jzw+00cVSMny8XB3yMY=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.12 h1:ZD2+BSw9vFsNlKYIasSNt3uDbjqqXIBcM13UJv/Lx2k=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.12/go.mod h1:Ms4zlcVBbXbiP7EVLhl+lgjvA/a7YphqQ3Ih3174EmI=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.22 h1:V51LGlOq/1VsDsHUdoklAQi7rMmx4qQubvFYAlP2254=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.22/go.mod h1:4Pzhyz8hJOm2bepgl+NjvRx8vlUFAIIvJnZ/MkcNPpU=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.29 h1:DRebniUGZ2MqiiIVmQJ04vIXr918hubdHMnarSLEWyU=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.29/go.mod h1:LfRkPCD8YHDM2E5eTkos2UpwYeZnBcVarTa8L59bJHA=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.29 h1:hiME6pBzC7OTl9LMtlyTWBuEl1f4QBcUmFDKC7MLXtc=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.29/go.mod h1:G7RP+uhagpKtKhd1BM9N6JQqjCcGEU47K5lBVZQyRQw=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.104.0 h1:ta8csKy5vN91F3i5gGR85lFV0srBqySEji7Jroes6rE=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.104.0/go.mod h1:77ZAgynvx1txMvDG8gGWoWkO1augYDxkp9JElWFgjQU=
+github.com/aws/smithy-go v1.27.1 h1:4T340VFndXtADGF52gYa1POyL7s9E4Z1OeZ1hCscIw8=
+github.com/aws/smithy-go v1.27.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
@@ -103,8 +103,8 @@ github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6N
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/dgraph-io/badger/v4 v4.9.1 h1:DocZXZkg5JJHJPtUErA0ibyHxOVUDVoXLSCV6t8NC8w=
-github.com/dgraph-io/badger/v4 v4.9.1/go.mod h1:5/MEx97uzdPUHR4KtkNt8asfI2T4JiEiQlV7kWUo8c0=
+github.com/dgraph-io/badger/v4 v4.9.2 h1:Wb5qw8gElqwV1a8msHTeQKova9b1V10heFKMIiPd80E=
+github.com/dgraph-io/badger/v4 v4.9.2/go.mod h1:nJjaJTUOSsQEBhsq209FmwCvMJzEA3e74RjZw6V2pQI=
 github.com/dgraph-io/ristretto/v2 v2.4.0 h1:I/w09yLjhdcVD2QV192UJcq8dPBaAJb9pOuMyNy0XlU=
 github.com/dgraph-io/ristretto/v2 v2.4.0/go.mod h1:0KsrXtXvnv0EqnzyowllbVJB8yBonswa2lTCK2gGo9E=
 github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da h1:aIftn67I1fkbMa512G+w+Pxci9hJPB8oMnkcP3iZF38=
@@ -238,8 +238,8 @@ github.com/klauspost/cpuid/v2 v2.0.10/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuOb
 github.com/klauspost/cpuid/v2 v2.0.12/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
 github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
 github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
-github.com/klauspost/reedsolomon v1.14.0 h1:5YSZeclzSYg5nl349+GDG/agDtQ6MZiwUYXvVKN1Jx0=
-github.com/klauspost/reedsolomon v1.14.0/go.mod h1:yjqqjgMTQkBUHSG97/rm4zipffCNbCiZcB3kTqr++sQ=
+github.com/klauspost/reedsolomon v1.14.1 h1:swE9kzyWXD/wVG+l5Pe8bWnQ0giIY7D1GjCBKk3kG2U=
+github.com/klauspost/reedsolomon v1.14.1/go.mod h1:yjqqjgMTQkBUHSG97/rm4zipffCNbCiZcB3kTqr++sQ=
 github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
@@ -288,30 +288,30 @@ github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELU
 github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
 github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
 github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
-github.com/mulgadc/predastore v1.6.0 h1:a0xJtrmAvmLlc19B5xUR9upbkl7VlDntpAKnxbkM07E=
-github.com/mulgadc/predastore v1.6.0/go.mod h1:RJh0V0TG27s0q6n0o4CIQFu7CjjPYv4yGnKpIiP9tLs=
-github.com/mulgadc/viperblock v1.6.0 h1:mFx0eVOJoT3WL+X/GPrZOI0ySqerpiCtUW4vPpUxbtI=
-github.com/mulgadc/viperblock v1.6.0/go.mod h1:cwE2FVrvxUD+QaMWnAZeKVnxF8VtTG9zxiUaP1m5oL0=
+github.com/mulgadc/predastore v1.9.0 h1:nXwyblTso9ICbDkzSVx7j2j1FpCgPe/4c1du5Ge0k18=
+github.com/mulgadc/predastore v1.9.0/go.mod h1:FBFtu96XZlslhIJ2+Tv9NdR1S3Ks5XrWvJ+Q59iBxVA=
+github.com/mulgadc/viperblock v1.9.0 h1:rogwhMefiHIo7T8ApDYMQEs6UNGFNHtH8Bj2W7RzXU8=
+github.com/mulgadc/viperblock v1.9.0/go.mod h1:tq3msiqnmURyqEFd0oeMrVv3hpIgefXLurEn9exz+rI=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
-github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
-github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
-github.com/nats-io/nats-server/v2 v2.14.1 h1:wXs/a5fw9Hzm3CvuzLxGeIwpjPulSa7gMT3eSuhGkcg=
-github.com/nats-io/nats-server/v2 v2.14.1/go.mod h1:4N17zLpuS7WMbG8T9gsE2B7z9hC9PraPyulVBfpK6nU=
+github.com/nats-io/jwt/v2 v2.8.2 h1:XXRgB60MSTnqsRwejQurVDs/hcv2dkt+86GjI+I/bMc=
+github.com/nats-io/jwt/v2 v2.8.2/go.mod h1:Ag/56sq9OblL4JgdYufDd16Egb17Kr/8WwwuO/forVc=
+github.com/nats-io/nats-server/v2 v2.14.2 h1:Q7dRhCY03Y00rETFW3KV+KGaCIajlDfWgWUVgbMxyuk=
+github.com/nats-io/nats-server/v2 v2.14.2/go.mod h1:lWpb1bSpRELZfRdlMkdz8E7lbXKKyNe8RIn0vvepIHs=
 github.com/nats-io/nats.go v1.52.0 h1:n3avV4VBsCgsdwh71TppsTwtv+QdPs7ntSKM8qJLGsc=
 github.com/nats-io/nats.go v1.52.0/go.mod h1:26HypzazeOkyO3/mqd1zZd53STJN0EjCYF9Uy2ZOBno=
-github.com/nats-io/nkeys v0.4.15 h1:JACV5jRVO9V856KOapQ7x+EY8Jo3qw1vJt/9Jpwzkk4=
-github.com/nats-io/nkeys v0.4.15/go.mod h1:CpMchTXC9fxA5zrMo4KpySxNjiDVvr8ANOSZdiNfUrs=
+github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
+github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
 github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
 github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
 github.com/ovn-kubernetes/libovsdb v0.8.1 h1:M2J8bcJt5mXCom0HqzfEtuHkT80CTSQRcYG7acT8gf4=
 github.com/ovn-kubernetes/libovsdb v0.8.1/go.mod h1:ZlnHLzagmLOSvyd9qfxBIZp6wOSOw0IsRsc+6lNUGbU=
 github.com/pascaldekloe/goe v0.1.0 h1:cBOtyMzM9HTpWjXfbbunk26uA6nG3a8n06Wieeh0MwY=
 github.com/pascaldekloe/goe v0.1.0/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
-github.com/pelletier/go-toml/v2 v2.3.1 h1:MYEvvGnQjeNkRF1qUuGolNtNExTDwct51yp7olPtrEc=
-github.com/pelletier/go-toml/v2 v2.3.1/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
+github.com/pelletier/go-toml/v2 v2.4.0 h1:Mwu0mAkUKbittDs3/ADDWXqMmq3EOK2VHiuCkV00Row=
+github.com/pelletier/go-toml/v2 v2.4.0/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
 github.com/pierrec/lz4/v4 v4.1.14 h1:+fL8AQEZtz/ijeNnpduH0bROTu0O3NZAlPjQxGn8LwE=
 github.com/pierrec/lz4/v4 v4.1.14/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
@@ -353,8 +353,10 @@ github.com/pterm/pterm v0.12.36/go.mod h1:NjiL09hFhT/vWjQHSj1athJpx6H8cjpHXNAK5b
 github.com/pterm/pterm v0.12.40/go.mod h1:ffwPLwlbXxP+rxT0GsgDTzS3y3rmpAO1NMjUkGTYf8s=
 github.com/pterm/pterm v0.12.83 h1:ie+YmGmA727VuhxBlyGr74Ks+7McV6kT99IB8EU80aA=
 github.com/pterm/pterm v0.12.83/go.mod h1:xlgc6bFWyJIMtmLJvGim+L7jhSReilOlOnodeIYe4Tk=
-github.com/quic-go/quic-go v0.59.1 h1:0Gmua0HW1Tv7ANR7hUYwRyD0MG5OJfgvYSZasGZzBic=
-github.com/quic-go/quic-go v0.59.1/go.mod h1:upnsH4Ju1YkqpLXC305eW3yDZ4NfnNbmQRCMWS58IKU=
+github.com/quic-go/go-ossfuzz-seeds v0.1.0 h1:APacT+iIaNF6fd8AGEiN3bT/Jtkd2jz4v4TzM7MFjy0=
+github.com/quic-go/go-ossfuzz-seeds v0.1.0/go.mod h1:3IOHRbJIc+L6YKMwfDtJAM9Vj9k0YY4muhuyUYk5tbk=
+github.com/quic-go/quic-go v0.60.0 h1:xcQioE8OM66UQLeUMHltK1CCcOu3JbVB4JAQdDQSB+0=
+github.com/quic-go/quic-go v0.60.0/go.mod h1:wpKpjmPpftl30sL6pFh7REVpjbcCVy4zt2vDyK1TuJk=
 github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
 github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
 github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
@@ -426,14 +428,14 @@ golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnf
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
-golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
-golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
+golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
+golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa h1:FRnLl4eNAQl8hwxVVC17teOw8kdjVDVAiFMtgUdTSRQ=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa/go.mod h1:zk2irFbV9DP96SEBUUAy67IdHUaZuSnrz1n472HUCLE=
 golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
 golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
-golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM=
-golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU=
+golang.org/x/mod v0.36.0 h1:JJjpVx6myfUsUdAzZuOSTTmRE0PfZeNWzzvKrP7amb4=
+golang.org/x/mod v0.36.0/go.mod h1:moc6ELqsWcOw5Ef3xVprK5ul/MvtVvkIXLziUOICjUQ=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -444,8 +446,8 @@ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81R
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
-golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
-golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
+golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
+golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -454,8 +456,8 @@ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
-golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
+golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
+golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -486,33 +488,33 @@ golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
-golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
-golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
-golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa h1:efT73AJZfAAUV7SOip6pWGkwJDzIGiKBZGVzHYa+ve4=
-golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa/go.mod h1:kHjTxDEnAu6/Nl9lDkzjWpR+bmKfxeiRuSDlsMb70gE=
+golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
+golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6 h1:HjU6IWBiAgRIdAJ9/y1rwCn+UELEmwV+VsTLzj/W4sE=
+golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6/go.mod h1:Eqhaxk/wZsWEH8CRxLwj6xzEJbz7k1EFGqx7nyCoabE=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
-golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4=
-golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=
+golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc=
+golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
-golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
-golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
+golang.org/x/text v0.38.0 h1:sXmwo9DwP3OK9EZ7PqAdaooSGozfl/3a6/xJcbzPRhE=
+golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4=
 golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
 golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
 golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
-golang.org/x/tools v0.44.0 h1:UP4ajHPIcuMjT1GqzDWRlalUEoY+uzoZKnhOjbIPD2c=
-golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI=
+golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8=
+golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0=
 golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM=
 golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY=
 golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM=
@@ -536,8 +538,8 @@ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
-gopkg.in/ini.v1 v1.67.2 h1:JtOSMb9OuaCZKr7h5D/h6iii14sK0hLbplTc6frx4Ss=
-gopkg.in/ini.v1 v1.67.2/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
+gopkg.in/ini.v1 v1.67.3 h1:iM9Lhz5MRSGhHVGGwCuzG9KO8PoirCXj/m/qTmOJJQw=
+gopkg.in/ini.v1 v1.67.3/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
@@ -0,0 +1,112 @@
+// Package eksgw is the on-VM SigV4 client used by EKS helpers (eks-gateway-publish,
+// eks-token-webhook) to POST over HTTPS to the AWS gateway instead of NATS.
+// Signs with Predastore credentials for service "eks".
+package eksgw
+
+import (
+	"bytes"
+	"crypto/sha256"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/mulgadc/predastore/auth"
+	"github.com/mulgadc/spinifex/internal/tlsconfig"
+)
+
+// Client posts SigV4-signed requests to the AWS gateway for the "eks" service.
+type Client struct {
+	baseURL    string
+	accessKey  string
+	secretKey  string
+	region     string
+	httpClient *http.Client
+}
+
+// New builds a client. caPath optionally pins the gateway TLS CA; when empty the
+// client relies on the system trust store. region defaults to us-east-1 when
+// empty (SigV4 requires a non-empty region).
+func New(baseURL, caPath, accessKey, secretKey, region string) (*Client, error) {
+	if baseURL == "" {
+		return nil, fmt.Errorf("eksgw: baseURL is required")
+	}
+	if accessKey == "" || secretKey == "" {
+		return nil, fmt.Errorf("eksgw: access key and secret key are required")
+	}
+	if region == "" {
+		region = "us-east-1"
+	}
+
+	tlsCfg := &tls.Config{
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
+	}
+	if caPath != "" {
+		pem, err := os.ReadFile(caPath)
+		if err != nil {
+			return nil, fmt.Errorf("eksgw: read gateway CA %q: %w", caPath, err)
+		}
+		pool := x509.NewCertPool()
+		if !pool.AppendCertsFromPEM(pem) {
+			return nil, fmt.Errorf("eksgw: gateway CA %q has no usable certificates", caPath)
+		}
+		tlsCfg.RootCAs = pool
+	}
+
+	return &Client{
+		baseURL:   strings.TrimRight(baseURL, "/"),
+		accessKey: accessKey,
+		secretKey: secretKey,
+		region:    region,
+		httpClient: &http.Client{
+			Timeout:   10 * time.Second,
+			Transport: &http.Transport{TLSClientConfig: tlsCfg, MaxIdleConns: 2, IdleConnTimeout: 30 * time.Second},
+		},
+	}, nil
+}
+
+// Post SigV4-signs and sends body to path, returning the response body on 2xx.
+// Errors include the gateway status and body. No retry; callers wrap as needed.
+func (c *Client) Post(path string, body []byte) ([]byte, error) {
+	return c.do(http.MethodPost, path, body)
+}
+
+// Get SigV4-signs and sends a single GET to path (e.g.
+// "/clusters/alpha/internal-addons?accountId=..."). Same contract as Post: 2xx
+// body or an error carrying the gateway status. No retry.
+func (c *Client) Get(path string) ([]byte, error) {
+	return c.do(http.MethodGet, path, nil)
+}
+
+func (c *Client) do(method, path string, body []byte) ([]byte, error) {
+	req, err := http.NewRequest(method, c.baseURL+path, bytes.NewReader(body))
+	if err != nil {
+		return nil, fmt.Errorf("create request: %w", err)
+	}
+	if body != nil {
+		req.Header.Set("Content-Type", "application/x-amz-json-1.1")
+	}
+
+	sum := sha256.Sum256(body)
+	if err := auth.SignReq(req, c.accessKey, c.secretKey, hex.EncodeToString(sum[:]), "eks", c.region); err != nil {
+		return nil, fmt.Errorf("sign request: %w", err)
+	}
+
+	resp, err := c.httpClient.Do(req)
+	if err != nil {
+		return nil, fmt.Errorf("send request: %w", err)
+	}
+	defer resp.Body.Close()
+
+	respBody, _ := io.ReadAll(resp.Body)
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		return nil, fmt.Errorf("gateway returned %d: %s", resp.StatusCode, strings.TrimSpace(string(respBody)))
+	}
+	return respBody, nil
+}
@@ -1,12 +1,6 @@
-// Package fipsboot enforces FIPS 140-3 mode at process startup.
-//
-// Blank-import this package from every binary's main:
-//
-//	import _ "github.com/mulgadc/spinifex/internal/fipsboot"
-//
-// Builds with GOFIPS140=v1.0.0 link the Go Cryptographic Module v1.0.0
-// (NIST CMVP cert #5247). At runtime FIPS mode can still be turned off
-// via GODEBUG=fips140=off; init() panics in that case to close the hole.
+// Package fipsboot enforces FIPS 140-3 at startup via blank import in every binary.
+// Requires GOFIPS140=v1.0.0 at build time; panics if GODEBUG=fips140=off disables
+// the runtime FIPS mode.
 package fipsboot

 import (
@@ -1,34 +1,13 @@
 // Package tlsconfig pins the cluster-wide TLS 1.3 curve preference list.
-//
-// The list is hardcoded because mixed classification on a single cluster is
-// not supported and the certified FIPS build is itself the per-deployment
-// choice. A typo in a config string would silently downgrade where a missing
-// constant cannot.
+// Hardcoded to prevent silent downgrade via misconfiguration; the FIPS build
+// is the per-deployment choice.
 package tlsconfig

 import "crypto/tls"

-// Curves is the fixed TLS 1.3 curve allowlist:
-//
-//   - X25519MLKEM768     — Cat 3 ML-KEM hybrid, wins under Go 1.26 stdlib filter
-//   - SecP384r1MLKEM1024 — Cat 5 ML-KEM hybrid, fallback for peers that prefer it
-//   - X25519             — classical TLS 1.3 fallback for SDKs without ML-KEM (aws CLI v2 today)
-//   - CurveP384          — classical TLS 1.3 fallback (Java/.NET enterprise SDKs)
-//
-// PQ-capable peers (browsers since 2024, Go aws-sdk v2 with Go 1.24+) always
-// negotiate the hybrid first; classical fallback only triggers for peers that
-// cannot do ML-KEM. Those connections still get TLS 1.3 + AES-256-GCM but
-// remain HNDL-exposed — accepted trade-off until AWS CLI v2 / bundled
-// OpenSSL ships ML-KEM (currently lagging upstream OpenSSL 3.5).
-//
-// Weak/niche entries from the Go stdlib default (SecP256r1MLKEM768, P-256,
-// P-521) are intentionally excluded so they cannot be advertised.
-//
-// Go 1.26 stdlib treats tls.Config.CurvePreferences as an allowlist filter
-// applied to the stdlib default order, not as a user-supplied order.
-//
-// Callers must not mutate this slice. Assign it directly to
-// tls.Config.CurvePreferences; do not append to it.
+// Curves is the fixed TLS 1.3 curve allowlist: two ML-KEM hybrids (PQ-capable
+// peers always negotiate these) followed by two classical fallbacks for SDKs
+// without ML-KEM. Weak Go defaults (SecP256r1MLKEM768, P-256, P-521) excluded.
 var Curves = []tls.CurveID{
 	tls.X25519MLKEM768,
 	tls.SecP384r1MLKEM1024,
@@ -46,11 +46,9 @@ func TestCurves_ExcludesWeakPrimitives(t *testing.T) {
 	}
 }

-// TestIntegration_NegotiatesPQHybrid spins up an httptest TLS server
-// configured with Curves + TLS 1.3 minimum, dials it, and asserts a PQ
-// hybrid is negotiated at TLS 1.3. Per Go 1.26 stdlib filter behavior the
-// winner is X25519MLKEM768; the assertion checks for any approved hybrid
-// to remain stable across stdlib reorderings of our 4 entries.
+// TestIntegration_NegotiatesPQHybrid verifies that a Curves-configured server
+// negotiates a PQ hybrid at TLS 1.3; checks for any approved hybrid to stay
+// stable across Go stdlib reorderings.
 func TestIntegration_NegotiatesPQHybrid(t *testing.T) {
 	cert := selfSignedCert(t)
 	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
@@ -95,10 +93,8 @@ func TestIntegration_NegotiatesPQHybrid(t *testing.T) {
 	}
 }

-// TestIntegration_TLS12ClientRejected confirms MinVersion: tls.VersionTLS13
-// is enforced — a TLS 1.2-pinned client gets a handshake error rather than
-// silently downgrading. This is the negative test that closes the HNDL gap
-// on external surfaces.
+// TestIntegration_TLS12ClientRejected confirms MinVersion TLS 1.3 is enforced:
+// a TLS 1.2-pinned client must get a handshake error, not a silent downgrade.
 func TestIntegration_TLS12ClientRejected(t *testing.T) {
 	cert := selfSignedCert(t)
 	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
@@ -1,2 +0,0 @@
-.creds/
-*.retry
@@ -1,222 +0,0 @@
-# Ansible Dev Lifecycle (parallel to bash scripts)
-
-Experimental Ansible-based lifecycle for a single-node Spinifex dev host.
-Lives alongside the existing `scripts/dev-*.sh` and `scripts/reset-dev-env.sh`;
-neither set is removed. Decide which to keep after parity testing.
-
-Plan: `docs/development/improvements/ansible-dev-lifecycle.md` (mulga top-level).
-
-## Prerequisites
-
-Venv lives at sub-repo root (`spinifex/.venv/`) — one venv per sub-repo,
-shared by any future python tooling in spinifex. Bootstrap from `spinifex/`:
-
-```
-sudo apt install python3.13-venv
-python3 -m venv .venv
-source .venv/bin/activate
-pip install 'ansible>=9'
-ansible-galaxy collection install -r scripts/ansible/requirements.yml
-```
-
-Then `cd scripts/ansible` to run playbooks — `ansible.cfg` resolves the
-inventory and roles relative to CWD.
-
-## Playbooks
-
-| Playbook | Purpose | Bash parity |
-|---|---|---|
-| `playbooks/dev-preflight.yml` | Dependency + port check | `dev-setup.sh` §checks |
-| `playbooks/dev-teardown.yml` | Wipe state only (no reinstall) | `reset-dev-env.sh` §teardown |
-| `playbooks/dev-install.yml` | Build + install + init + smoketest on clean box | `dev-install.sh` |
-| `playbooks/dev-reset.yml` | Capture settings → teardown → build → install → init → smoketest | `reset-dev-env.sh` (full) |
-| `playbooks/dev-deploy.yml` | Rebuild + swap binaries/microvm artifacts + restart (no setup.sh, no smoketest) | `make deploy` |
-| `playbooks/dev-status.yml` | Read-only health snapshot (units, ports, OVN/OVS, gateway, counts, drift) | none |
-| `playbooks/dev-logs.yml` | Dump journald + `/var/log/spinifex/*` + status/networking/versions into a tarball | none |
-| `playbooks/dev-snapshot.yml` | Quiesce spinifex+OVN+OVS, tar state paths to a named bundle, restart | none |
-| `playbooks/dev-restore.yml` | Stop services, wipe + untar a named bundle, refresh CA, restart | none |
-| `playbooks/dev-version.yml` | Build-vs-install drift report (spx, plugin, microvm) + sub-repo HEADs | none |
-| `playbooks/dev-vpc.yml` | Idempotent dev VPC fixture (VPC + public subnet + IGW + default route); `-e dev_vpc_state=absent` tears down | partial overlap with `scripts/iac/examples/*` |
-
-Upcoming (not yet implemented):
-
- `dev-mode-start.yml` / `dev-mode-stop.yml` — parity with `start-dev.sh` / `stop-dev.sh`
-  (lowest priority — deprecated bash scripts, CI-only callers).
-
-## Invocation
-
-```
-ansible-playbook playbooks/dev-preflight.yml
-ansible-playbook playbooks/dev-teardown.yml
-ansible-playbook playbooks/dev-install.yml
-ansible-playbook playbooks/dev-reset.yml
-ansible-playbook playbooks/dev-deploy.yml
-ansible-playbook playbooks/dev-status.yml
-ansible-playbook playbooks/dev-logs.yml
-ansible-playbook playbooks/dev-snapshot.yml -e snapshot_name=before-merge
-ansible-playbook playbooks/dev-restore.yml  -e snapshot_name=before-merge
-ansible-playbook playbooks/dev-version.yml
-ansible-playbook playbooks/dev-vpc.yml
-```
-
-Or via `make` (from `spinifex/`):
-
-```
-make ansible-dev-preflight
-make ansible-dev-teardown
-make ansible-dev-install
-make ansible-dev-reset
-make ansible-dev-deploy
-make ansible-dev-status
-make ansible-dev-logs
-make ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=before-merge'
-make ansible-dev-restore  ANSIBLE_EXTRA='-e snapshot_name=before-merge'
-make ansible-dev-version
-make ansible-dev-vpc
-make ansible-dev-vpc ANSIBLE_EXTRA='-e dev_vpc_state=absent'
-```
-
-### When to use which
-
- First-time / clean box → `ansible-dev-install`
- Iterate on Go code, microvm initramfs, lb-agent → `ansible-dev-deploy` (fast)
- Changed systemd units, helper scripts, logrotate, setup.sh → `ansible-dev-reset` (slow, full rebuild)
- Need a clean slate without reinstall → `ansible-dev-teardown`
- "Is my dev box healthy?" → `ansible-dev-status` (read-only, never mutates)
- Filing a bug / capturing the state of a broken box → `ansible-dev-logs` (writes `/tmp/spinifex-logs-<host>-<ts>.tar.gz`; override `-e logs_since=10min` or `-e logs_include_dmesg=false`)
- Checkpoint before a risky op (branch switch, viperblock surgery, ovn restart cycle) → `ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=<name>'`; roll back via `ansible-dev-restore` instead of a full `dev-reset`. Bundles live in `/var/lib/spinifex-snapshots/<name>.tar.gz` (root-owned, survives `dev-teardown`).
- "Do I need to `ansible-dev-deploy`?" → `ansible-dev-version` (drift-only check; faster than `dev-status`)
- Want a known-good VPC/subnet for manual EC2 testing without writing tofu → `ansible-dev-vpc` (smoketest will then find and use it automatically). Tear down with `ANSIBLE_EXTRA='-e dev_vpc_state=absent'`.
-
-## Variable overrides
-
-Defaults in `vars/defaults.yml`. Override per run with `-e`:
-
-```
-ansible-playbook playbooks/dev-teardown.yml -e spinifex_wipe_ssh_keys=true
-```
-
-Useful overrides:
-
-| Var | Default | What it does |
-|---|---|---|
-| `spinifex_wipe_legacy_home` | `true` | Remove `$HOME/spinifex` (legacy dev-mode path) |
-| `spinifex_wipe_ssh_keys` | `false` | Remove `~/.ssh/spinifex-key*` |
-| `spinifex_wipe_aws_creds` | `false` | Remove spinifex profile from `~/.aws/credentials` |
-| `teardown_process_wait_seconds` | `30` | Grace period for processes to exit |
-| `spinifex_region` | `ap-southeast-2` | Region passed to `spx admin init` |
-| `spinifex_az` | `{{ region }}a` | AZ passed to `spx admin init` |
-| `spinifex_external_mode` | `pool` | External networking mode — `pool` or empty (NAT removed) |
-| `spinifex_external_no_external` | `false` | `true` → emit `--no-external` and skip external networking |
-| `spinifex_external_pool_start` | `""` | **Required** for `pool` mode — first IP of the static WAN pool |
-| `spinifex_external_pool_end` | `""` | **Required** for `pool` mode — last IP of the static WAN pool |
-| `spinifex_external_gateway` | `""` | WAN gateway (auto-detected when blank) |
-| `spinifex_external_prefix_len` | `""` | WAN prefix length (auto-detected when blank) |
-| `spinifex_external_gateway_ip` | `""` | OVN router external-IP override (`--gateway-ip`) |
-
-`pool` mode is mandatory unless you set `spinifex_external_no_external=true`.
-`spx admin init` rejects the run without a `--external-pool` range, so the
-init role asserts `spinifex_external_pool_start/_end` upfront. Either persist
-them in `vars/defaults.yml`, or pass them at invocation:
-
-Direct `ansible-playbook` (use `-e` here — this is ansible's flag):
-
-```
-ansible-playbook playbooks/dev-install.yml \
-  -e spinifex_external_pool_start=192.168.1.150 \
-  -e spinifex_external_pool_end=192.168.1.250
-```
-
-Via `make` (use `EXTRA_VARS=` — make eats its own `-e`):
-
-```
-make ansible-dev-install \
-  EXTRA_VARS="spinifex_external_pool_start=192.168.1.150 spinifex_external_pool_end=192.168.1.250"
-```
-
-`dev-reset.yml` automatically captures `region`, `az`, `external_mode`,
-pool range, gateway, prefix-len and `gateway_ip` from the existing
-`/etc/spinifex/spinifex.toml` before teardown and replays them into
-`init`. Legacy `external_mode = "nat"` is coerced to `"pool"` with a warn.
-Use `-e` to override.
-
-## Helper scripts
-
-`scripts/aws-as-spinifex.sh` and `scripts/spx-as-spinifex.sh` wrap `aws`
-and `spx` invocations in `sg spinifex -c '...'` so they run with the
-`spinifex` supplementary group active. Both helpers preset
-`AWS_PROFILE=spinifex`. All AWS-calling tasks in the roles use these
-via `spinifex_aws_cli` / `spinifex_spx_cli` from `vars/defaults.yml`.
-
-Background: `/etc/spinifex/ca.pem` lives under `drwxr-x--- root:spinifex`,
-so the AWS CLI needs `spinifex` group membership to read the CA bundle.
-Ansible's `become_user` is a no-op when the target user equals the
-calling user (no `sudo -u` invoked → no `initgroups(3)` → supplementary
-groups inherited from the operator's shell). On a fresh login the
-operator's shell would have `spinifex` (added by `setup.sh`); on shells
-that predate the group add it does not. The helpers sidestep this
-entirely.
-
-## Leak catalog
-
-`roles/teardown/README.md` tracks every known state source. When teardown
-misses something, add it there and in `roles/teardown/tasks/main.yml`.
-
-## Multi-node cluster bootstrap
-
-`playbooks/cluster-bootstrap.yml` provisions a 2+ node Spinifex cluster
-across a long-running tofu env (`env1`, `env2`, `bryn`, etc). Inventory
-is generated dynamically from `scripts/tofu-cluster/envs/<env>.tfvars`
-via `inventory/tofu.py` — first node in `nodes = [...]` is the init host,
-the rest are joins.
-
-```bash
-make ansible-cluster-bootstrap ENV=env1 POOL=192.168.1.150-192.168.1.159
-```
-
-`POOL` is optional; omit for DHCP fallback. Equivalent without make:
-
-```bash
-cd scripts/ansible
-CLUSTER_ENV=env1 ansible-playbook -i inventory/tofu.py \
-    playbooks/cluster-bootstrap.yml \
-    -e cluster_env=env1 \
-    -e cluster_external_pool=192.168.1.150-192.168.1.159
-```
-
-Plays sequence:
-
-1. **runner (localhost)** — `build` role produces `/tmp/spinifex-local.tar.gz`.
-2. **all env hosts (parallel)** — copies tarball, stops any dev-layout
-   services (`~/spinifex/`), runs `preflight` + `install` roles. `setup.sh`
-   creates `/etc/spinifex`, the `spinifex` user/group, systemd units, and
-   installs `/usr/local/bin/spx`.
-3. **init host only** — `cluster-init` role runs `spx admin init` in the
-   background, waits for the formation server, captures the join token.
-   CA is then copied into `/usr/local/share/ca-certificates/spinifex-ca.crt`
-   **before** `spinifex.target` starts (required — predastore enforces
-   strict TLS verify against the system trust store).
-4. **join hosts (parallel)** — `cluster-join` role runs `spx admin join`
-   against the formation server, then installs CA, then starts
-   `spinifex.target`.
-5. **all hosts** — polls `https://<ip>:4432/health` until `awsgw=ok`.
-6. **init host only** — `cluster-post` role imports the SSH key + Ubuntu
-   AMI (both idempotent: skip if already present).
-
-Re-running the playbook against an already-bootstrapped env is safe —
-`setup.sh`, `spx admin init --force`, CA copy, key/AMI import are all
-idempotent.
-
-### Migration from `bootstrap.sh`
-
-`scripts/tofu-cluster/bootstrap.sh` ran services under `~/spinifex/` as
-`tf-user` via `start-dev.sh` with no systemd supervision, and installed
-the cluster CA *after* services started — predastore would crash with
-`x509: certificate signed by unknown authority` on every first boot.
-`cluster-bootstrap.yml` replaces that flow with production-layout
-(`/etc/spinifex/` + systemd) and correct CA-before-services ordering.
-The first run on an env previously bootstrapped by `bootstrap.sh` will
-stop the dev-layout services, install the prod layout, and re-init the
-cluster from scratch via `spx admin init --force`.
-
-`bootstrap.sh` stays in-tree for CI and pseudo-multinode envs.
@@ -1,17 +0,0 @@
-[defaults]
-inventory = inventory/localhost.yml
-roles_path = roles
-stdout_callback = ansible.builtin.default
-host_key_checking = False
-retry_files_enabled = False
-interpreter_python = auto_silent
-forks = 10
-
-[inventory]
-# inventory/tofu.py is an executable script inventory used by
-# cluster-bootstrap.yml — opted into via `-i inventory/tofu.py` on the
-# command line; not default to keep dev-* playbooks pointing at localhost.
-enable_plugins = host_list, script, auto, yaml, ini, toml
-
-[callback_default]
-result_format = yaml
@@ -1,6 +0,0 @@
---
-all:
-  hosts:
-    localhost:
-      ansible_connection: local
-      ansible_python_interpreter: "{{ ansible_playbook_python }}"
@@ -1,170 +0,0 @@
-#!/usr/bin/env python3
-"""Dynamic ansible inventory generator from tofu-cluster tfvars.
-
-Reads scripts/tofu-cluster/envs/<env>.tfvars (HCL subset) and emits ansible
-JSON inventory. First node in the tfvars `nodes = [...]` list is the init
-host (forms the cluster); the rest are join hosts.
-
-Usage:
-    inventory/tofu.py --list --env env1
-    inventory/tofu.py --host <ip> --env env1
-
-Selecting the env:
-    CLUSTER_ENV=env1 inventory/tofu.py --list
-    inventory/tofu.py --list --env env1
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import os
-import re
-import sys
-from pathlib import Path
-
-
-def _repo_root() -> Path:
-    here = Path(__file__).resolve()
-    for parent in here.parents:
-        if (parent / "go.work").exists() or (parent / ".gitmodules").exists():
-            return parent
-    raise SystemExit(f"could not locate mulga repo root from {here}")
-
-
-def _tfvars_path(env: str) -> Path:
-    root = _repo_root()
-    candidate = root / "scripts" / "tofu-cluster" / "envs" / f"{env}.tfvars"
-    if not candidate.exists():
-        raise SystemExit(f"tfvars file not found: {candidate}")
-    return candidate
-
-
-_STRING_RE = re.compile(r'^\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*=\s*"([^"]*)"\s*$')
-_NUMBER_RE = re.compile(r'^\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*=\s*([0-9]+)\s*$')
-_NODE_FIELD_RE = re.compile(r'(\w+)\s*=\s*"([^"]*)"')
-
-
-def _parse_tfvars(path: Path) -> dict:
-    """Minimal tfvars parser — handles the subset used by envs/*.tfvars.
-
-    Supports:
-      key = "string"
-      key = 12345
-      nodes = [
-        { name = "x", wan_ip = "1.2.3.4", ... },
-        ...
-      ]
-    """
-    out: dict = {"nodes": []}
-    text = path.read_text()
-
-    # Strip line comments
-    text = re.sub(r"#.*", "", text)
-
-    # Pull the nodes block out first, then process remaining lines.
-    nodes_match = re.search(r"nodes\s*=\s*\[(.+?)\]\s*$", text, re.DOTALL | re.MULTILINE)
-    if nodes_match:
-        nodes_body = nodes_match.group(1)
-        for entry in re.finditer(r"\{([^}]+)\}", nodes_body):
-            fields = dict(_NODE_FIELD_RE.findall(entry.group(1)))
-            if "name" in fields and "wan_ip" in fields:
-                out["nodes"].append(fields)
-        text = text.replace(nodes_match.group(0), "")
-
-    for line in text.splitlines():
-        if m := _STRING_RE.match(line):
-            out[m.group(1)] = m.group(2)
-        elif m := _NUMBER_RE.match(line):
-            out[m.group(1)] = int(m.group(2))
-
-    return out
-
-
-def _resolve_ssh_key(value: str | None) -> str:
-    if not value:
-        return os.path.expanduser("~/.ssh/tf-user-ap-southeast-2")
-    return os.path.expanduser(value)
-
-
-def _build_inventory(env: str) -> dict:
-    data = _parse_tfvars(_tfvars_path(env))
-    nodes = data.get("nodes") or []
-    if not nodes:
-        raise SystemExit(f"{env}.tfvars has no nodes")
-
-    ssh_user = data.get("ssh_user", "tf-user")
-    ssh_key = _resolve_ssh_key(data.get("ssh_public_key_path", "").replace(".pub", ""))
-
-    hostvars: dict[str, dict] = {}
-    all_hosts: list[str] = []
-    init_hosts: list[str] = []
-    join_hosts: list[str] = []
-
-    for idx, node in enumerate(nodes):
-        ip = node["wan_ip"]
-        role = "init" if idx == 0 else "join"
-        hostvars[ip] = {
-            "ansible_host": ip,
-            "ansible_user": ssh_user,
-            "ansible_ssh_private_key_file": ssh_key,
-            "ansible_ssh_common_args": "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR",
-            "spinifex_node_name": node["name"],
-            "spinifex_node_role": role,
-            "spinifex_node_index": idx,
-            "spinifex_wan_ip": ip,
-        }
-        all_hosts.append(ip)
-        (init_hosts if role == "init" else join_hosts).append(ip)
-
-    # Group-level vars derived from tfvars (operator can override with -e).
-    cluster_init_host = init_hosts[0]
-    group_vars = {
-        "cluster_env": env,
-        "cluster_init_host": cluster_init_host,
-        "cluster_init_ip": cluster_init_host,
-        "spinifex_node_count": len(nodes),
-        "spinifex_region": data.get("region", "ap-southeast-2"),
-        "spinifex_az": data.get("az", "ap-southeast-2a"),
-    }
-    if cidr := data.get("env_cidr"):
-        group_vars["cluster_env_cidr"] = cidr
-
-    inventory = {
-        "_meta": {"hostvars": hostvars},
-        env: {"hosts": all_hosts, "vars": group_vars},
-        "cluster": {"children": [env]},
-        "init": {"hosts": init_hosts},
-        "join": {"hosts": join_hosts},
-    }
-    return inventory
-
-
-def main() -> int:
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("--list", action="store_true", help="emit full inventory")
-    parser.add_argument("--host", help="emit hostvars for a single host (ignored — _meta carries them)")
-    parser.add_argument(
-        "--env",
-        default=os.environ.get("CLUSTER_ENV"),
-        help="tofu env name (matches scripts/tofu-cluster/envs/<env>.tfvars); also via CLUSTER_ENV env var",
-    )
-    args = parser.parse_args()
-
-    if not args.env:
-        sys.stderr.write("error: --env required (or CLUSTER_ENV)\n")
-        return 64
-
-    if args.host:
-        json.dump({}, sys.stdout)
-        sys.stdout.write("\n")
-        return 0
-
-    inventory = _build_inventory(args.env)
-    json.dump(inventory, sys.stdout, indent=2)
-    sys.stdout.write("\n")
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())
@@ -1,330 +0,0 @@
---
-# cluster-bootstrap.yml — multi-node Spinifex cluster bootstrap against a
-# tofu-provisioned env. Inventory comes from inventory/tofu.py which parses
-# scripts/tofu-cluster/envs/<env>.tfvars.
-#
-# Usage:
-#   cd spinifex/scripts/ansible
-#   ansible-playbook -i inventory/tofu.py playbooks/cluster-bootstrap.yml \
-#     -e cluster_env=env1 -e cluster_external_pool=192.168.1.150-192.168.1.159
-#
-# Or via the make wrapper:
-#   make ansible-cluster-bootstrap ENV=env1 POOL=192.168.1.150-192.168.1.159
-#
-# Behaviour:
-#   - all hosts: preflight + install (setup.sh) — production layout
-#   - init host: spx admin init (backgrounded) + CA install + spinifex.target
-#   - join hosts: spx admin join + CA install + spinifex.target (parallel)
-#   - init host: import ssh key + ubuntu AMI (idempotent)
-
- name: Cluster bootstrap — build artifact once on runner
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: assert cluster_env supplied
-      ansible.builtin.assert:
-        that: cluster_env | default('') | length > 0
-        fail_msg: "Pass -e cluster_env=<env> (matches scripts/tofu-cluster/envs/<env>.tfvars)"
-  roles:
-    - build
-
- name: Cluster bootstrap — install (all nodes, parallel)
-  hosts: "{{ cluster_env }}"
-  gather_facts: true
-  serial: 100%
-  vars:
-    install_run_ovn: false   # we drive setup-ovn ourselves with role-aware args
-    cluster_bootstrap_purge: true   # wipe stale single-node predastore.toml + raft data
-  vars_files:
-    - ../vars/defaults.yml
-  pre_tasks:
-    - name: copy build tarball to each node
-      ansible.builtin.copy:
-        src: "{{ build_tarball_path }}"
-        dest: "{{ build_tarball_path }}"
-        mode: "0644"
-
-    - name: stop dev-layout services if running (migration from ~/spinifex/)
-      become: true
-      # Bracket trick on the first letter prevents the pattern literal in the
-      # shell command line from matching the running shell itself (pkill -f
-      # against literal patterns is otherwise self-killing — rc=-9).
-      # `spx [a]dmin` catches lingering formation servers from prior bootstrap
-      # attempts that own port 4432 and steal joins from the new init.
-      ansible.builtin.shell:
-        cmd: |
-          pkill -9 -f "[b]in/spx service" 2>/dev/null || true
-          pkill -9 -f "spx [a]dmin" 2>/dev/null || true
-          pkill -9 -f "spinifex-[u]i" 2>/dev/null || true
-          pkill -9 -f "[n]ats-server" 2>/dev/null || true
-          pkill -9 -f "[p]redastore" 2>/dev/null || true
-          pkill -9 -f "[v]iperblock" 2>/dev/null || true
-          sleep 2
-          rm -rf "{{ ansible_env.HOME }}/spinifex" 2>/dev/null || true
-          exit 0
-      changed_when: false
-  roles:
-    - install
-
-# Wipe OVN/OVS state before setup-ovn runs. Without this, re-bootstrapping a
-# previously-built cluster leaves stale chassis rows in the SB DB, stale
-# external_ids on Open_vSwitch, stale br-int flows + ports, stale
-# systemd-networkd unit files, and stale ovs-monitor-ipsec strongSwan certs.
-# Symptom: ovn-controller flows reference dead encap endpoints; IPsec SAs
-# fail AUTHENTICATION_FAILED; new joins shadow ghosts under the same
-# node-name.
- name: Cluster bootstrap — OVN/OVS state reset (all nodes, parallel)
-  hosts: "{{ cluster_env }}"
-  gather_facts: false
-  serial: 100%
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: reset — stop OVN/OVS services
-      become: true
-      ansible.builtin.systemd:
-        name: "{{ item }}"
-        state: stopped
-      loop:
-        - ovn-controller
-        - ovn-central
-        - openvswitch-ipsec
-        - openvswitch-switch
-      failed_when: false
-
-    - name: reset — delete OVN NB/SB databases
-      become: true
-      ansible.builtin.file:
-        path: "{{ item }}"
-        state: absent
-      loop:
-        - "{{ ovn_nb_db }}"
-        - "{{ ovn_sb_db }}"
-
-    - name: reset — delete OVS conf.db + system-id (force fresh ovsdb)
-      become: true
-      ansible.builtin.file:
-        path: "{{ item }}"
-        state: absent
-      loop:
-        - /etc/openvswitch/conf.db
-        - /etc/openvswitch/.conf.db.~lock~
-        - /etc/openvswitch/system-id.conf
-
-    - name: reset — drop spinifex-managed systemd-networkd unit files
-      become: true
-      ansible.builtin.shell: |
-        rm -f /etc/systemd/network/14-spinifex-*.netdev \
-              /etc/systemd/network/15-spinifex-*.netdev \
-              /etc/systemd/network/15-spinifex-*.network \
-              /etc/systemd/network/16-spinifex-*.network
-        networkctl reload 2>/dev/null || true
-      changed_when: true
-
-    - name: reset — enumerate stale spinifex veth/tap netdevs
-      ansible.builtin.shell: |
-        ip -o link show | awk -F': ' '{print $2}' | cut -d'@' -f1 | \
-          grep -E '^(veth-wan-br|veth-wan-ovs|spx-ext-|spx-tap-|spx-)' || true
-      register: reset_links
-      changed_when: false
-
-    - name: reset — delete stale netdevs
-      become: true
-      ansible.builtin.command: "ip link delete {{ item }}"
-      loop: "{{ reset_links.stdout_lines }}"
-      changed_when: true
-      failed_when: false
-
-    - name: reset — clear ovs-monitor-ipsec strongSwan state
-      become: true
-      ansible.builtin.shell: |
-        rm -rf /var/lib/openvswitch/pki 2>/dev/null || true
-        rm -f /etc/ipsec.d/certs/ovs-* /etc/ipsec.d/private/ovs-* 2>/dev/null || true
-        rm -f /etc/ipsec.conf.d/ovs-* 2>/dev/null || true
-      changed_when: true
-
-    - name: reset — start openvswitch-switch with fresh conf.db
-      become: true
-      ansible.builtin.systemd:
-        name: openvswitch-switch
-        state: started
-        daemon_reload: true
-
- name: Cluster bootstrap — OVN central (init host only)
-  hosts: init
-  gather_facts: false
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: setup-ovn --management on init host
-      become: true
-      ansible.builtin.command:
-        cmd: >-
-          /usr/local/share/spinifex/setup-ovn.sh
-          --management
-          --encap-ip={{ spinifex_wan_ip }}
-          --node-name={{ hostvars[inventory_hostname]['spinifex_node_name'] }}
-      changed_when: true
-
- name: Cluster bootstrap — OVN chassis (join hosts, parallel)
-  hosts: join
-  gather_facts: false
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: setup-ovn --ovn-remote on join hosts
-      become: true
-      ansible.builtin.command:
-        cmd: >-
-          /usr/local/share/spinifex/setup-ovn.sh
-          --ovn-remote=tcp:{{ cluster_init_host }}:6642
-          --encap-ip={{ spinifex_wan_ip }}
-          --node-name={{ hostvars[inventory_hostname]['spinifex_node_name'] }}
-      changed_when: true
-
- name: Cluster bootstrap — init host (start formation server)
-  hosts: init
-  gather_facts: false
-  vars_files:
-    - ../vars/defaults.yml
-  roles:
-    - cluster-init
-
- name: Cluster bootstrap — join hosts run spx admin join (parallel)
-  hosts: join
-  gather_facts: false
-  serial: 100%
-  vars_files:
-    - ../vars/defaults.yml
-  roles:
-    - cluster-join
-
- name: Cluster bootstrap — wait for init to finish writing multi-node configs
-  hosts: init
-  gather_facts: false
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: wait for spx admin init async job to complete
-      become: true
-      ansible.builtin.async_status:
-        jid: "{{ cluster_init_async.ansible_job_id }}"
-      register: cluster_init_done
-      until: cluster_init_done.finished
-      retries: 60
-      delay: 5
-
-    - name: dump init log on failure
-      ansible.builtin.command: tail -100 /tmp/spx-admin-init.log
-      register: cluster_init_log
-      changed_when: false
-      when: cluster_init_done.rc | default(1) != 0
-
-    - name: assert init exited cleanly
-      ansible.builtin.assert:
-        that: cluster_init_done.rc == 0
-        fail_msg: |
-          spx admin init failed (rc={{ cluster_init_done.rc }})
-          {{ cluster_init_log.stdout | default('(log not captured)') }}
-
-    - name: confirm multi-node predastore.toml written
-      become: true
-      ansible.builtin.shell: |
-        if grep -q '127\.0\.0\.1' {{ spinifex_etc_dir }}/predastore/predastore.toml 2>/dev/null; then
-          echo "ERROR: predastore.toml still references loopback — multi-node config not written"
-          exit 1
-        fi
-        grep -c '^\[\[nodes\]\]' {{ spinifex_etc_dir }}/predastore/predastore.toml 2>/dev/null || echo 0
-      register: predastore_node_count
-      changed_when: false
-
- name: Cluster bootstrap — install CA on all nodes (before service start)
-  hosts: "{{ cluster_env }}"
-  gather_facts: false
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: install CA into system trust
-      become: true
-      ansible.builtin.copy:
-        src: "{{ spinifex_ca_src }}"
-        dest: "{{ spinifex_ca_trust_path }}"
-        mode: "0644"
-        remote_src: true
-
-    - name: rebuild CA bundle
-      become: true
-      ansible.builtin.command: update-ca-certificates
-      changed_when: true
-
-    # spinifex-ui reverse-proxies localhost:9999 (awsgw) and localhost:8443
-    # (predastore) — hardcoded in spinifex/services/spinifexui/spinifexui.go.
-    # spx admin init binds those services to BindIP only, so the loopback
-    # connection fails (BadGateway in UI). Patch to wildcard bind; cert
-    # already includes 127.0.0.1 in IP SANs.
-    - name: patch awsgw bind to 0.0.0.0 (UI proxies via localhost)
-      become: true
-      ansible.builtin.replace:
-        path: "{{ spinifex_config_file }}"
-        regexp: '^(\s*host\s*=\s*")[^"]+(:9999")'
-        replace: '\g<1>0.0.0.0\g<2>'
-
-    - name: patch predastore bind to 0.0.0.0 (UI proxies via localhost)
-      become: true
-      ansible.builtin.replace:
-        path: "{{ spinifex_config_file }}"
-        regexp: '^(\s*host\s*=\s*")[^"]+(:8443")'
-        replace: '\g<1>0.0.0.0\g<2>'
-
- name: Cluster bootstrap — start services (init first, then joins)
-  hosts: init
-  gather_facts: false
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: start spinifex.target on init
-      become: true
-      ansible.builtin.systemd:
-        name: spinifex.target
-        state: started
-
- name: Cluster bootstrap — start services on join hosts
-  hosts: join
-  gather_facts: false
-  serial: 100%
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: start spinifex.target on joiners
-      become: true
-      ansible.builtin.systemd:
-        name: spinifex.target
-        state: started
-
- name: Cluster bootstrap — readiness check (all nodes)
-  hosts: "{{ cluster_env }}"
-  gather_facts: false
-  vars_files:
-    - ../vars/defaults.yml
-  tasks:
-    - name: wait for awsgw responsiveness
-      ansible.builtin.uri:
-        url: "https://{{ spinifex_wan_ip }}:{{ cluster_formation_port }}/health"
-        validate_certs: false
-        return_content: true
-      register: cluster_health
-      until:
-        - cluster_health.status == 200
-        - cluster_health.json.service_health.awsgw == "ok"
-      retries: 30
-      delay: 5
-
- name: Cluster bootstrap — post-bootstrap fixtures (init host only)
-  hosts: init
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-  roles:
-    - cluster-post
@@ -1,14 +0,0 @@
---
-# In-place artifact swap: rebuilds + replaces binaries/microvm artifacts on
-# an already-installed node, then restarts spinifex.target. Use when iterating
-# on code without altering systemd units, users, dirs, or running the full
-# smoketest. For first-time or unit/script changes use dev-install / dev-reset.
- name: Dev deploy — build + swap binaries + restart (no setup.sh, no smoketest)
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-
-  roles:
-    - build
-    - deploy
@@ -1,67 +0,0 @@
---
- name: Dev install — build, install, init, smoketest (idempotent)
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-
-  pre_tasks:
-    - name: preflight
-      ansible.builtin.include_role:
-        name: preflight
-
-    - name: health — config file present?
-      ansible.builtin.stat:
-        path: "{{ spinifex_config_file }}"
-      register: health_config
-
-    - name: health — spinifex.target active?
-      become: true
-      ansible.builtin.command: systemctl is-active spinifex.target
-      register: health_target
-      changed_when: false
-      failed_when: false
-
-    - name: health — awsgw responds?
-      become: true
-      become_user: "{{ spinifex_invoking_user }}"
-      environment:
-        AWS_PROFILE: spinifex
-      ansible.builtin.command: aws ec2 describe-regions
-      register: health_gw
-      changed_when: false
-      failed_when: false
-
-    - name: health — spx binary installed?
-      ansible.builtin.stat:
-        path: /usr/local/bin/spx
-      register: health_spx
-
-    - name: health — set already_installed fact
-      ansible.builtin.set_fact:
-        spinifex_already_installed: >-
-          {{ health_config.stat.exists
-             and health_spx.stat.exists
-             and (health_target.stdout | trim) == 'active'
-             and health_gw.rc == 0 }}
-
-    - name: health — summary
-      ansible.builtin.debug:
-        msg: |
-          Health:
-            spx binary: {{ 'present' if health_spx.stat.exists else 'missing' }}
-            config:     {{ 'present' if health_config.stat.exists else 'missing' }}
-            target:     {{ health_target.stdout | trim }}
-            awsgw:      {{ 'ok' if health_gw.rc == 0 else 'not responding' }}
-          Already installed: {{ spinifex_already_installed }}
-          {% if spinifex_already_installed %}
-          → Skipping build/install/init. Smoketest only.
-          {% else %}
-          → Running full build → install → init → smoketest.
-          {% endif %}
-
-  roles:
-    - { role: build, when: not (spinifex_already_installed | default(false)) }
-    - { role: install, when: not (spinifex_already_installed | default(false)) }
-    - { role: init, when: not (spinifex_already_installed | default(false)) }
-    - smoketest
@@ -1,17 +0,0 @@
---
-# Capture spinifex + ovn/ovs + qemu/nbdkit logs from journald and
-# /var/log/spinifex/ into a single tarball for bug reports.
-# Tail mode is intentionally not provided — use `journalctl -u 'spinifex-*' -f`
-# directly; ansible is the wrong tool for follow.
- name: Dev logs — dump journald + varlog + status into a bundle
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-  vars:
-    logs_since: "1h"
-    logs_output_dir: "/tmp"
-    logs_include_dmesg: true
-
-  roles:
-    - logs
@@ -1,8 +0,0 @@
---
- name: Dev preflight — validate host dependencies
-  hosts: localhost
-  gather_facts: true
-  vars_files:
-    - ../vars/defaults.yml
-  roles:
-    - preflight
--- a/Show More
+++ b/Show More