release: update go.mod for v1.9.0

* feat: add predastore compaction E2E

Inject a short compaction interval into test-cluster predastore config via a
new --predastore-compaction-interval flag on admin init/join, gated so
production output stays byte-identical when unset. Add a harness helper that
drives write-then-delete churn against predastore S3, brackets it with
du-over-SSH usage, and asserts end-of-churn usage settles below 70% of the
churn peak; wire it into the multinode suite as TestPredastoreCompaction.

* fix: compaction E2E gate writes churn to its own bucket

The reclaim gate signed PutObject against the config-defined system
'predastore' bucket, which belongs to account 000000000000 with only a
static [[auth]] grant. The harness admin identity is a different IAM
account, so predastore returned 403 AccessDenied cross-account.

Create a per-run bucket the admin identity owns (AdministratorAccess)
and churn into it. Objects still erasure-code into distributed/nodes/
node-* so the du-based reclaim measurement is unchanged. The gateway
(:9999) does not proxy S3, so the client stays predastore-direct.

* test: run predastore compaction gate before destructive node tests

The compaction reclaim gate was declared last in the multinode suite, so it
measured shard-store usage after NodeFailure/NodeRecovery had torn down and
restarted node2. NodeRecovery only gates on NATS/gateway/EC2 readiness, not
predastore, so node2's shard store was still settling and the du baseline came
back empty. Move the test before the destructive node tests so it runs against
a pristine, fully-converged cluster primed by the volume tests.

* fix: race-free SSH output capture + empty-tolerant du probe

The compaction reclaim gate crashed with ParseInt("") when node1's du
probe returned empty combined output. Harden the probe and the SSH layer:

- ssh.Run used one bytes.Buffer for both session.Stdout and session.Stderr,
  which x/crypto/ssh copies in two concurrent goroutines — a data race on a
  non-concurrent-safe type. Switch to session.CombinedOutput, which guards the
  combined stream with a mutex-wrapped singleWriter.
- shardStoreUsageBytes now terminates the remote command with a sentinel echo.
  A missing sentinel is reported as output lost in SSH retrieval; a present
  sentinel with no number is treated as a 0 measurement (anomalous, logged)
  instead of crashing the gate.

The sentinel lets the next run distinguish a transport-layer drop from a
genuinely empty remote measurement.

* test(e2e): replace predastore du-reclaim gate with object lifecycle contract

The multinode compaction gate measured shard-store disk usage over SSH (du)
and asserted reclaim after churn. It could never pass: segments only seal at
the 4 GiB roll threshold, so test-scale churn produces no compactable segment,
and the du probe read 0 bytes before and after. Reclaim correctness is already
covered deterministically by predastore's store-level unit tests.

Delete AssertPredastoreCompaction and its du/churn/poll/sentinel machinery,
the harness unit tests that exercised them, and the multinode TestPredastore
Compaction entry. Add a single-node TestPredastoreObjectLifecycle that checks
the user-visible contract: an object is readable after PutObject, gone after
DeleteObject (GET returns NoSuchKey and ListObjectsV2 is empty), and the store
keeps serving fresh writes. newPredastoreS3 is retained, refactored to take a
host so it works outside the multinode cluster type.

* test(e2e): drop orphaned multinode TestPredastoreCompaction

Complete the du-reclaim -> object-lifecycle refactor: remove the
multinode TestPredastoreCompaction caller (and its now-unused context
and harness imports) that still referenced the deleted
harness.AssertPredastoreCompaction, which broke e2e-build. Wire the
single-node TestPredastoreObjectLifecycle entry that replaces it.

.github/pull_request_template.md

@@ -0,0 +1,7 @@
+## Summary
+
+<!-- What does this PR do, and why? -->
+
+## Changes
+
+<!-- The notable changes, as bullets. -->

.github/workflows/e2e-baremetal.yml

@@ -27,16 +27,16 @@ jobs:
       cancel-in-progress: false
     env:
       BM_ISO_CACHE: /var/cache/spinifex-iso-builder
-      ARTIFACT_DIR: /tmp/spinifex-e2e-baremetal-artifacts-${{ github.run_id }}
+      ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-20-${{ github.run_id }}
     steps:
       - name: Clean Workspace
-        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          sudo umount -R -l "${BM_ISO_CACHE}/rootfs" 2>/dev/null || true
+          if [ ! -c /dev/null ]; then sudo rm -f /dev/null; sudo mknod -m 666 /dev/null c 1 3; fi
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
 
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
         with:
-          # When called (nightly/release) the caller picks the ref (dev on
-          # schedule, the tag on release); standalone dispatch defaults to the
-          # triggering ref.
           ref: ${{ inputs.ref || github.ref }}
 
       - uses: actions/create-github-app-token@v3
@@ -46,7 +46,7 @@ jobs:
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
         with:
           repository: mulgadc/mulga
           token: ${{ steps.app-token.outputs.token }}
@@ -104,7 +104,6 @@ jobs:
             echo "Ubuntu 26.04 cloud image cached ($(du -h "$UBUNTU_IMG" | cut -f1))"
           fi
 
-
       - name: Install go-junit-report on runner
         run: go install github.com/jstemmer/go-junit-report/v2@latest
 
@@ -117,10 +116,6 @@ jobs:
           SPX_EXTERNAL_POOL: ${{ env.SPINIFEX_CI_POOL }}
         run: |
           mkdir -p "${ARTIFACT_DIR}"
-          # Capture combined bootstrap + e2e output so failures produce a
-          # downloadable, analyzable artifact (the suites run go test -v on the
-          # init node and stream back over SSH). pipefail keeps the script's
-          # real exit code through the tee.
           set -o pipefail
           sudo -E mulga/scripts/baremetal/run-baremetal-e2e.sh \
             --vmlinuz   "${BM_ISO_CACHE}/vmlinuz" \
@@ -142,12 +137,12 @@ jobs:
         with:
           junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
           log-dir: ${{ env.ARTIFACT_DIR }}
-          title: "Bare-Metal E2E failure analysis"
+          title: "E2E failure analysis (cell 20)"
 
       - uses: actions/upload-artifact@v7
         if: always()
         with:
-          name: e2e-baremetal-artifacts-${{ github.run_id }}
+          name: e2e-artifacts-cell-20-${{ github.run_id }}
           path: ${{ env.ARTIFACT_DIR }}
           if-no-files-found: warn
           retention-days: 14

.github/workflows/e2e-ddil.yml

@@ -21,10 +21,6 @@ jobs:
   ddil:
     name: DDIL Real Multi-Node
     runs-on: [ self-hosted, ci-multi ]
-    # Budget: ~45m go test + tofu provision/bootstrap/teardown on ci-multi.
-    # In Phase 1 the scenarios all t.Skip, so the real-world runtime is
-    # dominated by provisioning; the 60m ceiling leaves headroom for
-    # Phase 2/3 when assertions come online.
     timeout-minutes: 60
     env:
       TOFU_LOG: /tmp/e2e-tofu-ddil-${{ github.run_id }}.log
@@ -39,16 +35,12 @@ jobs:
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
-      # Full spinifex checkout: the Go harness must run on the runner
-      # (outside the cluster) so its SSH path survives scenario partitions
-      # that drop peer-to-peer cluster traffic. Running inside the cluster
-      # would self-sever when Scenario C isolates a node.
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
         name: Checkout Spinifex
         with:
           path: spinifex
 
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
         name: Checkout Mulga (tofu-cluster)
         with:
           repository: mulgadc/mulga
@@ -88,7 +80,6 @@ jobs:
         working-directory: mulga/scripts/tofu-cluster
         run: |
           set -euo pipefail
-          # shellcheck source=/dev/null
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           export TF_VAR_git_branch="${GIT_BRANCH}"
           quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
@@ -140,10 +131,6 @@ jobs:
             exit 1
           fi
 
-          # Guest SSH key: bootstrap-install.sh imports spinifex-key into EC2;
-          # the matching private material was generated on the primary node at
-          # ~/.ssh/spinifex-key. Pull it to the runner so the harness can SSH
-          # the witness via its MapPublicIpOnLaunch-assigned EIP.
           GUEST_KEY="${RUNNER_TEMP}/spinifex-key"
           scp -i "${SSH_KEY}" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
             "${SSH_USER}@${PRIMARY_IP}:~/.ssh/spinifex-key" "${GUEST_KEY}"
@@ -151,11 +138,6 @@ jobs:
 
           mkdir -p "${RUNNER_TEMP}/ddil-results"
           cd "${GITHUB_WORKSPACE}/spinifex"
-          # gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout,
-          # plus machine-readable jsonfile + JUnit XML artifacts. The JUnit
-          # file feeds the junit-report composite action below so failures
-          # become summary table rows + workflow annotations.
-          # shellcheck source=/dev/null
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           banner "DDIL scenarios"
           DDIL_NODES="${NODE_IPS}" \
@@ -185,10 +167,6 @@ jobs:
         if: failure()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          # Pull spinifex-* journals + QEMU process listings from every
-          # node into the artifact bundle so failures can be triaged
-          # without re-provisioning the cluster. Best-effort — a missing
-          # node should not mask the original test failure.
           set +e
           INVENTORY=$(tofu output -json inventory)
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
@@ -197,7 +175,6 @@ jobs:
           LOG_DIR="${RUNNER_TEMP}/ddil-results/cluster-logs"
           mkdir -p "${LOG_DIR}"
 
-          # shellcheck source=/dev/null
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           mapfile -t NODE_IPS < <(echo "$INVENTORY" | jq -r '.nodes[].wan_ip')
           for ip in "${NODE_IPS[@]}"; do
@@ -221,7 +198,6 @@ jobs:
         working-directory: mulga/scripts/tofu-cluster
         run: |
           set -euo pipefail
-          # shellcheck source=/dev/null
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
           quiet_run "tofu destroy"          "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false

.github/workflows/e2e-nightly.yml

@@ -40,6 +40,7 @@ jobs:
     timeout-minutes: 35
     env:
       TOFU_LOG: /tmp/e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}.log
+      ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}
 
     steps:
       # blank or scheduled run = full matrix.
@@ -60,7 +61,10 @@ jobs:
 
       - name: Clean Workspace
         if: steps.gate.outputs.selected == 'true'
-        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          sudo umount -R -l /var/cache/spinifex-iso-builder/rootfs 2>/dev/null || true
+          if [ ! -c /dev/null ]; then sudo rm -f /dev/null; sudo mknod -m 666 /dev/null c 1 3; fi
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         if: steps.gate.outputs.selected == 'true'
@@ -70,7 +74,7 @@ jobs:
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
         if: steps.gate.outputs.selected == 'true'
         with:
           repository: mulgadc/mulga
@@ -78,6 +82,11 @@ jobs:
           sparse-checkout: scripts/tofu-cluster
           path: mulga
 
+      - uses: actions/checkout@v7
+        if: steps.gate.outputs.selected == 'true'
+        with:
+          path: spinifex
+
       - name: Build Install Artifacts
         if: steps.gate.outputs.selected == 'true'
         working-directory: mulga/scripts/tofu-cluster
@@ -132,8 +141,18 @@ jobs:
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
-          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
-            "cd ~/e2e-tests && ./run-e2e.sh && ./run-cert-e2e.sh && ENDPOINT=https://${WAN_IP}:9999 ./run-lb-e2e.sh"
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
+             for SUITE in single cert lb; do \
+               echo \"::group::suite \$SUITE\"; \
+               date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
+               GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+                 ARTIFACT_DIR='${ARTIFACT_DIR}' \
+                 ./\$SUITE.test -test.v -test.timeout=30m \
+                 2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log'; \
+               rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
+               if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
+             done; exit \$FAIL"
 
       - name: Run Multi Node E2E
         if: steps.gate.outputs.selected == 'true' && matrix.topo == 'real-multi'
@@ -144,32 +163,57 @@ jobs:
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
 
-          NODE_IPS=""
+          NODES_CSV=""
           NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
           for i in $(seq 0 $((NODE_COUNT - 1))); do
-            NODE_IPS="${NODE_IPS} $(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")"
+            ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
+            NODES_CSV="${NODES_CSV:+$NODES_CSV,}$ip"
           done
 
-          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
-            "cd ~/e2e-tests && ./run-multinode-e2e.sh ${NODE_IPS} && ./run-cert-e2e.sh --multinode ${NODE_IPS} && ENDPOINT=https://${NODE1_IP}:9999 ./run-lb-e2e.sh --nodes ${NODE_IPS}"
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
+             for SUITE in multinode cert lb; do \
+               echo \"::group::suite \$SUITE\"; \
+               date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
+               GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
+                 SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
+                 SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
+                 ARTIFACT_DIR='${ARTIFACT_DIR}' \
+                 ./\$SUITE.test -test.v -test.timeout=30m \
+                 2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log'; \
+               rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
+               if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
+             done; exit \$FAIL"
 
       - name: Run Tofu Workbooks
         if: steps.gate.outputs.selected == 'true' && matrix.extra == 'tofu-examples'
         working-directory: mulga/scripts/tofu-cluster
+        # The driver emits go test -v markers per workbook; teeing to
+        # test-tofu.log feeds the same go-junit-report -> e2e-analyze -> upload
+        # chain the Go suites use, so cell 17 produces an RCA bundle too.
         run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
           INVENTORY=$(tofu output -json inventory)
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          date -u +%Y-%m-%dT%H:%M:%SZ > "${ARTIFACT_DIR}/test-tofu.start"
           ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
-            "cd ~/e2e-tests && ./run-tofu-examples-e2e.sh"
+            "cd ~/e2e-tests && ./run-tofu-examples-e2e.sh" \
+            2>&1 | tee "${ARTIFACT_DIR}/test-tofu.log"
+          rc=${PIPESTATUS[0]}
+          if [ $rc -ne 0 ]; then echo "tofu: FAILED" >> "${ARTIFACT_DIR}/suite-status.txt"; fi
+          exit $rc
 
       - name: Run Reboot Resilience E2E
         if: steps.gate.outputs.selected == 'true' && matrix.extra == 'reboot-single'
         working-directory: mulga/scripts/tofu-cluster
-        # Driver runs from the runner (NOT via ssh-and-run) so it survives the
+        # reboot.test runs from the runner (NOT via ssh-and-run) so it survives the
         # systemctl-reboot-induced ssh disconnect mid-run.
         run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
           INVENTORY=$(tofu output -json inventory)
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
@@ -178,31 +222,85 @@ jobs:
           REBOOT_DIR="${RUNNER_TEMP}/reboot-e2e"
           mkdir -p "$REBOOT_DIR"
           tar xf "${RUNNER_TEMP}/spinifex-tests.tar" -C "$REBOOT_DIR"
-          chmod +x "$REBOOT_DIR/run-reboot-e2e.sh"
+          chmod +x "$REBOOT_DIR/_bin/reboot.test"
 
-          WAN_IP="$WAN_IP" SSH_KEY="$SSH_KEY" "$REBOOT_DIR/run-reboot-e2e.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i "$SSH_KEY")
+          CREDS=$(ssh "${SSH_OPTS[@]}" tf-user@${WAN_IP} 'cat ~/.aws/credentials')
+          AWS_AK=$(echo "$CREDS" | awk -F '= *' '/^aws_access_key_id/ {print $2; exit}')
+          AWS_SK=$(echo "$CREDS" | awk -F '= *' '/^aws_secret_access_key/ {print $2; exit}')
+          echo "::add-mask::${AWS_AK}"; echo "::add-mask::${AWS_SK}"
 
-      - name: Dump Cluster Journals on Failure
-        if: failure() && steps.gate.outputs.selected == 'true'
+          date -u +%Y-%m-%dT%H:%M:%SZ > "${ARTIFACT_DIR}/test-reboot.start"
+          GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+            SPINIFEX_WAN_IP="${WAN_IP}" SPINIFEX_SSH_KEY="${SSH_KEY}" \
+            SPINIFEX_AWS_ENDPOINT="https://${WAN_IP}:9999" \
+            SPINIFEX_AWS_ACCESS_KEY_ID="${AWS_AK}" SPINIFEX_AWS_SECRET_ACCESS_KEY="${AWS_SK}" \
+            ARTIFACT_DIR="${ARTIFACT_DIR}" \
+            "$REBOOT_DIR/_bin/reboot.test" -test.v -test.timeout=30m \
+            2>&1 | tee "${ARTIFACT_DIR}/test-reboot.log"
+          rc=${PIPESTATUS[0]}
+          if [ $rc -ne 0 ]; then echo "reboot: FAILED" >> "${ARTIFACT_DIR}/suite-status.txt"; fi
+          exit $rc
+
+      - name: Collect e2e artifacts
+        if: always() && steps.gate.outputs.selected == 'true'
         working-directory: mulga/scripts/tofu-cluster
         run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
           INVENTORY=$(tofu output -json inventory 2>/dev/null) || { echo "no inventory; skipping"; exit 0; }
-          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
-          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path'); SSH_KEY="${SSH_KEY/#\~/$HOME}"
           NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY")
+
+          ssh "${SSH_OPTS[@]}" tf-user@${NODE1_IP} "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+
           for i in $(seq 0 $((NODE_COUNT - 1))); do
-            IP=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
-            echo "===== node$((i+1)) ($IP) ====="
-            for svc in spinifex-nats spinifex-predastore spinifex-viperblock spinifex-daemon spinifex-awsgw spinifex-vpcd dhcpcd; do
-              echo "--- $svc ---"
-              ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" tf-user@${IP} \
-                "sudo journalctl -u $svc --no-pager -n 200 2>/dev/null || echo '(not found)'" 2>&1 || echo "(node unreachable)"
-            done
-            echo "--- DHCP client state ---"
-            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" tf-user@${IP} \
-              "systemctl is-enabled dhcpcd.service 2>/dev/null; systemctl is-active dhcpcd.service 2>/dev/null; sudo ss -lupn 'sport = 68' 2>/dev/null; pgrep -laf dhcpcd 2>/dev/null" 2>&1 || echo "(node unreachable)"
+            IP=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip"); idx=$((i + 1))
+            mkdir -p "${ARTIFACT_DIR}/vm-console-node${idx}"
+            ssh "${SSH_OPTS[@]}" tf-user@${IP} 'sudo sh -c "cd /run/spinifex 2>/dev/null && tar -cf - console-*.log 2>/dev/null"' \
+              | tar -C "${ARTIFACT_DIR}/vm-console-node${idx}" -xf - 2>/dev/null
+            ssh "${SSH_OPTS[@]}" tf-user@${IP} 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "60 min ago"' \
+              > "${ARTIFACT_DIR}/spinifex-journal-node${idx}.log" 2>/dev/null
           done
 
+          cat "${ARTIFACT_DIR}"/spinifex-journal-node*.log \
+            > "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || true
+          echo "collected: $(ls "${ARTIFACT_DIR}" 2>/dev/null | wc -l) entries"
+
+      - name: Install go-junit-report
+        if: always() && steps.gate.outputs.selected == 'true'
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Convert test logs to JUnit XML
+        if: always() && steps.gate.outputs.selected == 'true'
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log); SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Analyze e2e failures
+        if: always() && steps.gate.outputs.selected == 'true'
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "E2E failure analysis (cell ${{ matrix.cell }})"
+
+      - uses: actions/upload-artifact@v7
+        if: always() && steps.gate.outputs.selected == 'true'
+        with:
+          name: e2e-artifacts-cell-${{ matrix.cell }}-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
       - name: Teardown
         if: always()
         working-directory: mulga/scripts/tofu-cluster
@@ -217,8 +315,7 @@ jobs:
           tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1
           ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
 
-  # Bare-metal single-node suite (PXE-reimaged physical hardware). Doesn't fit
-  # the tofu matrix shape (different runner/provisioning/120m timeout), so it
+  # Bare-metal single-node suite. Doesn't fit the tofu matrix shape, so it
   # runs as a sibling job via the reusable e2e-baremetal workflow. Gated like a
   # matrix cell ("cell 20"): every weeknight cron, or dispatch with 20 selected
   # (blank cells = all). Tests `dev` on schedule to match GIT_BRANCH above.

.github/workflows/e2e-suites.yml

@@ -0,0 +1,799 @@
+# E2E (per-suite blocks) — EXPERIMENTAL, workflow_dispatch only.
+#
+# Restructures the looped e2e.yml into env-grouped job blocks so each suite
+# surfaces as its own node in the Actions graph (mulga-siv-283):
+#
+#   single_provision ─┬─ single_suite (matrix: single, cert, lb, eks, iam)
+#                     └─ single_teardown (if: always)
+#   multi_provision  ─┬─ multi_suite  (matrix: multinode)
+#                     └─ multi_teardown (if: always)
+#
+# Constraint: tofu state is runner-filesystem-local, so *_provision and
+# *_teardown must execute on the same self-hosted runner (one runner per label
+# today). Teardown reuses the persisted mulga/ tofu-cluster workspace and does
+# NOT re-checkout mulga (that would wipe local state). Suite-matrix legs reach
+# the already-provisioned VM over its WAN IP using the key path the provision
+# job emits; they share the runner home dir, so the key file persists.
+#
+# This file does NOT post the gating `e2e` commit status — it runs alongside
+# e2e.yml for validation. Once green across a few manual runs, fold the
+# triggers from e2e.yml here and retire the loop.
+
+name: E2E (suite blocks)
+
+on:
+  workflow_dispatch:
+    inputs:
+      single_suites:
+        description: "Single-node suites to run as matrix legs"
+        required: false
+        default: "single cert lb eks iam"
+      run_multinode:
+        description: "Also run the multinode env block"
+        type: boolean
+        required: false
+        default: true
+      test_run:
+        description: "Optional -test.run regex (empty = all tests in suite)"
+        required: false
+        default: ""
+
+permissions:
+  contents: read
+
+env:
+  GIT_BRANCH: ${{ github.ref_name }}
+  TEST_RUN: ${{ inputs.test_run || '' }}
+
+jobs:
+  # ---------------------------------------------------------------------------
+  # Single-node env
+  # ---------------------------------------------------------------------------
+  single_provision:
+    name: Single env · provision
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 40
+    outputs:
+      wan_ip: ${{ steps.vm.outputs.wan_ip }}
+      ssh_key: ${{ steps.vm.outputs.ssh_key }}
+      suites: ${{ steps.suites.outputs.list }}
+      matrix: ${{ steps.suites.outputs.matrix }}
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
+    steps:
+      - name: Reap Orphan Procs
+        run: |
+          set +e
+          UID_SELF=$(id -u)
+          WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in
+                "$WS"|"$WS"/*) echo "reap $sig pid=$pid cwd=$cwd"; kill "-$sig" "$pid" 2>/dev/null ;;
+              esac
+            done
+          }
+          reap TERM; sleep 3; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          true
+
+      - name: Clean Workspace
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - uses: actions/checkout@v7
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/tofu-cluster
+          path: mulga
+
+      - uses: actions/checkout@v7
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Re-enable Go toolchain auto-download
+        run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
+
+      - name: Compute suite matrix
+        id: suites
+        run: |
+          LIST="${{ inputs.single_suites || 'single cert lb eks iam' }}"
+          echo "list=${LIST}" >> "$GITHUB_OUTPUT"
+          JSON=$(printf '%s\n' $LIST | jq -R . | jq -cs .)
+          echo "matrix=${JSON}" >> "$GITHUB_OUTPUT"
+
+      - name: Reclaim docker build cache
+        run: |
+          docker container prune -f || true
+          docker image prune -f || true
+          docker buildx prune -af --filter "until=24h" || true
+          docker system df || true
+
+      - name: Build Install Artifacts
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Provision VM
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          export TF_VAR_git_branch="${GIT_BRANCH}"
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "VM provisioned"
+
+      - name: Bootstrap via Binary Install
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Resolve VM SSH details
+        id: vm
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          echo "wan_ip=${WAN_IP}" >> "$GITHUB_OUTPUT"
+          echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
+
+      - name: Build eks-node AMI (libguestfs, runner)
+        if: contains(steps.suites.outputs.list, 'eks')
+        working-directory: spinifex
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "build eks-node AMI (libguestfs — no host devices)"
+          sudo apt-get update -qq
+          sudo apt-get install -y --no-install-recommends libguestfs-tools qemu-utils
+          export LIBGUESTFS_BACKEND=direct
+          sudo chmod 0644 /boot/vmlinuz-* || true
+          group_open "make build-eks-node-image"
+          make build-eks-node-image
+          group_close
+          ok "raw: $(du -h /tmp/eks-node-image-build/eks-node-alpine.raw | cut -f1)"
+
+      - name: Import eks-node AMI on VM
+        if: contains(steps.suites.outputs.list, 'eks')
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          eval "$(grep -E '^(ALPINE_VERSION|AMI_NAME|SYSTEM_TAG)=' \
+            spinifex/scripts/images/eks-node/manifest.conf)"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          banner "import eks-node AMI to VM image store"
+          scp "${SSH_OPTS[@]}" \
+            /tmp/eks-node-image-build/eks-node-alpine.raw \
+            "$VM":/tmp/eks-node-alpine.raw
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "spx admin images import --file /tmp/eks-node-alpine.raw \
+               --distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
+               --boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}" >/dev/null
+          ok "eks-node AMI imported"
+
+      - name: Install kubectl on VM
+        if: contains(steps.suites.outputs.list, 'eks')
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          banner "install kubectl on VM (eks data-plane subtest)"
+          ssh "${SSH_OPTS[@]}" "$VM" 'bash -s' <<'EOF'
+          set -euo pipefail
+          KVER=v1.32.5
+          case "$(uname -m)" in
+            x86_64)  ARCH=amd64 ;;
+            aarch64) ARCH=arm64 ;;
+            *) echo "unsupported arch $(uname -m)" >&2; exit 1 ;;
+          esac
+          cd /tmp
+          curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl"
+          curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl.sha256"
+          echo "$(cat kubectl.sha256)  kubectl" | sha256sum -c -
+          sudo install -m 0755 kubectl /usr/local/bin/kubectl
+          kubectl version --client
+          EOF
+          ok "kubectl installed on VM"
+
+      - name: Build E2E test binaries on runner
+        working-directory: spinifex/tests/e2e
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          # teardown always builds: the run-scoped sweep binary runs in the
+          # teardown job, it is never a selectable suite leg.
+          SUITES_BUILD="${{ steps.suites.outputs.list }} teardown"
+          banner "build ${SUITES_BUILD// /, }"
+          group_open "go test -c"
+          make e2e-build SUITES_BUILD="${SUITES_BUILD}"
+          group_close
+          for f in _bin/*.test; do
+            ok "$(basename "$f") $(stat -c '%s' "$f" | numfmt --to=iec)"
+          done
+
+      - name: Ship test binaries to VM
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          ssh "${SSH_OPTS[@]}" "$VM" "rm -rf \$HOME/spinifex-e2e-bin && mkdir -p \$HOME/spinifex-e2e-bin"
+          tar -C spinifex/tests/e2e/_bin -cf - . | ssh "${SSH_OPTS[@]}" "$VM" "tar -xf - -C \$HOME/spinifex-e2e-bin"
+          ok "shipped binaries to $VM"
+
+      - name: Start packet capture on VM br-wan
+        run: |
+          set +e
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          scp "${SSH_OPTS[@]}" \
+            "${GITHUB_WORKSPACE}/spinifex/.github/scripts/start-net-capture.sh" \
+            "$VM":/tmp/start-net-capture.sh
+          ssh "${SSH_OPTS[@]}" "$VM" "bash /tmp/start-net-capture.sh" || true
+
+  single_suite:
+    name: Single env · suite
+    needs: single_provision
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 45
+    strategy:
+      fail-fast: false
+      matrix:
+        suite: ${{ fromJSON(needs.single_provision.outputs.matrix) }}
+    env:
+      WAN_IP: ${{ needs.single_provision.outputs.wan_ip }}
+      SSH_KEY: ${{ needs.single_provision.outputs.ssh_key }}
+      ARTIFACT_DIR: /tmp/spinifex-e2e-single-${{ github.run_id }}-${{ matrix.suite }}
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Install go-junit-report
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Run ${{ matrix.suite }}
+        id: e2e
+        run: |
+          set +e
+          SUITE="${{ matrix.suite }}"
+          VM_HOST="tf-user@${WAN_IP}"
+          TEST_RUN_FLAG=""
+          [ -n "${TEST_RUN}" ] && TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+          cleanup_remote() {
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" "$VM_HOST" \
+              "pkill -u tf-user -TERM -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; true" \
+              2>/dev/null || true
+          }
+          trap cleanup_remote EXIT INT TERM
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" "$VM_HOST" \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/spinifex-e2e-bin; \
+             R=\$'\033[0m' B=\$'\033[1m' C=\$'\033[36m' G=\$'\033[32m' RED=\$'\033[31m'; \
+             printf '\n%s%s━━ suite %s ━━%s\n' \"\$B\" \"\$C\" \"${SUITE}\" \"\$R\"; \
+             date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-${SUITE}.start'; \
+             GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+               SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+               ARTIFACT_DIR='${ARTIFACT_DIR}' \
+               ./${SUITE}.test -test.v -test.timeout=30m ${TEST_RUN_FLAG} \
+               2>&1 | tee '${ARTIFACT_DIR}/test-${SUITE}.log' \
+                    | sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'; \
+             exit \${PIPESTATUS[0]}"
+          echo "rc=$?" >> "$GITHUB_OUTPUT"
+          exit 0
+
+      - name: Collect suite artifacts from VM
+        if: always()
+        run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${SSH_KEY}")
+          VM=tf-user@${WAN_IP}
+          ssh "${SSH_OPTS[@]}" "$VM" "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+          true
+
+      - name: Convert test log to JUnit XML
+        if: always()
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log); SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          title: "E2E ${{ matrix.suite }} results"
+
+      - name: Analyze e2e failures
+        if: always()
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "E2E ${{ matrix.suite }} failure analysis"
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-single-${{ matrix.suite }}-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
+      - name: Fail job if suite failed
+        if: always()
+        run: |
+          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
+            echo "::error::suite ${{ matrix.suite }} failed (rc=${{ steps.e2e.outputs.rc }})"
+            exit 1
+          fi
+
+  single_teardown:
+    name: Single env · teardown
+    needs: [ single_provision, single_suite ]
+    if: always()
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 20
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-single-td-${{ github.run_id }}.log
+      ARTIFACT_DIR: /tmp/spinifex-e2e-single-diag-${{ github.run_id }}
+      WAN_IP: ${{ needs.single_provision.outputs.wan_ip }}
+      SSH_KEY: ${{ needs.single_provision.outputs.ssh_key }}
+    steps:
+      # Reuse the persisted mulga/ tofu-cluster workspace from single_provision.
+      # Re-checkout only spinifex (for ci-fmt + diag scripts); checking out mulga
+      # again would wipe the local tofu state and orphan the VM.
+      - uses: actions/checkout@v7
+        with:
+          path: spinifex
+
+      - name: Capture network/IPsec diagnostics on VM
+        if: ${{ needs.single_provision.result == 'success' && env.WAN_IP != '' }}
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}")
+          VM=tf-user@${WAN_IP}
+          scp "${SSH_OPTS[@]}" \
+            "${GITHUB_WORKSPACE}/spinifex/.github/scripts/capture-net-diag.sh" \
+            "$VM":/tmp/capture-net-diag.sh 2>/dev/null
+          ssh "${SSH_OPTS[@]}" "$VM" "bash /tmp/capture-net-diag.sh '${ARTIFACT_DIR}/net-diag'" 2>/dev/null
+          ssh "${SSH_OPTS[@]}" "$VM" 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "90 min ago"' \
+            > "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null
+          mkdir -p "${ARTIFACT_DIR}/lb-console"
+          ssh "${SSH_OPTS[@]}" "$VM" 'sudo sh -c "cd /run/spinifex 2>/dev/null && tar -cf - console-*.log 2>/dev/null"' \
+            | tar -C "${ARTIFACT_DIR}/lb-console" -xf - 2>/dev/null
+          true
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-single-diag-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: ignore
+          retention-days: 14
+
+      # Reclaim any resource leaked by a crashed/interrupted suite before the VM
+      # is destroyed. Scoped to this run's tag, best-effort, never fails the job.
+      - name: Sweep leaked run resources on VM
+        if: ${{ needs.single_provision.result == 'success' && env.WAN_IP != '' }}
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}")
+          VM=tf-user@${WAN_IP}
+          banner "sweep run-tagged resources (run ${{ github.run_id }})"
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "cd \$HOME/spinifex-e2e-bin 2>/dev/null && \
+             SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+             SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+             ./teardown.test -test.v -test.run TestSweepRunResources -test.timeout=10m" \
+            2>&1 || true
+          true
+
+      - name: Teardown VM
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          if [ -n "${WAN_IP}" ] && [ -n "${SSH_KEY}" ]; then
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}" \
+              tf-user@${WAN_IP} \
+              'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
+              2>/dev/null || true
+          fi
+          UID_SELF=$(id -u); WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in "$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;; esac
+            done
+          }
+          reap TERM; sleep 2; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
+          ok "teardown complete"
+
+  # ---------------------------------------------------------------------------
+  # Multinode env
+  # ---------------------------------------------------------------------------
+  multi_provision:
+    name: Multi env · provision
+    if: ${{ inputs.run_multinode }}
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 45
+    outputs:
+      node1_ip: ${{ steps.cluster.outputs.node1_ip }}
+      ssh_key: ${{ steps.cluster.outputs.ssh_key }}
+      nodes_csv: ${{ steps.cluster.outputs.nodes_csv }}
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
+    steps:
+      - name: Reap Orphan Procs
+        run: |
+          set +e
+          UID_SELF=$(id -u); WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in "$WS"|"$WS"/*) echo "reap $sig pid=$pid"; kill "-$sig" "$pid" 2>/dev/null ;; esac
+            done
+          }
+          reap TERM; sleep 3; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          true
+
+      - name: Clean Workspace
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - uses: actions/checkout@v7
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/tofu-cluster
+          path: mulga
+
+      - uses: actions/checkout@v7
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Re-enable Go toolchain auto-download
+        run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
+
+      - name: Reclaim docker build cache
+        run: |
+          docker container prune -f || true
+          docker image prune -f || true
+          docker buildx prune -af --filter "until=24h" || true
+          docker system df || true
+
+      - name: Build Install Artifacts
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Provision VMs
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          export TF_VAR_git_branch="${GIT_BRANCH}"
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "3-node cluster provisioned"
+
+      - name: Bootstrap via Binary Install
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Resolve cluster SSH details
+        id: cluster
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path'); SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          NODES_CSV=""
+          for i in $(seq 0 $((NODE_COUNT - 1))); do
+            ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
+            if [ -z "$NODES_CSV" ]; then NODES_CSV="$ip"; else NODES_CSV="$NODES_CSV,$ip"; fi
+          done
+          echo "node1_ip=${NODE1_IP}" >> "$GITHUB_OUTPUT"
+          echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
+          echo "nodes_csv=${NODES_CSV}" >> "$GITHUB_OUTPUT"
+
+      - name: Build multinode binary on runner
+        working-directory: spinifex/tests/e2e
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "build multinode, teardown"
+          group_open "go test -c"
+          make e2e-build SUITES_BUILD="multinode teardown"
+          group_close
+          ok "$(basename _bin/multinode.test) $(stat -c '%s' _bin/multinode.test | numfmt --to=iec)"
+
+      - name: Ship test binaries to node1
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.cluster.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.cluster.outputs.node1_ip }}
+          ssh "${SSH_OPTS[@]}" "$VM" "rm -rf \$HOME/spinifex-e2e-bin && mkdir -p \$HOME/spinifex-e2e-bin"
+          tar -C spinifex/tests/e2e/_bin -cf - . | ssh "${SSH_OPTS[@]}" "$VM" "tar -xf - -C \$HOME/spinifex-e2e-bin"
+          ok "shipped binaries to $VM"
+
+      - name: Start packet capture on all nodes
+        run: |
+          set +e
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.cluster.outputs.ssh_key }}")
+          IFS=',' read -ra NODE_LIST <<< "${{ steps.cluster.outputs.nodes_csv }}"
+          for ip in "${NODE_LIST[@]}"; do
+            [ -z "$ip" ] && continue
+            scp "${SSH_OPTS[@]}" \
+              "${GITHUB_WORKSPACE}/spinifex/.github/scripts/start-net-capture.sh" \
+              tf-user@"$ip":/tmp/start-net-capture.sh
+            ssh "${SSH_OPTS[@]}" tf-user@"$ip" "bash /tmp/start-net-capture.sh" || true
+          done
+
+  multi_suite:
+    name: Multi env · suite
+    needs: multi_provision
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 60
+    strategy:
+      fail-fast: false
+      matrix:
+        suite: [ multinode ]
+    env:
+      NODE1_IP: ${{ needs.multi_provision.outputs.node1_ip }}
+      SSH_KEY: ${{ needs.multi_provision.outputs.ssh_key }}
+      NODES_CSV: ${{ needs.multi_provision.outputs.nodes_csv }}
+      ARTIFACT_DIR: /tmp/spinifex-e2e-multi-${{ github.run_id }}-${{ matrix.suite }}
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Install go-junit-report
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Run ${{ matrix.suite }}
+        id: e2e
+        run: |
+          set +e
+          SUITE="${{ matrix.suite }}"
+          VM_HOST="tf-user@${NODE1_IP}"
+          TEST_RUN_FLAG=""
+          [ -n "${TEST_RUN}" ] && TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+          cleanup_remote() {
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" "$VM_HOST" \
+              "pkill -u tf-user -TERM -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; true" \
+              2>/dev/null || true
+          }
+          trap cleanup_remote EXIT INT TERM
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" "$VM_HOST" \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/spinifex-e2e-bin; \
+             printf '\n━━ suite %s ━━\n' \"${SUITE}\"; \
+             date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-${SUITE}.start'; \
+             GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
+               SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+               SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
+               SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
+               ARTIFACT_DIR='${ARTIFACT_DIR}' \
+               ./${SUITE}.test -test.v -test.timeout=30m ${TEST_RUN_FLAG} \
+               2>&1 | tee '${ARTIFACT_DIR}/test-${SUITE}.log' \
+                    | sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'; \
+             exit \${PIPESTATUS[0]}"
+          echo "rc=$?" >> "$GITHUB_OUTPUT"
+          exit 0
+
+      - name: Collect suite artifacts from node1
+        if: always()
+        run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${SSH_KEY}")
+          ssh "${SSH_OPTS[@]}" tf-user@${NODE1_IP} "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+          true
+
+      - name: Convert test log to JUnit XML
+        if: always()
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log); SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          title: "Multi-Node E2E ${{ matrix.suite }} results"
+
+      - name: Analyze e2e failures
+        if: always()
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "Multi-Node E2E ${{ matrix.suite }} failure analysis"
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-multi-${{ matrix.suite }}-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
+      - name: Fail job if suite failed
+        if: always()
+        run: |
+          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
+            echo "::error::suite ${{ matrix.suite }} failed (rc=${{ steps.e2e.outputs.rc }})"
+            exit 1
+          fi
+
+  multi_teardown:
+    name: Multi env · teardown
+    needs: [ multi_provision, multi_suite ]
+    if: always() && inputs.run_multinode
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 20
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-multi-td-${{ github.run_id }}.log
+      SSH_KEY: ${{ needs.multi_provision.outputs.ssh_key }}
+      NODES_CSV: ${{ needs.multi_provision.outputs.nodes_csv }}
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          path: spinifex
+
+      # Reclaim leaked run-tagged resources before the cluster is destroyed.
+      - name: Sweep leaked run resources on node1
+        if: ${{ needs.multi_provision.result == 'success' && needs.multi_provision.outputs.node1_ip != '' }}
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}")
+          VM=tf-user@${{ needs.multi_provision.outputs.node1_ip }}
+          banner "sweep run-tagged resources (run ${{ github.run_id }})"
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "cd \$HOME/spinifex-e2e-bin 2>/dev/null && \
+             SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
+             SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+             ./teardown.test -test.v -test.run TestSweepRunResources -test.timeout=10m" \
+            2>&1 || true
+          true
+
+      - name: Teardown cluster
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          if [ -n "${NODES_CSV}" ] && [ -n "${SSH_KEY}" ]; then
+            IFS=',' read -ra NODE_LIST <<< "${NODES_CSV}"
+            for ip in "${NODE_LIST[@]}"; do
+              ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}" \
+                tf-user@"$ip" \
+                'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
+                2>/dev/null || true
+            done
+          fi
+          UID_SELF=$(id -u); WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in "$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;; esac
+            done
+          }
+          reap TERM; sleep 2; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
+          ok "teardown complete"
+
+  e2e_suites_status:
+    name: E2E suite blocks · summary
+    if: always()
+    needs: [ single_suite, multi_suite ]
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - name: Summarize
+        run: |
+          echo "## E2E suite-block results" >> "$GITHUB_STEP_SUMMARY"
+          echo "single_suite: ${{ needs.single_suite.result }}" >> "$GITHUB_STEP_SUMMARY"
+          echo "multi_suite:  ${{ needs.multi_suite.result }}" >> "$GITHUB_STEP_SUMMARY"
+          case "${{ needs.single_suite.result }}" in success|skipped) ;; *) exit 1 ;; esac
+          case "${{ needs.multi_suite.result }}" in success|skipped) ;; *) exit 1 ;; esac

.github/workflows/e2e.yml

@@ -14,13 +14,10 @@ on:
       - .github/workflows/e2e.yml
   workflow_dispatch:
     inputs:
-      sha:
-        description: "PR head SHA to report status on"
-        required: false
       suites:
-        description: "Space-separated Make targets to run (e.g. 'e2e-single e2e-multinode e2e-cert e2e-lb e2e-eks')"
+        description: "Space-separated Make targets to run (e.g. 'e2e-single e2e-multinode e2e-cert e2e-lb e2e-eks e2e-iam')"
         required: false
-        default: "e2e-single e2e-multinode e2e-cert e2e-lb"
+        default: "e2e-single e2e-multinode e2e-cert e2e-lb e2e-iam"
       test_run:
         description: "Optional regex for -test.run (e.g. '^TestMultinodeIPSec$'). Empty = run all tests in suite."
         required: false
@@ -32,19 +29,55 @@ permissions:
 
 env:
   GIT_BRANCH: ${{ github.ref_name }}
-  SUITES: ${{ inputs.suites || 'e2e-single e2e-multinode e2e-cert e2e-lb' }}
+  SUITES: ${{ inputs.suites || 'e2e-single e2e-multinode e2e-cert e2e-lb e2e-iam' }}
   TEST_RUN: ${{ inputs.test_run || '' }}
 
 jobs:
+  # Shadow-mode suite selector. Computes which suites the diff *would* run and logs it to the job summary.
+  # Does NOT gate or override the real SUITES env yet — it runs alongside the full matrix so we
+  # can compare predicted vs actual before flipping it on.
+  select_suites:
+    name: Select Suites (shadow)
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: go.mod
+          cache: false
+      - name: Compute targeted suite selection
+        run: |
+          set -euo pipefail
+          BASE="${{ github.event.before }}"
+          HEAD="${{ github.sha }}"
+          # First push to a branch reports an all-zero before-SHA; fall back to
+          # the first parent so the diff is still meaningful.
+          if [ -z "$BASE" ] || [ "$BASE" = "0000000000000000000000000000000000000000" ]; then
+            BASE="${HEAD}~1"
+          fi
+          echo "## Targeted suite selection (shadow)" >> "$GITHUB_STEP_SUMMARY"
+          echo '```' >> "$GITHUB_STEP_SUMMARY"
+          echo "base=$BASE head=$HEAD" >> "$GITHUB_STEP_SUMMARY"
+          go run ./tests/e2e/select-suites/cmd/select-suites \
+            --manifest docs/service-interfaces.yaml \
+            --base-sha "$BASE" --head-sha "$HEAD" --repo-root . \
+            2>>"$GITHUB_STEP_SUMMARY" \
+            | tee -a "$GITHUB_STEP_SUMMARY"
+          echo '```' >> "$GITHUB_STEP_SUMMARY"
+          echo "Shadow only — actual run uses SUITES='${SUITES}'." >> "$GITHUB_STEP_SUMMARY"
+
   e2e_single:
     name: Single-Node E2E
     # Skip when the dispatch input requests no in-VM suites (e.g.
-    # suites='e2e-reboot' only). Push triggers and the default SUITES list
+    # suites='e2e-multinode' only). Push triggers and the default SUITES list
     # both include single/cert/lb so the inputs.suites null-check keeps the
     # job running on plain PR pushes.
-    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-lb') || contains(github.event.inputs.suites, 'e2e-eks')
-    runs-on: [ self-hosted, ci-single ]
-    timeout-minutes: 45
+    if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-lb') || contains(github.event.inputs.suites, 'e2e-eks') || contains(github.event.inputs.suites, 'e2e-iam')
+    runs-on: [self-hosted, ci-single]
+    timeout-minutes: 50
     env:
       TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
       ARTIFACT_DIR: /tmp/spinifex-e2e-artifacts-${{ github.run_id }}
@@ -71,14 +104,13 @@ jobs:
 
       - name: Clean Workspace
         run: |
-          # Detach any leaked mounts under the workspace before deleting. A
-          # killed image build can leave a `mount --bind /dev` under its build
-          # dir; if `rm -rf` traverses it as root it wipes the host's /dev
-          # (deletes /dev/null and wedges the box). umount deepest-first, then
-          # forbid rm from crossing any mount boundary as a hard backstop.
           awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
             | sort -r | xargs -r -I{} sudo umount -l -R {} || true
           sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
+          # ARTIFACT_DIR lives in /tmp, outside the workspace, so a re-run on
+          # the same self-hosted runner reuses it. Wipe it so a prior attempt's
+          # stale suite-status.txt / analysis dirs can't leak into this attempt.
+          rm -rf "${ARTIFACT_DIR}"
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -87,14 +119,14 @@ jobs:
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
         with:
           repository: mulgadc/mulga
           token: ${{ steps.app-token.outputs.token }}
           sparse-checkout: scripts/tofu-cluster
           path: mulga
 
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
         with:
           path: spinifex
 
@@ -106,10 +138,10 @@ jobs:
       - name: Re-enable Go toolchain auto-download
         run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
 
-      - name: Prune rootless docker cache
+      - name: Reclaim docker build cache
         run: |
           docker container prune -f || true
-          docker image prune -af || true
+          docker image prune -f || true
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
@@ -184,7 +216,6 @@ jobs:
         run: |
           set -euo pipefail
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          # Pull import args straight from the manifest so they never drift.
           eval "$(grep -E '^(ALPINE_VERSION|AMI_NAME|SYSTEM_TAG)=' \
             spinifex/scripts/images/eks-node/manifest.conf)"
           SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
@@ -238,7 +269,7 @@ jobs:
         run: |
           set -euo pipefail
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          SUITES_BUILD=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(reboot|multinode)$' | tr '\n' ' ')
+          SUITES_BUILD=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(multinode)$' | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_BUILD}" | tr -d ' ')" ]; then
             echo "no in-VM suites requested (SUITES=${SUITES}); nothing to build"
             exit 0
@@ -280,7 +311,7 @@ jobs:
         id: e2e
         run: |
           set +e
-          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(reboot|multinode)$' | tr '\n' ' ')
+          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(multinode)$' | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
             echo "no in-VM suites requested (SUITES=${SUITES}); skipping"
             echo "rc=0" >> "$GITHUB_OUTPUT"
@@ -436,16 +467,16 @@ jobs:
       - name: Fail job if any suite failed
         if: always()
         run: |
-          if [ "${{ steps.e2e.outputs.rc }}" != "0" ] || [ -s "${ARTIFACT_DIR}/suite-status.txt" ]; then
+          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
             echo "::error::E2E suites failed (rc=${{ steps.e2e.outputs.rc }})"
             cat "${ARTIFACT_DIR}/suite-status.txt" 2>/dev/null || true
             exit 1
           fi
 
-  e2e_real_multi:
-    name: Real Multi-Node E2E
+  e2e_multi:
+    name: Multi-Node E2E
     if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-multinode')
-    runs-on: [ self-hosted, ci-multi ]
+    runs-on: [self-hosted, ci-multi]
     timeout-minutes: 60
     env:
       TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
@@ -473,14 +504,13 @@ jobs:
 
       - name: Clean Workspace
         run: |
-          # Detach any leaked mounts under the workspace before deleting. A
-          # killed image build can leave a `mount --bind /dev` under its build
-          # dir; if `rm -rf` traverses it as root it wipes the host's /dev
-          # (deletes /dev/null and wedges the box). umount deepest-first, then
-          # forbid rm from crossing any mount boundary as a hard backstop.
           awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
             | sort -r | xargs -r -I{} sudo umount -l -R {} || true
           sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
+          # ARTIFACT_DIR lives in /tmp, outside the workspace, so a re-run on
+          # the same self-hosted runner reuses it. Wipe it so a prior attempt's
+          # stale suite-status.txt / analysis dirs can't leak into this attempt.
+          rm -rf "${ARTIFACT_DIR}"
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -489,14 +519,14 @@ jobs:
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
         with:
           repository: mulgadc/mulga
           token: ${{ steps.app-token.outputs.token }}
           sparse-checkout: scripts/tofu-cluster
           path: mulga
 
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
         with:
           path: spinifex
 
@@ -508,10 +538,10 @@ jobs:
       - name: Re-enable Go toolchain auto-download
         run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
 
-      - name: Prune rootless docker cache
+      - name: Reclaim docker build cache
         run: |
           docker container prune -f || true
-          docker image prune -af || true
+          docker image prune -f || true
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
@@ -568,12 +598,15 @@ jobs:
         run: |
           set -euo pipefail
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          # single + iam are single-node-only suites — their fixtures skip
+          # unless SPINIFEX_MODE=single, so building/running them on multinode
+          # is wasted work.
           # eks is excluded on multinode: EKS KV is a clustered (Raft) JetStream
           # stream and create-cluster needs an elected leader, but multinode
           # JetStream clustering isn't solid yet (mulga-omq/z2z/5yg). EKS is
           # validated single-node only for now; re-enable here once clustered
           # JetStream + multinode EKS are a deliberate target.
-          SUITES_BUILD=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(reboot|single|eks)$' | tr '\n' ' ')
+          SUITES_BUILD=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(single|eks|iam)$' | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_BUILD}" | tr -d ' ')" ]; then
             echo "no multinode-eligible suites requested (SUITES=${SUITES}); nothing to build"
             exit 0
@@ -656,7 +689,7 @@ jobs:
         id: e2e
         run: |
           set +e
-          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(reboot|single|eks)$' | tr '\n' ' ')
+          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(single|eks|iam)$' | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
             echo "no multinode-eligible suites requested (SUITES=${SUITES}); skipping"
             echo "rc=0" >> "$GITHUB_OUTPUT"
@@ -851,303 +884,33 @@ jobs:
       - name: Fail job if any suite failed
         if: always()
         run: |
-          if [ "${{ steps.e2e.outputs.rc }}" != "0" ] || [ -s "${ARTIFACT_DIR}/suite-status.txt" ]; then
+          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
             echo "::error::Multi-node E2E suites failed (rc=${{ steps.e2e.outputs.rc }})"
             cat "${ARTIFACT_DIR}/suite-status.txt" 2>/dev/null || true
             exit 1
           fi
 
-  e2e_reboot:
-    name: Reboot Resilience E2E
-    # Opt-in: only fires when the SUITES dispatch input lists e2e-reboot.
-    # The reboot test reboots the spinifex VM, so it must run on the runner
-    # (not shipped to the VM like the cert/lb/single suites). Runs in
-    # parallel with e2e_single — both target [self-hosted, ci-single]
-    # and GitHub Actions serialises on the underlying runner allocation,
-    # so no explicit needs: ordering is required.
-    if: contains(github.event.inputs.suites, 'e2e-reboot')
-    runs-on: [ self-hosted, ci-single ]
-    timeout-minutes: 60
-    env:
-      TOFU_LOG: /tmp/e2e-tofu-reboot-${{ github.run_id }}.log
-      ARTIFACT_DIR: /tmp/spinifex-e2e-reboot-artifacts-${{ github.run_id }}
-    steps:
-      - name: Reap Orphan Procs
-        run: |
-          set +e
-          UID_SELF=$(id -u)
-          WS="${GITHUB_WORKSPACE}"
-          reap() {
-            local sig="$1"
-            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
-              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
-              case "$cwd" in
-                "$WS"|"$WS"/*) echo "reap $sig pid=$pid cwd=$cwd"; kill "-$sig" "$pid" 2>/dev/null ;;
-              esac
-            done
-          }
-          reap TERM
-          sleep 3
-          reap KILL
-          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
-          true
-
-      - name: Clean Workspace
-        run: |
-          # Detach any leaked mounts under the workspace before deleting. A
-          # killed image build can leave a `mount --bind /dev` under its build
-          # dir; if `rm -rf` traverses it as root it wipes the host's /dev
-          # (deletes /dev/null and wedges the box). umount deepest-first, then
-          # forbid rm from crossing any mount boundary as a hard backstop.
-          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
-            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
-          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
-
-      - uses: actions/create-github-app-token@v3
-        id: app-token
-        with:
-          client-id: ${{ secrets.APP_ID }}
-          private-key: ${{ secrets.APP_PRIVATE_KEY }}
-          repositories: mulga
-
-      - uses: actions/checkout@v6
-        with:
-          repository: mulgadc/mulga
-          token: ${{ steps.app-token.outputs.token }}
-          sparse-checkout: scripts/tofu-cluster
-          path: mulga
-
-      - uses: actions/checkout@v6
-        with:
-          path: spinifex
-
-      - uses: actions/setup-go@v6
-        with:
-          go-version-file: spinifex/go.mod
-          cache: false
-
-      - name: Re-enable Go toolchain auto-download
-        run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
-
-      - name: Prune rootless docker cache
-        run: |
-          docker container prune -f || true
-          docker image prune -af || true
-          docker buildx prune -af --filter "until=24h" || true
-          docker system df || true
-
-      - name: Build Install Artifacts
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          ./build-install-artifacts.sh "${GIT_BRANCH}" \
-            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
-            "${RUNNER_TEMP}/setup.sh" \
-            "${RUNNER_TEMP}/spinifex-tests.tar"
-
-      - name: Provision VM
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          set -euo pipefail
-          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          export TF_VAR_git_branch="${GIT_BRANCH}"
-          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
-          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
-          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
-          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
-          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
-          ok "VM provisioned"
-
-      - name: Bootstrap via Binary Install
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
-          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
-            "${RUNNER_TEMP}/setup.sh" \
-            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
-            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
-
-      - name: Resolve VM SSH details
-        id: vm
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          INVENTORY=$(tofu output -json inventory)
-          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
-          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
-          SSH_KEY="${SSH_KEY/#\~/$HOME}"
-          echo "wan_ip=${WAN_IP}" >> "$GITHUB_OUTPUT"
-          echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
-
-      - name: Build reboot.test on runner
-        working-directory: spinifex/tests/e2e
-        run: |
-          set -euo pipefail
-          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          banner "build reboot"
-          group_open "go test -c"
-          make e2e-build SUITES_BUILD="reboot"
-          group_close
-          ok "$(basename _bin/reboot.test) $(stat -c '%s' _bin/reboot.test | numfmt --to=iec)"
-
-      - name: Fetch AWS credentials from VM
-        id: creds
-        run: |
-          set -euo pipefail
-          SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
-                    -i "${{ steps.vm.outputs.ssh_key }}")
-          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
-          CREDS=$(ssh "${SSH_OPTS[@]}" "$VM" 'cat ~/.aws/credentials')
-          AWS_AK=$(echo "$CREDS" | awk -F '= *' '/^aws_access_key_id/ {print $2; exit}')
-          AWS_SK=$(echo "$CREDS" | awk -F '= *' '/^aws_secret_access_key/ {print $2; exit}')
-          if [ -z "$AWS_AK" ] || [ -z "$AWS_SK" ]; then
-            echo "::error::Failed to extract spinifex AWS credentials from ${{ steps.vm.outputs.wan_ip }}"
-            exit 1
-          fi
-          echo "::add-mask::${AWS_AK}"
-          echo "::add-mask::${AWS_SK}"
-          echo "ak=${AWS_AK}" >> "$GITHUB_OUTPUT"
-          echo "sk=${AWS_SK}" >> "$GITHUB_OUTPUT"
-
-      - name: Run reboot suite on runner
-        id: e2e
-        run: |
-          set +e
-          mkdir -p "${ARTIFACT_DIR}"
-          date -u +%Y-%m-%dT%H:%M:%SZ > "${ARTIFACT_DIR}/test-reboot.start"
-          R=$'\033[0m'; B=$'\033[1m'; C=$'\033[36m'
-          printf '\n%s%s━━ suite reboot ━━%s\n' "$B" "$C" "$R"
-
-          if [ -n "${TEST_RUN}" ]; then
-            TEST_RUN_FLAG="-test.run=${TEST_RUN}"
-          else
-            TEST_RUN_FLAG=""
-          fi
-          GITHUB_ACTIONS=true FORCE_COLOR=1 \
-            SPINIFEX_E2E=1 SPINIFEX_MODE=single \
-            SPINIFEX_WAN_IP="${{ steps.vm.outputs.wan_ip }}" \
-            SPINIFEX_SSH_KEY="${{ steps.vm.outputs.ssh_key }}" \
-            SPINIFEX_AWS_ENDPOINT="https://${{ steps.vm.outputs.wan_ip }}:9999" \
-            SPINIFEX_AWS_ACCESS_KEY_ID="${{ steps.creds.outputs.ak }}" \
-            SPINIFEX_AWS_SECRET_ACCESS_KEY="${{ steps.creds.outputs.sk }}" \
-            ARTIFACT_DIR="${ARTIFACT_DIR}" \
-            ./spinifex/tests/e2e/_bin/reboot.test -test.v -test.timeout=30m ${TEST_RUN_FLAG} \
-            2>&1 | tee "${ARTIFACT_DIR}/test-reboot.log" \
-                 | sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'
-          rc=${PIPESTATUS[0]}
-          echo "rc=${rc}" >> "$GITHUB_OUTPUT"
-          exit 0
-
-      - name: Collect VM journals
-        if: always()
-        run: |
-          set +e
-          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          mkdir -p "${ARTIFACT_DIR}"
-          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 \
-                    -i "${{ steps.vm.outputs.ssh_key }}")
-          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
-          ssh "${SSH_OPTS[@]}" "$VM" 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "60 min ago"' \
-            > "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || true
-          ok "journal: $(wc -l < "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || echo 0) lines"
-
-      - name: Install go-junit-report
-        if: always()
-        run: |
-          go install github.com/jstemmer/go-junit-report/v2@latest
-          echo "$HOME/go/bin" >> "$GITHUB_PATH"
-
-      - name: Convert test log to JUnit XML
-        if: always()
-        run: |
-          shopt -s nullglob
-          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
-            BASE=$(basename "$LOG" .log)
-            SUITE=${BASE#test-}
-            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
-          done
-
-      - name: Publish test summary
-        if: always()
-        uses: ./spinifex/.github/actions/junit-report
-        with:
-          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
-          title: "E2E reboot results"
-
-      - name: Analyze e2e failures
-        if: always()
-        uses: ./spinifex/.github/actions/e2e-analyze
-        with:
-          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
-          log-dir: ${{ env.ARTIFACT_DIR }}
-          title: "Reboot E2E failure analysis"
-
-      - uses: actions/upload-artifact@v7
-        if: always()
-        with:
-          name: e2e-reboot-artifacts-${{ github.run_id }}
-          path: ${{ env.ARTIFACT_DIR }}
-          if-no-files-found: warn
-          retention-days: 14
-
-      - name: Teardown
-        if: always()
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          set +e
-          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          UID_SELF=$(id -u)
-          WS="${GITHUB_WORKSPACE}"
-          reap() {
-            local sig="$1"
-            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
-              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
-              case "$cwd" in
-                "$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;;
-              esac
-            done
-          }
-          reap TERM
-          sleep 2
-          reap KILL
-          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
-          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
-          quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
-          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
-          ok "teardown complete"
-
-      - name: Fail job if reboot suite failed
-        if: always()
-        run: |
-          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
-            echo "::error::Reboot suite failed (rc=${{ steps.e2e.outputs.rc }})"
-            exit 1
-          fi
-
   e2e_status:
     name: Report E2E Status
     if: always()
-    needs: [ e2e_single, e2e_real_multi, e2e_reboot ]
+    needs: [e2e_single, e2e_multi]
     runs-on: ubuntu-latest
     timeout-minutes: 5
     steps:
       - name: Report Status to PR
         env:
           GH_TOKEN: ${{ github.token }}
-          REPORT_SHA: ${{ inputs.sha || github.sha }}
+          REPORT_SHA: ${{ github.sha }}
         run: |
-          if [[ -z "${REPORT_SHA}" ]]; then
-            echo "::error::no SHA available (inputs.sha and github.sha both empty)"
-            exit 1
-          fi
-          single_ok=true; multi_ok=true; reboot_ok=true
+          single_ok=true; multi_ok=true
           case "${{ needs.e2e_single.result }}" in success|skipped) ;; *) single_ok=false ;; esac
-          case "${{ needs.e2e_real_multi.result }}" in success|skipped) ;; *) multi_ok=false ;; esac
-          case "${{ needs.e2e_reboot.result }}" in success|skipped) ;; *) reboot_ok=false ;; esac
-          if $single_ok && $multi_ok && $reboot_ok; then
+          case "${{ needs.e2e_multi.result }}" in success|skipped) ;; *) multi_ok=false ;; esac
+          if $single_ok && $multi_ok; then
             STATE="success"
             DESC="E2E suites passed"
           else
             STATE="failure"
-            DESC="E2E failed: single=${{ needs.e2e_single.result }} multi=${{ needs.e2e_real_multi.result }} reboot=${{ needs.e2e_reboot.result }}"
+            DESC="E2E failed: single=${{ needs.e2e_single.result }} multi=${{ needs.e2e_multi.result }}"
           fi
           gh api "repos/${{ github.repository }}/statuses/${REPORT_SHA}" \
             -f state="$STATE" \

.github/workflows/publish-eks-ami.yml

@@ -26,23 +26,23 @@ concurrency:
 jobs:
   publish:
     name: Build + publish eks-node AMI
-    runs-on: ubuntu-24.04
+    runs-on: [ self-hosted, ci-single ]
 
     steps:
       - name: Checkout Spinifex
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           path: spinifex
 
       - name: Checkout Viperblock
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
           ref: dev
           path: viperblock
 
       - name: Checkout Predastore
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
           ref: dev
@@ -51,8 +51,12 @@ jobs:
       - name: Setup Go
         uses: actions/setup-go@v6
         with:
-          cache-dependency-path: "spinifex/go.sum"
           go-version-file: "spinifex/go.mod"
+          # Self-hosted runner: the Go module + build cache already persists on
+          # local disk between runs. setup-go's cache-save ("Post Setup Go")
+          # otherwise re-uploads hundreds of MB to the Actions cache backend,
+          # taking longer than the build itself.
+          cache: false
 
       - name: Setup Go Workspace
         run: go work init ./spinifex ./viperblock ./predastore

.github/workflows/release.yml

@@ -9,21 +9,21 @@ permissions:
 
 jobs:
   distro-amd64:
-    runs-on: ubuntu-24.04
+    runs-on: ubuntu-26.04
     steps:
       - name: Checkout Spinifex
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           path: spinifex
 
       - name: Checkout Predastore
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
           path: predastore
 
       - name: Checkout Viperblock
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
           path: viperblock
@@ -34,7 +34,7 @@ jobs:
       - name: Set up Go
         uses: actions/setup-go@v6
         with:
-          go-version: '1.26.4'
+          go-version: "1.26.4"
 
       - name: Build microVM artifacts (lb-agent + kernel + initramfs)
         run: cd spinifex && make build-microvm-image
@@ -54,21 +54,21 @@ jobs:
             spinifex/scripts/setup.sh
 
   distro-arm64:
-    runs-on: ubuntu-24.04-arm
+    runs-on: ubuntu-26.04-arm
     steps:
       - name: Checkout Spinifex
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           path: spinifex
 
       - name: Checkout Predastore
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
           path: predastore
 
       - name: Checkout Viperblock
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
           path: viperblock
@@ -91,7 +91,7 @@ jobs:
 
   release:
     needs: [ distro-amd64, distro-arm64 ]
-    runs-on: ubuntu-24.04
+    runs-on: ubuntu-26.04
     steps:
       - name: Download amd64 Artifacts
         uses: actions/download-artifact@v8
@@ -132,12 +132,6 @@ jobs:
     steps:
       - name: Clean Workspace
         run: |
-          # This runner (ci-single) is shared with the iso job, which mounts a
-          # kernel-global devtmpfs at <workspace>/mulga/scripts/iso-builder/.cache/rootfs/dev.
-          # If that job is killed its EXIT-trap unmount never runs and the bind
-          # leaks; a plain rm -rf then traverses it and deletes the host's /dev
-          # (wipes /dev/null, wedges the box). Detach any leaked mounts under the
-          # workspace first, then forbid rm from crossing a mount boundary.
           awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
             | sort -r | xargs -r -I{} sudo umount -l -R {} || true
           sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
@@ -149,7 +143,7 @@ jobs:
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
         with:
           repository: mulgadc/mulga
           token: ${{ steps.app-token.outputs.token }}
@@ -186,10 +180,6 @@ jobs:
           tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
           ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
 
-  # Bare-metal single-node suite on the real ISO/squashfs + PXE-boot path —
-  # the only release check that exercises hardware install. Runs in parallel
-  # with `verify` (both need: release). Promotion and ISO upload both hard-gate
-  # on this passing.
   baremetal:
     needs: release
     permissions:
@@ -198,11 +188,9 @@ jobs:
     uses: ./.github/workflows/e2e-baremetal.yml
     secrets: inherit
 
-  # Promote the draft only after BOTH the tofu install (`verify`) and the
-  # bare-metal hardware boot (`baremetal`) pass — runs only if both succeed.
   promote:
     needs: [ verify, baremetal ]
-    runs-on: ubuntu-24.04
+    runs-on: ubuntu-26.04
     env:
       TAG: ${{ github.ref_name }}
       REPO: ${{ github.repository }}
@@ -223,7 +211,7 @@ jobs:
   mark-failed:
     needs: [ verify, baremetal ]
     if: failure()
-    runs-on: ubuntu-24.04
+    runs-on: ubuntu-26.04
     env:
       TAG: ${{ github.ref_name }}
       REPO: ${{ github.repository }}
@@ -248,8 +236,6 @@ jobs:
       - name: Clean Workspace
         run: |
           ROOTFS="${GITHUB_WORKSPACE}/mulga/scripts/iso-builder/.cache/rootfs"
-          # Mirror make clean's umount sequence unconditionally — lazy unmount
-          # detaches immediately even if the mount is busy, so rm -rf is safe.
           sudo umount -l -R "${ROOTFS}" 2>/dev/null || true
           sudo umount -l "${ROOTFS}/dev/pts" 2>/dev/null || true
           sudo umount -l "${ROOTFS}/dev" 2>/dev/null || true
@@ -265,25 +251,25 @@ jobs:
           repositories: mulga
 
       - name: Checkout Spinifex
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           path: spinifex
           ref: ${{ github.ref }}
 
       - name: Checkout Predastore
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
           path: predastore
 
       - name: Checkout Viperblock
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
           path: viperblock
 
       - name: Checkout Mulga iso-builder
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           repository: mulgadc/mulga
           token: ${{ steps.app-token.outputs.token }}
@@ -335,9 +321,6 @@ jobs:
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
-          # Gate the R2 upload on BOTH the tofu install (`verify`) and the
-          # real-hardware boot (`baremetal`, a reusable workflow whose job shows
-          # as "baremetal / ..."). Upload only if both conclude success.
           echo "Waiting for verify + baremetal jobs to complete..."
           while true; do
             JOBS=$(gh api \

.github/workflows/spinifex.yml

@@ -23,24 +23,24 @@ permissions:
 jobs:
   build_and_test:
     name: Build and Test
-    runs-on: ubuntu-24.04
+    runs-on: ubuntu-26.04
 
     steps:
       - name: Checkout Spinifex
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           path: spinifex
           fetch-depth: 2
 
       - name: Checkout Viperblock
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
           ref: dev
           path: viperblock
 
       - name: Checkout Predastore
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
           ref: dev
@@ -58,6 +58,9 @@ jobs:
       - name: Build Spinifex
         run: make -C spinifex build
 
+      - name: Manifest check + drift lint
+        run: make -C spinifex manifest-check manifest-lint
+
       - name: Run Tests with Coverage
         run: make -C spinifex test-cover
 
@@ -69,23 +72,23 @@ jobs:
 
   race_detection:
     name: Race Detection
-    runs-on: ubuntu-24.04
+    runs-on: ubuntu-26.04
 
     steps:
       - name: Checkout Spinifex
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           path: spinifex
 
       - name: Checkout Viperblock
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
           ref: dev
           path: viperblock
 
       - name: Checkout Predastore
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
           ref: dev
@@ -105,23 +108,23 @@ jobs:
 
   lint_and_security:
     name: Lint and Security
-    runs-on: ubuntu-24.04
+    runs-on: ubuntu-26.04
 
     steps:
       - name: Checkout Spinifex
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           path: spinifex
 
       - name: Checkout Viperblock
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           repository: mulgadc/viperblock
           ref: dev
           path: viperblock
 
       - name: Checkout Predastore
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           repository: mulgadc/predastore
           ref: dev

.github/workflows/ui.yml

@@ -23,14 +23,14 @@ permissions:
 jobs:
   ui_build_and_test:
     name: UI Build and Test
-    runs-on: ubuntu-24.04
+    runs-on: ubuntu-26.04
     defaults:
       run:
         working-directory: spinifex/services/spinifexui/frontend
 
     steps:
       - name: Checkout Code
-        uses: actions/checkout@v6
+        uses: actions/checkout@v7
         with:
           sparse-checkout: spinifex/services/spinifexui/frontend
 

.gitignore

@@ -37,13 +37,3 @@ Thumbs.db
 
 # Test reports
 *-report.txt
-
-# Internal docs (moved to private mulga repo)
-docs/INTERNAL.md
-docs/PRIVATE.md
-docs/DEVNOTES.md
-docs/PLAN.md
-docs/TODO.md
-docs/development/
-docs/internal/
-scripts/iac/aws/spinifex-test.pem

INSTALL.md

@@ -1,3 +0,0 @@
-# Spinifex Installation
-
-For documentation on using and installing Spinifex see [https://docs.mulgadc.com](https://docs.mulgadc.com/)

Makefile

@@ -6,17 +6,12 @@ ARCH := $(shell uname -m)
 ifeq ($(ARCH),x86_64)
   GO_ARCH := amd64
   AWS_ARCH := x86_64
-  # On x86, we can run ARM VMs via emulation
-  QEMU_PACKAGES := qemu-system-x86 qemu-system-arm
 else ifeq ($(ARCH),aarch64)
   GO_ARCH := arm64
   AWS_ARCH := aarch64
-  # On ARM, we can run x86 VMs via emulation
-  QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
 else ifeq ($(ARCH),arm64)
   GO_ARCH := arm64
   AWS_ARCH := aarch64
-  QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
 else
   $(error Unsupported architecture: $(ARCH). Only x86_64 and aarch64/arm64 are supported.)
 endif
@@ -115,14 +110,10 @@ install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
 
-go_run:
-	@echo -e "\n....Running $(GO_PROJECT_NAME)...."
-	$(GOPATH)/bin/$(GO_PROJECT_NAME)
-
 # Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
 # Use this before committing to catch CI failures locally.
 preflight:
-	@$(MAKE) --no-print-directory QUIET=1 manifest-check lint govulncheck test-cover diff-coverage test-race test-harness
+	@$(MAKE) --no-print-directory QUIET=1 manifest-check manifest-lint lint govulncheck test-cover diff-coverage test-race test-harness
 	@echo -e "\n ✅ Preflight passed — safe to commit."
 
 # E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
@@ -139,6 +130,17 @@ manifest-check:
 	@echo -e "\n....Checking service-interfaces.yaml...."
 	@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml
 
+# Drift guards (Bead 5): direct-create fixture lint + NATS subject lint,
+# ratcheted against tests/e2e/manifest-lint/baseline.txt. Fails only on NEW
+# drift beyond the baseline.
+manifest-lint:
+	@echo -e "\n....Linting manifest drift (fixtures + subjects)...."
+	@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root .
+
+# Accept current drift into the baseline. Run after an intentional change.
+manifest-lint-update:
+	@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root . -update
+
 # Run unit tests
 test:
 	@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
@@ -168,13 +170,8 @@ diff-coverage: test-cover
 
 bench:
 	@echo -e "\n....Running benchmarks for $(GO_PROJECT_NAME)...."
-	$(MAKE) easyjson
 	LOG_IGNORE=1 go test -benchmem -run=. -bench=. ./...
 
-run:
-	$(MAKE) go_build
-	$(MAKE) go_run
-
 # Fast iteration: build + install binary + restart all services.
 # Microvm artifacts are reinstalled when they already exist on disk — the rule's
 # input timestamps drive a rebuild only if anything actually changed. On a fresh
@@ -201,11 +198,10 @@ clean:
 
 install-system:
 	@echo -e "\n....Installing system dependencies for $(ARCH)...."
-	@echo "QEMU packages: $(QEMU_PACKAGES)"
 	sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
 		-o Dpkg::Options::="--force-confdef" \
 		-o Dpkg::Options::="--force-confold" \
-		nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
+		nbdkit nbdkit-plugin-dev pkg-config qemu-system-x86 qemu-system-arm qemu-utils qemu-kvm \
 		ovmf qemu-efi-aarch64 \
 		libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
 		iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
@@ -234,19 +230,15 @@ install-aws:
 quickinstall: install-system install-go install-aws
 	@echo -e "\n✅ Quickinstall complete for $(ARCH)."
 	@echo "   Please ensure /usr/local/go/bin is in your PATH."
-	@echo "   Installed: Go ($(GO_ARCH)), AWS CLI ($(AWS_ARCH)), QEMU ($(QEMU_PACKAGES))"
 
-# Lint all Go code via golangci-lint (replaces check-format, vet, gosec, staticcheck)
 lint:
 	@echo "Running golangci-lint..."
 	$(_Q)golangci-lint run ./...
 	@echo "  golangci-lint ok"
 
-# Auto-fix all linter issues that have fixers
 fix:
 	golangci-lint run --fix ./...
 
-# Govulncheck — dependency vulnerability scanning (not covered by golangci-lint)
 govulncheck:
 	@echo "Running govulncheck..."
 	$(_Q)go tool govulncheck ./...
@@ -295,65 +287,7 @@ distro-arm64:
 distro-clean:
 	rm -rf dist/
 
-# Ansible dev lifecycle (experimental, parallel to dev-*.sh / reset-dev-env.sh).
-# See scripts/ansible/README.md and docs/development/improvements/ansible-dev-lifecycle.md.
-#
-# Variable overrides: pass EXTRA_VARS="key=val key2=val2" to forward as
-# `--extra-vars` to ansible-playbook. Plain `-e` on the make command line is
-# make's own flag and does NOT reach ansible.
-#   make ansible-dev-install EXTRA_VARS="spinifex_external_pool_start=192.168.1.90 spinifex_external_pool_end=192.168.1.99"
-EXTRA_VARS ?=
-_ANSIBLE_EXTRA = $(if $(strip $(EXTRA_VARS)),--extra-vars "$(EXTRA_VARS)",)
-
-ansible-dev-preflight:
-	cd scripts/ansible && ansible-playbook playbooks/dev-preflight.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-teardown:
-	cd scripts/ansible && ansible-playbook playbooks/dev-teardown.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-install:
-	cd scripts/ansible && ansible-playbook playbooks/dev-install.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-reset:
-	cd scripts/ansible && ansible-playbook playbooks/dev-reset.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-deploy:
-	cd scripts/ansible && ansible-playbook playbooks/dev-deploy.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-status:
-	cd scripts/ansible && ansible-playbook playbooks/dev-status.yml
-
-ansible-dev-logs:
-	cd scripts/ansible && ansible-playbook playbooks/dev-logs.yml
-
-# Snapshot / restore require an explicit -e snapshot_name=<name>. The
-# wrapper passes ANSIBLE_EXTRA through so devs can run
-# `make ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=before-merge'`.
-ansible-dev-snapshot:
-	cd scripts/ansible && ansible-playbook playbooks/dev-snapshot.yml $(ANSIBLE_EXTRA)
-
-ansible-dev-restore:
-	cd scripts/ansible && ansible-playbook playbooks/dev-restore.yml $(ANSIBLE_EXTRA)
-
-ansible-dev-version:
-	cd scripts/ansible && ansible-playbook playbooks/dev-version.yml
-
-ansible-dev-vpc:
-	cd scripts/ansible && ansible-playbook playbooks/dev-vpc.yml $(ANSIBLE_EXTRA)
-
-# Multi-node cluster bootstrap against a tofu env (env1, env2, bryn, etc).
-# Inventory is generated from scripts/tofu-cluster/envs/<env>.tfvars.
-#   make ansible-cluster-bootstrap ENV=env1 POOL=192.168.1.150-192.168.1.159
-ENV ?=
-POOL ?=
-ansible-cluster-bootstrap:
-	@test -n "$(ENV)" || { echo "ENV=<env> required (e.g. env1, env2)"; exit 64; }
-	cd scripts/ansible && CLUSTER_ENV=$(ENV) ansible-playbook -i inventory/tofu.py playbooks/cluster-bootstrap.yml \
-		-e cluster_env=$(ENV) \
-		$(if $(POOL),-e cluster_external_pool=$(POOL),) \
-		$(_ANSIBLE_EXTRA)
-
-.PHONY: build build-ui build-installer build-lb-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race diff-coverage bench run test-actions test-harness manifest-check diff-coverage bench run \
+.PHONY: build build-ui build-installer build-lb-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-harness manifest-check manifest-lint manifest-lint-update \
 	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
 	lint fix govulncheck \

build/systemd/spinifex-awsgw.service

@@ -6,6 +6,7 @@ Wants=spinifex-nats.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-gw
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh

build/systemd/spinifex-daemon.service

@@ -2,66 +2,29 @@
 Description=Spinifex Daemon (VM Orchestration)
 PartOf=spinifex.target
 After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.service ovn-controller.service openvswitch-ipsec.service
-# Wants=openvswitch-ipsec.service so systemd brings the IPsec userspace
-# daemon up before spinifex flips other_config:ipsec_encapsulation=true.
-# Without that daemon, OVS holds the cert pointers unused and strongSwan
-# never gets IKE config — tunnel traffic is encrypted with no SAs and
-# silently dropped. Wants= (not Requires=) keeps daemon bootable on
-# single-node / lab hosts where the package is intentionally absent.
+# Wants (not Requires): stay bootable when openvswitch-ipsec is absent
 Wants=openvswitch-ipsec.service
-# No Wants=spinifex-nats.service: Wants= reactivates a stopped unit during
-# `systemctl restart spinifex-daemon`, which would mask the standalone-startup
-# path the Tier 1 DDIL Scenario B exercises. spinifex.target Wants= covers
-# cold boot.
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-daemon
 Group=spinifex
-# DDIL Tier 1 (siv-17): no ExecStartPre wait-for-nats. The daemon starts in
-# standalone mode and reconnects to NATS in the background — process-exiting
-# on a missing NATS would kill the local VM management plane.
-#
-# Opt-in strict-startup (§1d-strict, siv-44): set SPINIFEX_REQUIRE_NATS=1 to
-# restore pre-DDIL fail-fast UX. First NATS connect uses a ~30s bounded retry
-# and the daemon aborts on timeout. Intended for dev/test/single-node deploys
-# where silent NATS misconfig is more confusing than helpful. Do NOT set this
-# in the shipped unit — it re-introduces the SPOF that Tier 1 removed.
 ExecStart=/usr/local/bin/spx service spinifex start
 Restart=on-failure
 RestartSec=5
-# DDIL Tier 1: KillMode=process so systemd only signals the
-# daemon's main PID, not the cgroup. QEMU children stay alive across
-# `systemctl restart spinifex-daemon` and the new daemon reattaches to them
-# via the persisted local state file + pidfile + QMP socket. With the
-# default control-group mode, restart would terminate every guest VM —
-# defeating Scenario B's "daemon restart preserves running instances".
+# KillMode=process: guest QEMU survives daemon restart; the new daemon reattaches (DDIL)
 KillMode=process
-# 15s upper bound on the SIGTERM grace window. The new SIGTERM handler
-# skips VM teardown (it now just persists state and exits), so 15s is
-# plenty of slack for the cluster-manager + NATS unsubscribe path.
 TimeoutStopSec=15
+# RG-4: infra tier — reaped only after every guest
 OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
 Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex/
 Environment=SPINIFEX_WAL_DIR=/var/lib/spinifex/spinifex/
-# DDIL Tier 1: QEMU pidfiles, QMP sockets, and the
-# shared qemu.log all resolve via utils.pidPath() — which falls back
-# to os.TempDir() (=/tmp) when $HOME/spinifex doesn't exist. With
-# PrivateTmp=yes, each daemon start gets a fresh /tmp namespace, so
-# pidfiles written by QEMU (running in the OLD daemon's namespace)
-# are invisible to the new daemon. The /local/instances handler then
-# reports PID=0 and Scenario B fails its "live QEMU PID" assertion.
-# Pin XDG_RUNTIME_DIR to the host-wide tmpfs at /run/spinifex (created
-# by tmpfiles.d, already in ReadWritePaths) so pidPath() returns a
-# stable, namespace-independent path that both daemon incarnations and
-# QEMU children resolve identically.
+# Pin to host tmpfs: QEMU pidfiles/QMP sockets must survive PrivateTmp namespace churn (DDIL)
 Environment=XDG_RUNTIME_DIR=/run/spinifex
 
-# Filesystem access — NoNewPrivileges omitted (daemon uses sudo for ip, ovs-vsctl, dhcpcd)
-# CAP_SYS_ADMIN: required for GPU passthrough sysfs operations (vfio group management).
-# CAP_DAC_OVERRIDE: sysfs driver bind/unbind paths (/sys/bus/pci/drivers/vfio-pci/bind,
-# unbind) are root:root 0200; non-root processes need CAP_DAC_OVERRIDE to open them.
+# RG-9: daemon tier — privileged by necessity (GPU vfio). NoNewPrivileges omitted (RG-10: sudo ip/ovs-vsctl/dhcpcd)
 AmbientCapabilities=CAP_SYS_ADMIN CAP_DAC_OVERRIDE
 ProtectSystem=full
 PrivateDevices=no
@@ -73,7 +36,7 @@ DeviceAllow=/dev/vfio/vfio rw
 ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
 ReadWritePaths=/var/lib/spinifex/spinifex /var/log/spinifex /run/spinifex /run/spinifex/nbd
 
-# Hardening
+# RG-9: hardening baseline
 ProtectHome=yes
 PrivateTmp=yes
 RestrictSUIDSGID=yes
@@ -88,8 +51,6 @@ ProtectControlGroups=yes
 ProtectProc=invisible
 SystemCallArchitectures=native
 MemoryDenyWriteExecute=yes
-# DHCP lease acquisition moved to spinifex-vpcd (see dhcp-allocate-error.md);
-# the daemon no longer needs AF_PACKET or the softened kernel protections.
 RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
 ProtectHostname=yes
 KeyringMode=private
@@ -97,8 +58,7 @@ RemoveIPC=yes
 LimitCORE=0
 LimitNOFILE=65536
 LimitNPROC=4096
-# VFIO GPU passthrough requires locking all guest RAM into physical memory.
-# Without this, QEMU fails with RLIMIT_MEMLOCK exceeded when pinning guest pages.
+# LimitMEMLOCK=infinity: VFIO pins all guest RAM
 LimitMEMLOCK=infinity
 
 [Install]

build/systemd/spinifex-guests.slice

@@ -0,0 +1,8 @@
+[Unit]
+Description=Spinifex guest VM tier
+Before=slices.target
+
+[Slice]
+# RG-3: child of spinifex.slice (spinifex-guests -> spinifex), sibling of
+# spinifex-system. Parent for the per-instance spinifex-instance-<id>.slice
+# leaves (0002 RG-7). Inherits the spinifex.slice ceiling; no extra limit here.

build/systemd/spinifex-nats.service

@@ -5,6 +5,7 @@ Before=spinifex-predastore.service spinifex-daemon.service spinifex-awsgw.servic
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-nats
 Group=spinifex
 ExecStart=/usr/local/bin/spx service nats start

build/systemd/spinifex-predastore.service

@@ -6,6 +6,7 @@ Wants=spinifex-nats.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-storage
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh

build/systemd/spinifex-system.slice

@@ -0,0 +1,9 @@
+[Unit]
+Description=Spinifex control/infra tier (NATS, predastore, viperblock, daemon, awsgw, vpcd, ui)
+Before=slices.target
+
+[Slice]
+# RG-3: child of spinifex.slice (spinifex-system -> spinifex), inherits the
+# parent ceiling, no extra limit. Holds the 7 control/infra services. Guest QEMU
+# moves to the sibling spinifex-guests.slice (per-instance placement lands in
+# 0002 RG-7); until then the daemon-forked guests stay under spinifex-daemon.service.

build/systemd/spinifex-ui.service

@@ -5,6 +5,7 @@ After=spinifex-awsgw.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-ui
 Group=spinifex
 ExecStart=/usr/local/bin/spx service spinifex-ui start

build/systemd/spinifex-viperblock.service

@@ -6,6 +6,7 @@ Wants=spinifex-nats.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-viperblock
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh

build/systemd/spinifex-vpcd.service

@@ -6,6 +6,7 @@ Wants=spinifex-nats.service ovn-controller.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-vpcd
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
@@ -16,45 +17,23 @@ OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
 Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/vpcd/
 
-# vpcd MUST share the HOST mount namespace. IMDS binds one listener per VPC inside
-# a per-VPC netns entered in-process via setns(CLONE_NEWNET); the /run/netns/<vpc>
-# bind-mount that `ip netns add` creates must be host-visible (for diagnostics) and
-# must survive a vpcd restart (so the listener can rebind). ANY filesystem-sandbox
-# directive — ProtectSystem / ProtectHome / PrivateTmp / ProtectKernel* /
-# ProtectControlGroups / ProtectProc / Read{Only,Write}Paths — forks a PRIVATE
-# mount namespace that traps that bind-mount inside vpcd, leaving the host a stale
-# handle that fails setns(2) with EINVAL (CI runs 26920490454, 26922587471). Those
-# directives are therefore omitted by design, and so is the MountFlags=shared /
-# spinifex-netns.service propagation workaround they required.
-#
-# Boundary that REMAINS: DAC (User=spinifex-vpcd) plus the seccomp / capability /
-# namespace / address-family restrictions below. The durable fix that restores
-# filesystem sandboxing AND drops CAP_SYS_ADMIN is the process-per-netns model
-# (`ip netns exec` helper instead of in-process setns) — see
-# docs/development/improvements/imds-l2-datapath.md Phase 5.
+# RG-9: network tier. vpcd MUST share the host mount namespace — IMDS enters
+# per-VPC netns via setns(), and any filesystem-sandbox directive (ProtectSystem/
+# Home, PrivateTmp, ProtectKernel*, ProtectControlGroups/Proc, Read*Paths) forks a
+# private mount ns that traps the /run/netns bind-mount and breaks setns(2). Those,
+# plus MemoryDenyWriteExecute, are omitted by design — not regressions.
 PrivateDevices=no
-
-# Hardening that does NOT fork a mount namespace (kept)
 RestrictSUIDSGID=no
 LockPersonality=yes
 RestrictRealtime=yes
 ProtectClock=yes
-# IMDS enters the per-VPC netns via setns(CLONE_NEWNET); the `ip -n <netns>` /
-# `ip netns exec` plumbing also unshares a mount namespace (CLONE_NEWNS, to give
-# the netns its own /sys), so `mnt` must be allowlisted alongside `net` or the
-# first in-netns command fails with EPERM.
+# mnt alongside net: in-netns `ip` unshares a mount ns for its own /sys
 RestrictNamespaces=net mnt
 SystemCallArchitectures=native
 MemoryDenyWriteExecute=no
-# AF_PACKET: required by the in-process DHCP client (github.com/insomniacslk/dhcp
-# / nclient4) for raw L2 BOOTP on br-wan before an IP is configured on the
-# interface. See docs/development/bugs/dhcp-allocate-error.md.
+# AF_PACKET: raw L2 BOOTP for the in-process DHCP client before the iface has an IP
 RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK AF_PACKET
-# CAP_NET_RAW: open AF_PACKET sockets.
-# CAP_NET_BIND_SERVICE: covers unicast RENEW to UDP :68.
-# CAP_NET_ADMIN: already used for ovs-vsctl / ip link wiring on the bridge.
-# CAP_SYS_ADMIN: setns(CLONE_NEWNET) into per-VPC IMDS netns to bind the listener.
-# Note, disabled for now, sudo issue with ovs
+# Caps: NET_ADMIN (ovs/ip), NET_RAW (AF_PACKET), NET_BIND_SERVICE (:68), SETUID/SETGID, SYS_ADMIN (setns into VPC netns)
 AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID CAP_SYS_ADMIN
 CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID CAP_SYS_ADMIN
 NoNewPrivileges=no

build/systemd/spinifex.slice

@@ -0,0 +1,13 @@
+[Unit]
+Description=Spinifex resource-bounded slice (host ceiling)
+Before=slices.target
+
+[Slice]
+# RG-1: no CPUWeight — slice roots are siblings; capping below the default would
+# let user.slice out-prioritize the VM fleet. Operator CPU priority lives on
+# system.slice (system.slice.d/spinifex-reserve.conf).
+# RG-1/RG-5: throttle+reclaim at High, hard wall at Max (% of physical RAM).
+MemoryHigh=80%
+MemoryMax=90%
+# RG-5: spill to the full swap device before the MemoryMax wall; matched to swap size.
+MemorySwapMax=8G

cmd/ecr-credential-provider/main.go

@@ -0,0 +1,368 @@
+// ecr-credential-provider is a kubelet exec credential provider that lets a
+// Spinifex EKS worker node pull images from the internal Spinifex ECR registry.
+//
+// kubelet invokes it per-pull, feeding a CredentialProviderRequest on STDIN and
+// reading a CredentialProviderResponse on STDOUT. The provider fetches the
+// node IAM-role credentials from IMDS, SigV4-signs GetAuthorizationToken against
+// the AWS gateway's ECR endpoint, and returns the minted AWS:<jwt> as Basic auth
+// for the requested registry host.
+//
+// Static node config (gateway URL, CA, region) is read from the cloud-init env
+// file /etc/spinifex-eks/agent.env (KEY=value); real env vars override it.
+package main
+
+import (
+	"bufio"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/base64"
+	"encoding/json"
+	"fmt"
+	"io"
+	"log/slog"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/aws/credentials"
+	"github.com/aws/aws-sdk-go/aws/session"
+	"github.com/aws/aws-sdk-go/service/ecr"
+	"github.com/mulgadc/spinifex/internal/tlsconfig"
+
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
+)
+
+const (
+	credProviderAPIVersion = "credentialprovider.kubelet.k8s.io/v1"
+	defaultCacheDuration   = "10h"
+	// expiryMargin trims the token's real TTL so kubelet re-invokes the provider
+	// before the JWT actually expires mid-run.
+	expiryMargin = 30 * time.Minute
+
+	defaultEnvFile   = "/etc/spinifex-eks/agent.env"
+	defaultGatewayCA = "/etc/spinifex-eks/gateway-ca.pem"
+	defaultIMDSBase  = "http://169.254.169.254/latest"
+	imdsTokenTTL     = "21600"
+)
+
+// CredentialProviderRequest is the kubelet exec-provider request on STDIN.
+type CredentialProviderRequest struct {
+	APIVersion string `json:"apiVersion"`
+	Kind       string `json:"kind"`
+	Image      string `json:"image"`
+}
+
+// AuthConfig is a single registry credential entry.
+type AuthConfig struct {
+	Username string `json:"username"`
+	Password string `json:"password"`
+}
+
+// CredentialProviderResponse is the kubelet exec-provider response on STDOUT.
+type CredentialProviderResponse struct {
+	APIVersion    string                `json:"apiVersion"`
+	Kind          string                `json:"kind"`
+	CacheKeyType  string                `json:"cacheKeyType"`
+	CacheDuration string                `json:"cacheDuration"`
+	Auth          map[string]AuthConfig `json:"auth"`
+}
+
+// config holds the static node settings the provider needs.
+type config struct {
+	GatewayURL string
+	GatewayCA  string
+	Region     string
+	IMDSBase   string
+}
+
+func main() {
+	cfg := loadConfig(defaultEnvFile)
+
+	httpClient, err := gatewayHTTPClient(cfg.GatewayCA)
+	if err != nil {
+		slog.Error("ecr-credential-provider: build gateway client", "err", err)
+		emitEmpty(os.Stdout)
+		os.Exit(1)
+	}
+
+	if err := run(os.Stdin, os.Stdout, cfg, httpClient); err != nil {
+		slog.Error("ecr-credential-provider: run failed", "err", err)
+		// Still emit a valid empty-auth response so kubelet has a parseable reply.
+		emitEmpty(os.Stdout)
+		os.Exit(1)
+	}
+}
+
+// run is the testable core: it reads a CredentialProviderRequest from in,
+// resolves credentials via IMDS + the gateway, and writes a
+// CredentialProviderResponse to out. The httpClient must trust the gateway CA.
+func run(in io.Reader, out io.Writer, cfg config, httpClient *http.Client) error {
+	var req CredentialProviderRequest
+	if err := json.NewDecoder(in).Decode(&req); err != nil {
+		return fmt.Errorf("decode request: %w", err)
+	}
+	if req.Image == "" {
+		return fmt.Errorf("request image is empty")
+	}
+	imageHost := hostFromImage(req.Image)
+
+	creds, err := fetchIMDSCredentials(httpClient, cfg.IMDSBase)
+	if err != nil {
+		return fmt.Errorf("fetch IMDS credentials: %w", err)
+	}
+
+	username, password, proxyHost, expiresAt, err := getAuthToken(cfg, httpClient, creds)
+	if err != nil {
+		return fmt.Errorf("get authorization token: %w", err)
+	}
+
+	auth := AuthConfig{Username: username, Password: password}
+	authMap := map[string]AuthConfig{imageHost: auth}
+	if proxyHost != "" && proxyHost != imageHost {
+		authMap[proxyHost] = auth
+	}
+
+	resp := CredentialProviderResponse{
+		APIVersion:    credProviderAPIVersion,
+		Kind:          "CredentialProviderResponse",
+		CacheKeyType:  "Registry",
+		CacheDuration: cacheDurationFrom(expiresAt),
+		Auth:          authMap,
+	}
+	return json.NewEncoder(out).Encode(&resp)
+}
+
+// imdsCreds mirrors the IMDS security-credentials JSON shape.
+type imdsCreds struct {
+	Code            string `json:"Code"`
+	AccessKeyID     string `json:"AccessKeyId"`
+	SecretAccessKey string `json:"SecretAccessKey"`
+	Token           string `json:"Token"`
+	Expiration      string `json:"Expiration"`
+}
+
+// fetchIMDSCredentials resolves the node role name then its credentials via
+// IMDSv2 (PUT token, then GET with the token header).
+func fetchIMDSCredentials(client *http.Client, base string) (imdsCreds, error) {
+	base = strings.TrimRight(base, "/")
+	token := imdsV2Token(client, base)
+
+	roleName, err := imdsGet(client, base+"/meta-data/iam/security-credentials/", token)
+	if err != nil {
+		return imdsCreds{}, fmt.Errorf("fetch role name: %w", err)
+	}
+	roleName = strings.TrimSpace(roleName)
+	if roleName == "" {
+		return imdsCreds{}, fmt.Errorf("IMDS returned empty role name")
+	}
+
+	body, err := imdsGet(client, base+"/meta-data/iam/security-credentials/"+roleName, token)
+	if err != nil {
+		return imdsCreds{}, fmt.Errorf("fetch role credentials: %w", err)
+	}
+	var creds imdsCreds
+	if err := json.Unmarshal([]byte(body), &creds); err != nil {
+		return imdsCreds{}, fmt.Errorf("decode role credentials: %w", err)
+	}
+	if creds.AccessKeyID == "" || creds.SecretAccessKey == "" {
+		return imdsCreds{}, fmt.Errorf("IMDS credentials missing access/secret key")
+	}
+	return creds, nil
+}
+
+// imdsV2Token requests an IMDSv2 session token; returns "" if the service does
+// not enforce v2 (tokenless v1 then applies).
+func imdsV2Token(client *http.Client, base string) string {
+	req, err := http.NewRequest(http.MethodPut, base+"/api/token", nil)
+	if err != nil {
+		return ""
+	}
+	req.Header.Set("X-Aws-Ec2-Metadata-Token-Ttl-Seconds", imdsTokenTTL)
+	resp, err := client.Do(req)
+	if err != nil {
+		return ""
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return ""
+	}
+	body, _ := io.ReadAll(resp.Body)
+	return strings.TrimSpace(string(body))
+}
+
+func imdsGet(client *http.Client, url, token string) (string, error) {
+	req, err := http.NewRequest(http.MethodGet, url, nil)
+	if err != nil {
+		return "", err
+	}
+	if token != "" {
+		req.Header.Set("X-Aws-Ec2-Metadata-Token", token)
+	}
+	resp, err := client.Do(req)
+	if err != nil {
+		return "", err
+	}
+	defer resp.Body.Close()
+	body, _ := io.ReadAll(resp.Body)
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		return "", fmt.Errorf("IMDS %s returned %d: %s", url, resp.StatusCode, strings.TrimSpace(string(body)))
+	}
+	return string(body), nil
+}
+
+// getAuthToken builds an aws-sdk ECR client pointed at the gateway and calls
+// GetAuthorizationToken, decoding the AWS:<jwt> token into Basic-auth parts. It
+// returns the username, password, the ProxyEndpoint host, and the token expiry.
+func getAuthToken(cfg config, httpClient *http.Client, creds imdsCreds) (username, password, proxyHost string, expiresAt time.Time, err error) {
+	region := cfg.Region
+	if region == "" {
+		region = "us-east-1"
+	}
+	sess, serr := session.NewSession(&aws.Config{
+		Region:      aws.String(region),
+		Endpoint:    aws.String(cfg.GatewayURL),
+		Credentials: credentials.NewStaticCredentials(creds.AccessKeyID, creds.SecretAccessKey, creds.Token),
+		HTTPClient:  httpClient,
+		DisableSSL:  aws.Bool(false),
+	})
+	if serr != nil {
+		return "", "", "", time.Time{}, fmt.Errorf("new session: %w", serr)
+	}
+
+	out, gerr := ecr.New(sess).GetAuthorizationToken(&ecr.GetAuthorizationTokenInput{})
+	if gerr != nil {
+		return "", "", "", time.Time{}, fmt.Errorf("GetAuthorizationToken: %w", gerr)
+	}
+	if len(out.AuthorizationData) == 0 || out.AuthorizationData[0].AuthorizationToken == nil {
+		return "", "", "", time.Time{}, fmt.Errorf("GetAuthorizationToken returned no authorization data")
+	}
+	data := out.AuthorizationData[0]
+
+	raw, derr := base64.StdEncoding.DecodeString(aws.StringValue(data.AuthorizationToken))
+	if derr != nil {
+		return "", "", "", time.Time{}, fmt.Errorf("decode authorization token: %w", derr)
+	}
+	user, pass, ok := strings.Cut(string(raw), ":")
+	if !ok {
+		return "", "", "", time.Time{}, fmt.Errorf("authorization token not in user:password form")
+	}
+
+	if data.ExpiresAt != nil {
+		expiresAt = aws.TimeValue(data.ExpiresAt)
+	}
+	return user, pass, hostFromImage(aws.StringValue(data.ProxyEndpoint)), expiresAt, nil
+}
+
+// cacheDurationFrom derives a Go-duration string from the token expiry minus a
+// safety margin, falling back to the default when expiry is unknown or too near.
+func cacheDurationFrom(expiresAt time.Time) string {
+	if expiresAt.IsZero() {
+		return defaultCacheDuration
+	}
+	d := time.Until(expiresAt) - expiryMargin
+	if d <= 0 {
+		return defaultCacheDuration
+	}
+	return d.Round(time.Second).String()
+}
+
+// gatewayHTTPClient builds an HTTP client trusting the gateway CA at caPath.
+// When caPath is empty it relies on the system trust store.
+func gatewayHTTPClient(caPath string) (*http.Client, error) {
+	tlsCfg := &tls.Config{
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
+	}
+	if caPath != "" {
+		pem, err := os.ReadFile(caPath)
+		if err != nil {
+			return nil, fmt.Errorf("read gateway CA %q: %w", caPath, err)
+		}
+		pool := x509.NewCertPool()
+		if !pool.AppendCertsFromPEM(pem) {
+			return nil, fmt.Errorf("gateway CA %q has no usable certificates", caPath)
+		}
+		tlsCfg.RootCAs = pool
+	}
+	return &http.Client{
+		Timeout:   15 * time.Second,
+		Transport: &http.Transport{TLSClientConfig: tlsCfg, MaxIdleConns: 2, IdleConnTimeout: 30 * time.Second},
+	}, nil
+}
+
+// loadConfig reads the cloud-init env file then lets real env vars override. The
+// IMDS base is fixed (not in the env file) but overridable for tests.
+func loadConfig(envFile string) config {
+	env := parseEnvFile(envFile)
+	get := func(key string) string {
+		if v := os.Getenv(key); v != "" {
+			return v
+		}
+		return env[key]
+	}
+
+	cfg := config{
+		GatewayURL: get("EKS_GATEWAY_URL"),
+		GatewayCA:  get("EKS_GATEWAY_CA"),
+		Region:     get("EKS_REGION"),
+		IMDSBase:   get("EKS_IMDS_BASE"),
+	}
+	if cfg.GatewayCA == "" {
+		cfg.GatewayCA = defaultGatewayCA
+	}
+	if cfg.IMDSBase == "" {
+		cfg.IMDSBase = defaultIMDSBase
+	}
+	return cfg
+}
+
+// parseEnvFile reads a simple KEY=value file; missing files yield an empty map.
+func parseEnvFile(path string) map[string]string {
+	out := map[string]string{}
+	f, err := os.Open(path)
+	if err != nil {
+		return out
+	}
+	defer f.Close()
+	scanner := bufio.NewScanner(f)
+	for scanner.Scan() {
+		line := strings.TrimSpace(scanner.Text())
+		if line == "" || strings.HasPrefix(line, "#") {
+			continue
+		}
+		key, val, ok := strings.Cut(line, "=")
+		if !ok {
+			continue
+		}
+		out[strings.TrimSpace(key)] = strings.TrimSpace(val)
+	}
+	return out
+}
+
+// hostFromImage returns the host[:port] portion of an image ref or URL. It strips
+// any scheme and trims at the first path separator.
+func hostFromImage(image string) string {
+	if i := strings.Index(image, "://"); i >= 0 {
+		image = image[i+3:]
+	}
+	if i := strings.IndexByte(image, '/'); i >= 0 {
+		image = image[:i]
+	}
+	return image
+}
+
+// emitEmpty writes a valid CredentialProviderResponse with no auth entries.
+func emitEmpty(out io.Writer) {
+	resp := CredentialProviderResponse{
+		APIVersion:    credProviderAPIVersion,
+		Kind:          "CredentialProviderResponse",
+		CacheKeyType:  "Registry",
+		CacheDuration: defaultCacheDuration,
+		Auth:          map[string]AuthConfig{},
+	}
+	if err := json.NewEncoder(out).Encode(&resp); err != nil {
+		slog.Error("ecr-credential-provider: encode empty response", "err", err)
+	}
+}

cmd/ecr-credential-provider/main_test.go

@@ -0,0 +1,138 @@
+package main
+
+import (
+	"bytes"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/base64"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+// fakeBackend serves both the IMDS role/credentials endpoints and the gateway
+// ECR GetAuthorizationToken on one TLS server, routed by request path.
+func fakeBackend(t *testing.T) *httptest.Server {
+	t.Helper()
+	mux := http.NewServeMux()
+
+	// IMDSv2 token.
+	mux.HandleFunc("/latest/api/token", func(w http.ResponseWriter, r *http.Request) {
+		if r.Method != http.MethodPut {
+			w.WriteHeader(http.StatusMethodNotAllowed)
+			return
+		}
+		_, _ = w.Write([]byte("imds-v2-token"))
+	})
+	// IMDS role name.
+	mux.HandleFunc("/latest/meta-data/iam/security-credentials/", func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path == "/latest/meta-data/iam/security-credentials/" {
+			if r.Header.Get("X-Aws-Ec2-Metadata-Token") != "imds-v2-token" {
+				w.WriteHeader(http.StatusUnauthorized)
+				return
+			}
+			_, _ = w.Write([]byte("node-role"))
+			return
+		}
+		// Role credentials.
+		_ = json.NewEncoder(w).Encode(map[string]string{
+			"Code":            "Success",
+			"Type":            "AWS-HMAC",
+			"AccessKeyId":     "AKIATEST",
+			"SecretAccessKey": "secrettest",
+			"Token":           "sessiontoken",
+			"AccountId":       "123456789012",
+		})
+	})
+	// Gateway ECR GetAuthorizationToken (X-Amz-Target dispatch).
+	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+		if !strings.Contains(r.Header.Get("X-Amz-Target"), "GetAuthorizationToken") {
+			w.WriteHeader(http.StatusBadRequest)
+			return
+		}
+		authToken := base64.StdEncoding.EncodeToString([]byte("AWS:testjwt"))
+		w.Header().Set("Content-Type", "application/x-amz-json-1.1")
+		_ = json.NewEncoder(w).Encode(map[string]any{
+			"authorizationData": []map[string]any{{
+				"authorizationToken": authToken,
+				"proxyEndpoint":      "https://123456789012.dkr.ecr.us-east-1.spinifex.internal",
+			}},
+		})
+	})
+
+	srv := httptest.NewTLSServer(mux)
+	t.Cleanup(srv.Close)
+	return srv
+}
+
+func clientTrusting(srv *httptest.Server) *http.Client {
+	pool := x509.NewCertPool()
+	pool.AddCert(srv.Certificate())
+	return &http.Client{Transport: &http.Transport{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12}}}
+}
+
+func TestRun_ReturnsBasicAuth(t *testing.T) {
+	srv := fakeBackend(t)
+	client := clientTrusting(srv)
+
+	cfg := config{
+		GatewayURL: srv.URL,
+		Region:     "us-east-1",
+		IMDSBase:   srv.URL + "/latest",
+	}
+
+	const image = "123456789012.dkr.ecr.us-east-1.spinifex.internal/spinifex-demo:latest"
+	reqBody, err := json.Marshal(CredentialProviderRequest{
+		APIVersion: credProviderAPIVersion,
+		Kind:       "CredentialProviderRequest",
+		Image:      image,
+	})
+	if err != nil {
+		t.Fatalf("marshal request: %v", err)
+	}
+
+	var out bytes.Buffer
+	if err := run(bytes.NewReader(reqBody), &out, cfg, client); err != nil {
+		t.Fatalf("run: %v", err)
+	}
+
+	var resp CredentialProviderResponse
+	if err := json.Unmarshal(out.Bytes(), &resp); err != nil {
+		t.Fatalf("unmarshal response: %v\n%s", err, out.String())
+	}
+
+	if resp.Kind != "CredentialProviderResponse" {
+		t.Errorf("Kind = %q, want CredentialProviderResponse", resp.Kind)
+	}
+	if resp.CacheKeyType != "Registry" {
+		t.Errorf("CacheKeyType = %q, want Registry", resp.CacheKeyType)
+	}
+
+	host := "123456789012.dkr.ecr.us-east-1.spinifex.internal"
+	auth, ok := resp.Auth[host]
+	if !ok {
+		t.Fatalf("auth map missing host %q; got %v", host, resp.Auth)
+	}
+	if auth.Username != "AWS" {
+		t.Errorf("username = %q, want AWS", auth.Username)
+	}
+	if auth.Password != "testjwt" {
+		t.Errorf("password = %q, want testjwt", auth.Password)
+	}
+}
+
+func TestHostFromImage(t *testing.T) {
+	cases := map[string]string{
+		"acct.dkr.ecr.us-east-1.spinifex.internal/demo:latest": "acct.dkr.ecr.us-east-1.spinifex.internal",
+		"https://acct.dkr.ecr.us-east-1.spinifex.internal":     "acct.dkr.ecr.us-east-1.spinifex.internal",
+		"host:9999/repo:tag": "host:9999",
+		"plainhost":          "plainhost",
+	}
+	for in, want := range cases {
+		if got := hostFromImage(in); got != want {
+			t.Errorf("hostFromImage(%q) = %q, want %q", in, got, want)
+		}
+	}
+}

cmd/eks-gateway-fetch/main.go

@@ -0,0 +1,139 @@
+// eks-gateway-fetch runs inside the EKS K3s control-plane VM and fetches a
+// control-plane resource from the host through the AWS gateway, instead of
+// dialing core NATS directly. It is the read-side companion to
+// eks-gateway-publish.
+//
+// It SigV4-signs (service "eks") an HTTPS GET to the gateway, retrying with
+// backoff until the gateway returns 2xx or the attempt budget is exhausted (so
+// a degraded link surfaces as a non-zero exit), then emits the staged add-ons
+// as tab-separated lines the on-VM addon-sync shell agent consumes without a
+// JSON parser:
+//
+//	<addonName>\t<addonVersion>\t<serviceAccountRoleArn>\t<base64(configurationValues)>
+//
+// The configuration values are base64-encoded so embedded tabs/newlines cannot
+// break the line framing.
+//
+// Usage:
+//
+//	eks-gateway-fetch -resource addons   # GET /clusters/{cluster}/internal-addons/{accountId}
+//
+// Flags default to environment variables seeded by cloud-init:
+// EKS_GATEWAY_URL, EKS_GATEWAY_CA, EKS_ACCESS_KEY, EKS_SECRET_KEY, EKS_REGION,
+// EKS_ACCOUNT_ID, EKS_CLUSTER_NAME.
+package main
+
+import (
+	"bufio"
+	"encoding/base64"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"log/slog"
+	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/eksgw"
+
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
+)
+
+// stagedAddon mirrors the gateway's StagedAddonManifest wire shape; duplicated
+// here to keep this tiny VM binary free of the handler package's dependency
+// graph.
+type stagedAddon struct {
+	AddonName             string `json:"addonName"`
+	AddonVersion          string `json:"addonVersion"`
+	ServiceAccountRoleArn string `json:"serviceAccountRoleArn"`
+	ConfigurationValues   string `json:"configurationValues"`
+}
+
+type internalAddonsResponse struct {
+	Addons []stagedAddon `json:"addons"`
+}
+
+const (
+	maxAttempts = 30
+	retryDelay  = 5 * time.Second
+)
+
+func main() {
+	var (
+		gatewayURL  string
+		gatewayCA   string
+		accessKey   string
+		secretKey   string
+		region      string
+		accountID   string
+		clusterName string
+		resource    string
+	)
+
+	flag.StringVar(&gatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
+	flag.StringVar(&gatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
+	flag.StringVar(&accessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
+	flag.StringVar(&secretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
+	flag.StringVar(&region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
+	flag.StringVar(&accountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
+	flag.StringVar(&clusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
+	flag.StringVar(&resource, "resource", "addons", "Resource to fetch: addons")
+	flag.Parse()
+
+	switch {
+	case accountID == "":
+		fatal("--account-id is required (or set EKS_ACCOUNT_ID)")
+	case clusterName == "":
+		fatal("--cluster is required (or set EKS_CLUSTER_NAME)")
+	case resource != "addons":
+		fatal("--resource must be addons")
+	}
+
+	client, err := eksgw.New(gatewayURL, gatewayCA, accessKey, secretKey, region)
+	if err != nil {
+		fatal(fmt.Sprintf("build gateway client: %v", err))
+	}
+	path := "/clusters/" + clusterName + "/internal-addons/" + accountID
+
+	var lastErr error
+	for attempt := 1; attempt <= maxAttempts; attempt++ {
+		body, err := client.Get(path)
+		if err != nil {
+			lastErr = err
+			slog.Warn("eks-gateway-fetch: attempt failed", "resource", resource, "attempt", attempt, "err", err)
+			if attempt < maxAttempts {
+				time.Sleep(retryDelay)
+			}
+			continue
+		}
+		if werr := emitAddonsTSV(os.Stdout, body); werr != nil {
+			fatal(fmt.Sprintf("render response: %v", werr))
+		}
+		return
+	}
+	fatal(fmt.Sprintf("fetch failed after %d attempts: %v", maxAttempts, lastErr))
+}
+
+// emitAddonsTSV decodes the internal-addons response and writes one TSV line per
+// staged add-on to w. An empty add-on set writes nothing (the agent then GCs
+// every locally-rendered manifest), which is the correct steady state for a
+// cluster with no managed add-ons.
+func emitAddonsTSV(out io.Writer, body []byte) error {
+	var resp internalAddonsResponse
+	if err := json.Unmarshal(body, &resp); err != nil {
+		return fmt.Errorf("unmarshal internal-addons response: %w", err)
+	}
+	w := bufio.NewWriter(out)
+	for _, a := range resp.Addons {
+		cfg := base64.StdEncoding.EncodeToString([]byte(a.ConfigurationValues))
+		if _, err := fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", a.AddonName, a.AddonVersion, a.ServiceAccountRoleArn, cfg); err != nil {
+			return err
+		}
+	}
+	return w.Flush()
+}
+
+func fatal(msg string) {
+	slog.Error("eks-gateway-fetch: " + msg)
+	os.Exit(1)
+}

cmd/eks-gateway-fetch/main_test.go

@@ -0,0 +1,44 @@
+package main
+
+import (
+	"bytes"
+	"encoding/base64"
+	"testing"
+)
+
+func TestEmitAddonsTSV(t *testing.T) {
+	body := []byte(`{"addons":[
+		{"addonName":"spinifex-noop","addonVersion":"0.1.0"},
+		{"addonName":"aws-load-balancer-controller","addonVersion":"2.8.1",
+		 "serviceAccountRoleArn":"arn:aws:iam::111122223333:role/alb",
+		 "configurationValues":"{\"replicaCount\":2}"}
+	]}`)
+
+	var buf bytes.Buffer
+	if err := emitAddonsTSV(&buf, body); err != nil {
+		t.Fatalf("emitAddonsTSV: %v", err)
+	}
+	cfg := base64.StdEncoding.EncodeToString([]byte(`{"replicaCount":2}`))
+	want := "spinifex-noop\t0.1.0\t\t\n" +
+		"aws-load-balancer-controller\t2.8.1\tarn:aws:iam::111122223333:role/alb\t" + cfg + "\n"
+	if buf.String() != want {
+		t.Fatalf("TSV mismatch:\n got: %q\nwant: %q", buf.String(), want)
+	}
+}
+
+func TestEmitAddonsTSV_Empty(t *testing.T) {
+	var buf bytes.Buffer
+	if err := emitAddonsTSV(&buf, []byte(`{"addons":[]}`)); err != nil {
+		t.Fatalf("emitAddonsTSV: %v", err)
+	}
+	if buf.Len() != 0 {
+		t.Fatalf("empty addon set must emit nothing, got %q", buf.String())
+	}
+}
+
+func TestEmitAddonsTSV_BadJSON(t *testing.T) {
+	var buf bytes.Buffer
+	if err := emitAddonsTSV(&buf, []byte(`{not-json`)); err == nil {
+		t.Fatal("expected error on malformed JSON")
+	}
+}

cmd/eks-gateway-publish/main.go

@@ -69,7 +69,7 @@ func main() {
 	flag.StringVar(&region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
 	flag.StringVar(&accountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
 	flag.StringVar(&clusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
-	flag.StringVar(&channel, "channel", "", "Publish channel: bootstrap|state")
+	flag.StringVar(&channel, "channel", "", "Publish channel: bootstrap|state|addon")
 	flag.StringVar(&kind, "kind", "", "Bootstrap subject kind (bootstrap channel only)")
 	flag.Parse()
 
@@ -78,8 +78,8 @@ func main() {
 		fatal("--account-id is required (or set EKS_ACCOUNT_ID)")
 	case clusterName == "":
 		fatal("--cluster is required (or set EKS_CLUSTER_NAME)")
-	case channel != "bootstrap" && channel != "state":
-		fatal("--channel must be bootstrap or state")
+	case channel != "bootstrap" && channel != "state" && channel != "addon":
+		fatal("--channel must be bootstrap, state or addon")
 	case channel == "bootstrap" && kind == "":
 		fatal("--kind is required for the bootstrap channel")
 	}

cmd/eks-webhook-cert/main.go

@@ -0,0 +1,147 @@
+// Command eks-webhook-cert mints, then reuses, a self-signed ECDSA-P256 serving
+// certificate for an in-cluster admission webhook and prints its base64 PEM so
+// the addon renderer can embed it as the webhook Secret + each caBundle.
+//
+// The eks-node AMI carries no openssl, so this is the Go equivalent of the
+// upstream Helm chart's install-time genSelfSignedCert: one cert per cluster,
+// cached on disk under -dir, stable across addon-sync ticks. The single cert is
+// its own CA (IsCA + ServerAuth), so the same PEM serves as both the TLS leaf
+// and the caBundle the apiserver trusts.
+//
+// Usage:
+//
+//	eks-webhook-cert -dir /var/lib/spinifex-eks/webhook-certs/<addon> \
+//	    -cn <svc>.<ns>.svc -dns <svc>.<ns>.svc,<svc>.<ns>.svc.cluster.local
+//
+// Output (one line): <ca_b64>\t<tls_crt_b64>\t<tls_key_b64>
+package main
+
+import (
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/base64"
+	"encoding/pem"
+	"flag"
+	"fmt"
+	"log/slog"
+	"math/big"
+	"os"
+	"path/filepath"
+	"strings"
+	"time"
+)
+
+func main() {
+	dir := flag.String("dir", "", "directory to cache tls.crt/tls.key in")
+	cn := flag.String("cn", "", "certificate common name")
+	dnsCSV := flag.String("dns", "", "comma-separated DNS SANs")
+	flag.Parse()
+
+	if *dir == "" || *cn == "" || *dnsCSV == "" {
+		fatal("eks-webhook-cert: -dir, -cn and -dns are required")
+	}
+	dnsNames := splitCSV(*dnsCSV)
+	if len(dnsNames) == 0 {
+		fatal("eks-webhook-cert: -dns lists no names")
+	}
+
+	certPEM, keyPEM, err := ensureCert(*dir, *cn, dnsNames)
+	if err != nil {
+		fatal("eks-webhook-cert: " + err.Error())
+	}
+	b := base64.StdEncoding.EncodeToString
+	fmt.Printf("%s\t%s\t%s\n", b(certPEM), b(certPEM), b(keyPEM))
+}
+
+// ensureCert returns the cached cert/key under dir, minting and persisting a
+// fresh self-signed pair (for cn + dnsNames) when none is present or readable.
+func ensureCert(dir, cn string, dnsNames []string) (certPEM, keyPEM []byte, err error) {
+	certPath := filepath.Join(dir, "tls.crt")
+	keyPath := filepath.Join(dir, "tls.key")
+
+	if c, cerr := os.ReadFile(certPath); cerr == nil {
+		if k, kerr := os.ReadFile(keyPath); kerr == nil {
+			if _, perr := x509.ParseCertificate(decodeFirst(c)); perr == nil && len(k) > 0 {
+				return c, k, nil
+			}
+		}
+	}
+
+	certPEM, keyPEM, err = generateSelfSigned(cn, dnsNames)
+	if err != nil {
+		return nil, nil, err
+	}
+	if err = os.MkdirAll(dir, 0o700); err != nil {
+		return nil, nil, fmt.Errorf("mkdir %s: %w", dir, err)
+	}
+	if err = os.WriteFile(certPath, certPEM, 0o644); err != nil {
+		return nil, nil, fmt.Errorf("write %s: %w", certPath, err)
+	}
+	if err = os.WriteFile(keyPath, keyPEM, 0o600); err != nil {
+		return nil, nil, fmt.Errorf("write %s: %w", keyPath, err)
+	}
+	return certPEM, keyPEM, nil
+}
+
+// generateSelfSigned mints an ECDSA-P256 self-signed cert valid as both a CA and
+// a server cert, with the given common name and DNS SANs.
+func generateSelfSigned(cn string, dnsNames []string) (certPEM, keyPEM []byte, err error) {
+	priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		return nil, nil, fmt.Errorf("generate key: %w", err)
+	}
+	serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
+	if err != nil {
+		return nil, nil, fmt.Errorf("serial: %w", err)
+	}
+	now := time.Now().UTC()
+	tmpl := x509.Certificate{
+		SerialNumber:          serial,
+		Subject:               pkix.Name{CommonName: cn},
+		NotBefore:             now.Add(-time.Hour),
+		NotAfter:              now.AddDate(10, 0, 0),
+		KeyUsage:              x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment | x509.KeyUsageCertSign,
+		ExtKeyUsage:           []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+		BasicConstraintsValid: true,
+		IsCA:                  true,
+		DNSNames:              dnsNames,
+	}
+	der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &priv.PublicKey, priv)
+	if err != nil {
+		return nil, nil, fmt.Errorf("create cert: %w", err)
+	}
+	keyDER, err := x509.MarshalECPrivateKey(priv)
+	if err != nil {
+		return nil, nil, fmt.Errorf("marshal key: %w", err)
+	}
+	certPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
+	keyPEM = pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
+	return certPEM, keyPEM, nil
+}
+
+// decodeFirst returns the DER bytes of the first PEM block, or nil.
+func decodeFirst(pemBytes []byte) []byte {
+	blk, _ := pem.Decode(pemBytes)
+	if blk == nil {
+		return nil
+	}
+	return blk.Bytes
+}
+
+func splitCSV(s string) []string {
+	var out []string
+	for p := range strings.SplitSeq(s, ",") {
+		if p = strings.TrimSpace(p); p != "" {
+			out = append(out, p)
+		}
+	}
+	return out
+}
+
+func fatal(msg string) {
+	slog.Error(msg)
+	os.Exit(1)
+}

cmd/eks-webhook-cert/main_test.go

@@ -0,0 +1,81 @@
+package main
+
+import (
+	"crypto/x509"
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+const (
+	testCN  = "aws-load-balancer-webhook-service.kube-system.svc"
+	testDNS = "aws-load-balancer-webhook-service.kube-system.svc.cluster.local"
+)
+
+func parseLeaf(t *testing.T, certPEM []byte) *x509.Certificate {
+	t.Helper()
+	leaf, err := x509.ParseCertificate(decodeFirst(certPEM))
+	if err != nil {
+		t.Fatalf("parse cert: %v", err)
+	}
+	return leaf
+}
+
+func TestEnsureCertMintsServingCert(t *testing.T) {
+	dir := t.TempDir()
+	certPEM, keyPEM, err := ensureCert(dir, testCN, []string{testCN, testDNS})
+	if err != nil {
+		t.Fatalf("ensureCert: %v", err)
+	}
+	if len(keyPEM) == 0 {
+		t.Fatal("empty key PEM")
+	}
+	leaf := parseLeaf(t, certPEM)
+
+	// Both webhook service DNS names must be present, or the apiserver's TLS
+	// dial to the webhook fails SAN verification.
+	for _, want := range []string{testCN, testDNS} {
+		if err := leaf.VerifyHostname(want); err != nil {
+			t.Errorf("SAN %q missing: %v", want, err)
+		}
+	}
+	if !leaf.IsCA {
+		t.Error("cert must be a CA so the same PEM serves as the webhook caBundle")
+	}
+
+	// The single self-signed cert must verify against itself as a root — this is
+	// exactly what the apiserver does with caBundle == the serving cert.
+	roots := x509.NewCertPool()
+	roots.AddCert(leaf)
+	if _, err := leaf.Verify(x509.VerifyOptions{
+		Roots:     roots,
+		DNSName:   testCN,
+		KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+	}); err != nil {
+		t.Errorf("self-signed cert does not verify as its own CA: %v", err)
+	}
+}
+
+func TestEnsureCertReusesCachedPair(t *testing.T) {
+	dir := t.TempDir()
+	cert1, key1, err := ensureCert(dir, testCN, []string{testCN})
+	if err != nil {
+		t.Fatalf("first ensureCert: %v", err)
+	}
+	cert2, key2, err := ensureCert(dir, testCN, []string{testCN})
+	if err != nil {
+		t.Fatalf("second ensureCert: %v", err)
+	}
+	if string(cert1) != string(cert2) || string(key1) != string(key2) {
+		t.Error("ensureCert regenerated instead of reusing the cached pair; a churning cert re-applies the addon every tick")
+	}
+
+	// The private key must not be world-readable on disk.
+	info, err := os.Stat(filepath.Join(dir, "tls.key"))
+	if err != nil {
+		t.Fatalf("stat key: %v", err)
+	}
+	if info.Mode().Perm()&0o077 != 0 {
+		t.Errorf("tls.key mode %o is group/other-accessible", info.Mode().Perm())
+	}
+}

cmd/spinifex/cmd/admin.go

@@ -115,6 +115,16 @@ Each phase waits for all nodes to ACK before proceeding to the next.`,
 	Run: runClusterShutdown,
 }
 
+var clusterDrainDHCPCmd = &cobra.Command{
+	Use:   "drain-dhcp",
+	Short: "Release all upstream DHCP leases held by vpcd",
+	Long: `Ask each vpcd to DHCPRELEASE every external-pool DHCP lease it currently
+holds, returning them to the upstream DHCP server. Run this on teardown before
+stopping services: an env reset otherwise strands held leases on the upstream
+server until their TTL expires, eventually exhausting the upstream scope.`,
+	Run: runClusterDrainDHCP,
+}
+
 var adminInitCmd = &cobra.Command{
 	Use:   "init",
 	Short: "Initialize Spinifex platform configuration",
@@ -245,6 +255,9 @@ func init() {
 	clusterShutdownCmd.Flags().Duration("timeout", 120*time.Second, "Maximum time to wait per phase")
 	clusterShutdownCmd.Flags().Bool("dry-run", false, "Print phase plan without executing")
 
+	clusterCmd.AddCommand(clusterDrainDHCPCmd)
+	clusterDrainDHCPCmd.Flags().Duration("timeout", 30*time.Second, "Reply-collection window for vpcd drain responders")
+
 	adminCmd.AddCommand(imagesCmd)
 	imagesCmd.AddCommand(imagesImportCmd)
 	imagesCmd.AddCommand(imagesListCmd)
@@ -285,6 +298,7 @@ func init() {
 	adminInitCmd.Flags().String("predastore-nodes", "", "Comma-separated IPs for multi-node Predastore cluster (e.g., 10.11.12.1,10.11.12.2,10.11.12.3). Requires >= 3 nodes.")
 	adminInitCmd.Flags().String("formation-timeout", "10m", "Timeout for cluster formation (e.g., 5m, 30s)")
 	adminInitCmd.Flags().String("token-ttl", "30m", "Join token validity duration (e.g. 30m, 1h, 2h)")
+	adminInitCmd.Flags().Int("predastore-compaction-interval", 0, "Predastore compactor interval in seconds (0 = unset, uses built-in default). Test clusters set a short interval.")
 	adminInitCmd.Flags().String("cluster-name", "spinifex", "NATS cluster name")
 	adminInitCmd.Flags().Bool("no-telemetry", false, "Disable telemetry metrics sent during init (default: enabled)")
 	adminInitCmd.Flags().String("email", "", "Operator email address (used for update and security notifications)")
@@ -318,6 +332,7 @@ func init() {
 	adminJoinCmd.Flags().StringSlice("services", nil, "Services this node runs (default: all)")
 	adminJoinCmd.Flags().Bool("no-telemetry", false, "Disable telemetry metrics sent during join (default: enabled)")
 	adminJoinCmd.Flags().String("email", "", "Operator email address (used for update and security notifications)")
+	adminJoinCmd.Flags().Int("predastore-compaction-interval", 0, "Predastore compactor interval in seconds (0 = unset, uses built-in default). Test clusters set a short interval.")
 	adminJoinCmd.MarkFlagRequired("node")
 	adminJoinCmd.MarkFlagRequired("host")
 	adminJoinCmd.MarkFlagRequired("token")
@@ -833,6 +848,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	predastoreNodesStr, _ := cmd.Flags().GetString("predastore-nodes")
 	formationTimeoutStr, _ := cmd.Flags().GetString("formation-timeout")
 	tokenTTLStr, _ := cmd.Flags().GetString("token-ttl")
+	compactionInterval, _ := cmd.Flags().GetInt("predastore-compaction-interval")
 	clusterName, _ := cmd.Flags().GetString("cluster-name")
 	services, _ := cmd.Flags().GetStringSlice("services")
 
@@ -1114,6 +1130,17 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	fmt.Println("\n🔐 Generated predastore encryption key (per-node, never transmitted)")
 	fmt.Printf("   Key: %s\n", predastoreKeyPath)
 
+	// Viperblock at-rest encryption key is cluster-wide; on a single-node init
+	// there are no joiners, so generate it locally and enable encryption by
+	// default for all volumes created on this install.
+	viperblockKeyPath, err := writeViperblockEncryptionKey(configDir)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error generating viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	fmt.Println("\n🔐 Generated viperblock at-rest encryption key")
+	fmt.Printf("   Key: %s\n", viperblockKeyPath)
+
 	fmt.Printf("\n🔑 Generated admin credentials (save these — they won't be shown again):\n")
 	fmt.Printf("   Access Key:  %s\n", bootstrapResult.AdminAccessKey)
 	fmt.Printf("   Secret Key:  %s\n", bootstrapResult.AdminSecretKey)
@@ -1121,7 +1148,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	fmt.Printf("   AWS Profile: spinifex\n")
 
 	// Generate SSL certificates (with bind IP in SANs for multi-node support)
-	certPath := admin.GenerateCertificatesIfNeeded(configDir, force, bindIP)
+	certPath := admin.GenerateCertificatesIfNeeded(configDir, force, bindIP, region, config.DefaultAWSInternalSuffix)
 
 	// Generate per-node IPsec peer cert when cluster-wide IPsec is enabled
 	// (default true). Reuses the cluster CA — no intermediate strongSwan PKI.
@@ -1232,7 +1259,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		}
 
 		// Generate multi-node predastore.toml
-		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP)
+		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP, compactionInterval)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Error generating multi-node predastore config: %v\n", err)
 			os.Exit(1)
@@ -1273,8 +1300,9 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		ClusterRoutes: clusterRoutes,
 		ClusterName:   clusterName,
 
-		PredastoreNodeID: predastoreNodeID,
-		Services:         services,
+		PredastoreNodeID:          predastoreNodeID,
+		CompactionIntervalSeconds: compactionInterval,
+		Services:                  services,
 
 		OVNNBAddr: "tcp:127.0.0.1:6641",
 		OVNSBAddr: "tcp:127.0.0.1:6642",
@@ -1301,6 +1329,8 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 
 		GPUPassthrough: gpuPassthrough,
 		IPSecEnabled:   ipsecEnabled,
+
+		EncryptionKeyFile: viperblockKeyPath,
 	}
 
 	// Print external networking summary
@@ -1377,6 +1407,8 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 		os.Exit(1)
 	}
 
+	compactionInterval, _ := cmd.Flags().GetInt("predastore-compaction-interval")
+
 	joinToken, err := formation.GenerateJoinToken()
 	if err != nil {
 		fmt.Fprintf(os.Stderr, "❌ Error generating join token: %v\n", err)
@@ -1409,6 +1441,22 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	fmt.Println("\n🔐 Generated predastore encryption key (per-node, never transmitted)")
 	fmt.Printf("   Key: %s\n", predastoreKeyPath)
 
+	// Viperblock at-rest encryption key is cluster-wide and shared: the leader
+	// generates it once and distributes it to joiners via the formation server
+	// so a volume sealed on any node can be opened on any other.
+	viperblockKey, err := handlers_iam.GenerateMasterKey()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "❌ Error generating viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	viperblockKeyPath, err := saveViperblockEncryptionKey(configDir, viperblockKey)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "❌ Error saving viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	fmt.Println("\n🔐 Generated viperblock at-rest encryption key (cluster-wide, shared with joiners)")
+	fmt.Printf("   Key: %s\n", viperblockKeyPath)
+
 	// Read CA cert/key for distribution to joining nodes
 	caCertData, err := os.ReadFile(filepath.Join(configDir, "ca.pem"))
 	if err != nil {
@@ -1437,6 +1485,9 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	// Include master key in formation server for distribution to joining nodes
 	fs.SetMasterKey(base64.StdEncoding.EncodeToString(masterKey))
 
+	// Distribute the cluster-wide viperblock encryption key to joiners
+	fs.SetViperblockKey(base64.StdEncoding.EncodeToString(viperblockKey))
+
 	// Register self (init node) as the first node
 	selfNode := formation.NodeInfo{
 		Name:        node,
@@ -1501,7 +1552,7 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	var predastoreNodeID int
 	hasPredastoreConfig := len(predastoreNodes) >= 2
 	if hasPredastoreConfig {
-		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP)
+		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP, compactionInterval)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Error generating multi-node predastore config: %v\n", err)
 			os.Exit(1)
@@ -1538,12 +1589,15 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 		ClusterRoutes: clusterRoutes,
 		ClusterName:   clusterName,
 
-		PredastoreNodeID: predastoreNodeID,
-		Services:         services,
-		RemoteNodes:      buildRemoteNodes(allNodes, node),
+		PredastoreNodeID:          predastoreNodeID,
+		CompactionIntervalSeconds: compactionInterval,
+		Services:                  services,
+		RemoteNodes:               buildRemoteNodes(allNodes, node),
 
 		OperatorEmail: email,
 
+		EncryptionKeyFile: viperblockKeyPath,
+
 		// Init node runs ovn-central locally
 		OVNNBAddr: "tcp:127.0.0.1:6641",
 		OVNSBAddr: "tcp:127.0.0.1:6642",
@@ -1585,7 +1639,7 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	}
 	fmt.Println("\n📋 Next steps:")
 	fmt.Println("   1. Start services on ALL nodes:")
-	fmt.Println("      ./scripts/start-dev.sh")
+	fmt.Println("      sudo systemctl start spinifex.target")
 	fmt.Println()
 }
 
@@ -1608,6 +1662,7 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 	configDir, _ := cmd.Flags().GetString("config-dir")
 	clusterBind, _ := cmd.Flags().GetString("cluster-bind")
 	services, _ := cmd.Flags().GetStringSlice("services")
+	compactionInterval, _ := cmd.Flags().GetInt("predastore-compaction-interval")
 
 	email, _ := cmd.Flags().GetString("email")
 	email = strings.TrimSpace(email)
@@ -1889,8 +1944,30 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 	}
 	fmt.Printf("✅ Predastore encryption key generated: %s\n", predastoreKeyPath)
 
+	// Viperblock at-rest encryption key is cluster-wide: receive it from the
+	// leader rather than generating one, so this node can open volumes sealed
+	// elsewhere. Lenient on absence — an older leader that predates key
+	// distribution returns nothing; fall back to cleartext rather than fail
+	// the join.
+	var viperblockKeyPath string
+	if statusResp.ViperblockKey == "" {
+		fmt.Println("⚠️  Leader did not provide a viperblock encryption key; at-rest encryption disabled on this node")
+	} else {
+		viperblockKeyBytes, err := base64.StdEncoding.DecodeString(statusResp.ViperblockKey)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "❌ Error decoding viperblock encryption key: %v\n", err)
+			os.Exit(1)
+		}
+		viperblockKeyPath, err = saveViperblockEncryptionKey(configDir, viperblockKeyBytes)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "❌ Error saving viperblock encryption key: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Printf("✅ Viperblock encryption key received from leader: %s\n", viperblockKeyPath)
+	}
+
 	// Generate server cert signed by CA with this node's bind IP
-	if err := admin.GenerateServerCertOnly(configDir, bindIP); err != nil {
+	if err := admin.GenerateServerCertOnly(configDir, bindIP, region, config.DefaultAWSInternalSuffix); err != nil {
 		fmt.Fprintf(os.Stderr, "Error generating server certificate: %v\n", err)
 		os.Exit(1)
 	}
@@ -1933,7 +2010,7 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 	hasPredastoreConfig := len(predastoreNodes) >= 2
 
 	if hasPredastoreConfig {
-		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, creds.AccessKey, creds.SecretKey, creds.Region, creds.NatsToken, configDir, bindIP)
+		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, creds.AccessKey, creds.SecretKey, creds.Region, creds.NatsToken, configDir, bindIP, compactionInterval)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Error generating multi-node predastore config: %v\n", err)
 			os.Exit(1)
@@ -1974,12 +2051,15 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 		ClusterRoutes: clusterRoutes,
 		ClusterName:   creds.ClusterName,
 
-		PredastoreNodeID: predastoreNodeID,
-		Services:         services,
-		RemoteNodes:      buildRemoteNodes(statusResp.Nodes, node),
+		PredastoreNodeID:          predastoreNodeID,
+		CompactionIntervalSeconds: compactionInterval,
+		Services:                  services,
+		RemoteNodes:               buildRemoteNodes(statusResp.Nodes, node),
 
 		OperatorEmail: email,
 
+		EncryptionKeyFile: viperblockKeyPath,
+
 		// Joining nodes connect to the init node's OVN NB/SB DB
 		OVNNBAddr: fmt.Sprintf("tcp:%s:6641", leaderIP),
 		OVNSBAddr: fmt.Sprintf("tcp:%s:6642", leaderIP),
@@ -2265,6 +2345,7 @@ func runCertRenew(cmd *cobra.Command, _ []string) {
 type configDirs struct {
 	AWSGW      string
 	Predastore string
+	Viperblock string
 	NATS       string
 	Spinifex   string
 }
@@ -2274,10 +2355,11 @@ func createConfigSubdirs(configDir string) (configDirs, error) {
 	dirs := configDirs{
 		AWSGW:      filepath.Join(configDir, "awsgw"),
 		Predastore: filepath.Join(configDir, "predastore"),
+		Viperblock: filepath.Join(configDir, "viperblock"),
 		NATS:       filepath.Join(configDir, "nats"),
 		Spinifex:   filepath.Join(configDir, "spinifex"),
 	}
-	for _, dir := range []string{dirs.AWSGW, dirs.Predastore, dirs.NATS, dirs.Spinifex} {
+	for _, dir := range []string{dirs.AWSGW, dirs.Predastore, dirs.Viperblock, dirs.NATS, dirs.Spinifex} {
 		if err := os.MkdirAll(dir, 0700); err != nil {
 			return configDirs{}, fmt.Errorf("create directory %s: %w", dir, err)
 		}
@@ -2399,6 +2481,43 @@ func writePredastoreEncryptionKey(configDir string) (string, error) {
 	return keyPath, nil
 }
 
+// writeViperblockEncryptionKey generates a fresh 32-byte AES-256 master key for
+// viperblock at-rest encryption and writes it to
+// <configDir>/viperblock/encryption.key with mode 0600. Unlike the predastore
+// key, this key is cluster-wide and shared: the leader generates it once at init
+// and distributes it to joiners via the formation server, so a volume sealed on
+// any node can be opened on any other. Loaded at startup via masterkey.LoadShared.
+func writeViperblockEncryptionKey(configDir string) (string, error) {
+	viperblockDir := filepath.Join(configDir, "viperblock")
+	if err := os.MkdirAll(viperblockDir, 0750); err != nil {
+		return "", fmt.Errorf("create viperblock config dir: %w", err)
+	}
+	key, err := handlers_iam.GenerateMasterKey()
+	if err != nil {
+		return "", fmt.Errorf("generate viperblock encryption key: %w", err)
+	}
+	keyPath := filepath.Join(viperblockDir, "encryption.key")
+	if err := handlers_iam.SaveMasterKey(keyPath, key); err != nil {
+		return "", fmt.Errorf("save viperblock encryption key: %w", err)
+	}
+	return keyPath, nil
+}
+
+// saveViperblockEncryptionKey writes an already-generated 32-byte viperblock
+// master key (received from the formation leader) to
+// <configDir>/viperblock/encryption.key with mode 0600.
+func saveViperblockEncryptionKey(configDir string, key []byte) (string, error) {
+	viperblockDir := filepath.Join(configDir, "viperblock")
+	if err := os.MkdirAll(viperblockDir, 0750); err != nil {
+		return "", fmt.Errorf("create viperblock config dir: %w", err)
+	}
+	keyPath := filepath.Join(viperblockDir, "encryption.key")
+	if err := handlers_iam.SaveMasterKey(keyPath, key); err != nil {
+		return "", fmt.Errorf("save viperblock encryption key: %w", err)
+	}
+	return keyPath, nil
+}
+
 // writeSystemCredentials writes the system access key to a plaintext JSON file.
 // The daemon reads this at startup to inject credentials into ALB VM cloud-init
 // for SigV4-authenticated communication with the AWS gateway.

cmd/spinifex/cmd/admin_test.go

@@ -6,11 +6,15 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"strings"
 	"testing"
 
+	s3 "github.com/mulgadc/predastore/s3"
 	"github.com/mulgadc/spinifex/spinifex/admin"
 	"github.com/mulgadc/spinifex/spinifex/formation"
 	"github.com/mulgadc/spinifex/spinifex/utils"
+	toml "github.com/pelletier/go-toml/v2"
+	"github.com/spf13/cobra"
 	"github.com/spf13/viper"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -206,7 +210,7 @@ func TestPredastoreMultinodeTemplate_NatsURLLoopbackShim(t *testing.T) {
 		{ID: 3, Host: "10.0.0.3"},
 	}
 	content, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "0.0.0.0",
+		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "0.0.0.0", 0,
 	)
 	require.NoError(t, err)
 	assert.Contains(t, content, `nats_url = "nats://localhost:4222"`)
@@ -214,7 +218,7 @@ func TestPredastoreMultinodeTemplate_NatsURLLoopbackShim(t *testing.T) {
 
 	// Specific bind IP → stays as-is.
 	content2, err := admin.GenerateMultiNodePredastoreConfig(
-		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "10.11.12.1",
+		predastoreMultiNodeTemplate, nodes, "AK", "SK", "ap-southeast-2", "token", "/tmp", "10.11.12.1", 0,
 	)
 	require.NoError(t, err)
 	assert.Contains(t, content2, `nats_url = "nats://10.11.12.1:4222"`)
@@ -252,6 +256,74 @@ func TestWritePredastoreEncryptionKey(t *testing.T) {
 	assert.NotEqual(t, key1, key2, "per-node keys must differ")
 }
 
+// The viperblock key gates at-rest encryption for every volume on a fresh
+// install. masterkey.LoadShared rejects anything that isn't exactly 32 bytes at
+// 0640-or-stricter, so a regression in size/mode would silently fall back to
+// cleartext (empty path) or fail service startup.
+func TestWriteViperblockEncryptionKey(t *testing.T) {
+	configDir := t.TempDir()
+
+	keyPath, err := writeViperblockEncryptionKey(configDir)
+	require.NoError(t, err)
+	assert.Equal(t, filepath.Join(configDir, "viperblock", "encryption.key"), keyPath)
+
+	info, err := os.Stat(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, int64(32), info.Size(), "viperblock master key must be exactly 32 bytes")
+	assert.Equal(t, os.FileMode(0600), info.Mode().Perm(), "viperblock master key must be mode 0600")
+}
+
+// Joiners persist the leader's shared key via saveViperblockEncryptionKey
+// against a configDir with no viperblock/ subdir yet. The bytes round-trip
+// verbatim — the whole cluster shares one key, so any mutation would orphan
+// volumes sealed on other nodes.
+func TestSaveViperblockEncryptionKey_RoundTrip(t *testing.T) {
+	configDir := t.TempDir()
+	key := make([]byte, 32)
+	for i := range key {
+		key[i] = byte(i)
+	}
+
+	keyPath, err := saveViperblockEncryptionKey(configDir, key)
+	require.NoError(t, err)
+	assert.Equal(t, filepath.Join(configDir, "viperblock", "encryption.key"), keyPath)
+
+	got, err := os.ReadFile(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, key, got, "shared key must be written verbatim")
+}
+
+// A fresh install must render encryption_key_file so viperblockd enables
+// at-rest encryption; an empty field (upgrade / legacy) must omit the line
+// entirely so the volume stays in cleartext legacy mode.
+func TestSpinifexTomlTemplate_EncryptionKeyFile(t *testing.T) {
+	base := admin.ConfigSettings{
+		Node: "node1", Az: "ap-southeast-2a", Port: "4432", Region: "ap-southeast-2",
+		BindIP: "0.0.0.0", AccessKey: "AKIATEST", SecretKey: "SECRET",
+		AccountID: "123456789012", NatsToken: "token",
+		OVNNBAddr: "tcp:127.0.0.1:6641", OVNSBAddr: "tcp:127.0.0.1:6642",
+	}
+
+	dir := t.TempDir()
+	withKey := base
+	withKey.ConfigDir = dir
+	withKey.EncryptionKeyFile = "/etc/spinifex/viperblock/encryption.key"
+	path := filepath.Join(dir, "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, withKey))
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	assert.Contains(t, string(data), `encryption_key_file = "/etc/spinifex/viperblock/encryption.key"`)
+
+	dir2 := t.TempDir()
+	noKey := base
+	noKey.ConfigDir = dir2
+	path2 := filepath.Join(dir2, "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path2, spinifexTomlTemplate, noKey))
+	data2, err := os.ReadFile(path2)
+	require.NoError(t, err)
+	assert.NotContains(t, string(data2), "encryption_key_file", "blank key must omit the field")
+}
+
 // Joiners run `spx admin join` against a configDir that doesn't yet contain
 // a predastore/ subdir. The helper must create it (predastore subdir is
 // owned by spinifex-storage in prod, but the helper just needs the dir to
@@ -386,3 +458,81 @@ func TestApplyNetworkConfig_PropagatesPoolBindBridge(t *testing.T) {
 	assert.Equal(t, "dhcp", settings.PoolSource)
 	assert.Equal(t, "br-wan", settings.PoolBindBridge)
 }
+
+func renderSingleNodePredastore(t *testing.T, settings admin.ConfigSettings) string {
+	t.Helper()
+	path := filepath.Join(t.TempDir(), "predastore.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, predastoreTomlTemplate, settings))
+	b, err := os.ReadFile(path)
+	require.NoError(t, err)
+	return string(b)
+}
+
+func predastoreMultinodeNodes() []admin.PredastoreNodeConfig {
+	return []admin.PredastoreNodeConfig{
+		{ID: 1, Host: "10.0.0.1"}, {ID: 2, Host: "10.0.0.2"}, {ID: 3, Host: "10.0.0.3"},
+	}
+}
+
+// Unset knob must leave production config byte-identical: no [compaction] block
+// and the original trailing bytes unchanged (single-node has no trailing
+// newline, multinode keeps one).
+func TestPredastoreTemplates_UnsetCompactionEmitsNoBlock(t *testing.T) {
+	single := renderSingleNodePredastore(t, admin.ConfigSettings{
+		Region: "ap-southeast-2", BindIP: "0.0.0.0", ConfigDir: "/cfg",
+		AccessKey: "AK", SecretKey: "SK", NatsToken: "tok",
+	})
+	assert.NotContains(t, single, "[compaction]")
+	assert.True(t, strings.HasSuffix(single, `access_keys_bucket = "spinifex-iam-access-keys"`),
+		"unset single-node tail changed: %q", single[len(single)-40:])
+
+	multi, err := admin.GenerateMultiNodePredastoreConfig(
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", 0,
+	)
+	require.NoError(t, err)
+	assert.NotContains(t, multi, "[compaction]")
+	assert.True(t, strings.HasSuffix(multi, "access_keys_bucket = \"spinifex-iam-access-keys\"\n"),
+		"unset multinode tail changed: %q", multi[len(multi)-40:])
+}
+
+func TestPredastoreTemplates_SetCompactionEmitsParseableBlock(t *testing.T) {
+	const interval = 30
+
+	single := renderSingleNodePredastore(t, admin.ConfigSettings{
+		Region: "ap-southeast-2", BindIP: "0.0.0.0", ConfigDir: "/cfg",
+		AccessKey: "AK", SecretKey: "SK", NatsToken: "tok", CompactionIntervalSeconds: interval,
+	})
+	assert.Contains(t, single, "[compaction]")
+	var singleCfg s3.Config
+	require.NoError(t, toml.Unmarshal([]byte(single), &singleCfg))
+	assert.Equal(t, interval, singleCfg.Compaction.IntervalSeconds)
+
+	multi, err := admin.GenerateMultiNodePredastoreConfig(
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok", "/cfg", "10.0.0.1", interval,
+	)
+	require.NoError(t, err)
+	var multiCfg s3.Config
+	require.NoError(t, toml.Unmarshal([]byte(multi), &multiCfg))
+	assert.Equal(t, interval, multiCfg.Compaction.IntervalSeconds)
+}
+
+// The init/join flag must register with an unset (0) default and flow into the
+// rendered config via ConfigSettings.
+func TestAdminInitFlag_CompactionIntervalReachesRenderedConfig(t *testing.T) {
+	for _, c := range []*cobra.Command{adminInitCmd, adminJoinCmd} {
+		f := c.Flags().Lookup("predastore-compaction-interval")
+		require.NotNil(t, f, "%s missing --predastore-compaction-interval", c.Name())
+		assert.Equal(t, "0", f.DefValue)
+	}
+
+	require.NoError(t, adminInitCmd.Flags().Set("predastore-compaction-interval", "45"))
+	defer func() { _ = adminInitCmd.Flags().Set("predastore-compaction-interval", "0") }()
+
+	v, _ := adminInitCmd.Flags().GetInt("predastore-compaction-interval")
+	require.Equal(t, 45, v)
+
+	out := renderSingleNodePredastore(t, admin.ConfigSettings{
+		Region: "ap-southeast-2", BindIP: "0.0.0.0", CompactionIntervalSeconds: v,
+	})
+	assert.Contains(t, out, "interval_seconds = 45")
+}

cmd/spinifex/cmd/cluster.go

@@ -8,6 +8,7 @@ import (
 	"time"
 
 	"github.com/mulgadc/spinifex/spinifex/daemon"
+	"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
 	"github.com/nats-io/nats.go"
 	"github.com/spf13/cobra"
 )
@@ -160,6 +161,79 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
 	fmt.Printf("Cluster shutdown complete (%s)\n", time.Since(start).Round(time.Millisecond))
 }
 
+// runClusterDrainDHCP asks every vpcd to DHCPRELEASE all external-pool leases
+// it currently holds, returning them to the upstream DHCP server. Intended for
+// the teardown path: an env reset otherwise strands held leases until TTL
+// because vpcd's Stop() preserves them for adopt-on-reboot. Best-effort — if
+// the cluster is already partly down it warns and exits 0 so teardown proceeds.
+func runClusterDrainDHCP(cmd *cobra.Command, args []string) {
+	timeout, _ := cmd.Flags().GetDuration("timeout")
+	if timeout <= 0 {
+		timeout = 30 * time.Second
+	}
+
+	_, nc, err := loadConfigAndConnect()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: drain-dhcp could not connect (cluster may be down): %v\n", err)
+		return
+	}
+	defer nc.Close()
+
+	total, responders := drainDHCPLeases(nc, timeout)
+	fmt.Printf("DHCP drain complete: released %d lease(s) from %d vpcd responder(s)\n", total, responders)
+}
+
+// drainDHCPLeases publishes a drain request to vpc.dhcp.drain and collects
+// replies until timeout — one responder per AZ (vpcd uses a per-AZ queue
+// group). Returns the summed released count and the number of responders.
+func drainDHCPLeases(nc *nats.Conn, timeout time.Duration) (released, responders int) {
+	reqData, err := json.Marshal(struct{}{})
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		return 0, 0
+	}
+
+	inbox := nats.NewInbox()
+	sub, err := nc.SubscribeSync(inbox)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: drain-dhcp subscribe failed: %v\n", err)
+		return 0, 0
+	}
+	defer sub.Unsubscribe()
+
+	if err := nc.PublishRequest(dhcp.TopicDrain, inbox, reqData); err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: drain-dhcp publish failed: %v\n", err)
+		return 0, 0
+	}
+	nc.Flush()
+
+	deadline := time.Now().Add(timeout)
+	for {
+		remaining := time.Until(deadline)
+		if remaining <= 0 {
+			break
+		}
+		msg, err := sub.NextMsg(remaining)
+		if err != nil {
+			break
+		}
+		var reply struct {
+			Released int    `json:"released"`
+			Error    string `json:"error,omitempty"`
+		}
+		if err := json.Unmarshal(msg.Data, &reply); err != nil {
+			continue
+		}
+		responders++
+		if reply.Error != "" {
+			fmt.Fprintf(os.Stderr, "  vpcd drain error: %s\n", reply.Error)
+			continue
+		}
+		released += reply.Released
+	}
+	return released, responders
+}
+
 // collectShutdownACKs publishes a shutdown request and collects ACKs from nodes.
 func collectShutdownACKs(nc *nats.Conn, topic string, reqData []byte, nodeCount int, timeout time.Duration) ([]daemon.ShutdownACK, error) {
 	inbox := nats.NewInbox()

cmd/spinifex/cmd/cluster_test.go

@@ -0,0 +1,58 @@
+package cmd
+
+import (
+	"encoding/json"
+	"testing"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
+	"github.com/mulgadc/spinifex/spinifex/testutil"
+	"github.com/nats-io/nats.go"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// fakeDrainResponder subscribes a stand-in vpcd to vpc.dhcp.drain that
+// replies with the given released count (or error), mirroring
+// Manager.handleDrainMsg without a real lease store.
+func fakeDrainResponder(t *testing.T, nc *nats.Conn, released int, errMsg string) {
+	t.Helper()
+	sub, err := nc.Subscribe(dhcp.TopicDrain, func(msg *nats.Msg) {
+		body, _ := json.Marshal(struct {
+			Released int    `json:"released"`
+			Error    string `json:"error,omitempty"`
+		}{Released: released, Error: errMsg})
+		_ = msg.Respond(body)
+	})
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = sub.Unsubscribe() })
+}
+
+func TestDrainDHCPLeasesSumsReleased(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+	fakeDrainResponder(t, nc, 7, "")
+
+	released, responders := drainDHCPLeases(nc, 2*time.Second)
+	assert.Equal(t, 7, released)
+	assert.Equal(t, 1, responders)
+}
+
+func TestDrainDHCPLeasesReportsResponderError(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+	fakeDrainResponder(t, nc, 0, "list leases for drain: boom")
+
+	// An erroring responder still counts as a responder but contributes
+	// zero released leases.
+	released, responders := drainDHCPLeases(nc, 2*time.Second)
+	assert.Equal(t, 0, released)
+	assert.Equal(t, 1, responders)
+}
+
+func TestDrainDHCPLeasesNoResponders(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+
+	// No vpcd subscribed: the collection window elapses with zero replies.
+	released, responders := drainDHCPLeases(nc, 200*time.Millisecond)
+	assert.Equal(t, 0, released)
+	assert.Equal(t, 0, responders)
+}

cmd/spinifex/cmd/templates/predastore-multinode.toml

@@ -75,3 +75,8 @@ nats_token = "{{.NatsToken}}"
 nats_ca_cert = "{{.ConfigDir}}/ca.pem"
 master_key_path = "{{.ConfigDir}}/master.key"
 access_keys_bucket = "spinifex-iam-access-keys"
+{{- if .CompactionIntervalSeconds}}
+
+[compaction]
+interval_seconds = {{.CompactionIntervalSeconds}}
+{{- end}}

cmd/spinifex/cmd/templates/predastore.toml

@@ -103,4 +103,9 @@ nats_url = "nats://{{if eq .BindIP "0.0.0.0"}}localhost{{else}}{{.BindIP}}{{end}
 nats_token = "{{.NatsToken}}"
 nats_ca_cert = "{{.ConfigDir}}/ca.pem"
 master_key_path = "{{.ConfigDir}}/master.key"
-access_keys_bucket = "spinifex-iam-access-keys"
+access_keys_bucket = "spinifex-iam-access-keys"
+{{- if .CompactionIntervalSeconds}}
+
+[compaction]
+interval_seconds = {{.CompactionIntervalSeconds}}
+{{- end}}

cmd/spinifex/cmd/templates/spinifex.toml

@@ -8,6 +8,10 @@ node = "{{.Node}}"
 email = "{{.OperatorEmail}}"
 {{- end}}
 
+[aws]
+region = "{{.Region}}"
+internal_suffix = "spinifex.internal"
+
 [network]
 ipsec_enabled = {{.IPSecEnabled}}
 {{- if .ExternalMode}}
@@ -93,6 +97,9 @@ token = "{{.NatsToken}}"
 
 [nodes.{{.Node}}.viperblock]
 shardwal = false
+{{- if .EncryptionKeyFile}}
+encryption_key_file = "{{.EncryptionKeyFile}}"
+{{- end}}
 
 [nodes.{{.Node}}.vpcd]
 ovn_nb_addr = "{{.OVNNBAddr}}"

docs/COMMANDS.md

@@ -65,6 +65,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 | Command | Flags | Description |
 |---------|-------|-------------|
 | `spx admin cluster shutdown` | `--force` (shutdown even if nodes don't respond), `--timeout` (max wait per phase, default 120s), `--dry-run` (print phase plan without executing) | Performs coordinated, phased shutdown of entire cluster. Phases execute in order: GATE (stop API/UI) → DRAIN (stop VMs) → STORAGE (stop viperblock) → PERSIST (stop predastore) → INFRA (stop NATS/daemon). Each phase waits for all nodes to ACK before proceeding. Uses JetStream state tracking. |
+| `spx admin cluster drain-dhcp` | `--timeout` (reply-collection window, default 30s) | Asks each vpcd to DHCPRELEASE every external-pool DHCP lease it currently holds, returning them to the upstream DHCP server. Run on teardown before stopping services — an env reset otherwise strands held leases upstream until their TTL expires, eventually exhausting the upstream scope. Best-effort: warns and exits 0 if the cluster is already down. |
 
 ### Certificate Management
 
@@ -112,7 +113,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
-| `run-instances` | `--image-id`, `--instance-type`, `--count`, `--key-name`, `--user-data`, `--subnet-id`, `--block-device-mappings` (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination), `--placement` (GroupName), `--iam-instance-profile` (Name/Arn) | `--security-group-ids`, `--tag-specifications`, `--dry-run`, `--client-token`, `--disable-api-termination`, `--ebs-optimized`, `--network-interfaces`, `--private-ip-address`, `--monitoring`, `--credit-specification`, `--cpu-options`, `--metadata-options`, `--launch-template`, `--hibernate-options` | **DONE** |
+| `run-instances` | `--image-id`, `--instance-type`, `--count`, `--key-name`, `--user-data`, `--subnet-id`, `--security-group-ids`, `--tag-specifications` (instance-scoped), `--block-device-mappings` (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination), `--placement` (GroupName), `--iam-instance-profile` (Name/Arn), `--capacity-reservation-specification` (CapacityReservationTarget.CapacityReservationId, targeted-by-id only) | `--dry-run`, `--client-token`, `--disable-api-termination`, `--ebs-optimized`, `--network-interfaces`, `--private-ip-address`, `--monitoring`, `--credit-specification`, `--cpu-options`, `--metadata-options`, `--launch-template`, `--hibernate-options` | **DONE** |
 | `describe-instances` | `--instance-ids`, `--filters` (instance-state-name, instance-id, instance-type, vpc-id, subnet-id, tag:*, tag-key, tag-value) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
 | `start-instances` | `--instance-ids` | `--dry-run`, `--force` | **DONE** |
 | `stop-instances` | `--instance-ids` | `--force`, `--hibernate`, `--dry-run` | **DONE** |
@@ -131,6 +132,11 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 role ARN and rejects cross-account profile ARNs. Placement strategies: `spread`
 (1 per node, atomic CAS) and `cluster` (pin all to one node).
 
+`run-instances --capacity-reservation-specification` consumes a reserved slot on
+the node owning the targeted reservation (targeted-by-id only; `open`
+auto-matching is not supported). The combination with `--placement` (group) is
+rejected. `describe-instances` then reports the instance's `CapacityReservationId`.
+
 ### EC2 — IAM Instance Profile Associations
 
 Stored as `vm.VM.IamInstanceProfileArn` + `IamInstanceProfileAssociationId`
@@ -183,7 +189,7 @@ input but treat it as `uefi` at launch.
 | `attach-volume` | `--volume-id`, `--instance-id`, `--device` (auto-assigns `/dev/sd[f-p]`) | `--dry-run` | **DONE** |
 | `detach-volume` | `--volume-id`, `--instance-id` (optional), `--device`, `--force` | `--dry-run` | **DONE** |
 | `describe-volume-status` | `--volume-ids`, `--filters` (volume-id, volume-status.status, availability-zone) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
-| `describe-volumes-modifications` | — | `--volume-ids`, `--filters`, `--max-results` | **NOT STARTED** |
+| `describe-volumes-modifications` | `--volume-ids`, `--filters` (modification-state, original-iops, original-size, original-volume-type, start-time, target-iops, target-size, target-volume-type, volume-id) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
 
 ### EC2 — Snapshots
 
@@ -432,9 +438,9 @@ EIP commands are only registered when an external IPAM pool is configured.
 | `delete-dhcp-options` | — | `--dhcp-options-id`, `--dry-run` | **NOT STARTED** |
 | `describe-dhcp-options` | — | `--dhcp-options-ids`, `--filters`, `--max-results` | **NOT STARTED** |
 | `associate-dhcp-options` | — | `--dhcp-options-id`, `--vpc-id`, `--dry-run` | **NOT STARTED** |
-| `create-capacity-reservation` | — | `--instance-type`, `--instance-platform`, `--availability-zone`, `--instance-count`, `--end-date`, `--end-date-type`, `--instance-match-criteria`, `--tag-specifications` | **NOT STARTED** |
-| `cancel-capacity-reservation` | — | `--capacity-reservation-id`, `--dry-run` | **NOT STARTED** |
-| `describe-capacity-reservations` | — | `--capacity-reservation-ids`, `--filters`, `--max-results` | **NOT STARTED** |
+| `create-capacity-reservation` | `--instance-type`, `--instance-count`, `--availability-zone`, `--instance-platform`, `--instance-match-criteria`, `--tenancy` (default only), `--dry-run` | `--end-date`, `--end-date-type` (unlimited only), `--availability-zone-id`, `--tag-specifications` | **DONE** |
+| `cancel-capacity-reservation` | `--capacity-reservation-id`, `--dry-run` | — | **DONE** |
+| `describe-capacity-reservations` | `--capacity-reservation-ids`, `--filters` | `--max-results`, `--next-token` | **DONE** |
 | `modify-capacity-reservation` | — | `--capacity-reservation-id`, `--instance-count`, `--end-date`, `--end-date-type` | **NOT STARTED** |
 
 ### EC2 — Misc
@@ -488,6 +494,7 @@ bypasses policy evaluation entirely.
 | `create-policy` | `--policy-name`, `--policy-document`, `--path`, `--description` | `--tags` | **DONE** |
 | `get-policy` | `--policy-arn` | — | **DONE** |
 | `get-policy-version` | `--policy-arn`, `--version-id` | — | **DONE** |
+| `list-policy-versions` | `--policy-arn` | `--max-items`, `--marker` | **DONE** |
 | `list-policies` | — | `--scope`, `--only-attached`, `--path-prefix`, `--max-items`, `--marker` | **DONE** |
 | `delete-policy` | `--policy-arn` | — | **DONE** |
 | `attach-user-policy` | `--user-name`, `--policy-arn` | — | **DONE** |
@@ -507,6 +514,8 @@ bypasses policy evaluation entirely.
 | `attach-role-policy` | `--role-name`, `--policy-arn` | — | **DONE** |
 | `detach-role-policy` | `--role-name`, `--policy-arn` | — | **DONE** |
 | `list-attached-role-policies` | `--role-name`, `--path-prefix` | `--max-items`, `--marker` | **DONE** |
+| `list-role-policies` | `--role-name` | `--max-items`, `--marker` | **DONE** (gateway stub — inline policies unsupported, always returns empty) |
+| `get-role-policy` / `put-role-policy` / `delete-role-policy` | — | inline role policies | **NOT STARTED** |
 
 ### IAM — Instance Profiles
 
@@ -520,6 +529,19 @@ bypasses policy evaluation entirely.
 | `add-role-to-instance-profile` | `--instance-profile-name`, `--role-name` | — | **DONE** |
 | `remove-role-from-instance-profile` | `--instance-profile-name`, `--role-name` | — | **DONE** |
 
+### IAM — OIDC Providers
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `create-open-id-connect-provider` | `--url`, `--client-id-list`, `--thumbprint-list`, `--tags` | — | **DONE** |
+| `get-open-id-connect-provider` | `--open-id-connect-provider-arn` | — | **DONE** |
+| `list-open-id-connect-providers` | — | — | **DONE** |
+| `delete-open-id-connect-provider` | `--open-id-connect-provider-arn` | — | **DONE** |
+| `add-client-id-to-open-id-connect-provider` | — | `--open-id-connect-provider-arn`, `--client-id` | **NOT STARTED** |
+| `remove-client-id-from-open-id-connect-provider` | — | `--open-id-connect-provider-arn`, `--client-id` | **NOT STARTED** |
+| `update-open-id-connect-provider-thumbprint` | — | `--open-id-connect-provider-arn`, `--thumbprint-list` | **NOT STARTED** |
+| `tag-open-id-connect-provider` / `untag-open-id-connect-provider` | — | `--open-id-connect-provider-arn`, `--tags`/`--tag-keys` | **NOT STARTED** |
+
 ### IAM — Groups
 
 | Command | Implemented Flags | Missing Flags | Status |
@@ -544,16 +566,13 @@ bypasses policy evaluation entirely.
 | `get-caller-identity` | — | — | **DONE** |
 | `assume-role` | `--role-arn`, `--role-session-name`, `--duration-seconds` (900–min(role MaxSessionDuration, 43200)) | `--policy`, `--policy-arns` (→ `PackedPolicyTooLarge`); `--tags`, `--transitive-tag-keys` (→ `InvalidParameterValue`); `--serial-number`, `--token-code` (→ `InvalidParameterValue`); `--external-id`, `--source-identity` (accepted and logged, **not enforced** — no Condition evaluator in v1) | **DONE** |
 | `get-session-token` | `--duration-seconds` (900–129600, default 43200 = 12h; clamped, not rejected) | `--serial-number`, `--token-code` (MFA → `InvalidParameterValue`) | **DONE** |
-| `assume-role-with-web-identity` | — | `--role-arn`, `--role-session-name`, `--web-identity-token`, `--provider-id`, `--policy`, `--policy-arns`, `--duration-seconds` | **NOT STARTED** |
+| `assume-role-with-web-identity` | `--role-arn`, `--role-session-name`, `--web-identity-token`, `--duration-seconds` (900–43200, default 3600) | `--provider-id`; `--policy`, `--policy-arns` (→ `PackedPolicyTooLarge`) | **DONE** |
 | `assume-role-with-saml` | — | `--role-arn`, `--principal-arn`, `--saml-assertion`, `--policy`, `--policy-arns`, `--duration-seconds` | **NOT STARTED** |
 | `get-access-key-info` | — | `--access-key-id` | **NOT STARTED** |
 | `get-federation-token` | — | `--name`, `--policy`, `--policy-arns`, `--duration-seconds`, `--tags` | **NOT STARTED** |
 | `decode-authorization-message` | — | `--encoded-message` | **NOT STARTED** |
 
-Trust policies stored on roles (`AssumeRolePolicyDocument`) reject `Condition`,
-`NotPrincipal`, `NotAction`, empty-string `Action` elements, and empty
-`Principal` blocks at write time (`MalformedPolicyDocument`) — v1 has no
-Condition evaluator so accepting them would silently allow.
+Trust policies stored on roles (`AssumeRolePolicyDocument`) reject `Condition`, `NotPrincipal`, `NotAction`, empty-string `Action` elements, and empty `Principal` blocks at write time (`MalformedPolicyDocument`) — v1 has no Condition evaluator so accepting them would silently allow.
 
 ---
 
@@ -579,24 +598,42 @@ curl -i http://169.254.169.254/latest/meta-data/instance-id
 | Path | Method | Source | Status |
 |------|--------|--------|--------|
 | `/latest/api/token` | PUT | Issues an ENI-bound IMDSv2 token; `X-aws-ec2-metadata-token-ttl-seconds` ∈ [1, 21600] required | **DONE** |
+| `/` | GET | Supported API-version list (`2021-07-15`, `latest`); token-gated | **DONE** |
+| `/latest` | GET | Top-level tree listing (`dynamic`, `meta-data`, `user-data`) | **DONE** |
+| `/<date>/...` | GET/PUT | Any dated API version aliases to `/latest` (cloud-init parity) | **DONE** |
 | `/latest/meta-data/` | GET | Directory listing of supported children | **DONE** |
 | `/latest/meta-data/instance-id` | GET | `vm.ID` | **DONE** |
 | `/latest/meta-data/instance-type` | GET | `vm.InstanceType` | **DONE** |
 | `/latest/meta-data/ami-id` | GET | launch `ImageId` | **DONE** |
+| `/latest/meta-data/ami-launch-index` | GET | Per-instance launch index (`0..n-1`), contiguous on partial failure | **DONE** |
+| `/latest/meta-data/reservation-id` | GET | `DescribeInstances` `Reservation.ReservationId` | **DONE** |
+| `/latest/meta-data/instance-life-cycle` | GET | `on-demand` (Spot not modelled) | **DONE** |
 | `/latest/meta-data/local-ipv4` | GET | `ENIRecord.PrivateIpAddress` (== request source IP) | **DONE** |
 | `/latest/meta-data/public-ipv4` | GET | EIP, else instance public IP; empty body if none | **DONE** |
+| `/latest/meta-data/public-hostname` | GET | Mirrors `public-ipv4`; 404 when no public IP | **DONE** |
 | `/latest/meta-data/mac` | GET | `ENIRecord.MacAddress` | **DONE** |
 | `/latest/meta-data/security-groups` | GET | `ENIRecord.SecurityGroupIds`, newline-separated | **DONE** |
 | `/latest/meta-data/hostname`, `/local-hostname` | GET | Synthesised `ip-<dashed-ip>.<region>.compute.internal` | **DONE** |
 | `/latest/meta-data/placement/availability-zone` | GET | `ENIRecord.AvailabilityZone` | **DONE** |
 | `/latest/meta-data/placement/region` | GET | Derived from AZ (trailing letter stripped) | **DONE** |
+| `/latest/meta-data/services/{domain,partition}` | GET | Static: `amazonaws.com` / `aws` | **DONE** |
 | `/latest/meta-data/iam/info` | GET | `{InstanceProfileArn, InstanceProfileId}`; 404 if no profile | **DONE** |
 | `/latest/meta-data/iam/security-credentials/` | GET | Role name(s) under the profile, one per line; empty body if none | **DONE** |
 | `/latest/meta-data/iam/security-credentials/<role>` | GET | STS `AssumeRoleForInstance` → ASIA-prefixed temporary credential JSON | **DONE** |
+| `/latest/meta-data/public-keys/` | GET | `0=<keyName>` from the launch key pair; 404 if none | **DONE** |
+| `/latest/meta-data/public-keys/0/` | GET | `openssh-key` (format list for index 0) | **DONE** |
+| `/latest/meta-data/public-keys/0/openssh-key` | GET | Launch SSH public key, live-fetched from the key store; 404 if the key was deleted, 500 on backend fault | **DONE** |
 | `/latest/user-data` | GET | `vm.UserData`; 404 if none | **DONE** |
-| `/latest/dynamic/instance-identity/document` | GET | Needs a per-cluster signing key | **NOT STARTED** (404; lands with EKS IRSA) |
-| `/latest/meta-data/network/interfaces/...` | GET | Multi-ENI rendering | **NOT STARTED** (404) |
-| `/latest/meta-data/tags/...` | GET | Instance-tag metadata | **NOT STARTED** (404) |
+| `/latest/dynamic` | GET | Lists `instance-identity/` | **DONE** |
+| `/latest/dynamic/instance-identity` | GET | Lists `document` (signed forms listed when the signing key lands) | **DONE** |
+| `/latest/dynamic/instance-identity/document` | GET | Unsigned identity document from resolved ENI + instance facts | **DONE** |
+| `/latest/dynamic/instance-identity/{signature,pkcs7,rsa2048}` | GET | Signed forms; need a per-cluster signing key | **NOT STARTED** (404; lands with EKS IRSA) |
+| `/latest/meta-data/network/interfaces/macs/<mac>/...` | GET | Multi-ENI rendering (subnet-id, vpc-id, ipv4s, security-group-ids, …) | **NOT STARTED** (404) |
+| `/latest/meta-data/tags/instance/<key>` | GET | Instance-tag metadata; gated on `InstanceMetadataTags` enablement | **NOT STARTED** (404) |
+| `/latest/meta-data/block-device-mapping/...` | GET | `ami`/`root`/`ebsN`/`ephemeralN` device map | **NOT STARTED** (404) |
+| `/latest/meta-data/placement/{group-name,partition-number,availability-zone-id,host-id}` | GET | Placement extras beyond `availability-zone`/`region` | **NOT STARTED** (404) |
+| `/latest/meta-data/instance-action` | GET | `none` unless interruptible instances ship | **NOT STARTED** (404) |
+| `/latest/meta-data/spot/{instance-action,termination-time}` | GET | Only meaningful once Spot is modelled | **NOT STARTED** (404) |
 
 ---
 
@@ -632,10 +669,54 @@ The data plane uses a system-managed LB VM, launched automatically during `creat
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
-| `create-listener` | `--load-balancer-arn`, `--default-actions` (Type=forward, TargetGroupArn), `--protocol` (HTTP), `--port` | `--ssl-policy`, `--certificates`, `--alpn-policy`, `--mutual-authentication`, `--dry-run` | **DONE** |
+| `create-listener` | `--load-balancer-arn`, `--default-actions` (Type=forward, TargetGroupArn), `--protocol` (HTTP/HTTPS/TLS), `--port`, `--certificates` (HTTPS/TLS), `--ssl-policy` | `--alpn-policy`, `--mutual-authentication`, `--dry-run` | **DONE** |
+| `modify-listener` | `--listener-arn`, `--protocol`, `--port`, `--default-actions`, `--certificates`, `--ssl-policy` | `--alpn-policy`, `--mutual-authentication`, `--dry-run` | **DONE** |
 | `delete-listener` | `--listener-arn` | `--dry-run` | **DONE** |
 | `describe-listeners` | `--load-balancer-arn`, `--listener-arns` | `--page-size`, `--marker`, `--dry-run` | **DONE** |
 
+### ELBv2 — Listener Rules
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `create-rule` | `--listener-arn`, `--priority` (1–50000), `--conditions` (host-header, path-pattern, http-header, http-request-method, query-string, source-ip), `--actions` (forward, redirect, fixed-response), `--tags` | `--dry-run` | **DONE** |
+| `modify-rule` | `--rule-arn`, `--conditions`, `--actions` | `--dry-run` | **DONE** |
+| `delete-rule` | `--rule-arn` | `--dry-run` | **DONE** |
+| `describe-rules` | `--listener-arn`, `--rule-arns` | `--page-size`, `--marker` (parsed, not enforced) | **DONE** |
+| `set-rule-priorities` | `--rule-priorities` (RuleArn, Priority) | — | **DONE** |
+
+A synthetic `default` rule is derived from the listener's `DefaultActions`.
+
+### ELBv2 — Listener Certificates & SSL Policies
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `add-listener-certificates` | `--listener-arn`, `--certificates` | `--dry-run` | **DONE** |
+| `remove-listener-certificates` | `--listener-arn`, `--certificates` | `--dry-run` | **DONE** |
+| `describe-listener-certificates` | `--listener-arn` | `--page-size`, `--marker` | **DONE** |
+| `describe-ssl-policies` | `--names` | `--load-balancer-type`, `--page-size`, `--marker` | **DONE** (static catalog: `ELBSecurityPolicy-FS-1-2-Res-2019-08`, `ELBSecurityPolicy-TLS13-1-2-2021-06` — metadata only, no in-platform TLS termination) |
+
+The default certificate cannot be added/removed via these calls — set it on the listener.
+
+### ELBv2 — Attributes & Modify
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `describe-load-balancer-attributes` | `--load-balancer-arn` | — | **DONE** (stored values over per-type defaults) |
+| `modify-load-balancer-attributes` | `--load-balancer-arn`, `--attributes` | — | **DONE** (unknown keys → `ValidationError`) |
+| `describe-target-group-attributes` | `--target-group-arn` | — | **DONE** |
+| `modify-target-group-attributes` | `--target-group-arn`, `--attributes` (incl. `deregistration_delay.timeout_seconds`, `stickiness.*`) | — | **DONE** |
+| `modify-target-group` | `--target-group-arn`, `--health-check-*`, `--matcher` | `--target-type`/`--protocol`/`--vpc-id` (immutable) | **DONE** |
+| `describe-listener-attributes` | `--listener-arn` | — | **DONE** (stub — returns empty; not persisted) |
+| `modify-listener-attributes` | `--listener-arn`, `--attributes` | — | **DONE** (stub — echoes input; not persisted) |
+
+### ELBv2 — Network & Security
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `set-security-groups` | `--load-balancer-arn`, `--security-groups` | `--enforce-security-group-inbound-rules-on-private-link-traffic` | **DONE** (ALB only — NLB → `InvalidConfigurationRequest`) |
+| `set-subnets` | `--load-balancer-arn`, `--subnets`, `--subnet-mappings` | `--ip-address-type` | **DONE** (live ENI add/remove with rollback) |
+| `set-ip-address-type` | `--load-balancer-arn`, `--ip-address-type` (ipv4) | dualstack/IPv6 (rejected) | **DONE** |
+
 ### ELBv2 — Tags
 
 | Command | Implemented Flags | Missing Flags | Status |
@@ -646,18 +727,32 @@ The data plane uses a system-managed LB VM, launched automatically during `creat
 
 | Feature | Priority | Status |
 |---------|----------|--------|
-| Listener rules (CreateRule, DeleteRule, DescribeRules, ModifyRule) | High | **NOT STARTED** |
-| HTTPS/TLS termination (SSL certificates, policies, ALPN) | High | **NOT STARTED** |
-| Modify operations (`ModifyLoadBalancerAttributes`, `ModifyTargetGroup`, `ModifyTargetGroupAttributes`, `ModifyListener`) | Medium | **NOT STARTED** |
-| Connection draining (deregistration delay) | Medium | **NOT STARTED** |
-| Stickiness / session affinity | Medium | **NOT STARTED** |
-| Active health checking (API-driven, vs. HAProxy-only today) | Medium | **NOT STARTED** |
+| In-platform HTTPS/TLS termination (cert + SSL-policy APIs exist; data-plane TLS not terminated) | High | **NOT STARTED** |
+| ALPN policy, mutual TLS (mTLS) | Medium | **NOT STARTED** |
+| Listener attribute persistence (`Describe/ModifyListenerAttributes` are stubs) | Medium | **NOT STARTED** |
+| Active health checking (API-driven, vs. HAProxy/nginx-only today) | Medium | **NOT STARTED** |
 | IP and Lambda target types | Low | **NOT STARTED** |
 | S3 access log delivery | Low | **NOT STARTED** |
 | WAF integration | Low | **NOT STARTED** |
 
 ---
 
+## ACM (AWS Certificate Manager)
+
+Import-only — Spinifex stores externally-issued certificates for ELBv2 listener references; it does not issue certificates or validate domains (`RequestCertificate`). Certs are account-scoped; `describe`/`delete` enforce ownership.
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `import-certificate` | `--certificate`, `--private-key`, `--certificate-chain`, `--certificate-arn` (re-import) | `--tags` | **DONE** |
+| `describe-certificate` | `--certificate-arn` | — | **DONE** |
+| `list-certificates` | — | `--certificate-statuses`, `--includes`, `--max-items`, `--next-token` | **DONE** |
+| `delete-certificate` | `--certificate-arn` | — | **DONE** |
+| `request-certificate` | — | `--domain-name`, `--validation-method`, `--subject-alternative-names`, `--tags` | **NOT STARTED** |
+| `add-tags-to-certificate` / `list-tags-for-certificate` / `remove-tags-from-certificate` | — | `--certificate-arn`, `--tags`/`--tag-keys` | **NOT STARTED** |
+| `export-certificate` | — | `--certificate-arn`, `--passphrase` | **NOT STARTED** |
+
+---
+
 ## CloudWatch (Basic Monitoring)
 
 | Command | Implemented Flags | Missing Flags | Status |

docs/administration/spinifex-admin-cli/README.md

@@ -95,13 +95,6 @@ checksum, and registers it tagged `spinifex:managed-by=eks`. `eks create-cluster
 and `eks create-nodegroup` resolve the boot AMI by that tag, so no further
 configuration is needed.
 
-Operators publishing a freshly built node image to the mirror run the
-**Publish EKS node AMI** GitHub Actions workflow (manual dispatch; uses the
-`CF_R2_*` repo secrets) or, locally, `make publish-eks-node-image` (see
-`scripts/publish-system-image.sh`; requires `R2_ENDPOINT` + R2-scoped
-`AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`).
-
-
 ## Cluster Shutdown
 
 Coordinated, phased shutdown of the entire cluster (API/UI → VMs → viperblock → predastore → NATS/daemon):

docs/containers/ecr/README.md

@@ -0,0 +1,188 @@
+---
+title: "ECR (Container Registry)"
+description: "Store and serve container images from Spinifex's AWS-compatible Elastic Container Registry — create a repository, authenticate Docker, push and pull images, and let your EKS workers pull from it, using the AWS CLI, the Spinifex console, or Terraform."
+category: "Containers"
+tags:
+  - ecr
+  - containers
+  - docker
+  - registry
+  - oci
+sections:
+  - overview
+  - prerequisites
+  - instructions
+  - troubleshooting
+resources:
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "Terraform AWS Provider — ECR"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ecr_repository"
+  - title: "OCI Distribution Spec"
+    url: "https://github.com/opencontainers/distribution-spec"
+  - title: "Docker CLI Reference"
+    url: "https://docs.docker.com/reference/cli/docker/"
+---
+
+# ECR (Container Registry)
+
+> Spinifex's ECR service is an AWS-compatible, OCI-compliant container registry. You create repositories with `aws ecr`, authenticate Docker with a registry token, and `docker push` / `docker pull` exactly as you would on AWS — and your EKS clusters can pull from it directly.
+
+## Overview
+
+ECR on Spinifex serves a private container registry on the Spinifex gateway endpoint — the same `host:9999` you already use for the AWS API. Authentication is account-scoped by token, so you push and pull at the gateway host and never need extra DNS:
+
+```
+<spinifex-gateway-host>:9999/<repository>
+```
+
+`aws ecr describe-repositories` and `aws ecr get-login-password` return the exact host to use — always prefer that value. Where real DNS is configured, the AWS-parity per-account host `<account-id>.dkr.ecr.<region>.<your-spinifex-domain>:9999` also works.
+
+Each repository holds image manifests and their layers. Images are stored in object storage ([Predastore](https://github.com/mulgadc/predastore)) in a per-account bucket, with blobs content-addressed and de-duplicated across repositories. Authentication is token-based: `aws ecr get-login-password` mints a short-lived bearer token that `docker login` uses against the registry's `/v2/` endpoint.
+
+**What works today**
+
+- Repository lifecycle — `CreateRepository`, `DeleteRepository`, `DescribeRepositories`.
+- OCI push and pull — blobs, manifests, and tags, including cross-repository blob mounts.
+- Token auth — `GetAuthorizationToken` plus bearer/basic auth on `/v2/`.
+- Repository policies — set, get, and delete a repository policy document.
+
+**Current limitations**
+
+- **Image scanning is not supported** — scan APIs return an explicit "operation not supported" error.
+- **No automatic garbage collection yet** — deleting images does not yet reclaim underlying blobs.
+- Lifecycle-policy enforcement and tag-immutability enforcement are not active.
+
+## Prerequisites
+
+- **Spinifex running**, with the AWS CLI configured for the `spinifex` profile (see [Installing Spinifex](/docs/install)).
+- **Docker** installed locally to build, push, and pull images.
+- **Docker trusting the Spinifex CA.** The registry serves TLS signed by the Spinifex local CA. Install it into the host trust store once (Docker uses the host root pool for registry TLS, so this covers every account endpoint without per-host config):
+
+```bash
+sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-local-ca.crt
+sudo update-ca-certificates
+sudo systemctl restart docker
+```
+
+- **The registry endpoint** — the Spinifex gateway `host:9999`. `aws ecr get-login-password` and `aws ecr describe-repositories` return the exact `host:port`; copy it from there rather than constructing it by hand.
+- **For EKS pulls:** worker nodes need the `AmazonEC2ContainerRegistryReadOnly` policy on their node IAM role (the EKS prerequisites already include this) and network egress to the registry endpoint.
+
+## Instructions
+
+Create a repository and push an image using your preferred tool.
+
+:::tabs
+@tab AWS CLI
+
+### 1. Create a repository
+
+```bash
+export AWS_PROFILE=spinifex
+
+aws ecr create-repository --repository-name my-app
+aws ecr describe-repositories --repository-names my-app \
+  --query 'repositories[0].repositoryUri'
+```
+
+The `repositoryUri` is the value you tag and push to — the gateway `host:9999`
+with the repository path, port included (for example `192.0.2.10:9999/my-app`).
+Use it verbatim; don't hand-build a hostname.
+
+### 2. Authenticate Docker
+
+Take the registry host straight from the API so the `:9999` port is correct:
+
+```bash
+REGISTRY=$(aws ecr describe-repositories --repository-names my-app \
+  --query 'repositories[0].repositoryUri' --output text | cut -d/ -f1)
+
+aws ecr get-login-password --region <region> \
+  | docker login --username AWS --password-stdin "$REGISTRY"
+```
+
+### 3. Build, tag, and push
+
+```bash
+docker build -t my-app:latest .
+docker tag my-app:latest "$REGISTRY/my-app:latest"
+docker push "$REGISTRY/my-app:latest"
+```
+
+### 4. Verify and pull
+
+```bash
+aws ecr describe-images --repository-name my-app
+docker pull "$REGISTRY/my-app:latest"
+```
+
+@tab Spinifex UI
+
+From the left navigation open **ECR → Repositories**.
+
+<img src="ecr-list-repositories.png" alt="The ECR Repositories list in the Spinifex console, showing a repository with its registry URI, tag mutability, and creation date">
+
+### 1. Create a repository
+
+1. Click **Create Repository**.
+2. Enter the **repository name** and, optionally, set **tag mutability** (mutable or immutable).
+3. Submit — the repository appears in the list with its full registry URI.
+
+### 2. Copy the push commands
+
+Open the repository to see its detail page. The **Push commands** panel lists the exact `aws ecr get-login-password | docker login`, `docker build`, `docker tag`, and `docker push` commands, pre-filled with your registry host and repository URI — use the **Copy** button and run them locally.
+
+<img src="ecr-push-commands.png" alt="A repository detail page in the Spinifex console, with the Push commands panel showing the docker login, build, tag, and push commands">
+
+### 3. Manage the repository
+
+- The **Permissions** tab edits the repository's access policy.
+- The **Lifecycle** tab manages retention rules, and **Tag immutability** can be toggled from the detail page.
+- The **Scan** tab reflects that image scanning is not supported on Spinifex.
+- Delete a repository from the **Delete** action in the repositories list.
+
+@tab Terraform
+
+Repositories can be managed as code with the standard `aws_ecr_repository` resource, with the AWS provider pointed at Spinifex's `ecr` endpoint:
+
+```hcl
+provider "aws" {
+  region                      = "ap-southeast-2"
+  skip_credentials_validation = true
+  skip_requesting_account_id  = true
+
+  endpoints {
+    ecr = "https://<your-spinifex-gateway>"
+  }
+}
+
+resource "aws_ecr_repository" "my_app" {
+  name = "my-app"
+}
+
+output "repository_url" {
+  value = aws_ecr_repository.my_app.repository_url
+}
+```
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+```
+
+Then authenticate Docker and push using the `repository_url` output, following steps 2–4 in the AWS CLI tab.
+
+:::
+
+## Troubleshooting
+
+**`docker login` fails with an authorization error.** The token is account-scoped and short-lived. Re-run `aws ecr get-login-password` to mint a fresh token, and confirm the registry host in `docker login` matches your account's endpoint (`aws ecr describe-repositories` returns it).
+
+**`docker push` cannot reach the registry.** Use the exact host from `aws ecr describe-repositories` — the Spinifex gateway `host:9999`. Docker dials it directly, so the `:9999` port must be present (without it docker tries `:443` and fails). If you are using the AWS-parity `<account-id>.dkr.ecr…` hostname instead, that path needs DNS resolving it to the gateway — prefer the gateway host the API returns.
+
+**EKS workers cannot pull an image.** Confirm the node IAM role has `AmazonEC2ContainerRegistryReadOnly` and that the workers have egress to the registry endpoint (an Internet Gateway or NAT Gateway route). See the [EKS prerequisites](/docs/eks).
+
+**Image-scanning commands return an error.** Image scanning is not supported on Spinifex; the scan APIs intentionally reject these calls. Remove scan-on-push configuration from your tooling.
+
+**A pushed image still appears after deletion frees no space.** Garbage collection of unreferenced blobs is not yet automatic — deleting an image removes its manifest and tags but does not immediately reclaim the underlying layers.

docs/containers/eks/README.md

@@ -0,0 +1,248 @@
+---
+title: "EKS (Managed Kubernetes)"
+description: "Run a managed, AWS-compatible Kubernetes cluster on Spinifex — provision the control plane and worker node group, wire up the VPC, IAM, and security groups it needs, and deploy your first workload via the AWS CLI, the Spinifex console, or Terraform."
+category: "Containers"
+tags:
+  - eks
+  - kubernetes
+  - containers
+  - vpc
+  - iam
+sections:
+  - overview
+  - prerequisites
+  - instructions
+  - troubleshooting
+resources:
+  - title: "EKS Quickstart Workbook"
+    url: "https://github.com/mulgadc/spinifex/tree/main/docs/terraform-workbooks/eks-quickstart"
+  - title: "EKS HTTPS Ingress Workbook (LBC + ACM)"
+    url: "https://github.com/mulgadc/spinifex/tree/main/docs/terraform-workbooks/eks-https-ingress"
+  - title: "EKS GitOps Workbook (Argo CD + EBS-CSI)"
+    url: "https://github.com/mulgadc/spinifex/tree/main/docs/terraform-workbooks/eks-gitops-argocd"
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "kubectl Documentation"
+    url: "https://kubernetes.io/docs/reference/kubectl/"
+---
+
+# EKS (Managed Kubernetes)
+
+> Spinifex's EKS service gives you a managed, AWS-compatible Kubernetes API: you create a cluster and a node group, and Spinifex provisions and runs the control plane and workers for you. Standard tools — `aws eks`, `kubectl`, the Terraform AWS provider — work unchanged against your cluster.
+
+## Overview
+
+An EKS cluster on Spinifex has two parts: a **control plane** (the managed Kubernetes API server, provisioned on a Spinifex-managed VM and fronted by a network load balancer) and one or more **managed node groups** of worker VMs that run your pods. You provide the VPC, subnets, and IAM roles; Spinifex creates and reconciles the cluster, its load balancer, and its security groups.
+
+Authentication uses **EKS access entries** (the API authentication mode) — IAM principals are mapped to Kubernetes RBAC groups, and `kubectl` authenticates with `aws eks get-token` exactly as it does on AWS.
+
+**What you'll create:**
+
+| Resource | Purpose |
+|---|---|
+| VPC + subnets | The network the cluster and workers run in |
+| Internet Gateway / NAT Gateway | Egress so workers can pull container images |
+| Cluster IAM role | Lets the control plane manage cluster resources |
+| Node IAM role | Lets workers join the cluster and pull images |
+| EKS cluster | The managed Kubernetes API endpoint |
+| Managed node group | The worker VMs that run your pods |
+| Security-group rule | Opens your app's port to reach the workers |
+
+**Spinifex specifics**
+
+- **Auth mode is API-only.** `authentication_mode` must be `API`; the legacy `CONFIG_MAP` and `API_AND_CONFIG_MAP` modes are rejected. Grant access with access entries, not an `aws-auth` ConfigMap.
+- **Security groups are auto-managed.** Spinifex creates the cluster and node-group security groups deterministically (e.g. `eks-cluster-<name>-nodegroup-sg`); `vpc_config.security_group_ids` is ignored. To expose a workload you add an ingress rule to the auto-managed node-group SG.
+- **The worker image is fixed.** Node groups always boot Spinifex's `eks-node` image. `ami_type` is recorded but does not select the image, and the image must be registered on the cluster before you create a node group.
+- **Default Kubernetes version is `1.32`.**
+
+## Prerequisites
+
+Before creating a cluster you need the supporting infrastructure in place. The Terraform workbooks build all of this for you; if you are using the CLI or console, create it first.
+
+- **Spinifex running**, with the AWS CLI configured for the `spinifex` profile (see [Installing Spinifex](/docs/install)) and `kubectl` installed.
+- **The `eks-node` image** registered on the cluster. The console blocks cluster creation until it is present.
+- **A VPC with at least one subnet.** The control plane launches in your first subnet; workers run in the subnets you pass to the node group.
+- **Egress for the workers.** Workers must reach the internet to pull container images:
+  - **Public workers** — an **Internet Gateway** with a default route (`0.0.0.0/0 → IGW`) and public IPs.
+  - **Private workers** — a **NAT Gateway** in a public subnet with a default route (`0.0.0.0/0 → NAT`) from the private subnets.
+- **Two IAM roles:**
+  - A **cluster role** trusted by `eks.amazonaws.com` with `AmazonEKSClusterPolicy` attached.
+  - A **node role** trusted by `ec2.amazonaws.com` with `AmazonEKSWorkerNodePolicy`, `AmazonEKS_CNI_Policy`, and `AmazonEC2ContainerRegistryReadOnly` attached (the last lets workers pull images from [ECR](/docs/ecr)).
+
+## Instructions
+
+The same cluster can be created three ways. Pick your tool — each path reaches the same working cluster.
+
+:::tabs
+@tab AWS CLI
+
+### 1. Create the cluster
+
+Point the cluster at your VPC subnets and the cluster role, with API authentication and a public endpoint:
+
+```bash
+export AWS_PROFILE=spinifex
+
+aws eks create-cluster \
+  --name demo \
+  --role-arn arn:aws:iam::000000000000:role/eks-cluster-role \
+  --resources-vpc-config subnetIds=subnet-aaaa,subnet-bbbb,endpointPublicAccess=true \
+  --access-config authenticationMode=API \
+  --kubernetes-version 1.32
+```
+
+Wait for it to become active:
+
+```bash
+aws eks wait cluster-active --name demo
+aws eks describe-cluster --name demo --query 'cluster.status'
+```
+
+### 2. Add a node group
+
+```bash
+aws eks create-nodegroup \
+  --cluster-name demo \
+  --nodegroup-name default \
+  --node-role-arn arn:aws:iam::000000000000:role/eks-node-role \
+  --subnets subnet-aaaa subnet-bbbb \
+  --scaling-config minSize=1,maxSize=2,desiredSize=1 \
+  --instance-types t3.medium
+
+aws eks wait nodegroup-active --cluster-name demo --nodegroup-name default
+```
+
+### 3. Connect kubectl
+
+```bash
+aws eks update-kubeconfig --name demo
+kubectl get nodes
+```
+
+### 4. Expose a workload
+
+Workers join an auto-managed security group that only allows intra-cluster traffic. To reach a NodePort service, add one ingress rule to that SG (look it up by its deterministic name):
+
+```bash
+SG=$(aws ec2 describe-security-groups \
+  --filters Name=group-name,Values=eks-cluster-demo-nodegroup-sg \
+  --query 'SecurityGroups[0].GroupId' --output text)
+
+aws ec2 authorize-security-group-ingress \
+  --group-id "$SG" --protocol tcp --port 30080 --cidr 0.0.0.0/0
+```
+
+Then deploy and publish your app:
+
+```bash
+kubectl create deployment hello --image=nginxdemos/hello --replicas=2
+kubectl expose deployment hello --type=NodePort --port=80 \
+  --overrides='{"spec":{"ports":[{"port":80,"nodePort":30080}]}}'
+```
+
+@tab Spinifex UI
+
+The Spinifex console drives the same workflow through a guided form. From the left navigation open **EKS → Clusters**.
+
+<img src="eks-list-clusters.png" alt="The EKS Clusters list in the Spinifex console, with a Create Cluster button">
+
+### 1. Create the cluster
+
+1. Click **Create Cluster**.
+2. Enter a **name** and choose the **Kubernetes version** (default `1.32`).
+3. Choose the **cluster IAM role**, or create one inline from the dialog.
+4. Select the **VPC and subnets** the cluster will run in.
+5. Set **endpoint access** (public, private, or both) and submit.
+
+<img src="eks-create-cluster.png" alt="The Create Cluster form in the Spinifex console, showing name, Kubernetes version, IAM role, VPC, subnets, and endpoint access fields">
+
+The form notes that the authentication mode is API and that the legacy aws-auth ConfigMap is not supported. The cluster appears in the **Clusters** list as `CREATING` and flips to `ACTIVE` once the control plane is healthy.
+
+### 2. Add a node group
+
+1. Open the cluster and switch to the **Node groups** tab.
+2. Click **Add node group**, choose the **node IAM role**, **subnets**, **instance type**, and **scaling** (desired / min / max).
+3. Submit and wait for the node group to reach `ACTIVE`.
+
+### 3. Manage access and addons
+
+- The **Access** tab lists IAM principals and their cluster-access policies — add an access entry to let another user or role reach the cluster.
+- The **Addons** tab installs and shows the health of managed addons.
+- The **Networking** tab shows the cluster's VPC, subnets, endpoint access, and the auto-managed security groups.
+
+To run `kubectl` against the cluster, follow the AWS CLI tab's `update-kubeconfig` step.
+
+@tab Terraform
+
+The Terraform workbooks are the fastest way to a working cluster — they build the VPC, IAM roles, cluster, node group, and a demo workload in one `apply`. Start with **eks-quickstart**:
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-quickstart
+```
+
+The cluster config points the AWS provider at Spinifex's `eks`, `ec2`, `iam`, and `sts` endpoints, then creates the cluster and node group:
+
+```hcl
+resource "aws_eks_cluster" "this" {
+  name     = "eks-quickstart"
+  role_arn = aws_iam_role.cluster.arn
+  version  = "1.32"
+
+  access_config {
+    authentication_mode = "API"
+  }
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.a.id, aws_subnet.b.id]
+    endpoint_public_access  = true
+  }
+}
+
+resource "aws_eks_node_group" "default" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "default"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.a.id, aws_subnet.b.id]
+
+  scaling_config {
+    desired_size = 1
+    min_size     = 1
+    max_size     = 2
+  }
+
+  instance_types = ["t3.medium"]
+}
+```
+
+Apply it, then connect:
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+aws eks update-kubeconfig --name eks-quickstart
+kubectl get nodes
+```
+
+The workbooks form a ladder. **eks-quickstart** is a NodePort demo; **eks-https-ingress** adds private subnets, a NAT gateway, and HTTPS via the AWS Load Balancer Controller + ACM; **eks-gitops-argocd** delivers the app from a git repo with the Argo CD addon and persists state on an EBS-CSI (Viperblock-backed) volume. See the full templates linked in Resources.
+
+:::
+
+## Troubleshooting
+
+**Cluster stuck in `CREATING`.** The control plane is still booting. Check `aws eks describe-cluster --name <name> --query 'cluster.health'` for reported issues. The control plane needs egress to bootstrap, so confirm its subnet has an Internet Gateway route.
+
+**`kubectl` cannot connect.** Re-run `aws eks update-kubeconfig --name <name>` to refresh the endpoint and CA, and confirm the cluster is `ACTIVE`. `aws sts get-caller-identity` verifies your credentials reach the Spinifex endpoint.
+
+**Workers stay `NotReady`.** Almost always an egress or IAM problem:
+- Confirm the workers can pull images — a default route to an Internet Gateway (public) or NAT Gateway (private).
+- Confirm the node role has `AmazonEKSWorkerNodePolicy`, `AmazonEKS_CNI_Policy`, and `AmazonEC2ContainerRegistryReadOnly`.
+
+**Workload unreachable from outside the cluster.** The node-group security group only allows intra-cluster traffic by default. Add an ingress rule for your NodePort to `eks-cluster-<name>-nodegroup-sg`, as shown in the CLI tab.
+
+**`create-cluster` rejected with an authentication-mode error.** `authentication_mode` must be `API`. Remove any `CONFIG_MAP` / `API_AND_CONFIG_MAP` setting and grant access with access entries instead.
+
+**Addon install fails.** Only addons bundled into the registered `eks-node` image install successfully. If you pin an addon version in Terraform, leave `addon_version` unset so Spinifex selects the catalog default.

docs/reference-architectures/SMCI-H14-orchestration/README.md

@@ -26,6 +26,8 @@ sections:
 
 ## Overview
 
+Spinifex is an open-source infrastructure platform that brings core AWS services including EC2, EBS and S3 to bare-metal, edge, and on-prem environments. It exposes an EC2-compatible API, so any tooling that works against AWS (the `aws` CLI, Terraform, SDKs) works against a Spinifex node unchanged, with a single profile swap.
+
 This document serves as a tutorial for setting up multi-tenant AI workloads with Spinifex. The reference architecture for this tutorial is the Supermicro H14 platform - specifications listed below.
 
 The H14 chassis comes equipped with **8× AMD Instinct MI350X** (288 GB HBM3e each, 2.3 TB total). We used Spinifex to provision three isolated VMs, each assigned 2× MI350X via direct PCIe passthrough, and ran three computationally intensive workloads simultaneously: a YOLO11x vision model, a Qwen3-VL 235B FP8 vision language model, and a Llama 3.3 70B large language model.
@@ -55,9 +57,9 @@ GPU passthrough requires a host kernel and OS that supports vfio-pci. Ubuntu 24.
 ## Prerequisites
 
 - Supermicro H14 with 8× AMD Instinct MI350X installed
-- Host OS: **Ubuntu 24.04 LTS** (kernel 6.8+) or **Debian 13** (kernel 6.12+) — minimum for vfio-pci support
+- Host OS: **Ubuntu 26.04 LTS** (kernel 6.8+) or **Debian 13** (kernel 6.12+) — minimum for vfio-pci support
 - Spinifex installed and all services running (`systemctl status spinifex.target`)
-- AMD GPU AMI registered (`ami-ubuntu-amd-gpu`) — Ubuntu 26.04 LTS with ROCm-compatible kernel
+- AMD GPU AMI registered (`ubuntu-26.04-amd-gpu-x86_64`) — Ubuntu 26.04 LTS with ROCm-compatible kernel
 - AWS CLI configured with `AWS_PROFILE=spinifex` pointing at the Spinifex endpoint
 - SSH key pair imported into Spinifex, VPC and security group created (see [Launching Instances](/docs/launching-instances))
 

docs/reference-architectures/SMCI-MIG-orchestration/README.md

@@ -0,0 +1,556 @@
+---
+title: "Mixed AI Workloads on a Single H200 Chassis: Guest-Managed MIG"
+description: "Install Spinifex from source, configure host-local networking, attach Predastore storage, and run four concurrent AI workloads across MIG partitions managed inside guest VMs."
+category: "Reference Architectures"
+tags:
+  - nvidia
+  - h200
+  - mig
+  - vllm
+  - yolo
+  - predastore
+  - bare-metal
+resources:
+  - title: "Install from Source"
+    url: "/docs/install-source"
+  - title: "VPC Networking"
+    url: "/docs/vpc-networking"
+  - title: "NVIDIA MIG User Guide"
+    url: "https://docs.nvidia.com/datacenter/tesla/mig-user-guide/"
+sections:
+  - overview
+  - prerequisites
+  - instructions
+---
+
+## Overview
+
+Spinifex is an open-source infrastructure platform that brings core AWS services including EC2, EBS and S3 to bare-metal, edge, and on-prem environments. It exposes an AWS compatible API, so any tooling that works against AWS (the `aws` CLI, Terraform, SDKs) works against a Spinifex node unchanged, with a single profile swap.
+
+This guide documents a full bare-metal AI deployment on a single [Supermicro X13](https://www.supermicro.com/en/products/x13) chassis using NVIDIA H200 SXM GPUs. These GPUs include the [NVIDIA Multi-Instance GPU](https://www.nvidia.com/en-au/technologies/multi-instance-gpu/) (MIG) capability, which allows a single GPU to be "sliced" into up to seven independent GPU partitions that are hardware isolated. Each partition is capable of running its own workloads with reserved resources from the "host" GPU.
+
+<p align="center"><img src="../../../.github/assets/images/h200/X13.png" alt="Supermicro X13 8U GPU System"></p>
+
+In this setup, each EC2 instance receives an entire H200 via PCIe passthrough and manages its own MIG partitions. This gives each tenant full control over how they slice their GPU — including the ability to run heterogeneous workloads at different partition sizes on the same physical card.
+
+A key consideration for this example is that we do not have access to the router upstream of the X13 host, or knowledge of potentially available IP addresses for the VMs we provision on it. The first step in the Instructions section outlines how we attach our own local IP address range to the bridge on the host, effectively creating a pseudo-airgapped environment - although in this specific case outbound connectivity is still required for remote access to the host, downloading required packages etc, with this configuration it is *not* required for host/EC2 instance communication.
+
+### EC2 workload layout
+
+| EC2 Workload | IP | MIG config | Workload | Model |
+|---|---|---|---|---|
+| `vm-llama3b` | 192.168.10.X  | 7 × 1g.18gb | Chat inference × 7 | Llama-3.2-3B-Instruct |
+| `vm-qwen32b` | 192.168.10.X  | 2 × 3g.71gb | Chat inference × 2 | Qwen2.5-32B-Instruct |
+| `vm-llama70b` | 192.168.10.X | 1 × 7g.141gb | Chat inference × 1 | Llama-3.1-70B-FP8 |
+| `vm-yolo`    | 192.168.10.X | 2 × 3g.71gb | Object detection × 2 | YOLO11x + YOLO11s |
+
+Twelve concurrent inference endpoints in total: 7 fast 3B slots, 2 mid-tier 32B slots, 1 full-GPU 70B slot, and 2 real-time vision streams running side-by-side to compare detection models.
+
+
+### Platform
+
+| Component | Specification |
+|---|---|
+| **Chassis** | Supermicro X13 8U GPU System |
+| **CPUs** | 2× Intel Xeon Platinum 8568Y+ (48 cores each, 96 cores / 192 threads total) |
+| **RAM** | 2 TB DDR5-4800 (32× 64 GB SK Hynix DIMMs) |
+| **Storage** | 4× 7.68 TB KIOXIA CD6 NVMe SSDs (30.72 TB raw) |
+| **GPUs** | 8× NVIDIA H200 SXM5 (141 GiB HBM3e per GPU, ~1.13 TB total) — 4 used in this demo |
+| **Host OS** | Ubuntu 26.04 LTS |
+| **Orchestration** | Spinifex — EC2-compatible bare-metal API |
+| **Guest OS** | Ubuntu 26.04 LTS |
+| **GPU partitioning** | NVIDIA MIG — managed inside each guest VM |
+| **Block storage** | Predastore — S3-compatible, NVMe-backed |
+| **Container runtime** | Docker |
+| **Inference runtime** | vLLM (`vllm-openai` image from local registry) |
+| **Vision runtime** | Ultralytics YOLO11 |
+
+
+## Prerequisites
+
+### 1. Verify GPU visibility on host
+Before making any changes, verify that the host can see its GPUs:
+
+<img src="../../../.github/assets/images/h200/nvidia-smi.png" alt="NVIDIA SMI">
+
+
+## Instructions
+
+
+### 1. Configure host-local VPC networking
+
+Spinifex utilises bridged networking via OVN. In this example, the remote host exposes only one public IP address, attached to a single physical NIC. Before provisioning Spinifex, we must first ensure that `br-wan` exists and enslave the physical NIC to it.
+
+Since we have no access to an upstream router or knowledge of available IP address pools, we must also attach our own `192.168.10.0/24` address range to `br-wan` which will be used by the guest VMs to communicate with the host and each other.
+
+For example:
+
+```yaml
+# /etc/netplan/…
+bridges:
+  br-wan:
+    addresses:
+      - 192.168.10.1/24      # VM pool gateway — host-local
+      - 198.51.100.10/24     # existing WAN IP — unchanged
+    routes:
+      - to: default
+        via: 198.51.100.1
+```
+
+Then apply with `sudo netplan apply`. VMs, once provisioned, are reachable from the host at
+`192.168.10.x`. For internet access through the host's WAN interface:
+
+```bash
+sysctl -w net.ipv4.ip_forward=1
+iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o br-wan -j MASQUERADE
+```
+The exact process for this is described in the [VPC Networking](/docs/vpc-networking#host-local-subnet-no-upstream-router) guide.
+
+
+
+### 2. Install Spinifex
+
+Follow the [Single Node Install](/docs/install) guide. This process will install Spinifex and start Spinifex services, however `spinifex.toml` needs to be edited to finalise the changes made to the networking in the previous section, as described in the following section.
+
+### 3. Configure spinifex.toml and restart services
+
+Edit `/etc/spinifex/spinifex.toml` to point the external pool and VPCD at the bridge created in step 1:
+
+```toml
+[network]
+external_mode = "pool"
+
+[[network.external_pools]]
+name        = "wan"
+source      = "static"         # required — no upstream DHCP for this range
+range_start = "192.168.10.2"
+range_end   = "192.168.10.100"
+gateway     = "192.168.10.1"   # second address on br-wan
+prefix_len  = 24
+dns_servers = ["8.8.8.8"]
+```
+
+Then restart all services:
+
+```bash
+sudo systemctl start spinifex.target
+sudo systemctl status spinifex.target
+```
+
+### 4. Attach Predastore storage
+
+Spinifex distributes the Predastore object storage volume across multiple nodes using Reed–Solomon encoding for data redundancy. This chassis has four NVMe drives — one for the OS, three dedicated to data — so we back each Predastore storage node with its own physical drive, giving us fault-tolerant distributed storage on a single machine — conceptually similar to RAID 5, where data and parity are spread across drives so a single drive failure is recoverable.
+
+Confirm drive assignments with `lsblk` before proceeding, as device names vary between systems.
+
+```bash
+lsblk   # identify the OS drive and the three data drives
+
+# Stop services so Predastore isn't writing while we relocate its data directories
+sudo systemctl stop spinifex.target
+
+# --- Repeat the block below for each data drive (node-1/nvme-1, node-2/nvme-2, node-3/nvme-3) ---
+
+# Mount the physical drive at a stable path
+sudo mkdir -p /mnt/nvme-1
+sudo mount /dev/nvme1n1 /mnt/nvme-1
+sudo mkdir -p /mnt/nvme-1/nodes /mnt/nvme-1/db
+
+# Move Predastore's node data and metadata off the OS drive onto the physical NVMe
+sudo mv /var/lib/spinifex/predastore/distributed/nodes/node-1 /mnt/nvme-1/nodes/node-1
+sudo mv /var/lib/spinifex/predastore/distributed/db/node-1    /mnt/nvme-1/db/node-1
+
+# Symlink the original paths back so Predastore finds its data unchanged
+sudo ln -s /mnt/nvme-1/nodes/node-1 /var/lib/spinifex/predastore/distributed/nodes/node-1
+sudo ln -s /mnt/nvme-1/db/node-1    /var/lib/spinifex/predastore/distributed/db/node-1
+
+# Persist the mount across reboots
+echo "/dev/nvme1n1  /mnt/nvme-1  auto  defaults  0  2" | sudo tee -a /etc/fstab
+
+# --- End of per-drive block ---
+
+sudo systemctl start spinifex.target
+```
+
+Verify the nodes are healthy before proceeding:
+
+```bash
+export AWS_PROFILE=spinifex
+
+aws s3 ls --endpoint-url https://localhost:8443
+# Should return without error (empty bucket list is fine)
+```
+
+### 5. Enable GPU Passthrough
+Spinifex allows GPUs to be utilised by guest VMS via PCIe-passthrough. This can be enabled via `sudo spx admin gpu setup`. Then, after a reboot, run `sudo spx admin gpu enable`. The Spinifex banner should update to reflect GPU passthrough state after every step:
+
+<p align="center"><img src="../../../.github/assets/images/h200/spinifex-banner1.png" alt="GPU Enabled"></p>
+
+
+### 6. Import the GPU AMI
+
+The demo uses the standard Spinifex NVIDIA GPU AMI (`ubuntu-26.04-nvidia-gpu-x86_64`), which includes:
+
+- Ubuntu 26.04 LTS guest image
+- NVIDIA server driver (DKMS pre-built against the pinned kernel)
+- Docker CE + nvidia-container-toolkit (`--gpus` support enabled at boot)
+- Python 3 + venv, common utilities (tmux, curl, ffmpeg, etc.)
+
+```bash
+spx admin images import --name ubuntu-26.04-nvidia-gpu-x86_64
+```
+
+Confirm and set the AMI ID:
+```bash
+AMI_ID=$(aws ec2 describe-images \
+    --filters "Name=name,Values=ubuntu-26.04-nvidia-gpu-x86_64" \
+    --query 'Images[0].ImageId' --output text)
+```
+
+### 7. Create VPC resources
+
+```bash
+# Create VPC and subnet
+VPC_ID=$(aws ec2 create-vpc --cidr-block 192.168.10.0/24 \
+    --query 'Vpc.VpcId' --output text)
+SUBNET_ID=$(aws ec2 create-subnet --vpc-id $VPC_ID \
+    --cidr-block 192.168.10.0/24 \
+    --query 'Subnet.SubnetId' --output text)
+
+# Security group — allow SSH + all vLLM/YOLO ports
+SG_ID=$(aws ec2 create-security-group \
+    --group-name demo-sg --description "H200 MIG demo" \
+    --vpc-id $VPC_ID --query 'GroupId' --output text)
+
+aws ec2 authorize-security-group-ingress --group-id $SG_ID \
+    --protocol tcp --port 22 --cidr 0.0.0.0/0
+aws ec2 authorize-security-group-ingress --group-id $SG_ID \
+    --protocol tcp --port 8000-8011 --cidr 0.0.0.0/0
+```
+
+```bash
+# Create SSH key
+aws ec2 create-key-pair --key-name spinifex-key \
+  | jq -r '.KeyMaterial | rtrimstr("\n")' > ~/.ssh/spinifex-key
+
+chmod 600 ~/.ssh/spinifex-key
+ssh-keygen -y -f ~/.ssh/spinifex-key > ~/.ssh/spinifex-key.pub
+```
+
+Verify:
+
+```bash
+aws ec2 describe-key-pairs
+```
+
+### 8. Launch the four VMs
+
+Each VM gets a whole H200 via PCIe passthrough. The `p5e.4xlarge` instance type maps one H200 per VM:
+
+```bash
+aws ec2 run-instances \
+  --image-id $AMI_ID \
+  --instance-type p5e.4xlarge \
+  --key-name spinifex-key \
+  --subnet-id $SUBNET_ID \
+  --security-group-ids $SG_ID \
+  --count 4
+```
+
+Wait until all four reach running state, then take note of the public IPs assigned to each instance:
+
+```bash
+aws ec2 describe-instances \
+    --query 'Reservations[*].Instances[*].[InstanceId,PublicIpAddress,State.Name]' \
+    --output table
+```
+
+### 9. Enable MIG inside each VM
+
+MIG is enabled inside each guest VM, not on the host. SSH into each VM and enable it:
+
+```bash
+# Repeat on each VM IP
+ssh -i ~/.ssh/spinifex-key ec2-user@<vm-ip>
+
+sudo nvidia-smi -mig 1
+nvidia-smi | grep "MIG M."
+# Should show: MIG M.  Enabled
+```
+
+### 10. Create MIG partitions inside each VM
+
+With MIG enabled, we are now able to partition each VM's assigned H200 GPU into several separate GPU instances. This process assigns each GPU slice its own UUID, so each VM goes from seeing one whole GPU to seeing a number of "MIG devices":
+
+<p align="center"><img src="../../../.github/assets/images/h200/mig-activated.png" alt="NVIDIA SMI"></p>
+
+This allows us to run several separate workloads, each assigned to its own GPU instance, on the same physical GPU, and thus utilise more of the overall GPU's resources.
+
+Run `nvidia-smi mig -lgip` to see the available partition types. MIG profiles are named Xg.Ygb, where:
+
+* X is the number of GPU slices (GPU Instances, or GIs) allocated to the partition. On an H200 there are 7 allocatable GPU slices, so the largest profile is 7g.
+* Y is the amount of HBM memory allocated to that partition.
+
+Larger profiles also receive proportionally more SMs (Streaming Multiprocessors - analagous to CPU cores), cache, copy engines, encoders/decoders, and other GPU resources.
+
+Thus we partition the GPUs assigned to our VMs as follows:
+
+**vm-llama3b — 7 × 1g.18gb (Llama 3B):**
+```bash
+sudo nvidia-smi mig -cgi 1g.18gb,1g.18gb,1g.18gb,1g.18gb,1g.18gb,1g.18gb,1g.18gb -C
+nvidia-smi -L  # Verify 7 MIG devices
+```
+
+**vm-qwen32b — 2 × 3g.71gb (Qwen 32B):**
+```bash
+sudo nvidia-smi mig -cgi 3g.71gb,3g.71gb -C
+nvidia-smi -L  # Verify 2 MIG devices
+```
+
+**vm-llama70b — 1 × 7g.141gb (Llama 70B, full GPU):**
+```bash
+sudo nvidia-smi mig -cgi 7g.141gb -C
+nvidia-smi -L  # Verify 1 MIG device
+```
+
+**vm-yolo — 2 × 3g.71gb (YOLO):**
+```bash
+sudo nvidia-smi mig -cgi 3g.71gb,3g.71gb -C
+nvidia-smi -L  # Verify 2 MIG devices
+```
+
+Each MIG device gets a UUID of the form `MIG-xxxxxxxx-...`. These UUIDs are used to pin individual containers to their slice via `--gpus "device=<UUID>"` (for Docker) or `CUDA_VISIBLE_DEVICES=<UUID>` (for direct Python processes).
+
+### 11. Configure Docker to use the local image registry
+
+The vLLM image is served from the host's local Docker registry (`192.168.10.1:5000`).
+
+Setting up the local registry on the host:
+
+```bash
+# Configure Docker to trust the local registry address
+sudo tee /etc/docker/daemon.json > /dev/null <<'EOF'
+{
+  "insecure-registries": ["192.168.10.1:5000"]
+}
+EOF
+sudo systemctl restart docker
+
+# Start the registry container, bound to the bridge interface only
+docker run -d \
+    -p 192.168.10.1:5000:5000 \
+    --name registry \
+    --restart=always \
+    registry:2
+
+# Pull the vLLM image and push it into the local registry
+docker pull vllm/vllm-openai:latest
+docker tag vllm/vllm-openai:latest 192.168.10.1:5000/vllm-openai:latest
+docker push 192.168.10.1:5000/vllm-openai:latest
+
+# Verify the image is available
+curl http://192.168.10.1:5000/v2/_catalog
+# {"repositories":["vllm-openai"]}
+```
+
+Each VM then needs to trust it as an insecure registry:
+
+```bash
+# On each VM — vm-yolo uses direct Python, not Docker for YOLO
+sudo tee /etc/docker/daemon.json > /dev/null <<'EOF'
+{
+  "insecure-registries": ["192.168.10.1:5000"]
+}
+EOF
+sudo systemctl restart docker
+```
+
+### 12. Deploy the LLM workloads
+
+Each LLM is downloaded using the Huggingface CLI onto the respective VM. First install the CLI:
+
+```bash
+pip install -U "huggingface_hub[cli]"
+```
+
+Then download the model and determine its snapshot directory. This directory contains the model’s config.json, tokenizer files, and weight files, and will be used as MODEL_DIR when launching vLLM.
+
+```bash
+#On vm-llama3b
+hf download meta-llama/Llama-3.2-3B-Instruct
+
+MODEL_DIR=$(dirname "$(find ~/.cache/huggingface/hub -path '*Llama-3.2-3B-Instruct*' -name config.json | head -1)")
+echo "$MODEL_DIR"
+
+#On vm-qwen32b
+hf download Qwen/Qwen2.5-32B-Instruct
+
+MODEL_DIR=$(dirname "$(find ~/.cache/huggingface/hub -path '*Qwen2.5-32B-Instruct*' -name config.json | head -1)")
+echo "$MODEL_DIR"
+
+#On vm-llama70b
+hf download RedHatAI/Meta-Llama-3.1-70B-Instruct-FP8
+
+MODEL_DIR=$(dirname "$(find ~/.cache/huggingface/hub -path '*Meta-Llama-3.1-70B-Instruct-FP8*' -name config.json | head -1)")
+echo "$MODEL_DIR"
+```
+
+Then we run the docker containers as follows:
+
+**vm-llama3b — 7 × Llama-3.2-3B-Instruct (one container per MIG slice):**
+
+Enumerate the MIG UUIDs and start one vLLM container per slice:
+
+
+```bash
+mapfile -t UUIDS < <(nvidia-smi -L | grep -oP 'MIG-[0-9a-f-]+')
+for i in "${!UUIDS[@]}"; do
+    docker run -d --rm \
+        --name "vllm-$i" \
+        --gpus "device=${UUIDS[$i]}" \
+        --ipc host \
+        -p $((8000 + i)):8000 \
+        -v "${MODEL_DIR}:/models:ro" \
+        192.168.10.1:5000/vllm-openai:latest \
+        vllm serve /models \
+            --served-model-name llama-3b \
+            --dtype bfloat16 \
+            --max-model-len 4096 \
+            --gpu-memory-utilization 0.90 \
+            --port 8000
+done
+```
+
+**vm-qwen32b — 2 × Qwen2.5-32B-Instruct:**
+
+```bash
+mapfile -t UUIDS < <(nvidia-smi -L | grep -oP 'MIG-[0-9a-f-]+')
+for i in "${!UUIDS[@]}"; do
+    docker run -d --rm \
+        --name "qwen32b-$i" \
+        --gpus "device=${UUIDS[$i]}" \
+        --ipc host \
+        -p $((8000 + i)):8000 \
+        -v "${MODEL_DIR}:/models:ro" \
+        192.168.10.1:5000/vllm-openai:latest \
+        vllm serve /models \
+            --served-model-name qwen2.5-32b \
+            --dtype bfloat16 \
+            --max-model-len 4096 \
+            --gpu-memory-utilization 0.90 \
+            --port 8000
+done
+```
+
+**vm-llama70b — Llama-3.1-70B-Instruct-FP8 (full GPU slice):**
+
+```bash
+UUID=$(nvidia-smi -L | grep -oP 'MIG-[0-9a-f-]+' | head -1)
+docker run -d --rm \
+    --name vllm \
+    --gpus "device=${UUID}" \
+    --ipc host \
+    -p 8000:8000 \
+    -v "${MODEL_DIR}:/models:ro" \
+    192.168.10.1:5000/vllm-openai:latest \
+    vllm serve /models \
+        --served-model-name meta-llama-3.1-70b \
+        --dtype auto \
+        --max-model-len 8192 \
+        --gpu-memory-utilization 0.90 \
+        --port 8000
+```
+
+### 13. Deploy YOLO object detection (vm-yolo)
+
+For this demo we built a simple YOLO inference server that runs detection on a looping video file and serves the annotated output as an MJPEG stream. The same script runs twice on vm-yolo — once for YOLO11x (larger, more accurate) and once for YOLO11s (smaller, faster) — each pinned to its own MIG slice and listening on a different port.
+
+The MIG pinning works through `CUDA_VISIBLE_DEVICES`: setting it to a MIG UUID before launch scopes the process to that slice, and CUDA remaps it to device index 0 inside the process. From the script's perspective it always sees one GPU at index 0 — the MIG boundary is invisible to it. `YOLO_MODEL` and `PORT` are what actually differentiate the two instances:
+
+```bash
+python3 -m venv ~/yolo-venv
+~/yolo-venv/bin/pip install ultralytics fastapi "uvicorn[standard]" opencv-python-headless httpx
+
+# Get MIG UUIDs
+nvidia-smi -L
+# GPU 0: NVIDIA H200 (UUID: GPU-...)
+#   MIG 3g.71gb  Device 0: (UUID: MIG-<uuid0>)
+#   MIG 3g.71gb  Device 1: (UUID: MIG-<uuid1>)
+
+# YOLO11x on slice 0 — larger model, port 8010
+CUDA_VISIBLE_DEVICES=MIG-<uuid0> \
+  VIDEO_PATH=<path-to-video> \
+  YOLO_MODEL=yolo11x.pt \
+  YOLO_DEVICE=0 \
+  PORT=8010 \
+  ~/yolo-venv/bin/python ~/yolo_stream.py >> ~/yolo-x.log 2>&1 &
+
+# YOLO11s on slice 1 — smaller/faster model, port 8011
+CUDA_VISIBLE_DEVICES=MIG-<uuid1> \
+  VIDEO_PATH=<path-to-video> \
+  YOLO_MODEL=yolo11s.pt \
+  YOLO_DEVICE=0 \
+  PORT=8011 \
+  ~/yolo-venv/bin/python ~/yolo_stream.py >> ~/yolo-s.log 2>&1 &
+```
+
+Each instance exposes a `/video` endpoint serving a `multipart/x-mixed-replace` MJPEG stream, consumable directly by browsers and most HTTP clients. YOLO11x (~75 MB) and YOLO11s (~9 MB) weights download automatically on first run from the Ultralytics model hub.
+
+### 14. Dashboard
+
+We also built a simple host-side dashboard — another FastAPI application that proxies all the VM streams to the browser so only one port on the host needs to be exposed. Each VM endpoint is wired in by address at startup, mapping directly to the IPs assigned in step 8:
+
+```
+http://vm-yolo-IP:8010  →  vm-yolo  YOLO11x MJPEG stream
+http://vm-yolo-IP:8011  →  vm-yolo  YOLO11s MJPEG stream
+http://vm-llama3b-IP:8000   →  vm-llama3b  vLLM endpoint 0
+http://vm-llama3b-IP:8001   →  vm-llama3b  vLLM endpoint 1
+... (one entry per endpoint across all three LLM VMs)
+```
+
+The dashboard has two proxy patterns: MJPEG passthrough for the YOLO feeds (forwarding the raw boundary stream from vm-yolo to the browser), and SSE passthrough for the LLM token streams (subscribing to each vLLM endpoint and re-emitting tokens as server-sent events). Both use a reconnect loop so the browser connection stays open if a VM is temporarily unreachable.
+
+
+
+<p><video src="https://iso.mulgadc.com/h200-demo.mp4" controls width="100%" style="border-radius:6px"></video></p>
+
+The dashboard shows:
+- GPU allocation bars for all four EC2 instancess (proportional to MIG slice size)
+- Live streaming LLM responses per endpoint, colour-coded by tier
+- Side-by-side YOLO11x vs YOLO11s video feeds with FPS and detection counts
+
+The dashboard also displays the overall GPU utilisation, derived as a percentage of maximum power usage. Importantly, it demonstrates how Spinifex combined with NVIDIA's MIG capability enables the deployment of multiple heterogeneous workloads on owned hardware.
+
+### 15. Teardown
+
+```bash
+# On vm-yolo:
+# Stop YOLO processes
+pkill -f yolo_stream.py
+
+# On each LLM instance:
+# Stop docker containers
+
+docker stop $(docker ps -q)
+
+# Disable MIG inside each VM before terminating.
+sudo nvidia-smi mig -dci
+sudo nvidia-smi mig -dgi
+sudo nvidia-smi -mig 0
+
+# On the host:
+# Terminate all four instances — releases 4× H200 back to the Spinifex pool
+aws ec2 terminate-instances --instance-ids \
+    $(aws ec2 describe-instances \
+        --filters "Name=instance-state-name,Values=running" \
+        --query 'Reservations[*].Instances[*].InstanceId' \
+        --output text)
+```
+
+The four H200s are immediately returned to the host GPU pool once the instances terminate, ready for reallocation.
+
+### 16. Conclusion
+
+This document highlights how Spinifex can turn a single bare-metal chassis into a multi-tenant AI serving platform. Spinifex utilises the flexibility of PCIe passthrough combined with NVIDIA's MIG capability to allocate GPU resources in whichever configuration is required by the workload/s. This flexibility and fine-grain control ensures maximum GPU utilisation.
+
+Importantly, it does so with standard `aws ec2` CLI calls — `run-instances`, `describe-instances`, `terminate-instances` — against Spinifex's EC2-compatible endpoint. Teams already operating AWS infrastructure can point their existing tooling at a Spinifex node with a single profile change, against GPUs that sit in their own rack.

docs/reference-architectures/arcee-trinity-mi350x/README.md

@@ -31,6 +31,8 @@ sections:
 
 ## Overview
 
+Spinifex is an open-source infrastructure platform that brings core AWS services including EC2, EBS and S3 to bare-metal, edge, and on-prem environments. It exposes an EC2-compatible API, so any tooling that works against AWS (the `aws` CLI, Terraform, SDKs) works against a Spinifex node unchanged, with a single profile swap.
+
 ### About Trinity
 
 [Trinity](https://www.arcee.ai/trinity) is a family of open-weight language models from [Arcee AI](https://www.arcee.ai), built specifically for production agentic workloads: reliable tool calling, structured JSON output, coherent multi-turn conversations, and long-context reasoning. The family spans four sizes — Nano (6B), Mini (26B), Large Preview (400B), and Large Thinking (400B reasoning-optimised), with consistent capabilities and API surface across all of them. A workflow validated on Nano can be promoted to Large with no prompt changes.

docs/service-interfaces.yaml

@@ -84,6 +84,12 @@ services:
       - "spinifex.node.vms"
       - "spinifex.nodes.discover"
       - "spinifex.cluster.shutdown.progress"
+      # Coordinated cluster-shutdown phases (fan-out, no queue group).
+      - "spinifex.cluster.shutdown.gate"
+      - "spinifex.cluster.shutdown.drain"
+      - "spinifex.cluster.shutdown.storage"
+      - "spinifex.cluster.shutdown.persist"
+      - "spinifex.cluster.shutdown.infra"
       - "spinifex.storage.config"
       # ELBv2 control plane (handlers/elbv2/service_nats.go topic map).
       - "elbv2.CreateLoadBalancer"
@@ -133,6 +139,8 @@ services:
       - "ebs.delete"
       - "ebs.sync"
       - "ebs.snapshot.*"
+      - "ebs.config"
+      - "ebs.config.*"
       # → predastore (S3 / AMI / keypair store)
       - "s3.*"
       # IAM lifecycle event
@@ -181,6 +189,8 @@ services:
       - "ebs.delete"
       - "ebs.sync"
       - "ebs.snapshot.*"
+      - "ebs.config"
+      - "ebs.config.*"
     publishes:
       - "ebs.mount.response"
       - "ebs.unmount.response"
@@ -274,6 +284,22 @@ suites:
     # into per-test fixture-driven coverage so the selector can pick
     # subtests, not just whole suites.
 
+  e2e-iam:
+    path: tests/e2e/iam
+    covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+    # "Everything IAM": the IAM/STS control plane plus the instance-identity
+    # tests (instance-profile association + IMDS), which boot guest VMs — OVN
+    # via vpcd, AMI via predastore, root volume via viperblockd, lifecycle via
+    # spinifex_daemon. Single-mode, VM-booting leg.
+
+  e2e-ecr:
+    path: tests/e2e/ecr
+    covers: [awsgw, spinifex_daemon, predastore]
+    # ECR control plane (handlers/ecr, gateway/ecr*) + OCI data plane: blobs and
+    # manifests in predastore, control + auth bridge in awsgw, KV via
+    # spinifex_daemon. docker/crane/skopeo data-plane subtests skip when the
+    # client binary or registry-host resolution is absent.
+
   e2e-ddil:
     path: tests/e2e/ddil
     covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]

docs/terraform-workbooks/demo-app/.dockerignore

@@ -0,0 +1,4 @@
+README.md
+*.tfstate
+*.tfstate.backup
+.terraform

docs/terraform-workbooks/demo-app/Dockerfile

@@ -0,0 +1,17 @@
+# Spinifex EKS demo app — multi-stage build to a tiny distroless image.
+#
+# Stage 1 compiles a static binary; stage 2 ships only that binary on a
+# distroless base (no shell, no package manager) for a small, production-shaped
+# image suitable for pushing to ECR and pulling onto EKS workers.
+
+FROM golang:1.23-alpine AS build
+WORKDIR /src
+COPY go.mod ./
+COPY main.go ./
+RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/spinifex-demo .
+
+FROM gcr.io/distroless/static:nonroot
+COPY --from=build /out/spinifex-demo /spinifex-demo
+EXPOSE 8080
+USER nonroot:nonroot
+ENTRYPOINT ["/spinifex-demo"]

docs/terraform-workbooks/demo-app/README.md

@@ -0,0 +1,69 @@
+# Spinifex EKS demo app
+
+A small, Spinifex-themed web app shared by the EKS Terraform workbooks. It
+replaces `nginxdemos/hello` with something production-shaped that you build
+yourself and push to **ECR**, then pull onto your EKS workers.
+
+The page reports the pod, node, cluster, and region that answered the request,
+plus a hit counter — refresh and watch requests land on different pods across a
+multi-node cluster.
+
+## What it does
+
+- `GET /` — themed HTML page.
+- `GET /api/info` — JSON of the same facts.
+- `GET /healthz` — liveness/readiness probe (`200 ok`), used by the ALB target
+  group health check.
+
+All configuration is read from the environment, so one image serves every tier:
+
+| Variable | Purpose | Source |
+|---|---|---|
+| `POD_NAME`, `NODE_NAME`, `POD_NAMESPACE` | identity shown on the page | downward API |
+| `CLUSTER_NAME`, `AWS_REGION` | cluster/region labels | deployment env |
+| `APP_TITLE` | page heading | deployment env (default `Spinifex EKS`) |
+| `PORT` | listen port | default `8080` |
+| `DATA_DIR` | when set, the hit counter is persisted here and survives pod restarts | mounted **EBS-CSI PVC** in `eks-gitops-argocd` |
+
+When `DATA_DIR` points at a writable volume the page shows a **"persisted to EBS
+volume"** badge — the gitops workbook mounts a Viperblock-backed PersistentVolume
+there to demonstrate durable storage.
+
+## Build and push to ECR
+
+The image is named `spinifex-demo`. Build it once and push it to a repository in
+your Spinifex ECR; every workbook references it by its ECR URI.
+
+```bash
+export AWS_PROFILE=spinifex
+REGION=ap-southeast-2
+
+# 1. Create the repository (the workbooks also create it via Terraform; skip if
+#    it already exists).
+aws ecr create-repository --repository-name spinifex-demo --region "$REGION"
+
+# 2. Resolve the registry host and authenticate Docker.
+REGISTRY=$(aws ecr describe-repositories --repository-names spinifex-demo \
+  --region "$REGION" --query 'repositories[0].repositoryUri' --output text)
+REGISTRY_HOST=${REGISTRY%%/*}
+aws ecr get-login-password --region "$REGION" \
+  | docker login --username AWS --password-stdin "$REGISTRY_HOST"
+
+# 3. Build, tag, and push.
+docker build -t spinifex-demo:latest .
+docker tag spinifex-demo:latest "${REGISTRY}:latest"
+docker push "${REGISTRY}:latest"
+
+echo "Image: ${REGISTRY}:latest"
+```
+
+Pass the resulting `${REGISTRY}:latest` value to a workbook as the `demo_image`
+variable (or accept its default, which resolves the URI from the ECR repository
+the workbook creates).
+
+## Run locally
+
+```bash
+go run .            # http://localhost:8080
+DATA_DIR=/tmp/demo go run .   # exercise the persisted-counter path
+```

docs/terraform-workbooks/demo-app/go.mod

@@ -0,0 +1,3 @@
+module github.com/mulgadc/spinifex-demo
+
+go 1.23

docs/terraform-workbooks/demo-app/main.go

@@ -0,0 +1,214 @@
+// Command spinifex-demo is the demo web app served by the EKS Terraform
+// workbooks. It renders a Spinifex-themed page reporting the pod, node, cluster,
+// and region that answered the request, plus a hit counter.
+//
+// The counter is held in memory by default. When DATA_DIR points at a writable
+// directory (an EBS-CSI PersistentVolume in the gitops workbook), the counter is
+// persisted there and survives pod restarts — demonstrating Viperblock-backed
+// durable storage. All configuration comes from the environment so a single
+// image serves every workbook tier.
+package main
+
+import (
+	"encoding/json"
+	"html/template"
+	"log/slog"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strconv"
+	"sync"
+	"time"
+)
+
+// instance captures the per-pod facts surfaced on the page, populated from the
+// downward API and the deployment environment.
+type instance struct {
+	Pod       string
+	Node      string
+	Namespace string
+	Cluster   string
+	Region    string
+	Title     string
+	Persisted bool
+}
+
+func env(key, fallback string) string {
+	if v := os.Getenv(key); v != "" {
+		return v
+	}
+	return fallback
+}
+
+// counter tracks page hits, optionally persisting to dir/hits so the count
+// survives a pod restart when backed by a PersistentVolume.
+type counter struct {
+	mu   sync.Mutex
+	n    int64
+	path string
+}
+
+func newCounter(dir string) *counter {
+	c := &counter{}
+	if dir == "" {
+		return c
+	}
+	c.path = filepath.Join(dir, "hits")
+	if b, err := os.ReadFile(c.path); err == nil {
+		if n, perr := strconv.ParseInt(string(b), 10, 64); perr == nil {
+			c.n = n
+		}
+	}
+	return c
+}
+
+func (c *counter) inc() int64 {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.n++
+	if c.path != "" {
+		// Best-effort persist; a read-only or missing volume must not 500 the page.
+		if err := os.WriteFile(c.path, []byte(strconv.FormatInt(c.n, 10)), 0o644); err != nil {
+			slog.Warn("persist counter failed", "path", c.path, "err", err)
+		}
+	}
+	return c.n
+}
+
+func main() {
+	logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
+	slog.SetDefault(logger)
+
+	dataDir := os.Getenv("DATA_DIR")
+	hits := newCounter(dataDir)
+
+	base := instance{
+		Pod:       env("POD_NAME", "unknown"),
+		Node:      env("NODE_NAME", "unknown"),
+		Namespace: env("POD_NAMESPACE", "default"),
+		Cluster:   env("CLUSTER_NAME", "spinifex-eks"),
+		Region:    env("AWS_REGION", env("AWS_DEFAULT_REGION", "ap-southeast-2")),
+		Title:     env("APP_TITLE", "Spinifex EKS"),
+		Persisted: dataDir != "",
+	}
+
+	tmpl := template.Must(template.New("page").Parse(pageHTML))
+
+	mux := http.NewServeMux()
+
+	mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("ok"))
+	})
+
+	mux.HandleFunc("/api/info", func(w http.ResponseWriter, _ *http.Request) {
+		w.Header().Set("Content-Type", "application/json")
+		_ = json.NewEncoder(w).Encode(map[string]any{
+			"pod":       base.Pod,
+			"node":      base.Node,
+			"namespace": base.Namespace,
+			"cluster":   base.Cluster,
+			"region":    base.Region,
+			"hits":      hits.inc(),
+			"persisted": base.Persisted,
+		})
+	})
+
+	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path != "/" {
+			http.NotFound(w, r)
+			return
+		}
+		view := struct {
+			instance
+			Hits int64
+			Now  string
+		}{base, hits.inc(), time.Now().UTC().Format("2006-01-02 15:04:05 MST")}
+		w.Header().Set("Content-Type", "text/html; charset=utf-8")
+		if err := tmpl.Execute(w, view); err != nil {
+			slog.Error("render failed", "err", err)
+		}
+	})
+
+	addr := ":" + env("PORT", "8080")
+	slog.Info("spinifex-demo listening", "addr", addr, "cluster", base.Cluster, "persisted", base.Persisted)
+	srv := &http.Server{
+		Addr:              addr,
+		Handler:           mux,
+		ReadHeaderTimeout: 5 * time.Second,
+	}
+	if err := srv.ListenAndServe(); err != nil {
+		slog.Error("server exited", "err", err)
+		os.Exit(1)
+	}
+}
+
+const pageHTML = `<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>{{.Title}}</title>
+<style>
+  :root {
+    --bg: #0b0e14; --panel: #131823; --line: #232a39;
+    --fg: #e6edf3; --muted: #8b97a7; --accent: #f5b301; --accent2: #3ddc97;
+  }
+  * { box-sizing: border-box; }
+  body {
+    margin: 0; min-height: 100vh; background:
+      radial-gradient(1200px 600px at 80% -10%, rgba(245,179,1,.10), transparent 60%),
+      radial-gradient(900px 500px at -10% 110%, rgba(61,220,151,.08), transparent 55%), var(--bg);
+    color: var(--fg); font: 15px/1.55 ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif;
+    display: flex; align-items: center; justify-content: center; padding: 32px;
+  }
+  .card {
+    width: 100%; max-width: 640px; background: var(--panel);
+    border: 1px solid var(--line); border-radius: 16px; overflow: hidden;
+    box-shadow: 0 20px 60px rgba(0,0,0,.45);
+  }
+  .head { padding: 28px 32px; border-bottom: 1px solid var(--line); display: flex; align-items: center; gap: 16px; }
+  .logo { height: 44px; width: auto; flex: 0 0 auto; }
+  .head h1 { margin: 0; font-size: 20px; letter-spacing: .2px; }
+  .head p { margin: 2px 0 0; color: var(--muted); font-size: 13px; }
+  .body { padding: 24px 32px 8px; }
+  .grid { display: grid; grid-template-columns: 1fr 1fr; gap: 12px 20px; }
+  .kv { padding: 12px 14px; background: #0e131d; border: 1px solid var(--line); border-radius: 10px; }
+  .kv .k { color: var(--muted); font-size: 11px; text-transform: uppercase; letter-spacing: .08em; }
+  .kv .v { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 14px; word-break: break-all; margin-top: 3px; }
+  .hits { grid-column: 1 / -1; display: flex; align-items: baseline; justify-content: space-between; }
+  .hits .n { font-size: 30px; font-weight: 700; color: var(--accent); font-family: ui-monospace, monospace; }
+  .foot { padding: 16px 32px 24px; color: var(--muted); font-size: 12px; display: flex; justify-content: space-between; gap: 12px; }
+  .pill { color: var(--accent2); border: 1px solid rgba(61,220,151,.35); border-radius: 999px; padding: 2px 10px; font-size: 11px; }
+</style>
+</head>
+<body>
+  <div class="card">
+    <div class="head">
+      <svg class="logo" viewBox="0 0 364 463" fill="none" xmlns="http://www.w3.org/2000/svg" aria-hidden="true">
+        <path d="M272.523 171.205C247.773 197.595 211.323 215.815 182.103 237.425C165.663 249.575 122.633 281.165 151.093 301.825C185.533 326.835 271.333 304.985 301.013 277.585C317.193 262.645 316.693 242.485 294.703 234.075C270.913 224.975 243.483 231.925 219.353 235.755C218.543 235.885 216.823 236.805 216.993 235.345C227.053 228.885 236.103 220.925 245.933 214.135C255.403 207.585 267.053 198.225 278.643 198.015C314.693 197.345 351.803 215.135 361.203 252.215C376.833 313.885 303.333 362.255 249.833 366.635L280.913 352.235C292.563 345.305 304.353 338.105 314.653 329.215C317.213 327.005 319.073 324.145 321.243 322.035C321.753 321.535 327.053 319.265 324.003 319.125C296.253 337.335 263.793 349.415 231.323 356.335C220.173 358.715 208.323 360.995 196.963 361.285C197.243 362.845 198.643 362.525 199.663 362.845C208.823 365.795 219.313 367.545 228.843 369.005C230.163 370.005 228.733 370.415 227.683 370.555C219.063 371.665 209.923 373.305 201.293 373.745C176.843 374.985 152.203 372.435 128.573 366.425L115.293 328.755C114.153 328.865 114.073 329.895 113.783 330.725C112.613 334.015 107.893 358.025 106.313 358.365C93.6234 354.515 82.2334 345.915 72.9034 336.575L95.0434 296.915L59.7134 316.375C54.7434 305.145 53.8334 292.435 55.4934 280.355L101.213 257.645C86.9934 256.705 72.6134 258.955 58.4734 260.425C60.8734 253.495 63.8034 246.615 67.3634 240.195C68.6634 237.855 75.5634 226.305 77.3834 226.245C93.9334 229.415 111.773 230.895 128.363 227.675L92.3534 212.825L91.0334 211.045C98.3834 204.125 105.843 197.235 114.113 191.325C129.603 195.825 145.523 199.455 161.733 199.655L130.633 179.885C134.133 177.125 137.813 174.575 141.553 172.165C144.643 170.175 153.283 164.425 156.423 164.835C168.933 169.105 181.923 172.125 195.123 173.185C189.093 166.935 177.633 162.325 171.833 156.465C171.203 155.825 170.713 155.785 171.003 154.585C186.153 145.885 203.003 137.235 216.053 125.375C229.363 113.265 243.423 93.4848 233.853 75.2048C244.893 78.7148 254.393 92.6648 254.153 104.325C253.493 136.595 219.073 165.375 197.483 185.615C230.833 171.615 268.993 142.685 268.923 102.735C268.863 70.2248 239.263 57.1948 211.283 54.2348C210.103 61.3848 207.573 68.9448 203.863 75.1748C195.623 89.0148 178.803 97.6648 170.013 112.055C164.023 121.865 162.883 135.065 150.163 138.825C144.973 140.365 138.293 140.595 132.903 140.615L144.743 134.365C158.003 124.625 157.773 107.025 165.893 93.9248C164.953 92.5948 155.413 90.0748 153.183 89.5248C143.363 87.1048 126.713 84.0248 116.883 84.6948C113.843 84.9048 100.143 87.0348 98.9634 89.3748C97.2634 92.7648 105.533 107.255 107.973 110.575C108.463 111.245 109.463 111.075 109.513 111.145C109.973 111.745 109.663 113.165 108.373 112.675C107.443 112.315 101.853 108.205 100.593 107.235C90.8934 99.7648 79.8234 88.4748 71.7234 79.3648C70.4034 76.1748 68.6334 72.6148 71.7634 69.9848C88.8634 59.4048 105.793 48.3148 123.163 38.1948C174.603 8.20482 233.143 -25.6952 282.353 29.2448C323.153 74.8048 311.503 129.615 272.493 171.195L272.523 171.205ZM114.153 67.5948C127.613 65.2648 153.013 78.6248 157.193 58.8348C157.603 56.9048 156.493 55.7948 157.583 53.9148C158.403 52.5048 163.083 49.1048 164.713 47.7848C169.903 43.6048 175.823 40.3348 180.923 36.0148C161.113 39.2048 141.163 50.5648 123.853 60.5948C120.983 62.2548 117.873 63.9948 115.153 65.8548C114.453 66.3348 113.713 65.7348 114.143 67.5848L114.153 67.5948ZM21.1634 276.415C-11.8666 312.635 -4.35661 364.395 30.6734 396.135C65.5434 427.725 120.403 438.925 166.373 436.725C207.813 434.735 249.893 421.075 290.773 434.965C306.663 440.365 320.533 450.465 332.363 462.135C330.773 454.415 326.123 446.365 321.623 439.825C280.593 380.225 200.893 402.565 140.123 390.665C107.163 384.215 72.8934 369.135 52.8234 341.225C35.4734 317.105 32.7734 288.665 41.3634 260.475C39.6134 258.725 23.0334 274.355 21.1634 276.415ZM143.743 109.535C136.373 116.045 128.573 122.235 120.403 127.775L97.1634 139.135C116.843 139.885 137.193 128.635 143.753 109.545L143.743 109.535Z" fill="#f5b301"/>
+      </svg>
+      <div>
+        <h1>{{.Title}}</h1>
+        <p>Served by a pod on your Spinifex-managed Kubernetes cluster</p>
+      </div>
+    </div>
+    <div class="body">
+      <div class="grid">
+        <div class="kv"><div class="k">Pod</div><div class="v">{{.Pod}}</div></div>
+        <div class="kv"><div class="k">Node</div><div class="v">{{.Node}}</div></div>
+        <div class="kv"><div class="k">Cluster</div><div class="v">{{.Cluster}}</div></div>
+        <div class="kv"><div class="k">Region</div><div class="v">{{.Region}}</div></div>
+        <div class="kv hits">
+          <div><div class="k">Requests served</div>{{if .Persisted}}<span class="pill">persisted to EBS volume</span>{{end}}</div>
+          <div class="n">{{.Hits}}</div>
+        </div>
+      </div>
+    </div>
+    <div class="foot">
+      <span>Namespace: {{.Namespace}}</span>
+      <span>{{.Now}}</span>
+    </div>
+  </div>
+</body>
+</html>`

docs/terraform-workbooks/eks-gitops-argocd/README.md

@@ -0,0 +1,284 @@
+---
+title: "GitOps on EKS (Argo CD + EBS-CSI)"
+description: "Deliver a Spinifex-themed app to EKS with GitOps — the Argo CD addon syncs it from a git repo, an EBS-CSI (Viperblock) PersistentVolume holds its state, and it is served over HTTPS through the AWS Load Balancer Controller + ACM, using Terraform on Spinifex."
+category: "Terraform Workbooks"
+tags:
+  - terraform
+  - eks
+  - kubernetes
+  - argocd
+  - gitops
+  - ebs-csi
+  - storage
+  - workbook
+resources:
+  - title: "Argo CD"
+    url: "https://argo-cd.readthedocs.io/"
+  - title: "AWS Load Balancer Controller"
+    url: "https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/"
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "Terraform Kubernetes Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+---
+
+# Terraform: GitOps on EKS (Argo CD + EBS-CSI)
+
+> The top rung of the EKS ladder. It extends [EKS HTTPS Ingress](../eks-https-ingress) and changes how the app is delivered: the **Argo CD** addon syncs a more elaborate Spinifex-themed app from a **git repo**, an **EBS-CSI** PersistentVolume (Viperblock-backed) holds its state, and it is still served over **HTTPS** through the AWS Load Balancer Controller + ACM.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+This workbook keeps everything from [EKS HTTPS Ingress](../eks-https-ingress) — the public/private VPC, the NAT gateway, the LBC addon, the ACM cert, the HTTPS Ingress — and adds the parts you reach for once an app is real: **GitOps delivery** and **persistent storage**.
+
+Three changes over the previous rung:
+
+1. **Argo CD delivers the app.** Instead of Terraform applying the Deployment, the **argocd** addon syncs the app from a git repository ([`mulgadc/eks-demo-app`](https://github.com/mulgadc/eks-demo-app)). Terraform only registers the repo credential and creates the Argo CD `Application`; Argo CD reconciles the manifests and self-heals drift.
+2. **State lives on an EBS volume.** The **aws-ebs-csi-driver** addon ships a default gp3 StorageClass. The app's `PersistentVolumeClaim` (in the git repo) dynamically provisions a **Viperblock-backed EBS volume**; the demo's hit counter persists to it and survives pod restarts.
+3. **Access-entry RBAC.** A second IAM principal is granted read-only cluster access via an access entry bound to `AmazonEKSViewPolicy`.
+
+Terraform manages the cluster, the addons, the ACM cert, and the HTTPS Ingress; **Argo CD** manages the app's Deployment, Service, and PVC from git. The Ingress points at the Service Argo CD creates, so the app is reachable over HTTPS the moment Argo CD finishes its first sync.
+
+**What you'll learn:**
+
+- Installing the `argocd` and `aws-ebs-csi-driver` addons through the EKS API
+- Registering a private git repo with Argo CD and driving an `Application` from Terraform
+- Dynamically provisioning a Viperblock-backed EBS volume with a `PersistentVolumeClaim`
+- Splitting ownership: Terraform owns infra + Ingress, Argo CD owns the workload
+- Granting scoped read-only access with an EKS access entry
+
+**What gets created**
+
+| Resource | Name | Purpose |
+|---|---|---|
+| VPC + subnets | `eks-gitops-*` | Public/private network (10.32.0.0/16) with a NAT gateway |
+| IAM Roles | `eks-gitops-cluster-role`, `-node-role`, `-viewer` | Control-plane, worker, and read-only viewer roles |
+| ECR Repository | `spinifex-demo` | Holds the demo image the workers pull |
+| EKS Cluster | `eks-gitops` | Public + private endpoints; `managed-ingress=false` |
+| Node Group | `workers` | `node_desired_size` `t3.large` worker(s) — 1 or 3 |
+| Addons | `aws-load-balancer-controller`, `argocd`, `aws-ebs-csi-driver` | Ingress, GitOps delivery, persistent storage |
+| ACM Certificate | `eks-gitops-ingress` | Self-signed, imported; on the HTTPS listener |
+| Access Entry + Assoc. | `eks-gitops-viewer` → `AmazonEKSViewPolicy` | Read-only, cluster scope |
+| Argo CD repo Secret | `eks-demo-app-repo` | Credential for the private git repo |
+| Argo CD Application | `spinifex-demo` | Syncs the app from git |
+| K8s Ingress | `spinifex-demo` | `ingressClassName: alb`, HTTPS via the ACM cert |
+| Git-managed (by Argo CD) | Deployment, Service, **PVC** | The app + its Viperblock-backed volume |
+
+**Spinifex specifics**
+
+- The `argocd` and `aws-ebs-csi-driver` bundles must be baked into the `eks-node` AMI — `describe-addon` returns `no baked bundle` otherwise.
+- The EBS-CSI default StorageClass uses `provisioner: ebs.csi.aws.com`; a PVC against it provisions a **Viperblock-backed EBS volume**.
+- **Verify dynamic provisioning on a live cluster.** EBS-CSI dynamic provisioning depends on the running Spinifex EBS/Viperblock data plane and may not be available on every build yet. If the PVC stays `Pending`, the app still serves (the counter falls back to in-memory) — confirm the volume binds on a stack where the driver is healthy.
+- Leave addon versions unset (the AWS provider and the catalog disagree on the version-string format).
+- The Argo CD `Application` is a `kubernetes_manifest`, so the `argoproj.io` CRDs must exist before `apply` — apply the parent and let the addon reach `ACTIVE` first.
+
+**Prerequisites:**
+
+- Spinifex running with an `eks-node` image carrying the LBC, Argo CD, and EBS-CSI bundles
+- OpenTofu or Terraform, plus `kubectl`, the AWS CLI, and Docker
+- The demo image built and pushed to ECR (see [`../demo-app/README.md`](../demo-app/README.md))
+- A git repo with the app manifests ([`mulgadc/eks-demo-app`](https://github.com/mulgadc/eks-demo-app)); a read-only PAT if it is private
+
+## Instructions
+
+### Step 1. Get the Template
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-gitops-argocd
+```
+
+<!-- INCLUDE: main.tf lang:hcl -->
+
+### Step 2. Deploy the Cluster + Addons
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply        # add -var node_desired_size=3 for an HA-shaped cluster
+```
+
+Wait for the cluster to reach `ACTIVE` and for all three addons (`aws-load-balancer-controller`, `argocd`, `aws-ebs-csi-driver`) to report healthy:
+
+```bash
+aws eks list-addons --cluster-name eks-gitops
+aws eks describe-addon --cluster-name eks-gitops --addon-name argocd --query 'addon.status'
+```
+
+### Step 3. Build and Push the Demo Image
+
+```bash
+cd ../demo-app
+REGISTRY=$(cd ../eks-gitops-argocd && tofu output -raw ecr_repository_url)
+REGISTRY_HOST=${REGISTRY%%/*}
+aws ecr get-login-password | docker login --username AWS --password-stdin "$REGISTRY_HOST"
+docker build -t "${REGISTRY}:latest" .
+docker push "${REGISTRY}:latest"
+cd ../eks-gitops-argocd
+```
+
+Point the app's manifests at this image: set the image ref in `eks-demo-app/manifests` (a kustomize `images:` override) to your `${REGISTRY}:latest`, and push that to the git repo.
+
+### Step 4. Hand Delivery to Argo CD
+
+```bash
+cd workloads
+tofu init
+tofu apply \
+  -var git_repo_url=https://github.com/mulgadc/eks-demo-app.git \
+  -var git_token=<read-only-PAT>     # omit for a public repo
+```
+
+This registers the repo credential, creates the Argo CD `Application`, the demo app's HTTPS Ingress, and an HTTPS Ingress for the Argo CD UI itself. Argo CD then syncs the Deployment, Service, and PVC from git.
+
+<!-- INCLUDE: workloads/main.tf lang:hcl -->
+
+### Step 5. Watch the Sync and Open the App
+
+The demo app and the Argo CD UI share **one ALB** (an LBC IngressGroup). On
+`:443` the app is the catch-all and `argocd.eks-gitops.spinifex.local`
+host-routes to the UI; Argo CD also gets a hostless `:8443` listener so it is
+reachable on the raw ALB IP before DNS exists. Grab the shared ALB address once:
+
+```bash
+# Point kubectl at the cluster first (writes ~/.kube/config) — without this every
+# kubectl call fails to connect:
+aws eks update-kubeconfig --name eks-gitops --region ap-southeast-2
+
+kubectl -n argocd get applications
+kubectl get pods -o wide
+kubectl get pvc                    # the EBS-CSI volume should be Bound
+
+DNSNAME=$(kubectl get ingress spinifex-demo -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
+ALB_IP=$(aws elbv2 describe-load-balancers \
+  --query "LoadBalancers[?DNSName=='${DNSNAME}'].AvailabilityZones[].LoadBalancerAddresses[].IpAddress | [0]" \
+  --output text)
+echo "$ALB_IP"
+```
+
+**Reach it by raw IP (no DNS, no host header):**
+
+| Service | URL |
+|---|---|
+| Demo app | `https://<ALB_IP>/` |
+| Argo CD UI | `https://<ALB_IP>:8443/` |
+
+```bash
+curl -k https://"$ALB_IP"/            # app — returns the Spinifex page
+curl -k https://"$ALB_IP":8443/       # Argo CD UI
+```
+
+The bare ALB IP serves the **app** on `:443`. Hitting `https://<ALB_IP>/` and
+expecting Argo CD gives the app instead — use `:8443` for the UI. Once northstar
+(or Route 53) resolves the hostnames to the ALB, `app.eks-gitops.spinifex.local`
+and `argocd.eks-gitops.spinifex.local` both work on `:443`. Self-signed cert —
+accept the browser warning.
+
+The page shows the **"persisted to EBS volume"** badge and a hit counter that keeps climbing — delete the pod (`kubectl delete pod -l app=spinifex-demo`) and the count survives, proving the volume is durable.
+
+### Step 6. Open the Argo CD UI
+
+Managing deployments through the Argo CD console is the point of this workbook, so
+the UI is exposed on the **same ALB** as the app, not behind a port-forward. The
+`workloads` module adds a `NodePort` Service in front of `argocd-server` (the addon
+ships it `ClusterIP` only) and two Ingresses in the shared group — a host-routed
+one on `:443` and a hostless one on `:8443` for raw-IP access. `argocd-server`
+serves TLS on its own port, so both use `backend-protocol: HTTPS`.
+
+**Get the admin credentials.** Argo CD generates a one-time `admin` password into
+a Secret on install; the username is always `admin`:
+
+```bash
+kubectl -n argocd get secret argocd-initial-admin-secret \
+  -o jsonpath='{.data.password}' | base64 -d ; echo
+```
+
+(`tofu output argocd_admin_password_cmd` prints this same command.)
+
+Open the UI and log in as `admin`:
+
+- By raw IP: `https://<ALB_IP>:8443/`
+- By DNS once wired: `https://argocd.eks-gitops.spinifex.local`
+
+The `spinifex-demo` Application shows the sync status, the resource tree, and live
+diffs against git — change a manifest in the repo and watch Argo CD reconcile it.
+
+If you'd rather not expose the UI at all, port-forward instead:
+
+```bash
+kubectl -n argocd port-forward svc/argocd-server 8080:443
+# then open https://localhost:8080
+```
+
+### Cleanup
+
+Destroy in reverse — the Argo CD Application + Ingress first (so Argo CD prunes the workload and the LBC tears the ALB down), then the cluster:
+
+```bash
+cd workloads
+tofu destroy
+cd ..
+tofu destroy
+```
+
+## Troubleshooting
+
+### Application Won't Sync
+
+```bash
+kubectl -n argocd get applications
+kubectl -n argocd describe application spinifex-demo     # conditions show repo/auth errors
+kubectl -n argocd logs deploy/argocd-repo-server --tail=50
+```
+
+For a private repo, confirm the `eks-demo-app-repo` Secret exists in the `argocd` namespace with the `argocd.argoproj.io/secret-type=repository` label and a valid token.
+
+### `kubernetes_manifest` Fails: no matches for kind "Application"
+
+The `argoproj.io` CRDs aren't installed yet. Apply the parent module and wait for the `argocd` addon to reach `ACTIVE` before applying `workloads/`.
+
+### PVC Stuck in Pending
+
+```bash
+kubectl get pvc
+kubectl describe pvc <name>
+kubectl get storageclass
+```
+
+Confirm the `aws-ebs-csi-driver` addon is `ACTIVE` and the default StorageClass exists. EBS-CSI dynamic provisioning depends on the live Spinifex EBS/Viperblock data plane — verify on a stack where the driver is healthy. The app still serves with an in-memory counter if the volume never binds.
+
+### The Ingress Has No Address
+
+The LBC populates `status.loadBalancer` once it provisions the ALB. Check the addon and the controller logs:
+
+```bash
+kubectl describe ingress spinifex-demo
+kubectl -n kube-system logs deploy/aws-load-balancer-controller --tail=50
+```
+
+Confirm the cluster carries `spinifex.io/managed-ingress = "false"` so the LBC owns ingress.
+
+If the controller logs show `FailedBuildModel ... DescribeAvailabilityZones ...
+403 ... AccessDenied`, the node role is missing the LBC permissions. The
+controller runs with the node instance-profile credentials, so the
+`${var.cluster_name}-node-lbc` policy (`aws_iam_policy.node_lbc`) must be
+attached to the node role — re-run `tofu apply` on the parent module if it was
+provisioned before that policy existed.
+
+### Provider Connection Refused
+
+```bash
+sudo systemctl status spinifex.target
+curl -k https://localhost:9999/
+```

docs/terraform-workbooks/eks-gitops-argocd/main.tf

@@ -0,0 +1,745 @@
+# Example: GitOps on EKS with Argo CD + a persistent EBS-CSI volume
+#
+# The top rung of the EKS ladder. It extends eks-https-ingress (LBC + ACM HTTPS)
+# and changes how the app is delivered: instead of Terraform applying the
+# workload, the Argo CD addon syncs a more elaborate Spinifex-themed app from a
+# git repository, and the app stores state on a Viperblock-backed EBS volume
+# provisioned dynamically through the EBS-CSI driver.
+#
+# What it adds over eks-https-ingress:
+#   * The argocd addon — GitOps continuous delivery, installed via the EKS API.
+#   * The aws-ebs-csi-driver addon — dynamic EBS (Viperblock) PersistentVolumes.
+#   * An access entry granting a second IAM principal read-only cluster access.
+#
+# The workloads/ module registers the (private) git repo with Argo CD, creates an
+# Argo CD Application that syncs the app from it, and keeps the HTTPS Ingress
+# (LBC + ACM) pointing at the git-managed Service. The app's manifests — including
+# the PersistentVolumeClaim — live in the git repo (see ../../../../eks-demo-app).
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-gitops-argocd
+#   export AWS_PROFILE=spinifex
+#   tofu init && tofu apply
+#   # build + push the demo image to the ECR repo this creates (see README),
+#   # then: cd workloads && tofu init && tofu apply -var git_repo_url=... -var git_token=...
+#   # finally: kubectl get ingress spinifex-demo -o wide  → open https://<address>
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    tls = {
+      source  = "hashicorp/tls"
+      version = ">= 4.0"
+    }
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Variables
+# ---------------------------------------------------------------------------
+
+variable "region" {
+  type    = string
+  default = "ap-southeast-2"
+}
+
+variable "cluster_name" {
+  type    = string
+  default = "eks-gitops"
+}
+
+variable "k8s_version" {
+  type    = string
+  default = "1.32"
+}
+
+variable "node_instance_type" {
+  type    = string
+  default = "t3.medium"
+}
+
+variable "node_desired_size" {
+  type        = number
+  default     = 1
+  description = "Worker count. Use 1 for a single-node demo, or 3 for an HA-shaped cluster."
+
+  validation {
+    condition     = var.node_desired_size == 1 || var.node_desired_size == 3
+    error_message = "node_desired_size must be 1 or 3."
+  }
+}
+
+variable "node_port" {
+  type        = number
+  default     = 30080
+  description = "NodePort the demo Service is published on and the ALB forwards to"
+}
+
+variable "argocd_node_port" {
+  type        = number
+  default     = 30081
+  description = "NodePort the Argo CD UI Service is published on and its ALB forwards to"
+}
+
+variable "cert_common_name" {
+  type    = string
+  default = "eks-gitops.spinifex.local"
+}
+
+variable "api_public_access_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the public Kubernetes API endpoint; tighten in production"
+}
+
+variable "spinifex_endpoint" {
+  type    = string
+  default = "https://127.0.0.1:9999"
+}
+
+# ---------------------------------------------------------------------------
+# Providers
+# ---------------------------------------------------------------------------
+
+provider "aws" {
+  region = var.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+    ecr = var.spinifex_endpoint
+    acm = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+data "aws_availability_zones" "available" {
+  state = "available"
+}
+
+# ---------------------------------------------------------------------------
+# VPC — public subnets for the ALB + NAT, private subnets for the workers
+# ---------------------------------------------------------------------------
+
+resource "aws_vpc" "main" {
+  cidr_block           = "10.32.0.0/16"
+  enable_dns_hostnames = true
+  enable_dns_support   = true
+
+  tags = {
+    Name = "${var.cluster_name}-vpc"
+  }
+}
+
+resource "aws_internet_gateway" "igw" {
+  vpc_id = aws_vpc.main.id
+
+  tags = {
+    Name = "${var.cluster_name}-igw"
+  }
+}
+
+resource "aws_subnet" "public_a" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.32.1.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-public-a"
+  }
+}
+
+resource "aws_subnet" "public_b" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.32.2.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-public-b"
+  }
+}
+
+resource "aws_subnet" "private_a" {
+  vpc_id            = aws_vpc.main.id
+  cidr_block        = "10.32.11.0/24"
+  availability_zone = data.aws_availability_zones.available.names[0]
+
+  tags = {
+    Name = "${var.cluster_name}-private-a"
+  }
+}
+
+resource "aws_subnet" "private_b" {
+  vpc_id            = aws_vpc.main.id
+  cidr_block        = "10.32.12.0/24"
+  availability_zone = data.aws_availability_zones.available.names[0]
+
+  tags = {
+    Name = "${var.cluster_name}-private-b"
+  }
+}
+
+resource "aws_route_table" "public" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block = "0.0.0.0/0"
+    gateway_id = aws_internet_gateway.igw.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-public-rt"
+  }
+}
+
+resource "aws_route_table_association" "public_a" {
+  subnet_id      = aws_subnet.public_a.id
+  route_table_id = aws_route_table.public.id
+}
+
+resource "aws_route_table_association" "public_b" {
+  subnet_id      = aws_subnet.public_b.id
+  route_table_id = aws_route_table.public.id
+}
+
+# NAT gateway gives the private workers outbound internet to pull the demo image
+# from ECR. The workers still join the cluster over the in-VPC private endpoint.
+resource "aws_eip" "nat" {
+  domain = "vpc"
+
+  tags = {
+    Name = "${var.cluster_name}-nat-eip"
+  }
+}
+
+resource "aws_nat_gateway" "nat" {
+  allocation_id = aws_eip.nat.id
+  subnet_id     = aws_subnet.public_a.id
+
+  depends_on = [aws_internet_gateway.igw]
+
+  tags = {
+    Name = "${var.cluster_name}-nat"
+  }
+}
+
+resource "aws_route_table" "private" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block     = "0.0.0.0/0"
+    nat_gateway_id = aws_nat_gateway.nat.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-private-rt"
+  }
+}
+
+resource "aws_route_table_association" "private_a" {
+  subnet_id      = aws_subnet.private_a.id
+  route_table_id = aws_route_table.private.id
+}
+
+resource "aws_route_table_association" "private_b" {
+  subnet_id      = aws_subnet.private_b.id
+  route_table_id = aws_route_table.private.id
+}
+
+# ---------------------------------------------------------------------------
+# IAM — cluster role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "cluster" {
+  name = "${var.cluster_name}-cluster-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "eks.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "cluster" {
+  role       = aws_iam_role.cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — node role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "node" {
+  name = "${var.cluster_name}-node-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_worker" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_cni" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_ecr" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+}
+
+# The AWS Load Balancer Controller runs with the node's instance-profile
+# credentials (Spinifex wires creds at the node level, not IRSA — see the addon
+# block below), so the permissions it needs to manage ALBs must live on the node
+# role. This mirrors the upstream AWSLoadBalancerControllerIAMPolicy, trimmed to
+# the actions an ALB Ingress exercises (Shield/WAF/Cognito are disabled on the
+# controller). Resources are "*" since Spinifex evaluates grants by action.
+# A customer-managed policy + attachment is used (Spinifex implements
+# CreatePolicy/AttachRolePolicy, not inline PutRolePolicy).
+resource "aws_iam_policy" "node_lbc" {
+  name = "${var.cluster_name}-node-lbc"
+
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Effect = "Allow"
+        Action = [
+          "ec2:DescribeAccountAttributes",
+          "ec2:DescribeAddresses",
+          "ec2:DescribeAvailabilityZones",
+          "ec2:DescribeInternetGateways",
+          "ec2:DescribeVpcs",
+          "ec2:DescribeSubnets",
+          "ec2:DescribeSecurityGroups",
+          "ec2:DescribeInstances",
+          "ec2:DescribeNetworkInterfaces",
+          "ec2:DescribeTags",
+          "ec2:GetCoipPoolUsage",
+          "ec2:DescribeCoipPools",
+          "ec2:CreateSecurityGroup",
+          "ec2:CreateTags",
+          "ec2:DeleteTags",
+          "ec2:AuthorizeSecurityGroupIngress",
+          "ec2:RevokeSecurityGroupIngress",
+          "ec2:DeleteSecurityGroup",
+          "elasticloadbalancing:*",
+          "acm:ListCertificates",
+          "acm:DescribeCertificate",
+          "acm:GetCertificate",
+          "iam:CreateServiceLinkedRole",
+          "iam:ListServerCertificates",
+          "iam:GetServerCertificate",
+          "tag:GetResources",
+          "tag:TagResources",
+          "wafv2:GetWebACLForResource",
+          "shield:GetSubscriptionState"
+        ]
+        Resource = "*"
+      }
+    ]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_lbc" {
+  role       = aws_iam_role.node.name
+  policy_arn = aws_iam_policy.node_lbc.arn
+}
+
+# The EBS CSI driver also runs with the node's instance-profile credentials, so
+# the volume-lifecycle permissions it needs must live on the node role too. This
+# mirrors the upstream AmazonEBSCSIDriverPolicy. Resources are "*" since Spinifex
+# evaluates grants by action; the CreateVolume/CreateSnapshot/CreateTags grants
+# upstream scope with request/resource tag conditions, omitted here because
+# Spinifex does not yet evaluate IAM tag conditions.
+resource "aws_iam_policy" "node_ebs_csi" {
+  name = "${var.cluster_name}-node-ebs-csi"
+
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Effect = "Allow"
+        Action = [
+          "ec2:CreateSnapshot",
+          "ec2:AttachVolume",
+          "ec2:DetachVolume",
+          "ec2:ModifyVolume",
+          "ec2:DescribeAvailabilityZones",
+          "ec2:DescribeInstances",
+          "ec2:DescribeSnapshots",
+          "ec2:DescribeTags",
+          "ec2:DescribeVolumes",
+          "ec2:DescribeVolumesModifications",
+          "ec2:CreateTags",
+          "ec2:DeleteTags",
+          "ec2:CreateVolume",
+          "ec2:DeleteVolume",
+          "ec2:DeleteSnapshot"
+        ]
+        Resource = "*"
+      }
+    ]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_ebs_csi" {
+  role       = aws_iam_role.node.name
+  policy_arn = aws_iam_policy.node_ebs_csi.arn
+}
+
+# ---------------------------------------------------------------------------
+# IAM — a second principal to grant read-only cluster access to
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "viewer" {
+  name = "${var.cluster_name}-viewer"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+# ---------------------------------------------------------------------------
+# ECR — repository the workers pull the demo image from
+# ---------------------------------------------------------------------------
+
+resource "aws_ecr_repository" "demo" {
+  name = "spinifex-demo"
+
+  # tofu destroy must remove the repo even though it still holds the pushed demo
+  # image; without this, DeleteRepository (force=false) fails RepositoryNotEmpty.
+  force_delete = true
+}
+
+# ---------------------------------------------------------------------------
+# EKS cluster — public + private endpoints, LBC-owned ingress
+#
+# The spinifex.io/managed-ingress = "false" tag disables K3s' built-in
+# traefik/servicelb so the AWS Load Balancer Controller owns ingress, matching
+# how ingress works on AWS EKS.
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_cluster" "this" {
+  name     = var.cluster_name
+  role_arn = aws_iam_role.cluster.arn
+  version  = var.k8s_version
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.private_a.id, aws_subnet.private_b.id]
+    endpoint_public_access  = true
+    endpoint_private_access = true
+    public_access_cidrs     = [var.api_public_access_cidr]
+  }
+
+  access_config {
+    authentication_mode                         = "API"
+    bootstrap_cluster_creator_admin_permissions = true
+  }
+
+  depends_on = [aws_iam_role_policy_attachment.cluster]
+
+  # Spinifex's DescribeCluster doesn't echo these back the way the AWS provider
+  # expects: access_config reads as absent (and its ForceNew
+  # bootstrap_cluster_creator_admin_permissions would replace the live cluster on
+  # re-add), and vpc_config.security_group_ids reads back as drift. Ignore both so
+  # later applies (e.g. adding the Argo CD NodePort rule) neither replace the
+  # cluster nor detach its security groups.
+  lifecycle {
+    ignore_changes = [access_config, vpc_config[0].security_group_ids]
+  }
+
+  tags = {
+    Name                          = var.cluster_name
+    "spinifex.io/managed-ingress" = "false"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Managed node group — workers in the private subnets
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_node_group" "workers" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "workers"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.private_a.id, aws_subnet.private_b.id]
+
+  scaling_config {
+    desired_size = var.node_desired_size
+    min_size     = var.node_desired_size
+    max_size     = var.node_desired_size * 2
+  }
+
+  instance_types = [var.node_instance_type]
+  ami_type       = "AL2_x86_64"
+
+  depends_on = [
+    aws_iam_role_policy_attachment.node_worker,
+    aws_iam_role_policy_attachment.node_cni,
+    aws_iam_role_policy_attachment.node_ecr,
+    aws_iam_role_policy_attachment.node_lbc,
+    aws_iam_role_policy_attachment.node_ebs_csi,
+  ]
+
+  tags = {
+    Name = "${var.cluster_name}-workers"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Addon — AWS Load Balancer Controller
+#
+# Spinifex wires the controller's AWS credentials and ELB-eligible subnets at the
+# node level, so no IRSA role or subnet tagging is needed here. addon_version is
+# omitted: the AWS provider demands a v-prefixed version that the catalog rejects,
+# so let Spinifex default to its catalog version.
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_addon" "lbc" {
+  cluster_name                = aws_eks_cluster.this.name
+  addon_name                  = "aws-load-balancer-controller"
+  resolve_conflicts_on_create = "OVERWRITE"
+
+  depends_on = [aws_eks_node_group.workers]
+
+  tags = {
+    Name = "${var.cluster_name}-lbc"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Addon — Argo CD (GitOps delivery)
+#
+# Installed through the EKS API; Spinifex stages the bundle host-side and the
+# worker renders it into the K3s auto-deploy dir. The workloads module registers
+# the git repo and creates the Argo CD Application that syncs the demo app.
+# addon_version omitted (see the LBC note).
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_addon" "argocd" {
+  cluster_name                = aws_eks_cluster.this.name
+  addon_name                  = "argocd"
+  resolve_conflicts_on_create = "OVERWRITE"
+
+  depends_on = [aws_eks_node_group.workers]
+
+  tags = {
+    Name = "${var.cluster_name}-argocd"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Addon — EBS CSI driver (Viperblock-backed PersistentVolumes)
+#
+# Ships a default gp3 StorageClass (provisioner ebs.csi.aws.com). A PVC in the
+# demo app's git manifests dynamically provisions a Viperblock-backed EBS volume
+# so the app's state survives pod restarts. addon_version omitted (see LBC note).
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_addon" "ebs_csi" {
+  cluster_name                = aws_eks_cluster.this.name
+  addon_name                  = "aws-ebs-csi-driver"
+  resolve_conflicts_on_create = "OVERWRITE"
+
+  depends_on = [aws_eks_node_group.workers]
+
+  tags = {
+    Name = "${var.cluster_name}-ebs-csi"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Access entry — grant the viewer principal read-only access (API auth mode)
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_access_entry" "viewer" {
+  cluster_name  = aws_eks_cluster.this.name
+  principal_arn = aws_iam_role.viewer.arn
+  type          = "STANDARD"
+}
+
+resource "aws_eks_access_policy_association" "viewer" {
+  cluster_name  = aws_eks_cluster.this.name
+  principal_arn = aws_iam_role.viewer.arn
+  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
+
+  access_scope {
+    type = "cluster"
+  }
+
+  depends_on = [aws_eks_access_entry.viewer]
+}
+
+# ---------------------------------------------------------------------------
+# TLS — self-signed certificate imported into ACM
+#
+# Spinifex ACM supports ImportCertificate (not RequestCertificate), so the cert
+# is generated locally by the tls provider and imported. The workloads Ingress
+# attaches it to the ALB's HTTPS listener by ARN.
+# ---------------------------------------------------------------------------
+
+resource "tls_private_key" "ingress" {
+  algorithm = "RSA"
+  rsa_bits  = 2048
+}
+
+resource "tls_self_signed_cert" "ingress" {
+  private_key_pem = tls_private_key.ingress.private_key_pem
+
+  subject {
+    common_name  = var.cert_common_name
+    organization = "Spinifex EKS Demo"
+  }
+
+  # Wildcard SAN so one ALB (shared IngressGroup) can host-route both the demo
+  # app (app.<cn>) and the Argo CD UI (argocd.<cn>) off this single cert.
+  dns_names             = [var.cert_common_name, "*.${var.cert_common_name}"]
+  validity_period_hours = 8760
+  early_renewal_hours   = 720
+
+  allowed_uses = [
+    "key_encipherment",
+    "digital_signature",
+    "server_auth",
+  ]
+}
+
+resource "aws_acm_certificate" "ingress" {
+  private_key      = tls_private_key.ingress.private_key_pem
+  certificate_body = tls_self_signed_cert.ingress.cert_pem
+
+  tags = {
+    Name = "${var.cluster_name}-ingress"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Let the ALB reach the workers' NodePort
+#
+# Spinifex auto-manages the nodegroup SG and admits only intra-cluster traffic.
+# The LBC-provisioned ALB lives in this VPC, so admit the VPC CIDR on the
+# NodePort. Look the SG up by its deterministic name and add one rule.
+# ---------------------------------------------------------------------------
+
+data "aws_security_group" "nodegroup" {
+  filter {
+    name   = "group-name"
+    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
+  }
+
+  filter {
+    name   = "vpc-id"
+    values = [aws_vpc.main.id]
+  }
+
+  depends_on = [aws_eks_node_group.workers]
+}
+
+resource "aws_vpc_security_group_ingress_rule" "nodeport_from_vpc" {
+  security_group_id = data.aws_security_group.nodegroup.id
+  cidr_ipv4         = aws_vpc.main.cidr_block
+  from_port         = var.node_port
+  to_port           = var.node_port
+  ip_protocol       = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-alb-nodeport"
+  }
+}
+
+# Same, for the Argo CD UI NodePort its ALB forwards to.
+resource "aws_vpc_security_group_ingress_rule" "argocd_nodeport_from_vpc" {
+  security_group_id = data.aws_security_group.nodegroup.id
+  cidr_ipv4         = aws_vpc.main.cidr_block
+  from_port         = var.argocd_node_port
+  to_port           = var.argocd_node_port
+  ip_protocol       = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-argocd-nodeport"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Outputs
+# ---------------------------------------------------------------------------
+
+output "cluster_name" {
+  value = aws_eks_cluster.this.name
+}
+
+output "region" {
+  value = var.region
+}
+
+output "node_port" {
+  value = var.node_port
+}
+
+output "argocd_node_port" {
+  value = var.argocd_node_port
+}
+
+output "node_desired_size" {
+  value = var.node_desired_size
+}
+
+output "certificate_arn" {
+  value = aws_acm_certificate.ingress.arn
+}
+
+output "cert_common_name" {
+  value = var.cert_common_name
+}
+
+output "viewer_principal_arn" {
+  value = aws_iam_role.viewer.arn
+}
+
+output "ecr_repository_url" {
+  value       = aws_ecr_repository.demo.repository_url
+  description = "Push the demo-app image here, then apply the workloads module"
+}
+
+output "ingress_address_hint" {
+  value = "The demo app and Argo CD UI share one ALB (LBC IngressGroup), host-routed: app.${var.cert_common_name} and argocd.${var.cert_common_name}. Get the address: kubectl get ingress spinifex-demo -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{\"\\n\"}'."
+}
+
+output "argocd_ingress_hint" {
+  value = "Same ALB as the app. Open https://argocd.${var.cert_common_name} (admin password: kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d). Resolve the host to the ALB address, or curl -k --resolve."
+}
+
+output "update_kubeconfig" {
+  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
+}

docs/terraform-workbooks/eks-gitops-argocd/workloads/main.tf

@@ -0,0 +1,370 @@
+# GitOps workload for eks-gitops-argocd — separate root module / state.
+#
+# Instead of applying the app directly, this module hands delivery to Argo CD:
+# it registers the (private) git repo as an Argo CD repository credential and
+# creates an Argo CD Application that syncs the demo app's manifests from it. The
+# app's Deployment, Service, and PersistentVolumeClaim live in the git repo (see
+# ../../../../eks-demo-app); the HTTPS Ingress (LBC + ACM) stays here and points
+# at the Service that Argo CD creates.
+#
+# The Argo CD Application is a kubernetes_manifest, so the argoproj.io CRDs must
+# already exist — apply the parent module (which installs the argocd addon) and
+# let it reach ACTIVE before applying this module.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-gitops-argocd
+#   tofu init && tofu apply                       # parent: cluster + addons + ACM
+#   cd workloads && tofu init
+#   tofu apply -var git_repo_url=https://github.com/mulgadc/eks-demo-app.git \
+#              -var git_token=<a-read-only-PAT>
+#   # teardown is the reverse — destroy here first, then the parent.
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.20"
+    }
+  }
+}
+
+variable "spinifex_endpoint" {
+  type    = string
+  default = "https://127.0.0.1:9999"
+}
+
+variable "git_repo_url" {
+  type        = string
+  default     = "https://github.com/mulgadc/eks-demo-app.git"
+  description = "Git repo Argo CD syncs the demo app from"
+}
+
+variable "git_revision" {
+  type        = string
+  default     = "main"
+  description = "Git branch, tag, or commit Argo CD tracks"
+}
+
+variable "git_path" {
+  type        = string
+  default     = "manifests"
+  description = "Path within the repo holding the app manifests"
+}
+
+variable "git_username" {
+  type        = string
+  default     = "git"
+  description = "Username for the git credential (any non-empty value for a PAT)"
+}
+
+variable "git_token" {
+  type        = string
+  default     = ""
+  sensitive   = true
+  description = "Read-only personal access token for the private repo. Leave empty for a public repo."
+}
+
+provider "aws" {
+  region = data.terraform_remote_state.infra.outputs.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+data "terraform_remote_state" "infra" {
+  backend = "local"
+
+  config = {
+    path = "../terraform.tfstate"
+  }
+}
+
+locals {
+  cluster_name     = data.terraform_remote_state.infra.outputs.cluster_name
+  region           = data.terraform_remote_state.infra.outputs.region
+  cert_arn         = data.terraform_remote_state.infra.outputs.certificate_arn
+  argocd_node_port = data.terraform_remote_state.infra.outputs.argocd_node_port
+  cert_cn          = data.terraform_remote_state.infra.outputs.cert_common_name
+  private_repo     = var.git_token != ""
+
+  # The demo app and the Argo CD UI share one ALB via an LBC IngressGroup; LBC
+  # gives each backend its own target group and host-routes between them.
+  alb_group   = local.cluster_name
+  app_host    = "app.${local.cert_cn}"
+  argocd_host = "argocd.${local.cert_cn}"
+}
+
+data "aws_eks_cluster" "this" {
+  name = local.cluster_name
+}
+
+provider "kubernetes" {
+  host                   = data.aws_eks_cluster.this.endpoint
+  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
+
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    command     = "aws"
+    args        = ["eks", "get-token", "--cluster-name", local.cluster_name, "--region", local.region]
+  }
+}
+
+# Repository credential for the private repo. Argo CD picks up Secrets in its
+# namespace labelled argocd.argoproj.io/secret-type=repository. Created only when
+# a token is supplied (a public repo needs no credential).
+resource "kubernetes_secret_v1" "repo" {
+  count = local.private_repo ? 1 : 0
+
+  metadata {
+    name      = "eks-demo-app-repo"
+    namespace = "argocd"
+    labels = {
+      "argocd.argoproj.io/secret-type" = "repository"
+    }
+  }
+
+  data = {
+    type     = "git"
+    url      = var.git_repo_url
+    username = var.git_username
+    password = var.git_token
+  }
+}
+
+# Argo CD Application: syncs the demo app from the git repo into the default
+# namespace, self-healing and pruning so the cluster tracks the repo.
+resource "kubernetes_manifest" "demo_app" {
+  manifest = {
+    apiVersion = "argoproj.io/v1alpha1"
+    kind       = "Application"
+
+    # The resources finalizer makes deleting this Application cascade-prune the
+    # synced resources (Deployment, Service, PVC). Without it, tofu destroy drops
+    # the Application CR but orphans the live PVC, whose Delete-reclaim PV then
+    # leaks an EBS volume once the cluster (and ebs-csi) is torn down.
+    metadata = {
+      name       = "spinifex-demo"
+      namespace  = "argocd"
+      finalizers = ["resources-finalizer.argocd.argoproj.io"]
+    }
+
+    spec = {
+      project = "default"
+
+      source = {
+        repoURL        = var.git_repo_url
+        targetRevision = var.git_revision
+        path           = var.git_path
+      }
+
+      destination = {
+        server    = "https://kubernetes.default.svc"
+        namespace = "default"
+      }
+
+      syncPolicy = {
+        automated = {
+          prune    = true
+          selfHeal = true
+        }
+        syncOptions = ["CreateNamespace=true"]
+      }
+    }
+  }
+
+  depends_on = [kubernetes_secret_v1.repo]
+}
+
+# HTTPS Ingress (LBC + ACM), carried over from eks-https-ingress. It points at
+# the spinifex-demo Service that Argo CD creates from the git manifests.
+resource "kubernetes_ingress_v1" "demo" {
+  metadata {
+    name      = "spinifex-demo"
+    namespace = "default"
+
+    annotations = {
+      "alb.ingress.kubernetes.io/scheme"           = "internet-facing"
+      "alb.ingress.kubernetes.io/target-type"      = "instance"
+      "alb.ingress.kubernetes.io/group.name"       = local.alb_group
+      "alb.ingress.kubernetes.io/listen-ports"     = "[{\"HTTP\":80},{\"HTTPS\":443}]"
+      "alb.ingress.kubernetes.io/certificate-arn"  = local.cert_arn
+      "alb.ingress.kubernetes.io/ssl-redirect"     = "443"
+      "alb.ingress.kubernetes.io/healthcheck-path" = "/healthz"
+    }
+  }
+
+  spec {
+    ingress_class_name = "alb"
+
+    # No host condition: the app is the ALB's catch-all on :443, so it answers on
+    # the raw ALB IP as well as app.<cn> once DNS is wired. Argo CD's host rule is
+    # more specific and still wins for argocd.<cn>.
+    rule {
+      http {
+        path {
+          path      = "/"
+          path_type = "Prefix"
+
+          backend {
+            service {
+              name = "spinifex-demo"
+              port {
+                number = 80
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+
+  depends_on = [kubernetes_manifest.demo_app]
+}
+
+# ---------------------------------------------------------------------------
+# Argo CD UI exposure — managing the cluster through the GitOps console is the
+# point of this demo, so the UI gets the same HTTPS-Ingress treatment as the app
+# rather than a port-forward. The argocd addon ships argocd-server as a ClusterIP
+# only; expose it via a NodePort the ALB targets. argocd-server serves TLS on
+# 8080 (no --insecure), so the ALB speaks HTTPS to the backend.
+#
+# Same group.name as the demo Ingress, so LBC folds both onto ONE ALB. On :443
+# the app is the catch-all and argocd.<cn> host-routes to the UI. Argo CD also
+# gets a hostless :8443 listener (argocd_ip below) so the UI is reachable on the
+# raw ALB IP before DNS exists — https://<alb-ip>:8443.
+# ---------------------------------------------------------------------------
+
+resource "kubernetes_service_v1" "argocd_server_nodeport" {
+  metadata {
+    name      = "argocd-server-nodeport"
+    namespace = "argocd"
+  }
+
+  spec {
+    type     = "NodePort"
+    selector = { "app.kubernetes.io/name" = "argocd-server" }
+
+    port {
+      port        = 443
+      target_port = 8080
+      node_port   = local.argocd_node_port
+      protocol    = "TCP"
+    }
+  }
+}
+
+resource "kubernetes_ingress_v1" "argocd" {
+  metadata {
+    name      = "argocd-server"
+    namespace = "argocd"
+
+    annotations = {
+      "alb.ingress.kubernetes.io/scheme"               = "internet-facing"
+      "alb.ingress.kubernetes.io/target-type"          = "instance"
+      "alb.ingress.kubernetes.io/group.name"           = local.alb_group
+      "alb.ingress.kubernetes.io/listen-ports"         = "[{\"HTTPS\":443}]"
+      "alb.ingress.kubernetes.io/certificate-arn"      = local.cert_arn
+      "alb.ingress.kubernetes.io/backend-protocol"     = "HTTPS"
+      "alb.ingress.kubernetes.io/healthcheck-protocol" = "HTTPS"
+      "alb.ingress.kubernetes.io/healthcheck-path"     = "/healthz"
+    }
+  }
+
+  spec {
+    ingress_class_name = "alb"
+
+    rule {
+      host = local.argocd_host
+
+      http {
+        path {
+          path      = "/"
+          path_type = "Prefix"
+
+          backend {
+            service {
+              name = kubernetes_service_v1.argocd_server_nodeport.metadata[0].name
+              port {
+                number = 443
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+}
+
+# Hostless Argo CD listener on :8443 of the same ALB, so the UI is reachable on
+# the raw ALB IP (https://<alb-ip>:8443) without DNS or a host header. argocd.<cn>
+# on :443 still works via the host-routed ingress above.
+resource "kubernetes_ingress_v1" "argocd_ip" {
+  metadata {
+    name      = "argocd-server-ip"
+    namespace = "argocd"
+
+    annotations = {
+      "alb.ingress.kubernetes.io/scheme"               = "internet-facing"
+      "alb.ingress.kubernetes.io/target-type"          = "instance"
+      "alb.ingress.kubernetes.io/group.name"           = local.alb_group
+      "alb.ingress.kubernetes.io/listen-ports"         = "[{\"HTTPS\":8443}]"
+      "alb.ingress.kubernetes.io/certificate-arn"      = local.cert_arn
+      "alb.ingress.kubernetes.io/backend-protocol"     = "HTTPS"
+      "alb.ingress.kubernetes.io/healthcheck-protocol" = "HTTPS"
+      "alb.ingress.kubernetes.io/healthcheck-path"     = "/healthz"
+    }
+  }
+
+  spec {
+    ingress_class_name = "alb"
+
+    rule {
+      http {
+        path {
+          path      = "/"
+          path_type = "Prefix"
+
+          backend {
+            service {
+              name = kubernetes_service_v1.argocd_server_nodeport.metadata[0].name
+              port {
+                number = 443
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+}
+
+output "application_name" {
+  value = kubernetes_manifest.demo_app.manifest.metadata.name
+}
+
+output "alb_address_hint" {
+  value = "One shared ALB. Get its IP: kubectl get ingress spinifex-demo -o jsonpath='{.status.loadBalancer.ingress[0].ip}{\"\\n\"}' (or [0].hostname). Reach by raw IP — app: https://<alb-ip>/  Argo CD: https://<alb-ip>:8443/ . By DNS — app: ${local.app_host}  Argo CD: ${local.argocd_host}."
+}
+
+output "argocd_admin_password_cmd" {
+  value = "kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d ; echo"
+}
+
+output "argocd_url_hint" {
+  value = "Open https://${local.argocd_host} (admin password: kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d). Resolve the host to the ALB address (northstar/CNAME), or curl -k --resolve ${local.argocd_host}:443:<alb-ip>."
+}

docs/terraform-workbooks/eks-https-ingress/README.md

@@ -0,0 +1,207 @@
+---
+title: "EKS HTTPS Ingress (LBC + ACM)"
+description: "Serve a Spinifex-themed demo app over HTTPS on EKS using the AWS Load Balancer Controller addon and an ACM certificate — an internet-facing ALB provisioned from a Kubernetes Ingress, using Terraform on Spinifex."
+category: "Terraform Workbooks"
+tags:
+  - terraform
+  - eks
+  - kubernetes
+  - ingress
+  - alb
+  - acm
+  - https
+  - workbook
+resources:
+  - title: "AWS Load Balancer Controller"
+    url: "https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/"
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "Terraform Kubernetes Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+---
+
+# Terraform: EKS HTTPS Ingress (LBC + ACM)
+
+> Builds on the quickstart. Instead of a raw NodePort, it serves the Spinifex-themed demo app over **HTTPS** through an ALB that the **AWS Load Balancer Controller** provisions from a Kubernetes **Ingress**, terminating TLS with a certificate held in **ACM** — the same pattern you'd use on AWS EKS.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+This is the second rung of the EKS ladder. It takes the cluster + demo app from [EKS Quickstart](../eks-quickstart) and publishes the app the way you would in production: a Kubernetes `Ingress` reconciled by the **AWS Load Balancer Controller (LBC)** into an internet-facing **Application Load Balancer** that terminates TLS with an **ACM** certificate.
+
+To make room for an internet-facing ALB with private workers, the network grows from the quickstart's two public subnets to a public/private split: the ALB and a **NAT gateway** sit in the public subnets, the workers sit in the private subnets and reach ECR through the NAT. The cluster is tagged `spinifex.io/managed-ingress = "false"`, which disables K3s' built-in traefik/servicelb so the LBC owns ingress.
+
+The LBC is installed as a managed **addon** (`aws_eks_addon`). Spinifex wires the controller's AWS credentials and the cluster's ELB-eligible subnets at the node level and injects those subnets into the `alb` `IngressClassParams`, so the Ingress needs **no IRSA role and no subnet tags** — just `ingressClassName: alb` and a handful of annotations.
+
+**What you'll learn:**
+
+- Installing the `aws-load-balancer-controller` addon and letting it own ingress
+- Disabling the built-in K3s ingress with the `spinifex.io/managed-ingress` tag
+- Importing a self-signed certificate into ACM and attaching it to an ALB listener
+- Driving an ALB entirely from a Kubernetes `Ingress` (`ingressClassName: alb`)
+- A public/private VPC with a NAT gateway for private workers
+
+**What gets created**
+
+| Resource | Name | Purpose |
+|---|---|---|
+| VPC | `eks-https-vpc` | Isolated network (10.31.0.0/16) |
+| Subnets | `eks-https-public-a/-b`, `eks-https-private-a/-b` | Public (ALB + NAT) and private (workers) |
+| Internet + NAT Gateway | `eks-https-igw`, `eks-https-nat` | Public egress; private-worker egress to ECR |
+| IAM Roles | `eks-https-cluster-role`, `eks-https-node-role` | Control-plane and worker roles |
+| ECR Repository | `spinifex-demo` | Holds the demo image the workers pull |
+| EKS Cluster | `eks-https` | Public + private endpoints; `managed-ingress=false` |
+| Node Group | `workers` | `node_desired_size` `t3.large` worker(s) — 1 or 3 |
+| Addon | `aws-load-balancer-controller` | Provisions the ALB from the Ingress |
+| ACM Certificate | `eks-https-ingress` | Self-signed, imported; attached to the HTTPS listener |
+| SG Ingress Rule | `eks-https-alb-nodeport` | Admits the VPC CIDR to the workers' NodePort |
+| K8s Deployment + Service | `spinifex-demo` | Themed demo image, 2 replicas, NodePort |
+| K8s Ingress | `spinifex-demo` | `ingressClassName: alb`, HTTPS via the ACM cert |
+
+**Spinifex specifics**
+
+- **The LBC needs no IRSA or subnet tagging here.** Credentials and ELB-eligible subnets are injected by Spinifex at the node level and into the `alb` `IngressClassParams`. The Ingress just sets `ingressClassName: alb` plus the `alb.ingress.kubernetes.io/*` annotations.
+- **`spinifex.io/managed-ingress = "false"`** disables the K3s built-in traefik/servicelb so the LBC owns ingress, matching AWS parity. Omit the tag (or set it `true`) and the built-in ingress stays on.
+- The ALB targets the workers' **NodePort** (`target-type: instance`). The one SG rule admits the VPC CIDR on that port so the in-VPC ALB can reach the workers; the worker SG is otherwise auto-managed.
+- Spinifex ACM supports **ImportCertificate only** — the cert is generated by the `tls` provider and imported, not requested.
+- **Leave `addon_version` unset** (see the [quickstart/addons notes](../eks-quickstart)) — the AWS provider and the Spinifex catalog disagree on the version string format, so let the server choose.
+
+**Prerequisites:**
+
+- Spinifex installed and running, with the `eks-node` image (carrying the LBC bundle) available
+- OpenTofu or Terraform, plus `kubectl`, the AWS CLI, and Docker
+- The demo image built and pushed to ECR (see [`../demo-app/README.md`](../demo-app/README.md))
+
+## Instructions
+
+### Step 1. Get the Template
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-https-ingress
+```
+
+Or create a `main.tf` file and paste the full configuration below.
+
+<!-- INCLUDE: main.tf lang:hcl -->
+
+### Step 2. Deploy the Cluster
+
+The cluster and the demo app are **two root modules with separate state**. Apply the cluster first (set `node_desired_size=3` for an HA-shaped cluster):
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+```
+
+This brings up the cluster, the workers, the NAT gateway, the LBC addon, the ACM cert, and the `spinifex-demo` ECR repository. Give it a few minutes to reach `ACTIVE` and for the LBC addon to report healthy.
+
+### Step 3. Build and Push the Demo Image
+
+```bash
+cd ../demo-app
+REGISTRY=$(cd ../eks-https-ingress && tofu output -raw ecr_repository_url)
+REGISTRY_HOST=${REGISTRY%%/*}
+aws ecr get-login-password | docker login --username AWS --password-stdin "$REGISTRY_HOST"
+docker build -t "${REGISTRY}:latest" .
+docker push "${REGISTRY}:latest"
+cd ../eks-https-ingress
+```
+
+### Step 4. Deploy the App and Ingress
+
+```bash
+cd workloads
+tofu init
+tofu apply
+```
+
+> **Why two modules?** The Kubernetes provider in `workloads/` reads the cluster endpoint from a live `data "aws_eks_cluster"` source, so it's only configured while the cluster exists. Always destroy `workloads/` before the cluster, or the provider falls back to `http://localhost:80` and reports `connection refused`. Keep `AWS_PROFILE=spinifex` exported for both applies.
+
+<!-- INCLUDE: workloads/main.tf lang:hcl -->
+
+### Step 5. Open the Demo over HTTPS
+
+The LBC takes a minute or two to provision the ALB after the Ingress is created. The Ingress publishes the ALB's **DNS name**, which has no resolver yet (northstar will add one) — resolve it to the ALB's public IP with `describe-load-balancers` so you can reach it without DNS or `/etc/hosts`:
+
+```bash
+# Point kubectl at the cluster first (writes ~/.kube/config) — without this every
+# kubectl call fails to connect:
+aws eks update-kubeconfig --name eks-https --region ap-southeast-2
+
+kubectl get ingress spinifex-demo -o wide
+DNSNAME=$(kubectl get ingress spinifex-demo -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
+IP=$(aws elbv2 describe-load-balancers \
+  --query "LoadBalancers[?DNSName=='${DNSNAME}'].AvailabilityZones[].LoadBalancerAddresses[].IpAddress | [0]" \
+  --output text)
+curl -k "https://$IP"          # self-signed cert: -k skips verification
+```
+
+Open `https://$IP` in a browser (accept the self-signed certificate). The Spinifex-themed page reports the pod, node, cluster, and region that answered — refresh to watch requests land on different replicas, now over HTTPS through the ALB.
+
+### Cleanup
+
+Destroy in reverse — the app/Ingress first (so the LBC tears the ALB down while the cluster is up), then the cluster:
+
+```bash
+cd workloads
+tofu destroy
+cd ..
+tofu destroy
+```
+
+## Troubleshooting
+
+### The Ingress Has No Address
+
+The LBC populates `status.loadBalancer` once it has provisioned the ALB. If it stays empty:
+
+```bash
+kubectl describe ingress spinifex-demo            # events show LBC reconcile errors
+kubectl -n kube-system logs deploy/aws-load-balancer-controller --tail=50
+aws eks describe-addon --cluster-name eks-https --addon-name aws-load-balancer-controller --query 'addon.status'
+```
+
+Confirm the addon is `ACTIVE` and the cluster carries `spinifex.io/managed-ingress = "false"` (otherwise the built-in ingress competes).
+
+> **Verify on a live cluster.** The LBC reconcile path (ALB creation, target registration, listener wiring) exercises the running Spinifex EKS data plane end to end. Stand this up on a live stack to confirm the ALB serves the page; the manifests here are the AWS-parity shape of that flow.
+
+### HTTPS Loads but Returns 502 / 504
+
+The ALB reached the workers but the backend didn't answer. Check the demo pods and the NodePort SG rule:
+
+```bash
+kubectl get pods -o wide
+kubectl get svc spinifex-demo
+aws ec2 describe-security-groups --filters "Name=group-name,Values=eks-cluster-eks-https-nodegroup-sg" \
+  --query 'SecurityGroups[0].IpPermissions'
+```
+
+The `eks-https-alb-nodeport` rule must admit the VPC CIDR on the NodePort, and the target group health check (`/healthz`) must pass.
+
+### Addon Fails With "no baked bundle"
+
+Spinifex only installs addons whose manifests are baked into the `eks-node` AMI. If `describe-addon` reports `CREATE_FAILED` with `no baked bundle`, the running worker image predates the LBC bundle — rebuild and republish the `eks-node` image, then recreate the addon.
+
+### Pods Stuck in ImagePullBackOff
+
+The private workers pull from ECR through the NAT gateway. Confirm the image was pushed (`tofu output ecr_repository_url`), the NAT route is in place, and the node role carries `AmazonEC2ContainerRegistryReadOnly`.
+
+### Provider Connection Refused
+
+```bash
+sudo systemctl status spinifex.target
+curl -k https://localhost:9999/
+```

docs/terraform-workbooks/eks-https-ingress/main.tf

@@ -0,0 +1,577 @@
+# Example: EKS HTTPS Ingress via the AWS Load Balancer Controller + ACM
+#
+# Builds on eks-quickstart. Instead of poking a NodePort open on the worker, this
+# workbook serves the Spinifex-themed demo app over HTTPS through an Application
+# Load Balancer that the AWS Load Balancer Controller (LBC) provisions from a
+# Kubernetes Ingress — the same pattern you'd use on AWS EKS.
+#
+# What it adds over the quickstart:
+#   * Public subnets for the ALB + a NAT gateway, private subnets for the workers.
+#   * The aws-load-balancer-controller addon, installed through the EKS API.
+#   * The cluster tag spinifex.io/managed-ingress = "false", which disables K3s'
+#     built-in traefik/servicelb so the LBC owns ingress (AWS parity).
+#   * A self-signed certificate imported into ACM and attached to the ALB's HTTPS
+#     listener via an Ingress annotation.
+#
+# The workloads/ module then creates a Deployment, a NodePort Service, and an
+# Ingress (ingressClassName: alb). The LBC reconciles that Ingress into an
+# internet-facing ALB that terminates TLS with the ACM cert and forwards to the
+# workers' NodePort. The cluster's ELB-eligible subnets are injected by Spinifex
+# into the alb IngressClassParams, so the Ingress needs no subnet annotations.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-https-ingress
+#   export AWS_PROFILE=spinifex
+#   tofu init && tofu apply
+#   # build + push the demo image to the ECR repo this creates (see README),
+#   # then: cd workloads && tofu init && tofu apply
+#   # finally: kubectl get ingress spinifex-demo -o wide  → open https://<address>
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    tls = {
+      source  = "hashicorp/tls"
+      version = ">= 4.0"
+    }
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Variables
+# ---------------------------------------------------------------------------
+
+variable "region" {
+  type    = string
+  default = "ap-southeast-2"
+}
+
+variable "cluster_name" {
+  type    = string
+  default = "eks-https"
+}
+
+variable "k8s_version" {
+  type    = string
+  default = "1.32"
+}
+
+variable "node_instance_type" {
+  type    = string
+  default = "t3.medium"
+}
+
+variable "node_desired_size" {
+  type        = number
+  default     = 1
+  description = "Worker count. Use 1 for a single-node demo, or 3 for an HA-shaped cluster."
+
+  validation {
+    condition     = var.node_desired_size == 1 || var.node_desired_size == 3
+    error_message = "node_desired_size must be 1 or 3."
+  }
+}
+
+variable "node_port" {
+  type        = number
+  default     = 30080
+  description = "NodePort the demo Service is published on and the ALB forwards to"
+}
+
+variable "cert_common_name" {
+  type    = string
+  default = "eks-https.spinifex.local"
+}
+
+variable "api_public_access_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the public Kubernetes API endpoint; tighten in production"
+}
+
+variable "spinifex_endpoint" {
+  type    = string
+  default = "https://127.0.0.1:9999"
+}
+
+# ---------------------------------------------------------------------------
+# Providers
+# ---------------------------------------------------------------------------
+
+provider "aws" {
+  region = var.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+    ecr = var.spinifex_endpoint
+    acm = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+data "aws_availability_zones" "available" {
+  state = "available"
+}
+
+# ---------------------------------------------------------------------------
+# VPC — public subnets for the ALB + NAT, private subnets for the workers
+# ---------------------------------------------------------------------------
+
+resource "aws_vpc" "main" {
+  cidr_block           = "10.31.0.0/16"
+  enable_dns_hostnames = true
+  enable_dns_support   = true
+
+  tags = {
+    Name = "${var.cluster_name}-vpc"
+  }
+}
+
+resource "aws_internet_gateway" "igw" {
+  vpc_id = aws_vpc.main.id
+
+  tags = {
+    Name = "${var.cluster_name}-igw"
+  }
+}
+
+resource "aws_subnet" "public_a" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.31.1.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-public-a"
+  }
+}
+
+resource "aws_subnet" "public_b" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.31.2.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-public-b"
+  }
+}
+
+resource "aws_subnet" "private_a" {
+  vpc_id            = aws_vpc.main.id
+  cidr_block        = "10.31.11.0/24"
+  availability_zone = data.aws_availability_zones.available.names[0]
+
+  tags = {
+    Name = "${var.cluster_name}-private-a"
+  }
+}
+
+resource "aws_subnet" "private_b" {
+  vpc_id            = aws_vpc.main.id
+  cidr_block        = "10.31.12.0/24"
+  availability_zone = data.aws_availability_zones.available.names[0]
+
+  tags = {
+    Name = "${var.cluster_name}-private-b"
+  }
+}
+
+resource "aws_route_table" "public" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block = "0.0.0.0/0"
+    gateway_id = aws_internet_gateway.igw.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-public-rt"
+  }
+}
+
+resource "aws_route_table_association" "public_a" {
+  subnet_id      = aws_subnet.public_a.id
+  route_table_id = aws_route_table.public.id
+}
+
+resource "aws_route_table_association" "public_b" {
+  subnet_id      = aws_subnet.public_b.id
+  route_table_id = aws_route_table.public.id
+}
+
+# NAT gateway gives the private workers outbound internet to pull the demo image
+# from ECR. The workers still join the cluster over the in-VPC private endpoint.
+resource "aws_eip" "nat" {
+  domain = "vpc"
+
+  tags = {
+    Name = "${var.cluster_name}-nat-eip"
+  }
+}
+
+resource "aws_nat_gateway" "nat" {
+  allocation_id = aws_eip.nat.id
+  subnet_id     = aws_subnet.public_a.id
+
+  depends_on = [aws_internet_gateway.igw]
+
+  tags = {
+    Name = "${var.cluster_name}-nat"
+  }
+}
+
+resource "aws_route_table" "private" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block     = "0.0.0.0/0"
+    nat_gateway_id = aws_nat_gateway.nat.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-private-rt"
+  }
+}
+
+resource "aws_route_table_association" "private_a" {
+  subnet_id      = aws_subnet.private_a.id
+  route_table_id = aws_route_table.private.id
+}
+
+resource "aws_route_table_association" "private_b" {
+  subnet_id      = aws_subnet.private_b.id
+  route_table_id = aws_route_table.private.id
+}
+
+# ---------------------------------------------------------------------------
+# IAM — cluster role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "cluster" {
+  name = "${var.cluster_name}-cluster-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "eks.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "cluster" {
+  role       = aws_iam_role.cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — node role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "node" {
+  name = "${var.cluster_name}-node-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_worker" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_cni" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_ecr" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+}
+
+# The AWS Load Balancer Controller runs with the node's instance-profile
+# credentials (Spinifex wires creds at the node level, not IRSA — see the addon
+# block below), so the permissions it needs to manage ALBs must live on the node
+# role. This mirrors the upstream AWSLoadBalancerControllerIAMPolicy, trimmed to
+# the actions an ALB Ingress exercises (Shield/WAF/Cognito are disabled on the
+# controller). Resources are "*" since Spinifex evaluates grants by action.
+# A customer-managed policy + attachment is used (Spinifex implements
+# CreatePolicy/AttachRolePolicy, not inline PutRolePolicy).
+resource "aws_iam_policy" "node_lbc" {
+  name = "${var.cluster_name}-node-lbc"
+
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Effect = "Allow"
+        Action = [
+          "ec2:DescribeAccountAttributes",
+          "ec2:DescribeAddresses",
+          "ec2:DescribeAvailabilityZones",
+          "ec2:DescribeInternetGateways",
+          "ec2:DescribeVpcs",
+          "ec2:DescribeSubnets",
+          "ec2:DescribeSecurityGroups",
+          "ec2:DescribeInstances",
+          "ec2:DescribeNetworkInterfaces",
+          "ec2:DescribeTags",
+          "ec2:GetCoipPoolUsage",
+          "ec2:DescribeCoipPools",
+          "ec2:CreateSecurityGroup",
+          "ec2:CreateTags",
+          "ec2:DeleteTags",
+          "ec2:AuthorizeSecurityGroupIngress",
+          "ec2:RevokeSecurityGroupIngress",
+          "ec2:DeleteSecurityGroup",
+          "elasticloadbalancing:*",
+          "acm:ListCertificates",
+          "acm:DescribeCertificate",
+          "acm:GetCertificate",
+          "iam:CreateServiceLinkedRole",
+          "iam:ListServerCertificates",
+          "iam:GetServerCertificate",
+          "tag:GetResources",
+          "tag:TagResources",
+          "wafv2:GetWebACLForResource",
+          "shield:GetSubscriptionState"
+        ]
+        Resource = "*"
+      }
+    ]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_lbc" {
+  role       = aws_iam_role.node.name
+  policy_arn = aws_iam_policy.node_lbc.arn
+}
+
+# ---------------------------------------------------------------------------
+# ECR — repository the workers pull the demo image from
+# ---------------------------------------------------------------------------
+
+resource "aws_ecr_repository" "demo" {
+  name = "spinifex-demo"
+
+  # tofu destroy must remove the repo even though it still holds the pushed demo
+  # image; without this, DeleteRepository (force=false) fails RepositoryNotEmpty.
+  force_delete = true
+}
+
+# ---------------------------------------------------------------------------
+# EKS cluster — public + private endpoints, LBC-owned ingress
+#
+# The spinifex.io/managed-ingress = "false" tag disables K3s' built-in
+# traefik/servicelb so the AWS Load Balancer Controller owns ingress, matching
+# how ingress works on AWS EKS.
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_cluster" "this" {
+  name     = var.cluster_name
+  role_arn = aws_iam_role.cluster.arn
+  version  = var.k8s_version
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.private_a.id, aws_subnet.private_b.id]
+    endpoint_public_access  = true
+    endpoint_private_access = true
+    public_access_cidrs     = [var.api_public_access_cidr]
+  }
+
+  access_config {
+    authentication_mode                         = "API"
+    bootstrap_cluster_creator_admin_permissions = true
+  }
+
+  depends_on = [aws_iam_role_policy_attachment.cluster]
+
+  tags = {
+    Name                          = var.cluster_name
+    "spinifex.io/managed-ingress" = "false"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Managed node group — workers in the private subnets
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_node_group" "workers" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "workers"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.private_a.id, aws_subnet.private_b.id]
+
+  scaling_config {
+    desired_size = var.node_desired_size
+    min_size     = var.node_desired_size
+    max_size     = var.node_desired_size * 2
+  }
+
+  instance_types = [var.node_instance_type]
+  ami_type       = "AL2_x86_64"
+
+  depends_on = [
+    aws_iam_role_policy_attachment.node_worker,
+    aws_iam_role_policy_attachment.node_cni,
+    aws_iam_role_policy_attachment.node_ecr,
+  ]
+
+  tags = {
+    Name = "${var.cluster_name}-workers"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Addon — AWS Load Balancer Controller
+#
+# Spinifex wires the controller's AWS credentials and ELB-eligible subnets at the
+# node level, so no IRSA role or subnet tagging is needed here. addon_version is
+# omitted: the AWS provider demands a v-prefixed version that the catalog rejects,
+# so let Spinifex default to its catalog version.
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_addon" "lbc" {
+  cluster_name                = aws_eks_cluster.this.name
+  addon_name                  = "aws-load-balancer-controller"
+  resolve_conflicts_on_create = "OVERWRITE"
+
+  depends_on = [aws_eks_node_group.workers]
+
+  tags = {
+    Name = "${var.cluster_name}-lbc"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# TLS — self-signed certificate imported into ACM
+#
+# Spinifex ACM supports ImportCertificate (not RequestCertificate), so the cert
+# is generated locally by the tls provider and imported. The workloads Ingress
+# attaches it to the ALB's HTTPS listener by ARN.
+# ---------------------------------------------------------------------------
+
+resource "tls_private_key" "ingress" {
+  algorithm = "RSA"
+  rsa_bits  = 2048
+}
+
+resource "tls_self_signed_cert" "ingress" {
+  private_key_pem = tls_private_key.ingress.private_key_pem
+
+  subject {
+    common_name  = var.cert_common_name
+    organization = "Spinifex EKS Demo"
+  }
+
+  dns_names             = [var.cert_common_name]
+  validity_period_hours = 8760
+  early_renewal_hours   = 720
+
+  allowed_uses = [
+    "key_encipherment",
+    "digital_signature",
+    "server_auth",
+  ]
+}
+
+resource "aws_acm_certificate" "ingress" {
+  private_key      = tls_private_key.ingress.private_key_pem
+  certificate_body = tls_self_signed_cert.ingress.cert_pem
+
+  tags = {
+    Name = "${var.cluster_name}-ingress"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Let the ALB reach the workers' NodePort
+#
+# Spinifex auto-manages the nodegroup SG and admits only intra-cluster traffic.
+# The LBC-provisioned ALB lives in this VPC, so admit the VPC CIDR on the
+# NodePort. Look the SG up by its deterministic name and add one rule.
+# ---------------------------------------------------------------------------
+
+data "aws_security_group" "nodegroup" {
+  filter {
+    name   = "group-name"
+    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
+  }
+
+  filter {
+    name   = "vpc-id"
+    values = [aws_vpc.main.id]
+  }
+
+  depends_on = [aws_eks_node_group.workers]
+}
+
+resource "aws_vpc_security_group_ingress_rule" "nodeport_from_vpc" {
+  security_group_id = data.aws_security_group.nodegroup.id
+  cidr_ipv4         = aws_vpc.main.cidr_block
+  from_port         = var.node_port
+  to_port           = var.node_port
+  ip_protocol       = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-alb-nodeport"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Outputs
+# ---------------------------------------------------------------------------
+
+output "cluster_name" {
+  value = aws_eks_cluster.this.name
+}
+
+output "region" {
+  value = var.region
+}
+
+output "node_port" {
+  value = var.node_port
+}
+
+output "node_desired_size" {
+  value = var.node_desired_size
+}
+
+output "certificate_arn" {
+  value = aws_acm_certificate.ingress.arn
+}
+
+output "ecr_repository_url" {
+  value       = aws_ecr_repository.demo.repository_url
+  description = "Push the demo-app image here, then apply the workloads module"
+}
+
+output "ingress_address_hint" {
+  value = "After applying workloads: kubectl get ingress spinifex-demo -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{\"\\n\"}' — then open https://<that-address> (self-signed cert: curl -k)."
+}
+
+output "update_kubeconfig" {
+  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
+}

docs/terraform-workbooks/eks-https-ingress/workloads/main.tf

@@ -0,0 +1,258 @@
+# Demo workload for eks-https-ingress — separate root module / state.
+#
+# Kept apart from the cluster root module on purpose: the kubernetes provider
+# below reads the cluster endpoint from a *live* data source, so it is only ever
+# configured while the cluster exists. Destroy this module before the parent and
+# the provider never falls back to localhost.
+#
+# Creates the Spinifex-themed demo Deployment, a NodePort Service, and an Ingress
+# (ingressClassName: alb). The AWS Load Balancer Controller reconciles the
+# Ingress into an internet-facing ALB that terminates TLS with the imported ACM
+# certificate and forwards to the workers' NodePort.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-https-ingress
+#   tofu init && tofu apply                    # parent: cluster + LBC + ACM
+#   cd workloads && tofu init && tofu apply    # this module: demo app + Ingress
+#   # teardown is the reverse — destroy here first, then the parent.
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.20"
+    }
+  }
+}
+
+variable "spinifex_endpoint" {
+  type    = string
+  default = "https://127.0.0.1:9999"
+}
+
+variable "replicas" {
+  type    = number
+  default = 2
+}
+
+variable "demo_image" {
+  type        = string
+  default     = ""
+  description = "Demo image ref. Defaults to the parent's ECR repository URL at :latest."
+}
+
+variable "inbound_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the ALB HTTPS listener"
+}
+
+provider "aws" {
+  region = data.terraform_remote_state.infra.outputs.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+# Cluster identity comes from the parent module's state; the live endpoint/CA
+# come from a data source so the provider is only configured while the cluster
+# is up.
+data "terraform_remote_state" "infra" {
+  backend = "local"
+
+  config = {
+    path = "../terraform.tfstate"
+  }
+}
+
+locals {
+  cluster_name = data.terraform_remote_state.infra.outputs.cluster_name
+  region       = data.terraform_remote_state.infra.outputs.region
+  node_port    = data.terraform_remote_state.infra.outputs.node_port
+  cert_arn     = data.terraform_remote_state.infra.outputs.certificate_arn
+  demo_image   = var.demo_image != "" ? var.demo_image : "${data.terraform_remote_state.infra.outputs.ecr_repository_url}:latest"
+}
+
+data "aws_eks_cluster" "this" {
+  name = local.cluster_name
+}
+
+provider "kubernetes" {
+  host                   = data.aws_eks_cluster.this.endpoint
+  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
+
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    command     = "aws"
+    args        = ["eks", "get-token", "--cluster-name", local.cluster_name, "--region", local.region]
+  }
+}
+
+# The Spinifex-themed demo app reports the pod, node, cluster, and region that
+# served the request. The pod and node names come from the downward API.
+resource "kubernetes_deployment_v1" "demo" {
+  metadata {
+    name      = "spinifex-demo"
+    namespace = "default"
+    labels    = { app = "spinifex-demo" }
+  }
+
+  spec {
+    replicas = var.replicas
+
+    selector {
+      match_labels = { app = "spinifex-demo" }
+    }
+
+    template {
+      metadata {
+        labels = { app = "spinifex-demo" }
+      }
+
+      spec {
+        container {
+          name  = "spinifex-demo"
+          image = local.demo_image
+
+          port {
+            container_port = 8080
+          }
+
+          env {
+            name = "POD_NAME"
+            value_from {
+              field_ref {
+                field_path = "metadata.name"
+              }
+            }
+          }
+
+          env {
+            name = "NODE_NAME"
+            value_from {
+              field_ref {
+                field_path = "spec.nodeName"
+              }
+            }
+          }
+
+          env {
+            name = "POD_NAMESPACE"
+            value_from {
+              field_ref {
+                field_path = "metadata.namespace"
+              }
+            }
+          }
+
+          env {
+            name  = "CLUSTER_NAME"
+            value = local.cluster_name
+          }
+
+          env {
+            name  = "AWS_REGION"
+            value = local.region
+          }
+
+          env {
+            name  = "APP_TITLE"
+            value = "Spinifex EKS — HTTPS Ingress"
+          }
+
+          readiness_probe {
+            http_get {
+              path = "/healthz"
+              port = 8080
+            }
+            initial_delay_seconds = 3
+            period_seconds        = 10
+          }
+        }
+      }
+    }
+  }
+}
+
+# NodePort Service: the ALB target group registers the workers on this port.
+resource "kubernetes_service_v1" "demo" {
+  metadata {
+    name      = "spinifex-demo"
+    namespace = "default"
+  }
+
+  spec {
+    selector = { app = "spinifex-demo" }
+    type     = "NodePort"
+
+    port {
+      port        = 80
+      target_port = 8080
+      node_port   = local.node_port
+    }
+  }
+
+  depends_on = [kubernetes_deployment_v1.demo]
+}
+
+# Ingress reconciled by the AWS Load Balancer Controller into an internet-facing
+# ALB. target-type instance registers the workers' NodePort; the ACM cert ARN
+# attaches to the HTTPS:443 listener; HTTP is redirected to HTTPS. ELB subnets
+# are injected by Spinifex into the alb IngressClassParams, so none are set here.
+resource "kubernetes_ingress_v1" "demo" {
+  metadata {
+    name      = "spinifex-demo"
+    namespace = "default"
+
+    annotations = {
+      "alb.ingress.kubernetes.io/scheme"           = "internet-facing"
+      "alb.ingress.kubernetes.io/target-type"      = "instance"
+      "alb.ingress.kubernetes.io/listen-ports"     = "[{\"HTTP\":80},{\"HTTPS\":443}]"
+      "alb.ingress.kubernetes.io/certificate-arn"  = local.cert_arn
+      "alb.ingress.kubernetes.io/ssl-redirect"     = "443"
+      "alb.ingress.kubernetes.io/healthcheck-path" = "/healthz"
+      "alb.ingress.kubernetes.io/inbound-cidrs"    = var.inbound_cidr
+    }
+  }
+
+  spec {
+    ingress_class_name = "alb"
+
+    rule {
+      http {
+        path {
+          path      = "/"
+          path_type = "Prefix"
+
+          backend {
+            service {
+              name = kubernetes_service_v1.demo.metadata[0].name
+              port {
+                number = 80
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+}
+
+output "ingress_name" {
+  value = kubernetes_ingress_v1.demo.metadata[0].name
+}

docs/terraform-workbooks/eks-quickstart/README.md

@@ -0,0 +1,214 @@
+---
+title: "EKS Quickstart"
+description: "Stand up a minimal managed-Kubernetes cluster and a browsable Spinifex-themed demo app — VPC, IAM roles, an EKS cluster, a one- or three-worker node group, an ECR repository, and a load-balanced web page — using Terraform on Spinifex."
+category: "Terraform Workbooks"
+tags:
+  - terraform
+  - eks
+  - kubernetes
+  - iam
+  - vpc
+  - workbook
+resources:
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "Terraform Kubernetes Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "OpenTofu"
+    url: "https://opentofu.org/"
+---
+
+# Terraform: EKS Quickstart
+
+> The smallest end-to-end EKS example on Spinifex: a VPC, the two IAM roles EKS needs, a one- or three-worker cluster, an ECR repository, and a Spinifex-themed demo app you can open in a browser.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+Provision a managed Kubernetes cluster on Spinifex with Terraform/OpenTofu — and finish with something you can actually see. As well as the cluster, this workbook creates an **ECR repository**, then deploys the Spinifex-themed demo app onto the cluster and publishes it on the worker's public IP. When `apply` finishes, open the **`demo_url`** output: the page reports the **pod, node, cluster, and region** that answered. Refresh it and the answering pod changes, so even a non-technical viewer can watch Kubernetes scheduling and load-balancing in real time.
+
+Under the hood it keeps things minimal: a VPC with two public subnets, an IAM role for the control plane, an IAM role for the workers with the three managed policies AWS-managed node groups expect, a public-endpoint `aws_eks_cluster`, and a node group sized by **`node_desired_size`** (1 for a single-node demo, or 3 for an HA-shaped cluster). You build the demo image and push it to the ECR repository, then Terraform uses the **Kubernetes provider** (authenticating exactly like `kubectl` does, via `aws eks get-token`) to deploy the Deployment + NodePort Service.
+
+**What you'll learn:**
+
+- Configuring the AWS provider to target Spinifex's `eks`, `ec2`, `iam` and `sts` endpoints
+- Creating the EKS cluster and worker IAM roles with faithful managed-policy attachments
+- Provisioning an `aws_eks_cluster` and a managed `aws_eks_node_group`
+- Pointing the Kubernetes provider at the new cluster and deploying a workload from the same `apply`
+- Opening one rule on the auto-managed worker security group to expose a NodePort
+
+**What gets created**
+
+| Resource | Name | Purpose |
+|---|---|---|
+| VPC | `eks-quickstart-vpc` | Isolated network (10.30.0.0/16) |
+| Subnets | `eks-quickstart-subnet-a/-b` | Public subnets for the cluster and worker |
+| Internet Gateway | `eks-quickstart-igw` | Egress so the worker can pull the demo image |
+| IAM Roles | `eks-quickstart-cluster-role`, `eks-quickstart-node-role` | Control-plane and worker roles |
+| ECR Repository | `spinifex-demo` | Holds the demo image the workers pull |
+| EKS Cluster | `eks-quickstart` | Public API endpoint, API auth mode, Kubernetes 1.32 |
+| Node Group | `default` | `node_desired_size` `t3.medium` worker(s) — 1 or 3 |
+| SG Ingress Rule | `eks-quickstart-demo-nodeport` | Opens the demo NodePort on the auto-managed worker SG |
+| K8s Deployment | `spinifex-demo` | The themed demo image from ECR, 2 replicas |
+| K8s Service | `spinifex-demo` | NodePort publishing the demo on the worker |
+
+**Spinifex specifics**
+
+- The cluster's security groups are **auto-managed** — `vpc_config.security_group_ids` is ignored. To reach a NodePort, this workbook looks the worker SG up by its deterministic name (`eks-cluster-<name>-nodegroup-sg`) and adds a single ingress rule.
+- The worker AMI is always Spinifex's `eks-node` image; `ami_type` is recorded but does **not** select the image.
+- `authentication_mode` must be `"API"` (the `API_AND_CONFIG_MAP` mode is rejected).
+- The workers need outbound internet (here via the IGW) to pull the demo container image.
+
+**Prerequisites:**
+
+- Spinifex installed and running (see [Installing Spinifex](/docs/install))
+- The Spinifex `eks-node` image available on the cluster
+- OpenTofu or Terraform, plus `kubectl` and the AWS CLI
+- Docker, to build and push the demo image to ECR
+
+## Instructions
+
+### Step 1. Get the Template
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-quickstart
+```
+
+Or create a `main.tf` file and paste the full configuration below.
+
+<!-- INCLUDE: main.tf lang:hcl -->
+
+### Step 2. Deploy
+
+The cluster and the demo app are **two root modules with separate state**. Apply the cluster first:
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+```
+
+This creates the cluster (which bootstraps a control-plane VM and brings up k3s — a few minutes in `CREATING`), launches the worker(s), creates the `spinifex-demo` ECR repository, and opens the NodePort.
+
+Set the worker count with `node_desired_size` (`1` for a single node, `3` for an HA-shaped cluster):
+
+```bash
+tofu apply -var node_desired_size=3
+```
+
+Once the cluster is `ACTIVE`, **build and push the demo image** to the ECR repository this created (full commands in [`../demo-app/README.md`](../demo-app/README.md)):
+
+```bash
+cd ../demo-app
+REGISTRY=$(cd ../eks-quickstart && tofu output -raw ecr_repository_url)
+REGISTRY_HOST=${REGISTRY%%/*}
+aws ecr get-login-password | docker login --username AWS --password-stdin "$REGISTRY_HOST"
+docker build -t "${REGISTRY}:latest" .
+docker push "${REGISTRY}:latest"
+cd ../eks-quickstart
+```
+
+Then deploy the demo app from the nested `workloads/` module — it defaults `demo_image` to the parent's ECR repository at `:latest`:
+
+```bash
+cd workloads
+tofu init
+tofu apply
+```
+
+> **Why two modules?** The Kubernetes provider in `workloads/` reads the cluster endpoint from a live `data "aws_eks_cluster"` source, so it's only ever configured while the cluster exists. Keeping it out of the cluster module means `destroy` never tries to refresh a workload against a cluster that's already gone — the failure mode where the provider falls back to `http://localhost:80` and reports `connection refused`. Always destroy `workloads/` before the cluster.
+
+> **Same profile for the Kubernetes provider.** The Kubernetes provider authenticates by shelling out to `aws eks get-token`, which has to reach the Spinifex STS endpoint. Keep `AWS_PROFILE=spinifex` exported for both applies.
+
+<!-- INCLUDE: workloads/main.tf lang:hcl -->
+
+### Step 3. Open the Demo
+
+Run from the cluster module directory (`cd ..` if you're still in `workloads/`):
+
+```bash
+tofu output demo_url
+```
+
+Open that URL in a browser. You'll see the Spinifex-themed page reporting the **pod, node, cluster, and region** that handled the request. Refresh a few times — with two replicas, the pod name alternates, demonstrating that the Service is load-balancing across the cluster.
+
+### Step 4. Inspect with kubectl (optional)
+
+```bash
+aws eks update-kubeconfig --name eks-quickstart --region ap-southeast-2
+kubectl get nodes
+kubectl get pods -o wide
+```
+
+`kubectl get pods -o wide` shows the demo pods and which node each landed on.
+
+### Cleanup
+
+Destroy in reverse — the demo app first (while the cluster is still up), then the cluster:
+
+```bash
+cd workloads
+tofu destroy
+cd ..
+tofu destroy
+```
+
+## Troubleshooting
+
+### Demo URL Doesn't Load
+
+The page is served from a NodePort on the worker's public IP. Work through the chain:
+
+```bash
+# Is the worker running with a public IP?
+aws ec2 describe-instances --filters "Name=tag:spinifex:eks-cluster,Values=eks-quickstart" \
+  --query 'Reservations[].Instances[].[InstanceId,State.Name,PublicIpAddress]' --output text
+
+# Did the demo pods roll out?
+kubectl get pods -o wide
+
+# Is the NodePort rule on the worker SG?
+aws ec2 describe-security-groups --filters "Name=group-name,Values=eks-cluster-eks-quickstart-nodegroup-sg" \
+  --query 'SecurityGroups[0].IpPermissions'
+```
+
+If the pods are `Pending`, the worker may not be `Ready` yet — give it a moment. If they're `ImagePullBackOff`, the worker can't pull from ECR: confirm you built and pushed the image (`tofu output ecr_repository_url`), that the IGW route and the worker's public IP are in place, and that the node role carries `AmazonEC2ContainerRegistryReadOnly`.
+
+### kubernetes provider: connection refused / Unauthorized
+
+The provider runs `aws eks get-token` against the Spinifex STS endpoint. Confirm the AWS CLI is still pointed at Spinifex and the cluster is `ACTIVE`:
+
+```bash
+aws sts get-caller-identity
+aws eks describe-cluster --name eks-quickstart --query 'cluster.status'
+```
+
+If the cluster was still `CREATING` when the provider first tried to connect, just re-run `tofu apply`.
+
+### Cluster Stuck in CREATING
+
+Control-plane bootstrap takes a few minutes. Confirm the underlying VM came up:
+
+```bash
+aws eks describe-cluster --name eks-quickstart --query 'cluster.status'
+aws ec2 describe-instances --profile spinifex
+```
+
+### Provider Connection Refused
+
+```bash
+sudo systemctl status spinifex.target
+curl -k https://localhost:9999/
+```

docs/terraform-workbooks/eks-quickstart/main.tf

@@ -0,0 +1,392 @@
+# Example: EKS Quickstart on Spinifex
+#
+# A minimal managed-Kubernetes cluster that ends in something you can see: a
+# VPC, the two IAM roles EKS needs, a one- or three-worker cluster, an ECR
+# repository, and the Spinifex-themed demo app deployed onto it by Terraform and
+# exposed on the worker's public IP.
+#
+# Once `apply` finishes, open the demo_url output in a browser. The page reports
+# the pod, node, cluster, and region that answered — refresh and it bounces
+# between the app's replicas, showing the cluster is really scheduling and
+# load-balancing pods.
+#
+# Demonstrates: VPC + subnets, EKS cluster + node IAM roles, an aws_eks_cluster
+# and managed aws_eks_node_group, an ECR repository the workers pull from, the
+# Kubernetes provider authenticating to the cluster, and a Deployment + NodePort
+# Service reachable from your browser.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-quickstart
+#   export AWS_PROFILE=spinifex
+#   tofu init && tofu apply
+#   # build + push the demo image to the ECR repo this creates (see README),
+#   # then: cd workloads && tofu init && tofu apply
+#   # finally open the demo_url output
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Variables
+# ---------------------------------------------------------------------------
+
+variable "region" {
+  type    = string
+  default = "ap-southeast-2"
+}
+
+variable "cluster_name" {
+  type    = string
+  default = "eks-quickstart"
+}
+
+variable "k8s_version" {
+  type        = string
+  default     = "1.32"
+  description = "Kubernetes minor version for the control plane and workers"
+}
+
+variable "instance_type" {
+  type    = string
+  default = "t3.medium"
+}
+
+variable "node_desired_size" {
+  type        = number
+  default     = 1
+  description = "Worker count. Use 1 for a single-node demo, or 3 for an HA-shaped cluster."
+
+  validation {
+    condition     = var.node_desired_size == 1 || var.node_desired_size == 3
+    error_message = "node_desired_size must be 1 or 3."
+  }
+}
+
+variable "node_port" {
+  type        = number
+  default     = 30080
+  description = "NodePort the demo Service is published on"
+}
+
+variable "browse_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the demo NodePort; tighten to your own IP in production"
+}
+
+variable "spinifex_endpoint" {
+  type        = string
+  default     = "https://127.0.0.1:9999"
+  description = "Spinifex AWS gateway endpoint"
+}
+
+# ---------------------------------------------------------------------------
+# Providers
+# ---------------------------------------------------------------------------
+
+provider "aws" {
+  region = var.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+    ecr = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+# ---------------------------------------------------------------------------
+# Data sources
+# ---------------------------------------------------------------------------
+
+data "aws_availability_zones" "available" {
+  state = "available"
+}
+
+# ---------------------------------------------------------------------------
+# VPC + two public subnets (workers get public IPs to pull the demo image)
+# ---------------------------------------------------------------------------
+
+resource "aws_vpc" "main" {
+  cidr_block           = "10.30.0.0/16"
+  enable_dns_hostnames = true
+  enable_dns_support   = true
+
+  tags = {
+    Name = "${var.cluster_name}-vpc"
+  }
+}
+
+resource "aws_internet_gateway" "igw" {
+  vpc_id = aws_vpc.main.id
+
+  tags = {
+    Name = "${var.cluster_name}-igw"
+  }
+}
+
+resource "aws_subnet" "a" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.30.1.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-subnet-a"
+  }
+}
+
+resource "aws_subnet" "b" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.30.2.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-subnet-b"
+  }
+}
+
+resource "aws_route_table" "public" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block = "0.0.0.0/0"
+    gateway_id = aws_internet_gateway.igw.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-public-rt"
+  }
+}
+
+resource "aws_route_table_association" "a" {
+  subnet_id      = aws_subnet.a.id
+  route_table_id = aws_route_table.public.id
+}
+
+resource "aws_route_table_association" "b" {
+  subnet_id      = aws_subnet.b.id
+  route_table_id = aws_route_table.public.id
+}
+
+# ---------------------------------------------------------------------------
+# IAM — EKS cluster role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "cluster" {
+  name = "${var.cluster_name}-cluster-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "eks.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "cluster" {
+  role       = aws_iam_role.cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — worker node role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "node" {
+  name = "${var.cluster_name}-node-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_worker" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_cni" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_ecr" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+}
+
+# ---------------------------------------------------------------------------
+# ECR — repository the workers pull the demo image from
+#
+# Build and push the demo-app image here before applying the workloads module
+# (see ../demo-app/README.md). The node role already carries
+# AmazonEC2ContainerRegistryReadOnly, so workers can pull from it.
+# ---------------------------------------------------------------------------
+
+resource "aws_ecr_repository" "demo" {
+  name = "spinifex-demo"
+
+  # tofu destroy must remove the repo even though it still holds the pushed demo
+  # image; without this, DeleteRepository (force=false) fails RepositoryNotEmpty.
+  force_delete = true
+}
+
+# ---------------------------------------------------------------------------
+# EKS cluster — public API endpoint, API (access-entry) auth mode
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_cluster" "this" {
+  name     = var.cluster_name
+  role_arn = aws_iam_role.cluster.arn
+  version  = var.k8s_version
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.a.id, aws_subnet.b.id]
+    endpoint_public_access  = true
+    endpoint_private_access = false
+  }
+
+  access_config {
+    authentication_mode                         = "API"
+    bootstrap_cluster_creator_admin_permissions = true
+  }
+
+  depends_on = [aws_iam_role_policy_attachment.cluster]
+
+  tags = {
+    Name = var.cluster_name
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Managed node group — one worker
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_node_group" "default" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "default"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.a.id, aws_subnet.b.id]
+
+  scaling_config {
+    desired_size = var.node_desired_size
+    min_size     = var.node_desired_size
+    max_size     = var.node_desired_size * 2
+  }
+
+  instance_types = [var.instance_type]
+  ami_type       = "AL2_x86_64"
+
+  depends_on = [
+    aws_iam_role_policy_attachment.node_worker,
+    aws_iam_role_policy_attachment.node_cni,
+    aws_iam_role_policy_attachment.node_ecr,
+  ]
+
+  tags = {
+    Name = "${var.cluster_name}-default"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Open the NodePort on the auto-managed nodegroup SG
+#
+# Spinifex creates the worker SG itself (vpc_config.security_group_ids is
+# ignored) and admits only intra-cluster traffic. To reach the demo NodePort
+# from a browser, look the SG up by its deterministic name and add one rule.
+# ---------------------------------------------------------------------------
+
+data "aws_security_group" "nodegroup" {
+  filter {
+    name   = "group-name"
+    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
+  }
+
+  filter {
+    name   = "vpc-id"
+    values = [aws_vpc.main.id]
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+resource "aws_vpc_security_group_ingress_rule" "nodeport" {
+  security_group_id = data.aws_security_group.nodegroup.id
+  cidr_ipv4         = var.browse_cidr
+  from_port         = var.node_port
+  to_port           = var.node_port
+  ip_protocol       = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-demo-nodeport"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Discover the worker's public IP for the demo URL
+# ---------------------------------------------------------------------------
+
+data "aws_instances" "workers" {
+  instance_tags = {
+    "spinifex:eks-cluster" = aws_eks_cluster.this.name
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+# ---------------------------------------------------------------------------
+# Outputs
+# ---------------------------------------------------------------------------
+
+output "cluster_name" {
+  value = aws_eks_cluster.this.name
+}
+
+output "region" {
+  value = var.region
+}
+
+output "node_port" {
+  value = var.node_port
+}
+
+output "node_desired_size" {
+  value = var.node_desired_size
+}
+
+output "ecr_repository_url" {
+  value       = aws_ecr_repository.demo.repository_url
+  description = "Push the demo-app image here, then apply the workloads module"
+}
+
+output "demo_url" {
+  value       = "http://${data.aws_instances.workers.public_ips[0]}:${var.node_port}"
+  description = "Open in a browser; refresh to see different pods answer"
+}
+
+output "update_kubeconfig" {
+  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
+}

docs/terraform-workbooks/eks-quickstart/workloads/main.tf

@@ -0,0 +1,198 @@
+# Demo workload for eks-quickstart — separate root module / state.
+#
+# Kept apart from the cluster root module on purpose: the kubernetes provider
+# below reads the cluster endpoint from a *live* data source, so it is only ever
+# configured while the cluster exists. Destroy this module before the parent and
+# the provider never falls back to localhost.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-quickstart
+#   tofu init && tofu apply                    # parent: cluster + infra
+#   cd workloads && tofu init && tofu apply    # this module: demo app
+#   # teardown is the reverse — destroy here first, then the parent.
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.20"
+    }
+  }
+}
+
+variable "spinifex_endpoint" {
+  type        = string
+  default     = "https://127.0.0.1:9999"
+  description = "Spinifex AWS gateway endpoint"
+}
+
+variable "replicas" {
+  type        = number
+  default     = 2
+  description = "Demo app replicas; refresh the page to see requests land on different pods"
+}
+
+variable "demo_image" {
+  type        = string
+  default     = ""
+  description = "Demo image ref. Defaults to the parent's ECR repository URL at :latest."
+}
+
+provider "aws" {
+  region = data.terraform_remote_state.infra.outputs.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+# Cluster identity comes from the parent module's state; the live endpoint/CA
+# come from a data source so the provider is only configured while the cluster
+# is up.
+data "terraform_remote_state" "infra" {
+  backend = "local"
+
+  config = {
+    path = "../terraform.tfstate"
+  }
+}
+
+locals {
+  cluster_name = data.terraform_remote_state.infra.outputs.cluster_name
+  region       = data.terraform_remote_state.infra.outputs.region
+  node_port    = data.terraform_remote_state.infra.outputs.node_port
+  demo_image   = var.demo_image != "" ? var.demo_image : "${data.terraform_remote_state.infra.outputs.ecr_repository_url}:latest"
+}
+
+data "aws_eks_cluster" "this" {
+  name = local.cluster_name
+}
+
+# Authenticates with the same `aws eks get-token` exec flow the generated
+# kubeconfig uses, so the Kubernetes provider can deploy the demo app.
+provider "kubernetes" {
+  host                   = data.aws_eks_cluster.this.endpoint
+  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
+
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    command     = "aws"
+    args        = ["eks", "get-token", "--cluster-name", local.cluster_name, "--region", local.region]
+  }
+}
+
+# The Spinifex-themed demo app reports the pod, node, cluster, and region that
+# served the request. With multiple replicas, refreshing the demo_url alternates
+# between them. The pod and node names come from the downward API.
+resource "kubernetes_deployment_v1" "demo" {
+  metadata {
+    name      = "spinifex-demo"
+    namespace = "default"
+    labels    = { app = "spinifex-demo" }
+  }
+
+  spec {
+    replicas = var.replicas
+
+    selector {
+      match_labels = { app = "spinifex-demo" }
+    }
+
+    template {
+      metadata {
+        labels = { app = "spinifex-demo" }
+      }
+
+      spec {
+        container {
+          name  = "spinifex-demo"
+          image = local.demo_image
+
+          port {
+            container_port = 8080
+          }
+
+          env {
+            name = "POD_NAME"
+            value_from {
+              field_ref {
+                field_path = "metadata.name"
+              }
+            }
+          }
+
+          env {
+            name = "NODE_NAME"
+            value_from {
+              field_ref {
+                field_path = "spec.nodeName"
+              }
+            }
+          }
+
+          env {
+            name = "POD_NAMESPACE"
+            value_from {
+              field_ref {
+                field_path = "metadata.namespace"
+              }
+            }
+          }
+
+          env {
+            name  = "CLUSTER_NAME"
+            value = local.cluster_name
+          }
+
+          env {
+            name  = "AWS_REGION"
+            value = local.region
+          }
+
+          readiness_probe {
+            http_get {
+              path = "/healthz"
+              port = 8080
+            }
+            initial_delay_seconds = 3
+            period_seconds        = 10
+          }
+        }
+      }
+    }
+  }
+}
+
+resource "kubernetes_service_v1" "demo" {
+  metadata {
+    name      = "spinifex-demo"
+    namespace = "default"
+  }
+
+  spec {
+    selector = { app = "spinifex-demo" }
+    type     = "NodePort"
+
+    port {
+      port        = 80
+      target_port = 8080
+      node_port   = local.node_port
+    }
+  }
+
+  depends_on = [kubernetes_deployment_v1.demo]
+}

docs/terraform-workbooks/s3-webapp/README.md

@@ -9,6 +9,9 @@ tags:
   - flask
   - webapp
   - workbook
+  - imds
+  - sts
+  - instance-profile
 resources:
   - title: "Terraform AWS Provider"
     url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
@@ -36,18 +39,21 @@ resources:
 
 Deploy an EC2 instance running a Flask file-sharing web application backed by S3 (Predastore). Users can upload files through a web form and browse uploaded content — demonstrating Terraform managing both compute and object storage together.
 
+This workbook uses the idiomatic AWS credential model: the instance is launched with an **IAM instance profile** and the app pulls short-lived STS credentials from **IMDS** (`169.254.169.254`) through boto3's default credential chain. **No long-lived S3 keys are baked into the instance.** The Terraform run still uses the operator's admin credentials (to create the bucket, role, profile, and instance), but the running instance authenticates to S3 with credentials it fetches at runtime and which expire in ~1 hour.
+
 **Architecture:**
 
 <p align="center">
-  <img src="../../../.github/assets/diagrams/tf-s3-webapp.svg" alt="S3 webapp — browser to Flask EC2 instance to Predastore via S3 API" width="900">
+  <img src="../../../.github/assets/diagrams/tf-s3-webapp.svg" alt="S3 webapp — browser to Flask EC2 instance; the instance fetches short-lived STS credentials from IMDS (169.254.169.254) via its instance profile, then calls Predastore over the S3 API" width="900">
 </p>
 
 **What you'll learn:**
 
 - Configuring the AWS provider with both Spinifex and Predastore endpoints
 - Creating S3 buckets on Predastore via Terraform
-- Deploying a Python webapp with cloud-init that talks to S3
-- Passing credentials and configuration to instances via user-data
+- Defining an IAM role, least-privilege managed policy, and instance profile, and passing the role to an instance (`iam:PassRole`)
+- How the instance obtains short-lived credentials from IMDS via boto3's default credential chain — no static keys on the instance
+- Deploying a Python webapp with cloud-init that signs S3 requests with the STS session token
 
 **Prerequisites:**
 
@@ -55,6 +61,7 @@ Deploy an EC2 instance running a Flask file-sharing web application backed by S3
 - Predastore running (S3 API on port 8443)
 - An Ubuntu 26.04 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
 - OpenTofu or Terraform installed
+- The operator identity running the Terraform apply (`AWS_PROFILE=spinifex`) must be allowed to manage IAM and pass the role: `iam:CreateRole`, `iam:CreatePolicy`, `iam:AttachRolePolicy`, `iam:CreateInstanceProfile`, `iam:AddRoleToInstanceProfile`, and `iam:PassRole` on the new role. The bootstrap admin profile satisfies this.
 - The EC2 instance must be able to reach Predastore — use the host's br-wan IP, not localhost
 
 ## Instructions
@@ -141,16 +148,39 @@ sudo systemctl status s3-webapp
 sudo journalctl -u s3-webapp --no-pager -n 50
 ```
 
-### Upload Fails or Files Not Appearing
+### Upload Fails with `403 AccessDenied`
 
-Check that the S3 credentials in `/opt/webapp/.env` are correct and that the bucket exists:
+The instance reached S3 but was not authorized. The `.env` file deliberately holds **no** keys — credentials come from IMDS:
 
 ```bash
 ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
-cat /opt/webapp/.env
-/opt/webapp/venv/bin/python -c "import boto3; print('boto3 OK')"
+cat /opt/webapp/.env   # S3_ENDPOINT / S3_BUCKET / S3_REGION only — no keys
 ```
 
+Check that:
+
+- the managed policy is attached to the role (`s3:ListBucket` on the bucket, `s3:GetObject`/`s3:PutObject` on its objects), and the policy's bucket name matches `bucket_name`;
+- the assumed-role session's account matches the bucket owner's account — Predastore enforces bucket ownership. Both are created by the same operator identity, so they align by construction; a mismatch surfaces as `403`.
+
+### `NoCredentialsError` / No Credentials
+
+boto3 could not obtain credentials, which means the IMDS credential chain did not resolve. From the instance:
+
+```bash
+ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
+
+# IMDSv2: fetch a token, then the role name behind the instance profile
+TOKEN=$(curl -sX PUT http://169.254.169.254/latest/api/token \
+  -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')
+curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
+  http://169.254.169.254/latest/meta-data/iam/security-credentials/
+
+# Confirm the resolved identity — expect assumed-role/s3-webapp-role/<instance-id>
+aws sts get-caller-identity --endpoint-url https://<predastore_host>
+```
+
+If the role name is empty, the instance profile was not attached — confirm `iam_instance_profile` on the instance and that `iam:PassRole` is allowed for the operator.
+
 ### AMI Not Found
 
 Ensure you have imported an Ubuntu 26.04 image:

docs/terraform-workbooks/s3-webapp/main.tf

@@ -8,13 +8,25 @@
 # Architecture:
 #
 #   Browser ──HTTP──▶ EC2 Instance (Flask webapp, port 80)
-#                         │
-#                         ▼ S3 API (boto3)
+#                         │  ▲ IMDS: short-lived STS creds (ASIA + token)
+#                         │  └──── 169.254.169.254
+#                         ▼ S3 API (boto3, SigV4 with session token)
 #                     Predastore (port 8443)
 #
+# Credentials:
+#   The Terraform run uses the operator's admin keys (s3_access_key /
+#   s3_secret_key) to create the bucket, IAM role, instance profile, and
+#   instance. The INSTANCE receives no long-lived secret — boto3's default
+#   credential chain pulls short-lived STS credentials from IMDS
+#   (169.254.169.254), resolved via the attached instance profile -> role.
+#
 # Prerequisites:
 #   - Spinifex services running (gateway on port 9999)
 #   - Predastore running (S3 API on port 8443)
+#   - The operator identity (AWS_PROFILE=spinifex) must be allowed
+#     iam:CreateRole / iam:CreatePolicy / iam:AttachRolePolicy /
+#     iam:CreateInstanceProfile / iam:AddRoleToInstanceProfile and
+#     iam:PassRole on the new role.
 #   - The EC2 instance must be able to reach the Predastore endpoint.
 #     Set `predastore_host` to the IP reachable from inside the VPC
 #     (e.g. the host's br-wan IP, NOT localhost).
@@ -80,13 +92,13 @@ variable "predastore_host" {
 
 variable "s3_access_key" {
   type        = string
-  description = "S3 access key for Predastore"
+  description = "Operator S3 access key for the Terraform run (creates the bucket on Predastore); the instance uses IMDS, not this key"
 }
 
 variable "s3_secret_key" {
   type        = string
   sensitive   = true
-  description = "S3 secret key for Predastore"
+  description = "Operator S3 secret key for the Terraform run; the instance uses IMDS, not this key"
 }
 
 variable "bucket_name" {
@@ -154,6 +166,55 @@ resource "aws_s3_bucket" "uploads" {
   bucket = var.bucket_name
 }
 
+# ---------------------------------------------------------------------------
+# IAM Instance Role — least-privilege S3 access fetched at runtime via IMDS
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "webapp" {
+  name = "s3-webapp-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Principal = { Service = "ec2.amazonaws.com" }
+      Action    = "sts:AssumeRole"
+    }]
+  })
+}
+
+resource "aws_iam_policy" "webapp_s3" {
+  name = "s3-webapp-policy"
+
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Sid      = "ListBucket"
+        Effect   = "Allow"
+        Action   = ["s3:ListBucket"]
+        Resource = ["arn:aws:s3:::${var.bucket_name}"]
+      },
+      {
+        Sid      = "ObjectRW"
+        Effect   = "Allow"
+        Action   = ["s3:GetObject", "s3:PutObject"]
+        Resource = ["arn:aws:s3:::${var.bucket_name}/*"]
+      }
+    ]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "webapp" {
+  role       = aws_iam_role.webapp.name
+  policy_arn = aws_iam_policy.webapp_s3.arn
+}
+
+resource "aws_iam_instance_profile" "webapp" {
+  name = "s3-webapp-profile"
+  role = aws_iam_role.webapp.name
+}
+
 # ---------------------------------------------------------------------------
 # SSH Key Pair
 # ---------------------------------------------------------------------------
@@ -273,9 +334,14 @@ resource "aws_instance" "webapp" {
   subnet_id              = aws_subnet.public.id
   vpc_security_group_ids = [aws_security_group.webapp.id]
   key_name               = aws_key_pair.webapp.key_name
+  iam_instance_profile   = aws_iam_instance_profile.webapp.name
 
   associate_public_ip_address = true
 
+  # The role's permissions must exist before the instance boots and makes its
+  # first IMDS-credentialed S3 call
+  depends_on = [aws_iam_role_policy_attachment.webapp]
+
   user_data_base64 = base64encode(<<-USERDATA
     #!/bin/bash
     set -euo pipefail
@@ -293,8 +359,6 @@ resource "aws_instance" "webapp" {
     cat > /opt/webapp/.env <<'ENVFILE'
     S3_ENDPOINT=https://${var.predastore_host}
     S3_BUCKET=${var.bucket_name}
-    S3_ACCESS_KEY=${var.s3_access_key}
-    S3_SECRET_KEY=${var.s3_secret_key}
     S3_REGION=${var.region}
     ENVFILE
 
@@ -321,8 +385,6 @@ resource "aws_instance" "webapp" {
     s3 = boto3.client(
         "s3",
         endpoint_url=env["S3_ENDPOINT"],
-        aws_access_key_id=env["S3_ACCESS_KEY"],
-        aws_secret_access_key=env["S3_SECRET_KEY"],
         region_name=env["S3_REGION"],
         verify=False,
         config=Config(s3={"addressing_style": "path"}),

docs/terraform-workbooks/s3-webapp/terraform.tfvars.example

@@ -2,6 +2,11 @@
 #
 # The predastore_host must be reachable from INSIDE the VPC guest — not
 # localhost. Use the host's br-wan or LAN IP, e.g. "192.168.1.10:8443".
+#
+# s3_access_key / s3_secret_key are the operator credentials for the Terraform
+# run only — they create the bucket on Predastore and the IAM role/profile.
+# They are NOT passed to the instance; the instance pulls short-lived STS
+# credentials from IMDS via its instance profile.
 
 predastore_host = "192.168.1.10:8443"
 s3_access_key   = "AKIAIOSFODNN7EXAMPLE"

go.mod

@@ -13,21 +13,21 @@ require (
 	github.com/insomniacslk/dhcp v0.0.0-20260407060928-11b94ed970f2
 	github.com/kdomanski/iso9660 v0.4.0
 	github.com/klauspost/cpuid/v2 v2.3.0
-	github.com/mulgadc/predastore v1.7.0
-	github.com/mulgadc/viperblock v1.7.0
+	github.com/mulgadc/predastore v1.9.0
+	github.com/mulgadc/viperblock v1.9.0
 	github.com/nats-io/nats-server/v2 v2.14.2
 	github.com/nats-io/nats.go v1.52.0
 	github.com/ovn-kubernetes/libovsdb v0.8.1
-	github.com/pelletier/go-toml/v2 v2.3.1
+	github.com/pelletier/go-toml/v2 v2.4.0
 	github.com/pterm/pterm v0.12.83
 	github.com/spf13/cobra v1.10.2
 	github.com/spf13/viper v1.21.0
 	github.com/stretchr/testify v1.11.1
 	go.uber.org/goleak v1.3.0
-	golang.org/x/crypto v0.52.0
-	golang.org/x/sync v0.20.0
-	golang.org/x/sys v0.45.0
-	gopkg.in/ini.v1 v1.67.2
+	golang.org/x/crypto v0.53.0
+	golang.org/x/sync v0.21.0
+	golang.org/x/sys v0.46.0
+	gopkg.in/ini.v1 v1.67.3
 	gopkg.in/yaml.v3 v3.0.1
 )
 
@@ -38,7 +38,7 @@ require (
 	github.com/antithesishq/antithesis-sdk-go v0.7.0 // indirect
 	github.com/armon/go-metrics v0.4.1 // indirect
 	github.com/atotto/clipboard v0.1.4 // indirect
-	github.com/aws/aws-sdk-go-v2 v1.41.12 // indirect
+	github.com/aws/aws-sdk-go-v2 v1.42.0 // indirect
 	github.com/aws/smithy-go v1.27.1 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
@@ -57,7 +57,7 @@ require (
 	github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
 	github.com/containerd/console v1.0.5 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/dgraph-io/badger/v4 v4.9.1 // indirect
+	github.com/dgraph-io/badger/v4 v4.9.2 // indirect
 	github.com/dgraph-io/ristretto/v2 v2.4.0 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
@@ -85,7 +85,7 @@ require (
 	github.com/jmespath/go-jmespath v0.4.0 // indirect
 	github.com/josharian/native v1.1.0 // indirect
 	github.com/klauspost/compress v1.18.6 // indirect
-	github.com/klauspost/reedsolomon v1.14.0 // indirect
+	github.com/klauspost/reedsolomon v1.14.1 // indirect
 	github.com/leodido/go-urn v1.4.0 // indirect
 	github.com/lithammer/fuzzysearch v1.1.8 // indirect
 	github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
@@ -126,13 +126,13 @@ require (
 	go.opentelemetry.io/otel/trace v1.42.0 // indirect
 	go.yaml.in/yaml/v2 v2.4.4 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
-	golang.org/x/mod v0.35.0 // indirect
-	golang.org/x/net v0.55.0 // indirect
-	golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa // indirect
-	golang.org/x/term v0.43.0 // indirect
-	golang.org/x/text v0.37.0 // indirect
+	golang.org/x/mod v0.36.0 // indirect
+	golang.org/x/net v0.56.0 // indirect
+	golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6 // indirect
+	golang.org/x/term v0.44.0 // indirect
+	golang.org/x/text v0.38.0 // indirect
 	golang.org/x/time v0.15.0 // indirect
-	golang.org/x/tools v0.44.0 // indirect
+	golang.org/x/tools v0.45.0 // indirect
 	golang.org/x/vuln v1.1.4 // indirect
 	google.golang.org/protobuf v1.36.11 // indirect
 )

go.sum

@@ -31,28 +31,28 @@ github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z
 github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
 github.com/aws/aws-sdk-go v1.55.8 h1:JRmEUbU52aJQZ2AjX4q4Wu7t4uZjOu71uyNmaWlUkJQ=
 github.com/aws/aws-sdk-go v1.55.8/go.mod h1:ZkViS9AqA6otK+JBBNH2++sx1sgxrPKcSzPPvQkUtXk=
-github.com/aws/aws-sdk-go-v2 v1.41.12 h1:DIKX2c31ekm9RA2D9FBj1EWXx++9AdAqRw+e78Tq2Ck=
-github.com/aws/aws-sdk-go-v2 v1.41.12/go.mod h1:27+ACypSLljLAEKsCYOmrjKh83vuTRkuAe9Uv/3A4bg=
+github.com/aws/aws-sdk-go-v2 v1.42.0 h1:XvXMJTkFQtpBKIWZnmr9ZEOc2InWM2yldjXEJ/bymhA=
+github.com/aws/aws-sdk-go-v2 v1.42.0/go.mod h1:27+ACypSLljLAEKsCYOmrjKh83vuTRkuAe9Uv/3A4bg=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.13 h1:p1BBrg/Hhp6uK7zpejeI8QFXHJeC/mynzi04Sl03k9g=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.13/go.mod h1:8cIfkE9MDhkRZGpQ22aV6/lkYeYSozpz16Smrs5x4Ls=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.22 h1:SHfH6wyPsEgG7fVsi5rQxWEt7tuIcN2PGhb1mTFv6tE=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.22/go.mod h1:54nO8lKD4aQPOntM/VTWjnR+DYzTwx0YkSMZMhAgewQ=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.28 h1:Xf2j7NdVcUKomlZ4iihOP4AZ3Fzlr8h4yKpXeP+OFPg=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.28/go.mod h1:O8cDo1dW63jU7ki//kRe1z+tLGcpnD1jrouitsQddDw=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.28 h1:KqIfN9kpkKkcBqBbNpNGTIrXO6ExTUvFKvXkC+YAzVo=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.28/go.mod h1:uxtQiKvLtNS4iXVsH2McVD/ls8FKN/uUhe1hGxPjrw0=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.29 h1:VkE9FuzTQVjBBrnj4+oCdxCLFIz7aqLYKUCjtvxVcOs=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.29/go.mod h1:H32Z2Qth9b+9LqjyBsCnozMQ8H2N7YBUDVXwbs0iggg=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.24 h1:2hQqYCV9yqyePQ9o6dCrZc/zO8U3TwPr9mIKlZnPu/I=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.24/go.mod h1:IDwpACtwqHLISdzfwUUNq4P9DsB/h5BLg4FwJPNfqFY=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.29 h1:f3vKqSo13fhTYb+JEcXwXefZQE26I1FB5eTSniU67ko=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.29/go.mod h1:MzoLFUArKGpGD+ukmPiTPG1X5x4o6M2kq4v2dr1FiEc=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.29 h1:RdwIf/CuUsvJX3RgJagbOyotl/cxoLY4xviKuE7p2GY=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.29/go.mod h1:71wt8W2EgswdZy9Mf9KNnzxZ3TiZlv4caKghPktDOkA=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.30 h1:VTGy885W5DKBxWRUJbym9hytNaYzsyaPkCHGRRMAOhU=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.30/go.mod h1:AS0HycUvJRFvTt613AYDOgO2jzw+00cVSMny8XB3yMY=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.12 h1:ZD2+BSw9vFsNlKYIasSNt3uDbjqqXIBcM13UJv/Lx2k=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.12/go.mod h1:Ms4zlcVBbXbiP7EVLhl+lgjvA/a7YphqQ3Ih3174EmI=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.21 h1:FsZxbPiVgEHYofziwfylouMki8b1Z7mI4CMU/7bhwBA=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.21/go.mod h1:Mmm30OV+JLXYQUcbSd84THnv3P5JtjhVDujLwMqRG0U=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.28 h1:axj4mEDletwKmTm/9jR+DkIMmCfcn5vE4jBMAAN+3Vg=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.28/go.mod h1:3Aaz69M0jqfSHLKqxgolgUBFT4hpwSNc7DzC95orEi8=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.28 h1:li8rTZAAb22g4UsxbjwMdaNVWbgVcDzPqI7nDTI+mF4=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.28/go.mod h1:/brXioSGIMEdcBFoubpSdmighSVp6poP+mma/wB7iHA=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.103.2 h1:b4ikkRk22T4xYkEgaWc3Voe+3xbt5YbbFhNehOWyUiY=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.103.2/go.mod h1:Gp7eHZ0NZ8ZK5RXpoIUp/C8OeAmJqpCgdwEK1D/QOek=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.22 h1:V51LGlOq/1VsDsHUdoklAQi7rMmx4qQubvFYAlP2254=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.22/go.mod h1:4Pzhyz8hJOm2bepgl+NjvRx8vlUFAIIvJnZ/MkcNPpU=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.29 h1:DRebniUGZ2MqiiIVmQJ04vIXr918hubdHMnarSLEWyU=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.29/go.mod h1:LfRkPCD8YHDM2E5eTkos2UpwYeZnBcVarTa8L59bJHA=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.29 h1:hiME6pBzC7OTl9LMtlyTWBuEl1f4QBcUmFDKC7MLXtc=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.29/go.mod h1:G7RP+uhagpKtKhd1BM9N6JQqjCcGEU47K5lBVZQyRQw=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.104.0 h1:ta8csKy5vN91F3i5gGR85lFV0srBqySEji7Jroes6rE=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.104.0/go.mod h1:77ZAgynvx1txMvDG8gGWoWkO1augYDxkp9JElWFgjQU=
 github.com/aws/smithy-go v1.27.1 h1:4T340VFndXtADGF52gYa1POyL7s9E4Z1OeZ1hCscIw8=
 github.com/aws/smithy-go v1.27.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
@@ -103,8 +103,8 @@ github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6N
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/dgraph-io/badger/v4 v4.9.1 h1:DocZXZkg5JJHJPtUErA0ibyHxOVUDVoXLSCV6t8NC8w=
-github.com/dgraph-io/badger/v4 v4.9.1/go.mod h1:5/MEx97uzdPUHR4KtkNt8asfI2T4JiEiQlV7kWUo8c0=
+github.com/dgraph-io/badger/v4 v4.9.2 h1:Wb5qw8gElqwV1a8msHTeQKova9b1V10heFKMIiPd80E=
+github.com/dgraph-io/badger/v4 v4.9.2/go.mod h1:nJjaJTUOSsQEBhsq209FmwCvMJzEA3e74RjZw6V2pQI=
 github.com/dgraph-io/ristretto/v2 v2.4.0 h1:I/w09yLjhdcVD2QV192UJcq8dPBaAJb9pOuMyNy0XlU=
 github.com/dgraph-io/ristretto/v2 v2.4.0/go.mod h1:0KsrXtXvnv0EqnzyowllbVJB8yBonswa2lTCK2gGo9E=
 github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da h1:aIftn67I1fkbMa512G+w+Pxci9hJPB8oMnkcP3iZF38=
@@ -238,8 +238,8 @@ github.com/klauspost/cpuid/v2 v2.0.10/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuOb
 github.com/klauspost/cpuid/v2 v2.0.12/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
 github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
 github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
-github.com/klauspost/reedsolomon v1.14.0 h1:5YSZeclzSYg5nl349+GDG/agDtQ6MZiwUYXvVKN1Jx0=
-github.com/klauspost/reedsolomon v1.14.0/go.mod h1:yjqqjgMTQkBUHSG97/rm4zipffCNbCiZcB3kTqr++sQ=
+github.com/klauspost/reedsolomon v1.14.1 h1:swE9kzyWXD/wVG+l5Pe8bWnQ0giIY7D1GjCBKk3kG2U=
+github.com/klauspost/reedsolomon v1.14.1/go.mod h1:yjqqjgMTQkBUHSG97/rm4zipffCNbCiZcB3kTqr++sQ=
 github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
@@ -288,10 +288,10 @@ github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELU
 github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
 github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
 github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
-github.com/mulgadc/predastore v1.7.0 h1:TuCU1oxs5EUJCqL3MybpyH1JR3WyxLHIj4Fa/sM9a5k=
-github.com/mulgadc/predastore v1.7.0/go.mod h1:0334T4L4Dby54TCuugpumyOhtvHQX+WSxQ/REDNSk9o=
-github.com/mulgadc/viperblock v1.7.0 h1:+SAWg/Dcyb03PC+5ZB5Ab4HWum9PFWJ5KfeVkcivowY=
-github.com/mulgadc/viperblock v1.7.0/go.mod h1:F/LOVydipnyjX9QMkAuf/gjdWeYjaniGHU4emwJwq+U=
+github.com/mulgadc/predastore v1.9.0 h1:nXwyblTso9ICbDkzSVx7j2j1FpCgPe/4c1du5Ge0k18=
+github.com/mulgadc/predastore v1.9.0/go.mod h1:FBFtu96XZlslhIJ2+Tv9NdR1S3Ks5XrWvJ+Q59iBxVA=
+github.com/mulgadc/viperblock v1.9.0 h1:rogwhMefiHIo7T8ApDYMQEs6UNGFNHtH8Bj2W7RzXU8=
+github.com/mulgadc/viperblock v1.9.0/go.mod h1:tq3msiqnmURyqEFd0oeMrVv3hpIgefXLurEn9exz+rI=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
@@ -310,8 +310,8 @@ github.com/ovn-kubernetes/libovsdb v0.8.1 h1:M2J8bcJt5mXCom0HqzfEtuHkT80CTSQRcYG
 github.com/ovn-kubernetes/libovsdb v0.8.1/go.mod h1:ZlnHLzagmLOSvyd9qfxBIZp6wOSOw0IsRsc+6lNUGbU=
 github.com/pascaldekloe/goe v0.1.0 h1:cBOtyMzM9HTpWjXfbbunk26uA6nG3a8n06Wieeh0MwY=
 github.com/pascaldekloe/goe v0.1.0/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
-github.com/pelletier/go-toml/v2 v2.3.1 h1:MYEvvGnQjeNkRF1qUuGolNtNExTDwct51yp7olPtrEc=
-github.com/pelletier/go-toml/v2 v2.3.1/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
+github.com/pelletier/go-toml/v2 v2.4.0 h1:Mwu0mAkUKbittDs3/ADDWXqMmq3EOK2VHiuCkV00Row=
+github.com/pelletier/go-toml/v2 v2.4.0/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
 github.com/pierrec/lz4/v4 v4.1.14 h1:+fL8AQEZtz/ijeNnpduH0bROTu0O3NZAlPjQxGn8LwE=
 github.com/pierrec/lz4/v4 v4.1.14/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
@@ -428,14 +428,14 @@ golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnf
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
-golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
-golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
+golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
+golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa h1:FRnLl4eNAQl8hwxVVC17teOw8kdjVDVAiFMtgUdTSRQ=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa/go.mod h1:zk2irFbV9DP96SEBUUAy67IdHUaZuSnrz1n472HUCLE=
 golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
 golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
-golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM=
-golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU=
+golang.org/x/mod v0.36.0 h1:JJjpVx6myfUsUdAzZuOSTTmRE0PfZeNWzzvKrP7amb4=
+golang.org/x/mod v0.36.0/go.mod h1:moc6ELqsWcOw5Ef3xVprK5ul/MvtVvkIXLziUOICjUQ=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -446,8 +446,8 @@ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81R
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
-golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
-golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
+golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
+golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -456,8 +456,8 @@ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
-golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
+golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
+golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -488,33 +488,33 @@ golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
-golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
-golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
-golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa h1:efT73AJZfAAUV7SOip6pWGkwJDzIGiKBZGVzHYa+ve4=
-golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa/go.mod h1:kHjTxDEnAu6/Nl9lDkzjWpR+bmKfxeiRuSDlsMb70gE=
+golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
+golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6 h1:HjU6IWBiAgRIdAJ9/y1rwCn+UELEmwV+VsTLzj/W4sE=
+golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6/go.mod h1:Eqhaxk/wZsWEH8CRxLwj6xzEJbz7k1EFGqx7nyCoabE=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
-golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4=
-golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=
+golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc=
+golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
-golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
-golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
+golang.org/x/text v0.38.0 h1:sXmwo9DwP3OK9EZ7PqAdaooSGozfl/3a6/xJcbzPRhE=
+golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4=
 golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
 golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
 golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
-golang.org/x/tools v0.44.0 h1:UP4ajHPIcuMjT1GqzDWRlalUEoY+uzoZKnhOjbIPD2c=
-golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI=
+golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8=
+golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0=
 golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM=
 golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY=
 golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM=
@@ -538,8 +538,8 @@ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
-gopkg.in/ini.v1 v1.67.2 h1:JtOSMb9OuaCZKr7h5D/h6iii14sK0hLbplTc6frx4Ss=
-gopkg.in/ini.v1 v1.67.2/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
+gopkg.in/ini.v1 v1.67.3 h1:iM9Lhz5MRSGhHVGGwCuzG9KO8PoirCXj/m/qTmOJJQw=
+gopkg.in/ini.v1 v1.67.3/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

internal/eksgw/client.go

@@ -1,8 +1,6 @@
-// Package eksgw is the on-VM SigV4 client EKS control-plane helpers use to
-// reach the AWS gateway, instead of dialing core NATS. Both eks-gateway-publish
-// (bootstrap + state relay) and eks-token-webhook (TokenReview relay) sign with
-// the system (Predastore) credentials for service "eks" and POST over HTTPS;
-// the gateway authorizes via IAM and relays to the host-side NATS/STS/KV.
+// Package eksgw is the on-VM SigV4 client used by EKS helpers (eks-gateway-publish,
+// eks-token-webhook) to POST over HTTPS to the AWS gateway instead of NATS.
+// Signs with Predastore credentials for service "eks".
 package eksgw
 
 import (
@@ -73,16 +71,27 @@ func New(baseURL, caPath, accessKey, secretKey, region string) (*Client, error)
 	}, nil
 }
 
-// Post SigV4-signs and sends a single POST of body to path (e.g.
-// "/clusters/alpha/token-review"). It returns the response body on a 2xx, or an
-// error (including the gateway status + body) otherwise. No retry — callers that
-// need it wrap Post in their own loop.
+// Post SigV4-signs and sends body to path, returning the response body on 2xx.
+// Errors include the gateway status and body. No retry; callers wrap as needed.
 func (c *Client) Post(path string, body []byte) ([]byte, error) {
-	req, err := http.NewRequest(http.MethodPost, c.baseURL+path, bytes.NewReader(body))
+	return c.do(http.MethodPost, path, body)
+}
+
+// Get SigV4-signs and sends a single GET to path (e.g.
+// "/clusters/alpha/internal-addons?accountId=..."). Same contract as Post: 2xx
+// body or an error carrying the gateway status. No retry.
+func (c *Client) Get(path string) ([]byte, error) {
+	return c.do(http.MethodGet, path, nil)
+}
+
+func (c *Client) do(method, path string, body []byte) ([]byte, error) {
+	req, err := http.NewRequest(method, c.baseURL+path, bytes.NewReader(body))
 	if err != nil {
 		return nil, fmt.Errorf("create request: %w", err)
 	}
-	req.Header.Set("Content-Type", "application/x-amz-json-1.1")
+	if body != nil {
+		req.Header.Set("Content-Type", "application/x-amz-json-1.1")
+	}
 
 	sum := sha256.Sum256(body)
 	if err := auth.SignReq(req, c.accessKey, c.secretKey, hex.EncodeToString(sum[:]), "eks", c.region); err != nil {

internal/fipsboot/fipsboot.go

@@ -1,12 +1,6 @@
-// Package fipsboot enforces FIPS 140-3 mode at process startup.
-//
-// Blank-import this package from every binary's main:
-//
-//	import _ "github.com/mulgadc/spinifex/internal/fipsboot"
-//
-// Builds with GOFIPS140=v1.0.0 link the Go Cryptographic Module v1.0.0
-// (NIST CMVP cert #5247). At runtime FIPS mode can still be turned off
-// via GODEBUG=fips140=off; init() panics in that case to close the hole.
+// Package fipsboot enforces FIPS 140-3 at startup via blank import in every binary.
+// Requires GOFIPS140=v1.0.0 at build time; panics if GODEBUG=fips140=off disables
+// the runtime FIPS mode.
 package fipsboot
 
 import (

internal/tlsconfig/curves.go

@@ -1,34 +1,13 @@
 // Package tlsconfig pins the cluster-wide TLS 1.3 curve preference list.
-//
-// The list is hardcoded because mixed classification on a single cluster is
-// not supported and the certified FIPS build is itself the per-deployment
-// choice. A typo in a config string would silently downgrade where a missing
-// constant cannot.
+// Hardcoded to prevent silent downgrade via misconfiguration; the FIPS build
+// is the per-deployment choice.
 package tlsconfig
 
 import "crypto/tls"
 
-// Curves is the fixed TLS 1.3 curve allowlist:
-//
-//   - X25519MLKEM768     — Cat 3 ML-KEM hybrid, wins under Go 1.26 stdlib filter
-//   - SecP384r1MLKEM1024 — Cat 5 ML-KEM hybrid, fallback for peers that prefer it
-//   - X25519             — classical TLS 1.3 fallback for SDKs without ML-KEM (aws CLI v2 today)
-//   - CurveP384          — classical TLS 1.3 fallback (Java/.NET enterprise SDKs)
-//
-// PQ-capable peers (browsers since 2024, Go aws-sdk v2 with Go 1.24+) always
-// negotiate the hybrid first; classical fallback only triggers for peers that
-// cannot do ML-KEM. Those connections still get TLS 1.3 + AES-256-GCM but
-// remain HNDL-exposed — accepted trade-off until AWS CLI v2 / bundled
-// OpenSSL ships ML-KEM (currently lagging upstream OpenSSL 3.5).
-//
-// Weak/niche entries from the Go stdlib default (SecP256r1MLKEM768, P-256,
-// P-521) are intentionally excluded so they cannot be advertised.
-//
-// Go 1.26 stdlib treats tls.Config.CurvePreferences as an allowlist filter
-// applied to the stdlib default order, not as a user-supplied order.
-//
-// Callers must not mutate this slice. Assign it directly to
-// tls.Config.CurvePreferences; do not append to it.
+// Curves is the fixed TLS 1.3 curve allowlist: two ML-KEM hybrids (PQ-capable
+// peers always negotiate these) followed by two classical fallbacks for SDKs
+// without ML-KEM. Weak Go defaults (SecP256r1MLKEM768, P-256, P-521) excluded.
 var Curves = []tls.CurveID{
 	tls.X25519MLKEM768,
 	tls.SecP384r1MLKEM1024,

internal/tlsconfig/curves_test.go

@@ -46,11 +46,9 @@ func TestCurves_ExcludesWeakPrimitives(t *testing.T) {
 	}
 }
 
-// TestIntegration_NegotiatesPQHybrid spins up an httptest TLS server
-// configured with Curves + TLS 1.3 minimum, dials it, and asserts a PQ
-// hybrid is negotiated at TLS 1.3. Per Go 1.26 stdlib filter behavior the
-// winner is X25519MLKEM768; the assertion checks for any approved hybrid
-// to remain stable across stdlib reorderings of our 4 entries.
+// TestIntegration_NegotiatesPQHybrid verifies that a Curves-configured server
+// negotiates a PQ hybrid at TLS 1.3; checks for any approved hybrid to stay
+// stable across Go stdlib reorderings.
 func TestIntegration_NegotiatesPQHybrid(t *testing.T) {
 	cert := selfSignedCert(t)
 	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
@@ -95,10 +93,8 @@ func TestIntegration_NegotiatesPQHybrid(t *testing.T) {
 	}
 }
 
-// TestIntegration_TLS12ClientRejected confirms MinVersion: tls.VersionTLS13
-// is enforced — a TLS 1.2-pinned client gets a handshake error rather than
-// silently downgrading. This is the negative test that closes the HNDL gap
-// on external surfaces.
+// TestIntegration_TLS12ClientRejected confirms MinVersion TLS 1.3 is enforced:
+// a TLS 1.2-pinned client must get a handshake error, not a silent downgrade.
 func TestIntegration_TLS12ClientRejected(t *testing.T) {
 	cert := selfSignedCert(t)
 	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {

scripts/build-microvm-image.sh

@@ -81,30 +81,46 @@ EOF
         --initdb \
         $ALPINE_PACKAGES
 
-# Strategy 2/3: container runtime (dev machines)
-else
-    if command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1; then
-        CONTAINER_TOOL=docker
-    elif command -v podman >/dev/null 2>&1; then
-        CONTAINER_TOOL=podman
-    else
-        echo "ERROR: no rootfs build tool available." >&2
-        echo "       Install one of: apk (Alpine), docker, podman" >&2
-        exit 1
-    fi
+# Strategy 2: docker -> BuildKit local export.
+# Do NOT use `docker export` here. On Docker's overlay2 driver it leaks the
+# container's overlay mount (the mount survives `docker rm`), orphaning layers
+# under /var/lib/docker that only a daemon restart reclaims; on a busy CI host
+# this grows without bound. `buildx --output type=local` writes the image
+# filesystem straight to a directory and releases every mount, so nothing leaks.
+elif command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1; then
+    echo "[build-microvm-image] rootfs: docker buildx --output type=local (alpine:${ALPINE_VERSION})"
+    BUILD_CTX=$(mktemp -d)
+    cat > "$BUILD_CTX/Dockerfile" <<DOCKERFILE
+FROM alpine:${ALPINE_VERSION}
+RUN apk add --no-cache ${ALPINE_PACKAGES}
+DOCKERFILE
+    docker buildx build \
+        --platform linux/amd64 \
+        --output "type=local,dest=${CHROOT_DIR}" \
+        "$BUILD_CTX"
+    rm -rf "$BUILD_CTX"
 
-    echo "[build-microvm-image] rootfs: $CONTAINER_TOOL export (alpine:${ALPINE_VERSION})"
-    CONTAINER_CID=$($CONTAINER_TOOL run -d \
+# Strategy 3: podman -> run + export. Podman's store does not exhibit the docker
+# overlay2 export-mount leak, and podman has no buildx.
+elif command -v podman >/dev/null 2>&1; then
+    echo "[build-microvm-image] rootfs: podman export (alpine:${ALPINE_VERSION})"
+    CONTAINER_TOOL=podman
+    CONTAINER_CID=$(podman run -d \
         "alpine:${ALPINE_VERSION}" \
         sh -c "apk add --no-cache ${ALPINE_PACKAGES}")
 
-    exit_code=$($CONTAINER_TOOL wait "$CONTAINER_CID")
+    exit_code=$(podman wait "$CONTAINER_CID")
     if [ "$exit_code" != "0" ]; then
         echo "ERROR: package installation failed in container (exit $exit_code)" >&2
         exit 1
     fi
 
-    $CONTAINER_TOOL export "$CONTAINER_CID" | tar -x -C "$CHROOT_DIR"
+    podman export "$CONTAINER_CID" | tar -x -C "$CHROOT_DIR"
+
+else
+    echo "ERROR: no rootfs build tool available." >&2
+    echo "       Install one of: apk (Alpine), docker, podman" >&2
+    exit 1
 fi
 
 # --- /dev: empty mountpoint only ---

scripts/build-system-image.sh

@@ -26,10 +26,11 @@ PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
 export LIBGUESTFS_BACKEND="${LIBGUESTFS_BACKEND:-direct}"
 
 usage() {
-    echo "Usage: $0 <manifest.conf> [--import]"
+    echo "Usage: $0 <manifest.conf> [--import] [--no-cache]"
     echo ""
     echo "  manifest.conf  Path to image manifest (see scripts/images/ for examples)"
     echo "  --import       Import the built image as an AMI"
+    echo "  --no-cache     Always rebuild; ignore a recently-built cached raw"
     exit 1
 }
 
@@ -47,10 +48,12 @@ fi
 
 DO_IMPORT=false
 QUIET=false
+NO_CACHE=false
 for arg in "$@"; do
     case "$arg" in
-        --import) DO_IMPORT=true ;;
-        --quiet)  QUIET=true ;;
+        --import)   DO_IMPORT=true ;;
+        --quiet)    QUIET=true ;;
+        --no-cache) NO_CACHE=true ;;
     esac
 done
 
@@ -184,7 +187,9 @@ echo "Lock acquired"
 
 # If the raw image was built recently (< 10 min), skip the entire build.
 # This avoids duplicate work when concurrent CI jobs build the same image.
-if [[ -f "$OUTPUT_RAW" ]] && [[ $(( $(date +%s) - $(stat -c %Y "$OUTPUT_RAW") )) -lt 600 ]]; then
+# --no-cache (e.g. the publish path) forces a fresh build so a stale raw from
+# a prior/concurrent run on a persistent runner is never republished.
+if [[ "$NO_CACHE" == false ]] && [[ -f "$OUTPUT_RAW" ]] && [[ $(( $(date +%s) - $(stat -c %Y "$OUTPUT_RAW") )) -lt 600 ]]; then
     echo "=== Skipping build — $OUTPUT_RAW is fresh (< 10 min old) ==="
 
     # Restore stdout if suppressed, then jump to import

scripts/clone-deps.sh

@@ -75,5 +75,5 @@ echo ""
 echo "🎉 Development environment setup complete!"
 echo ""
 echo "Next step:"
-echo "Run development setup to build services: ./scripts/dev-setup.sh"
+echo "Run development setup to build, install, and start services: ./scripts/dev-install.sh"
 echo ""

scripts/dev-install.sh

@@ -44,9 +44,9 @@ sudo systemctl stop spinifex.target 2>/dev/null || true
 sudo systemctl reset-failed 'spinifex-*' 2>/dev/null || true
 
 # Remove stale files owned by the dev user in production paths.
-# Previous dev-mode installs (admin init without sudo, start-dev.sh) leave
-# files owned by tf-user that service users (spinifex-nats, etc.) can't read
-# under systemd's ProtectSystem=strict sandboxing.
+# A prior admin init run without sudo can leave files owned by tf-user that
+# service users (spinifex-nats, etc.) can't read under systemd's
+# ProtectSystem=strict sandboxing.
 for dir in /var/lib/spinifex /var/log/spinifex /etc/spinifex; do
     if [ -d "$dir" ]; then
         # Remove PID files, stale logs, and the legacy ~/spinifex/config symlink

scripts/dev-setup.sh

@@ -1,180 +0,0 @@
-#!/bin/bash
-
-# Spinifex Platform Development Environment Setup
-# This script sets up a complete development environment for the Spinifex platform
-
-set -euo pipefail
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
-MULGA_ROOT="$(cd "$PROJECT_ROOT/.." && pwd)"
-
-# Default configuration
-NATS_PORT=4222
-PREDASTORE_PORT=8443
-SPINIFEX_GATEWAY_PORT=9999
-
-echo "🚀 Setting up Spinifex development environment..."
-echo "Project root: $PROJECT_ROOT"
-
-# Function to check if command exists
-command_exists() {
-    command -v "$1" >/dev/null 2>&1
-}
-
-# Function to check if port is available
-port_available() {
-    ! nc -z localhost "$1" 2>/dev/null
-}
-
-# Check dependencies
-echo "🔍 Checking dependencies..."
-
-# Check for required commands
-required_commands=("go" "make")
-missing_required=false
-for cmd in "${required_commands[@]}"; do
-    if command_exists "$cmd"; then
-        echo "✅ $cmd found"
-    else
-        echo "❌ $cmd not found. Please install $cmd"
-        missing_required=true
-        case "$cmd" in
-            "go")
-                echo "   Install Go: https://golang.org/dl/"
-                ;;
-        esac
-    fi
-done
-if [ "$missing_required" = true ]; then
-    echo "❌ Missing required dependencies. Cannot continue."
-    exit 1
-fi
-
-# Check optional commands
-optional_commands=("air" "nbdkit")
-for cmd in "${optional_commands[@]}"; do
-    if command_exists "$cmd"; then
-        echo "✅ $cmd found (optional)"
-    else
-        echo "⚠️  $cmd not found (optional)"
-        case "$cmd" in
-            "air")
-                echo "   Install air for hot reloading: go install github.com/air-verse/air@latest"
-                ;;
-            "nbdkit")
-                echo "   Install nbdkit: sudo apt-get install nbdkit (Ubuntu/Debian)"
-                ;;
-        esac
-    fi
-done
-
-# Check OVN/OVS commands (required for VPC networking)
-echo ""
-echo "🔍 Checking OVN/OVS dependencies..."
-ovn_commands=("ovs-vsctl" "ovn-controller" "ovn-nbctl")
-ovn_ready=true
-for cmd in "${ovn_commands[@]}"; do
-    if command_exists "$cmd"; then
-        echo "✅ $cmd found"
-    else
-        echo "⚠️  $cmd not found (required for VPC networking)"
-        ovn_ready=false
-    fi
-done
-if [ "$ovn_ready" = false ]; then
-    echo "   Install OVN: sudo apt-get install ovn-central ovn-host openvswitch-switch"
-    echo "   Or run: make quickinstall"
-fi
-
-# Initialize br-int if OVS is available
-if command_exists "ovs-vsctl"; then
-    echo ""
-    echo "🌐 Checking OVS br-int bridge..."
-    if ovs-vsctl br-exists br-int 2>/dev/null; then
-        echo "✅ br-int already exists"
-    else
-        echo "⚠️  br-int not found. Run ./scripts/setup-ovn.sh to initialize OVN networking"
-    fi
-fi
-
-# Check component repositories
-echo ""
-echo "📦 Checking component repositories..."
-
-components=("viperblock" "predastore")
-for component in "${components[@]}"; do
-    component_path="$MULGA_ROOT/$component"
-    if [[ -d "$component_path" ]]; then
-        echo "✅ $component found at $component_path"
-        if [[ -f "$component_path/go.mod" ]]; then
-            echo "   Go module verified"
-        else
-            echo "   ⚠️  No go.mod found in $component"
-        fi
-    else
-        echo "❌ $component not found at $component_path"
-        echo "   Run: ./scripts/clone-deps.sh"
-    fi
-done
-
-# Check port availability
-echo ""
-echo "🔌 Checking port availability..."
-
-ports=("$NATS_PORT:NATS" "$PREDASTORE_PORT:Predastore" "$SPINIFEX_GATEWAY_PORT:Spinifex Gateway")
-for port_info in "${ports[@]}"; do
-    port="${port_info%:*}"
-    service="${port_info#*:}"
-    if port_available "$port"; then
-        echo "✅ Port $port available for $service"
-    else
-        echo "⚠️  Port $port already in use (needed for $service)"
-    fi
-done
-
-# Build Spinifex first (needed for admin init)
-echo ""
-echo "🔨 Building Spinifex..."
-cd "$PROJECT_ROOT"
-make build
-echo "✅ Spinifex built successfully"
-
-# Build components
-echo ""
-echo "🔨 Building components..."
-
-# Build Viperblock
-if [[ -d "$MULGA_ROOT/viperblock" ]]; then
-    echo "📦 Building Viperblock..."
-    cd "$MULGA_ROOT/viperblock"
-    make build
-    echo "✅ Viperblock built successfully"
-else
-    echo "⚠️  Viperblock directory not found, skipping build"
-fi
-
-# Build Predastore
-if [[ -d "$MULGA_ROOT/predastore" ]]; then
-    echo "📦 Building Predastore..."
-    cd "$MULGA_ROOT/predastore"
-    make build
-    echo "✅ Predastore built successfully"
-else
-    echo "⚠️  Predastore directory not found, skipping build"
-fi
-
-echo ""
-echo "Development environment setup complete!"
-echo ""
-echo "Recommended: Use scripts/dev-install.sh for a production-parity setup"
-echo "  (systemd, privilege separation, production paths)."
-echo ""
-echo "  scripts/dev-install.sh  # One-time: build, install, init, start via systemd"
-echo "  make deploy             # Daily: rebuild + copy binary + restart all"
-echo "  make reinstall          # Changed a .service file? Re-run full setup"
-echo ""
-echo "Legacy workflow (deprecated, retained for pseudo-multinode CI only):"
-echo "  ./scripts/setup-ovn.sh --management"
-echo "  sudo ./bin/spx admin init --region ap-southeast-2 --az ap-southeast-2a --node node1 --nodes 1"
-echo "  ./scripts/start-dev.sh"

scripts/iac/README.md

@@ -1,9 +0,0 @@
-# IAC
-
-Collection of OpenTofu/Terraform templates for creating Spinifex instances.
-
-| Template | Description |
-|---|---|
-| [proxmox/](proxmox/) | Provisions a 3-node Spinifex cluster on Proxmox VE bare-metal hypervisors |
-| [aws/](aws/) | Provisions infrastructure using the Terraform AWS provider, with MulgaOS AWS-compatible API as the local endpoint |
-| [proxmox-template/](proxmox-template/) | Builds a Proxmox VM template with all Spinifex dependencies pre-installed (QEMU, Go, repos, modules) |

scripts/iac/aws/README.md

@@ -1,206 +0,0 @@
-# MulgaOS AWS Infrastructure as Code
-
-This directory contains OpenTofu/Terraform configuration files for deploying infrastructure on MulgaOS using AWS-compatible APIs.
-
-## Prerequisites
-
-1. **OpenTofu or Terraform installed**
-   ```bash
-   # Install OpenTofu (recommended)
-   # See: https://opentofu.org/docs/intro/install/
-
-   # Or use Terraform
-   # See: https://developer.hashicorp.com/terraform/install
-   ```
-
-2. **MulgaOS services running**
-   - `spinifex daemon` - Core daemon with NATS subscriptions
-   - `awsd` - AWS API gateway (default: http://localhost:8080)
-   - `predastore` - S3-compatible storage
-   - `viperblock` - EBS-compatible block storage
-
-3. **AWS credentials configured**
-   ```bash
-   # Option 1: Environment variables
-   export AWS_ACCESS_KEY_ID=your_access_key
-   export AWS_SECRET_ACCESS_KEY=your_secret_key
-
-   # Option 2: AWS CLI profile
-   aws configure --profile spinifex
-   export AWS_PROFILE=spinifex
-   ```
-
-4. **AMI available**
-   Before running, ensure you have a Debian 13 AMI imported:
-   ```bash
-   # Import an AMI using the spinifex CLI or aws CLI
-   aws --endpoint-url http://localhost:8080 ec2 describe-images
-   ```
-
-## Configuration
-
-### Environment Variables
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `TF_VAR_mulgaos_endpoint` | `http://localhost:8080` | MulgaOS API endpoint |
-| `TF_VAR_region` | `ap-southeast-2` | AWS region (for compatibility) |
-| `AWS_ACCESS_KEY_ID` | - | MulgaOS access key |
-| `AWS_SECRET_ACCESS_KEY` | - | MulgaOS secret key |
-
-### terraform.tfvars (optional)
-
-Create a `terraform.tfvars` file for custom settings:
-
-```hcl
-mulgaos_endpoint = "http://192.168.1.100:8080"
-region           = "us-west-2"
-```
-
-## Usage
-
-### Initialize
-
-```bash
-cd iac/aws
-tofu init
-# or: terraform init
-```
-
-### Validate
-
-```bash
-tofu validate
-# or: terraform validate
-```
-
-### Plan
-
-```bash
-tofu plan
-# or: terraform plan
-```
-
-### Apply
-
-```bash
-tofu apply
-# or: terraform apply
-```
-
-### Destroy
-
-```bash
-tofu destroy
-# or: terraform destroy
-```
-
-## What Gets Created
-
-The `main.tf` creates:
-
-1. **VPC** (`10.1.1.0/16`) - Virtual Private Cloud
-2. **Internet Gateway** - For public internet access
-3. **Public Subnet** (`10.1.2.0/24`) - With auto-assign public IP
-4. **Route Table** - Routes `0.0.0.0/0` through IGW
-5. **Security Group** - Allows SSH (22), HTTP (80), HTTPS (443) inbound
-6. **Placement Group** - Spread strategy for HA
-7. **3 EC2 Instances** - t3.small Debian 13 VMs
-8. **SSH Key Pair** - ED25519 key saved to `spinifex-test.pem`
-
-## SSH Access
-
-After `tofu apply`, SSH into instances:
-
-```bash
-# Using the generated key
-ssh -i spinifex-test.pem admin@<public_ip>
-
-# For local development with QEMU port forwarding
-# Find the forwarded port from QEMU args (-netdev user,id=net0,hostfwd=tcp:127.0.0.1:PORT-:22)
-ssh -p PORT -i spinifex-test.pem admin@127.0.0.1
-```
-
-## Troubleshooting
-
-### "No valid credential sources found"
-
-Ensure AWS credentials are set:
-```bash
-export AWS_ACCESS_KEY_ID=your_key
-export AWS_SECRET_ACCESS_KEY=your_secret
-```
-
-### "Error: No AMI found"
-
-Import a Debian 13 AMI first, or modify `main.tf` to use a different AMI filter.
-
-### Connection refused
-
-Ensure MulgaOS services are running:
-```bash
-# Check if awsd gateway is running
-curl http://localhost:8080/health
-
-# Check daemon status
-spinifex daemon status
-```
-
-### "InvalidAction" errors
-
-Some AWS operations may not be implemented yet. Check `TODO.md` for the current implementation status.
-
-## Customization
-
-### Using a different AMI
-
-Modify the `data "aws_ami"` block in `main.tf`:
-
-```hcl
-data "aws_ami" "custom" {
-  most_recent = true
-  owners      = ["self"]  # Your imported AMIs
-
-  filter {
-    name   = "name"
-    values = ["my-custom-ami-*"]
-  }
-}
-```
-
-### Changing instance count
-
-Modify the `count` parameter:
-
-```hcl
-resource "aws_instance" "vm" {
-  count = 5  # Change from 3 to 5
-  # ...
-}
-```
-
-### Different instance type
-
-```hcl
-resource "aws_instance" "vm" {
-  instance_type = "t3.medium"  # Change from t3.small
-  # ...
-}
-```
-
-## Files
-
-| File | Description |
-|------|-------------|
-| `main.tf` | Main Terraform/OpenTofu configuration |
-| `spinifex-test.pem` | Generated SSH private key (after apply) |
-| `.terraform/` | Provider plugins (after init) |
-| `terraform.tfstate` | State file (after apply) |
-| `terraform.tfvars` | Custom variables (optional) |
-
-## Related Documentation
-
-- [MulgaOS DEV.md](../../DEV.md) - Development guide
-- [MulgaOS TODO.md](../../TODO.md) - Implementation status
-- [OpenTofu Documentation](https://opentofu.org/docs/)
-- [AWS Provider Documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs)

scripts/iac/aws/basic/main.tf

@@ -1,307 +0,0 @@
-# Spinifex IaC Compatibility Test — "basic" suite
-#
-# Exercises every AWS resource Spinifex currently supports via tofu apply,
-# WITHOUT route tables, NACLs, or other unimplemented networking.
-#
-# Usage:
-#   cd spinifex/scripts/iac/aws/basic
-#   tofu init
-#   tofu plan
-#   tofu apply        # full round-trip against a running Spinifex cluster
-#   tofu destroy      # clean teardown
-
-terraform {
-  required_version = ">= 1.6.0"
-
-  required_providers {
-    aws = {
-      source  = "hashicorp/aws"
-      version = ">= 5.0"
-    }
-    tls = {
-      source  = "hashicorp/tls"
-      version = ">= 4.0"
-    }
-    local = {
-      source  = "hashicorp/local"
-      version = ">= 2.0"
-    }
-  }
-}
-
-# ---------------------------------------------------------------------------
-# Variables
-# ---------------------------------------------------------------------------
-
-variable "region" {
-  type    = string
-  default = "ap-southeast-2"
-}
-
-variable "mulgaos_endpoint" {
-  type        = string
-  default     = "https://localhost:9999"
-  description = "Spinifex AWS gateway endpoint"
-}
-
-# ---------------------------------------------------------------------------
-# Provider
-# ---------------------------------------------------------------------------
-
-provider "aws" {
-  region = var.region
-
-  endpoints {
-    ec2 = var.mulgaos_endpoint
-    s3  = var.mulgaos_endpoint
-    iam = var.mulgaos_endpoint
-    sts = var.mulgaos_endpoint
-  }
-
-  skip_credentials_validation = true
-  skip_metadata_api_check     = true
-  skip_requesting_account_id  = true
-  skip_region_validation      = true
-}
-
-# ---------------------------------------------------------------------------
-# Data sources
-# ---------------------------------------------------------------------------
-
-data "aws_availability_zones" "available" {
-  state = "available"
-}
-
-data "aws_ami" "debian13" {
-  most_recent = true
-  owners      = ["000000000000"]
-
-  filter {
-    name   = "name"
-    values = ["*debian-13*"]
-  }
-
-  filter {
-    name   = "virtualization-type"
-    values = ["hvm"]
-  }
-
-  filter {
-    name   = "root-device-type"
-    values = ["ebs"]
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 1. VPC + Subnet (no route table needed)
-# ---------------------------------------------------------------------------
-
-resource "aws_vpc" "test" {
-  cidr_block           = "10.99.0.0/16"
-  enable_dns_hostnames = true
-  enable_dns_support   = true
-
-  tags = {
-    Name = "spx-basic-test-vpc"
-  }
-}
-
-resource "aws_subnet" "test" {
-  vpc_id                  = aws_vpc.test.id
-  cidr_block              = "10.99.1.0/24"
-  availability_zone       = data.aws_availability_zones.available.names[0]
-  map_public_ip_on_launch = true
-
-  tags = {
-    Name = "spx-basic-test-subnet"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 2. Internet Gateway (attach to VPC, but no route table referencing it)
-# ---------------------------------------------------------------------------
-
-resource "aws_internet_gateway" "test" {
-  vpc_id = aws_vpc.test.id
-
-  tags = {
-    Name = "spx-basic-test-igw"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 3. Security Group — inbound SSH + HTTP, all outbound
-# ---------------------------------------------------------------------------
-
-resource "aws_security_group" "test" {
-  name        = "spx-basic-test-sg"
-  description = "Basic test: SSH + HTTP inbound, all outbound"
-  vpc_id      = aws_vpc.test.id
-
-  ingress {
-    description = "SSH"
-    from_port   = 22
-    to_port     = 22
-    protocol    = "tcp"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  ingress {
-    description = "HTTP"
-    from_port   = 80
-    to_port     = 80
-    protocol    = "tcp"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  egress {
-    description = "All outbound"
-    from_port   = 0
-    to_port     = 0
-    protocol    = "-1"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  tags = {
-    Name = "spx-basic-test-sg"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 4. Key Pair
-# ---------------------------------------------------------------------------
-
-resource "tls_private_key" "test" {
-  algorithm = "ED25519"
-}
-
-resource "aws_key_pair" "test" {
-  key_name   = "spx-basic-test-key"
-  public_key = tls_private_key.test.public_key_openssh
-}
-
-resource "local_file" "test_pem" {
-  filename        = "${path.module}/spx-basic-test.pem"
-  content         = tls_private_key.test.private_key_openssh
-  file_permission = "0600"
-}
-
-# ---------------------------------------------------------------------------
-# 5. Placement Group
-# ---------------------------------------------------------------------------
-
-resource "aws_placement_group" "test" {
-  name     = "spx-basic-test-spread"
-  strategy = "spread"
-}
-
-# ---------------------------------------------------------------------------
-# 6. EC2 Instance
-# ---------------------------------------------------------------------------
-
-resource "aws_instance" "test" {
-  ami           = data.aws_ami.debian13.id
-  instance_type = "t3.small"
-
-  subnet_id              = aws_subnet.test.id
-  vpc_security_group_ids = [aws_security_group.test.id]
-  key_name               = aws_key_pair.test.key_name
-  placement_group        = aws_placement_group.test.name
-
-  associate_public_ip_address = true
-
-  tags = {
-    Name        = "spx-basic-test-vm"
-    Environment = "test"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 7. EBS Volume + Attachment
-# ---------------------------------------------------------------------------
-
-resource "aws_ebs_volume" "data" {
-  availability_zone = data.aws_availability_zones.available.names[0]
-  size              = 10
-  type              = "gp3"
-
-  tags = {
-    Name = "spx-basic-test-data-vol"
-  }
-}
-
-
-# ---------------------------------------------------------------------------
-# 8. EBS Snapshot (of the standalone volume)
-# ---------------------------------------------------------------------------
-
-resource "aws_ebs_snapshot" "data" {
-  volume_id = aws_ebs_volume.data.id
-
-  tags = {
-    Name = "spx-basic-test-snapshot"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 9. Elastic IP + Association
-#    Requires external IPAM (pool mode) — uncomment when available.
-# ---------------------------------------------------------------------------
-
-# resource "aws_eip" "test" {
-#   domain = "vpc"
-#
-#   tags = {
-#     Name = "spx-basic-test-eip"
-#   }
-# }
-#
-# resource "aws_eip_association" "test" {
-#   allocation_id = aws_eip.test.id
-#   instance_id   = aws_instance.test.id
-# }
-
-# ---------------------------------------------------------------------------
-# 10. Standalone Network Interface (not attached — just tests CRUD)
-# ---------------------------------------------------------------------------
-
-resource "aws_network_interface" "extra" {
-  subnet_id       = aws_subnet.test.id
-  security_groups = [aws_security_group.test.id]
-
-  tags = {
-    Name = "spx-basic-test-eni"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# Outputs
-# ---------------------------------------------------------------------------
-
-output "vpc_id" {
-  value = aws_vpc.test.id
-}
-
-output "instance_id" {
-  value = aws_instance.test.id
-}
-
-output "instance_public_ip" {
-  value = aws_instance.test.public_ip
-}
-
-# output "eip_public_ip" {
-#   value = aws_eip.test.public_ip
-# }
-
-output "volume_id" {
-  value = aws_ebs_volume.data.id
-}
-
-output "snapshot_id" {
-  value = aws_ebs_snapshot.data.id
-}
-
-output "ssh_command" {
-  value = "ssh -i spx-basic-test.pem admin@${aws_instance.test.public_ip}"
-}

scripts/iac/aws/main.tf

@@ -1,217 +0,0 @@
-terraform {
-  required_version = ">= 1.6.0"
-
-  required_providers {
-    aws = {
-      source  = "hashicorp/aws"
-      version = ">= 5.0"
-    }
-    tls = {
-      source  = "hashicorp/tls"
-      version = ">= 4.0"
-    }
-    local = {
-      source  = "hashicorp/local"
-      version = ">= 2.0"
-    }
-  }
-}
-
-variable "region" {
-  type    = string
-  default = "ap-southeast-2" # Sydney, change if you want
-}
-
-# MulgaOS AWS-compatible endpoints
-# Set these environment variables or use terraform.tfvars:
-#   export AWS_ACCESS_KEY_ID=your_access_key
-#   export AWS_SECRET_ACCESS_KEY=your_secret_key
-#   export TF_VAR_mulgaos_endpoint=https://localhost:9999
-variable "mulgaos_endpoint" {
-  type        = string
-  default     = "https://localhost:9999"
-  description = "MulgaOS API endpoint (awsd gateway)"
-}
-
-provider "aws" {
-  region = var.region
-
-  # MulgaOS custom endpoints
-  endpoints {
-    ec2 = var.mulgaos_endpoint
-    s3  = var.mulgaos_endpoint
-    iam = var.mulgaos_endpoint
-    sts = var.mulgaos_endpoint
-  }
-
-  # Skip AWS credential validation (MulgaOS handles auth)
-  skip_credentials_validation = true
-  skip_metadata_api_check     = true
-  skip_requesting_account_id  = true
-
-  # For local development, skip region validation
-  skip_region_validation = true
-}
-
-data "aws_availability_zones" "available" {
-  state = "available"
-}
-
-# Debian 13 AMI
-# For MulgaOS, need to import a local AMI first
-data "aws_ami" "debian13" {
-  most_recent = true
-  owners      = ["000000000000"] # Spinifex system images (on AWS use "136693071363" for Debian official)
-
-  filter {
-    name   = "name"
-    values = ["*debian-13*"]
-  }
-
-  filter {
-    name   = "virtualization-type"
-    values = ["hvm"]
-  }
-
-  filter {
-    name   = "root-device-type"
-    values = ["ebs"]
-  }
-}
-
-# Keypair: spinifex-test
-resource "tls_private_key" "hive_test" {
-  algorithm = "ED25519"
-}
-
-resource "aws_key_pair" "hive_test" {
-  key_name   = "spinifex-test"
-  public_key = tls_private_key.hive_test.public_key_openssh
-}
-
-# Save private key locally for SSH
-resource "local_file" "hive_test_pem" {
-  filename        = "${path.module}/spinifex-test.pem"
-  content         = tls_private_key.hive_test.private_key_openssh
-  file_permission = "0600"
-}
-
-# 1: VPC 10.1.0.0/16
-resource "aws_vpc" "main" {
-  cidr_block           = "10.1.0.0/16"
-  enable_dns_hostnames = true
-  enable_dns_support   = true
-
-  tags = {
-    Name = "spinifex-test-vpc"
-  }
-}
-
-# Internet access for a public subnet
-resource "aws_internet_gateway" "igw" {
-  vpc_id = aws_vpc.main.id
-
-  tags = {
-    Name = "spinifex-test-igw"
-  }
-}
-
-# 2: Subnet 10.1.2.0/24 (public)
-resource "aws_subnet" "public" {
-  vpc_id                  = aws_vpc.main.id
-  cidr_block              = "10.1.2.0/24"
-  availability_zone       = data.aws_availability_zones.available.names[0]
-  map_public_ip_on_launch = true
-
-  tags = {
-    Name = "spinifex-test-public-subnet"
-  }
-}
-
-resource "aws_route_table" "public" {
-  vpc_id = aws_vpc.main.id
-
-  route {
-    cidr_block = "0.0.0.0/0"
-    gateway_id = aws_internet_gateway.igw.id
-  }
-
-  tags = {
-    Name = "spinifex-test-public-rt"
-  }
-}
-
-resource "aws_route_table_association" "public" {
-  subnet_id      = aws_subnet.public.id
-  route_table_id = aws_route_table.public.id
-}
-
-# 3: Security group
-resource "aws_security_group" "web_ssh" {
-  name        = "spinifex-test-sg"
-  description = "Allow inbound 22,80,443; allow all outbound"
-  vpc_id      = aws_vpc.main.id
-
-  ingress {
-    description = "SSH"
-    from_port   = 22
-    to_port     = 22
-    protocol    = "tcp"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  ingress {
-    description = "HTTP"
-    from_port   = 80
-    to_port     = 80
-    protocol    = "tcp"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  ingress {
-    description = "HTTPS"
-    from_port   = 443
-    to_port     = 443
-    protocol    = "tcp"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  egress {
-    description = "All outbound"
-    from_port   = 0
-    to_port     = 0
-    protocol    = "-1"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  tags = {
-    Name = "spinifex-test-sg"
-  }
-}
-
-resource "aws_instance" "vm" {
-  count         = 3
-  ami           = data.aws_ami.debian13.id
-  instance_type = "t3.small"
-
-  subnet_id              = aws_subnet.public.id
-  vpc_security_group_ids = [aws_security_group.web_ssh.id]
-  key_name               = aws_key_pair.hive_test.key_name
-
-  associate_public_ip_address = true
-
-  tags = {
-    Name = "spinifex-test-vm-${count.index + 1}"
-  }
-}
-
-output "instance_public_ips" {
-  value = aws_instance.vm[*].public_ip
-}
-
-output "ssh_commands" {
-  value = [
-    for ip in aws_instance.vm[*].public_ip :
-    "ssh -i spinifex-test.pem admin@${ip}"
-  ]
-}

scripts/iac/configure-cluster.sh

@@ -1,204 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-# Configure a provisioned Spinifex cluster: clone repo, build, form cluster, start services.
-# Usage: configure-cluster.sh <cluster_name> <state_file>
-
-CLUSTER_NAME="${1:?Usage: configure-cluster.sh <cluster_name> <state_file>}"
-STATE_FILE="${2:?Usage: configure-cluster.sh <cluster_name> <state_file>}"
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-TOFU_DIR="$SCRIPT_DIR/proxmox"
-
-SSH_OPTS="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10"
-
-REGION="ap-southeast-2"
-AZ="ap-southeast-2a"
-
-inventory=$(tofu -chdir="$TOFU_DIR" output -json -state="$STATE_FILE" inventory)
-NODE_COUNT=$(echo "$inventory" | jq -r '.node_count')
-SSH_KEY=$(eval echo "$(echo "$inventory" | jq -r '.ssh_key_path')")
-SSH_USER=$(echo "$inventory" | jq -r '.ssh_user')
-
-# Build arrays of node IPs
-declare -a MGMT_IPS
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    MGMT_IPS+=("$(echo "$inventory" | jq -r ".nodes[$i].management")")
-done
-
-remote() {
-    local ip="$1"
-    shift
-    ssh -i "$SSH_KEY" $SSH_OPTS "$SSH_USER@$ip" "$@"
-}
-
-remote_bg() {
-    local ip="$1"
-    shift
-    ssh -i "$SSH_KEY" $SSH_OPTS "$SSH_USER@$ip" "$@" &
-}
-
-echo "==> Cluster '$CLUSTER_NAME': $NODE_COUNT nodes"
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    echo "    node$((i + 1)): ${MGMT_IPS[$i]}"
-done
-
-# --- Step 1: Wait for cloud-init ---
-echo ""
-echo "==> Waiting for cloud-init to complete on all nodes..."
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    ip="${MGMT_IPS[$i]}"
-    printf "    node%d (%s): " "$((i + 1))" "$ip"
-    attempts=0
-    while ! remote "$ip" "test -f /tmp/vendor-cloud-init-done" 2>/dev/null; do
-        attempts=$((attempts + 1))
-        if [ $attempts -ge 60 ]; then
-            echo "TIMEOUT"
-            echo "Error: cloud-init did not complete on $ip after 5 minutes"
-            exit 1
-        fi
-        sleep 5
-    done
-    echo "done"
-done
-
-# --- Step 2: Clone repo and build on all nodes ---
-echo ""
-echo "==> Cloning Spinifex repo and building on all nodes..."
-PIDS=()
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    ip="${MGMT_IPS[$i]}"
-    (
-        echo "    node$((i + 1)) ($ip): cloning..."
-        remote "$ip" "mkdir -p ~/Development/mulga && cd ~/Development/mulga && git clone https://github.com/mulgadc/spinifex.git && export PATH=\$PATH:/usr/local/go/bin && sudo make -C spinifex quickinstall"
-        echo "    node$((i + 1)) ($ip): clone-deps + dev-setup..."
-        remote "$ip" "export PATH=\$PATH:/usr/local/go/bin && cd ~/Development/mulga/spinifex && ./scripts/clone-deps.sh && ./scripts/dev-setup.sh"
-        echo "    node$((i + 1)) ($ip): build complete"
-    ) &
-    PIDS+=($!)
-done
-
-for pid in "${PIDS[@]}"; do
-    wait "$pid"
-done
-echo "    All nodes built successfully"
-
-# --- Step 3: Form cluster ---
-echo ""
-echo "==> Forming cluster..."
-
-# Build comma-separated IP lists
-ALL_MGMT_IPS=$(IFS=,; echo "${MGMT_IPS[*]}")
-NODE1_IP="${MGMT_IPS[0]}"
-
-# Node 1: init
-echo "    Starting init on node1 ($NODE1_IP)..."
-remote_bg "$NODE1_IP" "export PATH=\$PATH:/usr/local/go/bin && cd ~/Development/mulga/spinifex && ./bin/spx admin init \
-    --node node1 \
-    --bind $NODE1_IP \
-    --cluster-bind $NODE1_IP \
-    --cluster-routes $NODE1_IP:4248 \
-    --predastore-nodes $ALL_MGMT_IPS \
-    --port 4432 \
-    --nodes $NODE_COUNT \
-    --region $REGION \
-    --az $AZ \
-    --spinifex-dir ~/spinifex/ \
-    --config-dir ~/spinifex/config/"
-INIT_PID=$!
-
-# Wait for formation server to be ready
-echo "    Waiting for formation server on $NODE1_IP..."
-attempts=0
-while ! curl -sk --connect-timeout 3 "https://$NODE1_IP:4432/formation/health" >/dev/null 2>&1; do
-    attempts=$((attempts + 1))
-    if [ $attempts -ge 60 ]; then
-        echo "Error: Formation server did not start on $NODE1_IP after 5 minutes"
-        kill $INIT_PID 2>/dev/null || true
-        exit 1
-    fi
-    sleep 5
-done
-echo "    Formation server ready"
-
-# Read join token from init node
-JOIN_TOKEN=$(remote "$NODE1_IP" "cat ~/spinifex/config/join-token")
-if [ -z "$JOIN_TOKEN" ]; then
-    echo "Error: Failed to read join token from $NODE1_IP"
-    exit 1
-fi
-echo "    Join token loaded from node1"
-
-# Nodes 2..N: join
-if [ "$NODE_COUNT" -gt 1 ]; then
-    JOIN_PIDS=()
-    for i in $(seq 1 $((NODE_COUNT - 1))); do
-        ip="${MGMT_IPS[$i]}"
-        node_num=$((i + 1))
-        echo "    Starting join on node$node_num ($ip)..."
-        remote_bg "$ip" "export PATH=\$PATH:/usr/local/go/bin && cd ~/Development/mulga/spinifex && ./bin/spx admin join \
-            --node node$node_num \
-            --bind $ip \
-            --cluster-bind $ip \
-            --cluster-routes $NODE1_IP:4248 \
-            --host $NODE1_IP:4432 \
-            --token $JOIN_TOKEN \
-            --data-dir ~/spinifex/ \
-            --config-dir ~/spinifex/config/ \
-            --region $REGION \
-            --az $AZ"
-        JOIN_PIDS+=($!)
-    done
-
-    for pid in "${JOIN_PIDS[@]}"; do
-        wait "$pid"
-    done
-fi
-
-# Wait for init process to complete
-wait $INIT_PID
-echo "    Cluster formation complete"
-
-# --- Step 4: Start services ---
-echo ""
-echo "==> Starting services on all nodes..."
-PIDS=()
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    ip="${MGMT_IPS[$i]}"
-    (
-        echo "    node$((i + 1)) ($ip): starting services..."
-        remote "$ip" "export PATH=\$PATH:/usr/local/go/bin && cd ~/Development/mulga/spinifex && ./scripts/start-dev.sh"
-        echo "    node$((i + 1)) ($ip): services started"
-    ) &
-    PIDS+=($!)
-done
-
-for pid in "${PIDS[@]}"; do
-    wait "$pid"
-done
-
-# --- Step 5: Health check ---
-echo ""
-echo "==> Waiting for all nodes to become healthy..."
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    ip="${MGMT_IPS[$i]}"
-    printf "    node%d (%s): " "$((i + 1))" "$ip"
-    attempts=0
-    while true; do
-        health=$(curl -sk --connect-timeout 3 "https://$ip:4432/health" 2>/dev/null || echo "")
-        if [ -n "$health" ]; then
-            echo "healthy"
-            break
-        fi
-        attempts=$((attempts + 1))
-        if [ $attempts -ge 30 ]; then
-            echo "TIMEOUT"
-            echo "Warning: node$((i + 1)) did not become healthy after 150 seconds"
-            break
-        fi
-        sleep 5
-    done
-done
-
-echo ""
-echo "==> Cluster '$CLUSTER_NAME' configured successfully"

scripts/iac/proxmox-template/.env.example

@@ -1,60 +0,0 @@
-# Proxmox Template Builder — Environment Configuration
-#
-# Copy to .env and fill in your values:
-#   cp .env.example .env
-#
-# Then source it before running tofu:
-#   source .env && tofu plan
-
-# --- Proxmox API ---
-
-# API token for Proxmox VE (read by the provider directly)
-export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN_HERE"
-
-# Proxmox VE API endpoint URL
-export TF_VAR_proxmox_endpoint="https://pve-host.example.com:8006/"
-
-# --- Proxmox Node ---
-# Build the template on ONE node. After creation, use vzdump to copy to others.
-#
-#   name    - Proxmox node name (as shown in the Proxmox UI)
-#   address - SSH-reachable hostname or IP of the Proxmox host
-
-export TF_VAR_proxmox_node_name="node1"
-export TF_VAR_proxmox_node_address="node1.example.com"
-
-# --- SSH Access ---
-
-# SSH username on Proxmox hosts (used by provider for file uploads + qm commands)
-export TF_VAR_proxmox_ssh_username="terraform"
-
-# Path to SSH private key for Proxmox host access
-export TF_VAR_ssh_private_key_path="~/.ssh/proxmox-host-tf"
-
-# Path to SSH private key for VM access (must be the pair of the public key below)
-export TF_VAR_vm_ssh_private_key_path="~/.ssh/cloud-init"
-
-# Path to SSH public key injected into VMs via cloud-init (for tf-user)
-export TF_VAR_ssh_public_key_path="~/.ssh/cloud-init.pub"
-
-# --- Template ---
-
-# VM ID for the template (pick a high number to avoid conflicts)
-export TF_VAR_template_vmid=9000
-
-# Short tag for the base image (used in Proxmox template tags)
-export TF_VAR_base_image_tag="debian-12"
-
-# Storage and network for the build node
-export TF_VAR_datastore_id="local-lvm"
-export TF_VAR_bridge="vmbr0"
-
-# --- Optional ---
-
-# Base image — change this when testing a newer Debian/Ubuntu version:
-# export TF_VAR_base_image="local:iso/debian-13-genericcloud-amd64.img"
-# export TF_VAR_base_image_tag="debian-13"
-# export TF_VAR_template_vmid=9010
-
-# Clone a specific branch instead of main:
-# export TF_VAR_git_branch="feature/my-branch"

scripts/iac/proxmox-template/.gitignore

@@ -1,6 +0,0 @@
-.env
-.terraform/
-.terraform.lock.hcl
-*.tfstate
-*.tfstate.backup
-*.tfplan

scripts/iac/proxmox-template/README.md

@@ -1,124 +0,0 @@
-# Spinifex Node Template Builder
-
-Builds a Proxmox VM template with all Spinifex dependencies pre-installed. Cloning from this template reduces cluster spin-up from ~15 minutes to ~1-2 minutes (`git pull` + `make build`).
-
-## What's in the template
-
-| Component | Details |
-|---|---|
-| System packages | QEMU, nbdkit, libvirt, gcc, jq, curl, etc. (`make install-system`) |
-| Go | 1.26.4 (`make install-go`) |
-| AWS CLI | v2 (`make install-aws`) |
-| Repositories | spinifex, viperblock, predastore (with `go.work` configured) |
-| Module cache | `go mod download` completed for all repos |
-| Cloud images | Ubuntu 26.04 for nested VMs (`~/images/ubuntu-26.04.img`) |
-| Tuning | sysctl (rmem_max/wmem_max), kvm group, ufw disabled |
-| Generalized | Clean machine-id, SSH host keys, cloud-init — each clone boots fresh |
-
-## Quick Start
-
-```bash
-cd scripts/iac/proxmox-template
-
-# 1. Configure
-cp .env.example .env
-# Edit .env — set your Proxmox endpoint, API token, node, and SSH key paths.
-
-# 2. Set API token (not stored in .env for security)
-export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN"
-
-# 3. Build
-source .env
-tofu init
-tofu plan
-tofu apply
-```
-
-Build takes ~10-15 minutes (apt packages, Go, cloud image download). When done, you'll have a template on the build node at the configured VMID (default 9000).
-
-## Distribute to Other Nodes
-
-After `tofu apply`, the output prints ready-to-run commands. Example for a 3-node cluster:
-
-```bash
-# Export from build node
-ssh -i ~/.ssh/proxmox-host-tf terraform@<BUILD_NODE> \
-  'vzdump 9000 --dumpdir /tmp --mode stop --compress zstd'
-
-# Copy to other nodes
-scp -3 -i ~/.ssh/proxmox-host-tf \
-  terraform@<BUILD_NODE>:/tmp/vzdump-qemu-9000-*.vma.zst \
-  terraform@<TARGET_NODE>:/tmp/
-
-# Restore on each target node (use a unique VMID and the node's datastore)
-ssh -i ~/.ssh/proxmox-host-tf terraform@<TARGET_NODE> \
-  'qmrestore /tmp/vzdump-qemu-9000-*.vma.zst <NEW_VMID> --storage <DATASTORE> && qm template <NEW_VMID>'
-```
-
-After distributing, each node has its own copy of the template with a unique VMID.
-
-## Using the Template
-
-Update `scripts/iac/proxmox/vms.tf` to clone from the template instead of booting from a raw cloud image. Or clone manually from the Proxmox UI.
-
-On each cloned VM after boot:
-
-```bash
-# Pull latest code
-cd ~/Development/mulga/spinifex && git pull
-cd ~/Development/mulga/viperblock && git pull
-cd ~/Development/mulga/predastore && git pull
-
-# Build (~30s with cached modules)
-cd ~/Development/mulga/spinifex && export PATH=/usr/local/go/bin:$PATH && make build
-```
-
-## Testing a Different Base Image
-
-Upload the new image to Proxmox, then build a separate template:
-
-```bash
-# Debian 13 example
-export TF_VAR_base_image="local:iso/debian-13-genericcloud-amd64.img"
-export TF_VAR_base_image_tag="debian-13"
-export TF_VAR_template_vmid=9010   # Keep the debian-12 template at 9000
-
-source .env
-tofu apply
-```
-
-Templates are tagged in the Proxmox UI with the base image name (`debian-12`, `debian-13`, etc.) so you can tell them apart.
-
-## Rebuilding
-
-When dependencies change (new Go version, new system packages):
-
-```bash
-source .env
-tofu destroy     # Remove the old template
-tofu apply       # Build fresh
-# Then re-distribute with vzdump
-```
-
-## SSH Key Setup
-
-This template uses **two separate key pairs** (matching the `scripts/iac/proxmox/` setup):
-
-| Key | Used for | Variable |
-|---|---|---|
-| Proxmox host key | SSH to Proxmox hosts (provider, `qm template` command) | `ssh_private_key_path` |
-| VM / cloud-init key | SSH to VMs (cloud-init injection, provisioning) | `vm_ssh_private_key_path` + `ssh_public_key_path` |
-
-If you use the same key for both, set all three variables to the same key pair.
-
-## Files
-
-| File | Description |
-|------|-------------|
-| `main.tf` | Proxmox provider configuration |
-| `variables.tf` | All configurable variables with defaults |
-| `template-builder.tf` | VM creation, provisioning, template conversion |
-| `outputs.tf` | Template info + vzdump/restore commands |
-| `scripts/provision.sh` | Install deps, clone repos, cache modules, generalize |
-| `.env.example` | Environment variable template |
-| `.gitignore` | Ignores .env, state files, provider plugins |

scripts/iac/proxmox-template/main.tf

@@ -1,27 +0,0 @@
-terraform {
-  required_version = ">= 1.6.0"
-
-  required_providers {
-    proxmox = {
-      source  = "bpg/proxmox"
-      version = "0.52.0"
-    }
-  }
-}
-
-# API token is read from PROXMOX_VE_API_TOKEN env var automatically
-provider "proxmox" {
-  endpoint = var.proxmox_endpoint
-  insecure = true
-
-  ssh {
-    agent       = false
-    username    = var.proxmox_ssh_username
-    private_key = file(pathexpand(var.ssh_private_key_path))
-
-    node {
-      name    = var.proxmox_node_name
-      address = var.proxmox_node_address
-    }
-  }
-}

scripts/iac/proxmox-template/outputs.tf

@@ -1,43 +0,0 @@
-output "template_vmid" {
-  description = "VM ID of the created template"
-  value       = proxmox_virtual_environment_vm.builder.vm_id
-}
-
-output "template_node" {
-  description = "Proxmox node where the template lives"
-  value       = var.proxmox_node_name
-}
-
-output "base_image_tag" {
-  description = "Base image tag for this template"
-  value       = var.base_image_tag
-}
-
-# Ready-to-run command: export the template for distribution to other Proxmox nodes
-output "vzdump_export_command" {
-  description = "Run this to export the template (vzdump) for copying to other nodes"
-  value       = <<-EOT
-    ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@${var.proxmox_node_address} \
-      'vzdump ${var.template_vmid} --dumpdir /tmp --mode stop --compress zstd'
-  EOT
-}
-
-# After exporting, copy to each target node and restore
-output "distribute_commands" {
-  description = "Copy and restore the template on other Proxmox nodes"
-  value       = <<-EOT
-    # 1. Export from build node (${var.proxmox_node_name}):
-    ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@${var.proxmox_node_address} \
-      'vzdump ${var.template_vmid} --dumpdir /tmp --mode stop --compress zstd'
-
-    # 2. Copy to each target node:
-    scp -3 \
-      -i ${var.ssh_private_key_path} \
-      ${var.proxmox_ssh_username}@${var.proxmox_node_address}:/tmp/vzdump-qemu-${var.template_vmid}-*.vma.zst \
-      ${var.proxmox_ssh_username}@<TARGET_NODE_ADDRESS>:/tmp/
-
-    # 3. On each target node — restore and convert to template:
-    ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@<TARGET_NODE_ADDRESS> \
-      'qmrestore /tmp/vzdump-qemu-${var.template_vmid}-*.vma.zst <NEW_VMID> --storage <DATASTORE> && qm template <NEW_VMID>'
-  EOT
-}

scripts/iac/proxmox-template/scripts/provision.sh

@@ -1,176 +0,0 @@
-#!/bin/bash
-# provision.sh — Installs all Spinifex dependencies, clones repos, caches Go modules,
-# and generalizes the VM for use as a Proxmox template.
-#
-# Runs as tf-user via Terraform remote-exec. Uses sudo for system operations.
-# Environment: GIT_BRANCH (default: dev)
-
-set -euo pipefail
-
-GIT_BRANCH="${GIT_BRANCH:-dev}"
-WORK_DIR="$HOME/Development/mulga"
-
-echo "========================================"
-echo "  Spinifex Template Provisioning"
-echo "  Branch: ${GIT_BRANCH}"
-echo "  Arch:   $(uname -m)"
-echo "========================================"
-
-# ── 1. Wait for cloud-init ──────────────────────────────────────────────────
-echo ""
-echo "=== [1/8] Waiting for cloud-init to complete ==="
-cloud-init status --wait
-
-# ── 2. Clone repositories ───────────────────────────────────────────────────
-echo ""
-echo "=== [2/8] Cloning repositories ==="
-mkdir -p "$WORK_DIR"
-cd "$WORK_DIR"
-
-for repo in spinifex viperblock predastore; do
-    if [ ! -d "$repo" ]; then
-        echo "  Cloning $repo (branch: $GIT_BRANCH)..."
-        git clone -b "$GIT_BRANCH" "https://github.com/mulgadc/${repo}.git"
-    else
-        echo "  $repo already exists, skipping"
-    fi
-done
-
-# ── 3. Install system dependencies via Makefile ─────────────────────────────
-echo ""
-echo "=== [3/8] Installing system dependencies ==="
-cd "$WORK_DIR/spinifex"
-sudo make quickinstall
-
-# Ensure Go is in PATH for the rest of the script
-export PATH="/usr/local/go/bin:$HOME/go/bin:$PATH"
-echo "  Go version: $(go version)"
-echo "  AWS CLI:    $(aws --version)"
-
-# ── 4. Set up go.work ───────────────────────────────────────────────────────
-echo ""
-echo "=== [4/8] Creating go.work ==="
-
-# Detect Go version from go.mod
-GO_VERSION=$(grep '^go ' "$WORK_DIR/spinifex/go.mod" | awk '{print $2}')
-echo "  Using Go version from go.mod: ${GO_VERSION}"
-
-cat > "$WORK_DIR/spinifex/go.work" << GOWORK
-go ${GO_VERSION}
-
-use (
-    .
-    ../viperblock
-    ../viperblock/nbd
-    ../viperblock/nbd/libguestfs.org/nbdkit
-    ../predastore
-)
-GOWORK
-
-echo "  Created $WORK_DIR/spinifex/go.work"
-
-# ── 5. Download Go modules ──────────────────────────────────────────────────
-echo ""
-echo "=== [5/8] Downloading Go modules ==="
-cd "$WORK_DIR/spinifex"    && go mod download && echo "  spinifex: done"
-cd "$WORK_DIR/viperblock"  && go mod download && echo "  viperblock: done"
-cd "$WORK_DIR/predastore"  && go mod download && echo "  predastore: done"
-
-# Sub-modules
-if [ -f "$WORK_DIR/viperblock/nbd/go.mod" ]; then
-    cd "$WORK_DIR/viperblock/nbd" && go mod download && echo "  viperblock/nbd: done"
-fi
-if [ -f "$WORK_DIR/viperblock/nbd/libguestfs.org/nbdkit/go.mod" ]; then
-    cd "$WORK_DIR/viperblock/nbd/libguestfs.org/nbdkit" && go mod download && echo "  viperblock/nbd/nbdkit: done"
-fi
-
-# ── 6. Pre-download cloud images for nested VMs ─────────────────────────────
-echo ""
-echo "=== [6/8] Pre-downloading cloud images ==="
-mkdir -p "$HOME/images"
-
-ARCH=$(dpkg --print-architecture)
-IMAGE_URL=""
-if [ "$ARCH" = "amd64" ]; then
-    IMAGE_URL="https://cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-amd64.img"
-elif [ "$ARCH" = "arm64" ]; then
-    IMAGE_URL="https://cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-arm64.img"
-fi
-
-if [ -n "$IMAGE_URL" ]; then
-    echo "  Downloading Ubuntu 26.04 cloud image ($ARCH)..."
-    curl -L -o "$HOME/images/ubuntu-26.04.img" "$IMAGE_URL"
-    echo "  Saved to ~/images/ubuntu-26.04.img"
-else
-    echo "  Unknown arch ($ARCH), skipping cloud image download"
-fi
-
-# ── 7. System tuning ────────────────────────────────────────────────────────
-echo ""
-echo "=== [7/8] Applying system tuning ==="
-
-# Persistent sysctl for NATS / high-throughput networking
-sudo tee /etc/sysctl.d/99-spinifex.conf > /dev/null << 'SYSCTL'
-net.core.rmem_max=4194304
-net.core.wmem_max=4194304
-SYSCTL
-echo "  sysctl: net.core.rmem_max/wmem_max = 4MB"
-
-# Add user to kvm group for QEMU access
-sudo usermod -aG kvm "$(whoami)"
-echo "  Added $(whoami) to kvm group"
-
-# Disable ufw if present
-if command -v ufw >/dev/null 2>&1; then
-    sudo ufw disable 2>/dev/null || true
-    echo "  ufw disabled"
-fi
-
-# ── 8. Generalize for templating ────────────────────────────────────────────
-echo ""
-echo "=== [8/8] Cleaning up for template ==="
-
-# Apt caches
-sudo apt-get clean
-sudo rm -rf /var/lib/apt/lists/*
-echo "  Cleared apt cache"
-
-# Logs
-sudo find /var/log -type f -name "*.log" -exec truncate -s 0 {} \; 2>/dev/null || true
-sudo rm -rf /var/log/journal/*
-echo "  Truncated logs"
-
-# Temp files
-sudo rm -rf /tmp/* /var/tmp/*
-
-# Machine ID — systemd regenerates on next boot.
-# This is critical: without this, all clones get the same DHCP client ID → IP conflicts.
-sudo truncate -s 0 /etc/machine-id
-sudo rm -f /var/lib/dbus/machine-id
-echo "  Cleared machine-id (will regenerate on boot)"
-
-# SSH host keys — regenerated on boot by cloud-init or systemd
-sudo rm -f /etc/ssh/ssh_host_*
-echo "  Removed SSH host keys (will regenerate on boot)"
-
-# Cloud-init state — forces cloud-init to run again on next boot.
-# This ensures each clone gets its own hostname, SSH keys, user config, etc.
-sudo cloud-init clean
-echo "  Cleaned cloud-init state (will re-run on boot)"
-
-# Shell history
-history -c 2>/dev/null || true
-cat /dev/null > "$HOME/.bash_history" 2>/dev/null || true
-
-# Provisioning script itself
-rm -f /tmp/provision.sh
-
-# Sync filesystem
-sync
-
-echo ""
-echo "========================================"
-echo "  Template provisioning complete!"
-echo "  VM is ready to be shut down and"
-echo "  converted to a Proxmox template."
-echo "========================================"

scripts/iac/proxmox-template/template-builder.tf

@@ -1,130 +0,0 @@
-data "local_file" "ssh_public_key" {
-  filename = pathexpand(var.ssh_public_key_path)
-}
-
-# Minimal cloud-init: just enough to bootstrap SSH + guest agent.
-# Heavy provisioning is done via remote-exec after boot.
-resource "proxmox_virtual_environment_file" "cloud_config" {
-  content_type = "snippets"
-  datastore_id = "local"
-  node_name    = var.proxmox_node_name
-
-  source_raw {
-    # NOTE: #cloud-config MUST start at column 0 — cloud-init uses it as a magic header.
-    data = <<EOF
-#cloud-config
-package_update: true
-packages:
-  - qemu-guest-agent
-  - git
-  - make
-  - curl
-  - sudo
-  - ca-certificates
-runcmd:
-  - systemctl enable qemu-guest-agent
-  - systemctl start qemu-guest-agent
-EOF
-
-    file_name = "hive-template-cloud-config.yaml"
-  }
-}
-
-resource "proxmox_virtual_environment_vm" "builder" {
-  name      = "${var.template_name}-builder"
-  node_name = var.proxmox_node_name
-  vm_id     = var.template_vmid
-
-  tags = sort(["terraform", "hive-template", var.base_image_tag])
-
-  initialization {
-    datastore_id        = "local"
-    vendor_data_file_id = proxmox_virtual_environment_file.cloud_config.id
-
-    user_account {
-      username = var.vm_ssh_user
-      keys     = [trimspace(data.local_file.ssh_public_key.content)]
-    }
-
-    ip_config {
-      ipv4 {
-        address = "dhcp"
-      }
-    }
-  }
-
-  cpu {
-    cores = var.cpu_cores
-    type  = "host"
-  }
-
-  memory {
-    dedicated = var.memory_mb
-  }
-
-  agent {
-    enabled = true
-  }
-
-  network_device {
-    bridge = var.bridge
-  }
-
-  disk {
-    file_id      = var.base_image
-    datastore_id = var.datastore_id
-    interface    = "virtio0"
-    iothread     = true
-    discard      = "on"
-    size         = var.disk_size_gb
-  }
-
-  # SSH connection to the VM for provisioning.
-  # Uses the VM key (vm_ssh_private_key_path), NOT the Proxmox host key.
-  # Waits for guest agent to report an IP, then connects.
-  connection {
-    type        = "ssh"
-    user        = var.vm_ssh_user
-    private_key = file(pathexpand(var.vm_ssh_private_key_path))
-    host        = element([for addr in flatten(self.ipv4_addresses) : addr if !startswith(addr, "127.")], 0)
-    timeout     = "10m"
-  }
-
-  # Copy provisioning script to VM
-  provisioner "file" {
-    source      = "${path.module}/scripts/provision.sh"
-    destination = "/tmp/provision.sh"
-  }
-
-  # Run provisioning (installs deps, clones repos, caches modules, cleans up)
-  provisioner "remote-exec" {
-    inline = [
-      "chmod +x /tmp/provision.sh",
-      "GIT_BRANCH='${var.git_branch}' /tmp/provision.sh",
-    ]
-  }
-}
-
-# After provisioning completes, shut down the VM and convert to a Proxmox template.
-# This runs on the Proxmox host via SSH (not on the VM — it's being shut down).
-resource "null_resource" "convert_template" {
-  depends_on = [proxmox_virtual_environment_vm.builder]
-
-  triggers = {
-    builder_id = proxmox_virtual_environment_vm.builder.vm_id
-  }
-
-  provisioner "local-exec" {
-    command = <<-EOT
-      ssh -i '${pathexpand(var.ssh_private_key_path)}' \
-        -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
-        '${var.proxmox_ssh_username}@${var.proxmox_node_address}' \
-        "sudo qm shutdown ${proxmox_virtual_environment_vm.builder.vm_id} --timeout 60 && \
-         echo 'Waiting for VM to stop...' && \
-         while sudo qm status ${proxmox_virtual_environment_vm.builder.vm_id} | grep -q running; do sleep 2; done && \
-         echo 'Converting to template...' && \
-         sudo qm template ${proxmox_virtual_environment_vm.builder.vm_id} && \
-         echo 'Template ${var.template_vmid} created successfully'"
-    EOT
-  }
-}

scripts/iac/proxmox-template/variables.tf

@@ -1,117 +0,0 @@
-# --- Proxmox Connection ---
-
-variable "proxmox_endpoint" {
-  type        = string
-  description = "Proxmox VE API endpoint URL (e.g. https://pve.example.com:8006/)"
-}
-
-variable "proxmox_node_name" {
-  type        = string
-  description = "Proxmox node name to build the template on (as shown in the Proxmox UI)"
-}
-
-variable "proxmox_node_address" {
-  type        = string
-  description = "SSH-reachable hostname or IP of the Proxmox node"
-}
-
-variable "proxmox_ssh_username" {
-  type        = string
-  description = "SSH username for Proxmox host access"
-  default     = "terraform"
-}
-
-# --- SSH Keys ---
-# Proxmox host and VM typically use different key pairs.
-# - ssh_private_key_path  → authenticates to the Proxmox host (provider + qm commands)
-# - vm_ssh_private_key_path → authenticates to the VM for provisioning (remote-exec)
-# - ssh_public_key_path → injected into the VM via cloud-init
-
-variable "ssh_private_key_path" {
-  type        = string
-  description = "Path to SSH private key for Proxmox host access"
-}
-
-variable "vm_ssh_private_key_path" {
-  type        = string
-  description = "Path to SSH private key for VM access (must match ssh_public_key_path)"
-}
-
-variable "ssh_public_key_path" {
-  type        = string
-  description = "Path to SSH public key injected into the VM via cloud-init"
-}
-
-# --- Base Image ---
-
-variable "base_image" {
-  type        = string
-  description = "Proxmox image file ID for the base OS (e.g. local:iso/debian-12-genericcloud-amd64-20240211-1654.img)"
-  default     = "local:iso/debian-12-genericcloud-amd64-20240211-1654.img"
-}
-
-variable "base_image_tag" {
-  type        = string
-  description = "Short tag identifying the base image, used in template tags for tracking"
-  default     = "debian-12"
-}
-
-# --- Template Settings ---
-
-variable "template_name" {
-  type        = string
-  description = "Name for the resulting Proxmox template"
-  default     = "hive-node-template"
-}
-
-variable "template_vmid" {
-  type        = number
-  description = "VM ID for the template (pick a high number to avoid conflicts, e.g. 9000)"
-  default     = 9000
-}
-
-variable "vm_ssh_user" {
-  type        = string
-  description = "Username created in the VM via cloud-init"
-  default     = "tf-user"
-}
-
-# --- VM Resources ---
-
-variable "cpu_cores" {
-  type        = number
-  description = "CPU cores for the builder VM (more = faster builds)"
-  default     = 4
-}
-
-variable "memory_mb" {
-  type        = number
-  description = "Memory in MB for the builder VM"
-  default     = 8192
-}
-
-variable "disk_size_gb" {
-  type        = number
-  description = "Disk size in GB (needs room for OS + Go + modules + cloud images)"
-  default     = 32
-}
-
-variable "datastore_id" {
-  type        = string
-  description = "Proxmox storage for the VM disk"
-  default     = "local-lvm"
-}
-
-variable "bridge" {
-  type        = string
-  description = "Network bridge for the VM"
-  default     = "vmbr0"
-}
-
-# --- Git ---
-
-variable "git_branch" {
-  type        = string
-  description = "Git branch to clone for spinifex, viperblock, and predastore"
-  default     = "dev"
-}

scripts/iac/proxmox/.env.example

@@ -1,68 +0,0 @@
-# Proxmox IaC Environment Configuration
-# Copy this file to .env and fill in your values:
-#   cp .env.example .env
-#
-# Then source it before running tofu:
-#   source .env && tofu plan
-
-# --- Proxmox API ---
-
-# API token for Proxmox VE (read by the provider directly)
-export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN_HERE"
-
-# Proxmox VE API endpoint URL
-export TF_VAR_proxmox_endpoint="https://pve-host.example.com:8006/"
-
-# --- SSH Access ---
-
-# SSH username on Proxmox hosts (used by provider for file uploads)
-export TF_VAR_proxmox_ssh_username="terraform"
-
-# Path to SSH private key for Proxmox host access
-export TF_VAR_ssh_private_key_path="~/.ssh/proxmox-host-tf"
-
-# Path to SSH public key injected into VMs via cloud-init (for tf-user)
-export TF_VAR_ssh_public_key_path="~/.ssh/cloud-init.pub"
-
-# --- Proxmox Nodes ---
-# Physical Proxmox hosts where VMs are placed (round-robin).
-# You can define 1 or more nodes — VMs are distributed across them.
-#
-# Each node object:
-#   name          - Proxmox node name (as shown in the Proxmox UI)
-#   address       - SSH-reachable hostname or IP of the Proxmox host
-#   bridge        - Network bridge for VM interfaces (e.g. vmbr0, vmbr3)
-#   datastore_id  - Proxmox storage ID for VM disks (e.g. local, local-lvm)
-#   template_vmid - (optional) VMID of a pre-built template on this node.
-#                   Build with scripts/iac/proxmox-template/, then distribute
-#                   with vzdump. When set, VMs clone from the template (fast).
-#                   When omitted, VMs boot from os_image (slower).
-
-export TF_VAR_nodes='[
-  {
-    "name": "node1",
-    "address": "node1.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local"
-  },
-  {
-    "name": "node2",
-    "address": "node2.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local"
-  },
-  {
-    "name": "node3",
-    "address": "node3.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local"
-  }
-]'
-
-# Example with templates (VMIDs may differ per node after vzdump restore):
-#
-# export TF_VAR_nodes='[
-#   { "name": "node1", "address": "node1.example.com", "bridge": "vmbr0", "datastore_id": "local", "template_vmid": 9999 },
-#   { "name": "node2", "address": "node2.example.com", "bridge": "vmbr0", "datastore_id": "local-lvm", "template_vmid": 9997 },
-#   { "name": "node3", "address": "node3.example.com", "bridge": "vmbr0", "datastore_id": "ssd", "template_vmid": 9998 }
-# ]'

scripts/iac/proxmox/README.md

@@ -1,191 +0,0 @@
-# Proxmox IaC
-
-OpenTofu/Terraform templates for provisioning Spinifex development clusters on Proxmox VE.
-
-## Prerequisites
-
-- [OpenTofu](https://opentofu.org/) (or Terraform >= 1.6.0)
-- 1 or more Proxmox VE nodes with:
-  - A Terraform API token (`Datacenter > Permissions > API Tokens`)
-  - SSH access for a `terraform` user on each node
-  - Debian 13 cloud image uploaded to `local:iso/` on each node
-- SSH keypair for cloud-init VM access
-- A hardcoded Debian ISO is used, which needs to be pre-installed on each Proxmox host ("debian-13-genericcloud-amd64-20260518-2482.img") https://cloud.debian.org/images/cloud/trixie/20260518-2482/debian-13-genericcloud-amd64-20260518-2482.qcow2
-
-## Quick Start (spinifex-test.sh)
-
-The `spinifex-test.sh` wrapper script manages the full lifecycle: provision, configure, test, and destroy.
-
-```sh
-# Source environment
-source scripts/iac/proxmox/.env
-source ~/prox   # Proxmox API token
-
-# Provision a 3-node cluster
-./scripts/iac/spinifex-test.sh up my-cluster
-
-# Configure: clone repo, build, form cluster, start services
-./scripts/iac/spinifex-test.sh configure my-cluster
-
-# Run smoke tests
-./scripts/iac/spinifex-test.sh test my-cluster
-
-# Check cluster health
-./scripts/iac/spinifex-test.sh status my-cluster
-
-# SSH to a specific node
-./scripts/iac/spinifex-test.sh ssh my-cluster 2
-
-# Destroy when done
-./scripts/iac/spinifex-test.sh down my-cluster
-
-# Or run the full lifecycle (up → configure → test → down)
-./scripts/iac/spinifex-test.sh full my-cluster
-```
-
-### Options
-
-```
---node-count=N       Number of VMs (default: 3)
---memory-mb=N        Memory per VM in MB (default: 16384)
---cpu-cores=N        CPU cores per VM (default: 4)
---disk-size-gb=N     Disk size per VM in GB (default: 32)
-```
-
-Example with custom sizing:
-
-```sh
-./scripts/iac/spinifex-test.sh up my-cluster --node-count=5 --memory-mb=8192 --cpu-cores=2
-```
-
-Per-cluster state is stored in `scripts/iac/proxmox/clusters/<name>/terraform.tfstate`, allowing multiple clusters to coexist.
-
-## Manual Setup
-
-### 1. Configure environment
-
-```sh
-cp .env.example .env
-# Edit .env with your Proxmox endpoint, SSH keys, and node configuration
-```
-
-### 2. Set API token
-
-```sh
-export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN_SECRET"
-```
-
-### 3. Deploy
-
-```sh
-source .env
-tofu init
-tofu plan -var="cluster_name=dev1"
-tofu apply -var="cluster_name=dev1"
-```
-
-### Example `.env` file
-
-```sh
-# --- Proxmox API ---
-
-# API token - set manually via: export PROXMOX_VE_API_TOKEN="terraform@pve!provider=..."
-# Format: <user>@<realm>!<token-name>=<token-secret>
-
-# Proxmox VE API endpoint (the URL you use to access the web UI)
-export TF_VAR_proxmox_endpoint="https://pve1.lab.example.com:8006/"
-
-# --- SSH Access ---
-
-# SSH username on the Proxmox hosts (used by the provider for file uploads via SCP)
-export TF_VAR_proxmox_ssh_username="terraform"
-
-# Path to SSH private key that authenticates as the above user on each Proxmox host
-export TF_VAR_ssh_private_key_path="~/.ssh/proxmox-tf"
-
-# Path to SSH public key injected into VMs via cloud-init (the tf-user account)
-export TF_VAR_ssh_public_key_path="~/.ssh/proxmox-tf-cloudinit.pub"
-
-# --- Proxmox Nodes ---
-#
-# Physical Proxmox hosts where VMs are placed (round-robin).
-# Each node object:
-#   name         - Proxmox node name exactly as shown in the Proxmox UI sidebar
-#   address      - SSH-reachable hostname or IP of the Proxmox host
-#   bridge       - Linux bridge for VM network interfaces (check Proxmox UI > Node > Network)
-#   datastore_id - Proxmox storage pool for VM disks (check Proxmox UI > Node > Storage)
-
-export TF_VAR_nodes='[
-  {
-    "name": "pve1",
-    "address": "pve1.lab.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local-lvm"
-  },
-  {
-    "name": "pve2",
-    "address": "pve2.lab.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local-lvm"
-  },
-  {
-    "name": "pve3",
-    "address": "pve3.lab.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local-lvm"
-  }
-]'
-```
-
-### Environment variable reference
-
-| Variable | Required | Description |
-|---|---|---|
-| `PROXMOX_VE_API_TOKEN` | Yes | Proxmox API token (read by provider directly) |
-| `TF_VAR_proxmox_endpoint` | Yes | Proxmox VE API URL (e.g. `https://host:8006/`) |
-| `TF_VAR_proxmox_ssh_username` | No | SSH user on Proxmox hosts (default: `terraform`) |
-| `TF_VAR_ssh_private_key_path` | Yes | SSH private key for Proxmox host access |
-| `TF_VAR_ssh_public_key_path` | Yes | SSH public key injected into VMs via cloud-init |
-| `TF_VAR_nodes` | Yes | JSON array of Proxmox node objects |
-
-Each node object in `TF_VAR_nodes`:
-
-| Field | Example | Description |
-|---|---|---|
-| `name` | `pve1` | Proxmox node name (as shown in UI sidebar) |
-| `address` | `pve1.lab.example.com` | SSH-reachable hostname or IP |
-| `bridge` | `vmbr0` | Network bridge for VM interfaces |
-| `datastore_id` | `local-lvm` | Storage pool for VM disks |
-
-### Cluster variables (passed via `-var` or `TF_VAR_`)
-
-| Variable | Default | Description |
-|---|---|---|
-| `cluster_name` | (required) | Cluster name — used in VM names and tags |
-| `node_count` | `3` | Number of VMs to create (1-10) |
-| `cpu_cores` | `4` | CPU cores per VM |
-| `memory_mb` | `16384` | Memory per VM in MB |
-| `disk_size_gb` | `32` | Disk size per VM in GB |
-| `os_image` | `local:iso/debian-12-...` | Proxmox image for boot disk |
-
-## Destroy
-
-```sh
-source .env
-tofu destroy -var="cluster_name=dev1"
-```
-
-## SSH access
-
-Connect to provisioned VMs using the cloud-init public key:
-
-```sh
-ssh -i ~/.ssh/your-cloud-init-key tf-user@<VM_IP>
-```
-
-## Known issues
-
-- `~/spinifex/config/spinifex.toml` - Does not add node1, node2, node3 from config automatically
-- `~/spinifex/config/predastore/predastore.toml` - Uses previous static local node config, needs to use the IPs for each node in the cluster
-- On multi-node deployments, NATS on the primary can timeout waiting for other nodes to start, causing a race condition where NATS fails and all dependent services fail
-- When adding 3 nodes, `nats.conf` is not updated with the 3rd node. Each node's cluster name is hardcoded to `C1` instead of per-node names

scripts/iac/proxmox/cloud-config.tf

@@ -1,48 +0,0 @@
-data "local_file" "ssh_public_key" {
-  filename = pathexpand(var.ssh_public_key_path)
-}
-
-# When cloning from a template, packages are pre-installed — skip apt update/upgrade.
-locals {
-  cloud_config_template = <<-EOF
-    #cloud-config
-    runcmd:
-      - timedatectl set-timezone America/Los_Angeles
-      - systemctl enable qemu-guest-agent
-      - systemctl start qemu-guest-agent
-      - echo "done" > /tmp/vendor-cloud-init-done
-  EOF
-
-  cloud_config_image = <<-EOF
-    #cloud-config
-    package_update: true
-    package_upgrade: true
-    packages:
-      - qemu-guest-agent
-      - net-tools
-      - git
-      - make
-    runcmd:
-      - timedatectl set-timezone America/Los_Angeles
-      - systemctl enable qemu-guest-agent
-      - systemctl start qemu-guest-agent
-      - ufw disable
-      - sysctl -w net.core.rmem_max=4194304
-      - sysctl -w net.core.wmem_max=4194304
-      - usermod -aG kvm tf-user
-      - echo "done" > /tmp/vendor-cloud-init-done
-  EOF
-}
-
-resource "proxmox_virtual_environment_file" "cloud_config" {
-  count = var.node_count
-
-  content_type = "snippets"
-  datastore_id = "local"
-  node_name    = var.nodes[count.index % length(var.nodes)].name
-
-  source_raw {
-    data      = var.nodes[count.index % length(var.nodes)].template_vmid != null ? local.cloud_config_template : local.cloud_config_image
-    file_name = "${var.cluster_name}-${count.index + 1}-cloud-config.yaml"
-  }
-}

scripts/iac/proxmox/files.tf

@@ -1,17 +0,0 @@
-# Files already exist, skip
-
-#resource "proxmox_virtual_environment_download_file" "debian_cloud_image_neon" {
-#  content_type = "iso"
-#  datastore_id = "local"
-#  node_name    = "neon"
-#  url          = "https://cdimage.debian.org/images/cloud/bookworm/20240211-1654/debian-12-genericcloud-amd64-20240211-1654.qcow2"
-#  file_name    = "debian-12-genericcloud-amd64-20240211-1654.img"
-#}
-
-#resource "proxmox_virtual_environment_download_file" "debian_cloud_image_radon" {
-#  content_type = "iso"
-#  datastore_id = "local"
-#  node_name    = "radon"
-#  url          = "https://cdimage.debian.org/images/cloud/bookworm/20240211-1654/debian-12-genericcloud-amd64-20240211-1654.qcow2"
-#  file_name    = "debian-12-genericcloud-amd64-20240211-1654.img"
-#}

scripts/iac/proxmox/main.tf

@@ -1,28 +0,0 @@
-terraform {
-  required_providers {
-    proxmox = {
-      source  = "bpg/proxmox"
-      version = "0.52.0"
-    }
-  }
-}
-
-# API token is read from PROXMOX_VE_API_TOKEN env var automatically
-provider "proxmox" {
-  endpoint = var.proxmox_endpoint
-  insecure = true
-
-  ssh {
-    agent       = false
-    username    = var.proxmox_ssh_username
-    private_key = file(pathexpand(var.ssh_private_key_path))
-
-    dynamic "node" {
-      for_each = var.nodes
-      content {
-        name    = node.value.name
-        address = node.value.address
-      }
-    }
-  }
-}