release: update go.mod for v1.8.0

* feat: join nodegroup workers via cluster NLB endpoint

Under the Set-B managed-CP-VPC topology the control plane runs in the
system-account CP VPC while nodegroup workers launch in the customer
VPC. The agent user-data set K3S_URL to the in-VPC control-plane ENI IP
(:6443), which sits in a different, unpeered VPC and is unreachable, so
the join path was dead.

Point K3S_URL at the published cluster NLB endpoint (meta.Endpoint,
https://<frontend-ip>:443, the :443->:6443 listener kubectl already
uses). Guard CreateNodegroup on a non-empty endpoint.

mulga-siv-298

* feat: gate nodegroup ACTIVE on workers registering Ready

launchNodegroupInfra flipped the nodegroup to ACTIVE the moment
RunInstances returned, so a worker that booted but never joined left the
nodegroup falsely ACTIVE with no usable capacity.

Gate ACTIVE on the cluster's Ready-node count (meta.NodeCount, refreshed
by the cluster reconciler from the CP NATS state report — Ready nodes
only) rising by the desired worker count over the create-time baseline.
On timeout the nodegroup is marked CREATE_FAILED with its instance IDs
retained for the reclaim path. Mirrors the CP reconciler's
healthy-observe gating. Bounded by injectable timeout/poll knobs.

mulga-siv-299

* feat(eks): private endpoint access via cross-account Set A ENI

Make EndpointPrivateAccess real: in-VPC workers and kubectl reach the
control plane through a customer-VPC (Set A) resident ENI on the cluster
NLB's LB VM, removing the public-NLB hairpin and the NAT-GW egress path.

Teardown: the Set A ENI is an extra cross-account NIC on the LB VM, not
its primary, so the instance-terminate cascade never reclaims it. Detach
(store-clear) the stale attachment before deleting, so private clusters
delete cleanly instead of looping on InvalidNetworkInterface.InUse.

Refs: mulga-siv-301

* fix(vpc): honor SecurityGroupRuleIds in RevokeSecurityGroup ingress/egress

Modern Terraform (aws_vpc_security_group_ingress_rule) revokes per-rule
resources by SecurityGroupRuleId with empty IpPermissions. The handlers
only read IpPermissions, so they returned success while removing nothing;
the surviving SG-reference rule then pinned both SGs undeletable
(DependencyViolation), deadlocking destroy of mutually-referencing SGs.

Resolve SecurityGroupRuleIds to their stored rules and remove them
(malformed -> InvalidSecurityGroupRuleId.Malformed, absent -> NotFound).

Refs: mulga-siv-321

* fix(vpc): reap the main route table on DeleteVpc

CreateVpc auto-creates a main route table and DeleteRouteTable refuses to
delete a main RT (AWS-faithful), so nothing reclaimed it — every deleted
VPC leaked one orphaned main RT in the rtb KV bucket. DeleteVpc now reaps
the VPC's main route table before deleting the VPC record.

Refs: mulga-siv-322

* fix: make AttachRolePolicy and AssociateRouteTable atomic via KV CAS

Concurrent writes to a single role / route table record used a blind
read-modify-Put with no optimistic-concurrency guard, so parallel writers
racing from the same revision clobbered each other and lost updates.
Verified on env19: three parallel policy attachments to one node role kept
only one; two parallel subnet associations to one route table kept only
one (the provider then timed out waiting for state 'associated').

Route the mutating paths (AttachRolePolicy/DetachRolePolicy,
AssociateRouteTable/DisassociateRouteTable) through a revision-guarded
Update + bounded retry, matching the existing accountCounterBucket CAS
idiom. Add concurrent regression tests covering both lost updates.

* docs: add EKS Terraform workbooks (quickstart, addons-access, https-ingress)

Three OpenTofu workbooks demonstrating EKS on Spinifex, each ending in a
browsable load-balanced demo app:
- eks-quickstart: minimal single-node cluster + NodePort demo
- eks-addons-access: CoreDNS managed addon + access-entry RBAC
- eks-https-ingress: 3-node cluster behind an HTTPS ALB terminating TLS
  with an ACM-imported self-signed certificate

Each covers the Spinifex-specific endpoints, the auto-managed worker SG
NodePort pattern, and the omitted addon_version constraint.

.github/workflows/e2e-baremetal.yml

@@ -27,16 +27,16 @@ jobs:
       cancel-in-progress: false
     env:
       BM_ISO_CACHE: /var/cache/spinifex-iso-builder
-      ARTIFACT_DIR: /tmp/spinifex-e2e-baremetal-artifacts-${{ github.run_id }}
+      ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-20-${{ github.run_id }}
     steps:
       - name: Clean Workspace
-        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          sudo umount -R -l "${BM_ISO_CACHE}/rootfs" 2>/dev/null || true
+          if [ ! -c /dev/null ]; then sudo rm -f /dev/null; sudo mknod -m 666 /dev/null c 1 3; fi
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
 
       - uses: actions/checkout@v6
         with:
-          # When called (nightly/release) the caller picks the ref (dev on
-          # schedule, the tag on release); standalone dispatch defaults to the
-          # triggering ref.
           ref: ${{ inputs.ref || github.ref }}
 
       - uses: actions/create-github-app-token@v3
@@ -104,7 +104,6 @@ jobs:
             echo "Ubuntu 26.04 cloud image cached ($(du -h "$UBUNTU_IMG" | cut -f1))"
           fi
 
-
       - name: Install go-junit-report on runner
         run: go install github.com/jstemmer/go-junit-report/v2@latest
 
@@ -117,10 +116,6 @@ jobs:
           SPX_EXTERNAL_POOL: ${{ env.SPINIFEX_CI_POOL }}
         run: |
           mkdir -p "${ARTIFACT_DIR}"
-          # Capture combined bootstrap + e2e output so failures produce a
-          # downloadable, analyzable artifact (the suites run go test -v on the
-          # init node and stream back over SSH). pipefail keeps the script's
-          # real exit code through the tee.
           set -o pipefail
           sudo -E mulga/scripts/baremetal/run-baremetal-e2e.sh \
             --vmlinuz   "${BM_ISO_CACHE}/vmlinuz" \
@@ -142,12 +137,12 @@ jobs:
         with:
           junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
           log-dir: ${{ env.ARTIFACT_DIR }}
-          title: "Bare-Metal E2E failure analysis"
+          title: "E2E failure analysis (cell 20)"
 
       - uses: actions/upload-artifact@v7
         if: always()
         with:
-          name: e2e-baremetal-artifacts-${{ github.run_id }}
+          name: e2e-artifacts-cell-20-${{ github.run_id }}
           path: ${{ env.ARTIFACT_DIR }}
           if-no-files-found: warn
           retention-days: 14

.github/workflows/e2e-ddil.yml

@@ -21,10 +21,6 @@ jobs:
   ddil:
     name: DDIL Real Multi-Node
     runs-on: [ self-hosted, ci-multi ]
-    # Budget: ~45m go test + tofu provision/bootstrap/teardown on ci-multi.
-    # In Phase 1 the scenarios all t.Skip, so the real-world runtime is
-    # dominated by provisioning; the 60m ceiling leaves headroom for
-    # Phase 2/3 when assertions come online.
     timeout-minutes: 60
     env:
       TOFU_LOG: /tmp/e2e-tofu-ddil-${{ github.run_id }}.log
@@ -39,10 +35,6 @@ jobs:
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
-      # Full spinifex checkout: the Go harness must run on the runner
-      # (outside the cluster) so its SSH path survives scenario partitions
-      # that drop peer-to-peer cluster traffic. Running inside the cluster
-      # would self-sever when Scenario C isolates a node.
       - uses: actions/checkout@v6
         name: Checkout Spinifex
         with:
@@ -88,7 +80,6 @@ jobs:
         working-directory: mulga/scripts/tofu-cluster
         run: |
           set -euo pipefail
-          # shellcheck source=/dev/null
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           export TF_VAR_git_branch="${GIT_BRANCH}"
           quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
@@ -140,10 +131,6 @@ jobs:
             exit 1
           fi
 
-          # Guest SSH key: bootstrap-install.sh imports spinifex-key into EC2;
-          # the matching private material was generated on the primary node at
-          # ~/.ssh/spinifex-key. Pull it to the runner so the harness can SSH
-          # the witness via its MapPublicIpOnLaunch-assigned EIP.
           GUEST_KEY="${RUNNER_TEMP}/spinifex-key"
           scp -i "${SSH_KEY}" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
             "${SSH_USER}@${PRIMARY_IP}:~/.ssh/spinifex-key" "${GUEST_KEY}"
@@ -151,11 +138,6 @@ jobs:
 
           mkdir -p "${RUNNER_TEMP}/ddil-results"
           cd "${GITHUB_WORKSPACE}/spinifex"
-          # gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout,
-          # plus machine-readable jsonfile + JUnit XML artifacts. The JUnit
-          # file feeds the junit-report composite action below so failures
-          # become summary table rows + workflow annotations.
-          # shellcheck source=/dev/null
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           banner "DDIL scenarios"
           DDIL_NODES="${NODE_IPS}" \
@@ -185,10 +167,6 @@ jobs:
         if: failure()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          # Pull spinifex-* journals + QEMU process listings from every
-          # node into the artifact bundle so failures can be triaged
-          # without re-provisioning the cluster. Best-effort — a missing
-          # node should not mask the original test failure.
           set +e
           INVENTORY=$(tofu output -json inventory)
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
@@ -197,7 +175,6 @@ jobs:
           LOG_DIR="${RUNNER_TEMP}/ddil-results/cluster-logs"
           mkdir -p "${LOG_DIR}"
 
-          # shellcheck source=/dev/null
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           mapfile -t NODE_IPS < <(echo "$INVENTORY" | jq -r '.nodes[].wan_ip')
           for ip in "${NODE_IPS[@]}"; do
@@ -221,7 +198,6 @@ jobs:
         working-directory: mulga/scripts/tofu-cluster
         run: |
           set -euo pipefail
-          # shellcheck source=/dev/null
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
           quiet_run "tofu destroy"          "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false

.github/workflows/e2e-nightly.yml

@@ -40,6 +40,7 @@ jobs:
     timeout-minutes: 35
     env:
       TOFU_LOG: /tmp/e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}.log
+      ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}
 
     steps:
       # blank or scheduled run = full matrix.
@@ -60,7 +61,10 @@ jobs:
 
       - name: Clean Workspace
         if: steps.gate.outputs.selected == 'true'
-        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+        run: |
+          sudo umount -R -l /var/cache/spinifex-iso-builder/rootfs 2>/dev/null || true
+          if [ ! -c /dev/null ]; then sudo rm -f /dev/null; sudo mknod -m 666 /dev/null c 1 3; fi
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         if: steps.gate.outputs.selected == 'true'
@@ -78,6 +82,11 @@ jobs:
           sparse-checkout: scripts/tofu-cluster
           path: mulga
 
+      - uses: actions/checkout@v6
+        if: steps.gate.outputs.selected == 'true'
+        with:
+          path: spinifex
+
       - name: Build Install Artifacts
         if: steps.gate.outputs.selected == 'true'
         working-directory: mulga/scripts/tofu-cluster
@@ -132,8 +141,18 @@ jobs:
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
-          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
-            "cd ~/e2e-tests && ./run-e2e.sh && ./run-cert-e2e.sh && ENDPOINT=https://${WAN_IP}:9999 ./run-lb-e2e.sh"
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
+             for SUITE in single cert lb; do \
+               echo \"::group::suite \$SUITE\"; \
+               date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
+               GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+                 ARTIFACT_DIR='${ARTIFACT_DIR}' \
+                 ./\$SUITE.test -test.v -test.timeout=30m \
+                 2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log'; \
+               rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
+               if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
+             done; exit \$FAIL"
 
       - name: Run Multi Node E2E
         if: steps.gate.outputs.selected == 'true' && matrix.topo == 'real-multi'
@@ -144,32 +163,57 @@ jobs:
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
 
-          NODE_IPS=""
+          NODES_CSV=""
           NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
           for i in $(seq 0 $((NODE_COUNT - 1))); do
-            NODE_IPS="${NODE_IPS} $(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")"
+            ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
+            NODES_CSV="${NODES_CSV:+$NODES_CSV,}$ip"
           done
 
-          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
-            "cd ~/e2e-tests && ./run-multinode-e2e.sh ${NODE_IPS} && ./run-cert-e2e.sh --multinode ${NODE_IPS} && ENDPOINT=https://${NODE1_IP}:9999 ./run-lb-e2e.sh --nodes ${NODE_IPS}"
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
+             for SUITE in multinode cert lb; do \
+               echo \"::group::suite \$SUITE\"; \
+               date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
+               GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
+                 SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
+                 SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
+                 ARTIFACT_DIR='${ARTIFACT_DIR}' \
+                 ./\$SUITE.test -test.v -test.timeout=30m \
+                 2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log'; \
+               rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
+               if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
+             done; exit \$FAIL"
 
       - name: Run Tofu Workbooks
         if: steps.gate.outputs.selected == 'true' && matrix.extra == 'tofu-examples'
         working-directory: mulga/scripts/tofu-cluster
+        # The driver emits go test -v markers per workbook; teeing to
+        # test-tofu.log feeds the same go-junit-report -> e2e-analyze -> upload
+        # chain the Go suites use, so cell 17 produces an RCA bundle too.
         run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
           INVENTORY=$(tofu output -json inventory)
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          date -u +%Y-%m-%dT%H:%M:%SZ > "${ARTIFACT_DIR}/test-tofu.start"
           ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
-            "cd ~/e2e-tests && ./run-tofu-examples-e2e.sh"
+            "cd ~/e2e-tests && ./run-tofu-examples-e2e.sh" \
+            2>&1 | tee "${ARTIFACT_DIR}/test-tofu.log"
+          rc=${PIPESTATUS[0]}
+          if [ $rc -ne 0 ]; then echo "tofu: FAILED" >> "${ARTIFACT_DIR}/suite-status.txt"; fi
+          exit $rc
 
       - name: Run Reboot Resilience E2E
         if: steps.gate.outputs.selected == 'true' && matrix.extra == 'reboot-single'
         working-directory: mulga/scripts/tofu-cluster
-        # Driver runs from the runner (NOT via ssh-and-run) so it survives the
+        # reboot.test runs from the runner (NOT via ssh-and-run) so it survives the
         # systemctl-reboot-induced ssh disconnect mid-run.
         run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
           INVENTORY=$(tofu output -json inventory)
           WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
@@ -178,31 +222,85 @@ jobs:
           REBOOT_DIR="${RUNNER_TEMP}/reboot-e2e"
           mkdir -p "$REBOOT_DIR"
           tar xf "${RUNNER_TEMP}/spinifex-tests.tar" -C "$REBOOT_DIR"
-          chmod +x "$REBOOT_DIR/run-reboot-e2e.sh"
+          chmod +x "$REBOOT_DIR/_bin/reboot.test"
 
-          WAN_IP="$WAN_IP" SSH_KEY="$SSH_KEY" "$REBOOT_DIR/run-reboot-e2e.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i "$SSH_KEY")
+          CREDS=$(ssh "${SSH_OPTS[@]}" tf-user@${WAN_IP} 'cat ~/.aws/credentials')
+          AWS_AK=$(echo "$CREDS" | awk -F '= *' '/^aws_access_key_id/ {print $2; exit}')
+          AWS_SK=$(echo "$CREDS" | awk -F '= *' '/^aws_secret_access_key/ {print $2; exit}')
+          echo "::add-mask::${AWS_AK}"; echo "::add-mask::${AWS_SK}"
 
-      - name: Dump Cluster Journals on Failure
-        if: failure() && steps.gate.outputs.selected == 'true'
+          date -u +%Y-%m-%dT%H:%M:%SZ > "${ARTIFACT_DIR}/test-reboot.start"
+          GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+            SPINIFEX_WAN_IP="${WAN_IP}" SPINIFEX_SSH_KEY="${SSH_KEY}" \
+            SPINIFEX_AWS_ENDPOINT="https://${WAN_IP}:9999" \
+            SPINIFEX_AWS_ACCESS_KEY_ID="${AWS_AK}" SPINIFEX_AWS_SECRET_ACCESS_KEY="${AWS_SK}" \
+            ARTIFACT_DIR="${ARTIFACT_DIR}" \
+            "$REBOOT_DIR/_bin/reboot.test" -test.v -test.timeout=30m \
+            2>&1 | tee "${ARTIFACT_DIR}/test-reboot.log"
+          rc=${PIPESTATUS[0]}
+          if [ $rc -ne 0 ]; then echo "reboot: FAILED" >> "${ARTIFACT_DIR}/suite-status.txt"; fi
+          exit $rc
+
+      - name: Collect e2e artifacts
+        if: always() && steps.gate.outputs.selected == 'true'
         working-directory: mulga/scripts/tofu-cluster
         run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
           INVENTORY=$(tofu output -json inventory 2>/dev/null) || { echo "no inventory; skipping"; exit 0; }
-          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
-          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path'); SSH_KEY="${SSH_KEY/#\~/$HOME}"
           NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY")
+
+          ssh "${SSH_OPTS[@]}" tf-user@${NODE1_IP} "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+
           for i in $(seq 0 $((NODE_COUNT - 1))); do
-            IP=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
-            echo "===== node$((i+1)) ($IP) ====="
-            for svc in spinifex-nats spinifex-predastore spinifex-viperblock spinifex-daemon spinifex-awsgw spinifex-vpcd dhcpcd; do
-              echo "--- $svc ---"
-              ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" tf-user@${IP} \
-                "sudo journalctl -u $svc --no-pager -n 200 2>/dev/null || echo '(not found)'" 2>&1 || echo "(node unreachable)"
-            done
-            echo "--- DHCP client state ---"
-            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" tf-user@${IP} \
-              "systemctl is-enabled dhcpcd.service 2>/dev/null; systemctl is-active dhcpcd.service 2>/dev/null; sudo ss -lupn 'sport = 68' 2>/dev/null; pgrep -laf dhcpcd 2>/dev/null" 2>&1 || echo "(node unreachable)"
+            IP=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip"); idx=$((i + 1))
+            mkdir -p "${ARTIFACT_DIR}/vm-console-node${idx}"
+            ssh "${SSH_OPTS[@]}" tf-user@${IP} 'sudo sh -c "cd /run/spinifex 2>/dev/null && tar -cf - console-*.log 2>/dev/null"' \
+              | tar -C "${ARTIFACT_DIR}/vm-console-node${idx}" -xf - 2>/dev/null
+            ssh "${SSH_OPTS[@]}" tf-user@${IP} 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "60 min ago"' \
+              > "${ARTIFACT_DIR}/spinifex-journal-node${idx}.log" 2>/dev/null
           done
 
+          cat "${ARTIFACT_DIR}"/spinifex-journal-node*.log \
+            > "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || true
+          echo "collected: $(ls "${ARTIFACT_DIR}" 2>/dev/null | wc -l) entries"
+
+      - name: Install go-junit-report
+        if: always() && steps.gate.outputs.selected == 'true'
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Convert test logs to JUnit XML
+        if: always() && steps.gate.outputs.selected == 'true'
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log); SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Analyze e2e failures
+        if: always() && steps.gate.outputs.selected == 'true'
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "E2E failure analysis (cell ${{ matrix.cell }})"
+
+      - uses: actions/upload-artifact@v7
+        if: always() && steps.gate.outputs.selected == 'true'
+        with:
+          name: e2e-artifacts-cell-${{ matrix.cell }}-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
       - name: Teardown
         if: always()
         working-directory: mulga/scripts/tofu-cluster
@@ -217,8 +315,7 @@ jobs:
           tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1
           ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
 
-  # Bare-metal single-node suite (PXE-reimaged physical hardware). Doesn't fit
-  # the tofu matrix shape (different runner/provisioning/120m timeout), so it
+  # Bare-metal single-node suite. Doesn't fit the tofu matrix shape, so it
   # runs as a sibling job via the reusable e2e-baremetal workflow. Gated like a
   # matrix cell ("cell 20"): every weeknight cron, or dispatch with 20 selected
   # (blank cells = all). Tests `dev` on schedule to match GIT_BRANCH above.

.github/workflows/e2e-suites.yml

@@ -0,0 +1,799 @@
+# E2E (per-suite blocks) — EXPERIMENTAL, workflow_dispatch only.
+#
+# Restructures the looped e2e.yml into env-grouped job blocks so each suite
+# surfaces as its own node in the Actions graph (mulga-siv-283):
+#
+#   single_provision ─┬─ single_suite (matrix: single, cert, lb, eks)
+#                     └─ single_teardown (if: always)
+#   multi_provision  ─┬─ multi_suite  (matrix: multinode)
+#                     └─ multi_teardown (if: always)
+#
+# Constraint: tofu state is runner-filesystem-local, so *_provision and
+# *_teardown must execute on the same self-hosted runner (one runner per label
+# today). Teardown reuses the persisted mulga/ tofu-cluster workspace and does
+# NOT re-checkout mulga (that would wipe local state). Suite-matrix legs reach
+# the already-provisioned VM over its WAN IP using the key path the provision
+# job emits; they share the runner home dir, so the key file persists.
+#
+# This file does NOT post the gating `e2e` commit status — it runs alongside
+# e2e.yml for validation. Once green across a few manual runs, fold the
+# triggers from e2e.yml here and retire the loop.
+
+name: E2E (suite blocks)
+
+on:
+  workflow_dispatch:
+    inputs:
+      single_suites:
+        description: "Single-node suites to run as matrix legs"
+        required: false
+        default: "single cert lb eks"
+      run_multinode:
+        description: "Also run the multinode env block"
+        type: boolean
+        required: false
+        default: true
+      test_run:
+        description: "Optional -test.run regex (empty = all tests in suite)"
+        required: false
+        default: ""
+
+permissions:
+  contents: read
+
+env:
+  GIT_BRANCH: ${{ github.ref_name }}
+  TEST_RUN: ${{ inputs.test_run || '' }}
+
+jobs:
+  # ---------------------------------------------------------------------------
+  # Single-node env
+  # ---------------------------------------------------------------------------
+  single_provision:
+    name: Single env · provision
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 40
+    outputs:
+      wan_ip: ${{ steps.vm.outputs.wan_ip }}
+      ssh_key: ${{ steps.vm.outputs.ssh_key }}
+      suites: ${{ steps.suites.outputs.list }}
+      matrix: ${{ steps.suites.outputs.matrix }}
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
+    steps:
+      - name: Reap Orphan Procs
+        run: |
+          set +e
+          UID_SELF=$(id -u)
+          WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in
+                "$WS"|"$WS"/*) echo "reap $sig pid=$pid cwd=$cwd"; kill "-$sig" "$pid" 2>/dev/null ;;
+              esac
+            done
+          }
+          reap TERM; sleep 3; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          true
+
+      - name: Clean Workspace
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - uses: actions/checkout@v6
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/tofu-cluster
+          path: mulga
+
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Re-enable Go toolchain auto-download
+        run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
+
+      - name: Compute suite matrix
+        id: suites
+        run: |
+          LIST="${{ inputs.single_suites || 'single cert lb eks' }}"
+          echo "list=${LIST}" >> "$GITHUB_OUTPUT"
+          JSON=$(printf '%s\n' $LIST | jq -R . | jq -cs .)
+          echo "matrix=${JSON}" >> "$GITHUB_OUTPUT"
+
+      - name: Reclaim docker build cache
+        run: |
+          docker container prune -f || true
+          docker image prune -f || true
+          docker buildx prune -af --filter "until=24h" || true
+          docker system df || true
+
+      - name: Build Install Artifacts
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Provision VM
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          export TF_VAR_git_branch="${GIT_BRANCH}"
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "VM provisioned"
+
+      - name: Bootstrap via Binary Install
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Resolve VM SSH details
+        id: vm
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          echo "wan_ip=${WAN_IP}" >> "$GITHUB_OUTPUT"
+          echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
+
+      - name: Build eks-node AMI (libguestfs, runner)
+        if: contains(steps.suites.outputs.list, 'eks')
+        working-directory: spinifex
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "build eks-node AMI (libguestfs — no host devices)"
+          sudo apt-get update -qq
+          sudo apt-get install -y --no-install-recommends libguestfs-tools qemu-utils
+          export LIBGUESTFS_BACKEND=direct
+          sudo chmod 0644 /boot/vmlinuz-* || true
+          group_open "make build-eks-node-image"
+          make build-eks-node-image
+          group_close
+          ok "raw: $(du -h /tmp/eks-node-image-build/eks-node-alpine.raw | cut -f1)"
+
+      - name: Import eks-node AMI on VM
+        if: contains(steps.suites.outputs.list, 'eks')
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          eval "$(grep -E '^(ALPINE_VERSION|AMI_NAME|SYSTEM_TAG)=' \
+            spinifex/scripts/images/eks-node/manifest.conf)"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          banner "import eks-node AMI to VM image store"
+          scp "${SSH_OPTS[@]}" \
+            /tmp/eks-node-image-build/eks-node-alpine.raw \
+            "$VM":/tmp/eks-node-alpine.raw
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "spx admin images import --file /tmp/eks-node-alpine.raw \
+               --distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
+               --boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}" >/dev/null
+          ok "eks-node AMI imported"
+
+      - name: Install kubectl on VM
+        if: contains(steps.suites.outputs.list, 'eks')
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          banner "install kubectl on VM (eks data-plane subtest)"
+          ssh "${SSH_OPTS[@]}" "$VM" 'bash -s' <<'EOF'
+          set -euo pipefail
+          KVER=v1.32.5
+          case "$(uname -m)" in
+            x86_64)  ARCH=amd64 ;;
+            aarch64) ARCH=arm64 ;;
+            *) echo "unsupported arch $(uname -m)" >&2; exit 1 ;;
+          esac
+          cd /tmp
+          curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl"
+          curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl.sha256"
+          echo "$(cat kubectl.sha256)  kubectl" | sha256sum -c -
+          sudo install -m 0755 kubectl /usr/local/bin/kubectl
+          kubectl version --client
+          EOF
+          ok "kubectl installed on VM"
+
+      - name: Build E2E test binaries on runner
+        working-directory: spinifex/tests/e2e
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          # teardown always builds: the run-scoped sweep binary runs in the
+          # teardown job, it is never a selectable suite leg.
+          SUITES_BUILD="${{ steps.suites.outputs.list }} teardown"
+          banner "build ${SUITES_BUILD// /, }"
+          group_open "go test -c"
+          make e2e-build SUITES_BUILD="${SUITES_BUILD}"
+          group_close
+          for f in _bin/*.test; do
+            ok "$(basename "$f") $(stat -c '%s' "$f" | numfmt --to=iec)"
+          done
+
+      - name: Ship test binaries to VM
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          ssh "${SSH_OPTS[@]}" "$VM" "rm -rf \$HOME/spinifex-e2e-bin && mkdir -p \$HOME/spinifex-e2e-bin"
+          tar -C spinifex/tests/e2e/_bin -cf - . | ssh "${SSH_OPTS[@]}" "$VM" "tar -xf - -C \$HOME/spinifex-e2e-bin"
+          ok "shipped binaries to $VM"
+
+      - name: Start packet capture on VM br-wan
+        run: |
+          set +e
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
+          scp "${SSH_OPTS[@]}" \
+            "${GITHUB_WORKSPACE}/spinifex/.github/scripts/start-net-capture.sh" \
+            "$VM":/tmp/start-net-capture.sh
+          ssh "${SSH_OPTS[@]}" "$VM" "bash /tmp/start-net-capture.sh" || true
+
+  single_suite:
+    name: Single env · suite
+    needs: single_provision
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 45
+    strategy:
+      fail-fast: false
+      matrix:
+        suite: ${{ fromJSON(needs.single_provision.outputs.matrix) }}
+    env:
+      WAN_IP: ${{ needs.single_provision.outputs.wan_ip }}
+      SSH_KEY: ${{ needs.single_provision.outputs.ssh_key }}
+      ARTIFACT_DIR: /tmp/spinifex-e2e-single-${{ github.run_id }}-${{ matrix.suite }}
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Install go-junit-report
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Run ${{ matrix.suite }}
+        id: e2e
+        run: |
+          set +e
+          SUITE="${{ matrix.suite }}"
+          VM_HOST="tf-user@${WAN_IP}"
+          TEST_RUN_FLAG=""
+          [ -n "${TEST_RUN}" ] && TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+          cleanup_remote() {
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" "$VM_HOST" \
+              "pkill -u tf-user -TERM -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; true" \
+              2>/dev/null || true
+          }
+          trap cleanup_remote EXIT INT TERM
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" "$VM_HOST" \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/spinifex-e2e-bin; \
+             R=\$'\033[0m' B=\$'\033[1m' C=\$'\033[36m' G=\$'\033[32m' RED=\$'\033[31m'; \
+             printf '\n%s%s━━ suite %s ━━%s\n' \"\$B\" \"\$C\" \"${SUITE}\" \"\$R\"; \
+             date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-${SUITE}.start'; \
+             GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+               SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+               ARTIFACT_DIR='${ARTIFACT_DIR}' \
+               ./${SUITE}.test -test.v -test.timeout=30m ${TEST_RUN_FLAG} \
+               2>&1 | tee '${ARTIFACT_DIR}/test-${SUITE}.log' \
+                    | sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'; \
+             exit \${PIPESTATUS[0]}"
+          echo "rc=$?" >> "$GITHUB_OUTPUT"
+          exit 0
+
+      - name: Collect suite artifacts from VM
+        if: always()
+        run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${SSH_KEY}")
+          VM=tf-user@${WAN_IP}
+          ssh "${SSH_OPTS[@]}" "$VM" "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+          true
+
+      - name: Convert test log to JUnit XML
+        if: always()
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log); SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          title: "E2E ${{ matrix.suite }} results"
+
+      - name: Analyze e2e failures
+        if: always()
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "E2E ${{ matrix.suite }} failure analysis"
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-single-${{ matrix.suite }}-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
+      - name: Fail job if suite failed
+        if: always()
+        run: |
+          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
+            echo "::error::suite ${{ matrix.suite }} failed (rc=${{ steps.e2e.outputs.rc }})"
+            exit 1
+          fi
+
+  single_teardown:
+    name: Single env · teardown
+    needs: [ single_provision, single_suite ]
+    if: always()
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 20
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-single-td-${{ github.run_id }}.log
+      ARTIFACT_DIR: /tmp/spinifex-e2e-single-diag-${{ github.run_id }}
+      WAN_IP: ${{ needs.single_provision.outputs.wan_ip }}
+      SSH_KEY: ${{ needs.single_provision.outputs.ssh_key }}
+    steps:
+      # Reuse the persisted mulga/ tofu-cluster workspace from single_provision.
+      # Re-checkout only spinifex (for ci-fmt + diag scripts); checking out mulga
+      # again would wipe the local tofu state and orphan the VM.
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - name: Capture network/IPsec diagnostics on VM
+        if: ${{ needs.single_provision.result == 'success' && env.WAN_IP != '' }}
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}")
+          VM=tf-user@${WAN_IP}
+          scp "${SSH_OPTS[@]}" \
+            "${GITHUB_WORKSPACE}/spinifex/.github/scripts/capture-net-diag.sh" \
+            "$VM":/tmp/capture-net-diag.sh 2>/dev/null
+          ssh "${SSH_OPTS[@]}" "$VM" "bash /tmp/capture-net-diag.sh '${ARTIFACT_DIR}/net-diag'" 2>/dev/null
+          ssh "${SSH_OPTS[@]}" "$VM" 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "90 min ago"' \
+            > "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null
+          mkdir -p "${ARTIFACT_DIR}/lb-console"
+          ssh "${SSH_OPTS[@]}" "$VM" 'sudo sh -c "cd /run/spinifex 2>/dev/null && tar -cf - console-*.log 2>/dev/null"' \
+            | tar -C "${ARTIFACT_DIR}/lb-console" -xf - 2>/dev/null
+          true
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-single-diag-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: ignore
+          retention-days: 14
+
+      # Reclaim any resource leaked by a crashed/interrupted suite before the VM
+      # is destroyed. Scoped to this run's tag, best-effort, never fails the job.
+      - name: Sweep leaked run resources on VM
+        if: ${{ needs.single_provision.result == 'success' && env.WAN_IP != '' }}
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}")
+          VM=tf-user@${WAN_IP}
+          banner "sweep run-tagged resources (run ${{ github.run_id }})"
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "cd \$HOME/spinifex-e2e-bin 2>/dev/null && \
+             SPINIFEX_E2E=1 SPINIFEX_MODE=single \
+             SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+             ./teardown.test -test.v -test.run TestSweepRunResources -test.timeout=10m" \
+            2>&1 || true
+          true
+
+      - name: Teardown VM
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          if [ -n "${WAN_IP}" ] && [ -n "${SSH_KEY}" ]; then
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}" \
+              tf-user@${WAN_IP} \
+              'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
+              2>/dev/null || true
+          fi
+          UID_SELF=$(id -u); WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in "$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;; esac
+            done
+          }
+          reap TERM; sleep 2; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
+          ok "teardown complete"
+
+  # ---------------------------------------------------------------------------
+  # Multinode env
+  # ---------------------------------------------------------------------------
+  multi_provision:
+    name: Multi env · provision
+    if: ${{ inputs.run_multinode }}
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 45
+    outputs:
+      node1_ip: ${{ steps.cluster.outputs.node1_ip }}
+      ssh_key: ${{ steps.cluster.outputs.ssh_key }}
+      nodes_csv: ${{ steps.cluster.outputs.nodes_csv }}
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
+    steps:
+      - name: Reap Orphan Procs
+        run: |
+          set +e
+          UID_SELF=$(id -u); WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in "$WS"|"$WS"/*) echo "reap $sig pid=$pid"; kill "-$sig" "$pid" 2>/dev/null ;; esac
+            done
+          }
+          reap TERM; sleep 3; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          true
+
+      - name: Clean Workspace
+        run: |
+          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
+            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
+          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - uses: actions/checkout@v6
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/tofu-cluster
+          path: mulga
+
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Re-enable Go toolchain auto-download
+        run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
+
+      - name: Reclaim docker build cache
+        run: |
+          docker container prune -f || true
+          docker image prune -f || true
+          docker buildx prune -af --filter "until=24h" || true
+          docker system df || true
+
+      - name: Build Install Artifacts
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Provision VMs
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          export TF_VAR_git_branch="${GIT_BRANCH}"
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "3-node cluster provisioned"
+
+      - name: Bootstrap via Binary Install
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Resolve cluster SSH details
+        id: cluster
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path'); SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          NODES_CSV=""
+          for i in $(seq 0 $((NODE_COUNT - 1))); do
+            ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
+            if [ -z "$NODES_CSV" ]; then NODES_CSV="$ip"; else NODES_CSV="$NODES_CSV,$ip"; fi
+          done
+          echo "node1_ip=${NODE1_IP}" >> "$GITHUB_OUTPUT"
+          echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
+          echo "nodes_csv=${NODES_CSV}" >> "$GITHUB_OUTPUT"
+
+      - name: Build multinode binary on runner
+        working-directory: spinifex/tests/e2e
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "build multinode, teardown"
+          group_open "go test -c"
+          make e2e-build SUITES_BUILD="multinode teardown"
+          group_close
+          ok "$(basename _bin/multinode.test) $(stat -c '%s' _bin/multinode.test | numfmt --to=iec)"
+
+      - name: Ship test binaries to node1
+        run: |
+          set -euo pipefail
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.cluster.outputs.ssh_key }}")
+          VM=tf-user@${{ steps.cluster.outputs.node1_ip }}
+          ssh "${SSH_OPTS[@]}" "$VM" "rm -rf \$HOME/spinifex-e2e-bin && mkdir -p \$HOME/spinifex-e2e-bin"
+          tar -C spinifex/tests/e2e/_bin -cf - . | ssh "${SSH_OPTS[@]}" "$VM" "tar -xf - -C \$HOME/spinifex-e2e-bin"
+          ok "shipped binaries to $VM"
+
+      - name: Start packet capture on all nodes
+        run: |
+          set +e
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.cluster.outputs.ssh_key }}")
+          IFS=',' read -ra NODE_LIST <<< "${{ steps.cluster.outputs.nodes_csv }}"
+          for ip in "${NODE_LIST[@]}"; do
+            [ -z "$ip" ] && continue
+            scp "${SSH_OPTS[@]}" \
+              "${GITHUB_WORKSPACE}/spinifex/.github/scripts/start-net-capture.sh" \
+              tf-user@"$ip":/tmp/start-net-capture.sh
+            ssh "${SSH_OPTS[@]}" tf-user@"$ip" "bash /tmp/start-net-capture.sh" || true
+          done
+
+  multi_suite:
+    name: Multi env · suite
+    needs: multi_provision
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 60
+    strategy:
+      fail-fast: false
+      matrix:
+        suite: [ multinode ]
+    env:
+      NODE1_IP: ${{ needs.multi_provision.outputs.node1_ip }}
+      SSH_KEY: ${{ needs.multi_provision.outputs.ssh_key }}
+      NODES_CSV: ${{ needs.multi_provision.outputs.nodes_csv }}
+      ARTIFACT_DIR: /tmp/spinifex-e2e-multi-${{ github.run_id }}-${{ matrix.suite }}
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Install go-junit-report
+        run: |
+          go install github.com/jstemmer/go-junit-report/v2@latest
+          echo "$HOME/go/bin" >> "$GITHUB_PATH"
+
+      - name: Run ${{ matrix.suite }}
+        id: e2e
+        run: |
+          set +e
+          SUITE="${{ matrix.suite }}"
+          VM_HOST="tf-user@${NODE1_IP}"
+          TEST_RUN_FLAG=""
+          [ -n "${TEST_RUN}" ] && TEST_RUN_FLAG="-test.run=${TEST_RUN}"
+          cleanup_remote() {
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" "$VM_HOST" \
+              "pkill -u tf-user -TERM -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f \"/spinifex-e2e-bin/${SUITE}\.test( |\$)\" 2>/dev/null; true" \
+              2>/dev/null || true
+          }
+          trap cleanup_remote EXIT INT TERM
+          ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" "$VM_HOST" \
+            "set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/spinifex-e2e-bin; \
+             printf '\n━━ suite %s ━━\n' \"${SUITE}\"; \
+             date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-${SUITE}.start'; \
+             GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
+               SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+               SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
+               SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
+               ARTIFACT_DIR='${ARTIFACT_DIR}' \
+               ./${SUITE}.test -test.v -test.timeout=30m ${TEST_RUN_FLAG} \
+               2>&1 | tee '${ARTIFACT_DIR}/test-${SUITE}.log' \
+                    | sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'; \
+             exit \${PIPESTATUS[0]}"
+          echo "rc=$?" >> "$GITHUB_OUTPUT"
+          exit 0
+
+      - name: Collect suite artifacts from node1
+        if: always()
+        run: |
+          set +e
+          mkdir -p "${ARTIFACT_DIR}"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -i "${SSH_KEY}")
+          ssh "${SSH_OPTS[@]}" tf-user@${NODE1_IP} "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
+            | tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
+          true
+
+      - name: Convert test log to JUnit XML
+        if: always()
+        run: |
+          shopt -s nullglob
+          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
+            BASE=$(basename "$LOG" .log); SUITE=${BASE#test-}
+            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
+          done
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          title: "Multi-Node E2E ${{ matrix.suite }} results"
+
+      - name: Analyze e2e failures
+        if: always()
+        uses: ./spinifex/.github/actions/e2e-analyze
+        with:
+          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+          log-dir: ${{ env.ARTIFACT_DIR }}
+          title: "Multi-Node E2E ${{ matrix.suite }} failure analysis"
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-multi-${{ matrix.suite }}-${{ github.run_id }}
+          path: ${{ env.ARTIFACT_DIR }}
+          if-no-files-found: warn
+          retention-days: 14
+
+      - name: Fail job if suite failed
+        if: always()
+        run: |
+          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
+            echo "::error::suite ${{ matrix.suite }} failed (rc=${{ steps.e2e.outputs.rc }})"
+            exit 1
+          fi
+
+  multi_teardown:
+    name: Multi env · teardown
+    needs: [ multi_provision, multi_suite ]
+    if: always() && inputs.run_multinode
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 20
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-multi-td-${{ github.run_id }}.log
+      SSH_KEY: ${{ needs.multi_provision.outputs.ssh_key }}
+      NODES_CSV: ${{ needs.multi_provision.outputs.nodes_csv }}
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          path: spinifex
+
+      # Reclaim leaked run-tagged resources before the cluster is destroyed.
+      - name: Sweep leaked run resources on node1
+        if: ${{ needs.multi_provision.result == 'success' && needs.multi_provision.outputs.node1_ip != '' }}
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}")
+          VM=tf-user@${{ needs.multi_provision.outputs.node1_ip }}
+          banner "sweep run-tagged resources (run ${{ github.run_id }})"
+          ssh "${SSH_OPTS[@]}" "$VM" \
+            "cd \$HOME/spinifex-e2e-bin 2>/dev/null && \
+             SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
+             SPINIFEX_E2E_RUN_ID='${{ github.run_id }}' \
+             ./teardown.test -test.v -test.run TestSweepRunResources -test.timeout=10m" \
+            2>&1 || true
+          true
+
+      - name: Teardown cluster
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set +e
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          if [ -n "${NODES_CSV}" ] && [ -n "${SSH_KEY}" ]; then
+            IFS=',' read -ra NODE_LIST <<< "${NODES_CSV}"
+            for ip in "${NODE_LIST[@]}"; do
+              ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "${SSH_KEY}" \
+                tf-user@"$ip" \
+                'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
+                2>/dev/null || true
+            done
+          fi
+          UID_SELF=$(id -u); WS="${GITHUB_WORKSPACE}"
+          reap() {
+            local sig="$1"
+            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
+              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
+              case "$cwd" in "$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;; esac
+            done
+          }
+          reap TERM; sleep 2; reap KILL
+          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
+          ok "teardown complete"
+
+  e2e_suites_status:
+    name: E2E suite blocks · summary
+    if: always()
+    needs: [ single_suite, multi_suite ]
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - name: Summarize
+        run: |
+          echo "## E2E suite-block results" >> "$GITHUB_STEP_SUMMARY"
+          echo "single_suite: ${{ needs.single_suite.result }}" >> "$GITHUB_STEP_SUMMARY"
+          echo "multi_suite:  ${{ needs.multi_suite.result }}" >> "$GITHUB_STEP_SUMMARY"
+          case "${{ needs.single_suite.result }}" in success|skipped) ;; *) exit 1 ;; esac
+          case "${{ needs.multi_suite.result }}" in success|skipped) ;; *) exit 1 ;; esac

.github/workflows/e2e.yml

@@ -36,14 +36,50 @@ env:
   TEST_RUN: ${{ inputs.test_run || '' }}
 
 jobs:
+  # Shadow-mode suite selector. Computes which suites the diff *would* run and logs it to the job summary.
+  # Does NOT gate or override the real SUITES env yet — it runs alongside the full matrix so we
+  # can compare predicted vs actual before flipping it on.
+  select_suites:
+    name: Select Suites (shadow)
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: go.mod
+          cache: false
+      - name: Compute targeted suite selection
+        run: |
+          set -euo pipefail
+          BASE="${{ github.event.before }}"
+          HEAD="${{ github.sha }}"
+          # First push to a branch reports an all-zero before-SHA; fall back to
+          # the first parent so the diff is still meaningful.
+          if [ -z "$BASE" ] || [ "$BASE" = "0000000000000000000000000000000000000000" ]; then
+            BASE="${HEAD}~1"
+          fi
+          echo "## Targeted suite selection (shadow)" >> "$GITHUB_STEP_SUMMARY"
+          echo '```' >> "$GITHUB_STEP_SUMMARY"
+          echo "base=$BASE head=$HEAD" >> "$GITHUB_STEP_SUMMARY"
+          go run ./tests/e2e/select-suites/cmd/select-suites \
+            --manifest docs/service-interfaces.yaml \
+            --base-sha "$BASE" --head-sha "$HEAD" --repo-root . \
+            2>>"$GITHUB_STEP_SUMMARY" \
+            | tee -a "$GITHUB_STEP_SUMMARY"
+          echo '```' >> "$GITHUB_STEP_SUMMARY"
+          echo "Shadow only — actual run uses SUITES='${SUITES}'." >> "$GITHUB_STEP_SUMMARY"
+
   e2e_single:
     name: Single-Node E2E
     # Skip when the dispatch input requests no in-VM suites (e.g.
-    # suites='e2e-reboot' only). Push triggers and the default SUITES list
+    # suites='e2e-multinode' only). Push triggers and the default SUITES list
     # both include single/cert/lb so the inputs.suites null-check keeps the
     # job running on plain PR pushes.
     if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-lb') || contains(github.event.inputs.suites, 'e2e-eks')
-    runs-on: [ self-hosted, ci-single ]
+    runs-on: [self-hosted, ci-single]
     timeout-minutes: 45
     env:
       TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
@@ -71,11 +107,6 @@ jobs:
 
       - name: Clean Workspace
         run: |
-          # Detach any leaked mounts under the workspace before deleting. A
-          # killed image build can leave a `mount --bind /dev` under its build
-          # dir; if `rm -rf` traverses it as root it wipes the host's /dev
-          # (deletes /dev/null and wedges the box). umount deepest-first, then
-          # forbid rm from crossing any mount boundary as a hard backstop.
           awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
             | sort -r | xargs -r -I{} sudo umount -l -R {} || true
           sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
@@ -106,10 +137,10 @@ jobs:
       - name: Re-enable Go toolchain auto-download
         run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
 
-      - name: Prune rootless docker cache
+      - name: Reclaim docker build cache
         run: |
           docker container prune -f || true
-          docker image prune -af || true
+          docker image prune -f || true
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
@@ -184,7 +215,6 @@ jobs:
         run: |
           set -euo pipefail
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          # Pull import args straight from the manifest so they never drift.
           eval "$(grep -E '^(ALPINE_VERSION|AMI_NAME|SYSTEM_TAG)=' \
             spinifex/scripts/images/eks-node/manifest.conf)"
           SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
@@ -238,7 +268,7 @@ jobs:
         run: |
           set -euo pipefail
           source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          SUITES_BUILD=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(reboot|multinode)$' | tr '\n' ' ')
+          SUITES_BUILD=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(multinode)$' | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_BUILD}" | tr -d ' ')" ]; then
             echo "no in-VM suites requested (SUITES=${SUITES}); nothing to build"
             exit 0
@@ -280,7 +310,7 @@ jobs:
         id: e2e
         run: |
           set +e
-          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(reboot|multinode)$' | tr '\n' ' ')
+          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(multinode)$' | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
             echo "no in-VM suites requested (SUITES=${SUITES}); skipping"
             echo "rc=0" >> "$GITHUB_OUTPUT"
@@ -445,7 +475,7 @@ jobs:
   e2e_real_multi:
     name: Real Multi-Node E2E
     if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-multinode')
-    runs-on: [ self-hosted, ci-multi ]
+    runs-on: [self-hosted, ci-multi]
     timeout-minutes: 60
     env:
       TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
@@ -473,11 +503,6 @@ jobs:
 
       - name: Clean Workspace
         run: |
-          # Detach any leaked mounts under the workspace before deleting. A
-          # killed image build can leave a `mount --bind /dev` under its build
-          # dir; if `rm -rf` traverses it as root it wipes the host's /dev
-          # (deletes /dev/null and wedges the box). umount deepest-first, then
-          # forbid rm from crossing any mount boundary as a hard backstop.
           awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
             | sort -r | xargs -r -I{} sudo umount -l -R {} || true
           sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
@@ -508,10 +533,10 @@ jobs:
       - name: Re-enable Go toolchain auto-download
         run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
 
-      - name: Prune rootless docker cache
+      - name: Reclaim docker build cache
         run: |
           docker container prune -f || true
-          docker image prune -af || true
+          docker image prune -f || true
           docker buildx prune -af --filter "until=24h" || true
           docker system df || true
 
@@ -573,7 +598,7 @@ jobs:
           # JetStream clustering isn't solid yet (mulga-omq/z2z/5yg). EKS is
           # validated single-node only for now; re-enable here once clustered
           # JetStream + multinode EKS are a deliberate target.
-          SUITES_BUILD=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(reboot|single|eks)$' | tr '\n' ' ')
+          SUITES_BUILD=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(single|eks)$' | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_BUILD}" | tr -d ' ')" ]; then
             echo "no multinode-eligible suites requested (SUITES=${SUITES}); nothing to build"
             exit 0
@@ -656,7 +681,7 @@ jobs:
         id: e2e
         run: |
           set +e
-          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(reboot|single|eks)$' | tr '\n' ' ')
+          SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -vE '^(single|eks)$' | tr '\n' ' ')
           if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
             echo "no multinode-eligible suites requested (SUITES=${SUITES}); skipping"
             echo "rc=0" >> "$GITHUB_OUTPUT"
@@ -857,275 +882,10 @@ jobs:
             exit 1
           fi
 
-  e2e_reboot:
-    name: Reboot Resilience E2E
-    # Opt-in: only fires when the SUITES dispatch input lists e2e-reboot.
-    # The reboot test reboots the spinifex VM, so it must run on the runner
-    # (not shipped to the VM like the cert/lb/single suites). Runs in
-    # parallel with e2e_single — both target [self-hosted, ci-single]
-    # and GitHub Actions serialises on the underlying runner allocation,
-    # so no explicit needs: ordering is required.
-    if: contains(github.event.inputs.suites, 'e2e-reboot')
-    runs-on: [ self-hosted, ci-single ]
-    timeout-minutes: 60
-    env:
-      TOFU_LOG: /tmp/e2e-tofu-reboot-${{ github.run_id }}.log
-      ARTIFACT_DIR: /tmp/spinifex-e2e-reboot-artifacts-${{ github.run_id }}
-    steps:
-      - name: Reap Orphan Procs
-        run: |
-          set +e
-          UID_SELF=$(id -u)
-          WS="${GITHUB_WORKSPACE}"
-          reap() {
-            local sig="$1"
-            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
-              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
-              case "$cwd" in
-                "$WS"|"$WS"/*) echo "reap $sig pid=$pid cwd=$cwd"; kill "-$sig" "$pid" 2>/dev/null ;;
-              esac
-            done
-          }
-          reap TERM
-          sleep 3
-          reap KILL
-          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
-          true
-
-      - name: Clean Workspace
-        run: |
-          # Detach any leaked mounts under the workspace before deleting. A
-          # killed image build can leave a `mount --bind /dev` under its build
-          # dir; if `rm -rf` traverses it as root it wipes the host's /dev
-          # (deletes /dev/null and wedges the box). umount deepest-first, then
-          # forbid rm from crossing any mount boundary as a hard backstop.
-          awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
-            | sort -r | xargs -r -I{} sudo umount -l -R {} || true
-          sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
-
-      - uses: actions/create-github-app-token@v3
-        id: app-token
-        with:
-          client-id: ${{ secrets.APP_ID }}
-          private-key: ${{ secrets.APP_PRIVATE_KEY }}
-          repositories: mulga
-
-      - uses: actions/checkout@v6
-        with:
-          repository: mulgadc/mulga
-          token: ${{ steps.app-token.outputs.token }}
-          sparse-checkout: scripts/tofu-cluster
-          path: mulga
-
-      - uses: actions/checkout@v6
-        with:
-          path: spinifex
-
-      - uses: actions/setup-go@v6
-        with:
-          go-version-file: spinifex/go.mod
-          cache: false
-
-      - name: Re-enable Go toolchain auto-download
-        run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
-
-      - name: Prune rootless docker cache
-        run: |
-          docker container prune -f || true
-          docker image prune -af || true
-          docker buildx prune -af --filter "until=24h" || true
-          docker system df || true
-
-      - name: Build Install Artifacts
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          ./build-install-artifacts.sh "${GIT_BRANCH}" \
-            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
-            "${RUNNER_TEMP}/setup.sh" \
-            "${RUNNER_TEMP}/spinifex-tests.tar"
-
-      - name: Provision VM
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          set -euo pipefail
-          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          export TF_VAR_git_branch="${GIT_BRANCH}"
-          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
-          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
-          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
-          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
-          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
-          ok "VM provisioned"
-
-      - name: Bootstrap via Binary Install
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
-          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
-            "${RUNNER_TEMP}/setup.sh" \
-            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
-            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
-
-      - name: Resolve VM SSH details
-        id: vm
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          INVENTORY=$(tofu output -json inventory)
-          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
-          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
-          SSH_KEY="${SSH_KEY/#\~/$HOME}"
-          echo "wan_ip=${WAN_IP}" >> "$GITHUB_OUTPUT"
-          echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
-
-      - name: Build reboot.test on runner
-        working-directory: spinifex/tests/e2e
-        run: |
-          set -euo pipefail
-          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          banner "build reboot"
-          group_open "go test -c"
-          make e2e-build SUITES_BUILD="reboot"
-          group_close
-          ok "$(basename _bin/reboot.test) $(stat -c '%s' _bin/reboot.test | numfmt --to=iec)"
-
-      - name: Fetch AWS credentials from VM
-        id: creds
-        run: |
-          set -euo pipefail
-          SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
-                    -i "${{ steps.vm.outputs.ssh_key }}")
-          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
-          CREDS=$(ssh "${SSH_OPTS[@]}" "$VM" 'cat ~/.aws/credentials')
-          AWS_AK=$(echo "$CREDS" | awk -F '= *' '/^aws_access_key_id/ {print $2; exit}')
-          AWS_SK=$(echo "$CREDS" | awk -F '= *' '/^aws_secret_access_key/ {print $2; exit}')
-          if [ -z "$AWS_AK" ] || [ -z "$AWS_SK" ]; then
-            echo "::error::Failed to extract spinifex AWS credentials from ${{ steps.vm.outputs.wan_ip }}"
-            exit 1
-          fi
-          echo "::add-mask::${AWS_AK}"
-          echo "::add-mask::${AWS_SK}"
-          echo "ak=${AWS_AK}" >> "$GITHUB_OUTPUT"
-          echo "sk=${AWS_SK}" >> "$GITHUB_OUTPUT"
-
-      - name: Run reboot suite on runner
-        id: e2e
-        run: |
-          set +e
-          mkdir -p "${ARTIFACT_DIR}"
-          date -u +%Y-%m-%dT%H:%M:%SZ > "${ARTIFACT_DIR}/test-reboot.start"
-          R=$'\033[0m'; B=$'\033[1m'; C=$'\033[36m'
-          printf '\n%s%s━━ suite reboot ━━%s\n' "$B" "$C" "$R"
-
-          if [ -n "${TEST_RUN}" ]; then
-            TEST_RUN_FLAG="-test.run=${TEST_RUN}"
-          else
-            TEST_RUN_FLAG=""
-          fi
-          GITHUB_ACTIONS=true FORCE_COLOR=1 \
-            SPINIFEX_E2E=1 SPINIFEX_MODE=single \
-            SPINIFEX_WAN_IP="${{ steps.vm.outputs.wan_ip }}" \
-            SPINIFEX_SSH_KEY="${{ steps.vm.outputs.ssh_key }}" \
-            SPINIFEX_AWS_ENDPOINT="https://${{ steps.vm.outputs.wan_ip }}:9999" \
-            SPINIFEX_AWS_ACCESS_KEY_ID="${{ steps.creds.outputs.ak }}" \
-            SPINIFEX_AWS_SECRET_ACCESS_KEY="${{ steps.creds.outputs.sk }}" \
-            ARTIFACT_DIR="${ARTIFACT_DIR}" \
-            ./spinifex/tests/e2e/_bin/reboot.test -test.v -test.timeout=30m ${TEST_RUN_FLAG} \
-            2>&1 | tee "${ARTIFACT_DIR}/test-reboot.log" \
-                 | sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'
-          rc=${PIPESTATUS[0]}
-          echo "rc=${rc}" >> "$GITHUB_OUTPUT"
-          exit 0
-
-      - name: Collect VM journals
-        if: always()
-        run: |
-          set +e
-          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          mkdir -p "${ARTIFACT_DIR}"
-          SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 \
-                    -i "${{ steps.vm.outputs.ssh_key }}")
-          VM=tf-user@${{ steps.vm.outputs.wan_ip }}
-          ssh "${SSH_OPTS[@]}" "$VM" 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "60 min ago"' \
-            > "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || true
-          ok "journal: $(wc -l < "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || echo 0) lines"
-
-      - name: Install go-junit-report
-        if: always()
-        run: |
-          go install github.com/jstemmer/go-junit-report/v2@latest
-          echo "$HOME/go/bin" >> "$GITHUB_PATH"
-
-      - name: Convert test log to JUnit XML
-        if: always()
-        run: |
-          shopt -s nullglob
-          for LOG in "${ARTIFACT_DIR}"/test-*.log; do
-            BASE=$(basename "$LOG" .log)
-            SUITE=${BASE#test-}
-            go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
-          done
-
-      - name: Publish test summary
-        if: always()
-        uses: ./spinifex/.github/actions/junit-report
-        with:
-          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
-          title: "E2E reboot results"
-
-      - name: Analyze e2e failures
-        if: always()
-        uses: ./spinifex/.github/actions/e2e-analyze
-        with:
-          junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
-          log-dir: ${{ env.ARTIFACT_DIR }}
-          title: "Reboot E2E failure analysis"
-
-      - uses: actions/upload-artifact@v7
-        if: always()
-        with:
-          name: e2e-reboot-artifacts-${{ github.run_id }}
-          path: ${{ env.ARTIFACT_DIR }}
-          if-no-files-found: warn
-          retention-days: 14
-
-      - name: Teardown
-        if: always()
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          set +e
-          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
-          UID_SELF=$(id -u)
-          WS="${GITHUB_WORKSPACE}"
-          reap() {
-            local sig="$1"
-            for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
-              cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
-              case "$cwd" in
-                "$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;;
-              esac
-            done
-          }
-          reap TERM
-          sleep 2
-          reap KILL
-          find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
-          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
-          quiet_run "tofu destroy"          "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
-          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
-          ok "teardown complete"
-
-      - name: Fail job if reboot suite failed
-        if: always()
-        run: |
-          if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
-            echo "::error::Reboot suite failed (rc=${{ steps.e2e.outputs.rc }})"
-            exit 1
-          fi
-
   e2e_status:
     name: Report E2E Status
     if: always()
-    needs: [ e2e_single, e2e_real_multi, e2e_reboot ]
+    needs: [e2e_single, e2e_real_multi]
     runs-on: ubuntu-latest
     timeout-minutes: 5
     steps:
@@ -1138,16 +898,15 @@ jobs:
             echo "::error::no SHA available (inputs.sha and github.sha both empty)"
             exit 1
           fi
-          single_ok=true; multi_ok=true; reboot_ok=true
+          single_ok=true; multi_ok=true
           case "${{ needs.e2e_single.result }}" in success|skipped) ;; *) single_ok=false ;; esac
           case "${{ needs.e2e_real_multi.result }}" in success|skipped) ;; *) multi_ok=false ;; esac
-          case "${{ needs.e2e_reboot.result }}" in success|skipped) ;; *) reboot_ok=false ;; esac
-          if $single_ok && $multi_ok && $reboot_ok; then
+          if $single_ok && $multi_ok; then
             STATE="success"
             DESC="E2E suites passed"
           else
             STATE="failure"
-            DESC="E2E failed: single=${{ needs.e2e_single.result }} multi=${{ needs.e2e_real_multi.result }} reboot=${{ needs.e2e_reboot.result }}"
+            DESC="E2E failed: single=${{ needs.e2e_single.result }} multi=${{ needs.e2e_real_multi.result }}"
           fi
           gh api "repos/${{ github.repository }}/statuses/${REPORT_SHA}" \
             -f state="$STATE" \

.github/workflows/publish-eks-ami.yml

@@ -26,7 +26,7 @@ concurrency:
 jobs:
   publish:
     name: Build + publish eks-node AMI
-    runs-on: ubuntu-24.04
+    runs-on: [ self-hosted, ci-single ]
 
     steps:
       - name: Checkout Spinifex
@@ -51,8 +51,12 @@ jobs:
       - name: Setup Go
         uses: actions/setup-go@v6
         with:
-          cache-dependency-path: "spinifex/go.sum"
           go-version-file: "spinifex/go.mod"
+          # Self-hosted runner: the Go module + build cache already persists on
+          # local disk between runs. setup-go's cache-save ("Post Setup Go")
+          # otherwise re-uploads hundreds of MB to the Actions cache backend,
+          # taking longer than the build itself.
+          cache: false
 
       - name: Setup Go Workspace
         run: go work init ./spinifex ./viperblock ./predastore

.github/workflows/release.yml

@@ -34,7 +34,7 @@ jobs:
       - name: Set up Go
         uses: actions/setup-go@v6
         with:
-          go-version: '1.26.4'
+          go-version: "1.26.4"
 
       - name: Build microVM artifacts (lb-agent + kernel + initramfs)
         run: cd spinifex && make build-microvm-image
@@ -132,12 +132,6 @@ jobs:
     steps:
       - name: Clean Workspace
         run: |
-          # This runner (ci-single) is shared with the iso job, which mounts a
-          # kernel-global devtmpfs at <workspace>/mulga/scripts/iso-builder/.cache/rootfs/dev.
-          # If that job is killed its EXIT-trap unmount never runs and the bind
-          # leaks; a plain rm -rf then traverses it and deletes the host's /dev
-          # (wipes /dev/null, wedges the box). Detach any leaked mounts under the
-          # workspace first, then forbid rm from crossing a mount boundary.
           awk -v w="$GITHUB_WORKSPACE" '$2 ~ "^"w {print $2}' /proc/mounts \
             | sort -r | xargs -r -I{} sudo umount -l -R {} || true
           sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
@@ -186,10 +180,6 @@ jobs:
           tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
           ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
 
-  # Bare-metal single-node suite on the real ISO/squashfs + PXE-boot path —
-  # the only release check that exercises hardware install. Runs in parallel
-  # with `verify` (both need: release). Promotion and ISO upload both hard-gate
-  # on this passing.
   baremetal:
     needs: release
     permissions:
@@ -198,8 +188,6 @@ jobs:
     uses: ./.github/workflows/e2e-baremetal.yml
     secrets: inherit
 
-  # Promote the draft only after BOTH the tofu install (`verify`) and the
-  # bare-metal hardware boot (`baremetal`) pass — runs only if both succeed.
   promote:
     needs: [ verify, baremetal ]
     runs-on: ubuntu-24.04
@@ -248,8 +236,6 @@ jobs:
       - name: Clean Workspace
         run: |
           ROOTFS="${GITHUB_WORKSPACE}/mulga/scripts/iso-builder/.cache/rootfs"
-          # Mirror make clean's umount sequence unconditionally — lazy unmount
-          # detaches immediately even if the mount is busy, so rm -rf is safe.
           sudo umount -l -R "${ROOTFS}" 2>/dev/null || true
           sudo umount -l "${ROOTFS}/dev/pts" 2>/dev/null || true
           sudo umount -l "${ROOTFS}/dev" 2>/dev/null || true
@@ -335,9 +321,6 @@ jobs:
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
-          # Gate the R2 upload on BOTH the tofu install (`verify`) and the
-          # real-hardware boot (`baremetal`, a reusable workflow whose job shows
-          # as "baremetal / ..."). Upload only if both conclude success.
           echo "Waiting for verify + baremetal jobs to complete..."
           while true; do
             JOBS=$(gh api \

.github/workflows/spinifex.yml

@@ -58,6 +58,9 @@ jobs:
       - name: Build Spinifex
         run: make -C spinifex build
 
+      - name: Manifest check + drift lint
+        run: make -C spinifex manifest-check manifest-lint
+
       - name: Run Tests with Coverage
         run: make -C spinifex test-cover
 

INSTALL.md

@@ -1,3 +0,0 @@
-# Spinifex Installation
-
-For documentation on using and installing Spinifex see [https://docs.mulgadc.com](https://docs.mulgadc.com/)

Makefile

@@ -6,17 +6,12 @@ ARCH := $(shell uname -m)
 ifeq ($(ARCH),x86_64)
   GO_ARCH := amd64
   AWS_ARCH := x86_64
-  # On x86, we can run ARM VMs via emulation
-  QEMU_PACKAGES := qemu-system-x86 qemu-system-arm
 else ifeq ($(ARCH),aarch64)
   GO_ARCH := arm64
   AWS_ARCH := aarch64
-  # On ARM, we can run x86 VMs via emulation
-  QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
 else ifeq ($(ARCH),arm64)
   GO_ARCH := arm64
   AWS_ARCH := aarch64
-  QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
 else
   $(error Unsupported architecture: $(ARCH). Only x86_64 and aarch64/arm64 are supported.)
 endif
@@ -115,14 +110,10 @@ install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
 	sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
 
-go_run:
-	@echo -e "\n....Running $(GO_PROJECT_NAME)...."
-	$(GOPATH)/bin/$(GO_PROJECT_NAME)
-
 # Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
 # Use this before committing to catch CI failures locally.
 preflight:
-	@$(MAKE) --no-print-directory QUIET=1 manifest-check lint govulncheck test-cover diff-coverage test-race test-harness
+	@$(MAKE) --no-print-directory QUIET=1 manifest-check manifest-lint lint govulncheck test-cover diff-coverage test-race test-harness
 	@echo -e "\n ✅ Preflight passed — safe to commit."
 
 # E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
@@ -139,6 +130,17 @@ manifest-check:
 	@echo -e "\n....Checking service-interfaces.yaml...."
 	@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml
 
+# Drift guards (Bead 5): direct-create fixture lint + NATS subject lint,
+# ratcheted against tests/e2e/manifest-lint/baseline.txt. Fails only on NEW
+# drift beyond the baseline.
+manifest-lint:
+	@echo -e "\n....Linting manifest drift (fixtures + subjects)...."
+	@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root .
+
+# Accept current drift into the baseline. Run after an intentional change.
+manifest-lint-update:
+	@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root . -update
+
 # Run unit tests
 test:
 	@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
@@ -168,13 +170,8 @@ diff-coverage: test-cover
 
 bench:
 	@echo -e "\n....Running benchmarks for $(GO_PROJECT_NAME)...."
-	$(MAKE) easyjson
 	LOG_IGNORE=1 go test -benchmem -run=. -bench=. ./...
 
-run:
-	$(MAKE) go_build
-	$(MAKE) go_run
-
 # Fast iteration: build + install binary + restart all services.
 # Microvm artifacts are reinstalled when they already exist on disk — the rule's
 # input timestamps drive a rebuild only if anything actually changed. On a fresh
@@ -201,11 +198,10 @@ clean:
 
 install-system:
 	@echo -e "\n....Installing system dependencies for $(ARCH)...."
-	@echo "QEMU packages: $(QEMU_PACKAGES)"
 	sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
 		-o Dpkg::Options::="--force-confdef" \
 		-o Dpkg::Options::="--force-confold" \
-		nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
+		nbdkit nbdkit-plugin-dev pkg-config qemu-system-x86 qemu-system-arm qemu-utils qemu-kvm \
 		ovmf qemu-efi-aarch64 \
 		libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
 		iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
@@ -234,19 +230,15 @@ install-aws:
 quickinstall: install-system install-go install-aws
 	@echo -e "\n✅ Quickinstall complete for $(ARCH)."
 	@echo "   Please ensure /usr/local/go/bin is in your PATH."
-	@echo "   Installed: Go ($(GO_ARCH)), AWS CLI ($(AWS_ARCH)), QEMU ($(QEMU_PACKAGES))"
 
-# Lint all Go code via golangci-lint (replaces check-format, vet, gosec, staticcheck)
 lint:
 	@echo "Running golangci-lint..."
 	$(_Q)golangci-lint run ./...
 	@echo "  golangci-lint ok"
 
-# Auto-fix all linter issues that have fixers
 fix:
 	golangci-lint run --fix ./...
 
-# Govulncheck — dependency vulnerability scanning (not covered by golangci-lint)
 govulncheck:
 	@echo "Running govulncheck..."
 	$(_Q)go tool govulncheck ./...
@@ -295,65 +287,7 @@ distro-arm64:
 distro-clean:
 	rm -rf dist/
 
-# Ansible dev lifecycle (experimental, parallel to dev-*.sh / reset-dev-env.sh).
-# See scripts/ansible/README.md and docs/development/improvements/ansible-dev-lifecycle.md.
-#
-# Variable overrides: pass EXTRA_VARS="key=val key2=val2" to forward as
-# `--extra-vars` to ansible-playbook. Plain `-e` on the make command line is
-# make's own flag and does NOT reach ansible.
-#   make ansible-dev-install EXTRA_VARS="spinifex_external_pool_start=192.168.1.90 spinifex_external_pool_end=192.168.1.99"
-EXTRA_VARS ?=
-_ANSIBLE_EXTRA = $(if $(strip $(EXTRA_VARS)),--extra-vars "$(EXTRA_VARS)",)
-
-ansible-dev-preflight:
-	cd scripts/ansible && ansible-playbook playbooks/dev-preflight.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-teardown:
-	cd scripts/ansible && ansible-playbook playbooks/dev-teardown.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-install:
-	cd scripts/ansible && ansible-playbook playbooks/dev-install.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-reset:
-	cd scripts/ansible && ansible-playbook playbooks/dev-reset.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-deploy:
-	cd scripts/ansible && ansible-playbook playbooks/dev-deploy.yml $(_ANSIBLE_EXTRA)
-
-ansible-dev-status:
-	cd scripts/ansible && ansible-playbook playbooks/dev-status.yml
-
-ansible-dev-logs:
-	cd scripts/ansible && ansible-playbook playbooks/dev-logs.yml
-
-# Snapshot / restore require an explicit -e snapshot_name=<name>. The
-# wrapper passes ANSIBLE_EXTRA through so devs can run
-# `make ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=before-merge'`.
-ansible-dev-snapshot:
-	cd scripts/ansible && ansible-playbook playbooks/dev-snapshot.yml $(ANSIBLE_EXTRA)
-
-ansible-dev-restore:
-	cd scripts/ansible && ansible-playbook playbooks/dev-restore.yml $(ANSIBLE_EXTRA)
-
-ansible-dev-version:
-	cd scripts/ansible && ansible-playbook playbooks/dev-version.yml
-
-ansible-dev-vpc:
-	cd scripts/ansible && ansible-playbook playbooks/dev-vpc.yml $(ANSIBLE_EXTRA)
-
-# Multi-node cluster bootstrap against a tofu env (env1, env2, bryn, etc).
-# Inventory is generated from scripts/tofu-cluster/envs/<env>.tfvars.
-#   make ansible-cluster-bootstrap ENV=env1 POOL=192.168.1.150-192.168.1.159
-ENV ?=
-POOL ?=
-ansible-cluster-bootstrap:
-	@test -n "$(ENV)" || { echo "ENV=<env> required (e.g. env1, env2)"; exit 64; }
-	cd scripts/ansible && CLUSTER_ENV=$(ENV) ansible-playbook -i inventory/tofu.py playbooks/cluster-bootstrap.yml \
-		-e cluster_env=$(ENV) \
-		$(if $(POOL),-e cluster_external_pool=$(POOL),) \
-		$(_ANSIBLE_EXTRA)
-
-.PHONY: build build-ui build-installer build-lb-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race diff-coverage bench run test-actions test-harness manifest-check diff-coverage bench run \
+.PHONY: build build-ui build-installer build-lb-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-harness manifest-check manifest-lint manifest-lint-update \
 	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
 	lint fix govulncheck \

build/systemd/spinifex-awsgw.service

@@ -6,6 +6,7 @@ Wants=spinifex-nats.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-gw
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh

build/systemd/spinifex-daemon.service

@@ -2,66 +2,29 @@
 Description=Spinifex Daemon (VM Orchestration)
 PartOf=spinifex.target
 After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.service ovn-controller.service openvswitch-ipsec.service
-# Wants=openvswitch-ipsec.service so systemd brings the IPsec userspace
-# daemon up before spinifex flips other_config:ipsec_encapsulation=true.
-# Without that daemon, OVS holds the cert pointers unused and strongSwan
-# never gets IKE config — tunnel traffic is encrypted with no SAs and
-# silently dropped. Wants= (not Requires=) keeps daemon bootable on
-# single-node / lab hosts where the package is intentionally absent.
+# Wants (not Requires): stay bootable when openvswitch-ipsec is absent
 Wants=openvswitch-ipsec.service
-# No Wants=spinifex-nats.service: Wants= reactivates a stopped unit during
-# `systemctl restart spinifex-daemon`, which would mask the standalone-startup
-# path the Tier 1 DDIL Scenario B exercises. spinifex.target Wants= covers
-# cold boot.
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-daemon
 Group=spinifex
-# DDIL Tier 1 (siv-17): no ExecStartPre wait-for-nats. The daemon starts in
-# standalone mode and reconnects to NATS in the background — process-exiting
-# on a missing NATS would kill the local VM management plane.
-#
-# Opt-in strict-startup (§1d-strict, siv-44): set SPINIFEX_REQUIRE_NATS=1 to
-# restore pre-DDIL fail-fast UX. First NATS connect uses a ~30s bounded retry
-# and the daemon aborts on timeout. Intended for dev/test/single-node deploys
-# where silent NATS misconfig is more confusing than helpful. Do NOT set this
-# in the shipped unit — it re-introduces the SPOF that Tier 1 removed.
 ExecStart=/usr/local/bin/spx service spinifex start
 Restart=on-failure
 RestartSec=5
-# DDIL Tier 1: KillMode=process so systemd only signals the
-# daemon's main PID, not the cgroup. QEMU children stay alive across
-# `systemctl restart spinifex-daemon` and the new daemon reattaches to them
-# via the persisted local state file + pidfile + QMP socket. With the
-# default control-group mode, restart would terminate every guest VM —
-# defeating Scenario B's "daemon restart preserves running instances".
+# KillMode=process: guest QEMU survives daemon restart; the new daemon reattaches (DDIL)
 KillMode=process
-# 15s upper bound on the SIGTERM grace window. The new SIGTERM handler
-# skips VM teardown (it now just persists state and exits), so 15s is
-# plenty of slack for the cluster-manager + NATS unsubscribe path.
 TimeoutStopSec=15
+# RG-4: infra tier — reaped only after every guest
 OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
 Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex/
 Environment=SPINIFEX_WAL_DIR=/var/lib/spinifex/spinifex/
-# DDIL Tier 1: QEMU pidfiles, QMP sockets, and the
-# shared qemu.log all resolve via utils.pidPath() — which falls back
-# to os.TempDir() (=/tmp) when $HOME/spinifex doesn't exist. With
-# PrivateTmp=yes, each daemon start gets a fresh /tmp namespace, so
-# pidfiles written by QEMU (running in the OLD daemon's namespace)
-# are invisible to the new daemon. The /local/instances handler then
-# reports PID=0 and Scenario B fails its "live QEMU PID" assertion.
-# Pin XDG_RUNTIME_DIR to the host-wide tmpfs at /run/spinifex (created
-# by tmpfiles.d, already in ReadWritePaths) so pidPath() returns a
-# stable, namespace-independent path that both daemon incarnations and
-# QEMU children resolve identically.
+# Pin to host tmpfs: QEMU pidfiles/QMP sockets must survive PrivateTmp namespace churn (DDIL)
 Environment=XDG_RUNTIME_DIR=/run/spinifex
 
-# Filesystem access — NoNewPrivileges omitted (daemon uses sudo for ip, ovs-vsctl, dhcpcd)
-# CAP_SYS_ADMIN: required for GPU passthrough sysfs operations (vfio group management).
-# CAP_DAC_OVERRIDE: sysfs driver bind/unbind paths (/sys/bus/pci/drivers/vfio-pci/bind,
-# unbind) are root:root 0200; non-root processes need CAP_DAC_OVERRIDE to open them.
+# RG-9: daemon tier — privileged by necessity (GPU vfio). NoNewPrivileges omitted (RG-10: sudo ip/ovs-vsctl/dhcpcd)
 AmbientCapabilities=CAP_SYS_ADMIN CAP_DAC_OVERRIDE
 ProtectSystem=full
 PrivateDevices=no
@@ -73,7 +36,7 @@ DeviceAllow=/dev/vfio/vfio rw
 ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
 ReadWritePaths=/var/lib/spinifex/spinifex /var/log/spinifex /run/spinifex /run/spinifex/nbd
 
-# Hardening
+# RG-9: hardening baseline
 ProtectHome=yes
 PrivateTmp=yes
 RestrictSUIDSGID=yes
@@ -88,8 +51,6 @@ ProtectControlGroups=yes
 ProtectProc=invisible
 SystemCallArchitectures=native
 MemoryDenyWriteExecute=yes
-# DHCP lease acquisition moved to spinifex-vpcd (see dhcp-allocate-error.md);
-# the daemon no longer needs AF_PACKET or the softened kernel protections.
 RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
 ProtectHostname=yes
 KeyringMode=private
@@ -97,8 +58,7 @@ RemoveIPC=yes
 LimitCORE=0
 LimitNOFILE=65536
 LimitNPROC=4096
-# VFIO GPU passthrough requires locking all guest RAM into physical memory.
-# Without this, QEMU fails with RLIMIT_MEMLOCK exceeded when pinning guest pages.
+# LimitMEMLOCK=infinity: VFIO pins all guest RAM
 LimitMEMLOCK=infinity
 
 [Install]

build/systemd/spinifex-guests.slice

@@ -0,0 +1,8 @@
+[Unit]
+Description=Spinifex guest VM tier
+Before=slices.target
+
+[Slice]
+# RG-3: child of spinifex.slice (spinifex-guests -> spinifex), sibling of
+# spinifex-system. Parent for the per-instance spinifex-instance-<id>.slice
+# leaves (0002 RG-7). Inherits the spinifex.slice ceiling; no extra limit here.

build/systemd/spinifex-nats.service

@@ -5,6 +5,7 @@ Before=spinifex-predastore.service spinifex-daemon.service spinifex-awsgw.servic
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-nats
 Group=spinifex
 ExecStart=/usr/local/bin/spx service nats start

build/systemd/spinifex-predastore.service

@@ -6,6 +6,7 @@ Wants=spinifex-nats.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-storage
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh

build/systemd/spinifex-system.slice

@@ -0,0 +1,9 @@
+[Unit]
+Description=Spinifex control/infra tier (NATS, predastore, viperblock, daemon, awsgw, vpcd, ui)
+Before=slices.target
+
+[Slice]
+# RG-3: child of spinifex.slice (spinifex-system -> spinifex), inherits the
+# parent ceiling, no extra limit. Holds the 7 control/infra services. Guest QEMU
+# moves to the sibling spinifex-guests.slice (per-instance placement lands in
+# 0002 RG-7); until then the daemon-forked guests stay under spinifex-daemon.service.

build/systemd/spinifex-ui.service

@@ -5,6 +5,7 @@ After=spinifex-awsgw.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-ui
 Group=spinifex
 ExecStart=/usr/local/bin/spx service spinifex-ui start

build/systemd/spinifex-viperblock.service

@@ -6,6 +6,7 @@ Wants=spinifex-nats.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-viperblock
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh

build/systemd/spinifex-vpcd.service

@@ -6,6 +6,7 @@ Wants=spinifex-nats.service ovn-controller.service
 
 [Service]
 Type=simple
+Slice=spinifex-system.slice
 User=spinifex-vpcd
 Group=spinifex
 ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
@@ -16,45 +17,23 @@ OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
 Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/vpcd/
 
-# vpcd MUST share the HOST mount namespace. IMDS binds one listener per VPC inside
-# a per-VPC netns entered in-process via setns(CLONE_NEWNET); the /run/netns/<vpc>
-# bind-mount that `ip netns add` creates must be host-visible (for diagnostics) and
-# must survive a vpcd restart (so the listener can rebind). ANY filesystem-sandbox
-# directive — ProtectSystem / ProtectHome / PrivateTmp / ProtectKernel* /
-# ProtectControlGroups / ProtectProc / Read{Only,Write}Paths — forks a PRIVATE
-# mount namespace that traps that bind-mount inside vpcd, leaving the host a stale
-# handle that fails setns(2) with EINVAL (CI runs 26920490454, 26922587471). Those
-# directives are therefore omitted by design, and so is the MountFlags=shared /
-# spinifex-netns.service propagation workaround they required.
-#
-# Boundary that REMAINS: DAC (User=spinifex-vpcd) plus the seccomp / capability /
-# namespace / address-family restrictions below. The durable fix that restores
-# filesystem sandboxing AND drops CAP_SYS_ADMIN is the process-per-netns model
-# (`ip netns exec` helper instead of in-process setns) — see
-# docs/development/improvements/imds-l2-datapath.md Phase 5.
+# RG-9: network tier. vpcd MUST share the host mount namespace — IMDS enters
+# per-VPC netns via setns(), and any filesystem-sandbox directive (ProtectSystem/
+# Home, PrivateTmp, ProtectKernel*, ProtectControlGroups/Proc, Read*Paths) forks a
+# private mount ns that traps the /run/netns bind-mount and breaks setns(2). Those,
+# plus MemoryDenyWriteExecute, are omitted by design — not regressions.
 PrivateDevices=no
-
-# Hardening that does NOT fork a mount namespace (kept)
 RestrictSUIDSGID=no
 LockPersonality=yes
 RestrictRealtime=yes
 ProtectClock=yes
-# IMDS enters the per-VPC netns via setns(CLONE_NEWNET); the `ip -n <netns>` /
-# `ip netns exec` plumbing also unshares a mount namespace (CLONE_NEWNS, to give
-# the netns its own /sys), so `mnt` must be allowlisted alongside `net` or the
-# first in-netns command fails with EPERM.
+# mnt alongside net: in-netns `ip` unshares a mount ns for its own /sys
 RestrictNamespaces=net mnt
 SystemCallArchitectures=native
 MemoryDenyWriteExecute=no
-# AF_PACKET: required by the in-process DHCP client (github.com/insomniacslk/dhcp
-# / nclient4) for raw L2 BOOTP on br-wan before an IP is configured on the
-# interface. See docs/development/bugs/dhcp-allocate-error.md.
+# AF_PACKET: raw L2 BOOTP for the in-process DHCP client before the iface has an IP
 RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK AF_PACKET
-# CAP_NET_RAW: open AF_PACKET sockets.
-# CAP_NET_BIND_SERVICE: covers unicast RENEW to UDP :68.
-# CAP_NET_ADMIN: already used for ovs-vsctl / ip link wiring on the bridge.
-# CAP_SYS_ADMIN: setns(CLONE_NEWNET) into per-VPC IMDS netns to bind the listener.
-# Note, disabled for now, sudo issue with ovs
+# Caps: NET_ADMIN (ovs/ip), NET_RAW (AF_PACKET), NET_BIND_SERVICE (:68), SETUID/SETGID, SYS_ADMIN (setns into VPC netns)
 AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID CAP_SYS_ADMIN
 CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID CAP_SYS_ADMIN
 NoNewPrivileges=no

build/systemd/spinifex.slice

@@ -0,0 +1,13 @@
+[Unit]
+Description=Spinifex resource-bounded slice (host ceiling)
+Before=slices.target
+
+[Slice]
+# RG-1: no CPUWeight — slice roots are siblings; capping below the default would
+# let user.slice out-prioritize the VM fleet. Operator CPU priority lives on
+# system.slice (system.slice.d/spinifex-reserve.conf).
+# RG-1/RG-5: throttle+reclaim at High, hard wall at Max (% of physical RAM).
+MemoryHigh=80%
+MemoryMax=90%
+# RG-5: spill to the full swap device before the MemoryMax wall; matched to swap size.
+MemorySwapMax=8G

cmd/eks-gateway-fetch/main.go

@@ -0,0 +1,139 @@
+// eks-gateway-fetch runs inside the EKS K3s control-plane VM and fetches a
+// control-plane resource from the host through the AWS gateway, instead of
+// dialing core NATS directly. It is the read-side companion to
+// eks-gateway-publish.
+//
+// It SigV4-signs (service "eks") an HTTPS GET to the gateway, retrying with
+// backoff until the gateway returns 2xx or the attempt budget is exhausted (so
+// a degraded link surfaces as a non-zero exit), then emits the staged add-ons
+// as tab-separated lines the on-VM addon-sync shell agent consumes without a
+// JSON parser:
+//
+//	<addonName>\t<addonVersion>\t<serviceAccountRoleArn>\t<base64(configurationValues)>
+//
+// The configuration values are base64-encoded so embedded tabs/newlines cannot
+// break the line framing.
+//
+// Usage:
+//
+//	eks-gateway-fetch -resource addons   # GET /clusters/{cluster}/internal-addons/{accountId}
+//
+// Flags default to environment variables seeded by cloud-init:
+// EKS_GATEWAY_URL, EKS_GATEWAY_CA, EKS_ACCESS_KEY, EKS_SECRET_KEY, EKS_REGION,
+// EKS_ACCOUNT_ID, EKS_CLUSTER_NAME.
+package main
+
+import (
+	"bufio"
+	"encoding/base64"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"log/slog"
+	"os"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/eksgw"
+
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
+)
+
+// stagedAddon mirrors the gateway's StagedAddonManifest wire shape; duplicated
+// here to keep this tiny VM binary free of the handler package's dependency
+// graph.
+type stagedAddon struct {
+	AddonName             string `json:"addonName"`
+	AddonVersion          string `json:"addonVersion"`
+	ServiceAccountRoleArn string `json:"serviceAccountRoleArn"`
+	ConfigurationValues   string `json:"configurationValues"`
+}
+
+type internalAddonsResponse struct {
+	Addons []stagedAddon `json:"addons"`
+}
+
+const (
+	maxAttempts = 30
+	retryDelay  = 5 * time.Second
+)
+
+func main() {
+	var (
+		gatewayURL  string
+		gatewayCA   string
+		accessKey   string
+		secretKey   string
+		region      string
+		accountID   string
+		clusterName string
+		resource    string
+	)
+
+	flag.StringVar(&gatewayURL, "gateway", os.Getenv("EKS_GATEWAY_URL"), "Gateway URL (e.g. https://10.15.8.1:9999)")
+	flag.StringVar(&gatewayCA, "gateway-ca", os.Getenv("EKS_GATEWAY_CA"), "Path to gateway TLS CA PEM (optional; falls back to system trust)")
+	flag.StringVar(&accessKey, "access-key", os.Getenv("EKS_ACCESS_KEY"), "AWS access key ID")
+	flag.StringVar(&secretKey, "secret-key", os.Getenv("EKS_SECRET_KEY"), "AWS secret access key")
+	flag.StringVar(&region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
+	flag.StringVar(&accountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
+	flag.StringVar(&clusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
+	flag.StringVar(&resource, "resource", "addons", "Resource to fetch: addons")
+	flag.Parse()
+
+	switch {
+	case accountID == "":
+		fatal("--account-id is required (or set EKS_ACCOUNT_ID)")
+	case clusterName == "":
+		fatal("--cluster is required (or set EKS_CLUSTER_NAME)")
+	case resource != "addons":
+		fatal("--resource must be addons")
+	}
+
+	client, err := eksgw.New(gatewayURL, gatewayCA, accessKey, secretKey, region)
+	if err != nil {
+		fatal(fmt.Sprintf("build gateway client: %v", err))
+	}
+	path := "/clusters/" + clusterName + "/internal-addons/" + accountID
+
+	var lastErr error
+	for attempt := 1; attempt <= maxAttempts; attempt++ {
+		body, err := client.Get(path)
+		if err != nil {
+			lastErr = err
+			slog.Warn("eks-gateway-fetch: attempt failed", "resource", resource, "attempt", attempt, "err", err)
+			if attempt < maxAttempts {
+				time.Sleep(retryDelay)
+			}
+			continue
+		}
+		if werr := emitAddonsTSV(os.Stdout, body); werr != nil {
+			fatal(fmt.Sprintf("render response: %v", werr))
+		}
+		return
+	}
+	fatal(fmt.Sprintf("fetch failed after %d attempts: %v", maxAttempts, lastErr))
+}
+
+// emitAddonsTSV decodes the internal-addons response and writes one TSV line per
+// staged add-on to w. An empty add-on set writes nothing (the agent then GCs
+// every locally-rendered manifest), which is the correct steady state for a
+// cluster with no managed add-ons.
+func emitAddonsTSV(out io.Writer, body []byte) error {
+	var resp internalAddonsResponse
+	if err := json.Unmarshal(body, &resp); err != nil {
+		return fmt.Errorf("unmarshal internal-addons response: %w", err)
+	}
+	w := bufio.NewWriter(out)
+	for _, a := range resp.Addons {
+		cfg := base64.StdEncoding.EncodeToString([]byte(a.ConfigurationValues))
+		if _, err := fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", a.AddonName, a.AddonVersion, a.ServiceAccountRoleArn, cfg); err != nil {
+			return err
+		}
+	}
+	return w.Flush()
+}
+
+func fatal(msg string) {
+	slog.Error("eks-gateway-fetch: " + msg)
+	os.Exit(1)
+}

cmd/eks-gateway-fetch/main_test.go

@@ -0,0 +1,44 @@
+package main
+
+import (
+	"bytes"
+	"encoding/base64"
+	"testing"
+)
+
+func TestEmitAddonsTSV(t *testing.T) {
+	body := []byte(`{"addons":[
+		{"addonName":"spinifex-noop","addonVersion":"0.1.0"},
+		{"addonName":"aws-load-balancer-controller","addonVersion":"2.8.1",
+		 "serviceAccountRoleArn":"arn:aws:iam::111122223333:role/alb",
+		 "configurationValues":"{\"replicaCount\":2}"}
+	]}`)
+
+	var buf bytes.Buffer
+	if err := emitAddonsTSV(&buf, body); err != nil {
+		t.Fatalf("emitAddonsTSV: %v", err)
+	}
+	cfg := base64.StdEncoding.EncodeToString([]byte(`{"replicaCount":2}`))
+	want := "spinifex-noop\t0.1.0\t\t\n" +
+		"aws-load-balancer-controller\t2.8.1\tarn:aws:iam::111122223333:role/alb\t" + cfg + "\n"
+	if buf.String() != want {
+		t.Fatalf("TSV mismatch:\n got: %q\nwant: %q", buf.String(), want)
+	}
+}
+
+func TestEmitAddonsTSV_Empty(t *testing.T) {
+	var buf bytes.Buffer
+	if err := emitAddonsTSV(&buf, []byte(`{"addons":[]}`)); err != nil {
+		t.Fatalf("emitAddonsTSV: %v", err)
+	}
+	if buf.Len() != 0 {
+		t.Fatalf("empty addon set must emit nothing, got %q", buf.String())
+	}
+}
+
+func TestEmitAddonsTSV_BadJSON(t *testing.T) {
+	var buf bytes.Buffer
+	if err := emitAddonsTSV(&buf, []byte(`{not-json`)); err == nil {
+		t.Fatal("expected error on malformed JSON")
+	}
+}

cmd/eks-gateway-publish/main.go

@@ -69,7 +69,7 @@ func main() {
 	flag.StringVar(&region, "region", os.Getenv("EKS_REGION"), "AWS region for SigV4 signing")
 	flag.StringVar(&accountID, "account-id", os.Getenv("EKS_ACCOUNT_ID"), "Cluster account ID")
 	flag.StringVar(&clusterName, "cluster", os.Getenv("EKS_CLUSTER_NAME"), "Cluster name")
-	flag.StringVar(&channel, "channel", "", "Publish channel: bootstrap|state")
+	flag.StringVar(&channel, "channel", "", "Publish channel: bootstrap|state|addon")
 	flag.StringVar(&kind, "kind", "", "Bootstrap subject kind (bootstrap channel only)")
 	flag.Parse()
 
@@ -78,8 +78,8 @@ func main() {
 		fatal("--account-id is required (or set EKS_ACCOUNT_ID)")
 	case clusterName == "":
 		fatal("--cluster is required (or set EKS_CLUSTER_NAME)")
-	case channel != "bootstrap" && channel != "state":
-		fatal("--channel must be bootstrap or state")
+	case channel != "bootstrap" && channel != "state" && channel != "addon":
+		fatal("--channel must be bootstrap, state or addon")
 	case channel == "bootstrap" && kind == "":
 		fatal("--kind is required for the bootstrap channel")
 	}

cmd/spinifex/cmd/admin.go

@@ -115,6 +115,16 @@ Each phase waits for all nodes to ACK before proceeding to the next.`,
 	Run: runClusterShutdown,
 }
 
+var clusterDrainDHCPCmd = &cobra.Command{
+	Use:   "drain-dhcp",
+	Short: "Release all upstream DHCP leases held by vpcd",
+	Long: `Ask each vpcd to DHCPRELEASE every external-pool DHCP lease it currently
+holds, returning them to the upstream DHCP server. Run this on teardown before
+stopping services: an env reset otherwise strands held leases on the upstream
+server until their TTL expires, eventually exhausting the upstream scope.`,
+	Run: runClusterDrainDHCP,
+}
+
 var adminInitCmd = &cobra.Command{
 	Use:   "init",
 	Short: "Initialize Spinifex platform configuration",
@@ -245,6 +255,9 @@ func init() {
 	clusterShutdownCmd.Flags().Duration("timeout", 120*time.Second, "Maximum time to wait per phase")
 	clusterShutdownCmd.Flags().Bool("dry-run", false, "Print phase plan without executing")
 
+	clusterCmd.AddCommand(clusterDrainDHCPCmd)
+	clusterDrainDHCPCmd.Flags().Duration("timeout", 30*time.Second, "Reply-collection window for vpcd drain responders")
+
 	adminCmd.AddCommand(imagesCmd)
 	imagesCmd.AddCommand(imagesImportCmd)
 	imagesCmd.AddCommand(imagesListCmd)
@@ -1114,6 +1127,17 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	fmt.Println("\n🔐 Generated predastore encryption key (per-node, never transmitted)")
 	fmt.Printf("   Key: %s\n", predastoreKeyPath)
 
+	// Viperblock at-rest encryption key is cluster-wide; on a single-node init
+	// there are no joiners, so generate it locally and enable encryption by
+	// default for all volumes created on this install.
+	viperblockKeyPath, err := writeViperblockEncryptionKey(configDir)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error generating viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	fmt.Println("\n🔐 Generated viperblock at-rest encryption key")
+	fmt.Printf("   Key: %s\n", viperblockKeyPath)
+
 	fmt.Printf("\n🔑 Generated admin credentials (save these — they won't be shown again):\n")
 	fmt.Printf("   Access Key:  %s\n", bootstrapResult.AdminAccessKey)
 	fmt.Printf("   Secret Key:  %s\n", bootstrapResult.AdminSecretKey)
@@ -1301,6 +1325,8 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 
 		GPUPassthrough: gpuPassthrough,
 		IPSecEnabled:   ipsecEnabled,
+
+		EncryptionKeyFile: viperblockKeyPath,
 	}
 
 	// Print external networking summary
@@ -1409,6 +1435,22 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	fmt.Println("\n🔐 Generated predastore encryption key (per-node, never transmitted)")
 	fmt.Printf("   Key: %s\n", predastoreKeyPath)
 
+	// Viperblock at-rest encryption key is cluster-wide and shared: the leader
+	// generates it once and distributes it to joiners via the formation server
+	// so a volume sealed on any node can be opened on any other.
+	viperblockKey, err := handlers_iam.GenerateMasterKey()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "❌ Error generating viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	viperblockKeyPath, err := saveViperblockEncryptionKey(configDir, viperblockKey)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "❌ Error saving viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	fmt.Println("\n🔐 Generated viperblock at-rest encryption key (cluster-wide, shared with joiners)")
+	fmt.Printf("   Key: %s\n", viperblockKeyPath)
+
 	// Read CA cert/key for distribution to joining nodes
 	caCertData, err := os.ReadFile(filepath.Join(configDir, "ca.pem"))
 	if err != nil {
@@ -1437,6 +1479,9 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	// Include master key in formation server for distribution to joining nodes
 	fs.SetMasterKey(base64.StdEncoding.EncodeToString(masterKey))
 
+	// Distribute the cluster-wide viperblock encryption key to joiners
+	fs.SetViperblockKey(base64.StdEncoding.EncodeToString(viperblockKey))
+
 	// Register self (init node) as the first node
 	selfNode := formation.NodeInfo{
 		Name:        node,
@@ -1544,6 +1589,8 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 
 		OperatorEmail: email,
 
+		EncryptionKeyFile: viperblockKeyPath,
+
 		// Init node runs ovn-central locally
 		OVNNBAddr: "tcp:127.0.0.1:6641",
 		OVNSBAddr: "tcp:127.0.0.1:6642",
@@ -1585,7 +1632,7 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	}
 	fmt.Println("\n📋 Next steps:")
 	fmt.Println("   1. Start services on ALL nodes:")
-	fmt.Println("      ./scripts/start-dev.sh")
+	fmt.Println("      sudo systemctl start spinifex.target")
 	fmt.Println()
 }
 
@@ -1889,6 +1936,28 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 	}
 	fmt.Printf("✅ Predastore encryption key generated: %s\n", predastoreKeyPath)
 
+	// Viperblock at-rest encryption key is cluster-wide: receive it from the
+	// leader rather than generating one, so this node can open volumes sealed
+	// elsewhere. Lenient on absence — an older leader that predates key
+	// distribution returns nothing; fall back to cleartext rather than fail
+	// the join.
+	var viperblockKeyPath string
+	if statusResp.ViperblockKey == "" {
+		fmt.Println("⚠️  Leader did not provide a viperblock encryption key; at-rest encryption disabled on this node")
+	} else {
+		viperblockKeyBytes, err := base64.StdEncoding.DecodeString(statusResp.ViperblockKey)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "❌ Error decoding viperblock encryption key: %v\n", err)
+			os.Exit(1)
+		}
+		viperblockKeyPath, err = saveViperblockEncryptionKey(configDir, viperblockKeyBytes)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "❌ Error saving viperblock encryption key: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Printf("✅ Viperblock encryption key received from leader: %s\n", viperblockKeyPath)
+	}
+
 	// Generate server cert signed by CA with this node's bind IP
 	if err := admin.GenerateServerCertOnly(configDir, bindIP); err != nil {
 		fmt.Fprintf(os.Stderr, "Error generating server certificate: %v\n", err)
@@ -1980,6 +2049,8 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 
 		OperatorEmail: email,
 
+		EncryptionKeyFile: viperblockKeyPath,
+
 		// Joining nodes connect to the init node's OVN NB/SB DB
 		OVNNBAddr: fmt.Sprintf("tcp:%s:6641", leaderIP),
 		OVNSBAddr: fmt.Sprintf("tcp:%s:6642", leaderIP),
@@ -2265,6 +2336,7 @@ func runCertRenew(cmd *cobra.Command, _ []string) {
 type configDirs struct {
 	AWSGW      string
 	Predastore string
+	Viperblock string
 	NATS       string
 	Spinifex   string
 }
@@ -2274,10 +2346,11 @@ func createConfigSubdirs(configDir string) (configDirs, error) {
 	dirs := configDirs{
 		AWSGW:      filepath.Join(configDir, "awsgw"),
 		Predastore: filepath.Join(configDir, "predastore"),
+		Viperblock: filepath.Join(configDir, "viperblock"),
 		NATS:       filepath.Join(configDir, "nats"),
 		Spinifex:   filepath.Join(configDir, "spinifex"),
 	}
-	for _, dir := range []string{dirs.AWSGW, dirs.Predastore, dirs.NATS, dirs.Spinifex} {
+	for _, dir := range []string{dirs.AWSGW, dirs.Predastore, dirs.Viperblock, dirs.NATS, dirs.Spinifex} {
 		if err := os.MkdirAll(dir, 0700); err != nil {
 			return configDirs{}, fmt.Errorf("create directory %s: %w", dir, err)
 		}
@@ -2399,6 +2472,43 @@ func writePredastoreEncryptionKey(configDir string) (string, error) {
 	return keyPath, nil
 }
 
+// writeViperblockEncryptionKey generates a fresh 32-byte AES-256 master key for
+// viperblock at-rest encryption and writes it to
+// <configDir>/viperblock/encryption.key with mode 0600. Unlike the predastore
+// key, this key is cluster-wide and shared: the leader generates it once at init
+// and distributes it to joiners via the formation server, so a volume sealed on
+// any node can be opened on any other. Loaded at startup via masterkey.LoadShared.
+func writeViperblockEncryptionKey(configDir string) (string, error) {
+	viperblockDir := filepath.Join(configDir, "viperblock")
+	if err := os.MkdirAll(viperblockDir, 0750); err != nil {
+		return "", fmt.Errorf("create viperblock config dir: %w", err)
+	}
+	key, err := handlers_iam.GenerateMasterKey()
+	if err != nil {
+		return "", fmt.Errorf("generate viperblock encryption key: %w", err)
+	}
+	keyPath := filepath.Join(viperblockDir, "encryption.key")
+	if err := handlers_iam.SaveMasterKey(keyPath, key); err != nil {
+		return "", fmt.Errorf("save viperblock encryption key: %w", err)
+	}
+	return keyPath, nil
+}
+
+// saveViperblockEncryptionKey writes an already-generated 32-byte viperblock
+// master key (received from the formation leader) to
+// <configDir>/viperblock/encryption.key with mode 0600.
+func saveViperblockEncryptionKey(configDir string, key []byte) (string, error) {
+	viperblockDir := filepath.Join(configDir, "viperblock")
+	if err := os.MkdirAll(viperblockDir, 0750); err != nil {
+		return "", fmt.Errorf("create viperblock config dir: %w", err)
+	}
+	keyPath := filepath.Join(viperblockDir, "encryption.key")
+	if err := handlers_iam.SaveMasterKey(keyPath, key); err != nil {
+		return "", fmt.Errorf("save viperblock encryption key: %w", err)
+	}
+	return keyPath, nil
+}
+
 // writeSystemCredentials writes the system access key to a plaintext JSON file.
 // The daemon reads this at startup to inject credentials into ALB VM cloud-init
 // for SigV4-authenticated communication with the AWS gateway.

cmd/spinifex/cmd/admin_test.go

@@ -252,6 +252,74 @@ func TestWritePredastoreEncryptionKey(t *testing.T) {
 	assert.NotEqual(t, key1, key2, "per-node keys must differ")
 }
 
+// The viperblock key gates at-rest encryption for every volume on a fresh
+// install. masterkey.LoadShared rejects anything that isn't exactly 32 bytes at
+// 0640-or-stricter, so a regression in size/mode would silently fall back to
+// cleartext (empty path) or fail service startup.
+func TestWriteViperblockEncryptionKey(t *testing.T) {
+	configDir := t.TempDir()
+
+	keyPath, err := writeViperblockEncryptionKey(configDir)
+	require.NoError(t, err)
+	assert.Equal(t, filepath.Join(configDir, "viperblock", "encryption.key"), keyPath)
+
+	info, err := os.Stat(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, int64(32), info.Size(), "viperblock master key must be exactly 32 bytes")
+	assert.Equal(t, os.FileMode(0600), info.Mode().Perm(), "viperblock master key must be mode 0600")
+}
+
+// Joiners persist the leader's shared key via saveViperblockEncryptionKey
+// against a configDir with no viperblock/ subdir yet. The bytes round-trip
+// verbatim — the whole cluster shares one key, so any mutation would orphan
+// volumes sealed on other nodes.
+func TestSaveViperblockEncryptionKey_RoundTrip(t *testing.T) {
+	configDir := t.TempDir()
+	key := make([]byte, 32)
+	for i := range key {
+		key[i] = byte(i)
+	}
+
+	keyPath, err := saveViperblockEncryptionKey(configDir, key)
+	require.NoError(t, err)
+	assert.Equal(t, filepath.Join(configDir, "viperblock", "encryption.key"), keyPath)
+
+	got, err := os.ReadFile(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, key, got, "shared key must be written verbatim")
+}
+
+// A fresh install must render encryption_key_file so viperblockd enables
+// at-rest encryption; an empty field (upgrade / legacy) must omit the line
+// entirely so the volume stays in cleartext legacy mode.
+func TestSpinifexTomlTemplate_EncryptionKeyFile(t *testing.T) {
+	base := admin.ConfigSettings{
+		Node: "node1", Az: "ap-southeast-2a", Port: "4432", Region: "ap-southeast-2",
+		BindIP: "0.0.0.0", AccessKey: "AKIATEST", SecretKey: "SECRET",
+		AccountID: "123456789012", NatsToken: "token",
+		OVNNBAddr: "tcp:127.0.0.1:6641", OVNSBAddr: "tcp:127.0.0.1:6642",
+	}
+
+	dir := t.TempDir()
+	withKey := base
+	withKey.ConfigDir = dir
+	withKey.EncryptionKeyFile = "/etc/spinifex/viperblock/encryption.key"
+	path := filepath.Join(dir, "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, withKey))
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	assert.Contains(t, string(data), `encryption_key_file = "/etc/spinifex/viperblock/encryption.key"`)
+
+	dir2 := t.TempDir()
+	noKey := base
+	noKey.ConfigDir = dir2
+	path2 := filepath.Join(dir2, "spinifex.toml")
+	require.NoError(t, admin.GenerateConfigFile(path2, spinifexTomlTemplate, noKey))
+	data2, err := os.ReadFile(path2)
+	require.NoError(t, err)
+	assert.NotContains(t, string(data2), "encryption_key_file", "blank key must omit the field")
+}
+
 // Joiners run `spx admin join` against a configDir that doesn't yet contain
 // a predastore/ subdir. The helper must create it (predastore subdir is
 // owned by spinifex-storage in prod, but the helper just needs the dir to

cmd/spinifex/cmd/cluster.go

@@ -8,6 +8,7 @@ import (
 	"time"
 
 	"github.com/mulgadc/spinifex/spinifex/daemon"
+	"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
 	"github.com/nats-io/nats.go"
 	"github.com/spf13/cobra"
 )
@@ -160,6 +161,79 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
 	fmt.Printf("Cluster shutdown complete (%s)\n", time.Since(start).Round(time.Millisecond))
 }
 
+// runClusterDrainDHCP asks every vpcd to DHCPRELEASE all external-pool leases
+// it currently holds, returning them to the upstream DHCP server. Intended for
+// the teardown path: an env reset otherwise strands held leases until TTL
+// because vpcd's Stop() preserves them for adopt-on-reboot. Best-effort — if
+// the cluster is already partly down it warns and exits 0 so teardown proceeds.
+func runClusterDrainDHCP(cmd *cobra.Command, args []string) {
+	timeout, _ := cmd.Flags().GetDuration("timeout")
+	if timeout <= 0 {
+		timeout = 30 * time.Second
+	}
+
+	_, nc, err := loadConfigAndConnect()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: drain-dhcp could not connect (cluster may be down): %v\n", err)
+		return
+	}
+	defer nc.Close()
+
+	total, responders := drainDHCPLeases(nc, timeout)
+	fmt.Printf("DHCP drain complete: released %d lease(s) from %d vpcd responder(s)\n", total, responders)
+}
+
+// drainDHCPLeases publishes a drain request to vpc.dhcp.drain and collects
+// replies until timeout — one responder per AZ (vpcd uses a per-AZ queue
+// group). Returns the summed released count and the number of responders.
+func drainDHCPLeases(nc *nats.Conn, timeout time.Duration) (released, responders int) {
+	reqData, err := json.Marshal(struct{}{})
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		return 0, 0
+	}
+
+	inbox := nats.NewInbox()
+	sub, err := nc.SubscribeSync(inbox)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: drain-dhcp subscribe failed: %v\n", err)
+		return 0, 0
+	}
+	defer sub.Unsubscribe()
+
+	if err := nc.PublishRequest(dhcp.TopicDrain, inbox, reqData); err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: drain-dhcp publish failed: %v\n", err)
+		return 0, 0
+	}
+	nc.Flush()
+
+	deadline := time.Now().Add(timeout)
+	for {
+		remaining := time.Until(deadline)
+		if remaining <= 0 {
+			break
+		}
+		msg, err := sub.NextMsg(remaining)
+		if err != nil {
+			break
+		}
+		var reply struct {
+			Released int    `json:"released"`
+			Error    string `json:"error,omitempty"`
+		}
+		if err := json.Unmarshal(msg.Data, &reply); err != nil {
+			continue
+		}
+		responders++
+		if reply.Error != "" {
+			fmt.Fprintf(os.Stderr, "  vpcd drain error: %s\n", reply.Error)
+			continue
+		}
+		released += reply.Released
+	}
+	return released, responders
+}
+
 // collectShutdownACKs publishes a shutdown request and collects ACKs from nodes.
 func collectShutdownACKs(nc *nats.Conn, topic string, reqData []byte, nodeCount int, timeout time.Duration) ([]daemon.ShutdownACK, error) {
 	inbox := nats.NewInbox()

cmd/spinifex/cmd/cluster_test.go

@@ -0,0 +1,58 @@
+package cmd
+
+import (
+	"encoding/json"
+	"testing"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/network/external/dhcp"
+	"github.com/mulgadc/spinifex/spinifex/testutil"
+	"github.com/nats-io/nats.go"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// fakeDrainResponder subscribes a stand-in vpcd to vpc.dhcp.drain that
+// replies with the given released count (or error), mirroring
+// Manager.handleDrainMsg without a real lease store.
+func fakeDrainResponder(t *testing.T, nc *nats.Conn, released int, errMsg string) {
+	t.Helper()
+	sub, err := nc.Subscribe(dhcp.TopicDrain, func(msg *nats.Msg) {
+		body, _ := json.Marshal(struct {
+			Released int    `json:"released"`
+			Error    string `json:"error,omitempty"`
+		}{Released: released, Error: errMsg})
+		_ = msg.Respond(body)
+	})
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = sub.Unsubscribe() })
+}
+
+func TestDrainDHCPLeasesSumsReleased(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+	fakeDrainResponder(t, nc, 7, "")
+
+	released, responders := drainDHCPLeases(nc, 2*time.Second)
+	assert.Equal(t, 7, released)
+	assert.Equal(t, 1, responders)
+}
+
+func TestDrainDHCPLeasesReportsResponderError(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+	fakeDrainResponder(t, nc, 0, "list leases for drain: boom")
+
+	// An erroring responder still counts as a responder but contributes
+	// zero released leases.
+	released, responders := drainDHCPLeases(nc, 2*time.Second)
+	assert.Equal(t, 0, released)
+	assert.Equal(t, 1, responders)
+}
+
+func TestDrainDHCPLeasesNoResponders(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+
+	// No vpcd subscribed: the collection window elapses with zero replies.
+	released, responders := drainDHCPLeases(nc, 200*time.Millisecond)
+	assert.Equal(t, 0, released)
+	assert.Equal(t, 0, responders)
+}

cmd/spinifex/cmd/templates/spinifex.toml

@@ -93,6 +93,9 @@ token = "{{.NatsToken}}"
 
 [nodes.{{.Node}}.viperblock]
 shardwal = false
+{{- if .EncryptionKeyFile}}
+encryption_key_file = "{{.EncryptionKeyFile}}"
+{{- end}}
 
 [nodes.{{.Node}}.vpcd]
 ovn_nb_addr = "{{.OVNNBAddr}}"

docs/COMMANDS.md

@@ -65,6 +65,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 | Command | Flags | Description |
 |---------|-------|-------------|
 | `spx admin cluster shutdown` | `--force` (shutdown even if nodes don't respond), `--timeout` (max wait per phase, default 120s), `--dry-run` (print phase plan without executing) | Performs coordinated, phased shutdown of entire cluster. Phases execute in order: GATE (stop API/UI) → DRAIN (stop VMs) → STORAGE (stop viperblock) → PERSIST (stop predastore) → INFRA (stop NATS/daemon). Each phase waits for all nodes to ACK before proceeding. Uses JetStream state tracking. |
+| `spx admin cluster drain-dhcp` | `--timeout` (reply-collection window, default 30s) | Asks each vpcd to DHCPRELEASE every external-pool DHCP lease it currently holds, returning them to the upstream DHCP server. Run on teardown before stopping services — an env reset otherwise strands held leases upstream until their TTL expires, eventually exhausting the upstream scope. Best-effort: warns and exits 0 if the cluster is already down. |
 
 ### Certificate Management
 
@@ -183,7 +184,7 @@ input but treat it as `uefi` at launch.
 | `attach-volume` | `--volume-id`, `--instance-id`, `--device` (auto-assigns `/dev/sd[f-p]`) | `--dry-run` | **DONE** |
 | `detach-volume` | `--volume-id`, `--instance-id` (optional), `--device`, `--force` | `--dry-run` | **DONE** |
 | `describe-volume-status` | `--volume-ids`, `--filters` (volume-id, volume-status.status, availability-zone) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
-| `describe-volumes-modifications` | — | `--volume-ids`, `--filters`, `--max-results` | **NOT STARTED** |
+| `describe-volumes-modifications` | `--volume-ids`, `--filters` (modification-state, original-iops, original-size, original-volume-type, start-time, target-iops, target-size, target-volume-type, volume-id) | `--max-results`, `--next-token`, `--dry-run` | **DONE** |
 
 ### EC2 — Snapshots
 
@@ -488,6 +489,7 @@ bypasses policy evaluation entirely.
 | `create-policy` | `--policy-name`, `--policy-document`, `--path`, `--description` | `--tags` | **DONE** |
 | `get-policy` | `--policy-arn` | — | **DONE** |
 | `get-policy-version` | `--policy-arn`, `--version-id` | — | **DONE** |
+| `list-policy-versions` | `--policy-arn` | `--max-items`, `--marker` | **DONE** |
 | `list-policies` | — | `--scope`, `--only-attached`, `--path-prefix`, `--max-items`, `--marker` | **DONE** |
 | `delete-policy` | `--policy-arn` | — | **DONE** |
 | `attach-user-policy` | `--user-name`, `--policy-arn` | — | **DONE** |
@@ -507,6 +509,8 @@ bypasses policy evaluation entirely.
 | `attach-role-policy` | `--role-name`, `--policy-arn` | — | **DONE** |
 | `detach-role-policy` | `--role-name`, `--policy-arn` | — | **DONE** |
 | `list-attached-role-policies` | `--role-name`, `--path-prefix` | `--max-items`, `--marker` | **DONE** |
+| `list-role-policies` | `--role-name` | `--max-items`, `--marker` | **DONE** (gateway stub — inline policies unsupported, always returns empty) |
+| `get-role-policy` / `put-role-policy` / `delete-role-policy` | — | inline role policies | **NOT STARTED** |
 
 ### IAM — Instance Profiles
 
@@ -520,6 +524,19 @@ bypasses policy evaluation entirely.
 | `add-role-to-instance-profile` | `--instance-profile-name`, `--role-name` | — | **DONE** |
 | `remove-role-from-instance-profile` | `--instance-profile-name`, `--role-name` | — | **DONE** |
 
+### IAM — OIDC Providers
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `create-open-id-connect-provider` | `--url`, `--client-id-list`, `--thumbprint-list`, `--tags` | — | **DONE** |
+| `get-open-id-connect-provider` | `--open-id-connect-provider-arn` | — | **DONE** |
+| `list-open-id-connect-providers` | — | — | **DONE** |
+| `delete-open-id-connect-provider` | `--open-id-connect-provider-arn` | — | **DONE** |
+| `add-client-id-to-open-id-connect-provider` | — | `--open-id-connect-provider-arn`, `--client-id` | **NOT STARTED** |
+| `remove-client-id-from-open-id-connect-provider` | — | `--open-id-connect-provider-arn`, `--client-id` | **NOT STARTED** |
+| `update-open-id-connect-provider-thumbprint` | — | `--open-id-connect-provider-arn`, `--thumbprint-list` | **NOT STARTED** |
+| `tag-open-id-connect-provider` / `untag-open-id-connect-provider` | — | `--open-id-connect-provider-arn`, `--tags`/`--tag-keys` | **NOT STARTED** |
+
 ### IAM — Groups
 
 | Command | Implemented Flags | Missing Flags | Status |
@@ -544,16 +561,13 @@ bypasses policy evaluation entirely.
 | `get-caller-identity` | — | — | **DONE** |
 | `assume-role` | `--role-arn`, `--role-session-name`, `--duration-seconds` (900–min(role MaxSessionDuration, 43200)) | `--policy`, `--policy-arns` (→ `PackedPolicyTooLarge`); `--tags`, `--transitive-tag-keys` (→ `InvalidParameterValue`); `--serial-number`, `--token-code` (→ `InvalidParameterValue`); `--external-id`, `--source-identity` (accepted and logged, **not enforced** — no Condition evaluator in v1) | **DONE** |
 | `get-session-token` | `--duration-seconds` (900–129600, default 43200 = 12h; clamped, not rejected) | `--serial-number`, `--token-code` (MFA → `InvalidParameterValue`) | **DONE** |
-| `assume-role-with-web-identity` | — | `--role-arn`, `--role-session-name`, `--web-identity-token`, `--provider-id`, `--policy`, `--policy-arns`, `--duration-seconds` | **NOT STARTED** |
+| `assume-role-with-web-identity` | `--role-arn`, `--role-session-name`, `--web-identity-token`, `--duration-seconds` (900–43200, default 3600) | `--provider-id`; `--policy`, `--policy-arns` (→ `PackedPolicyTooLarge`) | **DONE** |
 | `assume-role-with-saml` | — | `--role-arn`, `--principal-arn`, `--saml-assertion`, `--policy`, `--policy-arns`, `--duration-seconds` | **NOT STARTED** |
 | `get-access-key-info` | — | `--access-key-id` | **NOT STARTED** |
 | `get-federation-token` | — | `--name`, `--policy`, `--policy-arns`, `--duration-seconds`, `--tags` | **NOT STARTED** |
 | `decode-authorization-message` | — | `--encoded-message` | **NOT STARTED** |
 
-Trust policies stored on roles (`AssumeRolePolicyDocument`) reject `Condition`,
-`NotPrincipal`, `NotAction`, empty-string `Action` elements, and empty
-`Principal` blocks at write time (`MalformedPolicyDocument`) — v1 has no
-Condition evaluator so accepting them would silently allow.
+Trust policies stored on roles (`AssumeRolePolicyDocument`) reject `Condition`, `NotPrincipal`, `NotAction`, empty-string `Action` elements, and empty `Principal` blocks at write time (`MalformedPolicyDocument`) — v1 has no Condition evaluator so accepting them would silently allow.
 
 ---
 
@@ -593,10 +607,21 @@ curl -i http://169.254.169.254/latest/meta-data/instance-id
 | `/latest/meta-data/iam/info` | GET | `{InstanceProfileArn, InstanceProfileId}`; 404 if no profile | **DONE** |
 | `/latest/meta-data/iam/security-credentials/` | GET | Role name(s) under the profile, one per line; empty body if none | **DONE** |
 | `/latest/meta-data/iam/security-credentials/<role>` | GET | STS `AssumeRoleForInstance` → ASIA-prefixed temporary credential JSON | **DONE** |
+| `/latest/meta-data/public-keys/` | GET | `0=<keyName>` from the launch key pair; 404 if none | **DONE** |
+| `/latest/meta-data/public-keys/0/` | GET | `openssh-key` (format list for index 0) | **DONE** |
+| `/latest/meta-data/public-keys/0/openssh-key` | GET | Launch SSH public key, live-fetched from the key store; 404 if the key was deleted, 500 on backend fault | **DONE** |
 | `/latest/user-data` | GET | `vm.UserData`; 404 if none | **DONE** |
-| `/latest/dynamic/instance-identity/document` | GET | Needs a per-cluster signing key | **NOT STARTED** (404; lands with EKS IRSA) |
-| `/latest/meta-data/network/interfaces/...` | GET | Multi-ENI rendering | **NOT STARTED** (404) |
-| `/latest/meta-data/tags/...` | GET | Instance-tag metadata | **NOT STARTED** (404) |
+| `/latest/dynamic/instance-identity/document` (+ `signature`, `pkcs7`, `rsa2048`) | GET | Needs a per-cluster signing key | **NOT STARTED** (404; lands with EKS IRSA) |
+| `/latest/meta-data/network/interfaces/macs/<mac>/...` | GET | Multi-ENI rendering (subnet-id, vpc-id, ipv4s, security-group-ids, …) | **NOT STARTED** (404) |
+| `/latest/meta-data/tags/instance/<key>` | GET | Instance-tag metadata; gated on `InstanceMetadataTags` enablement | **NOT STARTED** (404) |
+| `/latest/meta-data/public-hostname` | GET | No public DNS today; would mirror `public-ipv4` | **NOT STARTED** (404) |
+| `/latest/meta-data/reservation-id` | GET | Available on the stored `Reservation`; cheap, not yet surfaced | **NOT STARTED** (404) |
+| `/latest/meta-data/ami-launch-index` | GET | `0` for single-instance launches; trivial once multi-count is exercised | **NOT STARTED** (404) |
+| `/latest/meta-data/block-device-mapping/...` | GET | `ami`/`root`/`ebsN`/`ephemeralN` device map | **NOT STARTED** (404) |
+| `/latest/meta-data/placement/{group-name,partition-number,availability-zone-id,host-id}` | GET | Placement extras beyond `availability-zone`/`region` | **NOT STARTED** (404) |
+| `/latest/meta-data/instance-life-cycle`, `/instance-action` | GET | `on-demand`/`spot`; `none` unless interruptible instances ship | **NOT STARTED** (404) |
+| `/latest/meta-data/services/{domain,partition}` | GET | Static per-deployment values | **NOT STARTED** (404) |
+| `/latest/meta-data/spot/{instance-action,termination-time}` | GET | Only meaningful once Spot is modelled | **NOT STARTED** (404) |
 
 ---
 
@@ -632,10 +657,54 @@ The data plane uses a system-managed LB VM, launched automatically during `creat
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|
-| `create-listener` | `--load-balancer-arn`, `--default-actions` (Type=forward, TargetGroupArn), `--protocol` (HTTP), `--port` | `--ssl-policy`, `--certificates`, `--alpn-policy`, `--mutual-authentication`, `--dry-run` | **DONE** |
+| `create-listener` | `--load-balancer-arn`, `--default-actions` (Type=forward, TargetGroupArn), `--protocol` (HTTP/HTTPS/TLS), `--port`, `--certificates` (HTTPS/TLS), `--ssl-policy` | `--alpn-policy`, `--mutual-authentication`, `--dry-run` | **DONE** |
+| `modify-listener` | `--listener-arn`, `--protocol`, `--port`, `--default-actions`, `--certificates`, `--ssl-policy` | `--alpn-policy`, `--mutual-authentication`, `--dry-run` | **DONE** |
 | `delete-listener` | `--listener-arn` | `--dry-run` | **DONE** |
 | `describe-listeners` | `--load-balancer-arn`, `--listener-arns` | `--page-size`, `--marker`, `--dry-run` | **DONE** |
 
+### ELBv2 — Listener Rules
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `create-rule` | `--listener-arn`, `--priority` (1–50000), `--conditions` (host-header, path-pattern, http-header, http-request-method, query-string, source-ip), `--actions` (forward, redirect, fixed-response), `--tags` | `--dry-run` | **DONE** |
+| `modify-rule` | `--rule-arn`, `--conditions`, `--actions` | `--dry-run` | **DONE** |
+| `delete-rule` | `--rule-arn` | `--dry-run` | **DONE** |
+| `describe-rules` | `--listener-arn`, `--rule-arns` | `--page-size`, `--marker` (parsed, not enforced) | **DONE** |
+| `set-rule-priorities` | `--rule-priorities` (RuleArn, Priority) | — | **DONE** |
+
+A synthetic `default` rule is derived from the listener's `DefaultActions`.
+
+### ELBv2 — Listener Certificates & SSL Policies
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `add-listener-certificates` | `--listener-arn`, `--certificates` | `--dry-run` | **DONE** |
+| `remove-listener-certificates` | `--listener-arn`, `--certificates` | `--dry-run` | **DONE** |
+| `describe-listener-certificates` | `--listener-arn` | `--page-size`, `--marker` | **DONE** |
+| `describe-ssl-policies` | `--names` | `--load-balancer-type`, `--page-size`, `--marker` | **DONE** (static catalog: `ELBSecurityPolicy-FS-1-2-Res-2019-08`, `ELBSecurityPolicy-TLS13-1-2-2021-06` — metadata only, no in-platform TLS termination) |
+
+The default certificate cannot be added/removed via these calls — set it on the listener.
+
+### ELBv2 — Attributes & Modify
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `describe-load-balancer-attributes` | `--load-balancer-arn` | — | **DONE** (stored values over per-type defaults) |
+| `modify-load-balancer-attributes` | `--load-balancer-arn`, `--attributes` | — | **DONE** (unknown keys → `ValidationError`) |
+| `describe-target-group-attributes` | `--target-group-arn` | — | **DONE** |
+| `modify-target-group-attributes` | `--target-group-arn`, `--attributes` (incl. `deregistration_delay.timeout_seconds`, `stickiness.*`) | — | **DONE** |
+| `modify-target-group` | `--target-group-arn`, `--health-check-*`, `--matcher` | `--target-type`/`--protocol`/`--vpc-id` (immutable) | **DONE** |
+| `describe-listener-attributes` | `--listener-arn` | — | **DONE** (stub — returns empty; not persisted) |
+| `modify-listener-attributes` | `--listener-arn`, `--attributes` | — | **DONE** (stub — echoes input; not persisted) |
+
+### ELBv2 — Network & Security
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `set-security-groups` | `--load-balancer-arn`, `--security-groups` | `--enforce-security-group-inbound-rules-on-private-link-traffic` | **DONE** (ALB only — NLB → `InvalidConfigurationRequest`) |
+| `set-subnets` | `--load-balancer-arn`, `--subnets`, `--subnet-mappings` | `--ip-address-type` | **DONE** (live ENI add/remove with rollback) |
+| `set-ip-address-type` | `--load-balancer-arn`, `--ip-address-type` (ipv4) | dualstack/IPv6 (rejected) | **DONE** |
+
 ### ELBv2 — Tags
 
 | Command | Implemented Flags | Missing Flags | Status |
@@ -646,18 +715,32 @@ The data plane uses a system-managed LB VM, launched automatically during `creat
 
 | Feature | Priority | Status |
 |---------|----------|--------|
-| Listener rules (CreateRule, DeleteRule, DescribeRules, ModifyRule) | High | **NOT STARTED** |
-| HTTPS/TLS termination (SSL certificates, policies, ALPN) | High | **NOT STARTED** |
-| Modify operations (`ModifyLoadBalancerAttributes`, `ModifyTargetGroup`, `ModifyTargetGroupAttributes`, `ModifyListener`) | Medium | **NOT STARTED** |
-| Connection draining (deregistration delay) | Medium | **NOT STARTED** |
-| Stickiness / session affinity | Medium | **NOT STARTED** |
-| Active health checking (API-driven, vs. HAProxy-only today) | Medium | **NOT STARTED** |
+| In-platform HTTPS/TLS termination (cert + SSL-policy APIs exist; data-plane TLS not terminated) | High | **NOT STARTED** |
+| ALPN policy, mutual TLS (mTLS) | Medium | **NOT STARTED** |
+| Listener attribute persistence (`Describe/ModifyListenerAttributes` are stubs) | Medium | **NOT STARTED** |
+| Active health checking (API-driven, vs. HAProxy/nginx-only today) | Medium | **NOT STARTED** |
 | IP and Lambda target types | Low | **NOT STARTED** |
 | S3 access log delivery | Low | **NOT STARTED** |
 | WAF integration | Low | **NOT STARTED** |
 
 ---
 
+## ACM (AWS Certificate Manager)
+
+Import-only — Spinifex stores externally-issued certificates for ELBv2 listener references; it does not issue certificates or validate domains (`RequestCertificate`). Certs are account-scoped; `describe`/`delete` enforce ownership.
+
+| Command | Implemented Flags | Missing Flags | Status |
+|---------|-------------------|---------------|--------|
+| `import-certificate` | `--certificate`, `--private-key`, `--certificate-chain`, `--certificate-arn` (re-import) | `--tags` | **DONE** |
+| `describe-certificate` | `--certificate-arn` | — | **DONE** |
+| `list-certificates` | — | `--certificate-statuses`, `--includes`, `--max-items`, `--next-token` | **DONE** |
+| `delete-certificate` | `--certificate-arn` | — | **DONE** |
+| `request-certificate` | — | `--domain-name`, `--validation-method`, `--subject-alternative-names`, `--tags` | **NOT STARTED** |
+| `add-tags-to-certificate` / `list-tags-for-certificate` / `remove-tags-from-certificate` | — | `--certificate-arn`, `--tags`/`--tag-keys` | **NOT STARTED** |
+| `export-certificate` | — | `--certificate-arn`, `--passphrase` | **NOT STARTED** |
+
+---
+
 ## CloudWatch (Basic Monitoring)
 
 | Command | Implemented Flags | Missing Flags | Status |

docs/reference-architectures/SMCI-H14-orchestration/README.md

@@ -26,6 +26,8 @@ sections:
 
 ## Overview
 
+Spinifex is an open-source infrastructure platform that brings core AWS services including EC2, EBS and S3 to bare-metal, edge, and on-prem environments. It exposes an EC2-compatible API, so any tooling that works against AWS (the `aws` CLI, Terraform, SDKs) works against a Spinifex node unchanged, with a single profile swap.
+
 This document serves as a tutorial for setting up multi-tenant AI workloads with Spinifex. The reference architecture for this tutorial is the Supermicro H14 platform - specifications listed below.
 
 The H14 chassis comes equipped with **8× AMD Instinct MI350X** (288 GB HBM3e each, 2.3 TB total). We used Spinifex to provision three isolated VMs, each assigned 2× MI350X via direct PCIe passthrough, and ran three computationally intensive workloads simultaneously: a YOLO11x vision model, a Qwen3-VL 235B FP8 vision language model, and a Llama 3.3 70B large language model.

docs/reference-architectures/arcee-trinity-mi350x/README.md

@@ -31,6 +31,8 @@ sections:
 
 ## Overview
 
+Spinifex is an open-source infrastructure platform that brings core AWS services including EC2, EBS and S3 to bare-metal, edge, and on-prem environments. It exposes an EC2-compatible API, so any tooling that works against AWS (the `aws` CLI, Terraform, SDKs) works against a Spinifex node unchanged, with a single profile swap.
+
 ### About Trinity
 
 [Trinity](https://www.arcee.ai/trinity) is a family of open-weight language models from [Arcee AI](https://www.arcee.ai), built specifically for production agentic workloads: reliable tool calling, structured JSON output, coherent multi-turn conversations, and long-context reasoning. The family spans four sizes — Nano (6B), Mini (26B), Large Preview (400B), and Large Thinking (400B reasoning-optimised), with consistent capabilities and API surface across all of them. A workflow validated on Nano can be promoted to Large with no prompt changes.

docs/service-interfaces.yaml

@@ -84,6 +84,12 @@ services:
       - "spinifex.node.vms"
       - "spinifex.nodes.discover"
       - "spinifex.cluster.shutdown.progress"
+      # Coordinated cluster-shutdown phases (fan-out, no queue group).
+      - "spinifex.cluster.shutdown.gate"
+      - "spinifex.cluster.shutdown.drain"
+      - "spinifex.cluster.shutdown.storage"
+      - "spinifex.cluster.shutdown.persist"
+      - "spinifex.cluster.shutdown.infra"
       - "spinifex.storage.config"
       # ELBv2 control plane (handlers/elbv2/service_nats.go topic map).
       - "elbv2.CreateLoadBalancer"
@@ -133,6 +139,8 @@ services:
       - "ebs.delete"
       - "ebs.sync"
       - "ebs.snapshot.*"
+      - "ebs.config"
+      - "ebs.config.*"
       # → predastore (S3 / AMI / keypair store)
       - "s3.*"
       # IAM lifecycle event
@@ -181,6 +189,8 @@ services:
       - "ebs.delete"
       - "ebs.sync"
       - "ebs.snapshot.*"
+      - "ebs.config"
+      - "ebs.config.*"
     publishes:
       - "ebs.mount.response"
       - "ebs.unmount.response"

docs/terraform-workbooks/eks-addons-access/README.md

@@ -0,0 +1,181 @@
+---
+title: "EKS with Addons + Access-Entry RBAC"
+description: "Extend a single-node EKS cluster with a managed CoreDNS addon, access-entry RBAC, and a browsable load-balanced demo app, using Terraform on Spinifex."
+category: "Terraform Workbooks"
+tags:
+  - terraform
+  - eks
+  - kubernetes
+  - addons
+  - rbac
+  - iam
+  - workbook
+resources:
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "Terraform Kubernetes Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "OpenTofu"
+    url: "https://opentofu.org/"
+---
+
+# Terraform: EKS with Addons + Access-Entry RBAC
+
+> Builds on the quickstart by installing a managed addon (CoreDNS) and granting a second IAM principal read-only cluster access — then deploys the same browsable, load-balanced demo app so you can see the result.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+This workbook takes the same single-node cluster (and the same browsable demo app) from [EKS Quickstart](../eks-quickstart) and adds the two things you reach for on almost any real cluster:
+
+1. **A managed addon.** CoreDNS is installed through the EKS API (`aws_eks_addon`) rather than `kubectl apply`, so its lifecycle is managed by the cluster. The demo app is deployed *after* CoreDNS, so a healthy page also confirms in-cluster DNS is working.
+2. **Access-entry RBAC.** A second IAM role is granted read-only access via an `aws_eks_access_entry` plus an `aws_eks_access_policy_association` bound to the AWS-managed `AmazonEKSViewPolicy`. This is the modern access-entry path (`authentication_mode = "API"`) — no `aws-auth` ConfigMap.
+
+When `apply` finishes, open the **`demo_url`** output and refresh — the `nginxdemos/hello` page reports which pod served each request.
+
+**What you'll learn:**
+
+- Installing a managed EKS addon with `aws_eks_addon`
+- Registering an IAM principal with `aws_eks_access_entry` and scoping it with `aws_eks_access_policy_association`
+- The difference between the cluster creator's admin access and a scoped viewer
+- Deploying a workload with the Kubernetes provider once the addon is healthy
+
+**What gets created**
+
+| Resource | Name | Purpose |
+|---|---|---|
+| VPC | `eks-addons-vpc` | Isolated network (10.31.0.0/16) |
+| Subnets | `eks-addons-subnet-a/-b` | Public subnets for the cluster and worker |
+| Internet Gateway | `eks-addons-igw` | Egress so the worker can pull the demo image |
+| IAM Roles | `eks-addons-cluster-role`, `eks-addons-node-role` | Control-plane and worker roles |
+| IAM Role | `eks-addons-viewer` | Stand-in principal granted read-only access |
+| EKS Cluster | `eks-addons` | Public endpoint, API auth mode |
+| Node Group | `default` | One `t3.medium` worker |
+| Addon | `coredns` | Cluster DNS, managed via the EKS API |
+| Access Entry + Assoc. | `eks-addons-viewer` → `AmazonEKSViewPolicy` | Read-only, cluster scope |
+| SG Ingress Rule | `eks-addons-demo-nodeport` | Opens the demo NodePort on the worker SG |
+| K8s Deployment + Service | `hello` | `nginxdemos/hello`, 2 replicas, NodePort |
+
+**Spinifex specifics**
+
+- Addon catalog is limited. Spinifex currently serves `coredns`, `aws-load-balancer-controller` and `spinifex-noop`. There is no `vpc-cni`, `kube-proxy`, `ebs-csi` or `metrics-server`.
+- **Leave `addon_version` unset.** Spinifex defaults an unset version to its catalog default (`coredns` 1.11.1). Pinning is a dead end: the AWS provider rejects a bare `1.11.1` (it requires the `v`-prefixed `v1.11.1-eksbuild.N` form), while Spinifex's catalog only accepts the bare `1.11.1` — so no single string satisfies both.
+- Cluster access policies are the four AWS-managed `cluster-access-policy/Amazon EKS{ClusterAdmin,Admin,Edit,View}Policy` ARNs.
+- The worker SG is auto-managed and the worker AMI is always the `eks-node` image; the NodePort is exposed by adding one rule to the looked-up worker SG.
+
+**Prerequisites:**
+
+- Spinifex installed and running
+- The Spinifex `eks-node` image available on the cluster
+- OpenTofu or Terraform, plus `kubectl` and the AWS CLI
+
+## Instructions
+
+### Step 1. Get the Template
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-addons-access
+```
+
+Or create a `main.tf` file and paste the full configuration below.
+
+<!-- INCLUDE: main.tf lang:hcl -->
+
+### Step 2. Deploy
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+```
+
+Allow a few minutes for the cluster to reach `ACTIVE` and the worker to register. CoreDNS installs onto the worker, then Terraform deploys the demo app.
+
+> Keep `AWS_PROFILE=spinifex` exported for the whole `apply` — the Kubernetes provider runs `aws eks get-token` against the Spinifex STS endpoint.
+
+### Step 3. Open the Demo
+
+```bash
+tofu output demo_url
+```
+
+Open it in a browser and refresh — the `nginxdemos/hello` page shows the pod that answered, alternating between the two replicas.
+
+### Step 4. Verify the Addon and Access Entry
+
+```bash
+aws eks list-addons --cluster-name eks-addons
+aws eks describe-addon --cluster-name eks-addons --addon-name coredns --query 'addon.status'
+
+aws eks list-access-entries --cluster-name eks-addons
+aws eks list-associated-access-policies \
+  --cluster-name eks-addons \
+  --principal-arn "$(tofu output -raw viewer_principal_arn)"
+```
+
+The viewer principal is bound to `AmazonEKSViewPolicy` at cluster scope — it can read across the cluster but cannot mutate anything. The principal that ran `apply` keeps full cluster-admin from `bootstrap_cluster_creator_admin_permissions`.
+
+### Cleanup
+
+```bash
+tofu destroy
+```
+
+## Troubleshooting
+
+### Demo URL Doesn't Load
+
+Same chain as the quickstart — check the worker's public IP, that the demo pods are `Running`, and that the NodePort rule landed on the worker SG:
+
+```bash
+kubectl get pods -o wide
+aws ec2 describe-security-groups --filters "Name=group-name,Values=eks-cluster-eks-addons-nodegroup-sg" \
+  --query 'SecurityGroups[0].IpPermissions'
+```
+
+If the demo pods are stuck `ContainerCreating` with DNS errors, confirm CoreDNS is healthy first:
+
+```bash
+kubectl -n kube-system get deploy coredns
+```
+
+### Addon Stuck in CREATING / DEGRADED
+
+CoreDNS needs a schedulable worker. Confirm the node group is `ACTIVE` and the node is `Ready`:
+
+```bash
+aws eks describe-nodegroup --cluster-name eks-addons --nodegroup-name default --query 'nodegroup.status'
+kubectl get nodes
+```
+
+### Addon Version Rejected
+
+This workbook leaves `addon_version` unset so Spinifex picks its catalog default. If you add an explicit `addon_version`, expect a wall: the AWS provider requires the `v`-prefixed `v1.11.1-eksbuild.N` form, but Spinifex's catalog only knows the bare `1.11.1` — a value that passes one is rejected by the other. Prefer omitting the field.
+
+### Access Entry Not Taking Effect
+
+Assume the viewer role and confirm it can read but not write:
+
+```bash
+kubectl auth can-i list pods --all-namespaces   # yes
+kubectl auth can-i create deployments           # no
+```
+
+### Provider Connection Refused
+
+```bash
+sudo systemctl status spinifex.target
+curl -k https://localhost:9999/
+```

docs/terraform-workbooks/eks-addons-access/main.tf

@@ -0,0 +1,468 @@
+# Example: EKS with an Addon + Access-Entry RBAC (with a visible demo app)
+#
+# Builds on eks-quickstart: same single-worker cluster and browsable demo app,
+# but adds two things you reach for on a real cluster:
+#
+#   * a managed addon (CoreDNS) installed through the EKS API, and
+#   * a second IAM principal granted scoped, read-only access via an EKS access
+#     entry + access-policy association (authentication_mode = "API").
+#
+# After `apply`, open the demo_url output — the page reports the pod that
+# served it, so a refresh shows requests landing on different replicas. CoreDNS
+# being healthy is what lets in-cluster name resolution work.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-addons-access
+#   export AWS_PROFILE=spinifex
+#   tofu init && tofu apply
+#   # then open the demo_url output
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.20"
+    }
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Variables
+# ---------------------------------------------------------------------------
+
+variable "region" {
+  type    = string
+  default = "ap-southeast-2"
+}
+
+variable "cluster_name" {
+  type    = string
+  default = "eks-addons"
+}
+
+variable "k8s_version" {
+  type    = string
+  default = "1.32"
+}
+
+variable "instance_type" {
+  type    = string
+  default = "t3.medium"
+}
+
+variable "node_port" {
+  type    = number
+  default = 30080
+}
+
+variable "replicas" {
+  type    = number
+  default = 2
+}
+
+variable "browse_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the demo NodePort; tighten to your own IP in production"
+}
+
+variable "spinifex_endpoint" {
+  type    = string
+  default = "https://127.0.0.1:9999"
+}
+
+# ---------------------------------------------------------------------------
+# Providers
+# ---------------------------------------------------------------------------
+
+provider "aws" {
+  region = var.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+provider "kubernetes" {
+  host                   = aws_eks_cluster.this.endpoint
+  cluster_ca_certificate = base64decode(aws_eks_cluster.this.certificate_authority[0].data)
+
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    command     = "aws"
+    args        = ["eks", "get-token", "--cluster-name", aws_eks_cluster.this.name, "--region", var.region]
+  }
+}
+
+data "aws_availability_zones" "available" {
+  state = "available"
+}
+
+# ---------------------------------------------------------------------------
+# VPC + two public subnets
+# ---------------------------------------------------------------------------
+
+resource "aws_vpc" "main" {
+  cidr_block           = "10.31.0.0/16"
+  enable_dns_hostnames = true
+  enable_dns_support   = true
+
+  tags = {
+    Name = "${var.cluster_name}-vpc"
+  }
+}
+
+resource "aws_internet_gateway" "igw" {
+  vpc_id = aws_vpc.main.id
+
+  tags = {
+    Name = "${var.cluster_name}-igw"
+  }
+}
+
+resource "aws_subnet" "a" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.31.1.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-subnet-a"
+  }
+}
+
+resource "aws_subnet" "b" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.31.2.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-subnet-b"
+  }
+}
+
+resource "aws_route_table" "public" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block = "0.0.0.0/0"
+    gateway_id = aws_internet_gateway.igw.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-public-rt"
+  }
+}
+
+resource "aws_route_table_association" "a" {
+  subnet_id      = aws_subnet.a.id
+  route_table_id = aws_route_table.public.id
+}
+
+resource "aws_route_table_association" "b" {
+  subnet_id      = aws_subnet.b.id
+  route_table_id = aws_route_table.public.id
+}
+
+# ---------------------------------------------------------------------------
+# IAM — cluster role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "cluster" {
+  name = "${var.cluster_name}-cluster-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "eks.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "cluster" {
+  role       = aws_iam_role.cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — node role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "node" {
+  name = "${var.cluster_name}-node-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_worker" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_cni" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_ecr" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — a second principal to grant read-only cluster access to
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "viewer" {
+  name = "${var.cluster_name}-viewer"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+# ---------------------------------------------------------------------------
+# EKS cluster
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_cluster" "this" {
+  name     = var.cluster_name
+  role_arn = aws_iam_role.cluster.arn
+  version  = var.k8s_version
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.a.id, aws_subnet.b.id]
+    endpoint_public_access  = true
+    endpoint_private_access = false
+  }
+
+  access_config {
+    authentication_mode                         = "API"
+    bootstrap_cluster_creator_admin_permissions = true
+  }
+
+  depends_on = [aws_iam_role_policy_attachment.cluster]
+
+  tags = {
+    Name = var.cluster_name
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Node group
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_node_group" "default" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "default"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.a.id, aws_subnet.b.id]
+
+  scaling_config {
+    desired_size = 1
+    min_size     = 1
+    max_size     = 2
+  }
+
+  instance_types = [var.instance_type]
+  ami_type       = "AL2_x86_64"
+
+  depends_on = [
+    aws_iam_role_policy_attachment.node_worker,
+    aws_iam_role_policy_attachment.node_cni,
+    aws_iam_role_policy_attachment.node_ecr,
+  ]
+
+  tags = {
+    Name = "${var.cluster_name}-default"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Addon — CoreDNS
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_addon" "coredns" {
+  cluster_name                = aws_eks_cluster.this.name
+  addon_name                  = "coredns"
+  resolve_conflicts_on_create = "OVERWRITE"
+
+  # addon_version omitted on purpose. Spinifex defaults an unset version to its
+  # catalog default (coredns 1.11.1). The AWS provider rejects a bare "1.11.1"
+  # (it demands a v-prefixed form) which Spinifex's catalog would in turn
+  # reject — so pinning is a dead end here; let the server choose.
+  depends_on = [aws_eks_node_group.default]
+
+  tags = {
+    Name = "${var.cluster_name}-coredns"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Access entry — grant the viewer principal read-only access
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_access_entry" "viewer" {
+  cluster_name  = aws_eks_cluster.this.name
+  principal_arn = aws_iam_role.viewer.arn
+  type          = "STANDARD"
+}
+
+resource "aws_eks_access_policy_association" "viewer" {
+  cluster_name  = aws_eks_cluster.this.name
+  principal_arn = aws_iam_role.viewer.arn
+  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
+
+  access_scope {
+    type = "cluster"
+  }
+
+  depends_on = [aws_eks_access_entry.viewer]
+}
+
+# ---------------------------------------------------------------------------
+# Open the NodePort on the auto-managed nodegroup SG (see eks-quickstart)
+# ---------------------------------------------------------------------------
+
+data "aws_security_group" "nodegroup" {
+  filter {
+    name   = "group-name"
+    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
+  }
+
+  filter {
+    name   = "vpc-id"
+    values = [aws_vpc.main.id]
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+resource "aws_vpc_security_group_ingress_rule" "nodeport" {
+  security_group_id = data.aws_security_group.nodegroup.id
+  cidr_ipv4         = var.browse_cidr
+  from_port         = var.node_port
+  to_port           = var.node_port
+  ip_protocol       = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-demo-nodeport"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Demo workload — deployed after CoreDNS so name resolution is up
+# ---------------------------------------------------------------------------
+
+resource "kubernetes_deployment_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+    labels    = { app = "hello" }
+  }
+
+  spec {
+    replicas = var.replicas
+
+    selector {
+      match_labels = { app = "hello" }
+    }
+
+    template {
+      metadata {
+        labels = { app = "hello" }
+      }
+
+      spec {
+        container {
+          name  = "hello"
+          image = "nginxdemos/hello"
+
+          port {
+            container_port = 80
+          }
+        }
+      }
+    }
+  }
+
+  depends_on = [aws_eks_addon.coredns]
+}
+
+resource "kubernetes_service_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+  }
+
+  spec {
+    selector = { app = "hello" }
+    type     = "NodePort"
+
+    port {
+      port        = 80
+      target_port = 80
+      node_port   = var.node_port
+    }
+  }
+
+  depends_on = [kubernetes_deployment_v1.hello]
+}
+
+data "aws_instances" "workers" {
+  instance_tags = {
+    "spinifex:eks-cluster" = aws_eks_cluster.this.name
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+# ---------------------------------------------------------------------------
+# Outputs
+# ---------------------------------------------------------------------------
+
+output "cluster_name" {
+  value = aws_eks_cluster.this.name
+}
+
+output "viewer_principal_arn" {
+  value = aws_iam_role.viewer.arn
+}
+
+output "demo_url" {
+  value       = "http://${data.aws_instances.workers.public_ips[0]}:${var.node_port}"
+  description = "Open in a browser; refresh to see different pods answer"
+}
+
+output "update_kubeconfig" {
+  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
+}

docs/terraform-workbooks/eks-https-ingress/README.md

@@ -0,0 +1,227 @@
+---
+title: "3-Node EKS behind an HTTPS ALB (ACM TLS)"
+description: "Provision an HA-shaped EKS cluster with public + private API endpoints and three workers, deploy a demo app, and publish it through an internet-facing ALB that terminates TLS with an ACM-imported certificate, using Terraform on Spinifex."
+category: "Terraform Workbooks"
+tags:
+  - terraform
+  - eks
+  - kubernetes
+  - acm
+  - tls
+  - alb
+  - elbv2
+  - vpc
+  - workbook
+resources:
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "Terraform Kubernetes Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "OpenTofu"
+    url: "https://opentofu.org/"
+---
+
+# Terraform: 3-Node EKS behind an HTTPS ALB
+
+> The largest EKS workbook, and the clearest demo. An HA-shaped cluster with public + private API endpoints and three workers, running a demo app published over HTTPS through an internet-facing ALB that terminates TLS with an ACM-imported certificate.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+This workbook is sized for a **3-node Spinifex deployment** and is built to show off. It brings together the pieces from the smaller EKS examples, deploys a demo app across three workers, and fronts it with a TLS-terminating load balancer. When `apply` finishes, fetch the ALB's public IP and open `https://<ip>` — the `nginxdemos/hello` page reports which pod answered, and refreshing moves between the three replicas spread across the cluster. That single page demonstrates a multi-node Kubernetes cluster load-balancing behind HTTPS.
+
+- **Public + private API endpoints.** `endpoint_public_access` serves `kubectl` and the Kubernetes provider from your host (CIDR-restricted); `endpoint_private_access` exposes an in-VPC private endpoint the workers use to join.
+- **Private workers + NAT.** The three workers sit in private subnets. They *join* the cluster over the in-VPC private endpoint, and a NAT gateway gives them outbound access to pull the demo container image.
+- **CoreDNS** installed as a managed addon.
+- **TLS via ACM.** A self-signed certificate is generated with the `tls` provider and **imported into ACM** (Spinifex ACM supports `ImportCertificate`, not `RequestCertificate`). The ALB's HTTPS listener references the ACM ARN and forwards to the workers' NodePort.
+- **The app, deployed by Terraform.** The Kubernetes provider deploys the Deployment + NodePort Service, and one security-group rule lets the ALB reach the NodePort.
+
+<p align="center">
+  <img src="../../../.github/assets/diagrams/tf-eks-https-ingress.svg" alt="3-node EKS — public+private API endpoints, ALB in public subnets terminating TLS, three workers in private subnets behind a NAT gateway" width="900">
+</p>
+
+**What you'll learn:**
+
+- Running an EKS cluster with both public and private API endpoints
+- Placing workers in private subnets that join over the private endpoint and egress via NAT
+- Importing a certificate into ACM and terminating TLS on an ALB HTTPS listener
+- Wiring an ALB target group to managed node-group workers via a NodePort, including the SG rule that allows it
+- Deploying a load-balanced workload with the Kubernetes provider
+
+**What gets created**
+
+| Resource | Name | Purpose |
+|---|---|---|
+| VPC | `eks-https-vpc` | Isolated network (10.32.0.0/16) |
+| Public Subnets | `eks-https-public-a/-b` | Host the ALB and NAT gateway |
+| Private Subnets | `eks-https-private-a/-b` | Host the workers |
+| Internet Gateway | `eks-https-igw` | Internet for the public subnets |
+| NAT Gateway (+ EIP) | `eks-https-nat` | Outbound for the private workers (image pulls) |
+| Route Tables | `eks-https-public-rt`, `eks-https-private-rt` | Public via IGW; private via NAT |
+| IAM Roles | `eks-https-cluster-role`, `eks-https-node-role` | Control-plane and worker roles |
+| EKS Cluster | `eks-https` | Public (CIDR-restricted) + private endpoints |
+| Node Group | `workers` | Three `t3.large` workers in the private subnets |
+| Addon | `coredns` | Cluster DNS |
+| ACM Cert | `eks-https-ingress` | Self-signed cert imported into ACM |
+| ALB + HTTPS Listener | `eks-https-alb` | Internet-facing, terminates TLS on :443 |
+| Target Group | `eks-https-tg` | Instance targets on the NodePort |
+| SG Ingress Rule | `eks-https-alb-nodeport` | Lets the ALB SG reach the worker NodePort |
+| K8s Deployment + Service | `hello` | `nginxdemos/hello`, 3 replicas, NodePort |
+
+**Spinifex specifics**
+
+- ACM supports `ImportCertificate`, `DescribeCertificate`, `ListCertificates`, `DeleteCertificate` — there is no `RequestCertificate`/DNS-validation flow, hence the self-signed import.
+- The ALB HTTPS listener resolves the cert from ACM. Valid SSL policies are `ELBSecurityPolicy-2016-08` (default) and `ELBSecurityPolicy-TLS13-1-2-2021-06`.
+- The worker SG is auto-managed (`vpc_config.security_group_ids` is ignored) and admits only intra-cluster traffic. The ALB→NodePort rule is added by looking the SG up by its deterministic name and admitting the ALB SG.
+- Worker discovery uses the `spinifex:eks-cluster` tag Spinifex stamps on node-group instances; the target-group attachment count is the known `node_desired_size`.
+- Leave `addon_version` unset (see the addons workbook for why pinning it can't satisfy both the provider and the catalog).
+
+**Prerequisites:**
+
+- A 3-node Spinifex deployment, installed and running
+- The Spinifex `eks-node` image available on the cluster
+- OpenTofu or Terraform, plus `kubectl` and the AWS CLI
+
+## Instructions
+
+### Step 1. Get the Template
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-https-ingress
+```
+
+Or create a `main.tf` file and paste the full configuration below.
+
+<!-- INCLUDE: main.tf lang:hcl -->
+
+### Step 2. Deploy
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+```
+
+Expect several minutes: the control plane bootstraps, three workers launch and join over the private endpoint, the NAT gateway comes up, CoreDNS installs, the ALB is provisioned, and the Kubernetes provider rolls out the demo app.
+
+> **Keep `AWS_PROFILE=spinifex` exported** for the whole `apply` — the Kubernetes provider authenticates with `aws eks get-token` against the Spinifex STS endpoint.
+
+> **Tighten access in production.** `api_public_access_cidr` and `alb_ingress_cidr` both default to `0.0.0.0/0`. Set them to your own CIDR:
+>
+> ```bash
+> export TF_VAR_api_public_access_cidr="203.0.113.10/32"
+> export TF_VAR_alb_ingress_cidr="203.0.113.10/32"
+> ```
+
+### Step 3. Open the Demo over HTTPS
+
+The ALB DNS name (`*.elb.spinifex.local`) won't resolve from your host, so fetch its public IP:
+
+```bash
+ALB_IP=$(aws elbv2 describe-load-balancers --names eks-https-alb \
+  --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
+  --output text)
+```
+
+The certificate is self-signed, so a browser will warn (proceed anyway) and `curl` needs `-k`:
+
+```bash
+curl -k https://$ALB_IP
+curl -k https://$ALB_IP   # refresh: the "Server name" changes between pods
+```
+
+Open `https://$ALB_IP` in a browser and refresh to watch requests land on different pods across the three nodes.
+
+### Step 4. Inspect (optional)
+
+```bash
+aws eks update-kubeconfig --name eks-https --region ap-southeast-2
+kubectl get nodes
+kubectl get pods -o wide          # three replicas, one per node
+
+TG_ARN=$(aws elbv2 describe-target-groups --names eks-https-tg \
+  --query 'TargetGroups[0].TargetGroupArn' --output text)
+aws elbv2 describe-target-health --target-group-arn "$TG_ARN"
+```
+
+### Cleanup
+
+```bash
+tofu destroy
+```
+
+## Troubleshooting
+
+### Targets Unhealthy / ALB Returns 5xx
+
+The ALB health-checks `/` on the NodePort. Targets turn healthy once the demo pods are running *and* the SG rule admits the ALB. Check, in order:
+
+```bash
+kubectl get pods -o wide                                  # pods Running?
+aws ec2 describe-security-groups \
+  --filters "Name=group-name,Values=eks-cluster-eks-https-nodegroup-sg" \
+  --query 'SecurityGroups[0].IpPermissions'               # NodePort rule present?
+aws elbv2 describe-target-health --target-group-arn "$TG_ARN"
+```
+
+If pods are `ImagePullBackOff`, the workers can't reach Docker Hub — confirm the NAT gateway is `available`:
+
+```bash
+aws ec2 describe-nat-gateways --query 'NatGateways[].[NatGatewayId,State]'
+```
+
+### Workers Never Join
+
+The workers join over the private endpoint. Confirm it's enabled and the node group is progressing:
+
+```bash
+aws eks describe-cluster --name eks-https --query 'cluster.resourcesVpcConfig.endpointPrivateAccess'
+aws eks describe-nodegroup --cluster-name eks-https --nodegroup-name workers --query 'nodegroup.status'
+```
+
+### Target-Group Attachment Errors at Apply
+
+The attachment indexes the discovered worker IDs by `node_desired_size`. If `apply` errors with an index out of range, the workers weren't all tagged/running when the `aws_instances` lookup ran. Re-run `tofu apply` — the node group will be `ACTIVE` by then:
+
+```bash
+aws ec2 describe-instances --filters "Name=tag:spinifex:eks-cluster,Values=eks-https" \
+  --query 'Reservations[].Instances[].[InstanceId,State.Name]' --output text
+```
+
+### kubernetes provider: connection refused / Unauthorized
+
+The provider runs `aws eks get-token` against the Spinifex STS endpoint. If the cluster was still `CREATING` when the provider first connected, re-run `tofu apply`. Otherwise confirm:
+
+```bash
+aws sts get-caller-identity
+aws eks describe-cluster --name eks-https --query 'cluster.status'
+```
+
+### ACM Import / HTTPS Listener Errors
+
+```bash
+aws acm list-certificates
+aws acm describe-certificate --certificate-arn "$(tofu output -raw certificate_arn)" \
+  --query 'Certificate.Status'
+```
+
+A `CertificateNotFound` on the listener means the ACM ARN didn't resolve — check the cert exists and belongs to the same account/profile.
+
+### Provider Connection Refused
+
+```bash
+sudo systemctl status spinifex.target
+curl -k https://localhost:9999/
+```

docs/terraform-workbooks/eks-https-ingress/main.tf

@@ -0,0 +1,671 @@
+# Example: 3-Node EKS behind an HTTPS ALB (ACM TLS) with a load-balanced app
+#
+# The largest EKS workbook, and the one with the clearest payoff. It stands up
+# an HA-shaped cluster sized for a 3-node Spinifex deployment, deploys a demo
+# web app across the workers, and publishes it through an internet-facing ALB
+# that terminates TLS with a certificate held in ACM.
+#
+# After `apply`, fetch the ALB's public IP (see alb_public_ip_hint) and open
+# https://<ip> in a browser. The page reports which pod answered; refresh and
+# it moves between the three replicas spread across the cluster — a live picture
+# of a multi-node Kubernetes cluster load-balancing behind HTTPS.
+#
+# What it builds:
+#   * VPC with two public subnets (ALB + NAT) and two private subnets (workers).
+#   * A NAT gateway so the private workers can pull the demo container image.
+#   * EKS cluster with BOTH a public endpoint (kubectl from your host) and a
+#     private endpoint (the in-VPC path the workers use to join).
+#   * A managed node group of three workers (t3.large) in the private subnets.
+#   * CoreDNS addon.
+#   * A self-signed cert imported into ACM (Spinifex ACM supports
+#     ImportCertificate, not RequestCertificate), wired to the ALB's HTTPS
+#     listener, which forwards to the workers' NodePort.
+#   * The demo Deployment + NodePort Service, deployed by the Kubernetes
+#     provider, plus the one SG rule that lets the ALB reach the NodePort.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-https-ingress
+#   export AWS_PROFILE=spinifex
+#   tofu init && tofu apply
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    tls = {
+      source  = "hashicorp/tls"
+      version = ">= 4.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.20"
+    }
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Variables
+# ---------------------------------------------------------------------------
+
+variable "region" {
+  type    = string
+  default = "ap-southeast-2"
+}
+
+variable "cluster_name" {
+  type    = string
+  default = "eks-https"
+}
+
+variable "k8s_version" {
+  type    = string
+  default = "1.32"
+}
+
+variable "node_instance_type" {
+  type    = string
+  default = "t3.large"
+}
+
+variable "node_desired_size" {
+  type        = number
+  default     = 3
+  description = "Worker count; spread across a 3-node Spinifex deployment for HA"
+}
+
+variable "node_port" {
+  type        = number
+  default     = 30080
+  description = "NodePort the ALB forwards to and the demo Service is published on"
+}
+
+variable "cert_common_name" {
+  type    = string
+  default = "eks-https.spinifex.local"
+}
+
+variable "api_public_access_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the public Kubernetes API endpoint; tighten in production"
+}
+
+variable "alb_ingress_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the ALB HTTPS listener"
+}
+
+variable "spinifex_endpoint" {
+  type    = string
+  default = "https://127.0.0.1:9999"
+}
+
+# ---------------------------------------------------------------------------
+# Providers
+# ---------------------------------------------------------------------------
+
+provider "aws" {
+  region = var.region
+
+  endpoints {
+    ec2                    = var.spinifex_endpoint
+    iam                    = var.spinifex_endpoint
+    sts                    = var.spinifex_endpoint
+    eks                    = var.spinifex_endpoint
+    acm                    = var.spinifex_endpoint
+    elasticloadbalancingv2 = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+provider "kubernetes" {
+  host                   = aws_eks_cluster.this.endpoint
+  cluster_ca_certificate = base64decode(aws_eks_cluster.this.certificate_authority[0].data)
+
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    command     = "aws"
+    args        = ["eks", "get-token", "--cluster-name", aws_eks_cluster.this.name, "--region", var.region]
+  }
+}
+
+data "aws_availability_zones" "available" {
+  state = "available"
+}
+
+# ---------------------------------------------------------------------------
+# VPC — public subnets for the ALB + NAT, private subnets for the workers
+# ---------------------------------------------------------------------------
+
+resource "aws_vpc" "main" {
+  cidr_block           = "10.32.0.0/16"
+  enable_dns_hostnames = true
+  enable_dns_support   = true
+
+  tags = {
+    Name = "${var.cluster_name}-vpc"
+  }
+}
+
+resource "aws_internet_gateway" "igw" {
+  vpc_id = aws_vpc.main.id
+
+  tags = {
+    Name = "${var.cluster_name}-igw"
+  }
+}
+
+resource "aws_subnet" "public_a" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.32.1.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-public-a"
+  }
+}
+
+resource "aws_subnet" "public_b" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.32.2.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-public-b"
+  }
+}
+
+resource "aws_subnet" "private_a" {
+  vpc_id            = aws_vpc.main.id
+  cidr_block        = "10.32.11.0/24"
+  availability_zone = data.aws_availability_zones.available.names[0]
+
+  tags = {
+    Name = "${var.cluster_name}-private-a"
+  }
+}
+
+resource "aws_subnet" "private_b" {
+  vpc_id            = aws_vpc.main.id
+  cidr_block        = "10.32.12.0/24"
+  availability_zone = data.aws_availability_zones.available.names[0]
+
+  tags = {
+    Name = "${var.cluster_name}-private-b"
+  }
+}
+
+resource "aws_route_table" "public" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block = "0.0.0.0/0"
+    gateway_id = aws_internet_gateway.igw.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-public-rt"
+  }
+}
+
+resource "aws_route_table_association" "public_a" {
+  subnet_id      = aws_subnet.public_a.id
+  route_table_id = aws_route_table.public.id
+}
+
+resource "aws_route_table_association" "public_b" {
+  subnet_id      = aws_subnet.public_b.id
+  route_table_id = aws_route_table.public.id
+}
+
+# NAT gateway gives the private workers outbound internet to pull the demo
+# container image. The workers still *join* the cluster over the in-VPC private
+# endpoint, not through the NAT.
+resource "aws_eip" "nat" {
+  domain = "vpc"
+
+  tags = {
+    Name = "${var.cluster_name}-nat-eip"
+  }
+}
+
+resource "aws_nat_gateway" "nat" {
+  allocation_id = aws_eip.nat.id
+  subnet_id     = aws_subnet.public_a.id
+
+  depends_on = [aws_internet_gateway.igw]
+
+  tags = {
+    Name = "${var.cluster_name}-nat"
+  }
+}
+
+resource "aws_route_table" "private" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block     = "0.0.0.0/0"
+    nat_gateway_id = aws_nat_gateway.nat.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-private-rt"
+  }
+}
+
+resource "aws_route_table_association" "private_a" {
+  subnet_id      = aws_subnet.private_a.id
+  route_table_id = aws_route_table.private.id
+}
+
+resource "aws_route_table_association" "private_b" {
+  subnet_id      = aws_subnet.private_b.id
+  route_table_id = aws_route_table.private.id
+}
+
+# ---------------------------------------------------------------------------
+# IAM — cluster role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "cluster" {
+  name = "${var.cluster_name}-cluster-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "eks.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "cluster" {
+  role       = aws_iam_role.cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — node role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "node" {
+  name = "${var.cluster_name}-node-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_worker" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_cni" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_ecr" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+}
+
+# ---------------------------------------------------------------------------
+# EKS cluster — public + private endpoints
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_cluster" "this" {
+  name     = var.cluster_name
+  role_arn = aws_iam_role.cluster.arn
+  version  = var.k8s_version
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.private_a.id, aws_subnet.private_b.id]
+    endpoint_public_access  = true
+    endpoint_private_access = true
+    public_access_cidrs     = [var.api_public_access_cidr]
+  }
+
+  access_config {
+    authentication_mode                         = "API"
+    bootstrap_cluster_creator_admin_permissions = true
+  }
+
+  depends_on = [aws_iam_role_policy_attachment.cluster]
+
+  tags = {
+    Name = var.cluster_name
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Managed node group — three workers in the private subnets
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_node_group" "workers" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "workers"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.private_a.id, aws_subnet.private_b.id]
+
+  scaling_config {
+    desired_size = var.node_desired_size
+    min_size     = var.node_desired_size
+    max_size     = var.node_desired_size * 2
+  }
+
+  instance_types = [var.node_instance_type]
+  ami_type       = "AL2_x86_64"
+
+  depends_on = [
+    aws_iam_role_policy_attachment.node_worker,
+    aws_iam_role_policy_attachment.node_cni,
+    aws_iam_role_policy_attachment.node_ecr,
+  ]
+
+  tags = {
+    Name = "${var.cluster_name}-workers"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Addon — CoreDNS
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_addon" "coredns" {
+  cluster_name                = aws_eks_cluster.this.name
+  addon_name                  = "coredns"
+  resolve_conflicts_on_create = "OVERWRITE"
+
+  # addon_version omitted on purpose. Spinifex defaults an unset version to its
+  # catalog default (coredns 1.11.1). The AWS provider also rejects a bare
+  # "1.11.1" (it demands a v-prefixed form) which Spinifex's catalog would in
+  # turn reject — so pinning is a dead end here; let the server choose.
+  depends_on = [aws_eks_node_group.workers]
+
+  tags = {
+    Name = "${var.cluster_name}-coredns"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# TLS — self-signed certificate imported into ACM
+# ---------------------------------------------------------------------------
+
+resource "tls_private_key" "ingress" {
+  algorithm = "RSA"
+  rsa_bits  = 2048
+}
+
+resource "tls_self_signed_cert" "ingress" {
+  private_key_pem = tls_private_key.ingress.private_key_pem
+
+  subject {
+    common_name  = var.cert_common_name
+    organization = "Spinifex EKS Demo"
+  }
+
+  dns_names             = [var.cert_common_name]
+  validity_period_hours = 8760
+  early_renewal_hours   = 720
+
+  allowed_uses = [
+    "key_encipherment",
+    "digital_signature",
+    "server_auth",
+  ]
+}
+
+resource "aws_acm_certificate" "ingress" {
+  private_key      = tls_private_key.ingress.private_key_pem
+  certificate_body = tls_self_signed_cert.ingress.cert_pem
+
+  tags = {
+    Name = "${var.cluster_name}-ingress"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# ALB — internet-facing, HTTPS terminated with the ACM cert
+# ---------------------------------------------------------------------------
+
+resource "aws_security_group" "alb" {
+  name        = "${var.cluster_name}-alb-sg"
+  description = "ALB HTTPS inbound, NodePort to workers"
+  vpc_id      = aws_vpc.main.id
+
+  ingress {
+    description = "HTTPS"
+    from_port   = 443
+    to_port     = 443
+    protocol    = "tcp"
+    cidr_blocks = [var.alb_ingress_cidr]
+  }
+
+  egress {
+    description = "All outbound"
+    from_port   = 0
+    to_port     = 0
+    protocol    = "-1"
+    cidr_blocks = ["0.0.0.0/0"]
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-alb-sg"
+  }
+}
+
+resource "aws_lb" "ingress" {
+  name               = "${var.cluster_name}-alb"
+  internal           = false
+  load_balancer_type = "application"
+  security_groups    = [aws_security_group.alb.id]
+  subnets            = [aws_subnet.public_a.id, aws_subnet.public_b.id]
+
+  tags = {
+    Name = "${var.cluster_name}-alb"
+  }
+}
+
+resource "aws_lb_target_group" "workers" {
+  name        = "${var.cluster_name}-tg"
+  port        = var.node_port
+  protocol    = "HTTP"
+  target_type = "instance"
+  vpc_id      = aws_vpc.main.id
+
+  health_check {
+    path                = "/"
+    protocol            = "HTTP"
+    matcher             = "200-399"
+    healthy_threshold   = 2
+    unhealthy_threshold = 3
+    timeout             = 5
+    interval            = 10
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-tg"
+  }
+}
+
+# Discover the launched workers by their Spinifex EKS tag so the ALB can target
+# them. depends_on the node group: the lookup must run after the workers exist.
+data "aws_instances" "workers" {
+  instance_tags = {
+    "spinifex:eks-cluster" = aws_eks_cluster.this.name
+  }
+
+  depends_on = [aws_eks_node_group.workers]
+}
+
+# count is the known desired size, so it's resolvable at plan time even though
+# the worker IDs themselves are only known after the node group launches.
+resource "aws_lb_target_group_attachment" "workers" {
+  count            = var.node_desired_size
+  target_group_arn = aws_lb_target_group.workers.arn
+  target_id        = data.aws_instances.workers.ids[count.index]
+  port             = var.node_port
+}
+
+resource "aws_lb_listener" "https" {
+  load_balancer_arn = aws_lb.ingress.arn
+  port              = 443
+  protocol          = "HTTPS"
+  ssl_policy        = "ELBSecurityPolicy-TLS13-1-2-2021-06"
+  certificate_arn   = aws_acm_certificate.ingress.arn
+
+  default_action {
+    type             = "forward"
+    target_group_arn = aws_lb_target_group.workers.arn
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Let the ALB reach the workers' NodePort
+#
+# Spinifex auto-manages the nodegroup SG and admits only intra-cluster traffic.
+# Look it up by its deterministic name and admit the ALB SG on the NodePort.
+# ---------------------------------------------------------------------------
+
+data "aws_security_group" "nodegroup" {
+  filter {
+    name   = "group-name"
+    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
+  }
+
+  filter {
+    name   = "vpc-id"
+    values = [aws_vpc.main.id]
+  }
+
+  depends_on = [aws_eks_node_group.workers]
+}
+
+resource "aws_vpc_security_group_ingress_rule" "nodeport_from_alb" {
+  security_group_id            = data.aws_security_group.nodegroup.id
+  referenced_security_group_id = aws_security_group.alb.id
+  from_port                    = var.node_port
+  to_port                      = var.node_port
+  ip_protocol                  = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-alb-nodeport"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Demo workload — three replicas spread across the workers
+# ---------------------------------------------------------------------------
+
+resource "kubernetes_deployment_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+    labels    = { app = "hello" }
+  }
+
+  spec {
+    replicas = var.node_desired_size
+
+    selector {
+      match_labels = { app = "hello" }
+    }
+
+    template {
+      metadata {
+        labels = { app = "hello" }
+      }
+
+      spec {
+        container {
+          name  = "hello"
+          image = "nginxdemos/hello"
+
+          port {
+            container_port = 80
+          }
+        }
+
+        # Spread one replica per node so a refresh visibly hits different nodes.
+        topology_spread_constraint {
+          max_skew           = 1
+          topology_key       = "kubernetes.io/hostname"
+          when_unsatisfiable = "ScheduleAnyway"
+
+          label_selector {
+            match_labels = { app = "hello" }
+          }
+        }
+      }
+    }
+  }
+
+  depends_on = [aws_eks_addon.coredns]
+}
+
+resource "kubernetes_service_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+  }
+
+  spec {
+    selector = { app = "hello" }
+    type     = "NodePort"
+
+    port {
+      port        = 80
+      target_port = 80
+      node_port   = var.node_port
+    }
+  }
+
+  depends_on = [kubernetes_deployment_v1.hello]
+}
+
+# ---------------------------------------------------------------------------
+# Outputs
+# ---------------------------------------------------------------------------
+
+output "cluster_name" {
+  value = aws_eks_cluster.this.name
+}
+
+output "certificate_arn" {
+  value = aws_acm_certificate.ingress.arn
+}
+
+output "alb_dns_name" {
+  value = aws_lb.ingress.dns_name
+}
+
+output "alb_public_ip_hint" {
+  value = "aws elbv2 describe-load-balancers --names ${var.cluster_name}-alb --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' --output text"
+}
+
+output "demo_url_hint" {
+  value = "Fetch the ALB IP via alb_public_ip_hint, then open https://<that-ip> (self-signed cert: curl -k). Refresh to see different pods answer."
+}
+
+output "update_kubeconfig" {
+  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
+}

docs/terraform-workbooks/eks-quickstart/README.md

@@ -0,0 +1,173 @@
+---
+title: "EKS Quickstart"
+description: "Stand up a minimal managed-Kubernetes cluster and a browsable demo app — VPC, IAM roles, an EKS cluster, a single-worker node group, and a load-balanced web page — using Terraform on Spinifex."
+category: "Terraform Workbooks"
+tags:
+  - terraform
+  - eks
+  - kubernetes
+  - iam
+  - vpc
+  - workbook
+resources:
+  - title: "Terraform AWS Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
+  - title: "Terraform Kubernetes Provider"
+    url: "https://registry.terraform.io/providers/hashicorp/kubernetes/latest"
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "OpenTofu"
+    url: "https://opentofu.org/"
+---
+
+# Terraform: EKS Quickstart
+
+> The smallest end-to-end EKS example on Spinifex: a VPC, the two IAM roles EKS needs, a single-worker cluster, and a demo web app you can open in a browser.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Instructions](#instructions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+Provision a managed Kubernetes cluster on Spinifex with Terraform/OpenTofu — and finish with something you can actually see. As well as the cluster, this workbook deploys a small demo app onto it and publishes it on the worker's public IP. When `apply` finishes, open the **`demo_url`** output: the page is served by `nginxdemos/hello` and prints the name of the pod that answered. Refresh it and the answering pod changes, so even a non-technical viewer can watch Kubernetes scheduling and load-balancing in real time.
+
+Under the hood it keeps things minimal: a VPC with two public subnets, an IAM role for the control plane, an IAM role for the workers with the three managed policies AWS-managed node groups expect, a public-endpoint `aws_eks_cluster`, and a single-node `aws_eks_node_group`. Terraform then uses the **Kubernetes provider** (authenticating exactly like `kubectl` does, via `aws eks get-token`) to deploy the Deployment + NodePort Service.
+
+**What you'll learn:**
+
+- Configuring the AWS provider to target Spinifex's `eks`, `ec2`, `iam` and `sts` endpoints
+- Creating the EKS cluster and worker IAM roles with faithful managed-policy attachments
+- Provisioning an `aws_eks_cluster` and a managed `aws_eks_node_group`
+- Pointing the Kubernetes provider at the new cluster and deploying a workload from the same `apply`
+- Opening one rule on the auto-managed worker security group to expose a NodePort
+
+**What gets created**
+
+| Resource | Name | Purpose |
+|---|---|---|
+| VPC | `eks-quickstart-vpc` | Isolated network (10.30.0.0/16) |
+| Subnets | `eks-quickstart-subnet-a/-b` | Public subnets for the cluster and worker |
+| Internet Gateway | `eks-quickstart-igw` | Egress so the worker can pull the demo image |
+| IAM Roles | `eks-quickstart-cluster-role`, `eks-quickstart-node-role` | Control-plane and worker roles |
+| EKS Cluster | `eks-quickstart` | Public API endpoint, API auth mode, Kubernetes 1.32 |
+| Node Group | `default` | One `t3.medium` worker |
+| SG Ingress Rule | `eks-quickstart-demo-nodeport` | Opens the demo NodePort on the auto-managed worker SG |
+| K8s Deployment | `hello` | `nginxdemos/hello`, 2 replicas |
+| K8s Service | `hello` | NodePort publishing the demo on the worker |
+
+**Spinifex specifics**
+
+- The cluster's security groups are **auto-managed** — `vpc_config.security_group_ids` is ignored. To reach a NodePort, this workbook looks the worker SG up by its deterministic name (`eks-cluster-<name>-nodegroup-sg`) and adds a single ingress rule.
+- The worker AMI is always Spinifex's `eks-node` image; `ami_type` is recorded but does **not** select the image.
+- `authentication_mode` must be `"API"` (the `API_AND_CONFIG_MAP` mode is rejected).
+- The workers need outbound internet (here via the IGW) to pull the demo container image.
+
+**Prerequisites:**
+
+- Spinifex installed and running (see [Installing Spinifex](/docs/install))
+- The Spinifex `eks-node` image available on the cluster
+- OpenTofu or Terraform, plus `kubectl` and the AWS CLI
+
+## Instructions
+
+### Step 1. Get the Template
+
+```bash
+git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
+cd spinifex-tf
+git sparse-checkout set docs/terraform-workbooks
+cd docs/terraform-workbooks/eks-quickstart
+```
+
+Or create a `main.tf` file and paste the full configuration below.
+
+<!-- INCLUDE: main.tf lang:hcl -->
+
+### Step 2. Deploy
+
+```bash
+export AWS_PROFILE=spinifex
+tofu init
+tofu apply
+```
+
+This single `apply` does the whole job: it creates the cluster (which bootstraps a control-plane VM and brings up k3s — a few minutes in `CREATING`), launches the worker, opens the NodePort, then deploys the demo app and waits for the pods to roll out.
+
+> **Same profile for the Kubernetes provider.** The Kubernetes provider authenticates by shelling out to `aws eks get-token`, which has to reach the Spinifex STS endpoint. Keep `AWS_PROFILE=spinifex` exported for the whole `apply`.
+
+### Step 3. Open the Demo
+
+```bash
+tofu output demo_url
+```
+
+Open that URL in a browser. You'll see the `nginxdemos/hello` page showing the **Server name** (the pod that handled the request). Refresh a few times — with two replicas, the pod name alternates, demonstrating that the Service is load-balancing across the cluster.
+
+### Step 4. Inspect with kubectl (optional)
+
+```bash
+aws eks update-kubeconfig --name eks-quickstart --region ap-southeast-2
+kubectl get nodes
+kubectl get pods -o wide
+```
+
+`kubectl get pods -o wide` shows the demo pods and which node each landed on.
+
+### Cleanup
+
+```bash
+tofu destroy
+```
+
+## Troubleshooting
+
+### Demo URL Doesn't Load
+
+The page is served from a NodePort on the worker's public IP. Work through the chain:
+
+```bash
+# Is the worker running with a public IP?
+aws ec2 describe-instances --filters "Name=tag:spinifex:eks-cluster,Values=eks-quickstart" \
+  --query 'Reservations[].Instances[].[InstanceId,State.Name,PublicIpAddress]' --output text
+
+# Did the demo pods roll out?
+kubectl get pods -o wide
+
+# Is the NodePort rule on the worker SG?
+aws ec2 describe-security-groups --filters "Name=group-name,Values=eks-cluster-eks-quickstart-nodegroup-sg" \
+  --query 'SecurityGroups[0].IpPermissions'
+```
+
+If the pods are `Pending`, the worker may not be `Ready` yet — give it a moment. If they're `ImagePullBackOff`, the worker can't reach Docker Hub; confirm the IGW route and the worker's public IP.
+
+### kubernetes provider: connection refused / Unauthorized
+
+The provider runs `aws eks get-token` against the Spinifex STS endpoint. Confirm the AWS CLI is still pointed at Spinifex and the cluster is `ACTIVE`:
+
+```bash
+aws sts get-caller-identity
+aws eks describe-cluster --name eks-quickstart --query 'cluster.status'
+```
+
+If the cluster was still `CREATING` when the provider first tried to connect, just re-run `tofu apply`.
+
+### Cluster Stuck in CREATING
+
+Control-plane bootstrap takes a few minutes. Confirm the underlying VM came up:
+
+```bash
+aws eks describe-cluster --name eks-quickstart --query 'cluster.status'
+aws ec2 describe-instances --profile spinifex
+```
+
+### Provider Connection Refused
+
+```bash
+sudo systemctl status spinifex.target
+curl -k https://localhost:9999/
+```

docs/terraform-workbooks/eks-quickstart/main.tf

@@ -0,0 +1,428 @@
+# Example: EKS Quickstart on Spinifex
+#
+# A minimal managed-Kubernetes cluster that ends in something you can see: a
+# VPC, the two IAM roles EKS needs, a single-worker cluster, and a demo web app
+# deployed onto it by Terraform and exposed on the worker's public IP.
+#
+# Once `apply` finishes, open the demo_url output in a browser. The page is
+# served by the nginxdemos/hello image and prints the pod name that answered —
+# refresh and it bounces between the app's replicas, showing the cluster is
+# really scheduling and load-balancing pods.
+#
+# Demonstrates: VPC + subnets, EKS cluster + node IAM roles, an aws_eks_cluster
+# and managed aws_eks_node_group, the Kubernetes provider authenticating to the
+# cluster, and a Deployment + NodePort Service reachable from your browser.
+#
+# Usage:
+#   cd spinifex/docs/terraform-workbooks/eks-quickstart
+#   export AWS_PROFILE=spinifex
+#   tofu init && tofu apply
+#   # then open the demo_url output
+
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.40, < 6.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.20"
+    }
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Variables
+# ---------------------------------------------------------------------------
+
+variable "region" {
+  type    = string
+  default = "ap-southeast-2"
+}
+
+variable "cluster_name" {
+  type    = string
+  default = "eks-quickstart"
+}
+
+variable "k8s_version" {
+  type        = string
+  default     = "1.32"
+  description = "Kubernetes minor version for the control plane and workers"
+}
+
+variable "instance_type" {
+  type    = string
+  default = "t3.medium"
+}
+
+variable "node_port" {
+  type        = number
+  default     = 30080
+  description = "NodePort the demo Service is published on"
+}
+
+variable "replicas" {
+  type        = number
+  default     = 2
+  description = "Demo app replicas; refresh the page to see requests land on different pods"
+}
+
+variable "browse_cidr" {
+  type        = string
+  default     = "0.0.0.0/0"
+  description = "CIDR allowed to reach the demo NodePort; tighten to your own IP in production"
+}
+
+variable "spinifex_endpoint" {
+  type        = string
+  default     = "https://127.0.0.1:9999"
+  description = "Spinifex AWS gateway endpoint"
+}
+
+# ---------------------------------------------------------------------------
+# Providers
+# ---------------------------------------------------------------------------
+
+provider "aws" {
+  region = var.region
+
+  endpoints {
+    ec2 = var.spinifex_endpoint
+    iam = var.spinifex_endpoint
+    sts = var.spinifex_endpoint
+    eks = var.spinifex_endpoint
+  }
+
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+  skip_region_validation      = true
+}
+
+# Authenticates to the cluster with the same `aws eks get-token` exec flow the
+# generated kubeconfig uses, so the Kubernetes provider can deploy the demo app.
+provider "kubernetes" {
+  host                   = aws_eks_cluster.this.endpoint
+  cluster_ca_certificate = base64decode(aws_eks_cluster.this.certificate_authority[0].data)
+
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    command     = "aws"
+    args        = ["eks", "get-token", "--cluster-name", aws_eks_cluster.this.name, "--region", var.region]
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Data sources
+# ---------------------------------------------------------------------------
+
+data "aws_availability_zones" "available" {
+  state = "available"
+}
+
+# ---------------------------------------------------------------------------
+# VPC + two public subnets (workers get public IPs to pull the demo image)
+# ---------------------------------------------------------------------------
+
+resource "aws_vpc" "main" {
+  cidr_block           = "10.30.0.0/16"
+  enable_dns_hostnames = true
+  enable_dns_support   = true
+
+  tags = {
+    Name = "${var.cluster_name}-vpc"
+  }
+}
+
+resource "aws_internet_gateway" "igw" {
+  vpc_id = aws_vpc.main.id
+
+  tags = {
+    Name = "${var.cluster_name}-igw"
+  }
+}
+
+resource "aws_subnet" "a" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.30.1.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-subnet-a"
+  }
+}
+
+resource "aws_subnet" "b" {
+  vpc_id                  = aws_vpc.main.id
+  cidr_block              = "10.30.2.0/24"
+  availability_zone       = data.aws_availability_zones.available.names[0]
+  map_public_ip_on_launch = true
+
+  tags = {
+    Name = "${var.cluster_name}-subnet-b"
+  }
+}
+
+resource "aws_route_table" "public" {
+  vpc_id = aws_vpc.main.id
+
+  route {
+    cidr_block = "0.0.0.0/0"
+    gateway_id = aws_internet_gateway.igw.id
+  }
+
+  tags = {
+    Name = "${var.cluster_name}-public-rt"
+  }
+}
+
+resource "aws_route_table_association" "a" {
+  subnet_id      = aws_subnet.a.id
+  route_table_id = aws_route_table.public.id
+}
+
+resource "aws_route_table_association" "b" {
+  subnet_id      = aws_subnet.b.id
+  route_table_id = aws_route_table.public.id
+}
+
+# ---------------------------------------------------------------------------
+# IAM — EKS cluster role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "cluster" {
+  name = "${var.cluster_name}-cluster-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "eks.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "cluster" {
+  role       = aws_iam_role.cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
+}
+
+# ---------------------------------------------------------------------------
+# IAM — worker node role
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "node" {
+  name = "${var.cluster_name}-node-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Action    = "sts:AssumeRole"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "node_worker" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_cni" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_ecr" {
+  role       = aws_iam_role.node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+}
+
+# ---------------------------------------------------------------------------
+# EKS cluster — public API endpoint, API (access-entry) auth mode
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_cluster" "this" {
+  name     = var.cluster_name
+  role_arn = aws_iam_role.cluster.arn
+  version  = var.k8s_version
+
+  vpc_config {
+    subnet_ids              = [aws_subnet.a.id, aws_subnet.b.id]
+    endpoint_public_access  = true
+    endpoint_private_access = false
+  }
+
+  access_config {
+    authentication_mode                         = "API"
+    bootstrap_cluster_creator_admin_permissions = true
+  }
+
+  depends_on = [aws_iam_role_policy_attachment.cluster]
+
+  tags = {
+    Name = var.cluster_name
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Managed node group — one worker
+# ---------------------------------------------------------------------------
+
+resource "aws_eks_node_group" "default" {
+  cluster_name    = aws_eks_cluster.this.name
+  node_group_name = "default"
+  node_role_arn   = aws_iam_role.node.arn
+  subnet_ids      = [aws_subnet.a.id, aws_subnet.b.id]
+
+  scaling_config {
+    desired_size = 1
+    min_size     = 1
+    max_size     = 2
+  }
+
+  instance_types = [var.instance_type]
+  ami_type       = "AL2_x86_64"
+
+  depends_on = [
+    aws_iam_role_policy_attachment.node_worker,
+    aws_iam_role_policy_attachment.node_cni,
+    aws_iam_role_policy_attachment.node_ecr,
+  ]
+
+  tags = {
+    Name = "${var.cluster_name}-default"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Open the NodePort on the auto-managed nodegroup SG
+#
+# Spinifex creates the worker SG itself (vpc_config.security_group_ids is
+# ignored) and admits only intra-cluster traffic. To reach the demo NodePort
+# from a browser, look the SG up by its deterministic name and add one rule.
+# ---------------------------------------------------------------------------
+
+data "aws_security_group" "nodegroup" {
+  filter {
+    name   = "group-name"
+    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
+  }
+
+  filter {
+    name   = "vpc-id"
+    values = [aws_vpc.main.id]
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+resource "aws_vpc_security_group_ingress_rule" "nodeport" {
+  security_group_id = data.aws_security_group.nodegroup.id
+  cidr_ipv4         = var.browse_cidr
+  from_port         = var.node_port
+  to_port           = var.node_port
+  ip_protocol       = "tcp"
+
+  tags = {
+    Name = "${var.cluster_name}-demo-nodeport"
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Demo workload — deployed onto the cluster by Terraform
+#
+# nginxdemos/hello renders a page showing which pod served the request; with
+# multiple replicas, refreshing the demo_url alternates between them.
+# ---------------------------------------------------------------------------
+
+resource "kubernetes_deployment_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+    labels    = { app = "hello" }
+  }
+
+  spec {
+    replicas = var.replicas
+
+    selector {
+      match_labels = { app = "hello" }
+    }
+
+    template {
+      metadata {
+        labels = { app = "hello" }
+      }
+
+      spec {
+        container {
+          name  = "hello"
+          image = "nginxdemos/hello"
+
+          port {
+            container_port = 80
+          }
+        }
+      }
+    }
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+resource "kubernetes_service_v1" "hello" {
+  metadata {
+    name      = "hello"
+    namespace = "default"
+  }
+
+  spec {
+    selector = { app = "hello" }
+    type     = "NodePort"
+
+    port {
+      port        = 80
+      target_port = 80
+      node_port   = var.node_port
+    }
+  }
+
+  depends_on = [kubernetes_deployment_v1.hello]
+}
+
+# ---------------------------------------------------------------------------
+# Discover the worker's public IP for the demo URL
+# ---------------------------------------------------------------------------
+
+data "aws_instances" "workers" {
+  instance_tags = {
+    "spinifex:eks-cluster" = aws_eks_cluster.this.name
+  }
+
+  depends_on = [aws_eks_node_group.default]
+}
+
+# ---------------------------------------------------------------------------
+# Outputs
+# ---------------------------------------------------------------------------
+
+output "cluster_name" {
+  value = aws_eks_cluster.this.name
+}
+
+output "demo_url" {
+  value       = "http://${data.aws_instances.workers.public_ips[0]}:${var.node_port}"
+  description = "Open in a browser; refresh to see different pods answer"
+}
+
+output "update_kubeconfig" {
+  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
+}

docs/terraform-workbooks/s3-webapp/README.md

@@ -9,6 +9,9 @@ tags:
   - flask
   - webapp
   - workbook
+  - imds
+  - sts
+  - instance-profile
 resources:
   - title: "Terraform AWS Provider"
     url: "https://registry.terraform.io/providers/hashicorp/aws/latest"
@@ -36,18 +39,21 @@ resources:
 
 Deploy an EC2 instance running a Flask file-sharing web application backed by S3 (Predastore). Users can upload files through a web form and browse uploaded content — demonstrating Terraform managing both compute and object storage together.
 
+This workbook uses the idiomatic AWS credential model: the instance is launched with an **IAM instance profile** and the app pulls short-lived STS credentials from **IMDS** (`169.254.169.254`) through boto3's default credential chain. **No long-lived S3 keys are baked into the instance.** The Terraform run still uses the operator's admin credentials (to create the bucket, role, profile, and instance), but the running instance authenticates to S3 with credentials it fetches at runtime and which expire in ~1 hour.
+
 **Architecture:**
 
 <p align="center">
-  <img src="../../../.github/assets/diagrams/tf-s3-webapp.svg" alt="S3 webapp — browser to Flask EC2 instance to Predastore via S3 API" width="900">
+  <img src="../../../.github/assets/diagrams/tf-s3-webapp.svg" alt="S3 webapp — browser to Flask EC2 instance; the instance fetches short-lived STS credentials from IMDS (169.254.169.254) via its instance profile, then calls Predastore over the S3 API" width="900">
 </p>
 
 **What you'll learn:**
 
 - Configuring the AWS provider with both Spinifex and Predastore endpoints
 - Creating S3 buckets on Predastore via Terraform
-- Deploying a Python webapp with cloud-init that talks to S3
-- Passing credentials and configuration to instances via user-data
+- Defining an IAM role, least-privilege managed policy, and instance profile, and passing the role to an instance (`iam:PassRole`)
+- How the instance obtains short-lived credentials from IMDS via boto3's default credential chain — no static keys on the instance
+- Deploying a Python webapp with cloud-init that signs S3 requests with the STS session token
 
 **Prerequisites:**
 
@@ -55,6 +61,7 @@ Deploy an EC2 instance running a Flask file-sharing web application backed by S3
 - Predastore running (S3 API on port 8443)
 - An Ubuntu 26.04 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
 - OpenTofu or Terraform installed
+- The operator identity running the Terraform apply (`AWS_PROFILE=spinifex`) must be allowed to manage IAM and pass the role: `iam:CreateRole`, `iam:CreatePolicy`, `iam:AttachRolePolicy`, `iam:CreateInstanceProfile`, `iam:AddRoleToInstanceProfile`, and `iam:PassRole` on the new role. The bootstrap admin profile satisfies this.
 - The EC2 instance must be able to reach Predastore — use the host's br-wan IP, not localhost
 
 ## Instructions
@@ -141,16 +148,39 @@ sudo systemctl status s3-webapp
 sudo journalctl -u s3-webapp --no-pager -n 50
 ```
 
-### Upload Fails or Files Not Appearing
+### Upload Fails with `403 AccessDenied`
 
-Check that the S3 credentials in `/opt/webapp/.env` are correct and that the bucket exists:
+The instance reached S3 but was not authorized. The `.env` file deliberately holds **no** keys — credentials come from IMDS:
 
 ```bash
 ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
-cat /opt/webapp/.env
-/opt/webapp/venv/bin/python -c "import boto3; print('boto3 OK')"
+cat /opt/webapp/.env   # S3_ENDPOINT / S3_BUCKET / S3_REGION only — no keys
 ```
 
+Check that:
+
+- the managed policy is attached to the role (`s3:ListBucket` on the bucket, `s3:GetObject`/`s3:PutObject` on its objects), and the policy's bucket name matches `bucket_name`;
+- the assumed-role session's account matches the bucket owner's account — Predastore enforces bucket ownership. Both are created by the same operator identity, so they align by construction; a mismatch surfaces as `403`.
+
+### `NoCredentialsError` / No Credentials
+
+boto3 could not obtain credentials, which means the IMDS credential chain did not resolve. From the instance:
+
+```bash
+ssh -i s3-webapp-demo.pem ec2-user@<public_ip>
+
+# IMDSv2: fetch a token, then the role name behind the instance profile
+TOKEN=$(curl -sX PUT http://169.254.169.254/latest/api/token \
+  -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')
+curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
+  http://169.254.169.254/latest/meta-data/iam/security-credentials/
+
+# Confirm the resolved identity — expect assumed-role/s3-webapp-role/<instance-id>
+aws sts get-caller-identity --endpoint-url https://<predastore_host>
+```
+
+If the role name is empty, the instance profile was not attached — confirm `iam_instance_profile` on the instance and that `iam:PassRole` is allowed for the operator.
+
 ### AMI Not Found
 
 Ensure you have imported an Ubuntu 26.04 image:

docs/terraform-workbooks/s3-webapp/main.tf

@@ -8,13 +8,25 @@
 # Architecture:
 #
 #   Browser ──HTTP──▶ EC2 Instance (Flask webapp, port 80)
-#                         │
-#                         ▼ S3 API (boto3)
+#                         │  ▲ IMDS: short-lived STS creds (ASIA + token)
+#                         │  └──── 169.254.169.254
+#                         ▼ S3 API (boto3, SigV4 with session token)
 #                     Predastore (port 8443)
 #
+# Credentials:
+#   The Terraform run uses the operator's admin keys (s3_access_key /
+#   s3_secret_key) to create the bucket, IAM role, instance profile, and
+#   instance. The INSTANCE receives no long-lived secret — boto3's default
+#   credential chain pulls short-lived STS credentials from IMDS
+#   (169.254.169.254), resolved via the attached instance profile -> role.
+#
 # Prerequisites:
 #   - Spinifex services running (gateway on port 9999)
 #   - Predastore running (S3 API on port 8443)
+#   - The operator identity (AWS_PROFILE=spinifex) must be allowed
+#     iam:CreateRole / iam:CreatePolicy / iam:AttachRolePolicy /
+#     iam:CreateInstanceProfile / iam:AddRoleToInstanceProfile and
+#     iam:PassRole on the new role.
 #   - The EC2 instance must be able to reach the Predastore endpoint.
 #     Set `predastore_host` to the IP reachable from inside the VPC
 #     (e.g. the host's br-wan IP, NOT localhost).
@@ -80,13 +92,13 @@ variable "predastore_host" {
 
 variable "s3_access_key" {
   type        = string
-  description = "S3 access key for Predastore"
+  description = "Operator S3 access key for the Terraform run (creates the bucket on Predastore); the instance uses IMDS, not this key"
 }
 
 variable "s3_secret_key" {
   type        = string
   sensitive   = true
-  description = "S3 secret key for Predastore"
+  description = "Operator S3 secret key for the Terraform run; the instance uses IMDS, not this key"
 }
 
 variable "bucket_name" {
@@ -154,6 +166,55 @@ resource "aws_s3_bucket" "uploads" {
   bucket = var.bucket_name
 }
 
+# ---------------------------------------------------------------------------
+# IAM Instance Role — least-privilege S3 access fetched at runtime via IMDS
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "webapp" {
+  name = "s3-webapp-role"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Effect    = "Allow"
+      Principal = { Service = "ec2.amazonaws.com" }
+      Action    = "sts:AssumeRole"
+    }]
+  })
+}
+
+resource "aws_iam_policy" "webapp_s3" {
+  name = "s3-webapp-policy"
+
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Sid      = "ListBucket"
+        Effect   = "Allow"
+        Action   = ["s3:ListBucket"]
+        Resource = ["arn:aws:s3:::${var.bucket_name}"]
+      },
+      {
+        Sid      = "ObjectRW"
+        Effect   = "Allow"
+        Action   = ["s3:GetObject", "s3:PutObject"]
+        Resource = ["arn:aws:s3:::${var.bucket_name}/*"]
+      }
+    ]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "webapp" {
+  role       = aws_iam_role.webapp.name
+  policy_arn = aws_iam_policy.webapp_s3.arn
+}
+
+resource "aws_iam_instance_profile" "webapp" {
+  name = "s3-webapp-profile"
+  role = aws_iam_role.webapp.name
+}
+
 # ---------------------------------------------------------------------------
 # SSH Key Pair
 # ---------------------------------------------------------------------------
@@ -273,9 +334,14 @@ resource "aws_instance" "webapp" {
   subnet_id              = aws_subnet.public.id
   vpc_security_group_ids = [aws_security_group.webapp.id]
   key_name               = aws_key_pair.webapp.key_name
+  iam_instance_profile   = aws_iam_instance_profile.webapp.name
 
   associate_public_ip_address = true
 
+  # The role's permissions must exist before the instance boots and makes its
+  # first IMDS-credentialed S3 call
+  depends_on = [aws_iam_role_policy_attachment.webapp]
+
   user_data_base64 = base64encode(<<-USERDATA
     #!/bin/bash
     set -euo pipefail
@@ -293,8 +359,6 @@ resource "aws_instance" "webapp" {
     cat > /opt/webapp/.env <<'ENVFILE'
     S3_ENDPOINT=https://${var.predastore_host}
     S3_BUCKET=${var.bucket_name}
-    S3_ACCESS_KEY=${var.s3_access_key}
-    S3_SECRET_KEY=${var.s3_secret_key}
     S3_REGION=${var.region}
     ENVFILE
 
@@ -321,8 +385,6 @@ resource "aws_instance" "webapp" {
     s3 = boto3.client(
         "s3",
         endpoint_url=env["S3_ENDPOINT"],
-        aws_access_key_id=env["S3_ACCESS_KEY"],
-        aws_secret_access_key=env["S3_SECRET_KEY"],
         region_name=env["S3_REGION"],
         verify=False,
         config=Config(s3={"addressing_style": "path"}),

docs/terraform-workbooks/s3-webapp/terraform.tfvars.example

@@ -2,6 +2,11 @@
 #
 # The predastore_host must be reachable from INSIDE the VPC guest — not
 # localhost. Use the host's br-wan or LAN IP, e.g. "192.168.1.10:8443".
+#
+# s3_access_key / s3_secret_key are the operator credentials for the Terraform
+# run only — they create the bucket on Predastore and the IAM role/profile.
+# They are NOT passed to the instance; the instance pulls short-lived STS
+# credentials from IMDS via its instance profile.
 
 predastore_host = "192.168.1.10:8443"
 s3_access_key   = "AKIAIOSFODNN7EXAMPLE"

go.mod

@@ -13,8 +13,8 @@ require (
 	github.com/insomniacslk/dhcp v0.0.0-20260407060928-11b94ed970f2
 	github.com/kdomanski/iso9660 v0.4.0
 	github.com/klauspost/cpuid/v2 v2.3.0
-	github.com/mulgadc/predastore v1.7.0
-	github.com/mulgadc/viperblock v1.7.0
+	github.com/mulgadc/predastore v1.8.0
+	github.com/mulgadc/viperblock v1.8.0
 	github.com/nats-io/nats-server/v2 v2.14.2
 	github.com/nats-io/nats.go v1.52.0
 	github.com/ovn-kubernetes/libovsdb v0.8.1
@@ -24,10 +24,10 @@ require (
 	github.com/spf13/viper v1.21.0
 	github.com/stretchr/testify v1.11.1
 	go.uber.org/goleak v1.3.0
-	golang.org/x/crypto v0.52.0
-	golang.org/x/sync v0.20.0
-	golang.org/x/sys v0.45.0
-	gopkg.in/ini.v1 v1.67.2
+	golang.org/x/crypto v0.53.0
+	golang.org/x/sync v0.21.0
+	golang.org/x/sys v0.46.0
+	gopkg.in/ini.v1 v1.67.3
 	gopkg.in/yaml.v3 v3.0.1
 )
 
@@ -38,7 +38,7 @@ require (
 	github.com/antithesishq/antithesis-sdk-go v0.7.0 // indirect
 	github.com/armon/go-metrics v0.4.1 // indirect
 	github.com/atotto/clipboard v0.1.4 // indirect
-	github.com/aws/aws-sdk-go-v2 v1.41.12 // indirect
+	github.com/aws/aws-sdk-go-v2 v1.42.0 // indirect
 	github.com/aws/smithy-go v1.27.1 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
@@ -57,7 +57,7 @@ require (
 	github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
 	github.com/containerd/console v1.0.5 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/dgraph-io/badger/v4 v4.9.1 // indirect
+	github.com/dgraph-io/badger/v4 v4.9.2 // indirect
 	github.com/dgraph-io/ristretto/v2 v2.4.0 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
@@ -126,13 +126,13 @@ require (
 	go.opentelemetry.io/otel/trace v1.42.0 // indirect
 	go.yaml.in/yaml/v2 v2.4.4 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
-	golang.org/x/mod v0.35.0 // indirect
-	golang.org/x/net v0.55.0 // indirect
-	golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa // indirect
-	golang.org/x/term v0.43.0 // indirect
-	golang.org/x/text v0.37.0 // indirect
+	golang.org/x/mod v0.36.0 // indirect
+	golang.org/x/net v0.56.0 // indirect
+	golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6 // indirect
+	golang.org/x/term v0.44.0 // indirect
+	golang.org/x/text v0.38.0 // indirect
 	golang.org/x/time v0.15.0 // indirect
-	golang.org/x/tools v0.44.0 // indirect
+	golang.org/x/tools v0.45.0 // indirect
 	golang.org/x/vuln v1.1.4 // indirect
 	google.golang.org/protobuf v1.36.11 // indirect
 )

go.sum

@@ -31,28 +31,28 @@ github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z
 github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
 github.com/aws/aws-sdk-go v1.55.8 h1:JRmEUbU52aJQZ2AjX4q4Wu7t4uZjOu71uyNmaWlUkJQ=
 github.com/aws/aws-sdk-go v1.55.8/go.mod h1:ZkViS9AqA6otK+JBBNH2++sx1sgxrPKcSzPPvQkUtXk=
-github.com/aws/aws-sdk-go-v2 v1.41.12 h1:DIKX2c31ekm9RA2D9FBj1EWXx++9AdAqRw+e78Tq2Ck=
-github.com/aws/aws-sdk-go-v2 v1.41.12/go.mod h1:27+ACypSLljLAEKsCYOmrjKh83vuTRkuAe9Uv/3A4bg=
+github.com/aws/aws-sdk-go-v2 v1.42.0 h1:XvXMJTkFQtpBKIWZnmr9ZEOc2InWM2yldjXEJ/bymhA=
+github.com/aws/aws-sdk-go-v2 v1.42.0/go.mod h1:27+ACypSLljLAEKsCYOmrjKh83vuTRkuAe9Uv/3A4bg=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.13 h1:p1BBrg/Hhp6uK7zpejeI8QFXHJeC/mynzi04Sl03k9g=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.13/go.mod h1:8cIfkE9MDhkRZGpQ22aV6/lkYeYSozpz16Smrs5x4Ls=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.22 h1:SHfH6wyPsEgG7fVsi5rQxWEt7tuIcN2PGhb1mTFv6tE=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.22/go.mod h1:54nO8lKD4aQPOntM/VTWjnR+DYzTwx0YkSMZMhAgewQ=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.28 h1:Xf2j7NdVcUKomlZ4iihOP4AZ3Fzlr8h4yKpXeP+OFPg=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.28/go.mod h1:O8cDo1dW63jU7ki//kRe1z+tLGcpnD1jrouitsQddDw=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.28 h1:KqIfN9kpkKkcBqBbNpNGTIrXO6ExTUvFKvXkC+YAzVo=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.28/go.mod h1:uxtQiKvLtNS4iXVsH2McVD/ls8FKN/uUhe1hGxPjrw0=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.29 h1:VkE9FuzTQVjBBrnj4+oCdxCLFIz7aqLYKUCjtvxVcOs=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.29/go.mod h1:H32Z2Qth9b+9LqjyBsCnozMQ8H2N7YBUDVXwbs0iggg=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.24 h1:2hQqYCV9yqyePQ9o6dCrZc/zO8U3TwPr9mIKlZnPu/I=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.24/go.mod h1:IDwpACtwqHLISdzfwUUNq4P9DsB/h5BLg4FwJPNfqFY=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.29 h1:f3vKqSo13fhTYb+JEcXwXefZQE26I1FB5eTSniU67ko=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.29/go.mod h1:MzoLFUArKGpGD+ukmPiTPG1X5x4o6M2kq4v2dr1FiEc=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.29 h1:RdwIf/CuUsvJX3RgJagbOyotl/cxoLY4xviKuE7p2GY=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.29/go.mod h1:71wt8W2EgswdZy9Mf9KNnzxZ3TiZlv4caKghPktDOkA=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.30 h1:VTGy885W5DKBxWRUJbym9hytNaYzsyaPkCHGRRMAOhU=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.30/go.mod h1:AS0HycUvJRFvTt613AYDOgO2jzw+00cVSMny8XB3yMY=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.12 h1:ZD2+BSw9vFsNlKYIasSNt3uDbjqqXIBcM13UJv/Lx2k=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.12/go.mod h1:Ms4zlcVBbXbiP7EVLhl+lgjvA/a7YphqQ3Ih3174EmI=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.21 h1:FsZxbPiVgEHYofziwfylouMki8b1Z7mI4CMU/7bhwBA=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.21/go.mod h1:Mmm30OV+JLXYQUcbSd84THnv3P5JtjhVDujLwMqRG0U=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.28 h1:axj4mEDletwKmTm/9jR+DkIMmCfcn5vE4jBMAAN+3Vg=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.28/go.mod h1:3Aaz69M0jqfSHLKqxgolgUBFT4hpwSNc7DzC95orEi8=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.28 h1:li8rTZAAb22g4UsxbjwMdaNVWbgVcDzPqI7nDTI+mF4=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.28/go.mod h1:/brXioSGIMEdcBFoubpSdmighSVp6poP+mma/wB7iHA=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.103.2 h1:b4ikkRk22T4xYkEgaWc3Voe+3xbt5YbbFhNehOWyUiY=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.103.2/go.mod h1:Gp7eHZ0NZ8ZK5RXpoIUp/C8OeAmJqpCgdwEK1D/QOek=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.22 h1:V51LGlOq/1VsDsHUdoklAQi7rMmx4qQubvFYAlP2254=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.22/go.mod h1:4Pzhyz8hJOm2bepgl+NjvRx8vlUFAIIvJnZ/MkcNPpU=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.29 h1:DRebniUGZ2MqiiIVmQJ04vIXr918hubdHMnarSLEWyU=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.29/go.mod h1:LfRkPCD8YHDM2E5eTkos2UpwYeZnBcVarTa8L59bJHA=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.29 h1:hiME6pBzC7OTl9LMtlyTWBuEl1f4QBcUmFDKC7MLXtc=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.29/go.mod h1:G7RP+uhagpKtKhd1BM9N6JQqjCcGEU47K5lBVZQyRQw=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.103.3 h1:JRseEu/vIDMaWis4bSw0QbXL+cvIGc1XnX076H5ZXLE=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.103.3/go.mod h1:77ZAgynvx1txMvDG8gGWoWkO1augYDxkp9JElWFgjQU=
 github.com/aws/smithy-go v1.27.1 h1:4T340VFndXtADGF52gYa1POyL7s9E4Z1OeZ1hCscIw8=
 github.com/aws/smithy-go v1.27.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
@@ -103,8 +103,8 @@ github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6N
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/dgraph-io/badger/v4 v4.9.1 h1:DocZXZkg5JJHJPtUErA0ibyHxOVUDVoXLSCV6t8NC8w=
-github.com/dgraph-io/badger/v4 v4.9.1/go.mod h1:5/MEx97uzdPUHR4KtkNt8asfI2T4JiEiQlV7kWUo8c0=
+github.com/dgraph-io/badger/v4 v4.9.2 h1:Wb5qw8gElqwV1a8msHTeQKova9b1V10heFKMIiPd80E=
+github.com/dgraph-io/badger/v4 v4.9.2/go.mod h1:nJjaJTUOSsQEBhsq209FmwCvMJzEA3e74RjZw6V2pQI=
 github.com/dgraph-io/ristretto/v2 v2.4.0 h1:I/w09yLjhdcVD2QV192UJcq8dPBaAJb9pOuMyNy0XlU=
 github.com/dgraph-io/ristretto/v2 v2.4.0/go.mod h1:0KsrXtXvnv0EqnzyowllbVJB8yBonswa2lTCK2gGo9E=
 github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da h1:aIftn67I1fkbMa512G+w+Pxci9hJPB8oMnkcP3iZF38=
@@ -288,10 +288,10 @@ github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELU
 github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
 github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
 github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
-github.com/mulgadc/predastore v1.7.0 h1:TuCU1oxs5EUJCqL3MybpyH1JR3WyxLHIj4Fa/sM9a5k=
-github.com/mulgadc/predastore v1.7.0/go.mod h1:0334T4L4Dby54TCuugpumyOhtvHQX+WSxQ/REDNSk9o=
-github.com/mulgadc/viperblock v1.7.0 h1:+SAWg/Dcyb03PC+5ZB5Ab4HWum9PFWJ5KfeVkcivowY=
-github.com/mulgadc/viperblock v1.7.0/go.mod h1:F/LOVydipnyjX9QMkAuf/gjdWeYjaniGHU4emwJwq+U=
+github.com/mulgadc/predastore v1.8.0 h1:Di/PlAP8nIVjXMmjVPuS9UtqQkqMjR41RrC5J21vHmw=
+github.com/mulgadc/predastore v1.8.0/go.mod h1:TC6I4Z27RCe2gPqlxqUehJ2J+/jKggnH+4DyMq1H/ng=
+github.com/mulgadc/viperblock v1.8.0 h1:kVyM5RxomokVa1uA2kbhtS02kZ9aDXKVj4MvGeqsKiM=
+github.com/mulgadc/viperblock v1.8.0/go.mod h1:18zxN4szsp5h6w8nTqkZWP0SWmIrLU3NdpKXDYNanls=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
@@ -428,14 +428,14 @@ golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnf
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
-golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
-golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
+golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
+golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa h1:FRnLl4eNAQl8hwxVVC17teOw8kdjVDVAiFMtgUdTSRQ=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa/go.mod h1:zk2irFbV9DP96SEBUUAy67IdHUaZuSnrz1n472HUCLE=
 golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
 golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
-golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM=
-golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU=
+golang.org/x/mod v0.36.0 h1:JJjpVx6myfUsUdAzZuOSTTmRE0PfZeNWzzvKrP7amb4=
+golang.org/x/mod v0.36.0/go.mod h1:moc6ELqsWcOw5Ef3xVprK5ul/MvtVvkIXLziUOICjUQ=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -446,8 +446,8 @@ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81R
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
-golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
-golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
+golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
+golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -456,8 +456,8 @@ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
-golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
+golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
+golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -488,33 +488,33 @@ golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
-golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
-golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
-golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa h1:efT73AJZfAAUV7SOip6pWGkwJDzIGiKBZGVzHYa+ve4=
-golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa/go.mod h1:kHjTxDEnAu6/Nl9lDkzjWpR+bmKfxeiRuSDlsMb70gE=
+golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
+golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6 h1:HjU6IWBiAgRIdAJ9/y1rwCn+UELEmwV+VsTLzj/W4sE=
+golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6/go.mod h1:Eqhaxk/wZsWEH8CRxLwj6xzEJbz7k1EFGqx7nyCoabE=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
-golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4=
-golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=
+golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc=
+golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
-golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
-golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
+golang.org/x/text v0.38.0 h1:sXmwo9DwP3OK9EZ7PqAdaooSGozfl/3a6/xJcbzPRhE=
+golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4=
 golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
 golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
 golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
-golang.org/x/tools v0.44.0 h1:UP4ajHPIcuMjT1GqzDWRlalUEoY+uzoZKnhOjbIPD2c=
-golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI=
+golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8=
+golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0=
 golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM=
 golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY=
 golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM=
@@ -538,8 +538,8 @@ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
-gopkg.in/ini.v1 v1.67.2 h1:JtOSMb9OuaCZKr7h5D/h6iii14sK0hLbplTc6frx4Ss=
-gopkg.in/ini.v1 v1.67.2/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
+gopkg.in/ini.v1 v1.67.3 h1:iM9Lhz5MRSGhHVGGwCuzG9KO8PoirCXj/m/qTmOJJQw=
+gopkg.in/ini.v1 v1.67.3/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

internal/eksgw/client.go

@@ -1,8 +1,6 @@
-// Package eksgw is the on-VM SigV4 client EKS control-plane helpers use to
-// reach the AWS gateway, instead of dialing core NATS. Both eks-gateway-publish
-// (bootstrap + state relay) and eks-token-webhook (TokenReview relay) sign with
-// the system (Predastore) credentials for service "eks" and POST over HTTPS;
-// the gateway authorizes via IAM and relays to the host-side NATS/STS/KV.
+// Package eksgw is the on-VM SigV4 client used by EKS helpers (eks-gateway-publish,
+// eks-token-webhook) to POST over HTTPS to the AWS gateway instead of NATS.
+// Signs with Predastore credentials for service "eks".
 package eksgw
 
 import (
@@ -73,16 +71,27 @@ func New(baseURL, caPath, accessKey, secretKey, region string) (*Client, error)
 	}, nil
 }
 
-// Post SigV4-signs and sends a single POST of body to path (e.g.
-// "/clusters/alpha/token-review"). It returns the response body on a 2xx, or an
-// error (including the gateway status + body) otherwise. No retry — callers that
-// need it wrap Post in their own loop.
+// Post SigV4-signs and sends body to path, returning the response body on 2xx.
+// Errors include the gateway status and body. No retry; callers wrap as needed.
 func (c *Client) Post(path string, body []byte) ([]byte, error) {
-	req, err := http.NewRequest(http.MethodPost, c.baseURL+path, bytes.NewReader(body))
+	return c.do(http.MethodPost, path, body)
+}
+
+// Get SigV4-signs and sends a single GET to path (e.g.
+// "/clusters/alpha/internal-addons?accountId=..."). Same contract as Post: 2xx
+// body or an error carrying the gateway status. No retry.
+func (c *Client) Get(path string) ([]byte, error) {
+	return c.do(http.MethodGet, path, nil)
+}
+
+func (c *Client) do(method, path string, body []byte) ([]byte, error) {
+	req, err := http.NewRequest(method, c.baseURL+path, bytes.NewReader(body))
 	if err != nil {
 		return nil, fmt.Errorf("create request: %w", err)
 	}
-	req.Header.Set("Content-Type", "application/x-amz-json-1.1")
+	if body != nil {
+		req.Header.Set("Content-Type", "application/x-amz-json-1.1")
+	}
 
 	sum := sha256.Sum256(body)
 	if err := auth.SignReq(req, c.accessKey, c.secretKey, hex.EncodeToString(sum[:]), "eks", c.region); err != nil {

internal/fipsboot/fipsboot.go

@@ -1,12 +1,6 @@
-// Package fipsboot enforces FIPS 140-3 mode at process startup.
-//
-// Blank-import this package from every binary's main:
-//
-//	import _ "github.com/mulgadc/spinifex/internal/fipsboot"
-//
-// Builds with GOFIPS140=v1.0.0 link the Go Cryptographic Module v1.0.0
-// (NIST CMVP cert #5247). At runtime FIPS mode can still be turned off
-// via GODEBUG=fips140=off; init() panics in that case to close the hole.
+// Package fipsboot enforces FIPS 140-3 at startup via blank import in every binary.
+// Requires GOFIPS140=v1.0.0 at build time; panics if GODEBUG=fips140=off disables
+// the runtime FIPS mode.
 package fipsboot
 
 import (

internal/tlsconfig/curves.go

@@ -1,34 +1,13 @@
 // Package tlsconfig pins the cluster-wide TLS 1.3 curve preference list.
-//
-// The list is hardcoded because mixed classification on a single cluster is
-// not supported and the certified FIPS build is itself the per-deployment
-// choice. A typo in a config string would silently downgrade where a missing
-// constant cannot.
+// Hardcoded to prevent silent downgrade via misconfiguration; the FIPS build
+// is the per-deployment choice.
 package tlsconfig
 
 import "crypto/tls"
 
-// Curves is the fixed TLS 1.3 curve allowlist:
-//
-//   - X25519MLKEM768     — Cat 3 ML-KEM hybrid, wins under Go 1.26 stdlib filter
-//   - SecP384r1MLKEM1024 — Cat 5 ML-KEM hybrid, fallback for peers that prefer it
-//   - X25519             — classical TLS 1.3 fallback for SDKs without ML-KEM (aws CLI v2 today)
-//   - CurveP384          — classical TLS 1.3 fallback (Java/.NET enterprise SDKs)
-//
-// PQ-capable peers (browsers since 2024, Go aws-sdk v2 with Go 1.24+) always
-// negotiate the hybrid first; classical fallback only triggers for peers that
-// cannot do ML-KEM. Those connections still get TLS 1.3 + AES-256-GCM but
-// remain HNDL-exposed — accepted trade-off until AWS CLI v2 / bundled
-// OpenSSL ships ML-KEM (currently lagging upstream OpenSSL 3.5).
-//
-// Weak/niche entries from the Go stdlib default (SecP256r1MLKEM768, P-256,
-// P-521) are intentionally excluded so they cannot be advertised.
-//
-// Go 1.26 stdlib treats tls.Config.CurvePreferences as an allowlist filter
-// applied to the stdlib default order, not as a user-supplied order.
-//
-// Callers must not mutate this slice. Assign it directly to
-// tls.Config.CurvePreferences; do not append to it.
+// Curves is the fixed TLS 1.3 curve allowlist: two ML-KEM hybrids (PQ-capable
+// peers always negotiate these) followed by two classical fallbacks for SDKs
+// without ML-KEM. Weak Go defaults (SecP256r1MLKEM768, P-256, P-521) excluded.
 var Curves = []tls.CurveID{
 	tls.X25519MLKEM768,
 	tls.SecP384r1MLKEM1024,

internal/tlsconfig/curves_test.go

@@ -46,11 +46,9 @@ func TestCurves_ExcludesWeakPrimitives(t *testing.T) {
 	}
 }
 
-// TestIntegration_NegotiatesPQHybrid spins up an httptest TLS server
-// configured with Curves + TLS 1.3 minimum, dials it, and asserts a PQ
-// hybrid is negotiated at TLS 1.3. Per Go 1.26 stdlib filter behavior the
-// winner is X25519MLKEM768; the assertion checks for any approved hybrid
-// to remain stable across stdlib reorderings of our 4 entries.
+// TestIntegration_NegotiatesPQHybrid verifies that a Curves-configured server
+// negotiates a PQ hybrid at TLS 1.3; checks for any approved hybrid to stay
+// stable across Go stdlib reorderings.
 func TestIntegration_NegotiatesPQHybrid(t *testing.T) {
 	cert := selfSignedCert(t)
 	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
@@ -95,10 +93,8 @@ func TestIntegration_NegotiatesPQHybrid(t *testing.T) {
 	}
 }
 
-// TestIntegration_TLS12ClientRejected confirms MinVersion: tls.VersionTLS13
-// is enforced — a TLS 1.2-pinned client gets a handshake error rather than
-// silently downgrading. This is the negative test that closes the HNDL gap
-// on external surfaces.
+// TestIntegration_TLS12ClientRejected confirms MinVersion TLS 1.3 is enforced:
+// a TLS 1.2-pinned client must get a handshake error, not a silent downgrade.
 func TestIntegration_TLS12ClientRejected(t *testing.T) {
 	cert := selfSignedCert(t)
 	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {

scripts/build-microvm-image.sh

@@ -81,30 +81,46 @@ EOF
         --initdb \
         $ALPINE_PACKAGES
 
-# Strategy 2/3: container runtime (dev machines)
-else
-    if command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1; then
-        CONTAINER_TOOL=docker
-    elif command -v podman >/dev/null 2>&1; then
-        CONTAINER_TOOL=podman
-    else
-        echo "ERROR: no rootfs build tool available." >&2
-        echo "       Install one of: apk (Alpine), docker, podman" >&2
-        exit 1
-    fi
+# Strategy 2: docker -> BuildKit local export.
+# Do NOT use `docker export` here. On Docker's overlay2 driver it leaks the
+# container's overlay mount (the mount survives `docker rm`), orphaning layers
+# under /var/lib/docker that only a daemon restart reclaims; on a busy CI host
+# this grows without bound. `buildx --output type=local` writes the image
+# filesystem straight to a directory and releases every mount, so nothing leaks.
+elif command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1; then
+    echo "[build-microvm-image] rootfs: docker buildx --output type=local (alpine:${ALPINE_VERSION})"
+    BUILD_CTX=$(mktemp -d)
+    cat > "$BUILD_CTX/Dockerfile" <<DOCKERFILE
+FROM alpine:${ALPINE_VERSION}
+RUN apk add --no-cache ${ALPINE_PACKAGES}
+DOCKERFILE
+    docker buildx build \
+        --platform linux/amd64 \
+        --output "type=local,dest=${CHROOT_DIR}" \
+        "$BUILD_CTX"
+    rm -rf "$BUILD_CTX"
 
-    echo "[build-microvm-image] rootfs: $CONTAINER_TOOL export (alpine:${ALPINE_VERSION})"
-    CONTAINER_CID=$($CONTAINER_TOOL run -d \
+# Strategy 3: podman -> run + export. Podman's store does not exhibit the docker
+# overlay2 export-mount leak, and podman has no buildx.
+elif command -v podman >/dev/null 2>&1; then
+    echo "[build-microvm-image] rootfs: podman export (alpine:${ALPINE_VERSION})"
+    CONTAINER_TOOL=podman
+    CONTAINER_CID=$(podman run -d \
         "alpine:${ALPINE_VERSION}" \
         sh -c "apk add --no-cache ${ALPINE_PACKAGES}")
 
-    exit_code=$($CONTAINER_TOOL wait "$CONTAINER_CID")
+    exit_code=$(podman wait "$CONTAINER_CID")
     if [ "$exit_code" != "0" ]; then
         echo "ERROR: package installation failed in container (exit $exit_code)" >&2
         exit 1
     fi
 
-    $CONTAINER_TOOL export "$CONTAINER_CID" | tar -x -C "$CHROOT_DIR"
+    podman export "$CONTAINER_CID" | tar -x -C "$CHROOT_DIR"
+
+else
+    echo "ERROR: no rootfs build tool available." >&2
+    echo "       Install one of: apk (Alpine), docker, podman" >&2
+    exit 1
 fi
 
 # --- /dev: empty mountpoint only ---

scripts/build-system-image.sh

@@ -26,10 +26,11 @@ PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
 export LIBGUESTFS_BACKEND="${LIBGUESTFS_BACKEND:-direct}"
 
 usage() {
-    echo "Usage: $0 <manifest.conf> [--import]"
+    echo "Usage: $0 <manifest.conf> [--import] [--no-cache]"
     echo ""
     echo "  manifest.conf  Path to image manifest (see scripts/images/ for examples)"
     echo "  --import       Import the built image as an AMI"
+    echo "  --no-cache     Always rebuild; ignore a recently-built cached raw"
     exit 1
 }
 
@@ -47,10 +48,12 @@ fi
 
 DO_IMPORT=false
 QUIET=false
+NO_CACHE=false
 for arg in "$@"; do
     case "$arg" in
-        --import) DO_IMPORT=true ;;
-        --quiet)  QUIET=true ;;
+        --import)   DO_IMPORT=true ;;
+        --quiet)    QUIET=true ;;
+        --no-cache) NO_CACHE=true ;;
     esac
 done
 
@@ -184,7 +187,9 @@ echo "Lock acquired"
 
 # If the raw image was built recently (< 10 min), skip the entire build.
 # This avoids duplicate work when concurrent CI jobs build the same image.
-if [[ -f "$OUTPUT_RAW" ]] && [[ $(( $(date +%s) - $(stat -c %Y "$OUTPUT_RAW") )) -lt 600 ]]; then
+# --no-cache (e.g. the publish path) forces a fresh build so a stale raw from
+# a prior/concurrent run on a persistent runner is never republished.
+if [[ "$NO_CACHE" == false ]] && [[ -f "$OUTPUT_RAW" ]] && [[ $(( $(date +%s) - $(stat -c %Y "$OUTPUT_RAW") )) -lt 600 ]]; then
     echo "=== Skipping build — $OUTPUT_RAW is fresh (< 10 min old) ==="
 
     # Restore stdout if suppressed, then jump to import

scripts/clone-deps.sh

@@ -75,5 +75,5 @@ echo ""
 echo "🎉 Development environment setup complete!"
 echo ""
 echo "Next step:"
-echo "Run development setup to build services: ./scripts/dev-setup.sh"
+echo "Run development setup to build, install, and start services: ./scripts/dev-install.sh"
 echo ""

scripts/dev-install.sh

@@ -44,9 +44,9 @@ sudo systemctl stop spinifex.target 2>/dev/null || true
 sudo systemctl reset-failed 'spinifex-*' 2>/dev/null || true
 
 # Remove stale files owned by the dev user in production paths.
-# Previous dev-mode installs (admin init without sudo, start-dev.sh) leave
-# files owned by tf-user that service users (spinifex-nats, etc.) can't read
-# under systemd's ProtectSystem=strict sandboxing.
+# A prior admin init run without sudo can leave files owned by tf-user that
+# service users (spinifex-nats, etc.) can't read under systemd's
+# ProtectSystem=strict sandboxing.
 for dir in /var/lib/spinifex /var/log/spinifex /etc/spinifex; do
     if [ -d "$dir" ]; then
         # Remove PID files, stale logs, and the legacy ~/spinifex/config symlink

scripts/dev-setup.sh

@@ -1,180 +0,0 @@
-#!/bin/bash
-
-# Spinifex Platform Development Environment Setup
-# This script sets up a complete development environment for the Spinifex platform
-
-set -euo pipefail
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
-MULGA_ROOT="$(cd "$PROJECT_ROOT/.." && pwd)"
-
-# Default configuration
-NATS_PORT=4222
-PREDASTORE_PORT=8443
-SPINIFEX_GATEWAY_PORT=9999
-
-echo "🚀 Setting up Spinifex development environment..."
-echo "Project root: $PROJECT_ROOT"
-
-# Function to check if command exists
-command_exists() {
-    command -v "$1" >/dev/null 2>&1
-}
-
-# Function to check if port is available
-port_available() {
-    ! nc -z localhost "$1" 2>/dev/null
-}
-
-# Check dependencies
-echo "🔍 Checking dependencies..."
-
-# Check for required commands
-required_commands=("go" "make")
-missing_required=false
-for cmd in "${required_commands[@]}"; do
-    if command_exists "$cmd"; then
-        echo "✅ $cmd found"
-    else
-        echo "❌ $cmd not found. Please install $cmd"
-        missing_required=true
-        case "$cmd" in
-            "go")
-                echo "   Install Go: https://golang.org/dl/"
-                ;;
-        esac
-    fi
-done
-if [ "$missing_required" = true ]; then
-    echo "❌ Missing required dependencies. Cannot continue."
-    exit 1
-fi
-
-# Check optional commands
-optional_commands=("air" "nbdkit")
-for cmd in "${optional_commands[@]}"; do
-    if command_exists "$cmd"; then
-        echo "✅ $cmd found (optional)"
-    else
-        echo "⚠️  $cmd not found (optional)"
-        case "$cmd" in
-            "air")
-                echo "   Install air for hot reloading: go install github.com/air-verse/air@latest"
-                ;;
-            "nbdkit")
-                echo "   Install nbdkit: sudo apt-get install nbdkit (Ubuntu/Debian)"
-                ;;
-        esac
-    fi
-done
-
-# Check OVN/OVS commands (required for VPC networking)
-echo ""
-echo "🔍 Checking OVN/OVS dependencies..."
-ovn_commands=("ovs-vsctl" "ovn-controller" "ovn-nbctl")
-ovn_ready=true
-for cmd in "${ovn_commands[@]}"; do
-    if command_exists "$cmd"; then
-        echo "✅ $cmd found"
-    else
-        echo "⚠️  $cmd not found (required for VPC networking)"
-        ovn_ready=false
-    fi
-done
-if [ "$ovn_ready" = false ]; then
-    echo "   Install OVN: sudo apt-get install ovn-central ovn-host openvswitch-switch"
-    echo "   Or run: make quickinstall"
-fi
-
-# Initialize br-int if OVS is available
-if command_exists "ovs-vsctl"; then
-    echo ""
-    echo "🌐 Checking OVS br-int bridge..."
-    if ovs-vsctl br-exists br-int 2>/dev/null; then
-        echo "✅ br-int already exists"
-    else
-        echo "⚠️  br-int not found. Run ./scripts/setup-ovn.sh to initialize OVN networking"
-    fi
-fi
-
-# Check component repositories
-echo ""
-echo "📦 Checking component repositories..."
-
-components=("viperblock" "predastore")
-for component in "${components[@]}"; do
-    component_path="$MULGA_ROOT/$component"
-    if [[ -d "$component_path" ]]; then
-        echo "✅ $component found at $component_path"
-        if [[ -f "$component_path/go.mod" ]]; then
-            echo "   Go module verified"
-        else
-            echo "   ⚠️  No go.mod found in $component"
-        fi
-    else
-        echo "❌ $component not found at $component_path"
-        echo "   Run: ./scripts/clone-deps.sh"
-    fi
-done
-
-# Check port availability
-echo ""
-echo "🔌 Checking port availability..."
-
-ports=("$NATS_PORT:NATS" "$PREDASTORE_PORT:Predastore" "$SPINIFEX_GATEWAY_PORT:Spinifex Gateway")
-for port_info in "${ports[@]}"; do
-    port="${port_info%:*}"
-    service="${port_info#*:}"
-    if port_available "$port"; then
-        echo "✅ Port $port available for $service"
-    else
-        echo "⚠️  Port $port already in use (needed for $service)"
-    fi
-done
-
-# Build Spinifex first (needed for admin init)
-echo ""
-echo "🔨 Building Spinifex..."
-cd "$PROJECT_ROOT"
-make build
-echo "✅ Spinifex built successfully"
-
-# Build components
-echo ""
-echo "🔨 Building components..."
-
-# Build Viperblock
-if [[ -d "$MULGA_ROOT/viperblock" ]]; then
-    echo "📦 Building Viperblock..."
-    cd "$MULGA_ROOT/viperblock"
-    make build
-    echo "✅ Viperblock built successfully"
-else
-    echo "⚠️  Viperblock directory not found, skipping build"
-fi
-
-# Build Predastore
-if [[ -d "$MULGA_ROOT/predastore" ]]; then
-    echo "📦 Building Predastore..."
-    cd "$MULGA_ROOT/predastore"
-    make build
-    echo "✅ Predastore built successfully"
-else
-    echo "⚠️  Predastore directory not found, skipping build"
-fi
-
-echo ""
-echo "Development environment setup complete!"
-echo ""
-echo "Recommended: Use scripts/dev-install.sh for a production-parity setup"
-echo "  (systemd, privilege separation, production paths)."
-echo ""
-echo "  scripts/dev-install.sh  # One-time: build, install, init, start via systemd"
-echo "  make deploy             # Daily: rebuild + copy binary + restart all"
-echo "  make reinstall          # Changed a .service file? Re-run full setup"
-echo ""
-echo "Legacy workflow (deprecated, retained for pseudo-multinode CI only):"
-echo "  ./scripts/setup-ovn.sh --management"
-echo "  sudo ./bin/spx admin init --region ap-southeast-2 --az ap-southeast-2a --node node1 --nodes 1"
-echo "  ./scripts/start-dev.sh"

scripts/iac/README.md

@@ -1,9 +0,0 @@
-# IAC
-
-Collection of OpenTofu/Terraform templates for creating Spinifex instances.
-
-| Template | Description |
-|---|---|
-| [proxmox/](proxmox/) | Provisions a 3-node Spinifex cluster on Proxmox VE bare-metal hypervisors |
-| [aws/](aws/) | Provisions infrastructure using the Terraform AWS provider, with MulgaOS AWS-compatible API as the local endpoint |
-| [proxmox-template/](proxmox-template/) | Builds a Proxmox VM template with all Spinifex dependencies pre-installed (QEMU, Go, repos, modules) |

scripts/iac/aws/README.md

@@ -1,206 +0,0 @@
-# MulgaOS AWS Infrastructure as Code
-
-This directory contains OpenTofu/Terraform configuration files for deploying infrastructure on MulgaOS using AWS-compatible APIs.
-
-## Prerequisites
-
-1. **OpenTofu or Terraform installed**
-   ```bash
-   # Install OpenTofu (recommended)
-   # See: https://opentofu.org/docs/intro/install/
-
-   # Or use Terraform
-   # See: https://developer.hashicorp.com/terraform/install
-   ```
-
-2. **MulgaOS services running**
-   - `spinifex daemon` - Core daemon with NATS subscriptions
-   - `awsd` - AWS API gateway (default: http://localhost:8080)
-   - `predastore` - S3-compatible storage
-   - `viperblock` - EBS-compatible block storage
-
-3. **AWS credentials configured**
-   ```bash
-   # Option 1: Environment variables
-   export AWS_ACCESS_KEY_ID=your_access_key
-   export AWS_SECRET_ACCESS_KEY=your_secret_key
-
-   # Option 2: AWS CLI profile
-   aws configure --profile spinifex
-   export AWS_PROFILE=spinifex
-   ```
-
-4. **AMI available**
-   Before running, ensure you have a Debian 13 AMI imported:
-   ```bash
-   # Import an AMI using the spinifex CLI or aws CLI
-   aws --endpoint-url http://localhost:8080 ec2 describe-images
-   ```
-
-## Configuration
-
-### Environment Variables
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `TF_VAR_mulgaos_endpoint` | `http://localhost:8080` | MulgaOS API endpoint |
-| `TF_VAR_region` | `ap-southeast-2` | AWS region (for compatibility) |
-| `AWS_ACCESS_KEY_ID` | - | MulgaOS access key |
-| `AWS_SECRET_ACCESS_KEY` | - | MulgaOS secret key |
-
-### terraform.tfvars (optional)
-
-Create a `terraform.tfvars` file for custom settings:
-
-```hcl
-mulgaos_endpoint = "http://192.168.1.100:8080"
-region           = "us-west-2"
-```
-
-## Usage
-
-### Initialize
-
-```bash
-cd iac/aws
-tofu init
-# or: terraform init
-```
-
-### Validate
-
-```bash
-tofu validate
-# or: terraform validate
-```
-
-### Plan
-
-```bash
-tofu plan
-# or: terraform plan
-```
-
-### Apply
-
-```bash
-tofu apply
-# or: terraform apply
-```
-
-### Destroy
-
-```bash
-tofu destroy
-# or: terraform destroy
-```
-
-## What Gets Created
-
-The `main.tf` creates:
-
-1. **VPC** (`10.1.1.0/16`) - Virtual Private Cloud
-2. **Internet Gateway** - For public internet access
-3. **Public Subnet** (`10.1.2.0/24`) - With auto-assign public IP
-4. **Route Table** - Routes `0.0.0.0/0` through IGW
-5. **Security Group** - Allows SSH (22), HTTP (80), HTTPS (443) inbound
-6. **Placement Group** - Spread strategy for HA
-7. **3 EC2 Instances** - t3.small Debian 13 VMs
-8. **SSH Key Pair** - ED25519 key saved to `spinifex-test.pem`
-
-## SSH Access
-
-After `tofu apply`, SSH into instances:
-
-```bash
-# Using the generated key
-ssh -i spinifex-test.pem admin@<public_ip>
-
-# For local development with QEMU port forwarding
-# Find the forwarded port from QEMU args (-netdev user,id=net0,hostfwd=tcp:127.0.0.1:PORT-:22)
-ssh -p PORT -i spinifex-test.pem admin@127.0.0.1
-```
-
-## Troubleshooting
-
-### "No valid credential sources found"
-
-Ensure AWS credentials are set:
-```bash
-export AWS_ACCESS_KEY_ID=your_key
-export AWS_SECRET_ACCESS_KEY=your_secret
-```
-
-### "Error: No AMI found"
-
-Import a Debian 13 AMI first, or modify `main.tf` to use a different AMI filter.
-
-### Connection refused
-
-Ensure MulgaOS services are running:
-```bash
-# Check if awsd gateway is running
-curl http://localhost:8080/health
-
-# Check daemon status
-spinifex daemon status
-```
-
-### "InvalidAction" errors
-
-Some AWS operations may not be implemented yet. Check `TODO.md` for the current implementation status.
-
-## Customization
-
-### Using a different AMI
-
-Modify the `data "aws_ami"` block in `main.tf`:
-
-```hcl
-data "aws_ami" "custom" {
-  most_recent = true
-  owners      = ["self"]  # Your imported AMIs
-
-  filter {
-    name   = "name"
-    values = ["my-custom-ami-*"]
-  }
-}
-```
-
-### Changing instance count
-
-Modify the `count` parameter:
-
-```hcl
-resource "aws_instance" "vm" {
-  count = 5  # Change from 3 to 5
-  # ...
-}
-```
-
-### Different instance type
-
-```hcl
-resource "aws_instance" "vm" {
-  instance_type = "t3.medium"  # Change from t3.small
-  # ...
-}
-```
-
-## Files
-
-| File | Description |
-|------|-------------|
-| `main.tf` | Main Terraform/OpenTofu configuration |
-| `spinifex-test.pem` | Generated SSH private key (after apply) |
-| `.terraform/` | Provider plugins (after init) |
-| `terraform.tfstate` | State file (after apply) |
-| `terraform.tfvars` | Custom variables (optional) |
-
-## Related Documentation
-
-- [MulgaOS DEV.md](../../DEV.md) - Development guide
-- [MulgaOS TODO.md](../../TODO.md) - Implementation status
-- [OpenTofu Documentation](https://opentofu.org/docs/)
-- [AWS Provider Documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs)

scripts/iac/aws/basic/main.tf

@@ -1,307 +0,0 @@
-# Spinifex IaC Compatibility Test — "basic" suite
-#
-# Exercises every AWS resource Spinifex currently supports via tofu apply,
-# WITHOUT route tables, NACLs, or other unimplemented networking.
-#
-# Usage:
-#   cd spinifex/scripts/iac/aws/basic
-#   tofu init
-#   tofu plan
-#   tofu apply        # full round-trip against a running Spinifex cluster
-#   tofu destroy      # clean teardown
-
-terraform {
-  required_version = ">= 1.6.0"
-
-  required_providers {
-    aws = {
-      source  = "hashicorp/aws"
-      version = ">= 5.0"
-    }
-    tls = {
-      source  = "hashicorp/tls"
-      version = ">= 4.0"
-    }
-    local = {
-      source  = "hashicorp/local"
-      version = ">= 2.0"
-    }
-  }
-}
-
-# ---------------------------------------------------------------------------
-# Variables
-# ---------------------------------------------------------------------------
-
-variable "region" {
-  type    = string
-  default = "ap-southeast-2"
-}
-
-variable "mulgaos_endpoint" {
-  type        = string
-  default     = "https://localhost:9999"
-  description = "Spinifex AWS gateway endpoint"
-}
-
-# ---------------------------------------------------------------------------
-# Provider
-# ---------------------------------------------------------------------------
-
-provider "aws" {
-  region = var.region
-
-  endpoints {
-    ec2 = var.mulgaos_endpoint
-    s3  = var.mulgaos_endpoint
-    iam = var.mulgaos_endpoint
-    sts = var.mulgaos_endpoint
-  }
-
-  skip_credentials_validation = true
-  skip_metadata_api_check     = true
-  skip_requesting_account_id  = true
-  skip_region_validation      = true
-}
-
-# ---------------------------------------------------------------------------
-# Data sources
-# ---------------------------------------------------------------------------
-
-data "aws_availability_zones" "available" {
-  state = "available"
-}
-
-data "aws_ami" "debian13" {
-  most_recent = true
-  owners      = ["000000000000"]
-
-  filter {
-    name   = "name"
-    values = ["*debian-13*"]
-  }
-
-  filter {
-    name   = "virtualization-type"
-    values = ["hvm"]
-  }
-
-  filter {
-    name   = "root-device-type"
-    values = ["ebs"]
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 1. VPC + Subnet (no route table needed)
-# ---------------------------------------------------------------------------
-
-resource "aws_vpc" "test" {
-  cidr_block           = "10.99.0.0/16"
-  enable_dns_hostnames = true
-  enable_dns_support   = true
-
-  tags = {
-    Name = "spx-basic-test-vpc"
-  }
-}
-
-resource "aws_subnet" "test" {
-  vpc_id                  = aws_vpc.test.id
-  cidr_block              = "10.99.1.0/24"
-  availability_zone       = data.aws_availability_zones.available.names[0]
-  map_public_ip_on_launch = true
-
-  tags = {
-    Name = "spx-basic-test-subnet"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 2. Internet Gateway (attach to VPC, but no route table referencing it)
-# ---------------------------------------------------------------------------
-
-resource "aws_internet_gateway" "test" {
-  vpc_id = aws_vpc.test.id
-
-  tags = {
-    Name = "spx-basic-test-igw"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 3. Security Group — inbound SSH + HTTP, all outbound
-# ---------------------------------------------------------------------------
-
-resource "aws_security_group" "test" {
-  name        = "spx-basic-test-sg"
-  description = "Basic test: SSH + HTTP inbound, all outbound"
-  vpc_id      = aws_vpc.test.id
-
-  ingress {
-    description = "SSH"
-    from_port   = 22
-    to_port     = 22
-    protocol    = "tcp"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  ingress {
-    description = "HTTP"
-    from_port   = 80
-    to_port     = 80
-    protocol    = "tcp"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  egress {
-    description = "All outbound"
-    from_port   = 0
-    to_port     = 0
-    protocol    = "-1"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  tags = {
-    Name = "spx-basic-test-sg"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 4. Key Pair
-# ---------------------------------------------------------------------------
-
-resource "tls_private_key" "test" {
-  algorithm = "ED25519"
-}
-
-resource "aws_key_pair" "test" {
-  key_name   = "spx-basic-test-key"
-  public_key = tls_private_key.test.public_key_openssh
-}
-
-resource "local_file" "test_pem" {
-  filename        = "${path.module}/spx-basic-test.pem"
-  content         = tls_private_key.test.private_key_openssh
-  file_permission = "0600"
-}
-
-# ---------------------------------------------------------------------------
-# 5. Placement Group
-# ---------------------------------------------------------------------------
-
-resource "aws_placement_group" "test" {
-  name     = "spx-basic-test-spread"
-  strategy = "spread"
-}
-
-# ---------------------------------------------------------------------------
-# 6. EC2 Instance
-# ---------------------------------------------------------------------------
-
-resource "aws_instance" "test" {
-  ami           = data.aws_ami.debian13.id
-  instance_type = "t3.small"
-
-  subnet_id              = aws_subnet.test.id
-  vpc_security_group_ids = [aws_security_group.test.id]
-  key_name               = aws_key_pair.test.key_name
-  placement_group        = aws_placement_group.test.name
-
-  associate_public_ip_address = true
-
-  tags = {
-    Name        = "spx-basic-test-vm"
-    Environment = "test"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 7. EBS Volume + Attachment
-# ---------------------------------------------------------------------------
-
-resource "aws_ebs_volume" "data" {
-  availability_zone = data.aws_availability_zones.available.names[0]
-  size              = 10
-  type              = "gp3"
-
-  tags = {
-    Name = "spx-basic-test-data-vol"
-  }
-}
-
-
-# ---------------------------------------------------------------------------
-# 8. EBS Snapshot (of the standalone volume)
-# ---------------------------------------------------------------------------
-
-resource "aws_ebs_snapshot" "data" {
-  volume_id = aws_ebs_volume.data.id
-
-  tags = {
-    Name = "spx-basic-test-snapshot"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 9. Elastic IP + Association
-#    Requires external IPAM (pool mode) — uncomment when available.
-# ---------------------------------------------------------------------------
-
-# resource "aws_eip" "test" {
-#   domain = "vpc"
-#
-#   tags = {
-#     Name = "spx-basic-test-eip"
-#   }
-# }
-#
-# resource "aws_eip_association" "test" {
-#   allocation_id = aws_eip.test.id
-#   instance_id   = aws_instance.test.id
-# }
-
-# ---------------------------------------------------------------------------
-# 10. Standalone Network Interface (not attached — just tests CRUD)
-# ---------------------------------------------------------------------------
-
-resource "aws_network_interface" "extra" {
-  subnet_id       = aws_subnet.test.id
-  security_groups = [aws_security_group.test.id]
-
-  tags = {
-    Name = "spx-basic-test-eni"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# Outputs
-# ---------------------------------------------------------------------------
-
-output "vpc_id" {
-  value = aws_vpc.test.id
-}
-
-output "instance_id" {
-  value = aws_instance.test.id
-}
-
-output "instance_public_ip" {
-  value = aws_instance.test.public_ip
-}
-
-# output "eip_public_ip" {
-#   value = aws_eip.test.public_ip
-# }
-
-output "volume_id" {
-  value = aws_ebs_volume.data.id
-}
-
-output "snapshot_id" {
-  value = aws_ebs_snapshot.data.id
-}
-
-output "ssh_command" {
-  value = "ssh -i spx-basic-test.pem admin@${aws_instance.test.public_ip}"
-}

scripts/iac/aws/main.tf

@@ -1,217 +0,0 @@
-terraform {
-  required_version = ">= 1.6.0"
-
-  required_providers {
-    aws = {
-      source  = "hashicorp/aws"
-      version = ">= 5.0"
-    }
-    tls = {
-      source  = "hashicorp/tls"
-      version = ">= 4.0"
-    }
-    local = {
-      source  = "hashicorp/local"
-      version = ">= 2.0"
-    }
-  }
-}
-
-variable "region" {
-  type    = string
-  default = "ap-southeast-2" # Sydney, change if you want
-}
-
-# MulgaOS AWS-compatible endpoints
-# Set these environment variables or use terraform.tfvars:
-#   export AWS_ACCESS_KEY_ID=your_access_key
-#   export AWS_SECRET_ACCESS_KEY=your_secret_key
-#   export TF_VAR_mulgaos_endpoint=https://localhost:9999
-variable "mulgaos_endpoint" {
-  type        = string
-  default     = "https://localhost:9999"
-  description = "MulgaOS API endpoint (awsd gateway)"
-}
-
-provider "aws" {
-  region = var.region
-
-  # MulgaOS custom endpoints
-  endpoints {
-    ec2 = var.mulgaos_endpoint
-    s3  = var.mulgaos_endpoint
-    iam = var.mulgaos_endpoint
-    sts = var.mulgaos_endpoint
-  }
-
-  # Skip AWS credential validation (MulgaOS handles auth)
-  skip_credentials_validation = true
-  skip_metadata_api_check     = true
-  skip_requesting_account_id  = true
-
-  # For local development, skip region validation
-  skip_region_validation = true
-}
-
-data "aws_availability_zones" "available" {
-  state = "available"
-}
-
-# Debian 13 AMI
-# For MulgaOS, need to import a local AMI first
-data "aws_ami" "debian13" {
-  most_recent = true
-  owners      = ["000000000000"] # Spinifex system images (on AWS use "136693071363" for Debian official)
-
-  filter {
-    name   = "name"
-    values = ["*debian-13*"]
-  }
-
-  filter {
-    name   = "virtualization-type"
-    values = ["hvm"]
-  }
-
-  filter {
-    name   = "root-device-type"
-    values = ["ebs"]
-  }
-}
-
-# Keypair: spinifex-test
-resource "tls_private_key" "hive_test" {
-  algorithm = "ED25519"
-}
-
-resource "aws_key_pair" "hive_test" {
-  key_name   = "spinifex-test"
-  public_key = tls_private_key.hive_test.public_key_openssh
-}
-
-# Save private key locally for SSH
-resource "local_file" "hive_test_pem" {
-  filename        = "${path.module}/spinifex-test.pem"
-  content         = tls_private_key.hive_test.private_key_openssh
-  file_permission = "0600"
-}
-
-# 1: VPC 10.1.0.0/16
-resource "aws_vpc" "main" {
-  cidr_block           = "10.1.0.0/16"
-  enable_dns_hostnames = true
-  enable_dns_support   = true
-
-  tags = {
-    Name = "spinifex-test-vpc"
-  }
-}
-
-# Internet access for a public subnet
-resource "aws_internet_gateway" "igw" {
-  vpc_id = aws_vpc.main.id
-
-  tags = {
-    Name = "spinifex-test-igw"
-  }
-}
-
-# 2: Subnet 10.1.2.0/24 (public)
-resource "aws_subnet" "public" {
-  vpc_id                  = aws_vpc.main.id
-  cidr_block              = "10.1.2.0/24"
-  availability_zone       = data.aws_availability_zones.available.names[0]
-  map_public_ip_on_launch = true
-
-  tags = {
-    Name = "spinifex-test-public-subnet"
-  }
-}
-
-resource "aws_route_table" "public" {
-  vpc_id = aws_vpc.main.id
-
-  route {
-    cidr_block = "0.0.0.0/0"
-    gateway_id = aws_internet_gateway.igw.id
-  }
-
-  tags = {
-    Name = "spinifex-test-public-rt"
-  }
-}
-
-resource "aws_route_table_association" "public" {
-  subnet_id      = aws_subnet.public.id
-  route_table_id = aws_route_table.public.id
-}
-
-# 3: Security group
-resource "aws_security_group" "web_ssh" {
-  name        = "spinifex-test-sg"
-  description = "Allow inbound 22,80,443; allow all outbound"
-  vpc_id      = aws_vpc.main.id
-
-  ingress {
-    description = "SSH"
-    from_port   = 22
-    to_port     = 22
-    protocol    = "tcp"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  ingress {
-    description = "HTTP"
-    from_port   = 80
-    to_port     = 80
-    protocol    = "tcp"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  ingress {
-    description = "HTTPS"
-    from_port   = 443
-    to_port     = 443
-    protocol    = "tcp"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  egress {
-    description = "All outbound"
-    from_port   = 0
-    to_port     = 0
-    protocol    = "-1"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  tags = {
-    Name = "spinifex-test-sg"
-  }
-}
-
-resource "aws_instance" "vm" {
-  count         = 3
-  ami           = data.aws_ami.debian13.id
-  instance_type = "t3.small"
-
-  subnet_id              = aws_subnet.public.id
-  vpc_security_group_ids = [aws_security_group.web_ssh.id]
-  key_name               = aws_key_pair.hive_test.key_name
-
-  associate_public_ip_address = true
-
-  tags = {
-    Name = "spinifex-test-vm-${count.index + 1}"
-  }
-}
-
-output "instance_public_ips" {
-  value = aws_instance.vm[*].public_ip
-}
-
-output "ssh_commands" {
-  value = [
-    for ip in aws_instance.vm[*].public_ip :
-    "ssh -i spinifex-test.pem admin@${ip}"
-  ]
-}

scripts/iac/configure-cluster.sh

@@ -1,204 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-# Configure a provisioned Spinifex cluster: clone repo, build, form cluster, start services.
-# Usage: configure-cluster.sh <cluster_name> <state_file>
-
-CLUSTER_NAME="${1:?Usage: configure-cluster.sh <cluster_name> <state_file>}"
-STATE_FILE="${2:?Usage: configure-cluster.sh <cluster_name> <state_file>}"
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-TOFU_DIR="$SCRIPT_DIR/proxmox"
-
-SSH_OPTS="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10"
-
-REGION="ap-southeast-2"
-AZ="ap-southeast-2a"
-
-inventory=$(tofu -chdir="$TOFU_DIR" output -json -state="$STATE_FILE" inventory)
-NODE_COUNT=$(echo "$inventory" | jq -r '.node_count')
-SSH_KEY=$(eval echo "$(echo "$inventory" | jq -r '.ssh_key_path')")
-SSH_USER=$(echo "$inventory" | jq -r '.ssh_user')
-
-# Build arrays of node IPs
-declare -a MGMT_IPS
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    MGMT_IPS+=("$(echo "$inventory" | jq -r ".nodes[$i].management")")
-done
-
-remote() {
-    local ip="$1"
-    shift
-    ssh -i "$SSH_KEY" $SSH_OPTS "$SSH_USER@$ip" "$@"
-}
-
-remote_bg() {
-    local ip="$1"
-    shift
-    ssh -i "$SSH_KEY" $SSH_OPTS "$SSH_USER@$ip" "$@" &
-}
-
-echo "==> Cluster '$CLUSTER_NAME': $NODE_COUNT nodes"
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    echo "    node$((i + 1)): ${MGMT_IPS[$i]}"
-done
-
-# --- Step 1: Wait for cloud-init ---
-echo ""
-echo "==> Waiting for cloud-init to complete on all nodes..."
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    ip="${MGMT_IPS[$i]}"
-    printf "    node%d (%s): " "$((i + 1))" "$ip"
-    attempts=0
-    while ! remote "$ip" "test -f /tmp/vendor-cloud-init-done" 2>/dev/null; do
-        attempts=$((attempts + 1))
-        if [ $attempts -ge 60 ]; then
-            echo "TIMEOUT"
-            echo "Error: cloud-init did not complete on $ip after 5 minutes"
-            exit 1
-        fi
-        sleep 5
-    done
-    echo "done"
-done
-
-# --- Step 2: Clone repo and build on all nodes ---
-echo ""
-echo "==> Cloning Spinifex repo and building on all nodes..."
-PIDS=()
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    ip="${MGMT_IPS[$i]}"
-    (
-        echo "    node$((i + 1)) ($ip): cloning..."
-        remote "$ip" "mkdir -p ~/Development/mulga && cd ~/Development/mulga && git clone https://github.com/mulgadc/spinifex.git && export PATH=\$PATH:/usr/local/go/bin && sudo make -C spinifex quickinstall"
-        echo "    node$((i + 1)) ($ip): clone-deps + dev-setup..."
-        remote "$ip" "export PATH=\$PATH:/usr/local/go/bin && cd ~/Development/mulga/spinifex && ./scripts/clone-deps.sh && ./scripts/dev-setup.sh"
-        echo "    node$((i + 1)) ($ip): build complete"
-    ) &
-    PIDS+=($!)
-done
-
-for pid in "${PIDS[@]}"; do
-    wait "$pid"
-done
-echo "    All nodes built successfully"
-
-# --- Step 3: Form cluster ---
-echo ""
-echo "==> Forming cluster..."
-
-# Build comma-separated IP lists
-ALL_MGMT_IPS=$(IFS=,; echo "${MGMT_IPS[*]}")
-NODE1_IP="${MGMT_IPS[0]}"
-
-# Node 1: init
-echo "    Starting init on node1 ($NODE1_IP)..."
-remote_bg "$NODE1_IP" "export PATH=\$PATH:/usr/local/go/bin && cd ~/Development/mulga/spinifex && ./bin/spx admin init \
-    --node node1 \
-    --bind $NODE1_IP \
-    --cluster-bind $NODE1_IP \
-    --cluster-routes $NODE1_IP:4248 \
-    --predastore-nodes $ALL_MGMT_IPS \
-    --port 4432 \
-    --nodes $NODE_COUNT \
-    --region $REGION \
-    --az $AZ \
-    --spinifex-dir ~/spinifex/ \
-    --config-dir ~/spinifex/config/"
-INIT_PID=$!
-
-# Wait for formation server to be ready
-echo "    Waiting for formation server on $NODE1_IP..."
-attempts=0
-while ! curl -sk --connect-timeout 3 "https://$NODE1_IP:4432/formation/health" >/dev/null 2>&1; do
-    attempts=$((attempts + 1))
-    if [ $attempts -ge 60 ]; then
-        echo "Error: Formation server did not start on $NODE1_IP after 5 minutes"
-        kill $INIT_PID 2>/dev/null || true
-        exit 1
-    fi
-    sleep 5
-done
-echo "    Formation server ready"
-
-# Read join token from init node
-JOIN_TOKEN=$(remote "$NODE1_IP" "cat ~/spinifex/config/join-token")
-if [ -z "$JOIN_TOKEN" ]; then
-    echo "Error: Failed to read join token from $NODE1_IP"
-    exit 1
-fi
-echo "    Join token loaded from node1"
-
-# Nodes 2..N: join
-if [ "$NODE_COUNT" -gt 1 ]; then
-    JOIN_PIDS=()
-    for i in $(seq 1 $((NODE_COUNT - 1))); do
-        ip="${MGMT_IPS[$i]}"
-        node_num=$((i + 1))
-        echo "    Starting join on node$node_num ($ip)..."
-        remote_bg "$ip" "export PATH=\$PATH:/usr/local/go/bin && cd ~/Development/mulga/spinifex && ./bin/spx admin join \
-            --node node$node_num \
-            --bind $ip \
-            --cluster-bind $ip \
-            --cluster-routes $NODE1_IP:4248 \
-            --host $NODE1_IP:4432 \
-            --token $JOIN_TOKEN \
-            --data-dir ~/spinifex/ \
-            --config-dir ~/spinifex/config/ \
-            --region $REGION \
-            --az $AZ"
-        JOIN_PIDS+=($!)
-    done
-
-    for pid in "${JOIN_PIDS[@]}"; do
-        wait "$pid"
-    done
-fi
-
-# Wait for init process to complete
-wait $INIT_PID
-echo "    Cluster formation complete"
-
-# --- Step 4: Start services ---
-echo ""
-echo "==> Starting services on all nodes..."
-PIDS=()
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    ip="${MGMT_IPS[$i]}"
-    (
-        echo "    node$((i + 1)) ($ip): starting services..."
-        remote "$ip" "export PATH=\$PATH:/usr/local/go/bin && cd ~/Development/mulga/spinifex && ./scripts/start-dev.sh"
-        echo "    node$((i + 1)) ($ip): services started"
-    ) &
-    PIDS+=($!)
-done
-
-for pid in "${PIDS[@]}"; do
-    wait "$pid"
-done
-
-# --- Step 5: Health check ---
-echo ""
-echo "==> Waiting for all nodes to become healthy..."
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    ip="${MGMT_IPS[$i]}"
-    printf "    node%d (%s): " "$((i + 1))" "$ip"
-    attempts=0
-    while true; do
-        health=$(curl -sk --connect-timeout 3 "https://$ip:4432/health" 2>/dev/null || echo "")
-        if [ -n "$health" ]; then
-            echo "healthy"
-            break
-        fi
-        attempts=$((attempts + 1))
-        if [ $attempts -ge 30 ]; then
-            echo "TIMEOUT"
-            echo "Warning: node$((i + 1)) did not become healthy after 150 seconds"
-            break
-        fi
-        sleep 5
-    done
-done
-
-echo ""
-echo "==> Cluster '$CLUSTER_NAME' configured successfully"

scripts/iac/proxmox-template/.env.example

@@ -1,60 +0,0 @@
-# Proxmox Template Builder — Environment Configuration
-#
-# Copy to .env and fill in your values:
-#   cp .env.example .env
-#
-# Then source it before running tofu:
-#   source .env && tofu plan
-
-# --- Proxmox API ---
-
-# API token for Proxmox VE (read by the provider directly)
-export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN_HERE"
-
-# Proxmox VE API endpoint URL
-export TF_VAR_proxmox_endpoint="https://pve-host.example.com:8006/"
-
-# --- Proxmox Node ---
-# Build the template on ONE node. After creation, use vzdump to copy to others.
-#
-#   name    - Proxmox node name (as shown in the Proxmox UI)
-#   address - SSH-reachable hostname or IP of the Proxmox host
-
-export TF_VAR_proxmox_node_name="node1"
-export TF_VAR_proxmox_node_address="node1.example.com"
-
-# --- SSH Access ---
-
-# SSH username on Proxmox hosts (used by provider for file uploads + qm commands)
-export TF_VAR_proxmox_ssh_username="terraform"
-
-# Path to SSH private key for Proxmox host access
-export TF_VAR_ssh_private_key_path="~/.ssh/proxmox-host-tf"
-
-# Path to SSH private key for VM access (must be the pair of the public key below)
-export TF_VAR_vm_ssh_private_key_path="~/.ssh/cloud-init"
-
-# Path to SSH public key injected into VMs via cloud-init (for tf-user)
-export TF_VAR_ssh_public_key_path="~/.ssh/cloud-init.pub"
-
-# --- Template ---
-
-# VM ID for the template (pick a high number to avoid conflicts)
-export TF_VAR_template_vmid=9000
-
-# Short tag for the base image (used in Proxmox template tags)
-export TF_VAR_base_image_tag="debian-12"
-
-# Storage and network for the build node
-export TF_VAR_datastore_id="local-lvm"
-export TF_VAR_bridge="vmbr0"
-
-# --- Optional ---
-
-# Base image — change this when testing a newer Debian/Ubuntu version:
-# export TF_VAR_base_image="local:iso/debian-13-genericcloud-amd64.img"
-# export TF_VAR_base_image_tag="debian-13"
-# export TF_VAR_template_vmid=9010
-
-# Clone a specific branch instead of main:
-# export TF_VAR_git_branch="feature/my-branch"

scripts/iac/proxmox-template/.gitignore

@@ -1,6 +0,0 @@
-.env
-.terraform/
-.terraform.lock.hcl
-*.tfstate
-*.tfstate.backup
-*.tfplan

scripts/iac/proxmox-template/README.md

@@ -1,124 +0,0 @@
-# Spinifex Node Template Builder
-
-Builds a Proxmox VM template with all Spinifex dependencies pre-installed. Cloning from this template reduces cluster spin-up from ~15 minutes to ~1-2 minutes (`git pull` + `make build`).
-
-## What's in the template
-
-| Component | Details |
-|---|---|
-| System packages | QEMU, nbdkit, libvirt, gcc, jq, curl, etc. (`make install-system`) |
-| Go | 1.26.4 (`make install-go`) |
-| AWS CLI | v2 (`make install-aws`) |
-| Repositories | spinifex, viperblock, predastore (with `go.work` configured) |
-| Module cache | `go mod download` completed for all repos |
-| Cloud images | Ubuntu 26.04 for nested VMs (`~/images/ubuntu-26.04.img`) |
-| Tuning | sysctl (rmem_max/wmem_max), kvm group, ufw disabled |
-| Generalized | Clean machine-id, SSH host keys, cloud-init — each clone boots fresh |
-
-## Quick Start
-
-```bash
-cd scripts/iac/proxmox-template
-
-# 1. Configure
-cp .env.example .env
-# Edit .env — set your Proxmox endpoint, API token, node, and SSH key paths.
-
-# 2. Set API token (not stored in .env for security)
-export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN"
-
-# 3. Build
-source .env
-tofu init
-tofu plan
-tofu apply
-```
-
-Build takes ~10-15 minutes (apt packages, Go, cloud image download). When done, you'll have a template on the build node at the configured VMID (default 9000).
-
-## Distribute to Other Nodes
-
-After `tofu apply`, the output prints ready-to-run commands. Example for a 3-node cluster:
-
-```bash
-# Export from build node
-ssh -i ~/.ssh/proxmox-host-tf terraform@<BUILD_NODE> \
-  'vzdump 9000 --dumpdir /tmp --mode stop --compress zstd'
-
-# Copy to other nodes
-scp -3 -i ~/.ssh/proxmox-host-tf \
-  terraform@<BUILD_NODE>:/tmp/vzdump-qemu-9000-*.vma.zst \
-  terraform@<TARGET_NODE>:/tmp/
-
-# Restore on each target node (use a unique VMID and the node's datastore)
-ssh -i ~/.ssh/proxmox-host-tf terraform@<TARGET_NODE> \
-  'qmrestore /tmp/vzdump-qemu-9000-*.vma.zst <NEW_VMID> --storage <DATASTORE> && qm template <NEW_VMID>'
-```
-
-After distributing, each node has its own copy of the template with a unique VMID.
-
-## Using the Template
-
-Update `scripts/iac/proxmox/vms.tf` to clone from the template instead of booting from a raw cloud image. Or clone manually from the Proxmox UI.
-
-On each cloned VM after boot:
-
-```bash
-# Pull latest code
-cd ~/Development/mulga/spinifex && git pull
-cd ~/Development/mulga/viperblock && git pull
-cd ~/Development/mulga/predastore && git pull
-
-# Build (~30s with cached modules)
-cd ~/Development/mulga/spinifex && export PATH=/usr/local/go/bin:$PATH && make build
-```
-
-## Testing a Different Base Image
-
-Upload the new image to Proxmox, then build a separate template:
-
-```bash
-# Debian 13 example
-export TF_VAR_base_image="local:iso/debian-13-genericcloud-amd64.img"
-export TF_VAR_base_image_tag="debian-13"
-export TF_VAR_template_vmid=9010   # Keep the debian-12 template at 9000
-
-source .env
-tofu apply
-```
-
-Templates are tagged in the Proxmox UI with the base image name (`debian-12`, `debian-13`, etc.) so you can tell them apart.
-
-## Rebuilding
-
-When dependencies change (new Go version, new system packages):
-
-```bash
-source .env
-tofu destroy     # Remove the old template
-tofu apply       # Build fresh
-# Then re-distribute with vzdump
-```
-
-## SSH Key Setup
-
-This template uses **two separate key pairs** (matching the `scripts/iac/proxmox/` setup):
-
-| Key | Used for | Variable |
-|---|---|---|
-| Proxmox host key | SSH to Proxmox hosts (provider, `qm template` command) | `ssh_private_key_path` |
-| VM / cloud-init key | SSH to VMs (cloud-init injection, provisioning) | `vm_ssh_private_key_path` + `ssh_public_key_path` |
-
-If you use the same key for both, set all three variables to the same key pair.
-
-## Files
-
-| File | Description |
-|------|-------------|
-| `main.tf` | Proxmox provider configuration |
-| `variables.tf` | All configurable variables with defaults |
-| `template-builder.tf` | VM creation, provisioning, template conversion |
-| `outputs.tf` | Template info + vzdump/restore commands |
-| `scripts/provision.sh` | Install deps, clone repos, cache modules, generalize |
-| `.env.example` | Environment variable template |
-| `.gitignore` | Ignores .env, state files, provider plugins |

scripts/iac/proxmox-template/main.tf

@@ -1,27 +0,0 @@
-terraform {
-  required_version = ">= 1.6.0"
-
-  required_providers {
-    proxmox = {
-      source  = "bpg/proxmox"
-      version = "0.52.0"
-    }
-  }
-}
-
-# API token is read from PROXMOX_VE_API_TOKEN env var automatically
-provider "proxmox" {
-  endpoint = var.proxmox_endpoint
-  insecure = true
-
-  ssh {
-    agent       = false
-    username    = var.proxmox_ssh_username
-    private_key = file(pathexpand(var.ssh_private_key_path))
-
-    node {
-      name    = var.proxmox_node_name
-      address = var.proxmox_node_address
-    }
-  }
-}

scripts/iac/proxmox-template/outputs.tf

@@ -1,43 +0,0 @@
-output "template_vmid" {
-  description = "VM ID of the created template"
-  value       = proxmox_virtual_environment_vm.builder.vm_id
-}
-
-output "template_node" {
-  description = "Proxmox node where the template lives"
-  value       = var.proxmox_node_name
-}
-
-output "base_image_tag" {
-  description = "Base image tag for this template"
-  value       = var.base_image_tag
-}
-
-# Ready-to-run command: export the template for distribution to other Proxmox nodes
-output "vzdump_export_command" {
-  description = "Run this to export the template (vzdump) for copying to other nodes"
-  value       = <<-EOT
-    ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@${var.proxmox_node_address} \
-      'vzdump ${var.template_vmid} --dumpdir /tmp --mode stop --compress zstd'
-  EOT
-}
-
-# After exporting, copy to each target node and restore
-output "distribute_commands" {
-  description = "Copy and restore the template on other Proxmox nodes"
-  value       = <<-EOT
-    # 1. Export from build node (${var.proxmox_node_name}):
-    ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@${var.proxmox_node_address} \
-      'vzdump ${var.template_vmid} --dumpdir /tmp --mode stop --compress zstd'
-
-    # 2. Copy to each target node:
-    scp -3 \
-      -i ${var.ssh_private_key_path} \
-      ${var.proxmox_ssh_username}@${var.proxmox_node_address}:/tmp/vzdump-qemu-${var.template_vmid}-*.vma.zst \
-      ${var.proxmox_ssh_username}@<TARGET_NODE_ADDRESS>:/tmp/
-
-    # 3. On each target node — restore and convert to template:
-    ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@<TARGET_NODE_ADDRESS> \
-      'qmrestore /tmp/vzdump-qemu-${var.template_vmid}-*.vma.zst <NEW_VMID> --storage <DATASTORE> && qm template <NEW_VMID>'
-  EOT
-}

scripts/iac/proxmox-template/scripts/provision.sh

@@ -1,176 +0,0 @@
-#!/bin/bash
-# provision.sh — Installs all Spinifex dependencies, clones repos, caches Go modules,
-# and generalizes the VM for use as a Proxmox template.
-#
-# Runs as tf-user via Terraform remote-exec. Uses sudo for system operations.
-# Environment: GIT_BRANCH (default: dev)
-
-set -euo pipefail
-
-GIT_BRANCH="${GIT_BRANCH:-dev}"
-WORK_DIR="$HOME/Development/mulga"
-
-echo "========================================"
-echo "  Spinifex Template Provisioning"
-echo "  Branch: ${GIT_BRANCH}"
-echo "  Arch:   $(uname -m)"
-echo "========================================"
-
-# ── 1. Wait for cloud-init ──────────────────────────────────────────────────
-echo ""
-echo "=== [1/8] Waiting for cloud-init to complete ==="
-cloud-init status --wait
-
-# ── 2. Clone repositories ───────────────────────────────────────────────────
-echo ""
-echo "=== [2/8] Cloning repositories ==="
-mkdir -p "$WORK_DIR"
-cd "$WORK_DIR"
-
-for repo in spinifex viperblock predastore; do
-    if [ ! -d "$repo" ]; then
-        echo "  Cloning $repo (branch: $GIT_BRANCH)..."
-        git clone -b "$GIT_BRANCH" "https://github.com/mulgadc/${repo}.git"
-    else
-        echo "  $repo already exists, skipping"
-    fi
-done
-
-# ── 3. Install system dependencies via Makefile ─────────────────────────────
-echo ""
-echo "=== [3/8] Installing system dependencies ==="
-cd "$WORK_DIR/spinifex"
-sudo make quickinstall
-
-# Ensure Go is in PATH for the rest of the script
-export PATH="/usr/local/go/bin:$HOME/go/bin:$PATH"
-echo "  Go version: $(go version)"
-echo "  AWS CLI:    $(aws --version)"
-
-# ── 4. Set up go.work ───────────────────────────────────────────────────────
-echo ""
-echo "=== [4/8] Creating go.work ==="
-
-# Detect Go version from go.mod
-GO_VERSION=$(grep '^go ' "$WORK_DIR/spinifex/go.mod" | awk '{print $2}')
-echo "  Using Go version from go.mod: ${GO_VERSION}"
-
-cat > "$WORK_DIR/spinifex/go.work" << GOWORK
-go ${GO_VERSION}
-
-use (
-    .
-    ../viperblock
-    ../viperblock/nbd
-    ../viperblock/nbd/libguestfs.org/nbdkit
-    ../predastore
-)
-GOWORK
-
-echo "  Created $WORK_DIR/spinifex/go.work"
-
-# ── 5. Download Go modules ──────────────────────────────────────────────────
-echo ""
-echo "=== [5/8] Downloading Go modules ==="
-cd "$WORK_DIR/spinifex"    && go mod download && echo "  spinifex: done"
-cd "$WORK_DIR/viperblock"  && go mod download && echo "  viperblock: done"
-cd "$WORK_DIR/predastore"  && go mod download && echo "  predastore: done"
-
-# Sub-modules
-if [ -f "$WORK_DIR/viperblock/nbd/go.mod" ]; then
-    cd "$WORK_DIR/viperblock/nbd" && go mod download && echo "  viperblock/nbd: done"
-fi
-if [ -f "$WORK_DIR/viperblock/nbd/libguestfs.org/nbdkit/go.mod" ]; then
-    cd "$WORK_DIR/viperblock/nbd/libguestfs.org/nbdkit" && go mod download && echo "  viperblock/nbd/nbdkit: done"
-fi
-
-# ── 6. Pre-download cloud images for nested VMs ─────────────────────────────
-echo ""
-echo "=== [6/8] Pre-downloading cloud images ==="
-mkdir -p "$HOME/images"
-
-ARCH=$(dpkg --print-architecture)
-IMAGE_URL=""
-if [ "$ARCH" = "amd64" ]; then
-    IMAGE_URL="https://cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-amd64.img"
-elif [ "$ARCH" = "arm64" ]; then
-    IMAGE_URL="https://cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-arm64.img"
-fi
-
-if [ -n "$IMAGE_URL" ]; then
-    echo "  Downloading Ubuntu 26.04 cloud image ($ARCH)..."
-    curl -L -o "$HOME/images/ubuntu-26.04.img" "$IMAGE_URL"
-    echo "  Saved to ~/images/ubuntu-26.04.img"
-else
-    echo "  Unknown arch ($ARCH), skipping cloud image download"
-fi
-
-# ── 7. System tuning ────────────────────────────────────────────────────────
-echo ""
-echo "=== [7/8] Applying system tuning ==="
-
-# Persistent sysctl for NATS / high-throughput networking
-sudo tee /etc/sysctl.d/99-spinifex.conf > /dev/null << 'SYSCTL'
-net.core.rmem_max=4194304
-net.core.wmem_max=4194304
-SYSCTL
-echo "  sysctl: net.core.rmem_max/wmem_max = 4MB"
-
-# Add user to kvm group for QEMU access
-sudo usermod -aG kvm "$(whoami)"
-echo "  Added $(whoami) to kvm group"
-
-# Disable ufw if present
-if command -v ufw >/dev/null 2>&1; then
-    sudo ufw disable 2>/dev/null || true
-    echo "  ufw disabled"
-fi
-
-# ── 8. Generalize for templating ────────────────────────────────────────────
-echo ""
-echo "=== [8/8] Cleaning up for template ==="
-
-# Apt caches
-sudo apt-get clean
-sudo rm -rf /var/lib/apt/lists/*
-echo "  Cleared apt cache"
-
-# Logs
-sudo find /var/log -type f -name "*.log" -exec truncate -s 0 {} \; 2>/dev/null || true
-sudo rm -rf /var/log/journal/*
-echo "  Truncated logs"
-
-# Temp files
-sudo rm -rf /tmp/* /var/tmp/*
-
-# Machine ID — systemd regenerates on next boot.
-# This is critical: without this, all clones get the same DHCP client ID → IP conflicts.
-sudo truncate -s 0 /etc/machine-id
-sudo rm -f /var/lib/dbus/machine-id
-echo "  Cleared machine-id (will regenerate on boot)"
-
-# SSH host keys — regenerated on boot by cloud-init or systemd
-sudo rm -f /etc/ssh/ssh_host_*
-echo "  Removed SSH host keys (will regenerate on boot)"
-
-# Cloud-init state — forces cloud-init to run again on next boot.
-# This ensures each clone gets its own hostname, SSH keys, user config, etc.
-sudo cloud-init clean
-echo "  Cleaned cloud-init state (will re-run on boot)"
-
-# Shell history
-history -c 2>/dev/null || true
-cat /dev/null > "$HOME/.bash_history" 2>/dev/null || true
-
-# Provisioning script itself
-rm -f /tmp/provision.sh
-
-# Sync filesystem
-sync
-
-echo ""
-echo "========================================"
-echo "  Template provisioning complete!"
-echo "  VM is ready to be shut down and"
-echo "  converted to a Proxmox template."
-echo "========================================"

scripts/iac/proxmox-template/template-builder.tf

@@ -1,130 +0,0 @@
-data "local_file" "ssh_public_key" {
-  filename = pathexpand(var.ssh_public_key_path)
-}
-
-# Minimal cloud-init: just enough to bootstrap SSH + guest agent.
-# Heavy provisioning is done via remote-exec after boot.
-resource "proxmox_virtual_environment_file" "cloud_config" {
-  content_type = "snippets"
-  datastore_id = "local"
-  node_name    = var.proxmox_node_name
-
-  source_raw {
-    # NOTE: #cloud-config MUST start at column 0 — cloud-init uses it as a magic header.
-    data = <<EOF
-#cloud-config
-package_update: true
-packages:
-  - qemu-guest-agent
-  - git
-  - make
-  - curl
-  - sudo
-  - ca-certificates
-runcmd:
-  - systemctl enable qemu-guest-agent
-  - systemctl start qemu-guest-agent
-EOF
-
-    file_name = "hive-template-cloud-config.yaml"
-  }
-}
-
-resource "proxmox_virtual_environment_vm" "builder" {
-  name      = "${var.template_name}-builder"
-  node_name = var.proxmox_node_name
-  vm_id     = var.template_vmid
-
-  tags = sort(["terraform", "hive-template", var.base_image_tag])
-
-  initialization {
-    datastore_id        = "local"
-    vendor_data_file_id = proxmox_virtual_environment_file.cloud_config.id
-
-    user_account {
-      username = var.vm_ssh_user
-      keys     = [trimspace(data.local_file.ssh_public_key.content)]
-    }
-
-    ip_config {
-      ipv4 {
-        address = "dhcp"
-      }
-    }
-  }
-
-  cpu {
-    cores = var.cpu_cores
-    type  = "host"
-  }
-
-  memory {
-    dedicated = var.memory_mb
-  }
-
-  agent {
-    enabled = true
-  }
-
-  network_device {
-    bridge = var.bridge
-  }
-
-  disk {
-    file_id      = var.base_image
-    datastore_id = var.datastore_id
-    interface    = "virtio0"
-    iothread     = true
-    discard      = "on"
-    size         = var.disk_size_gb
-  }
-
-  # SSH connection to the VM for provisioning.
-  # Uses the VM key (vm_ssh_private_key_path), NOT the Proxmox host key.
-  # Waits for guest agent to report an IP, then connects.
-  connection {
-    type        = "ssh"
-    user        = var.vm_ssh_user
-    private_key = file(pathexpand(var.vm_ssh_private_key_path))
-    host        = element([for addr in flatten(self.ipv4_addresses) : addr if !startswith(addr, "127.")], 0)
-    timeout     = "10m"
-  }
-
-  # Copy provisioning script to VM
-  provisioner "file" {
-    source      = "${path.module}/scripts/provision.sh"
-    destination = "/tmp/provision.sh"
-  }
-
-  # Run provisioning (installs deps, clones repos, caches modules, cleans up)
-  provisioner "remote-exec" {
-    inline = [
-      "chmod +x /tmp/provision.sh",
-      "GIT_BRANCH='${var.git_branch}' /tmp/provision.sh",
-    ]
-  }
-}
-
-# After provisioning completes, shut down the VM and convert to a Proxmox template.
-# This runs on the Proxmox host via SSH (not on the VM — it's being shut down).
-resource "null_resource" "convert_template" {
-  depends_on = [proxmox_virtual_environment_vm.builder]
-
-  triggers = {
-    builder_id = proxmox_virtual_environment_vm.builder.vm_id
-  }
-
-  provisioner "local-exec" {
-    command = <<-EOT
-      ssh -i '${pathexpand(var.ssh_private_key_path)}' \
-        -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
-        '${var.proxmox_ssh_username}@${var.proxmox_node_address}' \
-        "sudo qm shutdown ${proxmox_virtual_environment_vm.builder.vm_id} --timeout 60 && \
-         echo 'Waiting for VM to stop...' && \
-         while sudo qm status ${proxmox_virtual_environment_vm.builder.vm_id} | grep -q running; do sleep 2; done && \
-         echo 'Converting to template...' && \
-         sudo qm template ${proxmox_virtual_environment_vm.builder.vm_id} && \
-         echo 'Template ${var.template_vmid} created successfully'"
-    EOT
-  }
-}

scripts/iac/proxmox-template/variables.tf

@@ -1,117 +0,0 @@
-# --- Proxmox Connection ---
-
-variable "proxmox_endpoint" {
-  type        = string
-  description = "Proxmox VE API endpoint URL (e.g. https://pve.example.com:8006/)"
-}
-
-variable "proxmox_node_name" {
-  type        = string
-  description = "Proxmox node name to build the template on (as shown in the Proxmox UI)"
-}
-
-variable "proxmox_node_address" {
-  type        = string
-  description = "SSH-reachable hostname or IP of the Proxmox node"
-}
-
-variable "proxmox_ssh_username" {
-  type        = string
-  description = "SSH username for Proxmox host access"
-  default     = "terraform"
-}
-
-# --- SSH Keys ---
-# Proxmox host and VM typically use different key pairs.
-# - ssh_private_key_path  → authenticates to the Proxmox host (provider + qm commands)
-# - vm_ssh_private_key_path → authenticates to the VM for provisioning (remote-exec)
-# - ssh_public_key_path → injected into the VM via cloud-init
-
-variable "ssh_private_key_path" {
-  type        = string
-  description = "Path to SSH private key for Proxmox host access"
-}
-
-variable "vm_ssh_private_key_path" {
-  type        = string
-  description = "Path to SSH private key for VM access (must match ssh_public_key_path)"
-}
-
-variable "ssh_public_key_path" {
-  type        = string
-  description = "Path to SSH public key injected into the VM via cloud-init"
-}
-
-# --- Base Image ---
-
-variable "base_image" {
-  type        = string
-  description = "Proxmox image file ID for the base OS (e.g. local:iso/debian-12-genericcloud-amd64-20240211-1654.img)"
-  default     = "local:iso/debian-12-genericcloud-amd64-20240211-1654.img"
-}
-
-variable "base_image_tag" {
-  type        = string
-  description = "Short tag identifying the base image, used in template tags for tracking"
-  default     = "debian-12"
-}
-
-# --- Template Settings ---
-
-variable "template_name" {
-  type        = string
-  description = "Name for the resulting Proxmox template"
-  default     = "hive-node-template"
-}
-
-variable "template_vmid" {
-  type        = number
-  description = "VM ID for the template (pick a high number to avoid conflicts, e.g. 9000)"
-  default     = 9000
-}
-
-variable "vm_ssh_user" {
-  type        = string
-  description = "Username created in the VM via cloud-init"
-  default     = "tf-user"
-}
-
-# --- VM Resources ---
-
-variable "cpu_cores" {
-  type        = number
-  description = "CPU cores for the builder VM (more = faster builds)"
-  default     = 4
-}
-
-variable "memory_mb" {
-  type        = number
-  description = "Memory in MB for the builder VM"
-  default     = 8192
-}
-
-variable "disk_size_gb" {
-  type        = number
-  description = "Disk size in GB (needs room for OS + Go + modules + cloud images)"
-  default     = 32
-}
-
-variable "datastore_id" {
-  type        = string
-  description = "Proxmox storage for the VM disk"
-  default     = "local-lvm"
-}
-
-variable "bridge" {
-  type        = string
-  description = "Network bridge for the VM"
-  default     = "vmbr0"
-}
-
-# --- Git ---
-
-variable "git_branch" {
-  type        = string
-  description = "Git branch to clone for spinifex, viperblock, and predastore"
-  default     = "dev"
-}

scripts/iac/proxmox/.env.example

@@ -1,68 +0,0 @@
-# Proxmox IaC Environment Configuration
-# Copy this file to .env and fill in your values:
-#   cp .env.example .env
-#
-# Then source it before running tofu:
-#   source .env && tofu plan
-
-# --- Proxmox API ---
-
-# API token for Proxmox VE (read by the provider directly)
-export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN_HERE"
-
-# Proxmox VE API endpoint URL
-export TF_VAR_proxmox_endpoint="https://pve-host.example.com:8006/"
-
-# --- SSH Access ---
-
-# SSH username on Proxmox hosts (used by provider for file uploads)
-export TF_VAR_proxmox_ssh_username="terraform"
-
-# Path to SSH private key for Proxmox host access
-export TF_VAR_ssh_private_key_path="~/.ssh/proxmox-host-tf"
-
-# Path to SSH public key injected into VMs via cloud-init (for tf-user)
-export TF_VAR_ssh_public_key_path="~/.ssh/cloud-init.pub"
-
-# --- Proxmox Nodes ---
-# Physical Proxmox hosts where VMs are placed (round-robin).
-# You can define 1 or more nodes — VMs are distributed across them.
-#
-# Each node object:
-#   name          - Proxmox node name (as shown in the Proxmox UI)
-#   address       - SSH-reachable hostname or IP of the Proxmox host
-#   bridge        - Network bridge for VM interfaces (e.g. vmbr0, vmbr3)
-#   datastore_id  - Proxmox storage ID for VM disks (e.g. local, local-lvm)
-#   template_vmid - (optional) VMID of a pre-built template on this node.
-#                   Build with scripts/iac/proxmox-template/, then distribute
-#                   with vzdump. When set, VMs clone from the template (fast).
-#                   When omitted, VMs boot from os_image (slower).
-
-export TF_VAR_nodes='[
-  {
-    "name": "node1",
-    "address": "node1.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local"
-  },
-  {
-    "name": "node2",
-    "address": "node2.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local"
-  },
-  {
-    "name": "node3",
-    "address": "node3.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local"
-  }
-]'
-
-# Example with templates (VMIDs may differ per node after vzdump restore):
-#
-# export TF_VAR_nodes='[
-#   { "name": "node1", "address": "node1.example.com", "bridge": "vmbr0", "datastore_id": "local", "template_vmid": 9999 },
-#   { "name": "node2", "address": "node2.example.com", "bridge": "vmbr0", "datastore_id": "local-lvm", "template_vmid": 9997 },
-#   { "name": "node3", "address": "node3.example.com", "bridge": "vmbr0", "datastore_id": "ssd", "template_vmid": 9998 }
-# ]'

scripts/iac/proxmox/README.md

@@ -1,191 +0,0 @@
-# Proxmox IaC
-
-OpenTofu/Terraform templates for provisioning Spinifex development clusters on Proxmox VE.
-
-## Prerequisites
-
-- [OpenTofu](https://opentofu.org/) (or Terraform >= 1.6.0)
-- 1 or more Proxmox VE nodes with:
-  - A Terraform API token (`Datacenter > Permissions > API Tokens`)
-  - SSH access for a `terraform` user on each node
-  - Debian 13 cloud image uploaded to `local:iso/` on each node
-- SSH keypair for cloud-init VM access
-- A hardcoded Debian ISO is used, which needs to be pre-installed on each Proxmox host ("debian-13-genericcloud-amd64-20260518-2482.img") https://cloud.debian.org/images/cloud/trixie/20260518-2482/debian-13-genericcloud-amd64-20260518-2482.qcow2
-
-## Quick Start (spinifex-test.sh)
-
-The `spinifex-test.sh` wrapper script manages the full lifecycle: provision, configure, test, and destroy.
-
-```sh
-# Source environment
-source scripts/iac/proxmox/.env
-source ~/prox   # Proxmox API token
-
-# Provision a 3-node cluster
-./scripts/iac/spinifex-test.sh up my-cluster
-
-# Configure: clone repo, build, form cluster, start services
-./scripts/iac/spinifex-test.sh configure my-cluster
-
-# Run smoke tests
-./scripts/iac/spinifex-test.sh test my-cluster
-
-# Check cluster health
-./scripts/iac/spinifex-test.sh status my-cluster
-
-# SSH to a specific node
-./scripts/iac/spinifex-test.sh ssh my-cluster 2
-
-# Destroy when done
-./scripts/iac/spinifex-test.sh down my-cluster
-
-# Or run the full lifecycle (up → configure → test → down)
-./scripts/iac/spinifex-test.sh full my-cluster
-```
-
-### Options
-
-```
---node-count=N       Number of VMs (default: 3)
---memory-mb=N        Memory per VM in MB (default: 16384)
---cpu-cores=N        CPU cores per VM (default: 4)
---disk-size-gb=N     Disk size per VM in GB (default: 32)
-```
-
-Example with custom sizing:
-
-```sh
-./scripts/iac/spinifex-test.sh up my-cluster --node-count=5 --memory-mb=8192 --cpu-cores=2
-```
-
-Per-cluster state is stored in `scripts/iac/proxmox/clusters/<name>/terraform.tfstate`, allowing multiple clusters to coexist.
-
-## Manual Setup
-
-### 1. Configure environment
-
-```sh
-cp .env.example .env
-# Edit .env with your Proxmox endpoint, SSH keys, and node configuration
-```
-
-### 2. Set API token
-
-```sh
-export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN_SECRET"
-```
-
-### 3. Deploy
-
-```sh
-source .env
-tofu init
-tofu plan -var="cluster_name=dev1"
-tofu apply -var="cluster_name=dev1"
-```
-
-### Example `.env` file
-
-```sh
-# --- Proxmox API ---
-
-# API token - set manually via: export PROXMOX_VE_API_TOKEN="terraform@pve!provider=..."
-# Format: <user>@<realm>!<token-name>=<token-secret>
-
-# Proxmox VE API endpoint (the URL you use to access the web UI)
-export TF_VAR_proxmox_endpoint="https://pve1.lab.example.com:8006/"
-
-# --- SSH Access ---
-
-# SSH username on the Proxmox hosts (used by the provider for file uploads via SCP)
-export TF_VAR_proxmox_ssh_username="terraform"
-
-# Path to SSH private key that authenticates as the above user on each Proxmox host
-export TF_VAR_ssh_private_key_path="~/.ssh/proxmox-tf"
-
-# Path to SSH public key injected into VMs via cloud-init (the tf-user account)
-export TF_VAR_ssh_public_key_path="~/.ssh/proxmox-tf-cloudinit.pub"
-
-# --- Proxmox Nodes ---
-#
-# Physical Proxmox hosts where VMs are placed (round-robin).
-# Each node object:
-#   name         - Proxmox node name exactly as shown in the Proxmox UI sidebar
-#   address      - SSH-reachable hostname or IP of the Proxmox host
-#   bridge       - Linux bridge for VM network interfaces (check Proxmox UI > Node > Network)
-#   datastore_id - Proxmox storage pool for VM disks (check Proxmox UI > Node > Storage)
-
-export TF_VAR_nodes='[
-  {
-    "name": "pve1",
-    "address": "pve1.lab.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local-lvm"
-  },
-  {
-    "name": "pve2",
-    "address": "pve2.lab.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local-lvm"
-  },
-  {
-    "name": "pve3",
-    "address": "pve3.lab.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local-lvm"
-  }
-]'
-```
-
-### Environment variable reference
-
-| Variable | Required | Description |
-|---|---|---|
-| `PROXMOX_VE_API_TOKEN` | Yes | Proxmox API token (read by provider directly) |
-| `TF_VAR_proxmox_endpoint` | Yes | Proxmox VE API URL (e.g. `https://host:8006/`) |
-| `TF_VAR_proxmox_ssh_username` | No | SSH user on Proxmox hosts (default: `terraform`) |
-| `TF_VAR_ssh_private_key_path` | Yes | SSH private key for Proxmox host access |
-| `TF_VAR_ssh_public_key_path` | Yes | SSH public key injected into VMs via cloud-init |
-| `TF_VAR_nodes` | Yes | JSON array of Proxmox node objects |
-
-Each node object in `TF_VAR_nodes`:
-
-| Field | Example | Description |
-|---|---|---|
-| `name` | `pve1` | Proxmox node name (as shown in UI sidebar) |
-| `address` | `pve1.lab.example.com` | SSH-reachable hostname or IP |
-| `bridge` | `vmbr0` | Network bridge for VM interfaces |
-| `datastore_id` | `local-lvm` | Storage pool for VM disks |
-
-### Cluster variables (passed via `-var` or `TF_VAR_`)
-
-| Variable | Default | Description |
-|---|---|---|
-| `cluster_name` | (required) | Cluster name — used in VM names and tags |
-| `node_count` | `3` | Number of VMs to create (1-10) |
-| `cpu_cores` | `4` | CPU cores per VM |
-| `memory_mb` | `16384` | Memory per VM in MB |
-| `disk_size_gb` | `32` | Disk size per VM in GB |
-| `os_image` | `local:iso/debian-12-...` | Proxmox image for boot disk |
-
-## Destroy
-
-```sh
-source .env
-tofu destroy -var="cluster_name=dev1"
-```
-
-## SSH access
-
-Connect to provisioned VMs using the cloud-init public key:
-
-```sh
-ssh -i ~/.ssh/your-cloud-init-key tf-user@<VM_IP>
-```
-
-## Known issues
-
-- `~/spinifex/config/spinifex.toml` - Does not add node1, node2, node3 from config automatically
-- `~/spinifex/config/predastore/predastore.toml` - Uses previous static local node config, needs to use the IPs for each node in the cluster
-- On multi-node deployments, NATS on the primary can timeout waiting for other nodes to start, causing a race condition where NATS fails and all dependent services fail
-- When adding 3 nodes, `nats.conf` is not updated with the 3rd node. Each node's cluster name is hardcoded to `C1` instead of per-node names

scripts/iac/proxmox/cloud-config.tf

@@ -1,48 +0,0 @@
-data "local_file" "ssh_public_key" {
-  filename = pathexpand(var.ssh_public_key_path)
-}
-
-# When cloning from a template, packages are pre-installed — skip apt update/upgrade.
-locals {
-  cloud_config_template = <<-EOF
-    #cloud-config
-    runcmd:
-      - timedatectl set-timezone America/Los_Angeles
-      - systemctl enable qemu-guest-agent
-      - systemctl start qemu-guest-agent
-      - echo "done" > /tmp/vendor-cloud-init-done
-  EOF
-
-  cloud_config_image = <<-EOF
-    #cloud-config
-    package_update: true
-    package_upgrade: true
-    packages:
-      - qemu-guest-agent
-      - net-tools
-      - git
-      - make
-    runcmd:
-      - timedatectl set-timezone America/Los_Angeles
-      - systemctl enable qemu-guest-agent
-      - systemctl start qemu-guest-agent
-      - ufw disable
-      - sysctl -w net.core.rmem_max=4194304
-      - sysctl -w net.core.wmem_max=4194304
-      - usermod -aG kvm tf-user
-      - echo "done" > /tmp/vendor-cloud-init-done
-  EOF
-}
-
-resource "proxmox_virtual_environment_file" "cloud_config" {
-  count = var.node_count
-
-  content_type = "snippets"
-  datastore_id = "local"
-  node_name    = var.nodes[count.index % length(var.nodes)].name
-
-  source_raw {
-    data      = var.nodes[count.index % length(var.nodes)].template_vmid != null ? local.cloud_config_template : local.cloud_config_image
-    file_name = "${var.cluster_name}-${count.index + 1}-cloud-config.yaml"
-  }
-}

scripts/iac/proxmox/files.tf

@@ -1,17 +0,0 @@
-# Files already exist, skip
-
-#resource "proxmox_virtual_environment_download_file" "debian_cloud_image_neon" {
-#  content_type = "iso"
-#  datastore_id = "local"
-#  node_name    = "neon"
-#  url          = "https://cdimage.debian.org/images/cloud/bookworm/20240211-1654/debian-12-genericcloud-amd64-20240211-1654.qcow2"
-#  file_name    = "debian-12-genericcloud-amd64-20240211-1654.img"
-#}
-
-#resource "proxmox_virtual_environment_download_file" "debian_cloud_image_radon" {
-#  content_type = "iso"
-#  datastore_id = "local"
-#  node_name    = "radon"
-#  url          = "https://cdimage.debian.org/images/cloud/bookworm/20240211-1654/debian-12-genericcloud-amd64-20240211-1654.qcow2"
-#  file_name    = "debian-12-genericcloud-amd64-20240211-1654.img"
-#}

scripts/iac/proxmox/main.tf

@@ -1,28 +0,0 @@
-terraform {
-  required_providers {
-    proxmox = {
-      source  = "bpg/proxmox"
-      version = "0.52.0"
-    }
-  }
-}
-
-# API token is read from PROXMOX_VE_API_TOKEN env var automatically
-provider "proxmox" {
-  endpoint = var.proxmox_endpoint
-  insecure = true
-
-  ssh {
-    agent       = false
-    username    = var.proxmox_ssh_username
-    private_key = file(pathexpand(var.ssh_private_key_path))
-
-    dynamic "node" {
-      for_each = var.nodes
-      content {
-        name    = node.value.name
-        address = node.value.address
-      }
-    }
-  }
-}

scripts/iac/proxmox/outputs.tf

@@ -1,14 +0,0 @@
-output "inventory" {
-  value = {
-    cluster_name = var.cluster_name
-    node_count   = var.node_count
-    nodes = [for i, vm in proxmox_virtual_environment_vm.hive_node : {
-      name       = "${var.cluster_name}-${i + 1}"
-      index      = i + 1
-      management = vm.ipv4_addresses[1][0]
-      data       = vm.ipv4_addresses[2][0]
-    }]
-    ssh_user     = "tf-user"
-    ssh_key_path = trimsuffix(var.ssh_public_key_path, ".pub")
-  }
-}

scripts/iac/proxmox/variables.tf

@@ -1,106 +0,0 @@
-variable "proxmox_endpoint" {
-  type        = string
-  description = "Proxmox VE API endpoint URL (e.g. https://pve.example.com:8006/)"
-
-  validation {
-    condition     = can(regex("^https?://", var.proxmox_endpoint))
-    error_message = "proxmox_endpoint must be a valid URL starting with http:// or https://"
-  }
-}
-
-variable "proxmox_ssh_username" {
-  type        = string
-  description = "SSH username for Proxmox host access"
-  default     = "terraform"
-}
-
-variable "ssh_private_key_path" {
-  type        = string
-  description = "Path to SSH private key for Proxmox host access"
-
-  validation {
-    condition     = length(var.ssh_private_key_path) > 0
-    error_message = "ssh_private_key_path must not be empty."
-  }
-}
-
-variable "ssh_public_key_path" {
-  type        = string
-  description = "Path to SSH public key injected into VMs via cloud-init"
-
-  validation {
-    condition     = length(var.ssh_public_key_path) > 0
-    error_message = "ssh_public_key_path must not be empty."
-  }
-}
-
-variable "nodes" {
-  type = list(object({
-    name          = string
-    address       = string
-    bridge        = string
-    datastore_id  = string
-    template_vmid = optional(number)
-  }))
-  description = "Proxmox hosts for VM placement (VMs are distributed round-robin). Set template_vmid to clone from a pre-built template instead of booting from os_image."
-
-  validation {
-    condition     = length(var.nodes) > 0
-    error_message = "At least one Proxmox node must be defined."
-  }
-
-  validation {
-    condition     = alltrue([for n in var.nodes : length(n.name) > 0 && length(n.address) > 0])
-    error_message = "Each node must have a non-empty name and address."
-  }
-
-  validation {
-    condition     = length(distinct([for n in var.nodes : n.name])) == length(var.nodes)
-    error_message = "Each node must have a unique name."
-  }
-}
-
-variable "cluster_name" {
-  type        = string
-  description = "Name for the Spinifex cluster (used in VM names and tags)"
-
-  validation {
-    condition     = can(regex("^[a-z0-9-]+$", var.cluster_name))
-    error_message = "cluster_name must contain only lowercase letters, numbers, and hyphens."
-  }
-}
-
-variable "node_count" {
-  type        = number
-  description = "Number of Spinifex VMs to create"
-  default     = 3
-
-  validation {
-    condition     = var.node_count >= 1 && var.node_count <= 10
-    error_message = "node_count must be between 1 and 10."
-  }
-}
-
-variable "cpu_cores" {
-  type        = number
-  description = "Number of CPU cores per VM"
-  default     = 4
-}
-
-variable "memory_mb" {
-  type        = number
-  description = "Memory in MB per VM"
-  default     = 16384
-}
-
-variable "disk_size_gb" {
-  type        = number
-  description = "Disk size in GB per VM"
-  default     = 32
-}
-
-variable "os_image" {
-  type        = string
-  description = "Proxmox image file ID for VM boot disk (used when node has no template_vmid)"
-  default     = "local:iso/debian-12-genericcloud-amd64-20240211-1654.img"
-}

scripts/iac/proxmox/vms.tf

@@ -1,74 +0,0 @@
-resource "proxmox_virtual_environment_vm" "hive_node" {
-  count = var.node_count
-
-  name        = "${var.cluster_name}-${count.index + 1}"
-  description = "Managed by Terraform"
-  tags        = ["terraform", "spinifex", var.cluster_name]
-  node_name   = var.nodes[count.index % length(var.nodes)].name
-
-  initialization {
-    datastore_id        = "local"
-    vendor_data_file_id = proxmox_virtual_environment_file.cloud_config[count.index].id
-
-    user_account {
-      username = "tf-user"
-      keys     = [trimspace(data.local_file.ssh_public_key.content)]
-    }
-
-    # Management
-    ip_config {
-      ipv4 {
-        address = "dhcp"
-      }
-    }
-
-    # Data/VM
-    ip_config {
-      ipv4 {
-        address = "dhcp"
-      }
-    }
-  }
-
-  cpu {
-    cores = var.cpu_cores
-    type  = "host"
-  }
-
-  memory {
-    dedicated = var.memory_mb
-  }
-
-  agent {
-    enabled = true
-  }
-
-  network_device {
-    bridge = var.nodes[count.index % length(var.nodes)].bridge
-  }
-
-  network_device {
-    bridge = var.nodes[count.index % length(var.nodes)].bridge
-  }
-
-  # Clone from template if the target node has template_vmid set, otherwise boot from cloud image.
-  dynamic "clone" {
-    for_each = var.nodes[count.index % length(var.nodes)].template_vmid != null ? [1] : []
-    content {
-      vm_id = var.nodes[count.index % length(var.nodes)].template_vmid
-      full  = true
-    }
-  }
-
-  dynamic "disk" {
-    for_each = var.nodes[count.index % length(var.nodes)].template_vmid == null ? [1] : []
-    content {
-      file_id      = var.os_image
-      datastore_id = var.nodes[count.index % length(var.nodes)].datastore_id
-      interface    = "virtio0"
-      iothread     = true
-      discard      = "on"
-      size         = var.disk_size_gb
-    }
-  }
-}

scripts/iac/spinifex-test.sh

@@ -1,223 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-TOFU_DIR="$SCRIPT_DIR/proxmox"
-CLUSTERS_DIR="$TOFU_DIR/clusters"
-
-usage() {
-    cat <<'USAGE'
-Usage: spinifex-test.sh <command> <cluster_name> [options]
-
-Commands:
-  up          Provision VMs with OpenTofu
-  configure   Clone repo, build, form cluster, start services
-  test        Run smoke tests against the cluster
-  down        Destroy VMs and clean up state
-  full        up → configure → test → down (preserves test exit code)
-  status      Show cluster health and node IPs
-  ssh <N>     SSH to node N (1-based)
-
-Options:
-  --node-count=N     Number of VMs (default: 3)
-  --memory-mb=N      Memory per VM in MB (default: 16384)
-  --cpu-cores=N      CPU cores per VM (default: 4)
-  --disk-size-gb=N   Disk size per VM in GB (default: 32)
-
-Examples:
-  ./scripts/iac/spinifex-test.sh up spinifex-test1
-  ./scripts/iac/spinifex-test.sh configure spinifex-test1
-  ./scripts/iac/spinifex-test.sh test spinifex-test1
-  ./scripts/iac/spinifex-test.sh status spinifex-test1
-  ./scripts/iac/spinifex-test.sh ssh spinifex-test1 2
-  ./scripts/iac/spinifex-test.sh down spinifex-test1
-  ./scripts/iac/spinifex-test.sh full spinifex-test1 --node-count=3
-USAGE
-    exit 1
-}
-
-[ $# -lt 2 ] && usage
-
-COMMAND="$1"
-CLUSTER_NAME="$2"
-shift 2
-
-# Parse options
-NODE_COUNT=3
-MEMORY_MB=16384
-CPU_CORES=4
-DISK_SIZE_GB=32
-SSH_NODE=""
-
-for arg in "$@"; do
-    case "$arg" in
-        --node-count=*) NODE_COUNT="${arg#*=}" ;;
-        --memory-mb=*)  MEMORY_MB="${arg#*=}" ;;
-        --cpu-cores=*)  CPU_CORES="${arg#*=}" ;;
-        --disk-size-gb=*) DISK_SIZE_GB="${arg#*=}" ;;
-        [0-9]*)         SSH_NODE="$arg" ;;
-        *) echo "Unknown option: $arg"; usage ;;
-    esac
-done
-
-STATE_DIR="$CLUSTERS_DIR/$CLUSTER_NAME"
-STATE_FILE="$STATE_DIR/terraform.tfstate"
-
-tofu_cmd() {
-    tofu -chdir="$TOFU_DIR" "$@"
-}
-
-get_inventory() {
-    tofu_cmd output -json -state="$STATE_FILE" inventory 2>/dev/null
-}
-
-get_node_ip() {
-    local index=$1
-    get_inventory | jq -r ".nodes[$((index - 1))].management"
-}
-
-get_ssh_key() {
-    get_inventory | jq -r ".ssh_key_path"
-}
-
-get_node_count() {
-    get_inventory | jq -r ".node_count"
-}
-
-cmd_up() {
-    echo "==> Provisioning cluster '$CLUSTER_NAME' ($NODE_COUNT nodes, ${MEMORY_MB}MB RAM, ${CPU_CORES} cores)"
-    mkdir -p "$STATE_DIR"
-
-    tofu_cmd init
-
-    tofu_cmd apply \
-        -state="$STATE_FILE" \
-        -var="cluster_name=$CLUSTER_NAME" \
-        -var="node_count=$NODE_COUNT" \
-        -var="memory_mb=$MEMORY_MB" \
-        -var="cpu_cores=$CPU_CORES" \
-        -var="disk_size_gb=$DISK_SIZE_GB" \
-        -auto-approve
-
-    echo ""
-    echo "==> Cluster '$CLUSTER_NAME' provisioned"
-    echo ""
-    get_inventory | jq '.nodes[] | {name, management, data}'
-}
-
-cmd_configure() {
-    echo "==> Configuring cluster '$CLUSTER_NAME'"
-    if [ ! -f "$STATE_FILE" ]; then
-        echo "Error: No state file found at $STATE_FILE. Run 'up' first."
-        exit 1
-    fi
-
-    "$SCRIPT_DIR/configure-cluster.sh" "$CLUSTER_NAME" "$STATE_FILE"
-}
-
-cmd_test() {
-    echo "==> Testing cluster '$CLUSTER_NAME'"
-    if [ ! -f "$STATE_FILE" ]; then
-        echo "Error: No state file found at $STATE_FILE. Run 'up' first."
-        exit 1
-    fi
-
-    "$SCRIPT_DIR/test-cluster.sh" "$CLUSTER_NAME" "$STATE_FILE"
-}
-
-cmd_down() {
-    echo "==> Destroying cluster '$CLUSTER_NAME'"
-    if [ ! -f "$STATE_FILE" ]; then
-        echo "Error: No state file found at $STATE_FILE."
-        exit 1
-    fi
-
-    tofu_cmd destroy \
-        -state="$STATE_FILE" \
-        -var="cluster_name=$CLUSTER_NAME" \
-        -var="node_count=$NODE_COUNT" \
-        -var="memory_mb=$MEMORY_MB" \
-        -var="cpu_cores=$CPU_CORES" \
-        -var="disk_size_gb=$DISK_SIZE_GB" \
-        -auto-approve
-
-    rm -rf "$STATE_DIR"
-    echo "==> Cluster '$CLUSTER_NAME' destroyed"
-}
-
-cmd_full() {
-    cmd_up
-
-    cmd_configure
-
-    local test_exit=0
-    cmd_test || test_exit=$?
-
-    cmd_down
-
-    exit $test_exit
-}
-
-cmd_status() {
-    if [ ! -f "$STATE_FILE" ]; then
-        echo "Error: No state file found at $STATE_FILE."
-        exit 1
-    fi
-
-    local inventory
-    inventory=$(get_inventory)
-    local count
-    count=$(echo "$inventory" | jq -r '.node_count')
-    local ssh_key
-    ssh_key=$(echo "$inventory" | jq -r '.ssh_key_path')
-
-    echo "==> Cluster '$CLUSTER_NAME' status ($count nodes)"
-    echo ""
-
-    for i in $(seq 1 "$count"); do
-        local ip
-        ip=$(echo "$inventory" | jq -r ".nodes[$((i - 1))].management")
-        local data_ip
-        data_ip=$(echo "$inventory" | jq -r ".nodes[$((i - 1))].data")
-        local name
-        name=$(echo "$inventory" | jq -r ".nodes[$((i - 1))].name")
-
-        printf "  %-20s mgmt=%-16s data=%-16s " "$name" "$ip" "$data_ip"
-
-        local health
-        if health=$(curl -sk --connect-timeout 3 "https://$ip:4432/health" 2>/dev/null); then
-            echo "health=$health"
-        else
-            echo "health=unreachable"
-        fi
-    done
-}
-
-cmd_ssh() {
-    if [ -z "$SSH_NODE" ]; then
-        echo "Usage: spinifex-test.sh ssh <cluster_name> <node_number>"
-        exit 1
-    fi
-
-    local ip
-    ip=$(get_node_ip "$SSH_NODE")
-    local ssh_key
-    ssh_key=$(get_ssh_key)
-
-    echo "==> SSH to $CLUSTER_NAME node $SSH_NODE ($ip)"
-    ssh -i "$(eval echo "$ssh_key")" \
-        -o StrictHostKeyChecking=no \
-        -o UserKnownHostsFile=/dev/null \
-        "tf-user@$ip"
-}
-
-case "$COMMAND" in
-    up)        cmd_up ;;
-    configure) cmd_configure ;;
-    test)      cmd_test ;;
-    down)      cmd_down ;;
-    full)      cmd_full ;;
-    status)    cmd_status ;;
-    ssh)       cmd_ssh ;;
-    *)         echo "Unknown command: $COMMAND"; usage ;;
-esac

scripts/iac/test-cluster.sh

@@ -1,71 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-# Run smoke tests against a configured Spinifex cluster.
-# Usage: test-cluster.sh <cluster_name> <state_file>
-
-CLUSTER_NAME="${1:?Usage: test-cluster.sh <cluster_name> <state_file>}"
-STATE_FILE="${2:?Usage: test-cluster.sh <cluster_name> <state_file>}"
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-TOFU_DIR="$SCRIPT_DIR/proxmox"
-
-inventory=$(tofu -chdir="$TOFU_DIR" output -json -state="$STATE_FILE" inventory)
-NODE1_IP=$(echo "$inventory" | jq -r '.nodes[0].management')
-
-AWS_ENDPOINT="https://$NODE1_IP:9999"
-PASS=0
-FAIL=0
-
-run_test() {
-    local name="$1"
-    shift
-    printf "  %-40s " "$name"
-    if output=$("$@" 2>&1); then
-        echo "PASS"
-        PASS=$((PASS + 1))
-    else
-        echo "FAIL"
-        echo "    $output" | head -5
-        FAIL=$((FAIL + 1))
-    fi
-}
-
-echo "==> Smoke tests for cluster '$CLUSTER_NAME'"
-echo "    Endpoint: $AWS_ENDPOINT"
-echo ""
-
-# Wait for gateway to be reachable
-echo "==> Waiting for AWS gateway..."
-attempts=0
-while ! curl -sk --connect-timeout 3 "$AWS_ENDPOINT" >/dev/null 2>&1; do
-    attempts=$((attempts + 1))
-    if [ $attempts -ge 30 ]; then
-        echo "Error: Gateway not reachable at $AWS_ENDPOINT after 150 seconds"
-        exit 1
-    fi
-    sleep 5
-done
-echo "    Gateway reachable"
-echo ""
-
-echo "==> Running tests..."
-
-run_test "describe-regions" \
-    aws --endpoint-url "$AWS_ENDPOINT" --no-verify-ssl --region ap-southeast-2 \
-    ec2 describe-regions
-
-run_test "describe-instance-types" \
-    aws --endpoint-url "$AWS_ENDPOINT" --no-verify-ssl --region ap-southeast-2 \
-    ec2 describe-instance-types --query 'InstanceTypes[0].InstanceType'
-
-run_test "describe-instances" \
-    aws --endpoint-url "$AWS_ENDPOINT" --no-verify-ssl --region ap-southeast-2 \
-    ec2 describe-instances
-
-echo ""
-echo "==> Results: $PASS passed, $FAIL failed"
-
-if [ $FAIL -gt 0 ]; then
-    exit 1
-fi

scripts/images/eks-node/addons/spinifex-noop/0.1.0/noop.yaml

@@ -0,0 +1,20 @@
+# spinifex-noop — managed-addon delivery-transport fixture. Two trivial,
+# dependency-free objects the addon e2e asserts on to prove the stage → fetch →
+# render → auto-deploy → ACTIVE → delete round-trip end-to-end, independent of
+# the CSI or load-balancer-controller bundles. Not a real workload.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: spinifex-noop
+  labels:
+    app.kubernetes.io/managed-by: spinifex-addon
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: spinifex-noop
+  namespace: spinifex-noop
+  labels:
+    app.kubernetes.io/managed-by: spinifex-addon
+data:
+  marker: "spinifex-noop addon delivered"

scripts/images/eks-node/eks-node-role.sh

@@ -10,8 +10,13 @@ set -eu
 # ROLE_FILE so later boots and the etcd-snapshot cron can branch without
 # re-parsing, then removes itself from the runlevel so it runs exactly once.
 #
-#   server → eks-token-webhook, k3s (server), k3s-first-boot (bootstrap publish)
-#   agent  → k3s-agent
+#   server      → eks-token-webhook, k3s (server), k3s-first-boot (bootstrap
+#                 publish), mulga-eks-state-report
+#   server-join → eks-token-webhook, k3s (server, joins the first server's etcd
+#                 quorum), mulga-eks-state-report. NO k3s-first-boot: the first
+#                 server already publishes the cluster-identical bootstrap
+#                 artifacts; a join re-publish only races the bootstrap bus.
+#   agent       → k3s-agent
 #
 # Paths are overridable via env so the selection logic is unit-testable (bats)
 # without root or a real /etc.
@@ -50,10 +55,25 @@ case "${ROLE}" in
         rc-update add k3s default
         rc-update add k3s-first-boot default
         rc-update add mulga-eks-state-report default
+        # Managed-addon delivery runs only on the primary server: it renders
+        # staged addon bundles into the K3s auto-deploy dir, which a single
+        # writer owns. HA multi-server addon delivery is tracked separately
+        # (mulga-siv-231.7); server-join nodes do not run it.
+        rc-update add mulga-eks-addon-sync default
         rc-service eks-token-webhook start
         rc-service k3s start
         rc-service k3s-first-boot start
         rc-service mulga-eks-state-report start
+        rc-service mulga-eks-addon-sync start
+        ;;
+    server-join)
+        log "configuring server-join role"
+        rc-update add eks-token-webhook default
+        rc-update add k3s default
+        rc-update add mulga-eks-state-report default
+        rc-service eks-token-webhook start
+        rc-service k3s start
+        rc-service mulga-eks-state-report start
         ;;
     agent)
         log "configuring agent role"

scripts/images/eks-node/eks-node-role_test.sh

@@ -66,6 +66,20 @@ check "server: starts k3s" "yes" "$(grep -q 'rc-service k3s start' "${CALLS}" &&
 check "server: no k3s-agent" "no" "$(grep -q 'k3s-agent' "${CALLS}" && echo yes || echo no)"
 check "server: self-disables" "yes" "$(grep -q 'rc-update del eks-node-role default' "${CALLS}" && echo yes || echo no)"
 
+# --- Case 1b: explicit server-join role ---
+run_case server_join
+printf 'SPINIFEX_K3S_ROLE=server-join\nEKS_CLUSTER_NAME=alpha\n' > "${EKS_NODE_ENVFILE}"
+invoke
+check "server-join: exit 0" 0 "${rc}"
+check "server-join: role file" "server-join" "$(cat "${ROLE_FILE}")"
+check "server-join: enables webhook" "yes" "$(grep -q 'rc-update add eks-token-webhook default' "${CALLS}" && echo yes || echo no)"
+check "server-join: enables k3s" "yes" "$(grep -q 'rc-update add k3s default' "${CALLS}" && echo yes || echo no)"
+check "server-join: enables state-report" "yes" "$(grep -q 'rc-update add mulga-eks-state-report default' "${CALLS}" && echo yes || echo no)"
+# Join servers must NOT re-publish bootstrap; the first server already did.
+check "server-join: no k3s-first-boot" "no" "$(grep -q 'k3s-first-boot' "${CALLS}" && echo yes || echo no)"
+check "server-join: no k3s-agent" "no" "$(grep -q 'k3s-agent' "${CALLS}" && echo yes || echo no)"
+check "server-join: self-disables" "yes" "$(grep -q 'rc-update del eks-node-role default' "${CALLS}" && echo yes || echo no)"
+
 # --- Case 2: explicit agent role ---
 run_case agent_explicit
 printf 'SPINIFEX_K3S_ROLE=agent\n' > "${EKS_NODE_ENVFILE}"

scripts/images/eks-node/manifest.conf

@@ -18,9 +18,9 @@ APK_PACKAGES="ca-certificates curl jq iptables ip6tables conntrack-tools cgroup-
 # eks-gateway-publish relays the bootstrap envelopes + state reports to the AWS
 # gateway over SigV4 HTTPS (server role only). The k3s binary is downloaded +
 # SHA256-verified by setup.sh (network in chroot).
-BUILD_COMMANDS="CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -o bin/eks-token-webhook ./cmd/eks-token-webhook && CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -o bin/eks-gateway-publish ./cmd/eks-gateway-publish"
+BUILD_COMMANDS="CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -o bin/eks-token-webhook ./cmd/eks-token-webhook && CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -o bin/eks-gateway-publish ./cmd/eks-gateway-publish && CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -o bin/eks-gateway-fetch ./cmd/eks-gateway-fetch"
 
-INSTALL_BINARIES="bin/eks-token-webhook:/usr/local/bin/eks-token-webhook bin/eks-gateway-publish:/usr/local/bin/eks-gateway-publish"
+INSTALL_BINARIES="bin/eks-token-webhook:/usr/local/bin/eks-token-webhook bin/eks-gateway-publish:/usr/local/bin/eks-gateway-publish bin/eks-gateway-fetch:/usr/local/bin/eks-gateway-fetch"
 
 # All role services ship; none but the selector are enabled at bake time.
 # eks-node-role picks server vs agent at first boot and enables the rest.
@@ -34,6 +34,9 @@ scripts/images/eks-node/k3s-first-boot.initd:/etc/init.d/k3s-first-boot \
 scripts/images/eks-node/k3s-first-boot.sh:/usr/local/sbin/k3s-first-boot \
 scripts/images/eks-node/mulga-eks-state-report.initd:/etc/init.d/mulga-eks-state-report \
 scripts/images/eks-node/mulga-eks-state-report.sh:/usr/local/sbin/mulga-eks-state-report \
+scripts/images/eks-node/mulga-eks-addon-sync.initd:/etc/init.d/mulga-eks-addon-sync \
+scripts/images/eks-node/mulga-eks-addon-sync.sh:/usr/local/sbin/mulga-eks-addon-sync \
+scripts/images/eks-node/addons/spinifex-noop/0.1.0/noop.yaml:/usr/share/spinifex-eks/addons/spinifex-noop/0.1.0/noop.yaml \
 scripts/images/eks-node/mulga-eks-etcd-snapshot.sh:/etc/periodic/daily/mulga-eks-etcd-snapshot"
 
 # Only crond (etcd snapshot) + the role selector at bake. The selector adds the

scripts/images/eks-node/mulga-eks-addon-sync.initd

@@ -0,0 +1,30 @@
+#!/sbin/openrc-run
+# mulga-eks-addon-sync — server-role service that renders Spinifex managed-addon
+# bundles into the K3s auto-deploy dir and reports per-addon delivery status to
+# the spinifex control plane every ADDON_SYNC_INTERVAL seconds. The host-side
+# daemon stages each addon in KV; the VM pulls the staged set through the AWS
+# gateway (it cannot read NATS/KV directly) and renders the baked bundles.
+# Enabled on the primary server role by eks-node-role.sh.
+
+description="Mulga EKS managed-addon delivery agent"
+command="/usr/local/sbin/mulga-eks-addon-sync"
+command_background="yes"
+pidfile="/run/mulga-eks-addon-sync.pid"
+output_log="/var/log/mulga-eks-addon-sync.log"
+error_log="/var/log/mulga-eks-addon-sync.log"
+
+# Cadence the sync loop sleeps between pulls. 30s keeps CreateAddon→ACTIVE
+# latency low without hammering the gateway.
+export ADDON_SYNC_INTERVAL="${ADDON_SYNC_INTERVAL:-30}"
+
+depend() {
+    need net
+    after k3s
+}
+
+start_pre() {
+    if [ ! -f /etc/spinifex-eks/first-boot.env ]; then
+        eerror "/etc/spinifex-eks/first-boot.env not present — cloud-init did not seed gateway creds"
+        return 1
+    fi
+}

scripts/images/eks-node/mulga-eks-addon-sync.sh

@@ -0,0 +1,199 @@
+#!/bin/sh
+set -eu
+
+# mulga-eks-addon-sync — renders Spinifex managed-addon bundles into the K3s
+# auto-deploy dir and reports per-addon delivery status back to the spinifex
+# control plane. The host-side daemon stages each CreateAddon as a manifest
+# descriptor in KV; the VM cannot read KV directly, so this agent pulls the
+# staged set through the AWS gateway (eks-gateway-fetch, SigV4) and renders the
+# baked bundle for each one. Status flows back via eks-gateway-publish on the
+# "addon" channel, where the reconciler CASes the AddonRecord CREATING→ACTIVE.
+#
+# Pull   (host→VM): GET /clusters/{cluster}/internal-addons/{accountId}
+# Report (VM→host): eks.addon.{accountID}.{clusterName}.status
+#
+# Runs server-role only (it owns the single auto-deploy dir); enabled by
+# eks-node-role.sh. Loops every ADDON_SYNC_INTERVAL seconds in service mode,
+# one-shot otherwise.
+
+ENVFILE=/etc/spinifex-eks/first-boot.env
+[ -f "${ENVFILE}" ] || { logger -t mulga-eks-addon-sync "${ENVFILE} missing — exiting"; exit 0; }
+# set -a so the sourced creds are exported to the eks-gateway-{fetch,publish}
+# children, which read EKS_ACCOUNT_ID etc. from their environment.
+set -a
+# shellcheck disable=SC1090
+. "${ENVFILE}"
+set +a
+
+: "${EKS_GATEWAY_URL:?}"
+: "${EKS_ACCESS_KEY:?}"
+: "${EKS_SECRET_KEY:?}"
+: "${EKS_ACCOUNT_ID:?}"
+: "${EKS_CLUSTER_NAME:?}"
+
+KUBECONFIG=${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml}
+export KUBECONFIG
+
+# Baked bundles live at BAKED_DIR/<addon>/<version>/*.yaml (air-gapped, shipped
+# in the AMI). Rendered output lands in the K3s auto-deploy dir under a stable
+# per-addon filename so GC can find orphans this agent owns.
+BAKED_DIR=/usr/share/spinifex-eks/addons
+DEPLOY_DIR=/var/lib/rancher/k3s/server/manifests
+RENDER_PREFIX=spinifex-addon-
+
+# log mirrors to syslog (on-VM /var/log) and, best-effort, to the serial
+# console — the host captures ttyS0 to a per-instance log, so this is the only
+# delivery diagnostics channel reachable from the host (the VM has no
+# host-routable address). Keep messages terse; one line per sync outcome.
+log() {
+    logger -t mulga-eks-addon-sync "$*"
+    echo "[mulga-eks-addon-sync] $*" > /dev/console 2>/dev/null || true
+}
+
+# report PHASE ADDON VERSION [MESSAGE] — publish one addon status report.
+report() {
+    _phase=$1; _addon=$2; _version=$3; _msg=${4:-}
+    printf '{"addon":"%s","version":"%s","phase":"%s","message":"%s","ts":%s}' \
+        "${_addon}" "${_version}" "${_phase}" "${_msg}" "$(date +%s)" \
+        | eks-gateway-publish -channel addon 2>&1 | logger -t mulga-eks-addon-sync || true
+}
+
+# render_addon ADDON VERSION ROLE_ARN — render the baked bundle into the
+# auto-deploy dir, substituting the IRSA role ARN. Returns non-zero (and reports
+# failed) when the baked bundle for ADDON/VERSION is absent.
+render_addon() {
+    _addon=$1; _version=$2; _role=$3
+    _src="${BAKED_DIR}/${_addon}/${_version}"
+    _dst="${DEPLOY_DIR}/${RENDER_PREFIX}${_addon}.yaml"
+
+    if [ ! -d "${_src}" ]; then
+        log "no baked bundle for ${_addon}/${_version} (looked in ${_src})"
+        report failed "${_addon}" "${_version}" "no baked bundle for version ${_version}"
+        return 1
+    fi
+
+    _tmp="${_dst}.tmp.$$"
+    : > "${_tmp}"
+    for _f in "${_src}"/*.yaml; do
+        [ -e "${_f}" ] || continue
+        sed -e "s|{{SERVICE_ACCOUNT_ROLE_ARN}}|${_role}|g" "${_f}" >> "${_tmp}"
+        printf '\n---\n' >> "${_tmp}"
+    done
+    # Idempotent: only replace the auto-deploy file when the rendered content
+    # actually changed. An unconditional mv bumps the file mtime every tick,
+    # which K3s' auto-deploy controller treats as a change and re-applies —
+    # clearing the Addon CR's .status.gvks each time, so the same-tick readiness
+    # check never observes a settled apply and the add-on stays CREATING forever.
+    if [ -e "${_dst}" ] && cmp -s "${_tmp}" "${_dst}"; then
+        rm -f "${_tmp}"
+        return 0
+    fi
+    mv -f "${_tmp}" "${_dst}"
+    return 0
+}
+
+# addon_gvks ADDON — echoes the GVKs K3s' auto-deploy controller applied for the
+# rendered manifest, empty until it has applied the objects. A non-empty value
+# means delivery succeeded. Addon-agnostic — confirms delivery, not deep per-pod
+# health (a later hardening item).
+#
+# K3s (v1.32.x) records the applied GVKs in the addon.k3s.cattle.io/gvks
+# *annotation* (e.g. "/v1, Kind=Namespace;/v1, Kind=ConfigMap"); the Addon CR
+# carries no .status subresource. The CR is matched by .spec.source (the
+# absolute path of the rendered file) rather than a guessed CR name, since K3s
+# derives the name from the path with its own transform.
+addon_gvks() {
+    _want="${DEPLOY_DIR}/${RENDER_PREFIX}$1.yaml"
+    kubectl -n kube-system get addon -o json 2>/dev/null \
+        | jq -r --arg src "${_want}" \
+            '.items[] | select(.spec.source==$src)
+             | .metadata.annotations["addon.k3s.cattle.io/gvks"] // ""' 2>/dev/null \
+        | grep -v '^$' | head -1 || true
+}
+
+# diag_addon ADDON — one-shot console dump of the live K3s state for a stuck
+# add-on, so a delivery that never reaches ready is diagnosable from the
+# host-captured serial console (the VM is otherwise unreachable). Fires once per
+# process via a sentinel to avoid spamming every tick.
+diag_addon() {
+    [ -f /tmp/.addon-diag-"$1" ] && return 0
+    : > /tmp/.addon-diag-"$1"
+    {
+        echo "----- addon-diag ${1} -----"
+        echo "[deploy dir]"; ls -l "${DEPLOY_DIR}" 2>&1
+        echo "[rendered file head]"; head -40 "${DEPLOY_DIR}/${RENDER_PREFIX}${1}.yaml" 2>&1
+        echo "[all addon CRs]"; kubectl get addon -A 2>&1
+        echo "[addon CR ${RENDER_PREFIX}${1} -o yaml]"; kubectl -n kube-system get addon "${RENDER_PREFIX}${1}" -o yaml 2>&1
+        echo "[namespace ${1}]"; kubectl get ns "${1}" 2>&1
+        echo "[configmaps in ${1}]"; kubectl -n "${1}" get cm 2>&1
+        echo "----- end addon-diag ${1} -----"
+    } > /dev/console 2>&1 || true
+}
+
+sync_once() {
+    _staged=$(mktemp)
+    _ferr=$(mktemp)
+    if ! eks-gateway-fetch -resource addons > "${_staged}" 2>"${_ferr}"; then
+        log "fetch staged addons failed — keeping current state: $(tr '\n' ' ' < "${_ferr}")"
+        rm -f "${_staged}" "${_ferr}"
+        return 0
+    fi
+    rm -f "${_ferr}"
+
+    # Names currently staged, for the GC pass below.
+    _names=$(cut -f1 "${_staged}")
+    _count=$(grep -c . "${_staged}" 2>/dev/null || echo 0)
+    log "fetch ok: ${_count} staged addon(s)"
+
+    # Render/refresh each staged addon and report its phase.
+    while IFS=$(printf '\t') read -r _addon _version _role _cfg_b64; do
+        [ -n "${_addon}" ] || continue
+        : "${_cfg_b64:-}"  # config values reserved for per-addon templating (follow-up)
+        if render_addon "${_addon}" "${_version}" "${_role}"; then
+            _gvks=$(addon_gvks "${_addon}")
+            if [ -n "${_gvks}" ]; then
+                log "addon ${_addon}/${_version}: applied + ready (gvks=${_gvks})"
+                report ready "${_addon}" "${_version}"
+            else
+                log "addon ${_addon}/${_version}: applied, not ready yet (no K3s Addon CR for ${RENDER_PREFIX}${_addon}.yaml with populated .status.gvks)"
+                diag_addon "${_addon}"
+                report applied "${_addon}" "${_version}"
+            fi
+        fi
+    done < "${_staged}"
+    rm -f "${_staged}"
+
+    # GC: remove rendered manifests this agent owns whose addon is no longer
+    # staged (DeleteAddon removed its descriptor). K3s does NOT delete the applied
+    # objects when the manifest file disappears, so delete them explicitly.
+    for _file in "${DEPLOY_DIR}/${RENDER_PREFIX}"*.yaml; do
+        [ -e "${_file}" ] || continue
+        _base=$(basename "${_file}")
+        _name=${_base#"${RENDER_PREFIX}"}
+        _name=${_name%.yaml}
+        if ! printf '%s\n' "${_names}" | grep -qx "${_name}"; then
+            log "unstaging addon ${_name} (no longer staged)"
+            # Copy the manifest aside, drop the original so K3s stops tracking it
+            # (and never re-applies), then delete the objects it rendered.
+            # --wait=false so namespace finalizers don't stall the sync loop.
+            _gc=$(mktemp)
+            cp "${_file}" "${_gc}"
+            rm -f "${_file}"
+            kubectl delete -f "${_gc}" --ignore-not-found --wait=false 2>&1 \
+                | logger -t mulga-eks-addon-sync || true
+            rm -f "${_gc}"
+        fi
+    done
+}
+
+mkdir -p "${DEPLOY_DIR}"
+
+if [ -n "${ADDON_SYNC_INTERVAL:-}" ] && [ "${ADDON_SYNC_INTERVAL}" -gt 0 ] 2>/dev/null; then
+    log "starting: cluster=${EKS_CLUSTER_NAME} account=${EKS_ACCOUNT_ID} interval=${ADDON_SYNC_INTERVAL}s gateway=${EKS_GATEWAY_URL}"
+    while true; do
+        sync_once
+        sleep "${ADDON_SYNC_INTERVAL}"
+    done
+else
+    sync_once
+fi

scripts/images/eks-node/setup.sh

@@ -43,9 +43,10 @@ ln -sf /usr/local/bin/k3s /usr/local/bin/ctr
 # Init scripts ship as 0644 from INSTALL_FILES; OpenRC requires 0755. Every
 # role's services are baked; the selector enables the right ones at first boot.
 chmod 0755 /etc/init.d/eks-node-role /etc/init.d/k3s /etc/init.d/k3s-agent \
-    /etc/init.d/eks-token-webhook /etc/init.d/k3s-first-boot /etc/init.d/mulga-eks-state-report
+    /etc/init.d/eks-token-webhook /etc/init.d/k3s-first-boot /etc/init.d/mulga-eks-state-report \
+    /etc/init.d/mulga-eks-addon-sync
 chmod 0755 /usr/local/sbin/eks-node-role /usr/local/sbin/k3s-first-boot \
-    /usr/local/sbin/mulga-eks-state-report
+    /usr/local/sbin/mulga-eks-state-report /usr/local/sbin/mulga-eks-addon-sync
 chmod 0755 /etc/periodic/daily/mulga-eks-etcd-snapshot
 
 # K3s server config — skeleton; cloud-init / first-boot fills in the

scripts/publish-system-image.sh

@@ -89,7 +89,9 @@ fi
 
 if [[ "$DO_BUILD" == true ]]; then
     echo "=== Building ${IMAGE_NAME} ==="
-    "${SCRIPT_DIR}/build-system-image.sh" "$MANIFEST"
+    # --no-cache: publishing must never reuse a stale raw left by a prior or
+    # concurrent build on a persistent runner.
+    "${SCRIPT_DIR}/build-system-image.sh" "$MANIFEST" --no-cache
 fi
 
 if [[ ! -f "$RAW_IMAGE" ]]; then

scripts/run-instance.sh

@@ -1,55 +0,0 @@
-#!/bin/sh
-
-# Query images, get Ubuntu image
-export AWS_PROFILE=spinifex
-
-# Detect architecture
-ARCH=$(uname -m)
-if [ "$ARCH" = "x86_64" ]; then
-    IMAGE_NAME="ami-ubuntu-26.04-x86_64"
-elif [ "$ARCH" = "aarch64" ] || [ "$ARCH" = "arm64" ]; then
-    IMAGE_NAME="ami-ubuntu-26.04-arm64"
-else
-    echo "Warning: Unknown architecture $ARCH, defaulting to x86_64"
-    IMAGE_NAME="ami-ubuntu-26.04-x86_64"
-fi
-
-# Query available instance types from Spinifex and pick the first micro type
-INSTANCE_TYPE=$(aws ec2 describe-instance-types --query "InstanceTypes[?contains(InstanceType, '.micro')].InstanceType | [0]" --output text)
-if [ -z "$INSTANCE_TYPE" ] || [ "$INSTANCE_TYPE" = "None" ]; then
-    echo "Error: Could not find available micro instance type"
-    echo "Available instance types:"
-    aws ec2 describe-instance-types --query "InstanceTypes[*].InstanceType" --output table
-    exit 1
-fi
-echo "Using instance type: $INSTANCE_TYPE"
-
-echo "Detected architecture: $ARCH"
-echo "Looking for image with Name: $IMAGE_NAME"
-
-# Query images and extract ImageId matching the IMAGE_NAME
-SPINIFEX_AMI=$(aws ec2 describe-images --query "Images[?Name=='$IMAGE_NAME'].ImageId" --output text)
-
-if [ -z "$SPINIFEX_AMI" ]; then
-    echo "Error: Could not find image with Name '$IMAGE_NAME'"
-    echo "Available images:"
-    aws ec2 describe-images --query "Images[*].[Name,ImageId]" --output table
-    exit 1
-fi
-
-export SPINIFEX_AMI=$SPINIFEX_AMI
-
-echo "Found ImageId: $SPINIFEX_AMI"
-export SPINIFEX_AMI
-
-echo "Launching instance"
-
-# Launch instance
-aws ec2 run-instances \
-  --image-id "$SPINIFEX_AMI" \
-  --instance-type "$INSTANCE_TYPE" \
-  --key-name spinifex-key \
-  --security-group-ids sg-0123456789abcdef0 \
-  --subnet-id subnet-6e7f829e \
-  --count 1
-

scripts/setup.sh

@@ -416,6 +416,14 @@ TMPEOF
     $SUDO chown "spinifex-gw:$SPINIFEX_GROUP" /etc/spinifex/awsgw
     $SUDO chmod 0750 /etc/spinifex/awsgw
 
+    # Viperblock's at-rest encryption key dir. 0750 group-traversable; the key
+    # itself is set to root:spinifex 0640 by SetServiceOwnership because both
+    # viperblockd (spinifex-viperblock) and the awsgw handlers (spinifex-gw)
+    # load it.
+    $SUDO mkdir -p /etc/spinifex/viperblock
+    $SUDO chown "spinifex-viperblock:$SPINIFEX_GROUP" /etc/spinifex/viperblock
+    $SUDO chmod 0750 /etc/spinifex/viperblock
+
     # Per-service data directories
     $SUDO mkdir -p /var/lib/spinifex/nats
     $SUDO chown "spinifex-nats:$SPINIFEX_GROUP" /var/lib/spinifex/nats
@@ -599,6 +607,20 @@ install_systemd() {
         info "  /etc/systemd/system/$(basename "$unit")"
     done
 
+    # Reserve RAM + CPU priority for system.slice (sshd, journald, the operator)
+    # so a maxed spinifex.slice cannot starve them — the "stay sshable" guarantee.
+    # Generated here rather than shipped as a staged file because the packaging
+    # globs flatten the systemd/ dir and would skip a nested drop-in directory;
+    # this mirrors the sshd-keygen drop-in pattern in build-rootfs.sh. MemoryMin
+    # is a guaranteed-unreclaimable floor, not a cap.
+    $SUDO mkdir -p /etc/systemd/system/system.slice.d
+    $SUDO tee /etc/systemd/system/system.slice.d/spinifex-reserve.conf > /dev/null << 'EOF'
+[Slice]
+MemoryMin=1G
+CPUWeight=300
+EOF
+    info "  /etc/systemd/system/system.slice.d/spinifex-reserve.conf"
+
     # daemon-reload / enable require a running systemd — skip inside the ISO
     # chroot. Unit files are still dropped into place; firstboot enables them.
     if [ "${ISO_BUILD:-0}" = "1" ]; then
@@ -704,6 +726,48 @@ print_summary() {
     echo ""
 }
 
+# --- Configure host swap (mulga-siv-251) ---
+# Provisions an 8G swapfile and lowers vm.swappiness so spinifex.slice
+# (MemorySwapMax=100%) has a backing device for graceful degradation under
+# memory pressure. Reverses the historical swap=0 assumption. Idempotent.
+setup_swap() {
+    stage "configuring host swap"
+
+    # ISO build runs in a chroot — cannot swapon, and baking an 8G file into the
+    # rootfs bloats the image. ISO hosts provision swap at firstboot instead.
+    if [ "${ISO_BUILD:-0}" = "1" ]; then
+        info "Swap setup skipped (ISO_BUILD=1; firstboot provisions swap)"
+        return
+    fi
+
+    local swapfile=/swapfile size_mb=8192
+
+    # Swap is a safety buffer, not a routine path: reclaim page cache first.
+    $SUDO tee /etc/sysctl.d/99-spinifex-swap.conf > /dev/null << 'EOF'
+# Spinifex: minimise swapping; swap backs spinifex.slice graceful degradation.
+vm.swappiness = 10
+EOF
+    $SUDO chmod 0644 /etc/sysctl.d/99-spinifex-swap.conf
+    $SUDO sysctl -q -w vm.swappiness=10 || true
+
+    if swapon --show=NAME --noheadings 2>/dev/null | grep -qx "$swapfile"; then
+        info "Swap already active ($swapfile)"
+        return
+    fi
+
+    if [ ! -f "$swapfile" ]; then
+        info "Creating ${size_mb}MiB $swapfile"
+        $SUDO fallocate -l "${size_mb}M" "$swapfile" 2>/dev/null \
+            || $SUDO dd if=/dev/zero of="$swapfile" bs=1M count="$size_mb" status=none
+        $SUDO chmod 0600 "$swapfile"
+        $SUDO mkswap "$swapfile" > /dev/null
+    fi
+    $SUDO swapon "$swapfile"
+    grep -q "^$swapfile " /etc/fstab 2>/dev/null \
+        || echo "$swapfile none swap sw 0 0" | $SUDO tee -a /etc/fstab > /dev/null
+    info "Swap enabled: $swapfile (${size_mb}MiB), vm.swappiness=10"
+}
+
 # --- Main ---
 main() {
     info "Spinifex installer"
@@ -739,6 +803,7 @@ main() {
     stage_enabled systemd    && install_systemd
     stage_enabled logrotate  && install_logrotate
     stage_enabled udev       && install_udev
+    stage_enabled swap       && setup_swap
 
     # Migrations are only safe on a live system (need a running NATS and a
     # persisted config file). Skip under ISO_BUILD and under any explicit

scripts/start-dev.sh

@@ -1,514 +0,0 @@
-#!/bin/bash
-
-# Start Spinifex development environment
-# This script starts all required services in the correct order using Spinifex service commands
-# Usage: ./scripts/start-dev.sh [--build] [data-dir]
-#   --build:  Rebuild all binaries before starting (default: skip build)
-#   data-dir: Optional data directory path (default: ~/spinifex)
-#
-# Environment variables:
-#   UI=false              Skip starting Spinifex UI (e.g., UI=false ./scripts/start-dev.sh)
-
-set -euo pipefail
-
-echo "WARNING: start-dev.sh is deprecated for local development." >&2
-echo "  Use 'make deploy' for daily iteration or 'scripts/dev-install.sh' for first-time setup." >&2
-echo "  start-dev.sh is retained only for pseudo-multinode CI (e2e_pseudo_multi)." >&2
-echo "" >&2
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
-MULGA_ROOT="$(cd "$PROJECT_ROOT/.." && pwd)"
-
-# Parse arguments
-BUILD=false
-DATA_DIR="$HOME/spinifex"
-for arg in "$@"; do
-    case "$arg" in
-        --build) BUILD=true ;;
-        *) DATA_DIR="$arg" ;;
-    esac
-done
-
-# Configuration paths
-# Use CONFIG_DIR environment variable if set, otherwise derive from DATA_DIR
-CONFIG_DIR="${CONFIG_DIR:-$DATA_DIR/config}"
-echo "Using data directory: $DATA_DIR"
-echo "Using configuration directory: $CONFIG_DIR"
-LOGS_DIR="$DATA_DIR/logs"
-WAL_DIR="$DATA_DIR/spinifex"
-
-echo "⚠️  Dev mode: all services run as $(whoami) without privilege separation"
-echo ""
-echo "🚀 Starting Spinifex development environment..."
-echo "Project root: $PROJECT_ROOT"
-echo "Data directory: $DATA_DIR"
-
-# Parse services from spinifex.toml — defaults to all if not set
-parse_services() {
-    local config_file="$CONFIG_DIR/spinifex.toml"
-    if [ -f "$config_file" ]; then
-        local svc_line=$(grep -m1 '^services' "$config_file" | sed 's/.*\[//;s/\].*//;s/"//g;s/,/ /g')
-        if [ -n "$svc_line" ]; then
-            echo "$svc_line"
-            return
-        fi
-    fi
-    echo "nats predastore viperblock daemon awsgw vpcd ui"
-}
-
-SERVICES=$(parse_services)
-has_service() {
-    local svc="$1"
-    echo "$SERVICES" | grep -qw "$svc"
-}
-echo "Services: $SERVICES"
-
-# Detect multi-node cluster from config
-is_multinode() {
-    local config_file="$CONFIG_DIR/spinifex.toml"
-    if [ -f "$config_file" ]; then
-        local node_count=$(grep -c '^\[nodes\.' "$config_file")
-        [ "$node_count" -gt 1 ]
-    else
-        return 1
-    fi
-}
-
-# Confirm configuration directory exists
-if [ ! -d "$CONFIG_DIR" ]; then
-    echo "⚠️  Configuration directory $CONFIG_DIR does not exist."
-    echo "Please init the spinifex environment using the CLI."
-    echo "spx admin init"
-    exit 1
-fi
-
-
-if [ -d "/mnt/ramdisk" ]; then
-
-    # Check if /mnt/ramdisk is mounted, if not mount it as tmpfs
-    if ! mountpoint -q /mnt/ramdisk; then
-        echo "💾 Mounting /mnt/ramdisk as tmpfs"
-        sudo mount -t tmpfs -o size=8G tmpfs /mnt/ramdisk/
-    fi
-
-    # If /mnt/ramdisk is mounted, use it for the WAL directory (for development)
-    if mountpoint -q "/mnt/ramdisk"; then
-        WAL_DIR="/mnt/ramdisk/"
-    fi
-
-else
-    echo "⚠️  /mnt/ramdisk not available, using $DATA_DIR/viperblock"
-fi
-
-# Change to project root for all commands
-cd "$PROJECT_ROOT"
-
-# Function to start service in background
-start_service() {
-    local name="$1"
-    local command="$2"
-    local pidfile="$LOGS_DIR/$name.pid"
-    local logfile="$LOGS_DIR/$name.log"
-
-    echo "📡 Starting $name..."
-    echo "   Command: $command"
-
-    # Start service and capture PID
-    nohup $command > "$logfile" 2>&1 &
-    local pid=$!
-    echo $pid > "$pidfile"
-
-    echo "   PID: $pid, Log: $logfile"
-
-    # Brief pause to let service start, then verify it's still alive
-    sleep 1
-    if ! kill -0 "$pid" 2>/dev/null; then
-        echo "   ❌ $name failed to start (exited immediately)"
-        echo "   Check log: $logfile"
-        tail -5 "$logfile" 2>/dev/null || true
-        exit 1
-    fi
-}
-
-# Function to start service in foreground (for final daemon)
-start_service_foreground() {
-    local name="$1"
-    local command="$2"
-
-    echo "📡 Starting $name in foreground..."
-    echo "   Command: $command"
-
-    # Setup signal handler to stop background services when daemon stops
-    trap 'echo ""; echo "🛑 Stopping all services..."; ./scripts/stop-dev.sh; exit 0' INT TERM
-
-    # Start the service in foreground
-    $command
-}
-
-# Function to set OOM score for a service (Linux only)
-# Protects infrastructure services from OOM killer (-500 = less likely to be killed)
-set_oom_score() {
-    local name="$1"
-    local score="$2"
-    local pid_file="$LOGS_DIR/${name}.pid"
-    if [ -f "$pid_file" ]; then
-        local pid=$(cat "$pid_file")
-        if [ -d "/proc/$pid" ]; then
-            echo "$score" > "/proc/$pid/oom_score_adj" 2>/dev/null && \
-                echo "  OOM score for $name (PID $pid): $score" || \
-                echo "  Warning: Could not set OOM score for $name"
-        fi
-    fi
-}
-
-# Function to check if service is responsive
-check_service() {
-    local name="$1"
-    local host="$2"
-    local port="$3"
-    local max_attempts="${4:-30}"
-    local attempt=1
-
-    echo "🔍 Checking $name connectivity on $host:$port..."
-
-    while [ $attempt -le $max_attempts ]; do
-        if nc -z "$host" "$port" 2>/dev/null; then
-            echo "   ✅ $name is responding on $host:$port"
-            return 0
-        fi
-        echo "   ⏳ Attempt $attempt/$max_attempts - waiting for $name..."
-        sleep 2
-        ((attempt++))
-    done
-
-    echo "   ❌ $name failed to start on $host:$port after $max_attempts attempts"
-    exit 1
-}
-
-# Pre-flight, compile latest (only with --build flag)
-if [ "$BUILD" = "true" ]; then
-    echo "✈️  Pre-flight, compiling latest..."
-
-    echo "   Building spinifex..."
-    make build
-
-    echo "   Building predastore..."
-    cd "$MULGA_ROOT/predastore" && make build
-    cd "$PROJECT_ROOT"
-
-    echo "   Building viperblock (nbdkit plugin)..."
-    cd "$MULGA_ROOT/viperblock" && make build
-    cd "$PROJECT_ROOT"
-
-    echo "   ✅ Build complete"
-else
-    echo "✈️  Skipping build (pass --build to rebuild)"
-fi
-
-# 0️⃣ Start OVN networking (required for VPC)
-echo ""
-echo "0️⃣  Starting OVN networking..."
-
-if ! pidof systemd >/dev/null 2>&1; then
-    echo "   ⚠️  No systemd (container?) — skipping OVN lifecycle management"
-elif ! command -v ovs-vsctl >/dev/null 2>&1; then
-    echo "   ❌ OVS not installed"
-    echo "   Run: make quickinstall && ./scripts/setup-ovn.sh --management"
-    exit 1
-else
-    # Start OVS + OVN services (may already be running)
-    sudo systemctl start openvswitch-switch
-    echo "   ✅ openvswitch-switch started"
-
-    # Start ovn-central if this node has it installed and enabled
-    if systemctl list-unit-files ovn-central.service >/dev/null 2>&1 && systemctl is-enabled ovn-central >/dev/null 2>&1; then
-        sudo systemctl start ovn-central
-        echo "   ✅ ovn-central started"
-    fi
-
-    sudo systemctl start ovn-controller
-    echo "   ✅ ovn-controller started"
-
-    # Verify br-int exists (created by setup-ovn.sh)
-    if ! sudo ovs-vsctl br-exists br-int 2>/dev/null; then
-        echo "   ❌ br-int not found — run ./scripts/setup-ovn.sh --management first"
-        exit 1
-    fi
-    echo "   ✅ br-int exists"
-fi
-
-# 1️⃣ Start NATS server
-echo ""
-if has_service "nats"; then
-    echo "1️⃣  Starting NATS server..."
-
-    if [ -f "$CONFIG_DIR/nats/nats.conf" ]; then
-        export SPINIFEX_CONFIG_PATH=$CONFIG_DIR/nats/nats.conf
-        echo " Using NATS config file: $CONFIG_DIR/nats/nats.conf"
-    else
-        echo " ⚠️ NATS config file not found at $CONFIG_DIR/nats/nats.conf, using defaults"
-        export SPINIFEX_NATS_HOST=0.0.0.0
-        export SPINIFEX_NATS_PORT=4222
-        export SPINIFEX_NATS_DATA_DIR=$DATA_DIR/nats/
-        export SPINIFEX_NATS_JETSTREAM=false
-    fi
-
-    # Extract NATS listen address from config (multi-node binds to specific IP, not 127.0.0.1)
-    NATS_CHECK_HOST="127.0.0.1"
-    if [ -f "$CONFIG_DIR/nats/nats.conf" ]; then
-        NATS_LISTEN_IP=$(grep -oP '^listen:\s*\K[^:]+' "$CONFIG_DIR/nats/nats.conf" 2>/dev/null || true)
-        if [ -n "$NATS_LISTEN_IP" ] && [ "$NATS_LISTEN_IP" != "0.0.0.0" ]; then
-            NATS_CHECK_HOST="$NATS_LISTEN_IP"
-        fi
-    fi
-
-    NATS_CMD="./bin/spx service nats start"
-    start_service "nats" "$NATS_CMD"
-    set_oom_score "nats" "-500"
-    check_service "NATS" "$NATS_CHECK_HOST" "4222"
-
-    # Wait for NATS JetStream to be ready before Predastore starts.
-    # Predastore lazily opens IAM KV buckets — it starts with config-only auth
-    # and activates IAM auth once the spinifex daemon creates the KV buckets.
-    # Wait for NATS JetStream to be ready before Predastore starts.
-    # Predastore lazily opens IAM KV buckets — it starts with config-only auth
-    # and activates IAM auth once the spinifex daemon creates the KV buckets.
-    # Use the monitoring HTTP endpoint (plaintext nc probe doesn't work with TLS).
-    # Monitoring is only on the primary node; secondary nodes skip this check.
-    NATS_HAS_MONITORING=false
-    if [ -f "$CONFIG_DIR/nats/nats.conf" ] && grep -q '^http:' "$CONFIG_DIR/nats/nats.conf"; then
-        NATS_HAS_MONITORING=true
-    fi
-
-    if [ "$NATS_HAS_MONITORING" = "true" ]; then
-        echo "🔍 Waiting for NATS JetStream..."
-        NATS_JS_READY=false
-        for i in $(seq 1 15); do
-            if curl -sf "http://127.0.0.1:8222/jsz" > /dev/null 2>&1; then
-                echo "   ✅ NATS JetStream is ready"
-                NATS_JS_READY=true
-                break
-            fi
-            sleep 1
-        done
-        if [ "$NATS_JS_READY" = "false" ]; then
-            echo "   ⚠️  NATS JetStream not confirmed ready — Predastore may fall back to config-only auth"
-        fi
-    else
-        echo "🔍 NATS monitoring not configured — skipping JetStream readiness check"
-        sleep 2
-    fi
-else
-    echo "1️⃣  Skipping NATS (not a local service)"
-fi
-
-# 2️⃣ Start Predastore
-echo ""
-if has_service "predastore"; then
-    echo "2️⃣  Starting Predastore..."
-    unset SPINIFEX_CONFIG_PATH
-
-    export SPINIFEX_PREDASTORE_BASE_PATH=$DATA_DIR/predastore/
-    export SPINIFEX_PREDASTORE_CONFIG_PATH=$CONFIG_DIR/predastore/predastore.toml
-    export SPINIFEX_PREDASTORE_TLS_CERT=$CONFIG_DIR/server.pem
-    export SPINIFEX_PREDASTORE_TLS_KEY=$CONFIG_DIR/server.key
-
-    # Per-node predastore encryption key. Self-heal for devs who skipped
-    # `spx admin init`: generate with umask 0177 so the file is 0600 from
-    # creation (predastore rejects group/other-readable keys outright).
-    ENCRYPTION_KEY="$CONFIG_DIR/predastore/encryption.key"
-    if [ ! -f "$ENCRYPTION_KEY" ]; then
-        mkdir -p "$CONFIG_DIR/predastore"
-        ( umask 0177 && openssl rand -out "$ENCRYPTION_KEY" 32 )
-        echo "   Generated predastore encryption key: $ENCRYPTION_KEY"
-    fi
-    export SPINIFEX_PREDASTORE_ENCRYPTION_KEY_FILE="$ENCRYPTION_KEY"
-
-    # Auto-detect Predastore host:port from spinifex.toml [nodes.<name>.predastore] section
-    PREDASTORE_BIND="0.0.0.0:8443"
-    if [ -f "$CONFIG_DIR/spinifex.toml" ]; then
-        DETECTED_PREDASTORE_HOST=$(awk -F'"' '/\[nodes\..*\.predastore\]/{found=1} found && /^host/{print $2; exit}' "$CONFIG_DIR/spinifex.toml")
-        if [ -n "$DETECTED_PREDASTORE_HOST" ]; then
-            PREDASTORE_BIND="$DETECTED_PREDASTORE_HOST"
-            echo "   Auto-detected Predastore bind=$PREDASTORE_BIND from spinifex.toml"
-        fi
-    fi
-    export SPINIFEX_PREDASTORE_HOST="${PREDASTORE_BIND%%:*}"
-    export SPINIFEX_PREDASTORE_PORT="${PREDASTORE_BIND##*:}"
-
-    export SPINIFEX_PREDASTORE_BACKEND=distributed
-
-    # Auto-detect Predastore NODE_ID from spinifex.toml if not already set
-    if [ -z "${SPINIFEX_PREDASTORE_NODE_ID:-}" ]; then
-        if [ -f "$CONFIG_DIR/spinifex.toml" ]; then
-            DETECTED_NODE_ID=$(awk -F'= *' '/node_id/{gsub(/ /,"",$2); print $2; exit}' "$CONFIG_DIR/spinifex.toml")
-            if [ -n "$DETECTED_NODE_ID" ] && [ "$DETECTED_NODE_ID" != "0" ]; then
-                export SPINIFEX_PREDASTORE_NODE_ID="$DETECTED_NODE_ID"
-                echo "   Auto-detected Predastore NODE_ID=$DETECTED_NODE_ID from spinifex.toml"
-            fi
-        fi
-    fi
-    export SPINIFEX_PREDASTORE_NODE_ID="${SPINIFEX_PREDASTORE_NODE_ID:-}"
-
-    PREDASTORE_CMD="./bin/spx service predastore start"
-    start_service "predastore" "$PREDASTORE_CMD"
-    set_oom_score "predastore" "-500"
-    if is_multinode; then
-        echo "   ⏭️  Skipping Predastore connectivity check (multi-node: needs quorum from peer nodes)"
-    else
-        check_service "Predastore" "$SPINIFEX_PREDASTORE_HOST" "$SPINIFEX_PREDASTORE_PORT"
-    fi
-else
-    echo "2️⃣  Skipping Predastore (not a local service)"
-fi
-
-
-# 3️⃣ Start Viperblock
-echo ""
-if has_service "viperblock"; then
-    echo "3️⃣  Starting Viperblock..."
-
-    # Determine base directory for Viperblock data, dev uses /mnt/ramdisk if available
-    if [ -d "/mnt/ramdisk" ] && [ -w "/mnt/ramdisk" ]; then
-        VB_BASE_DIR="/mnt/ramdisk/"
-    else
-        VB_BASE_DIR="$DATA_DIR/viperblock/"
-    fi
-
-    # Check if NBD plugin exists
-    NBD_PLUGIN_PATH="$MULGA_ROOT/viperblock/lib/nbdkit-viperblock-plugin.so"
-
-    if [ ! -f "$NBD_PLUGIN_PATH" ]; then
-        echo "   ⚠️  NBD plugin not found at $NBD_PLUGIN_PATH"
-        echo "   Building Viperblock NBD plugin..."
-        cd "$MULGA_ROOT/viperblock" && make build
-        cd "$PROJECT_ROOT"
-    fi
-
-    export SPINIFEX_CONFIG_PATH=$CONFIG_DIR/spinifex.toml
-    export SPINIFEX_VIPERBLOCK_PLUGIN_PATH=$NBD_PLUGIN_PATH
-    export SPINIFEX_BASE_DIR=$VB_BASE_DIR
-
-    VIPERBLOCK_CMD="./bin/spx service viperblock start"
-    start_service "viperblock" "$VIPERBLOCK_CMD"
-    set_oom_score "viperblock" "-500"
-else
-    echo "3️⃣  Skipping Viperblock (not a local service)"
-fi
-
-# 4️⃣ Start Spinifex Gateway/Daemon
-echo ""
-if has_service "daemon"; then
-    echo "4️⃣. Starting Spinifex Gateway..."
-
-    export SPINIFEX_CONFIG_PATH=$CONFIG_DIR/spinifex.toml
-    export SPINIFEX_BASE_DIR=$DATA_DIR/spinifex/
-    export SPINIFEX_WAL_DIR=$WAL_DIR
-
-    SPINIFEX_CMD="./bin/spx service spinifex start"
-    start_service "spinifex" "$SPINIFEX_CMD"
-    set_oom_score "spinifex" "-500"
-else
-    echo "4️⃣. Skipping Spinifex Gateway (not a local service)"
-fi
-
-
-# 5️⃣ Start AWS Gateway
-echo ""
-if has_service "awsgw"; then
-    echo "5️⃣. Starting AWS Gateway..."
-
-    unset SPINIFEX_NATS_HOST
-    unset SPINIFEX_PREDASTORE_HOST
-    export SPINIFEX_BASE_DIR=$DATA_DIR
-    export SPINIFEX_CONFIG_PATH=$CONFIG_DIR/spinifex.toml
-    export SPINIFEX_AWSGW_TLS_CERT=$CONFIG_DIR/server.pem
-    export SPINIFEX_AWSGW_TLS_KEY=$CONFIG_DIR/server.key
-
-    AWSGW_CMD="./bin/spx service awsgw start"
-    start_service "awsgw" "$AWSGW_CMD"
-    set_oom_score "awsgw" "-500"
-else
-    echo "5️⃣. Skipping AWS Gateway (not a local service)"
-fi
-
-
-# 6️⃣ Start vpcd (VPC daemon)
-echo ""
-if has_service "vpcd"; then
-    echo "6️⃣. Starting vpcd (VPC daemon)..."
-
-    export SPINIFEX_CONFIG_PATH=$CONFIG_DIR/spinifex.toml
-
-    VPCD_CMD="./bin/spx service vpcd start"
-    start_service "vpcd" "$VPCD_CMD"
-    set_oom_score "vpcd" "-500"
-else
-    echo "6️⃣. Skipping vpcd (not a local service)"
-fi
-
-# 7️⃣ Start Spinifex UI (skip with UI=false or if not a local service)
-if [ "${UI:-}" != "false" ] && has_service "ui"; then
-    echo ""
-    echo "7️⃣. Starting Spinifex UI..."
-
-    SPXUI_CMD="./bin/spx service spinifex-ui start"
-    start_service "spinifex-ui" "$SPXUI_CMD"
-    set_oom_score "spinifex-ui" "-500"
-else
-    echo ""
-    echo "7️⃣. Skipping Spinifex UI"
-fi
-
-
-echo ""
-echo "🔗 Service endpoints will be:"
-if [ "${UI:-}" != "false" ]; then
-    echo "   - Spinifex UI:       https://localhost:3000"
-fi
-echo "   - NATS:          nats://localhost:4222"
-echo "   - Predastore:    https://localhost:8443"
-echo "   - AWS Gateway:   https://localhost:9999"
-echo ""
-echo "📊 Monitor background service logs:"
-echo "   tail -f $LOGS_DIR/*.log"
-echo ""
-echo "🧪 Test with AWS CLI (once daemon is running):"
-echo "   export AWS_PROFILE=spinifex"
-echo "   aws ec2 describe-instances"
-echo ""
-
-# For multi-node clusters, check peer daemon health (best-effort)
-if is_multinode; then
-    echo "Checking cluster peer health..."
-    # Extract peer daemon hosts from config (host = "ip:port" under [nodes.X.daemon])
-    peer_hosts=$(awk '
-        /^\[nodes\./ { node=1; daemon=0 }
-        node && /^\[.*\.daemon\]/ { daemon=1 }
-        daemon && /^host/ { gsub(/[" ]/, "", $3); print $3; daemon=0 }
-    ' "$CONFIG_DIR/spinifex.toml")
-
-    for peer in $peer_hosts; do
-        attempts=0
-        max_attempts=5
-        while [ $attempts -lt $max_attempts ]; do
-            if curl -sk --connect-timeout 3 "https://$peer/health" > /dev/null 2>&1; then
-                echo "   $peer: healthy"
-                break
-            fi
-            attempts=$((attempts + 1))
-            if [ $attempts -lt $max_attempts ]; then
-                sleep 2
-            fi
-        done
-        if [ $attempts -ge $max_attempts ]; then
-            echo "   $peer: not responding (may still be starting)"
-        fi
-    done
-    echo ""
-fi
-
-# This will only be reached if air/daemon exits normally
-#echo ""
-#echo "🛑 Spinifex development environment stopped"

scripts/start-then-terminate-pending.sh

@@ -1,57 +0,0 @@
-#!/bin/sh
-# Launch instance then terminate it while still in `pending` state.
-
-set -e
-
-export AWS_PROFILE=spinifex
-
-ARCH=$(uname -m)
-case "$ARCH" in
-    x86_64)            IMAGE_NAME="ami-ubuntu-26.04-x86_64" ;;
-    aarch64|arm64)     IMAGE_NAME="ami-ubuntu-26.04-arm64" ;;
-    *)                 echo "Unknown arch $ARCH, default x86_64"; IMAGE_NAME="ami-ubuntu-26.04-x86_64" ;;
-esac
-
-INSTANCE_TYPE=$(aws ec2 describe-instance-types \
-    --query "InstanceTypes[?contains(InstanceType, '.micro')].InstanceType | [0]" \
-    --output text)
-[ -z "$INSTANCE_TYPE" ] || [ "$INSTANCE_TYPE" = "None" ] && { echo "no micro type"; exit 1; }
-
-AMI=$(aws ec2 describe-images --query "Images[?Name=='$IMAGE_NAME'].ImageId" --output text)
-[ -z "$AMI" ] && { echo "no AMI $IMAGE_NAME"; exit 1; }
-
-SUBNET=$(aws ec2 describe-subnets --query 'Subnets[0].SubnetId' --output text)
-SG=$(aws ec2 describe-security-groups --query 'SecurityGroups[0].GroupId' --output text)
-[ -z "$SUBNET" ] || [ "$SUBNET" = "None" ] && { echo "no subnet"; exit 1; }
-[ -z "$SG" ]     || [ "$SG" = "None" ]     && { echo "no SG"; exit 1; }
-
-echo "Launch: ami=$AMI type=$INSTANCE_TYPE subnet=$SUBNET sg=$SG"
-
-INSTANCE_ID=$(aws ec2 run-instances \
-    --image-id "$AMI" \
-    --instance-type "$INSTANCE_TYPE" \
-    --key-name spinifex-key \
-    --security-group-ids "$SG" \
-    --subnet-id "$SUBNET" \
-    --count 1 \
-    --query 'Instances[0].InstanceId' \
-    --output text)
-
-echo "Launched: $INSTANCE_ID — waiting for QEMU"
-
-# Wait until QEMU process exists (LaunchInstance.StartInstance fired) but
-# BEFORE TransitionState moves to running. Poll tight; terminate on first hit.
-TIMEOUT=60
-DEADLINE=$(( $(date +%s) + TIMEOUT ))
-QEMU_PID=""
-while [ "$(date +%s)" -lt "$DEADLINE" ]; do
-    QEMU_PID=$(pgrep -f "qemu.*$INSTANCE_ID" | head -1 || true)
-    [ -n "$QEMU_PID" ] && break
-done
-
-[ -z "$QEMU_PID" ] && { echo "ERR: QEMU never appeared"; exit 1; }
-
-echo "QEMU pid=$QEMU_PID — terminate now"
-aws ec2 terminate-instances --instance-ids "$INSTANCE_ID" \
-    --query 'TerminatingInstances[0].[InstanceId,PreviousState.Name,CurrentState.Name]' \
-    --output text

scripts/stop-dev.sh

@@ -1,201 +0,0 @@
-#!/bin/bash
-
-# Stop Spinifex development environment
-# This script stops all services started by start-dev.sh
-# Usage: ./scripts/stop-dev.sh
-# Note: Services are stopped using PID files, so data-dir is not required
-
-set -uo pipefail
-
-# Accept optional data directory argument
-DATA_DIR="${1:-$HOME/spinifex}"
-
-# Always use the logs directory for PID files — start-dev.sh writes shell PIDs
-# there, and Go services write PIDs to their own data dirs. This ensures
-# consistent stop behavior for both single-node and multi-node setups.
-PID_DIR="$DATA_DIR/logs"
-
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
-LOGS_DIR="$PROJECT_ROOT/data/logs"
-
-# Check if /mnt/ramdisk is mounted
-#if mountpoint -q /mnt/ramdisk; then
-    #echo "🛑 Stopping Spinifex development environment..."
-    #sudo umount /mnt/ramdisk
-#fi
-
-echo "Stopping Spinifex development environment..."
-
-# Parse services from spinifex.toml — defaults to all if not set
-CONFIG_DIR="${CONFIG_DIR:-$DATA_DIR/config}"
-parse_services() {
-    local config_file="$CONFIG_DIR/spinifex.toml"
-    if [ -f "$config_file" ]; then
-        local svc_line=$(grep -m1 '^services' "$config_file" | sed 's/.*\[//;s/\].*//;s/"//g;s/,/ /g')
-        if [ -n "$svc_line" ]; then
-            echo "$svc_line"
-            return
-        fi
-    fi
-    echo "nats predastore viperblock daemon awsgw vpcd ui"
-}
-
-SERVICES=$(parse_services)
-has_service() {
-    local svc="$1"
-    case "$svc" in
-        spinifex) svc="daemon" ;;
-        spinifex-ui) svc="ui" ;;
-    esac
-    echo "$SERVICES" | grep -qw "$svc"
-}
-
-# Detect multi-node cluster from config
-is_multinode() {
-    local config_file="$CONFIG_DIR/spinifex.toml"
-    if [ -f "$config_file" ]; then
-        # Count only top-level node sections [nodes.X], not subsections [nodes.X.Y]
-        local node_count=$(grep -cE '^\[nodes\.[^.]+\]' "$config_file")
-        [ "$node_count" -gt 1 ]
-    else
-        return 1
-    fi
-}
-
-# For multi-node clusters, delegate to coordinated shutdown via NATS.
-# Only when called without arguments (default path). When a data-dir is
-# explicitly provided (e.g., E2E per-node cleanup), use per-service stop.
-if is_multinode && [ -z "${SPINIFEX_FORCE_LOCAL_STOP:-}" ] && [ -z "${1:-}" ]; then
-    echo "Multi-node cluster detected. Using coordinated shutdown..."
-    echo "  (Set SPINIFEX_FORCE_LOCAL_STOP=1 to force per-service stop)"
-    exec $PROJECT_ROOT/bin/spx admin cluster shutdown
-fi
-
-# Function to stop a service. Sends SIGTERM and waits for the process to exit
-# so resources (ports, DB locks) are fully released before returning.
-stop_service() {
-    local name="$1"
-    local pidpath="$2"
-
-    echo "🛑 Stopping $name..."
-
-    local pid
-    pid=$(cat "$pidpath/$name.pid" 2>/dev/null) || { echo "⚠️  No PID file for $name"; return 1; }
-
-    kill -SIGTERM "$pid" 2>/dev/null || { rm -f "$pidpath/$name.pid"; echo "✅ $name already stopped"; echo ""; return 0; }
-
-    # Wait for process to fully exit (up to 15s) so ports/locks are released
-    local checks=0
-    while kill -0 "$pid" 2>/dev/null; do
-        sleep 1
-        checks=$((checks + 1))
-        if [ "$checks" -ge 15 ]; then
-            echo "  Force killing $name..."
-            kill -9 "$pid" 2>/dev/null || true
-            sleep 1
-            break
-        fi
-    done
-
-    rm -f "$pidpath/$name.pid"
-    echo "✅ $name stopped"
-    echo ""
-}
-
-# Stop services in reverse order
-echo "Stopping services..."
-echo ""
-# Stop Spinifex UI first (last to start)
-has_service "spinifex-ui" && stop_service "spinifex-ui" "$PID_DIR"
-
-# Stop Spinifex daemon/gateway (it will terminate running instances, unmount nbd devices)
-has_service "spinifex" && stop_service "spinifex" "$PID_DIR"
-
-# Wait for all QEMU instances to exit before stopping infrastructure services.
-# The daemon sends system_powerdown to VMs during shutdown, but may be killed
-# before VMs fully exit. We must wait here so viperblock/predastore aren't
-# pulled out from under running VMs.
-if pgrep -x qemu-system-x86_64 > /dev/null 2>&1; then
-    echo "⏳ Waiting for QEMU instances to exit..."
-    timeout=120
-    elapsed=0
-    while pgrep -x qemu-system-x86_64 > /dev/null 2>&1; do
-        if [ $elapsed -ge $timeout ]; then
-            echo "⚠️  Timeout waiting for QEMU processes, force killing..."
-            pkill -9 -x qemu-system-x86_64 2>/dev/null || true
-            sleep 1
-            break
-        fi
-        sleep 1
-        ((elapsed++))
-    done
-    echo "✅ All QEMU instances exited"
-fi
-
-# Stop vpcd (VPC daemon)
-has_service "vpcd" && stop_service "vpcd" "$PID_DIR"
-
-# Stop AWSGW
-has_service "awsgw" && stop_service "awsgw" "$PID_DIR"
-
-# Stop Viperblock
-has_service "viperblock" && stop_service "viperblock" "$PID_DIR"
-
-# Stop Predastore
-has_service "predastore" && stop_service "predastore" "$PID_DIR"
-
-# Stop NATS
-has_service "nats" && stop_service "nats" "$PID_DIR"
-
-# Stop OVN networking (prevents idle CPU burn when spinifex isn't running)
-# CRITICAL: Do NOT stop openvswitch-switch if OVS carries WAN traffic.
-# This can happen when an OVS bridge has a WAN IP (direct bridge mode) or
-# when a veth pair links an OVS bridge to a Linux WAN bridge (veth mode).
-# Stopping OVS would destroy the bridge and break WAN connectivity.
-# OVS is infrastructure — only stop ovn-controller and ovn-central.
-if pidof systemd >/dev/null 2>&1; then
-    echo "🛑 Stopping OVN networking..."
-    sudo systemctl stop ovn-controller 2>/dev/null && echo "✅ ovn-controller stopped" || true
-    if systemctl is-active --quiet ovn-central 2>/dev/null; then
-        sudo systemctl stop ovn-central 2>/dev/null && echo "✅ ovn-central stopped" || true
-    fi
-
-    # Only stop OVS if tearing it down won't kill WAN connectivity.
-    # Two cases where OVS is load-bearing for WAN:
-    #   1. An OVS bridge has a WAN IP directly (direct bridge mode)
-    #   2. A veth pair links a Linux bridge to an OVS bridge (veth mode) —
-    #      stopping OVS destroys the OVS bridge end, breaking the link
-    WAN_BRIDGE_HAS_IP=false
-    VETH_LINKS_WAN=false
-    for br in $(sudo ovs-vsctl list-br 2>/dev/null); do
-        if [ "$br" = "br-int" ]; then continue; fi
-        if ip -4 addr show "$br" 2>/dev/null | grep -q "inet "; then
-            WAN_BRIDGE_HAS_IP=true
-            break
-        fi
-    done
-    # Check if a veth pair links an OVS bridge to a Linux bridge with a WAN IP
-    if ip link show veth-wan-br >/dev/null 2>&1; then
-        VETH_LINKS_WAN=true
-    fi
-    if [ "$WAN_BRIDGE_HAS_IP" = true ]; then
-        echo "⚠️  Skipping openvswitch-switch stop ($br has WAN IP — stopping would kill connectivity)"
-    elif [ "$VETH_LINKS_WAN" = true ]; then
-        echo "⚠️  Skipping openvswitch-switch stop (veth pair links OVS to WAN bridge)"
-    else
-        sudo systemctl stop openvswitch-switch 2>/dev/null && echo "✅ openvswitch-switch stopped" || true
-    fi
-fi
-
-echo ""
-echo "✅ Spinifex development environment stopped"
-
-# Show any remaining spinifex-related processes
-remaining=$(pgrep -af '(bin/spx|spinifex-ui|nats-server|predastore|viperblock|vpcd|qemu-system)' | grep -v "stop-dev.sh" || true)
-if [[ -n "$remaining" ]]; then
-    echo ""
-    echo "⚠️  Some related processes may still be running:"
-    echo "$remaining"
-fi

spinifex/admin/admin.go

@@ -44,13 +44,12 @@ type ConfigSettings struct {
 	LogDir    string
 	ConfigDir string
 
-	// Add more fields as needed
 	Node   string
 	Az     string
 	Port   string
 	BindIP string
-	// AdvertiseIP is the off-host dial target rendered into the local node's
-	// [nodes.X].advertise field. Empty → callers fall back to BindIP.
+	// AdvertiseIP is the off-host dial target rendered into [nodes.X].advertise.
+	// Empty → callers fall back to BindIP.
 	AdvertiseIP string
 
 	// Cluster settings
@@ -82,10 +81,8 @@ type ConfigSettings struct {
 	PoolPrefixLen  int      // Subnet prefix length (default 24)
 	PoolDNSServers []string // DNS servers for VM DHCP (auto-detected from host)
 
-	// OperatorEmail is the address collected at install time (TUI, SPINIFEX_EMAIL,
-	// or `spx admin init --email`). Written under [operator] in spinifex.toml so
-	// reset-dev-env.sh can preserve it across wipes. Empty means no operator
-	// identity was supplied.
+	// OperatorEmail is the address collected at install time. Written under [operator]
+	// in spinifex.toml so it survives wipes. Empty means no identity was supplied.
 	OperatorEmail string
 
 	// Other nodes in the cluster (for config source of truth)
@@ -104,10 +101,15 @@ type ConfigSettings struct {
 	// Sets gpu_passthrough = true under [nodes.<node>.daemon].
 	GPUPassthrough bool
 
-	// IPSecEnabled toggles cluster-wide OVN native IPsec on intra-AZ Geneve
-	// (default true). Surfaced under [network] in spinifex.toml so the value
-	// is visible/auditable; the daemon reads it via cluster config.
+	// IPSecEnabled toggles cluster-wide OVN native IPsec on intra-AZ Geneve.
+	// Written under [network] in spinifex.toml; daemon reads it via cluster config.
 	IPSecEnabled bool
+
+	// EncryptionKeyFile is the path to the cluster-wide viperblock at-rest
+	// encryption key, rendered into [nodes.X.viperblock].encryption_key_file.
+	// Empty means no key was provisioned and volumes are written cleartext
+	// (legacy mode); the template omits the field entirely in that case.
+	EncryptionKeyFile string
 }
 
 // PredastoreNodeConfig describes a single Predastore node for multi-node config generation.
@@ -133,22 +135,19 @@ func GenerateConfigFiles(configs []ConfigFile, configSettings ConfigSettings) er
 	return nil
 }
 
-// generateConfigFile creates a configuration file from a template
+// GenerateConfigFile creates a configuration file from a template.
 func GenerateConfigFile(configPath string, configTemplate string, configSettings ConfigSettings) error {
-	// Parse the embedded template
 	tmpl, err := template.New("config").Parse(configTemplate)
 	if err != nil {
 		return fmt.Errorf("failed to parse template: %w", err)
 	}
 
-	// Create file with secure permissions
 	f, err := os.OpenFile(configPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
 	if err != nil {
 		return fmt.Errorf("failed to create config file: %w", err)
 	}
 	defer f.Close()
 
-	// Execute template
 	if err := tmpl.Execute(f, configSettings); err != nil {
 		return fmt.Errorf("failed to execute template: %w", err)
 	}
@@ -157,7 +156,6 @@ func GenerateConfigFile(configPath string, configTemplate string, configSettings
 }
 
 func GenerateCertificatesIfNeeded(configDir string, force bool, bindIP string) (caCertPath string) {
-	// Certificate paths
 	caCertPath = filepath.Join(configDir, "ca.pem")
 	caKeyPath := filepath.Join(configDir, "ca.key")
 	serverCertPath := filepath.Join(configDir, "server.pem")
@@ -171,7 +169,6 @@ func GenerateCertificatesIfNeeded(configDir string, force bool, bindIP string) (
 	if needsGeneration {
 		fmt.Println("\n🔐 Generating Certificate Authority and SSL certificates...")
 
-		// Step 1: Generate CA certificate
 		if err := GenerateCACert(caCertPath, caKeyPath); err != nil {
 			fmt.Fprintf(os.Stderr, "Error generating CA certificate: %v\n", err)
 			os.Exit(1)
@@ -180,7 +177,6 @@ func GenerateCertificatesIfNeeded(configDir string, force bool, bindIP string) (
 		fmt.Printf("   CA Certificate: %s\n", caCertPath)
 		fmt.Printf("   CA Key: %s\n", caKeyPath)
 
-		// Step 2: Generate server certificate signed by CA (with bind IP in SANs)
 		if err := GenerateSignedCert(serverCertPath, serverKeyPath, caCertPath, caKeyPath, bindIP); err != nil {
 			fmt.Fprintf(os.Stderr, "Error generating server certificate: %v\n", err)
 			os.Exit(1)
@@ -211,17 +207,14 @@ func GenerateServerCertOnly(configDir string, bindIP string) error {
 	serverCertPath := filepath.Join(configDir, "server.pem")
 	serverKeyPath := filepath.Join(configDir, "server.key")
 
-	// Verify CA files exist
 	if !FileExists(caCertPath) || !FileExists(caKeyPath) {
 		return fmt.Errorf("CA files not found in %s", configDir)
 	}
 
-	// Generate server cert signed by CA with this node's bind IP
 	return GenerateSignedCert(serverCertPath, serverKeyPath, caCertPath, caKeyPath, bindIP)
 }
 
 func CreateServiceDirectories(spxRoot string) {
-	// Create additional directories
 	dirs := []string{
 		filepath.Join(spxRoot, "images"),
 		filepath.Join(spxRoot, "amis"),
@@ -238,7 +231,6 @@ func CreateServiceDirectories(spxRoot string) {
 
 	fmt.Println("\n📁 Creating directory structure...")
 	for _, dir := range dirs {
-		// Check if directory exists
 		if _, err := os.Stat(dir); os.IsNotExist(err) {
 			if err := os.MkdirAll(dir, 0750); err != nil {
 				fmt.Fprintf(os.Stderr, "Warning: Could not create %s: %v\n", dir, err)
@@ -248,16 +240,12 @@ func CreateServiceDirectories(spxRoot string) {
 	fmt.Printf("✅ Directory structure created in %s\n", spxRoot)
 }
 
-// Helper functions
-
 func FileExists(path string) bool {
 	_, err := os.Stat(path)
 	return err == nil
 }
 
-// ChownRecursive changes ownership of a path and all its contents to the
-// specified username. Used after init to hand production directories back
-// to the service user when init runs as root via sudo.
+// ChownRecursive changes ownership of path and its contents to username.
 // Best-effort: errors are logged but do not halt the operation.
 func ChownRecursive(path, username string) {
 	u, err := user.Lookup(username)
@@ -306,7 +294,6 @@ func ChownRecursive(path, username string) {
 
 // SetServiceOwnership sets per-service ownership on data/config directories
 // and shared config files to root:spinifex with correct modes.
-// Keep in sync with setup.sh create_directories() and plan doc section 2.
 func SetServiceOwnership() {
 	grp, err := user.LookupGroup("spinifex")
 	if err != nil {
@@ -327,6 +314,7 @@ func SetServiceOwnership() {
 		"/var/lib/spinifex/nats":       "spinifex-nats",
 		"/etc/spinifex/predastore":     "spinifex-storage",
 		"/var/lib/spinifex/predastore": "spinifex-storage",
+		"/etc/spinifex/viperblock":     "spinifex-viperblock",
 		"/var/lib/spinifex/spinifex":   "spinifex-daemon",
 		"/var/lib/spinifex/viperblock": "spinifex-viperblock",
 		"/var/lib/spinifex/vpcd":       "spinifex-vpcd",
@@ -361,11 +349,12 @@ func SetServiceOwnership() {
 	// bootstrap.json lives in the awsgw data dir (not /etc/spinifex),
 	// so /etc/spinifex stays at 0750 (no group-write needed).
 	for path, mode := range map[string]os.FileMode{
-		"/etc/spinifex/spinifex.toml": 0640,
-		"/etc/spinifex/master.key":    0640,
-		"/etc/spinifex/server.pem":    0644,
-		"/etc/spinifex/server.key":    0640,
-		"/etc/spinifex/ca.pem":        0644,
+		"/etc/spinifex/spinifex.toml":             0640,
+		"/etc/spinifex/master.key":                0640,
+		"/etc/spinifex/viperblock/encryption.key": 0640,
+		"/etc/spinifex/server.pem":                0644,
+		"/etc/spinifex/server.key":                0640,
+		"/etc/spinifex/ca.pem":                    0644,
 	} {
 		if _, err := os.Stat(path); err != nil {
 			continue
@@ -379,12 +368,11 @@ func SetServiceOwnership() {
 	}
 }
 
-// updateAWSINIFile updates or creates an AWS INI file section with given key-value pairs
+// UpdateAWSINIFile updates or creates an AWS INI file section with the given key-value pairs.
 func UpdateAWSINIFile(path, section string, values map[string]string) error {
 	var cfg *ini.File
 	var err error
 
-	// Load existing file or create new one
 	if FileExists(path) {
 		cfg, err = ini.Load(path)
 		if err != nil {
@@ -394,25 +382,21 @@ func UpdateAWSINIFile(path, section string, values map[string]string) error {
 		cfg = ini.Empty()
 	}
 
-	// Get or create section
 	sec, err := cfg.NewSection(section)
 	if err != nil {
-		// Section already exists, get it
+		// Section already exists, get it.
 		sec, err = cfg.GetSection(section)
 		if err != nil {
 			return fmt.Errorf("failed to get section: %w", err)
 		}
 	}
 
-	// Set key-value pairs
 	for key, value := range values {
 		sec.Key(key).SetValue(value)
 	}
 
-	// Write atomically with 0600. ini.SaveTo uses os.Create (0666 & ~umask,
-	// typically 0644), which leaves ~/.aws/credentials world-readable. Render
-	// to a sibling temp file (os.CreateTemp is 0600) and rename into place so
-	// the secrets are never exposed, even briefly.
+	// Write atomically: ini.SaveTo uses os.Create (world-readable); render to a
+	// sibling temp file (0600) and rename to avoid briefly exposing secrets.
 	dir := filepath.Dir(path)
 	tmp, err := os.CreateTemp(dir, ".aws-ini-*.tmp")
 	if err != nil {
@@ -433,8 +417,7 @@ func UpdateAWSINIFile(path, section string, values map[string]string) error {
 	return nil
 }
 
-// generateAWSAccessKey generates an AWS-style access key
-// Format: AKIA + 16 random uppercase alphanumeric characters
+// GenerateAWSAccessKey generates an AWS-style access key (AKIA + 16 random alphanumeric chars).
 func GenerateAWSAccessKey() (string, error) {
 	const prefix = "AKIA"
 	const charset = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
@@ -452,8 +435,7 @@ func GenerateAWSAccessKey() (string, error) {
 	return prefix + string(result), nil
 }
 
-// generateAWSSecretKey generates an AWS-style secret key
-// 40 character base64-encoded string
+// GenerateAWSSecretKey generates a 40-character base64-encoded AWS-style secret key.
 func GenerateAWSSecretKey() (string, error) {
 	bytes := make([]byte, 30) // 30 bytes = 40 chars in base64
 	if _, err := rand.Read(bytes); err != nil {
@@ -488,15 +470,13 @@ func GenerateNATSToken() (string, error) {
 	return "nats_" + base64.URLEncoding.EncodeToString(bytes)[:32], nil
 }
 
-// GenerateCACert generates a Certificate Authority certificate and key
+// GenerateCACert generates a Certificate Authority certificate and key.
 func GenerateCACert(caCertPath, caKeyPath string) error {
-	// Generate CA private key
 	caPrivateKey, err := rsa.GenerateKey(rand.Reader, 4096)
 	if err != nil {
 		return fmt.Errorf("failed to generate CA private key: %w", err)
 	}
 
-	// Create CA certificate template
 	notBefore := time.Now()
 	notAfter := notBefore.Add(3650 * 24 * time.Hour) // 10 years
 
@@ -519,13 +499,11 @@ func GenerateCACert(caCertPath, caKeyPath string) error {
 		MaxPathLen:            1,
 	}
 
-	// Self-sign the CA certificate
 	caDerBytes, err := x509.CreateCertificate(rand.Reader, &caTemplate, &caTemplate, &caPrivateKey.PublicKey, caPrivateKey)
 	if err != nil {
 		return fmt.Errorf("failed to create CA certificate: %w", err)
 	}
 
-	// Write CA certificate to file
 	caCertOut, err := os.Create(caCertPath)
 	if err != nil {
 		return fmt.Errorf("failed to create CA cert file: %w", err)
@@ -536,7 +514,6 @@ func GenerateCACert(caCertPath, caKeyPath string) error {
 		return fmt.Errorf("failed to write CA cert: %w", err)
 	}
 
-	// Write CA private key to file
 	caKeyOut, err := os.OpenFile(caKeyPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
 	if err != nil {
 		return fmt.Errorf("failed to create CA key file: %w", err)
@@ -752,15 +729,13 @@ func GenerateSignedCertWithDNS(certPath, keyPath, caCertPath, caKeyPath string,
 	return nil
 }
 
-// generateSelfSignedCert generates a self-signed SSL certificate (legacy, kept for compatibility)
+// GenerateSelfSignedCert generates a self-signed SSL certificate (legacy, kept for compatibility).
 func GenerateSelfSignedCert(certPath, keyPath string) error {
-	// Generate private key
 	privateKey, err := rsa.GenerateKey(rand.Reader, 4096)
 	if err != nil {
 		return fmt.Errorf("failed to generate private key: %w", err)
 	}
 
-	// Create certificate template
 	notBefore := time.Now()
 	notAfter := notBefore.Add(3650 * 24 * time.Hour) // 10 years
 
@@ -784,13 +759,11 @@ func GenerateSelfSignedCert(certPath, keyPath string) error {
 		IPAddresses:           []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1")},
 	}
 
-	// Create certificate
 	derBytes, err := x509.CreateCertificate(rand.Reader, &template, &template, &privateKey.PublicKey, privateKey)
 	if err != nil {
 		return fmt.Errorf("failed to create certificate: %w", err)
 	}
 
-	// Write certificate to file
 	certOut, err := os.Create(certPath)
 	if err != nil {
 		return fmt.Errorf("failed to create cert file: %w", err)
@@ -801,7 +774,6 @@ func GenerateSelfSignedCert(certPath, keyPath string) error {
 		return fmt.Errorf("failed to write cert: %w", err)
 	}
 
-	// Write private key to file
 	keyOut, err := os.OpenFile(keyPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
 	if err != nil {
 		return fmt.Errorf("failed to create key file: %w", err)
@@ -821,11 +793,6 @@ func GenerateSelfSignedCert(certPath, keyPath string) error {
 }
 
 // SetupAWSCredentials updates ~/.aws/credentials and ~/.aws/config.
-// bindIP is the IP the AWS gateway listens on. If empty or "0.0.0.0", the
-// operator's local ~/.aws/config endpoint_url falls back to "localhost" (this
-// runs on the same box as the gateway). wanIP is reserved for a future
-// --operator-endpoint flag that will let remote operators point their CLI at
-// the host's WAN IP; today it is accepted but unused.
 // When running under sudo, writes to SUDO_USER's home instead of root's.
 func SetupAWSCredentials(accessKey, secretKey, region, certPath, bindIP, wanIP string) error {
 	_ = wanIP
@@ -834,8 +801,6 @@ func SetupAWSCredentials(accessKey, secretKey, region, certPath, bindIP, wanIP s
 		return err
 	}
 
-	// When running under sudo, write to the invoking user's home directory
-	// so the operator can use AWS_PROFILE=spinifex without sudo.
 	sudoUser := os.Getenv("SUDO_USER")
 	if os.Getuid() == 0 && sudoUser != "" {
 		if u, err := user.Lookup(sudoUser); err == nil {
@@ -851,21 +816,8 @@ func SetupAWSCredentials(accessKey, secretKey, region, certPath, bindIP, wanIP s
 	credPath := filepath.Join(awsDir, "credentials")
 	configPath := filepath.Join(awsDir, "config")
 
-	// Determine profile name
-	//profileName := "default"
-
-	// Use Spinifex as the default profile
 	profileName := "spinifex"
 
-	if FileExists(credPath) {
-		// Check if default profile already exists
-		cfg, err := ini.Load(credPath)
-		if err == nil && cfg.HasSection("default") {
-			profileName = "spinifex"
-		}
-	}
-
-	// Update credentials file
 	if err := UpdateAWSINIFile(credPath, profileName, map[string]string{
 		"aws_access_key_id":     accessKey,
 		"aws_secret_access_key": secretKey,
@@ -873,7 +825,6 @@ func SetupAWSCredentials(accessKey, secretKey, region, certPath, bindIP, wanIP s
 		return err
 	}
 
-	// Update config file
 	configSection := profileName
 	if profileName != "default" {
 		configSection = "profile " + profileName
@@ -961,9 +912,8 @@ func ParsePredastoreNodeIDFromConfig(tomlContent string, ip string) int {
 	return FindNodeIDByIP(cfg.DB, ip)
 }
 
-// SetMIGProfile idempotently writes mig_profile = "<profile>" for the given
-// node into spinifex.toml, preserving all other content and comments.
-// An empty profile clears the setting (writes mig_profile = "").
+// SetMIGProfile idempotently writes mig_profile = "<profile>" for the given node
+// into spinifex.toml. An empty profile clears the setting.
 func SetMIGProfile(tomlPath, node, profile string) error {
 	raw, err := os.ReadFile(tomlPath)
 	if err != nil {

spinifex/admin/dial.go

@@ -2,13 +2,8 @@ package admin
 
 import "net"
 
-// DialTarget returns a host:port suitable for an in-process client to dial,
-// given a configured listen address. Inputs of the form "0.0.0.0:N" or "[::]:N"
-// are rewritten to "127.0.0.1:N" or "[::1]:N" — a service listening on the
-// wildcard is reachable on loopback, and loopback is the TLS SAN that the
-// daemon's own cert always covers. Loopback, specific IPs, and bare hostnames
-// pass through unchanged. Malformed input (no host:port) passes through so
-// callers still see the original value in error messages.
+// DialTarget returns a host:port for in-process dialing. Wildcard listeners
+// ("0.0.0.0:N", "[::]:N") are rewritten to loopback; specific IPs pass through.
 func DialTarget(listenAddr string) string {
 	host, port, err := net.SplitHostPort(listenAddr)
 	if err != nil {
@@ -23,12 +18,8 @@ func DialTarget(listenAddr string) string {
 	return listenAddr
 }
 
-// AdvertiseHost picks the host that off-host clients should dial for a service
-// listening on listenAddr. If the listen host is a wildcard (0.0.0.0, ::, or
-// empty), the caller-supplied advertiseIP is returned; otherwise the listen
-// host itself is returned. Port is intentionally dropped — callers append the
-// port appropriate for the protocol (some services listen and advertise on
-// different ports).
+// AdvertiseHost returns the host off-host clients should dial. Wildcard listeners
+// return advertiseIP; specific listen hosts return themselves (port is dropped).
 func AdvertiseHost(listenAddr, advertiseIP string) string {
 	host, _, err := net.SplitHostPort(listenAddr)
 	if err != nil {

spinifex/admin/email.go

@@ -26,12 +26,8 @@ func ValidateEmail(addr string) error {
 	return nil
 }
 
-// ReadOperatorEmail extracts the [operator].email scalar from a spinifex.toml
-// file, or returns "" if the file doesn't exist, can't be read, or has no
-// [operator] section. Used by `spx admin init --force` (and reset flows) to
-// preserve a previously-captured email across re-inits when --email isn't
-// supplied on the new invocation. Minimal text scan — no TOML parser, since
-// we only need one scalar from one section.
+// ReadOperatorEmail extracts the [operator].email scalar from spinifex.toml,
+// returning "" on any error or missing section. Uses a text scan, not a full TOML parser.
 func ReadOperatorEmail(tomlPath string) string {
 	f, err := os.Open(tomlPath)
 	if err != nil {

spinifex/admin/images_remove.go

@@ -46,10 +46,8 @@ func (d Dependents) Empty() bool {
 	return len(d.Snapshots) == 0 && len(d.Volumes) == 0 && len(d.AMIs) == 0
 }
 
-// RemovePreview captures everything the CLI needs to render the confirmation
-// prompt and decide whether to proceed: AMI metadata (or zero values when the
-// config is missing/corrupt), backing-storage byte counts, and the dependent
-// resource set. PreviewRemoveSystemImage performs no deletions.
+// RemovePreview captures AMI metadata, byte counts, and dependents for the
+// CLI confirmation prompt. PreviewRemoveSystemImage performs no deletions.
 type RemovePreview struct {
 	ImageID       string
 	Name          string
@@ -74,11 +72,8 @@ func SnapPrefix(imageID string) string {
 	return "snap-" + imageID
 }
 
-// PreviewRemoveSystemImage walks the object store and gathers everything the
-// caller needs to decide whether to remove an admin-imported AMI. It performs
-// no mutations. Returned errors indicate unexpected I/O failures only —
-// AMI-not-found, corrupt config, and account-owned AMIs are reflected in the
-// preview fields, not as errors.
+// PreviewRemoveSystemImage gathers AMI metadata and dependents without mutating
+// any state. Not-found, corrupt config, and account-owned AMI are preview fields.
 func PreviewRemoveSystemImage(store objectstore.ObjectStore, bucket, imageID string) (*RemovePreview, error) {
 	if !strings.HasPrefix(imageID, "ami-") {
 		return nil, errors.New(awserrors.ErrorInvalidAMIIDMalformed)
@@ -125,16 +120,8 @@ func PreviewRemoveSystemImage(store objectstore.ObjectStore, bucket, imageID str
 	return preview, nil
 }
 
-// FindAMIDependents walks the bucket to find every resource that transitively
-// depends on the given admin-imported AMI:
-//
-//  1. snap-*/metadata.json whose VolumeID == imageID  → derivedSnaps
-//     (these are CopyImage-created snapshots that point back at the system AMI)
-//  2. vol-*/config.json whose VolumeMetadata.SnapshotID ∈ {snap-ami-<id>} ∪ derivedSnaps
-//  3. ami-*/config.json whose AMIMetadata.SnapshotID ∈ derivedSnaps
-//
-// The walk terminates one hop deep. CopyImage of an account AMI uses its own
-// VolumeID, not the system AMI's, so further recursion is not required today.
+// FindAMIDependents returns snapshots, volumes, and AMIs that depend on the
+// given system AMI. Walk terminates one hop deep.
 func FindAMIDependents(store objectstore.ObjectStore, bucket, imageID string) (Dependents, error) {
 	var deps Dependents
 
@@ -220,16 +207,9 @@ func FindAMIDependents(store objectstore.ObjectStore, bucket, imageID string) (D
 	return deps, nil
 }
 
-// RemoveSystemImage deletes an admin-imported AMI and its backing storage
-// after re-validating the preconditions:
-//
-//  1. ami-<id>/config.json exists and parses (skipped under --force = salvage)
-//  2. AMI is system-owned, not account-owned (--force bypasses)
-//  3. No volumes / snapshots / AMIs depend on it (--force bypasses)
-//
-// Deletion order is intentional: the config.json is deleted FIRST so the AMI
-// disappears from DescribeImages immediately, preventing new RunInstances
-// from racing the block cleanup.
+// RemoveSystemImage deletes an admin-imported AMI after re-validating that it
+// is system-owned and has no dependents (bypassed by --force). config.json is
+// deleted first so the AMI vanishes from DescribeImages before block cleanup.
 func RemoveSystemImage(store objectstore.ObjectStore, bucket string, opts RemoveImageOpts) (*RemoveImageResult, error) {
 	if !strings.HasPrefix(opts.ImageID, "ami-") {
 		return nil, errors.New(awserrors.ErrorInvalidAMIIDMalformed)
@@ -338,16 +318,14 @@ func readAMIConfig(store objectstore.ObjectStore, bucket, imageID string) (viper
 	}
 
 	var state viperblock.VBState
-	if err := json.Unmarshal(body, &state); err != nil {
+	if err := json.Unmarshal(viperblock.StateBody(body), &state); err != nil {
 		return viperblock.AMIMetadata{}, fmt.Errorf("%w: %s: %v", handlers_ec2_image.ErrCorruptAMIConfig, key, err)
 	}
 	return state.VolumeConfig.AMIMetadata, nil
 }
 
-// errCorruptVolumeConfig lets the dependency walk distinguish a vol-*/config.json
-// that can't be parsed (recovery is operator-owned, walk continues) from a
-// transient transport error (walk must fail closed — silently dropping a
-// dependent could lead to deleting blocks underneath a live volume).
+// errCorruptVolumeConfig distinguishes an unparse-able config (walk continues)
+// from a transport error (walk fails closed to prevent deleting live blocks).
 var errCorruptVolumeConfig = errors.New("corrupt volume config")
 
 // readVolumeConfig reads vol-<id>/config.json into VolumeConfig.
@@ -372,7 +350,7 @@ func readVolumeConfig(store objectstore.ObjectStore, bucket, volumeID string) (*
 	}
 
 	var w volumeConfigWrapper
-	if err := json.Unmarshal(body, &w); err != nil {
+	if err := json.Unmarshal(viperblock.StateBody(body), &w); err != nil {
 		return nil, fmt.Errorf("%w: %s: %v", errCorruptVolumeConfig, key, err)
 	}
 	return &w.VolumeConfig, nil

spinifex/admin/ipsec.go

@@ -33,33 +33,16 @@ const (
 // strongSwan config consumes them as-is.
 var oidExtKeyUsageIPSecIKE = asn1.ObjectIdentifier{1, 3, 6, 1, 5, 5, 7, 3, 17}
 
-// IPSecCertPaths returns the canonical paths for the per-node IPsec peer
-// credential pair under configDir. Both paths live in
-// {configDir}/ipsec/. The CA trust anchor is the existing cluster CA in
-// configDir/ca.pem; ovs-monitor-ipsec is pointed at that path directly.
+// IPSecCertPaths returns the canonical cert/key paths for the per-node IPsec peer
+// credential pair under {configDir}/ipsec/.
 func IPSecCertPaths(configDir string) (certPath, keyPath string) {
 	dir := filepath.Join(configDir, ipsecDirName)
 	return filepath.Join(dir, ipsecCertFileName), filepath.Join(dir, ipsecKeyFileName)
 }
 
-// GenerateIPSecPeerCert issues an IKEv2 peer certificate for this node,
-// signed by the existing cluster CA. The cert carries the IPSec-IKE
-// extended key usage so strongSwan accepts it as a peer credential.
-//
-// hostname is the node's primary hostname (used as Common Name + DNS SAN).
-// nodeIP is the node's primary cluster IP (added as an IP SAN). Additional
-// SANs discovered from the local machine's interfaces are not added here —
-// IPSec peer identity should be the explicit cluster identity only, not
-// every interface address, to keep peer-identity matching strict.
-//
-// Cross-cutting invariant: this dnsName SAN must equal the OVS chassis-id
-// (Open_vSwitch.external_ids:system-id) on every node, because
-// ovs-monitor-ipsec generates strongSwan configs with leftid=@<chassis-id>
-// / rightid=@<chassis-id> and charon validates those names against the
-// peer cert's dnsName SAN. setup-ovn.sh pins system-id to `hostname -s`
-// for exactly this reason — see the comment in setup-ovn.sh near
-// `system-id.conf`. Changing the cert identity here without updating
-// setup-ovn.sh (or vice versa) breaks IKE with AUTHENTICATION_FAILED.
+// GenerateIPSecPeerCert issues an IKEv2 peer certificate for this node signed
+// by the existing cluster CA. The cert carries the IPSec-IKE EKU so strongSwan
+// accepts it. The dnsName SAN must equal the OVS chassis-id (system-id).
 func GenerateIPSecPeerCert(configDir, caCertPath, caKeyPath, hostname, nodeIP string) error {
 	if hostname == "" {
 		return fmt.Errorf("ipsec peer cert: hostname required")
@@ -127,14 +110,8 @@ func GenerateIPSecPeerCert(configDir, caCertPath, caKeyPath, hostname, nodeIP st
 		return fmt.Errorf("ipsec peer cert: write key: %w", err)
 	}
 
-	// Symlink the cluster CA into charon's default cacerts scan dir.
-	// ovs-monitor-ipsec writes per-conn `cacert=` references, but
-	// strongSwan 6.0 only loads CAs declared in standalone `ca` sections
-	// or files placed in /etc/ipsec.d/cacerts at startup. Without this,
-	// charon authenticates the local side OK but rejects every inbound
-	// peer cert as `no trusted RSA public key found for '<peer>'` and
-	// no SAs negotiate. The dir is on charon's default AppArmor allowlist
-	// so no profile change is needed (unlike /etc/spinifex/).
+	// strongSwan 6.0 only loads CAs placed in /etc/ipsec.d/cacerts at startup;
+	// per-conn cacert= references are insufficient. Install the cluster CA there.
 	if err := installCAIntoCharonTrustStore(caCertPath); err != nil {
 		return fmt.Errorf("ipsec peer cert: install CA into charon trust store: %w", err)
 	}
@@ -166,15 +143,8 @@ func installCAIntoCharonTrustStore(caCertPath string) error {
 		return fmt.Errorf("rename %s -> %s: %w", staging, link, err)
 	}
 
-	// strongSwan scans /etc/ipsec.d/cacerts only at charon startup. If
-	// openvswitch-ipsec.service started before this install (the common
-	// case — the unit is enabled at provision time), charon is running
-	// with zero CAs loaded and every IKE_AUTH ends in `no trusted RSA
-	// public key found for '<peer>'`. Trigger a stroke rereadcacerts so
-	// the symlink we just placed is loaded without bouncing charon.
-	// Silent failure is expected when charon isn't running yet (admin
-	// init can run before setup-ovn.sh enables the unit); a later charon
-	// start will scan cacerts naturally.
+	// Trigger rereadcacerts so charon picks up the symlink without restart.
+	// Silent failure is expected when charon isn't running yet.
 	if err := charonRereadCAs(); err != nil {
 		slog.Debug("ipsec: rereadcacerts skipped", "err", err)
 	}

spinifex/admin/ipsec_test.go

@@ -127,12 +127,8 @@ func TestIPSecCertPaths(t *testing.T) {
 	assert.Equal(t, "/etc/spinifex/ipsec/peer.key", keyPath)
 }
 
-// TestInstallCAIntoCharonTrustStore_TriggersRereadCAs locks the fix for the
-// race where openvswitch-ipsec started before admin init's CA install: every
-// IKE_AUTH failed with `no trusted RSA public key found for '<peer>'` because
-// charon had scanned cacerts before the symlink was placed. The trust-store
-// installer must trigger `ipsec rereadcacerts` so an already-running charon
-// picks up the CA without a restart.
+// TestInstallCAIntoCharonTrustStore_TriggersRereadCAs ensures the installer
+// calls `ipsec rereadcacerts` so an already-running charon picks up the CA.
 func TestInstallCAIntoCharonTrustStore_TriggersRereadCAs(t *testing.T) {
 	origTrustDir := charonCATrustDir
 	charonCATrustDir = t.TempDir()
@@ -160,9 +156,7 @@ func TestInstallCAIntoCharonTrustStore_TriggersRereadCAs(t *testing.T) {
 }
 
 // TestInstallCAIntoCharonTrustStore_SwallowsRereadError verifies that a failed
-// rereadcacerts (e.g. charon not running, ipsec binary missing on a fresh
-// install) does not break admin init. The CA symlink must still be in place so
-// a later charon start picks it up via the normal cacerts scan.
+// rereadcacerts (charon not running) doesn't abort admin init.
 func TestInstallCAIntoCharonTrustStore_SwallowsRereadError(t *testing.T) {
 	origTrustDir := charonCATrustDir
 	charonCATrustDir = t.TempDir()

spinifex/admin/network_detect.go

@@ -54,7 +54,6 @@ func DetectNetwork() (*DetectedNetwork, error) {
 		return nil, fmt.Errorf("could not parse default route: %s", defaultRoute)
 	}
 
-	// Get all routes to find interfaces and their subnets
 	allRoutes, err := exec.Command("ip", "-4", "route", "show", "scope", "link").Output()
 	if err != nil {
 		return nil, fmt.Errorf("failed to get routes: %w", err)
@@ -149,7 +148,6 @@ func SuggestPoolRange(wan *DetectedInterface) (start, end string) {
 		return "", ""
 	}
 
-	// Calculate broadcast address
 	ip := ipNet.IP.To4()
 	mask := ipNet.Mask
 	broadcast := make(net.IP, 4)
@@ -182,13 +180,9 @@ func SuggestPoolRange(wan *DetectedInterface) (start, end string) {
 	return startIP.String(), endIP.String()
 }
 
-// isVirtualInterface returns true for interfaces that shouldn't be
-// considered as physical NICs. The defaultRouteDev parameter is the
-// interface carrying the default route — it's always considered physical
-// even if its name matches a virtual prefix (e.g., br-wan).
+// isVirtualInterface returns true for interfaces that shouldn't be considered
+// physical NICs. defaultRouteDev is always treated as physical.
 func isVirtualInterface(name string, defaultRouteDev string) bool {
-	// The default route interface is always considered physical — it's the
-	// WAN uplink, even if it's a bridge (br-wan, br-external, etc.).
 	if defaultRouteDev != "" && name == defaultRouteDev {
 		return false
 	}

spinifex/awsec2query/parser.go

@@ -13,27 +13,6 @@ import (
 // Callers should map this to AWS's MalformedQueryString error code.
 var ErrSliceTooLarge = errors.New("list parameter exceeds maximum entries")
 
-/*
-
-Sample args
-
-args["Action"] = "RunInstance"
-args["ImageId"] = "ami-123456789"
-args["MinCount"] = "1"
-args["MaxCount"] = "1"
-args["InstanceType"] = "t2.micro"
-args["KeyName"] = "my-key-pair"
-args["SecurityGroup.1"] = "sg-12345678"
-args["SecurityGroup.2"] = "sg-87654321"
-args["TagSpecification.1.ResourceType"] = "instance"
-args["TagSpecification.1.Tag.1.Key"] = "Name"
-args["TagSpecification.1.Tag.1.Value"] = "MyInstance"
-args["TagSpecification.2.ResourceType"] = "volume"
-args["TagSpecification.2.Tag.1.Key"] = "Environment"
-args["TagSpecification.2.Tag.1.Value"] = "Production"
-
-*/
-
 func QueryParamsToStruct(params map[string]string, out any) error {
 	v := reflect.ValueOf(out)
 
@@ -67,10 +46,8 @@ func setStructFields(v reflect.Value, params map[string]string, prefix string) e
 		// Check for locationName tag (AWS SDK uses this for query parameter names)
 		locationName := fieldType.Tag.Get("locationName")
 
-		// Try the field name, locationName, and title-cased locationName.
-		// AWS query params use title case (e.g. "ResourceId") but some SDK
-		// structs use camelCase locationName (e.g. "resourceId" in DeleteTagsInput
-		// vs "ResourceId" in CreateTagsInput).
+		// Try field name, locationName, and title-cased locationName.
+		// AWS query params use title case but some SDK structs use camelCase locationName.
 		queryKeys := []string{prefix + fieldName}
 		if locationName != "" && locationName != fieldName {
 			queryKeys = append(queryKeys, prefix+locationName)
@@ -153,13 +130,12 @@ func setFieldValue(field reflect.Value, value string) error {
 		elem := field.Type().Elem()
 		// Special case: []byte
 		if elem.Kind() == reflect.Uint8 {
-			// First try base64 (what the AWS API expects for PublicKeyMaterial).
+			// Try base64 first (AWS API expectation); fall back to raw bytes.
 			trimmed := strings.TrimSpace(value)
 			if decoded, err := base64.StdEncoding.DecodeString(trimmed); err == nil {
 				field.SetBytes(decoded)
 				return nil
 			}
-			// Some clients may send raw text. Fall back to raw bytes.
 			field.SetBytes([]byte(value))
 			return nil
 		}
@@ -195,8 +171,7 @@ func setFieldValue(field reflect.Value, value string) error {
 const maxSliceLen = 1024
 
 func setSliceField(field reflect.Value, params map[string]string, prefix string) error {
-	// Find all indexed items for this slice.
-	// Supports both EC2-style (Prefix.N) and IAM/ELBv2-style (Prefix.member.N) formats.
+	// Collect indexed items; supports EC2-style (Prefix.N) and IAM/ELBv2-style (Prefix.member.N).
 	indices := make(map[int]bool)
 	useMember := false
 	for key := range params {

spinifex/awserrors/awserrors.go

@@ -221,9 +221,7 @@ var (
 	ErrorInvalidOptionConflict                                 = "InvalidOption.Conflict"
 	ErrorInvalidPaginationToken                                = "InvalidPaginationToken"
 	ErrorInvalidParameter                                      = "InvalidParameter"
-	// ACM (AWS JSON 1.1). Suffix-less; the JSON error envelope appends
-	// "Exception" on the wire (InvalidArnException), matching aws-sdk-go acm
-	// ErrCode* values. Not-found reuses the shared ErrorResourceNotFound.
+	// ACM error: the JSON envelope appends "Exception" on the wire; not-found reuses ErrorResourceNotFound.
 	ErrorACMInvalidArn                                        = "InvalidArn"
 	ErrorInvalidParameterCombination                          = "InvalidParameterCombination"
 	ErrorInvalidParameterDependency                           = "InvalidParameterDependency"
@@ -372,9 +370,7 @@ var (
 	ErrorResourceCountExceeded                                = "ResourceCountExceeded"
 	ErrorResourceCountLimitExceeded                           = "ResourceCountLimitExceeded"
 	ErrorResourceLimitExceeded                                = "ResourceLimitExceeded"
-	// ErrorResourceNotFound ("ResourceNotFound") is the ACM not-found code. EKS
-	// must use ErrorEKSResourceNotFound ("ResourceNotFoundException", below) —
-	// the restjson code the aws eks client expects. Do not cross-wire the two.
+	// ErrorResourceNotFound is the ACM not-found code. EKS must use ErrorEKSResourceNotFound — do not cross-wire.
 	ErrorResourceNotFound                               = "ResourceNotFound"
 	ErrorEKSResourceInUse                               = "ResourceInUseException"
 	ErrorEKSResourceNotFound                            = "ResourceNotFoundException"
@@ -495,15 +491,21 @@ func ValidErrorCode(code string) string {
 	return ErrorServerInternal
 }
 
-// IsErrorCode reports whether err carries the named AWS error code. Service
-// handlers return these codes as plain strings (errors.New(code)), commonly
-// wrapped with %w by orchestration callers, so the match is a substring test
-// against err.Error() rather than equality. Use it to classify a delete error
-// as idempotent (already-gone) or retryable (still-in-use) in teardown paths.
+// IsErrorCode reports whether err carries the named AWS error code.
+// Uses substring matching because handlers return codes as plain strings, sometimes wrapped with %w.
 func IsErrorCode(err error, code string) bool {
 	return err != nil && strings.Contains(err.Error(), code)
 }
 
+// IsNotFound reports whether err carries an AWS "*.NotFound" error code — the
+// canonical "resource absent" signal. Destroy orchestration (teardown / GC /
+// reaper) uses this to treat an already-gone resource as success, so the API
+// handlers stay AWS-faithful (idempotency lives at the call site, not the
+// handler). Substring match mirrors IsErrorCode for %w-wrapped errors.
+func IsNotFound(err error) bool {
+	return err != nil && strings.Contains(err.Error(), ".NotFound")
+}
+
 var ErrorLookup = map[string]ErrorMessage{
 	ErrorAccountDisabled: {HTTPCode: 400, Message: "The functionality you have requested has been administratively disabled for this account."},
 	ErrorActiveVpcPeeringConnectionPerVpcLimitExceeded:         {HTTPCode: 400, Message: "You've reached the limit on the number of active VPC peering connections you can have for the specified VPC."},

spinifex/awserrors/awserrors_test.go

@@ -1,12 +1,13 @@
 package awserrors
 
-import "testing"
+import (
+	"errors"
+	"fmt"
+	"testing"
+)
 
-// TestErrorLookup_Structure asserts invariants of the ErrorLookup map without
-// duplicating every entry. Catches wholesale-deletion accidents, invalid HTTP
-// codes, empty messages, and regressions of SDK-surfaced codes — the failure
-// modes that actually matter — without locking in a 1:1 mirror that has to be
-// updated in lockstep with every new error code.
+// TestErrorLookup_Structure asserts ErrorLookup invariants: minimum size, valid HTTP codes, non-empty messages.
+// Avoids a 1:1 mirror that would need updating with every new error code.
 func TestErrorLookup_Structure(t *testing.T) {
 	if len(ErrorLookup) < 400 {
 		t.Fatalf("ErrorLookup unexpectedly small: %d entries", len(ErrorLookup))
@@ -68,3 +69,36 @@ func TestValidErrorCode(t *testing.T) {
 		})
 	}
 }
+
+// TestIsNotFound covers the destroy-orchestration tolerance helper: it matches
+// any AWS canonical InvalidX.NotFound (so teardown converges on already-gone
+// resources) and only those — a nil error or a non-NotFound failure is not
+// swallowed.
+func TestIsNotFound(t *testing.T) {
+	tests := []struct {
+		name string
+		err  error
+		want bool
+	}{
+		{"nil", nil, false},
+		{"vpc not found", errors.New(ErrorInvalidVpcIDNotFound), true},
+		{"subnet not found", errors.New(ErrorInvalidSubnetIDNotFound), true},
+		{"igw not found", errors.New(ErrorInvalidInternetGatewayIDNotFound), true},
+		{"natgw not found", errors.New(ErrorInvalidNatGatewayIDNotFound), true},
+		{"route table not found", errors.New(ErrorInvalidRouteTableIDNotFound), true},
+		{"eni not found", errors.New(ErrorInvalidNetworkInterfaceIDNotFound), true},
+		{"sg not found", errors.New(ErrorInvalidGroupNotFound), true},
+		{"volume not found", errors.New(ErrorInvalidVolumeNotFound), true},
+		{"wrapped not found", fmt.Errorf("delete cp vpc: %w", errors.New(ErrorInvalidVpcIDNotFound)), true},
+		{"dependency violation is not NotFound", errors.New(ErrorDependencyViolation), false},
+		{"in use is not NotFound", errors.New(ErrorInvalidNetworkInterfaceInUse), false},
+		{"server internal is not NotFound", errors.New(ErrorServerInternal), false},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if got := IsNotFound(tt.err); got != tt.want {
+				t.Errorf("IsNotFound(%v) = %v, want %v", tt.err, got, tt.want)
+			}
+		})
+	}
+}

spinifex/config/config.go

@@ -33,12 +33,8 @@ type ExternalPool struct {
 	DNSServers []string `mapstructure:"dns_servers"` // DNS servers for VM DHCP (auto-detected from host; fallback: 8.8.8.8, 1.1.1.1)
 	Region     string   `mapstructure:"region"`      // Scope to region (optional — empty means any region)
 	AZ         string   `mapstructure:"az"`          // Scope to AZ (optional — empty means any AZ in region)
-	// GwLrpRangeStart/End reserve a sub-range of the LAN for OVN gateway LRP IPs
-	// in centralized NAT mode (veth). Each VPC consumes one IP from this
-	// range so its gateway router port can ARP with a sender IP on the LAN
-	// subnet — link-local 169.254.0.1/30 makes upstream routers reject ARP per
-	// RFC 826. Must NOT overlap [RangeStart, RangeEnd] or IPAM and vpcd will
-	// fight over the same IP.
+	// GwLrpRangeStart/End reserve a sub-range for OVN gateway LRP IPs in centralized NAT mode.
+	// Must NOT overlap [RangeStart, RangeEnd] — link-local 169.254/16 is rejected by upstream routers.
 	GwLrpRangeStart string `mapstructure:"gw_lrp_range_start"`
 	GwLrpRangeEnd   string `mapstructure:"gw_lrp_range_end"`
 }
@@ -47,15 +43,12 @@ type ExternalPool struct {
 type NetworkConfig struct {
 	ExternalMode  string         `mapstructure:"external_mode"`  // "pool" or "" (disabled)
 	ExternalPools []ExternalPool `mapstructure:"external_pools"` // One or more IP pools
-	// IPSecEnabled toggles OVN native IPsec on every node for intra-AZ Geneve
-	// (AES-256-GCM via strongSwan). Default true. Operators can flip off for
-	// trusted single-rack lab topologies; production edge stays on.
+	// IPSecEnabled toggles OVN native IPsec (AES-256-GCM) on every node. Default true; disable only for trusted lab topologies.
 	IPSecEnabled bool `mapstructure:"ipsec_enabled"`
 }
 
 // BootstrapConfig holds the default VPC infrastructure IDs written by admin init.
-// vpcd reads this on startup to ensure OVN topology exists for the bootstrap VPC,
-// covering the case where admin init ran before services were started.
+// vpcd reads this on startup to ensure OVN topology exists for the bootstrap VPC.
 type BootstrapConfig struct {
 	AccountID  string `mapstructure:"account_id"`
 	VpcId      string `mapstructure:"vpc_id"`
@@ -70,8 +63,7 @@ type Config struct {
 	// Node config
 	Node string `json:"Node" mapstructure:"node"`
 	Host string `json:"Host" mapstructure:"host"` // Unique hostname or IP of this node
-	// AdvertiseIP is the off-host dial target for this node. Empty → callers
-	// fall back to Host for backward compat with pre-siv-8 cluster configs.
+	// AdvertiseIP is the off-host dial target. Empty falls back to Host for backward compat.
 	AdvertiseIP string   `json:"AdvertiseIP" mapstructure:"advertise"`
 	Region      string   `json:"Region" mapstructure:"region"`
 	AZ          string   `json:"AZ" mapstructure:"az"`
@@ -102,12 +94,8 @@ type AWSGWConfig struct {
 type ViperblockConfig struct {
 	ShardWAL *bool `json:"ShardWAL" mapstructure:"shardwal"` // Enable sharded WAL (default false when nil)
 
-	// EncryptionKeyFile is the path to the shared 32-byte AES-256 master key
-	// for viperblock at-rest encryption (chunks, WAL records, VBState/snapshot
-	// metadata). When empty, volumes are written in cleartext (legacy mode).
-	// When set, every VB constructed on this node — daemons and AWS handlers
-	// alike — must load it via masterkey.LoadShared so existing volumes can
-	// still be opened.
+	// EncryptionKeyFile is the path to the shared 32-byte AES-256 master key for viperblock at-rest encryption.
+	// Empty means cleartext. When set, all VB instances must load it via masterkey.LoadShared.
 	EncryptionKeyFile string `json:"EncryptionKeyFile" mapstructure:"encryption_key_file"`
 }
 
@@ -139,11 +127,9 @@ type GPUModelOverride struct {
 	Manufacturer string `json:"Manufacturer" mapstructure:"manufacturer"`
 	Name         string `json:"Name" mapstructure:"name"`
 	MemoryMiB    int64  `json:"MemoryMiB" mapstructure:"memory_mib"`
-	// XVGAOff forces x-vga=off for this GPU in QEMU passthrough, overriding the
-	// default (on for consumer GPUs, off for known datacenter cards).
+	// XVGAOff forces x-vga=off in QEMU passthrough, overriding the per-GPU default.
 	XVGAOff bool `json:"XVGAOff" mapstructure:"xvga_off"`
-	// MIGProfile overrides the daemon-level MIGProfile for this specific GPU.
-	// Takes the same format as DaemonConfig.MIGProfile (e.g. "1g.10gb").
+	// MIGProfile overrides the daemon-level MIGProfile for this GPU (same format, e.g. "1g.10gb").
 	MIGProfile string `json:"MIGProfile" mapstructure:"mig_profile"`
 }
 
@@ -156,9 +142,7 @@ type DaemonConfig struct {
 	MgmtBridge        string             `json:"MgmtBridge" mapstructure:"mgmt_bridge"`                // Linux bridge for system instance control plane (default "br-mgmt")
 	GPUPassthrough    bool               `json:"GPUPassthrough" mapstructure:"gpu_passthrough"`        // Enable VFIO GPU passthrough for g5.* instance types
 	GPUModelOverrides []GPUModelOverride `json:"GPUModelOverrides" mapstructure:"gpu_model_overrides"` // Dev/test GPU mappings not in the production model list
-	// MIGProfile enables NVIDIA MIG partitioning on all eligible GPUs on this node.
-	// Set to a profile name such as "1g.10gb" to activate; empty disables MIG.
-	// Individual GPUs can override this via GPUModelOverrides[].MIGProfile.
+	// MIGProfile enables NVIDIA MIG on all eligible GPUs (e.g. "1g.10gb"); empty disables. Per-GPU override via GPUModelOverrides[].MIGProfile.
 	MIGProfile string `json:"MIGProfile" mapstructure:"mig_profile"`
 }
 
@@ -180,8 +164,7 @@ type NATSSub struct {
 	Subject string `json:"Subject" mapstructure:"subject"`
 }
 
-// NodeBaseDir returns the BaseDir for the current node, or "" if the config
-// is nil, the node name is unset, or the node is not found in the Nodes map.
+// NodeBaseDir returns the BaseDir for the current node, or "" if config is nil, node is unset, or not found.
 func (cc *ClusterConfig) NodeBaseDir() string {
 	if cc == nil || cc.Node == "" {
 		slog.Warn("NodeBaseDir: no config or node name set, using global PID path")
@@ -201,8 +184,7 @@ func (cc *ClusterConfig) NodeBaseDir() string {
 // AllServices is the default service list when Services is empty (backward compat).
 var AllServices = []string{"nats", "predastore", "viperblock", "daemon", "awsgw", "vpcd", "ui"}
 
-// HasService reports whether the node runs the named service.
-// An empty Services list means all services (backward compat).
+// HasService reports whether the node runs the named service (empty list means all services).
 func (c Config) HasService(name string) bool {
 	services := c.Services
 	if len(services) == 0 {
@@ -225,8 +207,7 @@ func LoadConfig(configPath string) (*ClusterConfig, error) {
 	viper.SetEnvPrefix("SPINIFEX")
 	viper.AutomaticEnv()
 
-	// network.ipsec_enabled defaults to true (intra-AZ Geneve encrypted via OVN
-	// native IPsec). Operators must explicitly set false to disable.
+	// Default ipsec_enabled to true; operators must explicitly set false to disable.
 	viper.SetDefault("network.ipsec_enabled", true)
 
 	// Try to load config file if it exists
@@ -251,9 +232,7 @@ func LoadConfig(configPath string) (*ClusterConfig, error) {
 		return nil, fmt.Errorf("error unmarshaling config: %w", err)
 	}
 
-	// Normalize the local node's bind address: 0.0.0.0 means "listen on all
-	// interfaces" but is not a valid connect address. Only rewrite for the
-	// local node — remote nodes use real IPs that must not be changed.
+	// Rewrite 0.0.0.0 in Predastore.Host to 127.0.0.1 for the local node only (not a valid connect address).
 	if local, ok := config.Nodes[config.Node]; ok {
 		if strings.HasPrefix(local.Predastore.Host, "0.0.0.0") {
 			local.Predastore.Host = strings.Replace(local.Predastore.Host, "0.0.0.0", "127.0.0.1", 1)
@@ -268,9 +247,7 @@ func LoadConfig(configPath string) (*ClusterConfig, error) {
 	return &config, nil
 }
 
-// validateClusterConfig rejects legacy upstream-DHCP config and validates
-// external pool ranges. The upstream-DHCP-client model was replaced by static
-// WAN-pool allocation; legacy keys must be removed from the operator's TOML.
+// validateClusterConfig rejects legacy DHCP config keys and validates external pool ranges.
 func validateClusterConfig(cc *ClusterConfig) error {
 	if viper.IsSet("network.external_dhcp") {
 		return fmt.Errorf("config: [network] external_dhcp is no longer supported; remove the key (static WAN-pool allocation only)")