feat(eks): HA control plane — 3-node etcd quorum spread across distinct hosts (#335)

* feat(eks): node-targetable control-plane VM (HA spread transport prereq)

Groundwork for HA control-plane placement (mulga-siv-231.7.1): make the k3s
control-plane VM placeable on a specific Spinifex host.

- Add a sys.medium instance type (2 vCPU / 4 GB) and boot the control-plane VM
  as it instead of t3.medium. A sys.* type keeps the CP VM out of customer
  DescribeInstanceTypes and makes the daemon register the node-targeted
  system.LaunchInstance.{type}.{nodeID} subject (only sys.* types get it).
- Add K3sServerInput.TargetNodeID and a LaunchSystemInstanceOnNode method on
  the launcher: empty/local node launches in-process (unchanged single-CP
  path); any other node dispatches over the node-targeted subject, where the
  receiving daemon runs the launch and owns the VM.

The spread orchestrator (reserve N hosts, launch per host, verify, rollback,
<3-host fallback) lands separately; CreateCluster still launches locally
(TargetNodeID empty), so behaviour is unchanged.

* feat(eks): HA control-plane spread orchestrator

Place N=3 k3s server VMs across distinct Spinifex hosts via a per-cluster
spread placement group, with all-or-nothing rollback and single-CP fallback.

- placeControlPlane: capacity fan-out (HostScheduler) -> >=3-host gate ->
  ensure spread group (system account) -> ReserveSpreadNodes(Min=Max=3) ->
  parallel per-node LaunchK3sServerVM(TargetNodeID) -> node-vms placement
  verify -> FinalizeSpreadInstances. Any launch/verify/finalize failure
  terminates launched VMs, releases the reservation, drops the group.
- <3 schedulable hosts or a lost reservation race falls back to a single
  local control-plane VM (prior behaviour).
- ClusterMeta gains ControlPlaneNodes[] + ControlPlaneSpreadGroup; [0] is
  mirrored into the scalar ControlPlane* fields for back-compat. DeleteCluster
  tears down every CP VM and the spread group.
- Wire PlacementGroup + Scheduler into EKSServiceDeps + orchestration-deps gate.

* fix(eks): size HA control-plane spread on raw host headroom

SchedulableHosts matched the CP type by name in node.status's per-type
capacity list, but that list omits sys.* types (system types are hidden
from customers). The CP VM is sys.medium, so the match never hit and
SchedulableHosts returned 0 on every node — placeControlPlane always fell
back to a single CP, making the >=3-host spread unreachable in production.

System VMs consume host vCPU/memory like any guest, so size the fit against
each node's raw schedulable headroom (Total - Reserved - Alloc) and the
type's footprint instead. Adds instancetypes.SpecForSystemType for the
sys.* vCPU/memory spec and a pure nodeFitsSystemInstance helper.

* feat(eks): k3s server-join role for HA control-plane quorum

Servers 2..N now boot in join mode: server: https://<first-server>:6443 +
a shared cluster token, without cluster-init, so they join the first server's
embedded-etcd quorum instead of each cluster-initing an independent datastore.

- K3sServerInput gains JoinToken (shared cluster secret, set on every server)
  and ServerURL (set => join server, no cluster-init, server-join role).
- buildK3sUserData branches cluster-init vs server:, emits token:, and selects
  SPINIFEX_K3S_ROLE server vs server-join.
- launchControlPlaneSpread launches nodes[0] first (cluster-init), derives the
  join URL from its ENI IP, then launches the rest in parallel as join servers;
  a first-server failure skips the joins and rolls back.
- CreateCluster mints a per-cluster crypto-random token (GenerateK3sClusterToken)
  threaded into the placement template.
- eks-node-role.sh adds a server-join case: token-webhook + k3s + state-report,
  but not k3s-first-boot (the first server already publishes the bootstrap
  artifacts; a join re-publish only races the bootstrap bus).

* fix: open etcd-peer/kubelet ingress on EKS control-plane SG for HA join

Join servers (231.7.2) register over the supervisor port 6443 but their
embedded etcd could not peer with the quorum: the control-plane SG only
admitted 6443 from the VPC CIDR, omitting etcd client+peer 2379-2380 and
kubelet 10250 between control-plane nodes. A single cluster-init server never
exercises those ports, so the gap stayed hidden until multi-server HA — the
join registered but its etcd never replicated the raft log, leaving the node
perpetually NotReady.

Add EnsureControlPlaneHAIngress with self-referencing (cpSG->cpSG) rules for
etcd 2379-2380 and kubelet 10250, called from CreateCluster after
EnsureControlPlaneIngress. Verified live on a 3-node cluster: join servers
reach Ready and form a 3-member etcd quorum that survives loss of the original
first server.

* feat: register all control-plane ENI IPs as EKS cluster NLB targets

Register every HA control-plane server ENI IP in the cluster NLB target
group, not just the primary. The TG already health-checks :6443, so a
dead member drains and kubectl/apiserver traffic fails over to surviving
etcd-quorum members. Add RegisterClusterTargets batch helper; the single
RegisterClusterTarget now delegates to it.

scripts/images/eks-node/eks-node-role.sh

@@ -10,8 +10,13 @@ set -eu
 # ROLE_FILE so later boots and the etcd-snapshot cron can branch without
 # re-parsing, then removes itself from the runlevel so it runs exactly once.
 #
-#   server → eks-token-webhook, k3s (server), k3s-first-boot (bootstrap publish)
-#   agent  → k3s-agent
+#   server      → eks-token-webhook, k3s (server), k3s-first-boot (bootstrap
+#                 publish), mulga-eks-state-report
+#   server-join → eks-token-webhook, k3s (server, joins the first server's etcd
+#                 quorum), mulga-eks-state-report. NO k3s-first-boot: the first
+#                 server already publishes the cluster-identical bootstrap
+#                 artifacts; a join re-publish only races the bootstrap bus.
+#   agent       → k3s-agent
 #
 # Paths are overridable via env so the selection logic is unit-testable (bats)
 # without root or a real /etc.
@@ -55,6 +60,15 @@ case "${ROLE}" in
         rc-service k3s-first-boot start
         rc-service mulga-eks-state-report start
         ;;
+    server-join)
+        log "configuring server-join role"
+        rc-update add eks-token-webhook default
+        rc-update add k3s default
+        rc-update add mulga-eks-state-report default
+        rc-service eks-token-webhook start
+        rc-service k3s start
+        rc-service mulga-eks-state-report start
+        ;;
     agent)
         log "configuring agent role"
         rc-update add k3s-agent default

scripts/images/eks-node/eks-node-role_test.sh

@@ -66,6 +66,20 @@ check "server: starts k3s" "yes" "$(grep -q 'rc-service k3s start' "${CALLS}" &&
 check "server: no k3s-agent" "no" "$(grep -q 'k3s-agent' "${CALLS}" && echo yes || echo no)"
 check "server: self-disables" "yes" "$(grep -q 'rc-update del eks-node-role default' "${CALLS}" && echo yes || echo no)"
 
+# --- Case 1b: explicit server-join role ---
+run_case server_join
+printf 'SPINIFEX_K3S_ROLE=server-join\nEKS_CLUSTER_NAME=alpha\n' > "${EKS_NODE_ENVFILE}"
+invoke
+check "server-join: exit 0" 0 "${rc}"
+check "server-join: role file" "server-join" "$(cat "${ROLE_FILE}")"
+check "server-join: enables webhook" "yes" "$(grep -q 'rc-update add eks-token-webhook default' "${CALLS}" && echo yes || echo no)"
+check "server-join: enables k3s" "yes" "$(grep -q 'rc-update add k3s default' "${CALLS}" && echo yes || echo no)"
+check "server-join: enables state-report" "yes" "$(grep -q 'rc-update add mulga-eks-state-report default' "${CALLS}" && echo yes || echo no)"
+# Join servers must NOT re-publish bootstrap; the first server already did.
+check "server-join: no k3s-first-boot" "no" "$(grep -q 'k3s-first-boot' "${CALLS}" && echo yes || echo no)"
+check "server-join: no k3s-agent" "no" "$(grep -q 'k3s-agent' "${CALLS}" && echo yes || echo no)"
+check "server-join: self-disables" "yes" "$(grep -q 'rc-update del eks-node-role default' "${CALLS}" && echo yes || echo no)"
+
 # --- Case 2: explicit agent role ---
 run_case agent_explicit
 printf 'SPINIFEX_K3S_ROLE=agent\n' > "${EKS_NODE_ENVFILE}"