mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
74b91b5cc5
* feat(eks): node-targetable control-plane VM (HA spread transport prereq)
Groundwork for HA control-plane placement (mulga-siv-231.7.1): make the k3s
control-plane VM placeable on a specific Spinifex host.
- Add a sys.medium instance type (2 vCPU / 4 GB) and boot the control-plane VM
as it instead of t3.medium. A sys.* type keeps the CP VM out of customer
DescribeInstanceTypes and makes the daemon register the node-targeted
system.LaunchInstance.{type}.{nodeID} subject (only sys.* types get it).
- Add K3sServerInput.TargetNodeID and a LaunchSystemInstanceOnNode method on
the launcher: empty/local node launches in-process (unchanged single-CP
path); any other node dispatches over the node-targeted subject, where the
receiving daemon runs the launch and owns the VM.
The spread orchestrator (reserve N hosts, launch per host, verify, rollback,
<3-host fallback) lands separately; CreateCluster still launches locally
(TargetNodeID empty), so behaviour is unchanged.
* feat(eks): HA control-plane spread orchestrator
Place N=3 k3s server VMs across distinct Spinifex hosts via a per-cluster
spread placement group, with all-or-nothing rollback and single-CP fallback.
- placeControlPlane: capacity fan-out (HostScheduler) -> >=3-host gate ->
ensure spread group (system account) -> ReserveSpreadNodes(Min=Max=3) ->
parallel per-node LaunchK3sServerVM(TargetNodeID) -> node-vms placement
verify -> FinalizeSpreadInstances. Any launch/verify/finalize failure
terminates launched VMs, releases the reservation, drops the group.
- <3 schedulable hosts or a lost reservation race falls back to a single
local control-plane VM (prior behaviour).
- ClusterMeta gains ControlPlaneNodes[] + ControlPlaneSpreadGroup; [0] is
mirrored into the scalar ControlPlane* fields for back-compat. DeleteCluster
tears down every CP VM and the spread group.
- Wire PlacementGroup + Scheduler into EKSServiceDeps + orchestration-deps gate.
* fix(eks): size HA control-plane spread on raw host headroom
SchedulableHosts matched the CP type by name in node.status's per-type
capacity list, but that list omits sys.* types (system types are hidden
from customers). The CP VM is sys.medium, so the match never hit and
SchedulableHosts returned 0 on every node — placeControlPlane always fell
back to a single CP, making the >=3-host spread unreachable in production.
System VMs consume host vCPU/memory like any guest, so size the fit against
each node's raw schedulable headroom (Total - Reserved - Alloc) and the
type's footprint instead. Adds instancetypes.SpecForSystemType for the
sys.* vCPU/memory spec and a pure nodeFitsSystemInstance helper.
* feat(eks): k3s server-join role for HA control-plane quorum
Servers 2..N now boot in join mode: server: https://<first-server>:6443 +
a shared cluster token, without cluster-init, so they join the first server's
embedded-etcd quorum instead of each cluster-initing an independent datastore.
- K3sServerInput gains JoinToken (shared cluster secret, set on every server)
and ServerURL (set => join server, no cluster-init, server-join role).
- buildK3sUserData branches cluster-init vs server:, emits token:, and selects
SPINIFEX_K3S_ROLE server vs server-join.
- launchControlPlaneSpread launches nodes[0] first (cluster-init), derives the
join URL from its ENI IP, then launches the rest in parallel as join servers;
a first-server failure skips the joins and rolls back.
- CreateCluster mints a per-cluster crypto-random token (GenerateK3sClusterToken)
threaded into the placement template.
- eks-node-role.sh adds a server-join case: token-webhook + k3s + state-report,
but not k3s-first-boot (the first server already publishes the bootstrap
artifacts; a join re-publish only races the bootstrap bus).
* fix: open etcd-peer/kubelet ingress on EKS control-plane SG for HA join
Join servers (231.7.2) register over the supervisor port 6443 but their
embedded etcd could not peer with the quorum: the control-plane SG only
admitted 6443 from the VPC CIDR, omitting etcd client+peer 2379-2380 and
kubelet 10250 between control-plane nodes. A single cluster-init server never
exercises those ports, so the gap stayed hidden until multi-server HA — the
join registered but its etcd never replicated the raft log, leaving the node
perpetually NotReady.
Add EnsureControlPlaneHAIngress with self-referencing (cpSG->cpSG) rules for
etcd 2379-2380 and kubelet 10250, called from CreateCluster after
EnsureControlPlaneIngress. Verified live on a 3-node cluster: join servers
reach Ready and form a 3-member etcd quorum that survives loss of the original
first server.
* feat: register all control-plane ENI IPs as EKS cluster NLB targets
Register every HA control-plane server ENI IP in the cluster NLB target
group, not just the primary. The TG already health-checks :6443, so a
dead member drains and kubectl/apiserver traffic fails over to surviving
etcd-quorum members. Add RegisterClusterTargets batch helper; the single
RegisterClusterTarget now delegates to it.
118 lines
5.0 KiB
Bash
Executable File
118 lines
5.0 KiB
Bash
Executable File
#!/bin/sh
|
|
# Self-contained POSIX test for eks-node-role.sh role selection. No bats/root
|
|
# needed: rc-update / rc-service are stubbed on PATH and record their calls; the
|
|
# script's file paths are redirected into a temp dir via its EKS_NODE_* env knobs.
|
|
#
|
|
# Run: sh scripts/images/eks-node/eks-node-role_test.sh
|
|
set -eu
|
|
|
|
SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)
|
|
SELECTOR="${SCRIPT_DIR}/eks-node-role.sh"
|
|
WORK=$(mktemp -d)
|
|
trap 'rm -rf "${WORK}"' EXIT
|
|
|
|
# Stub rc-update / rc-service: append "cmd args" to ${CALLS} when invoked.
|
|
STUBBIN="${WORK}/bin"
|
|
mkdir -p "${STUBBIN}"
|
|
for tool in rc-update rc-service; do
|
|
cat > "${STUBBIN}/${tool}" <<EOF
|
|
#!/bin/sh
|
|
echo "${tool} \$*" >> "\${CALLS}"
|
|
exit 0
|
|
EOF
|
|
chmod +x "${STUBBIN}/${tool}"
|
|
done
|
|
|
|
PASS=0
|
|
FAIL=0
|
|
check() { # check <desc> <expected> <actual>
|
|
if [ "$2" = "$3" ]; then
|
|
PASS=$((PASS + 1))
|
|
else
|
|
FAIL=$((FAIL + 1))
|
|
echo "FAIL: $1"
|
|
echo " expected: [$2]"
|
|
echo " actual: [$3]"
|
|
fi
|
|
}
|
|
|
|
# run_case <name>: resets a per-case dir, exports the knobs; caller seeds env
|
|
# files then invokes the selector. Sets CASE (dir), CALLS (log), ROLE_FILE.
|
|
run_case() {
|
|
CASE="${WORK}/$1"
|
|
mkdir -p "${CASE}"
|
|
CALLS="${CASE}/calls"
|
|
: > "${CALLS}"
|
|
ROLE_FILE="${CASE}/role"
|
|
export CALLS
|
|
export EKS_NODE_ROLE_FILE="${ROLE_FILE}"
|
|
export EKS_NODE_ENVFILE="${CASE}/first-boot.env"
|
|
export EKS_NODE_AGENT_ENVFILE="${CASE}/agent.env"
|
|
}
|
|
|
|
# Run the selector, capturing its exit code in ${rc} without tripping set -e.
|
|
invoke() { rc=0; PATH="${STUBBIN}:${PATH}" sh "${SELECTOR}" >/dev/null 2>&1 || rc=$?; }
|
|
|
|
# --- Case 1: explicit server role ---
|
|
run_case server
|
|
printf 'SPINIFEX_K3S_ROLE=server\nEKS_CLUSTER_NAME=alpha\n' > "${EKS_NODE_ENVFILE}"
|
|
invoke
|
|
check "server: exit 0" 0 "${rc}"
|
|
check "server: role file" "server" "$(cat "${ROLE_FILE}")"
|
|
check "server: enables webhook" "yes" "$(grep -q 'rc-update add eks-token-webhook default' "${CALLS}" && echo yes || echo no)"
|
|
check "server: enables k3s" "yes" "$(grep -q 'rc-update add k3s default' "${CALLS}" && echo yes || echo no)"
|
|
check "server: enables k3s-first-boot" "yes" "$(grep -q 'rc-update add k3s-first-boot default' "${CALLS}" && echo yes || echo no)"
|
|
check "server: starts k3s" "yes" "$(grep -q 'rc-service k3s start' "${CALLS}" && echo yes || echo no)"
|
|
check "server: no k3s-agent" "no" "$(grep -q 'k3s-agent' "${CALLS}" && echo yes || echo no)"
|
|
check "server: self-disables" "yes" "$(grep -q 'rc-update del eks-node-role default' "${CALLS}" && echo yes || echo no)"
|
|
|
|
# --- Case 1b: explicit server-join role ---
|
|
run_case server_join
|
|
printf 'SPINIFEX_K3S_ROLE=server-join\nEKS_CLUSTER_NAME=alpha\n' > "${EKS_NODE_ENVFILE}"
|
|
invoke
|
|
check "server-join: exit 0" 0 "${rc}"
|
|
check "server-join: role file" "server-join" "$(cat "${ROLE_FILE}")"
|
|
check "server-join: enables webhook" "yes" "$(grep -q 'rc-update add eks-token-webhook default' "${CALLS}" && echo yes || echo no)"
|
|
check "server-join: enables k3s" "yes" "$(grep -q 'rc-update add k3s default' "${CALLS}" && echo yes || echo no)"
|
|
check "server-join: enables state-report" "yes" "$(grep -q 'rc-update add mulga-eks-state-report default' "${CALLS}" && echo yes || echo no)"
|
|
# Join servers must NOT re-publish bootstrap; the first server already did.
|
|
check "server-join: no k3s-first-boot" "no" "$(grep -q 'k3s-first-boot' "${CALLS}" && echo yes || echo no)"
|
|
check "server-join: no k3s-agent" "no" "$(grep -q 'k3s-agent' "${CALLS}" && echo yes || echo no)"
|
|
check "server-join: self-disables" "yes" "$(grep -q 'rc-update del eks-node-role default' "${CALLS}" && echo yes || echo no)"
|
|
|
|
# --- Case 2: explicit agent role ---
|
|
run_case agent_explicit
|
|
printf 'SPINIFEX_K3S_ROLE=agent\n' > "${EKS_NODE_ENVFILE}"
|
|
invoke
|
|
check "agent: exit 0" 0 "${rc}"
|
|
check "agent: role file" "agent" "$(cat "${ROLE_FILE}")"
|
|
check "agent: enables k3s-agent" "yes" "$(grep -q 'rc-update add k3s-agent default' "${CALLS}" && echo yes || echo no)"
|
|
check "agent: no server services" "no" "$(grep -qE 'eks-token-webhook|k3s-first-boot|add k3s default' "${CALLS}" && echo yes || echo no)"
|
|
|
|
# --- Case 3: agent inferred from agent.env (no explicit role) ---
|
|
run_case agent_inferred
|
|
printf 'K3S_URL=https://nlb:443\nK3S_TOKEN=abc\n' > "${EKS_NODE_AGENT_ENVFILE}"
|
|
invoke
|
|
check "infer: exit 0" 0 "${rc}"
|
|
check "infer: role file" "agent" "$(cat "${ROLE_FILE}")"
|
|
check "infer: enables k3s-agent" "yes" "$(grep -q 'rc-update add k3s-agent default' "${CALLS}" && echo yes || echo no)"
|
|
|
|
# --- Case 4: missing role + no env files → fail, no role file, no services ---
|
|
run_case missing
|
|
invoke
|
|
check "missing: nonzero exit" "1" "${rc}"
|
|
check "missing: no role file" "absent" "$([ -f "${ROLE_FILE}" ] && echo present || echo absent)"
|
|
check "missing: no rc calls" "" "$(cat "${CALLS}")"
|
|
|
|
# --- Case 5: already resolved (role file present) → no-op, exit 0 ---
|
|
run_case resolved
|
|
printf 'SPINIFEX_K3S_ROLE=server\n' > "${EKS_NODE_ENVFILE}"
|
|
printf 'server\n' > "${ROLE_FILE}"
|
|
invoke
|
|
check "resolved: exit 0" 0 "${rc}"
|
|
check "resolved: no rc calls" "" "$(cat "${CALLS}")"
|
|
|
|
echo "---"
|
|
echo "PASS=${PASS} FAIL=${FAIL}"
|
|
[ "${FAIL}" -eq 0 ]
|