mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Josh Siviour
74b91b5cc5
* feat(eks): node-targetable control-plane VM (HA spread transport prereq)
Groundwork for HA control-plane placement (mulga-siv-231.7.1): make the k3s
control-plane VM placeable on a specific Spinifex host.
- Add a sys.medium instance type (2 vCPU / 4 GB) and boot the control-plane VM
as it instead of t3.medium. A sys.* type keeps the CP VM out of customer
DescribeInstanceTypes and makes the daemon register the node-targeted
system.LaunchInstance.{type}.{nodeID} subject (only sys.* types get it).
- Add K3sServerInput.TargetNodeID and a LaunchSystemInstanceOnNode method on
the launcher: empty/local node launches in-process (unchanged single-CP
path); any other node dispatches over the node-targeted subject, where the
receiving daemon runs the launch and owns the VM.
The spread orchestrator (reserve N hosts, launch per host, verify, rollback,
<3-host fallback) lands separately; CreateCluster still launches locally
(TargetNodeID empty), so behaviour is unchanged.
* feat(eks): HA control-plane spread orchestrator
Place N=3 k3s server VMs across distinct Spinifex hosts via a per-cluster
spread placement group, with all-or-nothing rollback and single-CP fallback.
- placeControlPlane: capacity fan-out (HostScheduler) -> >=3-host gate ->
ensure spread group (system account) -> ReserveSpreadNodes(Min=Max=3) ->
parallel per-node LaunchK3sServerVM(TargetNodeID) -> node-vms placement
verify -> FinalizeSpreadInstances. Any launch/verify/finalize failure
terminates launched VMs, releases the reservation, drops the group.
- <3 schedulable hosts or a lost reservation race falls back to a single
local control-plane VM (prior behaviour).
- ClusterMeta gains ControlPlaneNodes[] + ControlPlaneSpreadGroup; [0] is
mirrored into the scalar ControlPlane* fields for back-compat. DeleteCluster
tears down every CP VM and the spread group.
- Wire PlacementGroup + Scheduler into EKSServiceDeps + orchestration-deps gate.
* fix(eks): size HA control-plane spread on raw host headroom
SchedulableHosts matched the CP type by name in node.status's per-type
capacity list, but that list omits sys.* types (system types are hidden
from customers). The CP VM is sys.medium, so the match never hit and
SchedulableHosts returned 0 on every node — placeControlPlane always fell
back to a single CP, making the >=3-host spread unreachable in production.
System VMs consume host vCPU/memory like any guest, so size the fit against
each node's raw schedulable headroom (Total - Reserved - Alloc) and the
type's footprint instead. Adds instancetypes.SpecForSystemType for the
sys.* vCPU/memory spec and a pure nodeFitsSystemInstance helper.
* feat(eks): k3s server-join role for HA control-plane quorum
Servers 2..N now boot in join mode: server: https://<first-server>:6443 +
a shared cluster token, without cluster-init, so they join the first server's
embedded-etcd quorum instead of each cluster-initing an independent datastore.
- K3sServerInput gains JoinToken (shared cluster secret, set on every server)
and ServerURL (set => join server, no cluster-init, server-join role).
- buildK3sUserData branches cluster-init vs server:, emits token:, and selects
SPINIFEX_K3S_ROLE server vs server-join.
- launchControlPlaneSpread launches nodes[0] first (cluster-init), derives the
join URL from its ENI IP, then launches the rest in parallel as join servers;
a first-server failure skips the joins and rolls back.
- CreateCluster mints a per-cluster crypto-random token (GenerateK3sClusterToken)
threaded into the placement template.
- eks-node-role.sh adds a server-join case: token-webhook + k3s + state-report,
but not k3s-first-boot (the first server already publishes the bootstrap
artifacts; a join re-publish only races the bootstrap bus).
* fix: open etcd-peer/kubelet ingress on EKS control-plane SG for HA join
Join servers (231.7.2) register over the supervisor port 6443 but their
embedded etcd could not peer with the quorum: the control-plane SG only
admitted 6443 from the VPC CIDR, omitting etcd client+peer 2379-2380 and
kubelet 10250 between control-plane nodes. A single cluster-init server never
exercises those ports, so the gap stayed hidden until multi-server HA — the
join registered but its etcd never replicated the raft log, leaving the node
perpetually NotReady.
Add EnsureControlPlaneHAIngress with self-referencing (cpSG->cpSG) rules for
etcd 2379-2380 and kubelet 10250, called from CreateCluster after
EnsureControlPlaneIngress. Verified live on a 3-node cluster: join servers
reach Ready and form a 3-member etcd quorum that survives loss of the original
first server.
* feat: register all control-plane ENI IPs as EKS cluster NLB targets
Register every HA control-plane server ENI IP in the cluster NLB target
group, not just the primary. The TG already health-checks :6443, so a
dead member drains and kubectl/apiserver traffic fails over to surviving
etcd-quorum members. Add RegisterClusterTargets batch helper; the single
RegisterClusterTarget now delegates to it.
86 lines
3.2 KiB
Bash
Executable File
86 lines
3.2 KiB
Bash
Executable File
#!/bin/sh
|
|
set -eu
|
|
|
|
# eks-node-role — first-boot role selector for the unified eks-node AMI.
|
|
#
|
|
# The same image boots either as a K3s control-plane server or a worker agent;
|
|
# the role is chosen per-instance at first boot from SPINIFEX_K3S_ROLE in the
|
|
# cloud-init env file. This script enables + starts the matching OpenRC services
|
|
# (none are in the default runlevel at bake time), records the resolved role to
|
|
# ROLE_FILE so later boots and the etcd-snapshot cron can branch without
|
|
# re-parsing, then removes itself from the runlevel so it runs exactly once.
|
|
#
|
|
# server → eks-token-webhook, k3s (server), k3s-first-boot (bootstrap
|
|
# publish), mulga-eks-state-report
|
|
# server-join → eks-token-webhook, k3s (server, joins the first server's etcd
|
|
# quorum), mulga-eks-state-report. NO k3s-first-boot: the first
|
|
# server already publishes the cluster-identical bootstrap
|
|
# artifacts; a join re-publish only races the bootstrap bus.
|
|
# agent → k3s-agent
|
|
#
|
|
# Paths are overridable via env so the selection logic is unit-testable (bats)
|
|
# without root or a real /etc.
|
|
|
|
ROLE_FILE="${EKS_NODE_ROLE_FILE:-/etc/spinifex-eks/role}"
|
|
ENVFILE="${EKS_NODE_ENVFILE:-/etc/spinifex-eks/first-boot.env}"
|
|
AGENT_ENVFILE="${EKS_NODE_AGENT_ENVFILE:-/etc/spinifex-eks/agent.env}"
|
|
LOGTAG="eks-node-role"
|
|
|
|
log() { echo "[${LOGTAG}] $*"; }
|
|
die() { log "ERROR: $*"; exit 1; }
|
|
|
|
# Already resolved on a prior boot — the role services are in the runlevel and
|
|
# this service should have been pulled, but guard anyway so a stray re-run is a
|
|
# no-op rather than a double-enable.
|
|
if [ -f "${ROLE_FILE}" ]; then
|
|
log "role already resolved ($(cat "${ROLE_FILE}")); nothing to do"
|
|
exit 0
|
|
fi
|
|
|
|
# Resolve the role. Server seeds SPINIFEX_K3S_ROLE in first-boot.env; a worker
|
|
# seed that predates the explicit var is inferred from the presence of agent.env
|
|
# (K3S_URL/K3S_TOKEN live there).
|
|
ROLE=""
|
|
if [ -f "${ENVFILE}" ]; then
|
|
ROLE=$(. "${ENVFILE}"; printf '%s' "${SPINIFEX_K3S_ROLE:-}")
|
|
fi
|
|
if [ -z "${ROLE}" ] && [ -f "${AGENT_ENVFILE}" ]; then
|
|
ROLE="agent"
|
|
fi
|
|
|
|
case "${ROLE}" in
|
|
server)
|
|
log "configuring server role"
|
|
rc-update add eks-token-webhook default
|
|
rc-update add k3s default
|
|
rc-update add k3s-first-boot default
|
|
rc-update add mulga-eks-state-report default
|
|
rc-service eks-token-webhook start
|
|
rc-service k3s start
|
|
rc-service k3s-first-boot start
|
|
rc-service mulga-eks-state-report start
|
|
;;
|
|
server-join)
|
|
log "configuring server-join role"
|
|
rc-update add eks-token-webhook default
|
|
rc-update add k3s default
|
|
rc-update add mulga-eks-state-report default
|
|
rc-service eks-token-webhook start
|
|
rc-service k3s start
|
|
rc-service mulga-eks-state-report start
|
|
;;
|
|
agent)
|
|
log "configuring agent role"
|
|
rc-update add k3s-agent default
|
|
rc-service k3s-agent start
|
|
;;
|
|
*)
|
|
die "SPINIFEX_K3S_ROLE missing or invalid: '${ROLE}'"
|
|
;;
|
|
esac
|
|
|
|
mkdir -p "$(dirname "${ROLE_FILE}")"
|
|
printf '%s\n' "${ROLE}" > "${ROLE_FILE}"
|
|
rc-update del eks-node-role default 2>/dev/null || true
|
|
log "role '${ROLE}' configured"
|