mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Josh Siviour
a22bc2f716
* feat(eks): nodegroup provisioning + K3s agent join (6d)
CreateNodegroup launches N customer-owned worker EC2 from the unified
eks-node AMI in agent role; each joins the cluster via K3S_URL + K3S_TOKEN
baked into cloud-init (first-boot.env SPINIFEX_K3S_ROLE=agent + agent.env).
Workers are customer-owned and DescribeInstances-visible (no ManagedBy,
no mgmt NIC) via the normal RunInstances path, matching real EKS.
K3S_URL targets the control-plane ENI's in-VPC private IP on :6443 — the
cluster NLB DNS endpoint is not resolvable in-VPC yet; switch to the NLB
endpoint on :443 once authoritative in-VPC DNS lands. K3S_TOKEN is the
decrypted cluster node-token persisted at server bootstrap.
Describe/List/Update(scaling)/Delete round out CRUD. Cluster<->nodegroup
SG ingress rules (apiserver 6443, flannel VXLAN 8472/udp, kubelet 10250,
intra-nodegroup) authorized idempotently. UpdateNodegroupVersion deferred.
* feat: gate EKS cluster health via NATS state reports
The K3s apiserver binds the VPC node-ip, unreachable from the host-side
spinifex daemon, so the reconciler cannot probe /healthz directly. Add a
control-plane state reporter that publishes {healthz, node_count} over the
management NATS bus the daemon already shares.
- eks-node server role runs mulga-eks-state-report as an OpenRC service,
publishing eks.state.{account}.{cluster}.server every 30s.
- ClusterReconciler subscribes to that subject, gates CREATING -> ACTIVE on
a healthy report, and records Ready node_count into ClusterMeta.
- node_count counts Ready nodes only; k3s retains terminated workers as
NotReady until pruned, so a raw count never drops on scale-down.
72 lines
2.5 KiB
Bash
Executable File
72 lines
2.5 KiB
Bash
Executable File
#!/bin/sh
|
|
set -eu
|
|
|
|
# eks-node-role — first-boot role selector for the unified eks-node AMI.
|
|
#
|
|
# The same image boots either as a K3s control-plane server or a worker agent;
|
|
# the role is chosen per-instance at first boot from SPINIFEX_K3S_ROLE in the
|
|
# cloud-init env file. This script enables + starts the matching OpenRC services
|
|
# (none are in the default runlevel at bake time), records the resolved role to
|
|
# ROLE_FILE so later boots and the etcd-snapshot cron can branch without
|
|
# re-parsing, then removes itself from the runlevel so it runs exactly once.
|
|
#
|
|
# server → eks-token-webhook, k3s (server), k3s-first-boot (bootstrap publish)
|
|
# agent → k3s-agent
|
|
#
|
|
# Paths are overridable via env so the selection logic is unit-testable (bats)
|
|
# without root or a real /etc.
|
|
|
|
ROLE_FILE="${EKS_NODE_ROLE_FILE:-/etc/spinifex-eks/role}"
|
|
ENVFILE="${EKS_NODE_ENVFILE:-/etc/spinifex-eks/first-boot.env}"
|
|
AGENT_ENVFILE="${EKS_NODE_AGENT_ENVFILE:-/etc/spinifex-eks/agent.env}"
|
|
LOGTAG="eks-node-role"
|
|
|
|
log() { echo "[${LOGTAG}] $*"; }
|
|
die() { log "ERROR: $*"; exit 1; }
|
|
|
|
# Already resolved on a prior boot — the role services are in the runlevel and
|
|
# this service should have been pulled, but guard anyway so a stray re-run is a
|
|
# no-op rather than a double-enable.
|
|
if [ -f "${ROLE_FILE}" ]; then
|
|
log "role already resolved ($(cat "${ROLE_FILE}")); nothing to do"
|
|
exit 0
|
|
fi
|
|
|
|
# Resolve the role. Server seeds SPINIFEX_K3S_ROLE in first-boot.env; a worker
|
|
# seed that predates the explicit var is inferred from the presence of agent.env
|
|
# (K3S_URL/K3S_TOKEN live there).
|
|
ROLE=""
|
|
if [ -f "${ENVFILE}" ]; then
|
|
ROLE=$(. "${ENVFILE}"; printf '%s' "${SPINIFEX_K3S_ROLE:-}")
|
|
fi
|
|
if [ -z "${ROLE}" ] && [ -f "${AGENT_ENVFILE}" ]; then
|
|
ROLE="agent"
|
|
fi
|
|
|
|
case "${ROLE}" in
|
|
server)
|
|
log "configuring server role"
|
|
rc-update add eks-token-webhook default
|
|
rc-update add k3s default
|
|
rc-update add k3s-first-boot default
|
|
rc-update add mulga-eks-state-report default
|
|
rc-service eks-token-webhook start
|
|
rc-service k3s start
|
|
rc-service k3s-first-boot start
|
|
rc-service mulga-eks-state-report start
|
|
;;
|
|
agent)
|
|
log "configuring agent role"
|
|
rc-update add k3s-agent default
|
|
rc-service k3s-agent start
|
|
;;
|
|
*)
|
|
die "SPINIFEX_K3S_ROLE missing or invalid: '${ROLE}'"
|
|
;;
|
|
esac
|
|
|
|
mkdir -p "$(dirname "${ROLE_FILE}")"
|
|
printf '%s\n' "${ROLE}" > "${ROLE_FILE}"
|
|
rc-update del eks-node-role default 2>/dev/null || true
|
|
log "role '${ROLE}' configured"
|