Files
spinifex/scripts/images/eks-node/setup.sh
T
Josh Siviour bc9556d1a9 fix(eks): managed-CP worker data-plane — apiserver reachability, node-local CoreDNS, SG teardown (#368)
* fix: persist instance TagSpecifications in RunInstance

RunInstance built the ec2.Instance metadata but never read
input.TagSpecifications, so DescribeInstances returned untagged
instances and tag filters matched nothing. EKS node groups discover
their workers by the spinifex:eks-cluster / spinifex:eks-nodegroup
tags, so scale-up convergence and worker reconciliation depended on
tags that were silently dropped.

Apply instance-scoped TagSpecifications onto the built instance so the
tags surface through DescribeInstances and tag filters.

* fix: serialize nodegroup reconcile via CAS to prevent scale overshoot

UpdateNodegroupConfig is dispatched on a queue group, so two
overlapping requests (e.g. an SDK retry mid-flight) each read
current = len(InstanceIDs) and each launched desired-current workers,
overshooting the target (1 -> 5 for desired=3) and stranding untracked
VMs via last-writer-wins on the unconditional PutNodegroupRecord.

Make every durable nodegroup-record mutation a compare-and-swap on the
KV revision. Scale-up launches one worker at a time and CAS-appends it
only while the record is below the live ScalingDesired, terminating any
surplus VM launched into an already-full record. Scale-down terminates
before the CAS truncation so a crash or retry leaves reclaimable dead
IDs rather than orphaned VMs. A concurrent regression test asserts two
overlapping scale-ups converge to exactly desired under -race.

* fix: return securityGroupRuleSet from AuthorizeSecurityGroup ingress/egress

The AWS provider's aws_vpc_security_group_ingress_rule reads
SecurityGroupRules[0].SecurityGroupRuleId on create; an empty ruleset
panics the plugin (Plugin did not respond), though the rule applies
server-side. Echo the created rules via sgRuleToSecurityGroupRule, the
same shape DescribeSecurityGroupRules already returns.

* fix: surface InsufficientInstanceCapacity from nodegroup scale, not 500

launchWorkersCAS wrapped the bare awserror code via fmt.Errorf, so the
gateway ErrorHandler's exact-match against ErrorLookup missed and remapped
to InternalError (500). An out-of-capacity scale now surfaces the real
InsufficientInstanceCapacity code (400); opaque internal failures still
wrap and stay 500. Adds awserrors.HasErrorCode for the membership test.

* fix: cut EKS node boot ~50s — drop cloud-final deadlock + syslinux menu

Worker/server user-data runcmd ended with `[ rc-service, local, start ]`,
which runs inside cloud-final while the OpenRC `local` service is ordered
after it — a deadlock broken only by OpenRC's ~50s start timeout, stalling
the k3s join. Run the IMDS route script directly instead and keep
`rc-update add local default` for reboot persistence.

setup.sh: cut the SYSLINUX boot-menu countdown 10s->~1s (kept nonzero so the
menu stays interruptible over serial).

Validated live on env19 CP: boot menu 1s, no `waiting for cloud-final`
stall, apiserver ready ~66s (was ~109s+).

mulga-siv-332

* feat: add argocd managed addon + bake bundle into eks-node AMI

The addon catalog advertised coredns and aws-load-balancer-controller, but
the only baked manifest shipped in the eks-node AMI was the spinifex-noop
fixture. Installing any real addon failed with 'no baked bundle for version'
once the VM-side renderer found nothing on disk.

Add argocd v3.0.23 as a real, demonstrable managed addon: catalog entry (no
IRSA), a flattened namespaced install manifest baked into the AMI, and the
manifest.conf bake line. The eks-addons-access workbook switches from coredns
(redundant on K3s, never baked) to argocd.

The baked bundle is generated from upstream install.yaml via kubectl kustomize
with namespace: argocd plus a prepended Namespace, since upstream is
namespace-agnostic and K3s auto-deploy applies with no -n flag.

* fix(eks): run CoreDNS as a node-local DaemonSet for worker DNS

k3s ships CoreDNS as a single-replica Deployment that tolerates the CP taint
and strands itself on the control-plane node. In the managed-CP topology the
worker<->CP pod overlay cannot cross the VPC boundary, so worker pods cannot
reach a CP-resident CoreDNS and lose all cluster + external DNS.

Skip the bundled CoreDNS (coredns.yaml.skip, staged in k3s.initd like traefik)
and deploy a Mulga-owned variant: a DaemonSet (one pod per node) plus kube-dns
internalTrafficPolicy=Local, so every node resolves via its own CoreDNS. Immune
to the cross-VPC overlay gap and topology-agnostic (single-node and managed-CP
both work). Image-side only; no handler change.

* fix(eks): revoke cross-referencing SG rules before delete on teardown

DeleteClusterSGs deleted the control-plane and nodegroup SGs directly, but they
cross-reference each other (EnsureNodegroupSGRules), and AWS refuses to delete a
security group still referenced by another SG's rule. Both deletes failed with
DependencyViolation, the SGs leaked, and the VPC could not be deleted (tofu
destroy hung on DeleteVpc).

Resolve both SG IDs, revoke all ingress on each to break the cp<->ng cycle, then
delete both. Adds RevokeSecurityGroupIngress to sgProvisioner (already on
VPCServiceImpl) and revokeAllSGIngress, with a regression test that models AWS's
DependencyViolation.

* fix(eks): advertise NLB front-end so worker pods reach the apiserver ClusterIP

In the managed-control-plane topology the apiserver advertised its Set-B
node-ip into the in-cluster `kubernetes` Endpoints. That IP lives in the
unpeered CP VPC and is unreachable from worker pods, so every worker pod
using the apiserver ClusterIP (CoreDNS kubernetes-plugin sync, controllers,
argocd redis secret-init) failed.

- k3s_server_vm.go: set advertise-address to the worker-reachable NLB
  front-end (PrivateEndpointIP, else EndpointIP); both are already cert SANs.
- nlb.go: add a second :6443 listener on the cluster NLB (same CP target
  group) so PrivateEndpointIP:6443 — the advertised endpoint — is served.
- security_groups.go: admit :6443 on the private-endpoint SG.

Validated on a live managed-CP cluster: worker CoreDNS /ready 200, all argocd
pods Running, kubectl logs on worker pods working.
2026-06-16 20:27:21 +10:00

106 lines
5.2 KiB
Bash
Executable File

#!/bin/sh
set -eu
# setup.sh — guest customisation for the unified eks-node AMI (server + agent).
#
# Runs inside the libguestfs appliance (via virt-customize --run) under
# build-system-image.sh after packages and binaries are installed. Downloads the
# pinned K3s binary, verifies its SHA256, drops it into /usr/local/bin, sets
# executable bits on all role init scripts + cron entries, and writes the K3s
# server config skeleton. The role (server vs agent) is selected per-instance at
# first boot by eks-node-role.
#
# Network access inside the appliance is provided by virt-customize --network.
# curl is in APK_PACKAGES.
K3S_VERSION="v1.32.5+k3s1"
K3S_URL_BASE="https://github.com/k3s-io/k3s/releases/download/${K3S_VERSION}"
# Pull the upstream signed checksums file and pin the amd64 line. A tampered
# release replacing both files would still produce a self-consistent download,
# so the SHA file URL is the trust anchor — anyone forging a Mulga AMI build
# would need to compromise the k3s-io GitHub release artefacts.
echo "[eks-node-setup] fetching k3s checksums ${K3S_VERSION}"
curl -fsSL -o /tmp/k3s-checksums.txt "${K3S_URL_BASE}/sha256sum-amd64.txt"
K3S_SHA256=$(awk '/[ \t]k3s$/{print $1; exit}' /tmp/k3s-checksums.txt)
if [ -z "${K3S_SHA256}" ]; then
echo "[eks-node-setup] could not parse k3s sha256 from upstream checksums"
cat /tmp/k3s-checksums.txt
exit 1
fi
echo "[eks-node-setup] downloading k3s ${K3S_VERSION} (sha256=${K3S_SHA256})"
curl -fsSL -o /usr/local/bin/k3s "${K3S_URL_BASE}/k3s"
echo "${K3S_SHA256} /usr/local/bin/k3s" > /tmp/k3s.sha256
if ! sha256sum -c /tmp/k3s.sha256; then
echo "[eks-node-setup] k3s SHA256 mismatch — refusing to bake AMI"
exit 1
fi
chmod 0755 /usr/local/bin/k3s
ln -sf /usr/local/bin/k3s /usr/local/bin/kubectl
ln -sf /usr/local/bin/k3s /usr/local/bin/crictl
ln -sf /usr/local/bin/k3s /usr/local/bin/ctr
# Init scripts ship as 0644 from INSTALL_FILES; OpenRC requires 0755. Every
# role's services are baked; the selector enables the right ones at first boot.
chmod 0755 /etc/init.d/eks-node-role /etc/init.d/k3s /etc/init.d/k3s-agent \
/etc/init.d/eks-token-webhook /etc/init.d/k3s-first-boot /etc/init.d/mulga-eks-state-report \
/etc/init.d/mulga-eks-addon-sync
chmod 0755 /usr/local/sbin/eks-node-role /usr/local/sbin/k3s-first-boot \
/usr/local/sbin/mulga-eks-state-report /usr/local/sbin/mulga-eks-addon-sync
chmod 0755 /etc/periodic/daily/mulga-eks-etcd-snapshot
# K3s server config — skeleton; cloud-init / first-boot fills in the
# per-cluster fields (cluster-cidr, service-cidr, token-file, etc). Agents
# ignore this file (they run `k3s agent`).
# IAM token-auth is wired here via kube-apiserver-arg: the eks-token-webhook
# service (ordered `before k3s`) writes its kubeconfig to
# /etc/spinifex-eks/token-webhook.kubeconfig before the apiserver reads it.
# This only affects bearer-token requests; anonymous and client-cert paths
# (the first-boot /readyz probe uses the admin kubeconfig) are unaffected.
mkdir -p /etc/rancher/k3s
cat > /etc/rancher/k3s/config.yaml.skel <<'EOF'
# Populated at first boot by cloud-init user-data via k3s-first-boot.sh.
# Fields documented at https://docs.k3s.io/installation/configuration.
# cluster-init selects the embedded etcd datastore. cloud-init write_files
# normally overrides this skeleton; kept consistent as the boot fallback.
cluster-init: true
disable:
- traefik
- servicelb
- local-storage
kube-apiserver-arg:
- "authentication-token-webhook-config-file=/etc/spinifex-eks/token-webhook.kubeconfig"
- "authentication-token-webhook-cache-ttl=5m"
EOF
# Shared state dir for both roles (agent.env / role marker live here).
mkdir -p /etc/spinifex-eks
# Sentinel file marker — k3s-first-boot (server role) self-disables after first
# success by checking for this path. Initial state is "not yet run".
touch /var/lib/spinifex-eks/first-boot.pending 2>/dev/null || {
mkdir -p /var/lib/spinifex-eks
touch /var/lib/spinifex-eks/first-boot.pending
}
# Bind /dev/console to the serial port so userspace boot output — OpenRC
# service starts, cloud-init, role selection, and k3s-first-boot diagnostics —
# reaches ttyS0, which the orchestrator captures to a host-side log. The stock
# Alpine cloud image lists `console=tty0` LAST in default_kernel_opts; Linux
# makes the last console= the controlling /dev/console, so userspace logs to
# the framebuffer and the serial capture sees only kernel dmesg. Reorder so
# ttyS0 is last in both the generator config and the rendered extlinux.conf.
sed -i \
's|console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0|console=tty0 console=ttyAMA0,115200n8 console=ttyS0,115200n8|' \
/etc/update-extlinux.conf /boot/extlinux.conf
# Cut the boot-menu countdown from 10s to ~1s. The stock cloud image waits 10s at
# the SYSLINUX menu before auto-booting — a fixed, network-independent tax on every
# VM start. Patch the generator config (timeout in seconds) and the rendered output
# (TIMEOUT in 1/10s) so a regenerate keeps the short value. A small nonzero value is
# kept so the menu stays interruptible over serial; TIMEOUT 0 would wait forever.
sed -i 's/^timeout=.*/timeout=1/' /etc/update-extlinux.conf
sed -i 's/^TIMEOUT[[:space:]].*/TIMEOUT 10/' /boot/extlinux.conf
echo "[eks-node-setup] done"