Files
spinifex/scripts/images/eks-node/eks-node-role.sh
T
Josh Siviour 7bbd99c792 feat(eks): multinode control-plane + IMDS/token-webhook CP fixes (426/427/428/495/496) (#459)
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity

Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.

EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
  (apiserver-network-proxy): CP servers run konnectivity-server behind the
  cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
  traffic rides the agent-initiated reverse tunnel instead of an unroutable
  CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
  replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
  flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
  tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
  workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.

EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
  clobber it; route mounted-volume state updates through the live VB with a
  retry for in-use marks racing the mount responder; keep the boot volume
  in-use across stop/crash unmount.

* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM

missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.

Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.

* fix(eks): let eks-token-webhook use IMDS creds when static keys absent

The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.

Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.

* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane

The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.

Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.

* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role

The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.

Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
2026-07-09 09:43:30 +10:00

98 lines
3.9 KiB
Bash
Executable File

#!/bin/sh
set -eu
# eks-node-role — first-boot role selector for the unified eks-node AMI.
#
# The same image boots either as a K3s control-plane server or a worker agent;
# the role is chosen per-instance at first boot from SPINIFEX_K3S_ROLE in the
# cloud-init env file. This script enables + starts the matching OpenRC services
# (none are in the default runlevel at bake time), records the resolved role to
# ROLE_FILE so later boots and the etcd-snapshot cron can branch without
# re-parsing, then removes itself from the runlevel so it runs exactly once.
#
# server → eks-token-webhook, k3s (server), k3s-first-boot (bootstrap
# publish), mulga-eks-state-report
# server-join → eks-token-webhook, k3s (server, joins the first server's etcd
# quorum), mulga-eks-state-report. NO k3s-first-boot: the first
# server already publishes the cluster-identical bootstrap
# artifacts; a join re-publish only races the bootstrap bus.
# agent → k3s-agent
#
# Paths are overridable via env so the selection logic is unit-testable (bats)
# without root or a real /etc.
ROLE_FILE="${EKS_NODE_ROLE_FILE:-/etc/spinifex-eks/role}"
ENVFILE="${EKS_NODE_ENVFILE:-/etc/spinifex-eks/first-boot.env}"
AGENT_ENVFILE="${EKS_NODE_AGENT_ENVFILE:-/etc/spinifex-eks/agent.env}"
LOGTAG="eks-node-role"
log() { echo "[${LOGTAG}] $*"; }
die() { log "ERROR: $*"; exit 1; }
# Already resolved on a prior boot — the role services are in the runlevel and
# this service should have been pulled, but guard anyway so a stray re-run is a
# no-op rather than a double-enable.
if [ -f "${ROLE_FILE}" ]; then
log "role already resolved ($(cat "${ROLE_FILE}")); nothing to do"
exit 0
fi
# Resolve the role. Server seeds SPINIFEX_K3S_ROLE in first-boot.env; a worker
# seed that predates the explicit var is inferred from the presence of agent.env
# (K3S_URL/K3S_TOKEN live there).
ROLE=""
if [ -f "${ENVFILE}" ]; then
ROLE=$(. "${ENVFILE}"; printf '%s' "${SPINIFEX_K3S_ROLE:-}")
fi
if [ -z "${ROLE}" ] && [ -f "${AGENT_ENVFILE}" ]; then
ROLE="agent"
fi
case "${ROLE}" in
server)
log "configuring server role"
rc-update add eks-token-webhook default
rc-update add k3s default
# konnectivity-server fronts apiserver egress; every server replica runs
# one so the agent's per-replica tunnel always lands a live server.
rc-update add konnectivity-server default
rc-update add k3s-first-boot default
rc-update add mulga-eks-state-report default
# Managed-addon delivery runs only on the primary server: it renders
# staged addon bundles into the K3s auto-deploy dir, which a single
# writer owns. HA multi-server addon delivery is tracked separately
# (mulga-siv-231.7); server-join nodes do not run it.
rc-update add mulga-eks-addon-sync default
rc-service eks-token-webhook start
rc-service k3s start
rc-service konnectivity-server start
rc-service k3s-first-boot start
rc-service mulga-eks-state-report start
rc-service mulga-eks-addon-sync start
;;
server-join)
log "configuring server-join role"
rc-update add eks-token-webhook default
rc-update add k3s default
rc-update add konnectivity-server default
rc-update add mulga-eks-state-report default
rc-service eks-token-webhook start
rc-service k3s start
rc-service konnectivity-server start
rc-service mulga-eks-state-report start
;;
agent)
log "configuring agent role"
rc-update add k3s-agent default
rc-service k3s-agent start
;;
*)
die "SPINIFEX_K3S_ROLE missing or invalid: '${ROLE}'"
;;
esac
mkdir -p "$(dirname "${ROLE_FILE}")"
printf '%s\n' "${ROLE}" > "${ROLE_FILE}"
rc-update del eks-node-role default 2>/dev/null || true
log "role '${ROLE}' configured"