mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 19:01:42 +00:00
7bbd99c792
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity
Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.
EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
(apiserver-network-proxy): CP servers run konnectivity-server behind the
cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
traffic rides the agent-initiated reverse tunnel instead of an unroutable
CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.
EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
clobber it; route mounted-volume state updates through the live VB with a
retry for in-use marks racing the mount responder; keep the boot volume
in-use across stop/crash unmount.
* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM
missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.
Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.
* fix(eks): let eks-token-webhook use IMDS creds when static keys absent
The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.
Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.
* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane
The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.
Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.
* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role
The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.
Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
98 lines
3.9 KiB
Bash
Executable File
98 lines
3.9 KiB
Bash
Executable File
#!/bin/sh
|
|
set -eu
|
|
|
|
# eks-node-role — first-boot role selector for the unified eks-node AMI.
|
|
#
|
|
# The same image boots either as a K3s control-plane server or a worker agent;
|
|
# the role is chosen per-instance at first boot from SPINIFEX_K3S_ROLE in the
|
|
# cloud-init env file. This script enables + starts the matching OpenRC services
|
|
# (none are in the default runlevel at bake time), records the resolved role to
|
|
# ROLE_FILE so later boots and the etcd-snapshot cron can branch without
|
|
# re-parsing, then removes itself from the runlevel so it runs exactly once.
|
|
#
|
|
# server → eks-token-webhook, k3s (server), k3s-first-boot (bootstrap
|
|
# publish), mulga-eks-state-report
|
|
# server-join → eks-token-webhook, k3s (server, joins the first server's etcd
|
|
# quorum), mulga-eks-state-report. NO k3s-first-boot: the first
|
|
# server already publishes the cluster-identical bootstrap
|
|
# artifacts; a join re-publish only races the bootstrap bus.
|
|
# agent → k3s-agent
|
|
#
|
|
# Paths are overridable via env so the selection logic is unit-testable (bats)
|
|
# without root or a real /etc.
|
|
|
|
ROLE_FILE="${EKS_NODE_ROLE_FILE:-/etc/spinifex-eks/role}"
|
|
ENVFILE="${EKS_NODE_ENVFILE:-/etc/spinifex-eks/first-boot.env}"
|
|
AGENT_ENVFILE="${EKS_NODE_AGENT_ENVFILE:-/etc/spinifex-eks/agent.env}"
|
|
LOGTAG="eks-node-role"
|
|
|
|
log() { echo "[${LOGTAG}] $*"; }
|
|
die() { log "ERROR: $*"; exit 1; }
|
|
|
|
# Already resolved on a prior boot — the role services are in the runlevel and
|
|
# this service should have been pulled, but guard anyway so a stray re-run is a
|
|
# no-op rather than a double-enable.
|
|
if [ -f "${ROLE_FILE}" ]; then
|
|
log "role already resolved ($(cat "${ROLE_FILE}")); nothing to do"
|
|
exit 0
|
|
fi
|
|
|
|
# Resolve the role. Server seeds SPINIFEX_K3S_ROLE in first-boot.env; a worker
|
|
# seed that predates the explicit var is inferred from the presence of agent.env
|
|
# (K3S_URL/K3S_TOKEN live there).
|
|
ROLE=""
|
|
if [ -f "${ENVFILE}" ]; then
|
|
ROLE=$(. "${ENVFILE}"; printf '%s' "${SPINIFEX_K3S_ROLE:-}")
|
|
fi
|
|
if [ -z "${ROLE}" ] && [ -f "${AGENT_ENVFILE}" ]; then
|
|
ROLE="agent"
|
|
fi
|
|
|
|
case "${ROLE}" in
|
|
server)
|
|
log "configuring server role"
|
|
rc-update add eks-token-webhook default
|
|
rc-update add k3s default
|
|
# konnectivity-server fronts apiserver egress; every server replica runs
|
|
# one so the agent's per-replica tunnel always lands a live server.
|
|
rc-update add konnectivity-server default
|
|
rc-update add k3s-first-boot default
|
|
rc-update add mulga-eks-state-report default
|
|
# Managed-addon delivery runs only on the primary server: it renders
|
|
# staged addon bundles into the K3s auto-deploy dir, which a single
|
|
# writer owns. HA multi-server addon delivery is tracked separately
|
|
# (mulga-siv-231.7); server-join nodes do not run it.
|
|
rc-update add mulga-eks-addon-sync default
|
|
rc-service eks-token-webhook start
|
|
rc-service k3s start
|
|
rc-service konnectivity-server start
|
|
rc-service k3s-first-boot start
|
|
rc-service mulga-eks-state-report start
|
|
rc-service mulga-eks-addon-sync start
|
|
;;
|
|
server-join)
|
|
log "configuring server-join role"
|
|
rc-update add eks-token-webhook default
|
|
rc-update add k3s default
|
|
rc-update add konnectivity-server default
|
|
rc-update add mulga-eks-state-report default
|
|
rc-service eks-token-webhook start
|
|
rc-service k3s start
|
|
rc-service konnectivity-server start
|
|
rc-service mulga-eks-state-report start
|
|
;;
|
|
agent)
|
|
log "configuring agent role"
|
|
rc-update add k3s-agent default
|
|
rc-service k3s-agent start
|
|
;;
|
|
*)
|
|
die "SPINIFEX_K3S_ROLE missing or invalid: '${ROLE}'"
|
|
;;
|
|
esac
|
|
|
|
mkdir -p "$(dirname "${ROLE_FILE}")"
|
|
printf '%s\n' "${ROLE}" > "${ROLE_FILE}"
|
|
rc-update del eks-node-role default 2>/dev/null || true
|
|
log "role '${ROLE}' configured"
|