Josh Siviour f7bf921ce7 fix(eks): NLB name overflow, nodegroup readiness scoping, launch retry, GPU konnectivity toleration (#511) ...

* fix(eks): keep derived ELBv2 names under 32 chars and surface CREATE_FAILED reason

Long cluster names made the derived "eks-<name>[-cp|-konn]" NLB/target-group
names exceed the ELBv2 32-char limit, failing cluster creation asynchronously
with no visible cause. safeELBv2Name now deterministically truncates the name
and appends a stable hash of the full cluster name when needed, so the same
name is derived on every call (create, lookup, delete). DescribeCluster also
now surfaces ClusterMeta.StatusReason as a Health.Issues entry when a cluster
is FAILED, instead of leaving callers to guess why.

* fix(eks): plumb per-nodegroup Ready-node counts through state reports

The on-VM state-report script now emits nodegroup_ready, a per-nodegroup Ready
count keyed by the eks.amazonaws.com/nodegroup label value, alongside the
existing cluster-wide node_count. ServerStateReport, the ClusterReconciler's
observe step, and SetClusterHealthState now carry this map through to
ClusterMeta.NodegroupNodeCounts, so per-nodegroup consumers (wired in the next
commit) can gate on a nodegroup's own workers instead of the cluster total.

* fix(eks): scope nodegroup Ready-gate per nodegroup and retry failed worker launches

waitWorkersReady compared the cluster-wide Ready-node count against a
nodegroup's baseline+desired, so in a multi-nodegroup cluster one nodegroup's
Ready workers could mask another whose own workers never registered. It now
reads meta.NodegroupNodeCounts[ng], scoped to nodes carrying the
eks.amazonaws.com/nodegroup=<name> label.

Separately, launchNodegroupInfra never retried a failed worker launch (a
transient QMP timeout or host capacity pressure), leaving the nodegroup stuck
CREATING with zero workers. The new launchWorkersUntilDesired wrapper relaunches
only the shortfall on failure, logging each retry, until the desired count is
reached or workerLaunchRetryTimeout elapses.

Both fixes are covered by regression tests: a two-nodegroup scenario where only
one has Ready workers, a transient-failure-then-success relaunch, and an
exhausted-retry-budget CREATE_FAILED path. Not covered here: a worker that dies
after the nodegroup is already ACTIVE — that needs an ongoing reconciler, which
this change does not add.

* fix(eks): tolerate GPU NoSchedule taint in konnectivity-agent DaemonSet

A GPU-only worker is tainted nvidia.com/gpu=present:NoSchedule, so the
konnectivity-agent DaemonSet (no tolerations) showed DESIRED=0 there and
kubectl logs/exec against pods on that node failed "No agent available".
Adds an Exists toleration for nvidia.com/gpu:NoSchedule.

Live-cluster validation (DESIRED count, kubectl exec) is deferred — no cluster
available to test against right now.

* fix: tolerate nvidia.com/gpu taint on ebs-csi-node and coredns DaemonSets so GPU workers get storage and DNS

2026-07-12 12:11:16 +10:00

..

coredns-mulga.yaml

fix(eks): NLB name overflow, nodegroup readiness scoping, launch retry, GPU konnectivity toleration (#511)

2026-07-12 12:11:16 +10:00