3 Commits

Author	SHA1	Message	Date
Josh Siviour	f7bf921ce7	fix(eks): NLB name overflow, nodegroup readiness scoping, launch retry, GPU konnectivity toleration (#511) ... * fix(eks): keep derived ELBv2 names under 32 chars and surface CREATE_FAILED reason Long cluster names made the derived "eks-<name>[-cp|-konn]" NLB/target-group names exceed the ELBv2 32-char limit, failing cluster creation asynchronously with no visible cause. safeELBv2Name now deterministically truncates the name and appends a stable hash of the full cluster name when needed, so the same name is derived on every call (create, lookup, delete). DescribeCluster also now surfaces ClusterMeta.StatusReason as a Health.Issues entry when a cluster is FAILED, instead of leaving callers to guess why. * fix(eks): plumb per-nodegroup Ready-node counts through state reports The on-VM state-report script now emits nodegroup_ready, a per-nodegroup Ready count keyed by the eks.amazonaws.com/nodegroup label value, alongside the existing cluster-wide node_count. ServerStateReport, the ClusterReconciler's observe step, and SetClusterHealthState now carry this map through to ClusterMeta.NodegroupNodeCounts, so per-nodegroup consumers (wired in the next commit) can gate on a nodegroup's own workers instead of the cluster total. * fix(eks): scope nodegroup Ready-gate per nodegroup and retry failed worker launches waitWorkersReady compared the cluster-wide Ready-node count against a nodegroup's baseline+desired, so in a multi-nodegroup cluster one nodegroup's Ready workers could mask another whose own workers never registered. It now reads meta.NodegroupNodeCounts[ng], scoped to nodes carrying the eks.amazonaws.com/nodegroup=<name> label. Separately, launchNodegroupInfra never retried a failed worker launch (a transient QMP timeout or host capacity pressure), leaving the nodegroup stuck CREATING with zero workers. The new launchWorkersUntilDesired wrapper relaunches only the shortfall on failure, logging each retry, until the desired count is reached or workerLaunchRetryTimeout elapses. Both fixes are covered by regression tests: a two-nodegroup scenario where only one has Ready workers, a transient-failure-then-success relaunch, and an exhausted-retry-budget CREATE_FAILED path. Not covered here: a worker that dies after the nodegroup is already ACTIVE — that needs an ongoing reconciler, which this change does not add. * fix(eks): tolerate GPU NoSchedule taint in konnectivity-agent DaemonSet A GPU-only worker is tainted nvidia.com/gpu=present:NoSchedule, so the konnectivity-agent DaemonSet (no tolerations) showed DESIRED=0 there and kubectl logs/exec against pods on that node failed "No agent available". Adds an Exists toleration for nvidia.com/gpu:NoSchedule. Live-cluster validation (DESIRED count, kubectl exec) is deferred — no cluster available to test against right now. * fix: tolerate nvidia.com/gpu taint on ebs-csi-node and coredns DaemonSets so GPU workers get storage and DNS	2026-07-12 12:11:16 +10:00
Josh Siviour	7bbd99c792	feat(eks): multinode control-plane + IMDS/token-webhook CP fixes (426/427/428/495/496) (#459) ... * feat(eks): multinode control-plane networking, IAM, and CoreDNS parity Integrates the EKS multinode fix set for the managed-control-plane topology so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it. EKS: - Route apiserver egress (exec/logs/webhooks) through konnectivity (apiserver-network-proxy): CP servers run konnectivity-server behind the cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod traffic rides the agent-initiated reverse tunnel instead of an unroutable CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every replica (HA-correct). - Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay flannel mesh SG rule authorized at cluster launch with ordered teardown. - Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay. - Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity. - Spread nodegroup workers across hosts; addon readiness falls back on workload rollout when gvks are absent. - elbv2: allow the proxy_protocol_v2.enabled target-group attribute. EBS: - Move volume attachment state to state.json so a live nbdkit VB cannot clobber it; route mounted-volume state updates through the live VB with a retry for in-use marks racing the mount responder; keep the boot volume in-use across stop/crash unmount. * fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM missingOrchestrationDeps gated on s.deps.IAM, but production wires only the lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only). The gate therefore reported missing=[IAM] on every node forever, so CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even though the IAM service was fully up via the provider. Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else deps.IAMProvider()) the nodegroup/role code already uses — so the protective check still rejects a node with no IAM service but passes when the provider supplies one. Add a regression test that sets only IAMProvider and asserts the gate is satisfied; the existing nil-both test still rejects. * fix(eks): let eks-token-webhook use IMDS creds when static keys absent The CP VM launches on an instance profile, so buildK3sUserData omits EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys before eksgw.New could fall back to the IMDS SDK chain, so the webhook aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver crashed on the missing --authentication-token-webhook-config-file file, stalling the cluster at CREATING until it timed out to FAILED. Drop the static-key precondition (matching the sibling eks-gateway-publish and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/ cluster gates. * fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt, which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a 169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing 169.254.169.254 and steering IMDS off the data NIC away from the per-tap br-imds datapath, where the host RST it. eks-gateway-publish then failed to sign requests and no bootstrap artifacts were ever published, so clusters sat CREATING until they timed out to FAILED. Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI) are unaffected and IMDS falls through the data NIC's default route to the br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net deletes any default route on a NIC declared NIC<n>_DEFAULT=0. * fix(eks): grant eks:WebhookTokenReview to the control-plane instance role The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway broker at POST /clusters/{name}/token-review, signing with the control-plane VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes against eks:WebhookTokenReview, but the role's inline policy granted only eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every relay and the apiserver returned 401 to all external kubectl. Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile re-asserts the inline policy on every control-plane instance-profile ensure, and the gateway evaluates authorization per request, so existing clusters converge without a rebuild once the shared role is updated.	2026-07-09 09:43:30 +10:00
Josh Siviour	bc9556d1a9	fix(eks): managed-CP worker data-plane — apiserver reachability, node-local CoreDNS, SG teardown (#368) ... * fix: persist instance TagSpecifications in RunInstance RunInstance built the ec2.Instance metadata but never read input.TagSpecifications, so DescribeInstances returned untagged instances and tag filters matched nothing. EKS node groups discover their workers by the spinifex:eks-cluster / spinifex:eks-nodegroup tags, so scale-up convergence and worker reconciliation depended on tags that were silently dropped. Apply instance-scoped TagSpecifications onto the built instance so the tags surface through DescribeInstances and tag filters. * fix: serialize nodegroup reconcile via CAS to prevent scale overshoot UpdateNodegroupConfig is dispatched on a queue group, so two overlapping requests (e.g. an SDK retry mid-flight) each read current = len(InstanceIDs) and each launched desired-current workers, overshooting the target (1 -> 5 for desired=3) and stranding untracked VMs via last-writer-wins on the unconditional PutNodegroupRecord. Make every durable nodegroup-record mutation a compare-and-swap on the KV revision. Scale-up launches one worker at a time and CAS-appends it only while the record is below the live ScalingDesired, terminating any surplus VM launched into an already-full record. Scale-down terminates before the CAS truncation so a crash or retry leaves reclaimable dead IDs rather than orphaned VMs. A concurrent regression test asserts two overlapping scale-ups converge to exactly desired under -race. * fix: return securityGroupRuleSet from AuthorizeSecurityGroup ingress/egress The AWS provider's aws_vpc_security_group_ingress_rule reads SecurityGroupRules[0].SecurityGroupRuleId on create; an empty ruleset panics the plugin (Plugin did not respond), though the rule applies server-side. Echo the created rules via sgRuleToSecurityGroupRule, the same shape DescribeSecurityGroupRules already returns. * fix: surface InsufficientInstanceCapacity from nodegroup scale, not 500 launchWorkersCAS wrapped the bare awserror code via fmt.Errorf, so the gateway ErrorHandler's exact-match against ErrorLookup missed and remapped to InternalError (500). An out-of-capacity scale now surfaces the real InsufficientInstanceCapacity code (400); opaque internal failures still wrap and stay 500. Adds awserrors.HasErrorCode for the membership test. * fix: cut EKS node boot ~50s — drop cloud-final deadlock + syslinux menu Worker/server user-data runcmd ended with `[ rc-service, local, start ]`, which runs inside cloud-final while the OpenRC `local` service is ordered after it — a deadlock broken only by OpenRC's ~50s start timeout, stalling the k3s join. Run the IMDS route script directly instead and keep `rc-update add local default` for reboot persistence. setup.sh: cut the SYSLINUX boot-menu countdown 10s->~1s (kept nonzero so the menu stays interruptible over serial). Validated live on env19 CP: boot menu 1s, no `waiting for cloud-final` stall, apiserver ready ~66s (was ~109s+). mulga-siv-332 * feat: add argocd managed addon + bake bundle into eks-node AMI The addon catalog advertised coredns and aws-load-balancer-controller, but the only baked manifest shipped in the eks-node AMI was the spinifex-noop fixture. Installing any real addon failed with 'no baked bundle for version' once the VM-side renderer found nothing on disk. Add argocd v3.0.23 as a real, demonstrable managed addon: catalog entry (no IRSA), a flattened namespaced install manifest baked into the AMI, and the manifest.conf bake line. The eks-addons-access workbook switches from coredns (redundant on K3s, never baked) to argocd. The baked bundle is generated from upstream install.yaml via kubectl kustomize with namespace: argocd plus a prepended Namespace, since upstream is namespace-agnostic and K3s auto-deploy applies with no -n flag. * fix(eks): run CoreDNS as a node-local DaemonSet for worker DNS k3s ships CoreDNS as a single-replica Deployment that tolerates the CP taint and strands itself on the control-plane node. In the managed-CP topology the worker<->CP pod overlay cannot cross the VPC boundary, so worker pods cannot reach a CP-resident CoreDNS and lose all cluster + external DNS. Skip the bundled CoreDNS (coredns.yaml.skip, staged in k3s.initd like traefik) and deploy a Mulga-owned variant: a DaemonSet (one pod per node) plus kube-dns internalTrafficPolicy=Local, so every node resolves via its own CoreDNS. Immune to the cross-VPC overlay gap and topology-agnostic (single-node and managed-CP both work). Image-side only; no handler change. * fix(eks): revoke cross-referencing SG rules before delete on teardown DeleteClusterSGs deleted the control-plane and nodegroup SGs directly, but they cross-reference each other (EnsureNodegroupSGRules), and AWS refuses to delete a security group still referenced by another SG's rule. Both deletes failed with DependencyViolation, the SGs leaked, and the VPC could not be deleted (tofu destroy hung on DeleteVpc). Resolve both SG IDs, revoke all ingress on each to break the cp<->ng cycle, then delete both. Adds RevokeSecurityGroupIngress to sgProvisioner (already on VPCServiceImpl) and revokeAllSGIngress, with a regression test that models AWS's DependencyViolation. * fix(eks): advertise NLB front-end so worker pods reach the apiserver ClusterIP In the managed-control-plane topology the apiserver advertised its Set-B node-ip into the in-cluster `kubernetes` Endpoints. That IP lives in the unpeered CP VPC and is unreachable from worker pods, so every worker pod using the apiserver ClusterIP (CoreDNS kubernetes-plugin sync, controllers, argocd redis secret-init) failed. - k3s_server_vm.go: set advertise-address to the worker-reachable NLB front-end (PrivateEndpointIP, else EndpointIP); both are already cert SANs. - nlb.go: add a second :6443 listener on the cluster NLB (same CP target group) so PrivateEndpointIP:6443 — the advertised endpoint — is served. - security_groups.go: admit :6443 on the private-endpoint SG. Validated on a live managed-CP cluster: worker CoreDNS /ready 200, all argocd pods Running, kubectl logs on worker pods working.	2026-06-16 20:27:21 +10:00