10 Commits

Author	SHA1	Message	Date
Josh Siviour	a3fb4cccec	fix(eks): backoff and terminal state on delete re-drive, CSI volume reclaim reporting, independent e2e subtests (#642) ... * fix: bound EKS DELETING re-drive with backoff and a terminal give-up reapCluster re-drove purgeClusterInfra on every GC tick with no attempt tracking, so a permanently-failing teardown (e.g. an unretriable DependencyViolation) was retried forever. Track attempt count and last attempt time on ClusterMeta, back off exponentially between re-drives, and give up after maxDeleteReapAttempts with a single error-level log. Status stays DELETING on give-up so the cluster's infra remains tracked and billable pending operator intervention, preserving the ADR-0006 §6 billable-before-sweep invariant. * fix: create the EKS e2e node role independently in each subtest runEBSCSIVolume string-built the worker node role ARN without ever creating it, only passing because runIRSAPod's t.Cleanup discarded DeleteRole's error while the role was still attached to its instance profile, leaving it behind for EBSCSIVolume to reuse by naming convention. Extract ensureNodeRole and call it independently from both subtests, detaching the role from its instance profile before deleting it and asserting that error instead of discarding it. * fix: reclaim CSI-provisioned EBS volumes on EKS cluster delete purgeClusterInfra tore down the control plane, nodegroups, and networking but never looked at EBS volumes, so any volume the in-cluster CSI driver had provisioned for a PVC was left behind untracked and unbilled for. Add a report-only reclaim step that scans for volumes tagged with the cluster's CSI owner tag and logs them for operator follow-up. Volumes are never auto-deleted: a PV's reclaim policy (Delete vs Retain) is a Kubernetes-side property the CSI driver does not tag onto the EC2 volume, so there is no safe way to tell from tags alone whether deleting a given volume is intended. This mirrors the volume-leak reaper's existing mark-and-alarm policy for the same reason. The cluster-owner tag requires the EBS CSI controller's --k8s-tag-cluster-id flag, added here to the controller manifest; without it a CSI volume carries no cluster-scoped tag and reclaim would have to guess across every cluster sharing the account. * fix: persist the last purge error on a wedged DELETING cluster DeleteReapExhausted told an operator a cluster's teardown had given up, but not why: nothing recorded the failing purge's error. Persist it on every failed re-drive, not only once the reaper exhausts its attempts, so it is visible on a still-retrying cluster too. Status stays DELETING throughout, preserving the billing invariant that untorn-down infra remains tracked. * fix: make the EKS e2e suite confirm its CSI volume is actually deleted The EBSCSIVolume subtest deleted its PVC on cleanup but never confirmed the underlying volume was reclaimed, so a CSI driver that silently failed to delete it would leak a volume with nobody noticing. The suite provisions and owns this volume, so unlike production teardown it can safely wait on its StorageClass's Delete reclaim policy: capture the CSI volumeHandle while the PVC is bound, then have cleanup poll EC2 for that exact volume to disappear before the addon/nodegroup that run the CSI controller are torn down.	2026-08-02 08:21:25 +10:00
tomnewton-mulga	6389b37433	fix(gpu): Blackwell GPU support fixes for EKS/ECS image builds and VFIO (#559) ... * fix(eks): GPU node AMI declares uefi boot mode The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at firmware with zero serial console output. * fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to initialize under the closed/proprietary kernel module: dmesg shows RmInitAdapter failing with "requires use of the NVIDIA open kernel modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no separate open variant (userspace tools only), so it is unchanged. * fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons A cluster with a GPU-only nodegroup has no untainted node: the CP carries CriticalAddonsOnly (EKS parity keeps user workloads off it) and every worker carries nvidia.com/gpu. Without tolerations for both, ebs-csi-controller and aws-load-balancer-controller have nowhere to schedule and stay Pending forever. Real EKS never needs this since its control plane isn't a schedulable node, so the stock upstream manifests carry no such toleration. * fix(images): give the libguestfs appliance a working resolver The guest's /etc/resolv.conf symlinks to systemd-resolved's stub (127.0.0.53), which isn't running inside the appliance chroot, so DNS is dead until package installs need it. Point it at a real resolver before package installs, then restore the stub symlink afterward so the shipped AMI still gets its DNS from systemd-resolved at boot like a stock cloud image. * fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind A GPU's IOMMU group can include an upstream PCIe bridge when the platform lacks ACS isolation between them. vfio-pci refuses to bind non-endpoint bridge devices, so Claim/ReclaimByAddress must filter bridge-class (0x06) members out of the group before touching driver bindings, rather than failing the whole claim or leaving the bridge rebound incorrectly. * fix(ecs): GPU node AMI declares uefi boot mode The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at firmware with zero serial console output. Same defect and same fix as the eks-node-gpu entry: both images are built by the same script from the same stock Ubuntu cloudimg, so both were EFI-only and only one was corrected. * fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary kernel module. NVIDIA ships no proprietary module for Blackwell (GB202) and later, so ECS GPU container-instances could not bind those cards on any driver branch. It is the third verbatim copy of the routine and was missed when the other two were fixed. Also pins the driver branch across all three copies. The branch was resolved at build time via apt-cache search | sort -rn | head -1, which sorts on the number in the package name — but that number is a branch alias, not a driver version (nvidia-dkms-535-server installs 580.126.20), so the ordering key need not match what installs. Resolving also let identical source produce different drivers on different days. The pin lives in setup.sh rather than manifest.conf because manifest variables are not passed into the guest: the setup script is uploaded and executed by virt-customize --run, with no environment carried across. * fix(eks): tolerate CP and GPU taints in argocd addon All six argocd Deployments and its StatefulSet carried no tolerations. A cluster with a GPU-only nodegroup has no untainted node to place them on: the control plane carries CriticalAddonsOnly and every worker carries nvidia.com/gpu, so the pods stay Pending forever. Same defect the ebs-csi-controller and load-balancer-controller addons already had. Real EKS never hits it because its control plane is not a schedulable node, so the stock upstream manifests carry no such toleration and the divergence is ours to maintain. --------- Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>	2026-07-27 12:11:18 +10:00
Josh Siviour	b41a193cbc	fix(eks): make provider-id.sh the single k3s node-label authority for nodegroup + GPU labels (#539)	2026-07-15 00:39:13 +10:00
Josh Siviour	f7bf921ce7	fix(eks): NLB name overflow, nodegroup readiness scoping, launch retry, GPU konnectivity toleration (#511) ... * fix(eks): keep derived ELBv2 names under 32 chars and surface CREATE_FAILED reason Long cluster names made the derived "eks-<name>[-cp|-konn]" NLB/target-group names exceed the ELBv2 32-char limit, failing cluster creation asynchronously with no visible cause. safeELBv2Name now deterministically truncates the name and appends a stable hash of the full cluster name when needed, so the same name is derived on every call (create, lookup, delete). DescribeCluster also now surfaces ClusterMeta.StatusReason as a Health.Issues entry when a cluster is FAILED, instead of leaving callers to guess why. * fix(eks): plumb per-nodegroup Ready-node counts through state reports The on-VM state-report script now emits nodegroup_ready, a per-nodegroup Ready count keyed by the eks.amazonaws.com/nodegroup label value, alongside the existing cluster-wide node_count. ServerStateReport, the ClusterReconciler's observe step, and SetClusterHealthState now carry this map through to ClusterMeta.NodegroupNodeCounts, so per-nodegroup consumers (wired in the next commit) can gate on a nodegroup's own workers instead of the cluster total. * fix(eks): scope nodegroup Ready-gate per nodegroup and retry failed worker launches waitWorkersReady compared the cluster-wide Ready-node count against a nodegroup's baseline+desired, so in a multi-nodegroup cluster one nodegroup's Ready workers could mask another whose own workers never registered. It now reads meta.NodegroupNodeCounts[ng], scoped to nodes carrying the eks.amazonaws.com/nodegroup=<name> label. Separately, launchNodegroupInfra never retried a failed worker launch (a transient QMP timeout or host capacity pressure), leaving the nodegroup stuck CREATING with zero workers. The new launchWorkersUntilDesired wrapper relaunches only the shortfall on failure, logging each retry, until the desired count is reached or workerLaunchRetryTimeout elapses. Both fixes are covered by regression tests: a two-nodegroup scenario where only one has Ready workers, a transient-failure-then-success relaunch, and an exhausted-retry-budget CREATE_FAILED path. Not covered here: a worker that dies after the nodegroup is already ACTIVE — that needs an ongoing reconciler, which this change does not add. * fix(eks): tolerate GPU NoSchedule taint in konnectivity-agent DaemonSet A GPU-only worker is tainted nvidia.com/gpu=present:NoSchedule, so the konnectivity-agent DaemonSet (no tolerations) showed DESIRED=0 there and kubectl logs/exec against pods on that node failed "No agent available". Adds an Exists toleration for nvidia.com/gpu:NoSchedule. Live-cluster validation (DESIRED count, kubectl exec) is deferred — no cluster available to test against right now. * fix: tolerate nvidia.com/gpu taint on ebs-csi-node and coredns DaemonSets so GPU workers get storage and DNS	2026-07-12 12:11:16 +10:00
Josh Siviour	9f349e86ca	feat(eks): expose NVIDIA GPUs to pods (Epic B) — VFIO QMP, worker registration, device-plugin GPU access (#507) ... * feat(eks): expose NVIDIA GPUs to pods via CDI device plugin GPU nodegroups now label workers nvidia.com/gpu.present=true and taint them nvidia.com/gpu=present:NoSchedule (via a k3s config.yaml.d drop-in, since k3s has no K3S_NODE_TAINT env), and stage a hidden nvidia-device-plugin addon that addon-sync renders into the K3s auto-deploy dir. The device plugin runs with --device-list-strategy=cdi-cri and is gated to GPU nodes by nodeSelector + toleration, advertising nvidia.com/gpu allocatable so pods land on GPU nodes and get the device injected via the CDI spec baked by mulga-cdi-generate. Relies on containerd 2.0.5 default enable_cdi=true (no config fork). * fix(vm): scale QMP greeting deadline for VFIO passthrough guests A GPU passthrough guest (-device vfio-pci + large -m) must lock and DMA-map its entire guest RAM through the IOMMU before QEMU's QMP monitor answers, which exceeds the fixed 30s greeting deadline and gets the VM SIGKILLed at launch. Parameterize the greeting timeout (NewQMPClientWithGreetingTimeout) and scale it to 180s for VMs with GPU attachments; plain VMs keep the 30s default and reconnects against an already-running QEMU keep it too. * fix(eks): register GPU workers reliably and expose GPUs to pods Three fixes let a GPU nodegroup worker boot, register, and advertise its GPU: - k3s-agent.service gains Delegate=yes (plus KillMode/TasksMax/limits and overlay/br_netfilter modprobe): k3s embeds containerd+kubelet in-process, so on cgroup-v2 the kubelet cannot create /kubepods without a delegated subtree and ContainerManager.Start fails before Node registration. - The GPU node label rides a k3s config drop-in (20-gpu.yaml node-label + node-taint) instead of K3S_NODE_LABEL, which k3s ignores; without the label the nvidia-device-plugin nodeSelector never matches. - Raise the nodegroup ready timeout to 20m for the ~7min GPU first boot. Adds the GPU e2e (VPC + NAT-GW egress harness) and a slog-restore unit test. * fix(eks): install ecr-credential-provider on the GPU node AMI The GPU worker's kubelet is started (from nodegroup userdata) with --image-credential-provider-config + --image-credential-provider-bin-dir= /usr/local/bin, but the eks-node-gpu AMI never built or installed the ecr-credential-provider binary. kubelet treats a referenced-but-missing provider binary as a fatal RegisterCredentialProviderPlugins error, so the node crash-loops and never registers (while the Alpine eks-node worker, which ships the binary, joins fine). Build + install it in the GPU manifest; the server-only binaries stay omitted since GPU nodes are agent-only. Also tee k3s-agent stderr (and its embedded kubelet klog) to the serial console: Ubuntu logs only to journald, which is off-box-invisible on a worker in an isolated customer VPC, so a kubelet that fails to register left no trace. * fix(eks): give the nvidia-device-plugin its own GPU access The CDI-mode device plugin (--device-list-strategy=cdi-cri) must run nvml.Init and generate the CDI spec in its own container, but the DS had no GPU wiring, so nvml.Init crash-looped and nvidia.com/gpu was never advertised. Run the DS under the k3s-auto-configured nvidia runtimeClass with NVIDIA_VISIBLE_DEVICES/CAPABILITIES=all, the host root mounted read-only at /driver-root so it can locate the driver libs + ld.so.cache, and /var/run/cdi mounted writable so the generated spec is visible to containerd. No enable_cdi drop-in is needed on k3s v1.32.5 / containerd 2.0.5. GPU workload pods still need only resources.limits nvidia.com/gpu. Live-validated end-to-end on the single-GPU host (RTX A1000): baked control-plane AMI, GPU nodegroup, nvidia.com/gpu allocatable, nvidia-smi pod, zero manual intervention. * test(ec2): drop obsolete restoreSlogDefault test Rebased onto dev, which dropped the restoreSlogDefault workaround in #506 (viperblock no longer mutates the global slog). The test referenced the now-deleted helper and no longer compiles. Real GPU/registration diff is independently covered (diff coverage 95.3%).	2026-07-12 10:38:36 +10:00
Josh Siviour	5025a37006	feat: Containers — internal-ECR pulls, DNS-free registry, EKS hardening + docs (#409) ... * docs: add Containers section with EKS and ECR service pages New docs/containers/{eks,ecr} pages covering each service via AWS CLI, the Spinifex console, and Terraform, plus the requisite VPC/IGW/NAT/SG/IAM setup. EKS page embeds console screenshots. Cross-links the EKS Terraform workbooks. * docs(eks): rework Terraform workbooks into a capability ladder Rework the three EKS workbooks into a coherent ladder and publish them to the Containers section. All tiers are single- or three-node capable via node_desired_size and serve a shared Spinifex-themed demo app from ECR, replacing nginxdemos/hello. - demo-app: new std-lib Go web app (themed page reporting pod/node/cluster/ region + hit counter; optional /data persistence), distroless image. - eks-quickstart: node_desired_size var, ECR repo + themed app over NodePort. - eks-https-ingress (was eks-addons-access): HTTPS via the aws-load-balancer- controller addon + ACM, public/private VPC with NAT, managed-ingress=false. - eks-gitops-argocd (was eks-https-ingress): Argo CD addon syncs the app from a git repo, aws-ebs-csi-driver PVC (Viperblock-backed) for state, access entry RBAC, HTTPS Ingress carried over. Pending live e2e verification (image build/push, apply, ALB + PVC bind). * fix(ecr): implement ListTagsForResource (empty tags) for client parity The AWS provider calls ecr:ListTagsForResource on every repository read, which returned 501 NotImplementedException and broke 'tofu apply' of any workbook that creates an aws_ecr_repository. Implement it to return an empty tag set (resource tagging stays a deferred feature) so reads succeed. Drop the tags block from the three EKS workbooks' ECR repos so the provider doesn't call the still-stubbed TagResource on create. * feat(eks): pull from internal ECR via kubelet credential provider Worker nodes could not pull custom images from the internal Spinifex ECR registry: the eks-node had no DNS for the registry host, no containerd CA trust, no port rewrite to :9999, and no pull credentials. Only public images worked. Add cmd/ecr-credential-provider, a kubelet exec credential provider that fetches the node IAM-role credentials from IMDS, calls GetAuthorizationToken against the gateway ECR endpoint, and returns the minted AWS:<jwt> as Basic auth for the registry host. Thread the gateway URL, account, region, and internal suffix into the agent userdata, which now writes the gateway CA, a registries.yaml (CA trust + :9999 endpoint), the kubelet credential-provider config and config.yaml.d drop-in, and an /etc/hosts entry mapping the registry host to the gateway IP. Ship the provider binary in the eks-node AMI. Also remove a stray paragraph from the admin-cli README. * fix: dial EKS ECR registry mirror by gateway IP cloud-init manage_etc_hosts rewrites /etc/hosts from the alpine template on every boot, wiping the registry-host mapping appended via bootcmd, so worker nodes could not resolve the internal ECR host and pulls failed with no such host. Point the k3s registries.yaml mirror endpoint and tls config at the gateway IP instead (the gateway cert carries an IP SAN), keeping the pod image ref port-less and dropping the DNS leg until northstar lands. Fall back to the hostname endpoint when the gateway is not an IP. * fix: attach node-role instance profile to EKS workers for ECR EKS workers launched with no IAM instance profile, so IMDS served no role and the ECR credential provider could not mint a registry token, leaving internal ECR pulls failing with 401. Real EKS implicitly creates an instance profile for the node role; do the same here. Wire a KV-backed IAM service into the EKS deps and, at worker launch, find-or-create an instance profile named for the node role, attach the role, and pass it to RunInstances so IMDS exposes the role to the credential provider. * feat: use mulga logo in EKS demo app * feat: add ACM certificate tag support (List/Add/RemoveTags) Implements ListTagsForCertificate, AddTagsToCertificate and RemoveTagsFromCertificate across the store, service, NATS, daemon and gateway layers. Import now persists tags. Fixes terraform aws_acm_certificate InvalidActionException on ListTagsForCertificate. * test: cover ACM tag gateway and NATS wrapper paths * feat: support EKS nodegroup tags (Tag/Untag/ListTags) Extends EKS tag ops to back nodegroup ARNs alongside cluster ARNs. Nodegroup create-time tags persist on the record and echo via DescribeNodegroup, giving the terraform-aws provider a clean default_tags round-trip. Fixes aws_eks_node_group TagResource 501 NotImplementedException. * fix: provision EKS ALB ingress via node credentials The aws-load-balancer-controller addon could not provision an ALB. Three fixes: - TLS: the controller's per-service ec2/elbv2/acm SDK clients install their own BaseEndpoint resolver and build an HTTP client that bypasses AWS_CA_BUNDLE, falling back to the Go system trust pool and rejecting the gateway cert as 'unknown authority'. Add SSL_CERT_FILE so the gateway CA joins the system pool. - Credentials: an empty AWS_ROLE_ARN beside the web-identity token file half-arms the SDK web-identity provider, which then signs anonymously (MissingAuthenticationToken). Inject the IRSA env only when a service-account role ARN is staged; otherwise omit it so the default chain falls back to the node instance-profile credentials served over IMDS. - Authorization: the node role lacked the ec2/elbv2 actions the controller needs. The eks-https-ingress workbook now attaches an AWSLoadBalancerController managed policy to the node role (CreatePolicy + AttachRolePolicy). * fix: add force_delete to EKS workbook demo ECR repos tofu destroy of an EKS workbook left the demo ECR repository behind: it still holds the pushed demo image, so DeleteRepository (force=false) returns RepositoryNotEmptyException. Add force_delete = true to each workbook's aws_ecr_repository.demo so teardown removes the repo with its images. * fix: force-delete EKS control-plane ENIs on teardown TerminateK3sServerVM deleted the CP ENI with a plain force=false DeleteNetworkInterface. eniIsLiveAttachment reports InUse whenever the record's Status/InstanceId/AttachmentId are set, so when the K3s VM is gone but the ENI record's attachment fields were never cleared the delete returned InvalidNetworkInterface.InUse permanently and EKSDeletingReaper re-drove the same non-idempotent call every 90s, wedging the cluster in DELETING. Teardown owns its CP ENIs and has already terminated the VM, so the attachment is authoritatively dead. Replace the InUse-retry budget loop with detach-then-delete (detachAndDeleteServerENI): DetachENI clears the stale attachment fields, then DeleteNetworkInterface passes the in-use guard. Both calls tolerate an already-gone ENI (NotFound) so a race with the async instance-terminate cascade resolves to idempotent success. purgeClusterInfra's private-endpoint ENI path reuses the same helper. * fix: detach+delete worker ENIs on EKS nodegroup teardown A terminated nodegroup worker could leave its primary ENI in spinifex-vpc-enis as in-use/attached, and a dangling ENI throws DependencyViolation on subnet/VPC delete, blocking tofu destroy. Two gaps: TerminateWorkerInstances did a local-only vmMgr.Get and silently skipped any worker owned by another node, stranding both the VM and its ENI on multi-node clusters. Route the terminate per-instance via ec2.cmd.<id> (mirroring the gateway TerminateInstances path) so the owning daemon runs the full teardown — detach + force-delete the primary ENI and clear its record. No responder / NotFound payload is treated as already-gone so a retried DeleteNodegroup stays idempotent. deleteInstanceENI (stopped-instance terminate) issued a force=false DeleteNetworkInterface with no prior detach: a stale in-use attachment made it fail InvalidNetworkInterface.InUse, leaving the ENI record behind after the instance was already written to the terminated bucket. Detach first, then delete, tolerating NotFound. * fix: heal volume State drift on detach/terminate teardown A detach/terminate could leave a volume's State empty instead of 'available'. DeleteVolume gated on State=='available' exactly, so it returned VolumeInUse for an unattached volume nothing was using, leaving it undeletable and blocking stack teardown. DescribeVolumes cosmetically rendered an empty State as 'available', masking the drift. - DeleteVolume now treats an empty State with no attachment as deletable (an attached volume still returns VolumeInUse). - DescribeVolumes derives an empty State from the attachment (attached -> in-use, unattached -> available) instead of blindly rendering available, so an empty-but-attached volume is no longer hidden. - UpdateVolumeState normalizes an unattached volume with an empty state to available, so a detach writeback that omits the state cannot strand the volume in drift. * fix: break all SG cross-references on EKS teardown DeleteClusterSGs broke the cp<->nodegroup cycle too narrowly. It revoked only ingress (IpPermissions), never egress (IpPermissionsEgress), so a cross-reference via an egress rule kept the SG pinned on DependencyViolation. It also resolved only the two named cluster SGs, so any OTHER SG in the VPC referencing them — the default SG, the LBC/ALB SG, the private-endpoint SG, user/TF SGs — was never revoked and pinned the referenced SG on delete, leaving the VPC stuck. Enumerate every non-default SG in the VPC and revoke BOTH ingress and egress on each before deleting, breaking any cycle order-independently, then delete only the cluster-owned SGs (referrer SGs are deleted by their own teardown). The default SG is left untouched. Since DeleteClusterSGs runs against both the customer VPC and the managed control-plane VPC, the broadened sweep covers both teardown paths. * feat: support TargetGroup ProtocolVersion in elbv2 CreateTargetGroup now stores ProtocolVersion (defaulting HTTP/HTTPS target groups to HTTP1) and DescribeTargetGroups/Create output return it. Without this the aws-load-balancer-controller sees a perpetual immutable-field mismatch (desired HTTP1 vs empty) and recreates the target group every reconcile, looping on DuplicateTargetGroupName once a shared ALB IngressGroup reconciles more than one target group. NLB protocols (TCP/UDP/TLS) keep it omitted to match AWS. * fix: omit IRSA env from ebs-csi addon when no role staged The ebs-csi controller manifest hardcoded AWS_ROLE_ARN={{SERVICE_ACCOUNT_ROLE_ARN}} plus the web-identity token volume. With no service-account role staged the placeholder rendered empty, half-arming the SDK web-identity provider, which panics 'role ARN is not set' and CrashLoops the controller, leaving PVCs Pending. Convert to the conditional {{IRSA_ENV}}/{{IRSA_VOLUME_MOUNT}}/{{IRSA_VOLUME}} markers like the LBC bundle so the IRSA env and token volume are omitted when no role is staged, falling back to node instance-profile credentials over IMDS. The EBS-specific AWS_EC2_ENDPOINT/AWS_ENDPOINT_URL_EC2 stay in both modes. Requires an eks-node AMI rebake to take effect. * docs: route gitops-argocd workbook through a shared ALB and expose ArgoCD UI Fold the demo app and ArgoCD ingresses onto one ALB via an LBC IngressGroup with host routing (app./argocd. on a wildcard cert) instead of one ALB per ingress, add a NodePort + HTTPS Ingress for the ArgoCD UI, and document the access steps. Add the node-role LBC and EBS-CSI IAM policies the controllers need (Spinifex wires controller creds at the node level, not IRSA) and an ignore_changes block for the cluster's access_config and vpc_config security_group_ids, which Spinifex DescribeCluster does not echo back, so the provider stops trying to replace the cluster on every plan. * fix: stop addon-sync substituting IRSA markers named in comments The ebs-csi controller manifest's header comment spelled out the literal {{IRSA_ENV}} / {{IRSA_VOLUME_MOUNT}} / {{IRSA_VOLUME}} marker tokens. addon-sync matched each marker by substring, so the comment lines were replaced with the env/mount blocks, injecting a bare YAML sequence between two documents. K3s' auto-deploy then rejected the whole file (ParseManifestFailed: cannot unmarshal array into unstructured), so the ebs-csi Addon CR never applied and the EKS addon hung in CREATING. Reword the comment to not contain the literal tokens, and anchor the marker substitution to lines that are exactly the marker (whitespace aside) so a future comment mentioning a marker can't retrigger it. * fix: disable K3s local-storage on EKS clusters in parity mode In AWS LB Controller parity mode (BuiltinIngress=false) the K3s server disabled traefik and servicelb but left local-storage enabled. Its local-path provisioner ships a StorageClass marked default, which races the EBS CSI ebs-gp3 default StorageClass — two defaults, so dynamic PVCs bind nondeterministically. EKS has no local-path, so disable local-storage too and let ebs-gp3 be the sole default. The built-in-ingress path is unchanged. * fix: re-encrypt HTTPS target groups in haproxy ALB backends An ALB target group with backend-protocol HTTPS (e.g. argocd-server, which serves TLS on its own port) rendered a plaintext haproxy server line. HAProxy's own check then spoke HTTP to a TLS port, failed, and marked the server DOWN, returning 503 — even though the lb-agent's separate HTTPS health probe reported the target healthy, so describe-target-health masked the fault. Emit check-ssl ssl verify none on the server line for HTTPS target groups so both the check and proxied traffic re-encrypt to the backend. * fix: reconnect wedged QMP client on next command A timed-out QMP decode under host contention left the cached client's shared json.Decoder mid-message, wedging every later command and breaking EBS AttachVolume/DetachVolume for the VM's lifetime. Mark the client Dead on encode/decode error and redial the retained socket path (re-running the qmp_capabilities handshake in place) on the next send. * fix: round-trip security-group rule descriptions SG ingress/egress rules dropped the per-rule Description on store and describe. The AWS Load Balancer Controller tags the worker-node SG rules it manages with a UserIdGroupPair description and identifies them by that value to revoke them on teardown. Without the round-trip the controller cannot match its rules, never revokes them, and the backend security group stays referenced — DeleteSecurityGroup returns DependencyViolation forever, hanging the ingress finalizer and tofu destroy with orphaned security groups. Persist Description on SGRule (excluded from sgRuleKey so it is metadata, not identity) and render it back through describe. * fix: reclaim customer-VPC nodegroup SGs on managed-CP-VPC teardown In the managed control-plane VPC topology, launchNodegroupInfra creates the cluster control-plane and nodegroup security groups in the customer VPC for worker<->control-plane networking, but purgeClusterInfra only reclaimed the security groups in the managed CP VPC. The customer-VPC SGs orphaned, cross-referenced each other, and pinned the customer VPC with DependencyViolation, hanging tofu destroy of the VPC. Reclaim them in the managed-CP-VPC branch too, matching the legacy customer-VPC branch. * fix: reclaim ALB EIP when system VM already gone on terminate TerminateSystemInstance only released an internet-facing ALB's Elastic IP on the local path, where the VM is still present in the manager. When the VM was already gone (health-reaped, crashed, or raced by ENI teardown), it took the remote/no-owner path and returned idempotent success without reconciling the EIP, leaking the wan-pool address still associated to a deleted ENI. Add EIPServiceImpl.ReleaseAddressByInstanceID, an idempotent account-agnostic scan that disassociates then releases any EIP recorded against an instance, and call it as a backstop in TerminateSystemInstance for both branches. * fix: cascade-prune demo PVC via Argo CD finalizer in eks-gitops workbook The demo Application carried no resources finalizer, so tofu destroy removed the Application CR without pruning the synced PVC. The PVC survived, the cluster was destroyed, and the Delete-reclaim PV's EBS volume orphaned. Add the resources-finalizer.argocd.argoproj.io finalizer so deleting the Application cascades to the PVC and ebs-csi reaps the volume before cluster teardown. * feat: raw-IP access + documented Argo CD creds in eks-gitops workbook The shared ALB host-routed everything, so the raw ALB IP 404'd in a browser without an /etc/hosts entry. Make the app the hostless catch-all on :443 and add a hostless Argo CD listener on :8443, so https://<alb-ip>/ serves the app and https://<alb-ip>:8443/ serves the UI with no DNS. Surface the admin-password command as a tofu output and document the credential + IP URLs in the README. * fix: reclaim ALB EIP on the owning-node terminate path The EIP backstop lived only in TerminateSystemInstance, but the ALB delete path goes serveSystemTerminateInstance -> terminateSystemInstanceLocal directly, bypassing it, and releaseSystemInstanceEIP relied on the VM's cached allocation which is often empty. Extract reclaimSystemInstanceEIP (authoritative KV reconcile via ReleaseAddressByInstanceID) and call it from terminateSystemInstanceLocal as well as the no-owner backstop, so an internet-facing ALB's EIP is released on every terminate route. * chore(eks): drop coredns from the managed addon catalog Cluster DNS ships as the baked node-local coredns-mulga DaemonSet, not as a managed addon — there is no bundled coredns manifest, so CreateAddon coredns would stage but fail VM-side render. Remove the phantom catalog entry so DescribeAddonVersions and CreateAddon no longer advertise it; repoint the addon tests that used coredns as their sample to argocd. * feat(ui): group EKS and ECR under a Containers nav section Fold the separate EKS and ECR sidebar groups into one Containers group with EKS (clusters) and ECR (repositories) entries. * fix(ui): satisfy strict null checks in ECR component tests Index mock.calls via optional chaining and narrow the getAllByRole result before clicking, clearing the TS2532/TS2345 errors that failed pnpm fix. * refactor(eks): hide internal fixture addons from the public catalog Add an AddonSpec.Hidden flag and mark spinifex-noop hidden: it stays creatable (lookupAddon finds it, so the delivery-transport e2e still drives it) but no longer surfaces in the unfiltered DescribeAddonVersions listing. * refactor(eks): remove K3s built-in ingress; LBC-only Drop the BuiltinIngress option and its managed-ingress tag opt-in. traefik, servicelb, and local-storage are now always disabled on EKS nodes (AWS parity — Service type=LoadBalancer and Ingress are the Load Balancer Controller's job). Removes the EKS_DEFER_TRAEFIK userdata flag, the k3s.initd traefik .skip staging, and the maybe_enable_traefik deferred-enable loop in the state reporter. * fix(ui): clear lint debt in VPC-wizard mutations and CIDR validation Parallelize the independent subnet-create and route-table-association loops in the create-VPC mutation with Promise.all (each task records its resource in the rollback list before any further call), and switch the CIDR-validation regex to named capture groups. Clears the no-await-in-loop and prefer-named-capture-group lint errors. * fix: carry gateway port in ECR registry endpoints; rename mulga.internal -> spinifex.internal The ECR registry is served on the gateway port (:9999), but proxyEndpoint and repositoryUri were emitted port-less, so docker login/tag/push dialed :443 and failed. Derive the port from the awsgw listen host and append it to the registry host (omitted for 443). Also rename the default internal DNS suffix. * docs: drop verbose ECR :9999 host note * docs: fix literal-asterisk emphasis (single-* not rendered) * docs: brace-wrap ${REGISTRY}:latest (zsh :l modifier ate the tag); document Docker CA trust * feat: default ECR registry URIs to the gateway host (no-DNS interim) The registry authenticates by token, not hostname, so the per-account <acct>.dkr.ecr name is cosmetic. Advertise the gateway's reachable bind host:port (already a cert SAN) in repositoryUri/proxyEndpoint so docker login/tag/push need no DNS; the parity host still works server-side where DNS exists. * fix: serve ECR registry on a DNS-free gateway host end to end Advertise the gateway's AdvertiseIP as the ECR registry host when the AWSGW bind is unspecified (0.0.0.0/::), so repository_url resolves to a reachable host:9999 carried in the server cert SANs rather than the unresolvable per-account parity name. Docker push then dials it directly with no DNS. Mirror the same host on EKS workers: the registries.yaml mirror keys and the credential provider matchImages now cover the port-less parity name, the port-ful parity name, and the gateway IP:9999, all redirecting to the dialed gateway endpoint. A port-ful pull from the API-returned repository_url matches the mirror instead of bypassing it into a failed DNS lookup. Net: docker push host, pod image host, and the API's repository_url are the same value, with no DNS or /etc/hosts. * docs: resolve eks-https Ingress ALB DNS name to its public IP (no DNS) * docs: resolve eks-gitops Ingress ALB DNS name to its public IP (no DNS) * docs: remind to run aws eks update-kubeconfig before kubectl in EKS workbooks	2026-06-23 15:29:14 +10:00
Josh Siviour	9d662f9889	feat: EKS EBS CSI driver (Viperblock) — egress, attach, node discovery, e2e (#402) ... * feat: bake aws-ebs-csi-driver addon + mdev by-id bridge Add the aws-ebs-csi-driver entry (v1.40.1) to the eks addon catalog and ship its baked manifests (RBAC, IRSA-wired controller, node DaemonSet, default gp3 StorageClass) under the eks-node image. The node plugin locates EBS volumes via /dev/disk/by-id, which busybox mdev does not populate. Add mulga-ebs-byid: an mdev rule + helper that symlinks nvme-Amazon_Elastic_Block_Store_<serial> from the virtio-blk serial, plus a boot oneshot that points the kernel hotplug helper at mdev and coldplugs volumes present at boot. * test: e2e EBS CSI volume — PVC bind + mount + reschedule survival Add a TestEKS/EBSCSIVolume subtest that installs the aws-ebs-csi-driver managed addon bound to an IRSA role, provisions a default gp3 PVC + pod, and asserts a marker written by the first pod survives a reschedule onto a fresh pod backed by the same PVC. Exercises the full data path: CreateVolume(Viperblock)/AttachVolume virtio-blk hotplug with serial set to the volume-id, mdev /dev/disk/by-id symlink, and ext4 mount. * feat: eks-node provider-id + topology/instance-type labels for EBS CSI mulga-eks-provider-id boot oneshot writes a K3s config.yaml.d drop-in from IMDS before K3s starts: kubelet provider-id plus the region, zone and instance-type node labels. The aws-ebs-csi-driver node plugin reads this metadata for NodeGetInfo (instance id from providerID, region/zone from topology labels, instance type for the volume-limit family lookup). k3s runs no AWS cloud-provider so none of it is set otherwise, which left the node plugin's metadata init failing, NodeGetInfo panicking in IsNitroInstanceType, and the CSINode carrying no topology key. Wire the service into the image manifest (INSTALL_FILES + ENABLE_SERVICES) and chmod it in setup. * test: e2e EBS CSI controller IRSA policy + failure diagnostics awsgw enforces IAM on assumed-role EC2 calls, so the controller's CreateVolume/AttachVolume need a permission policy on the IRSA role. Attach a customer-managed EC2 policy (AWS-managed ARNs are opaque and grant nothing) covering the EBS CSI action set; CreateVolume then succeeds. Add an OnFailure dump of the CSI control path (provisioner and plugin logs, pvc/pod describe, events, csinode) to the EBSCSIVolume subtest so provisioning/attach failures are diagnosable post-teardown. * feat: EBS CSI obeys control-plane taint; e2e worker nodegroup Drop the CriticalAddonsOnly/NoExecute tolerations from the EBS CSI controller + node manifests so the driver runs only on customer worker nodes, never the system control-plane VM. The e2e creates a 1-node worker nodegroup and the writer/reader pods drop their CP tolerations so the customer-owned volume attaches in the customer space. * fix: EBS CSI attach — decouple hotplug PCIe port from device name; iothread rollback virtio-blk-pci hot-plug bus was derived from the AWS device-name letter (/dev/sd[f-p]); the EBS CSI driver sends arbitrary names (/dev/xvdaa) that map to no slot, so device_add fell back to pcie.0 — which rejects hot-plug — and the attach hung. A failed attach also left the per-volume iothread object behind, so every retry died on object-add 'duplicate property'. nextHotplugEBSPort allocates the lowest free hotplug-ebs port from live query-block state, independent of the device name. delIothreadBestEffort removes the iothread on every attach rollback path. * fix: EBS CSI node discovery — un-shadow mdev by-id rule, mint nvme link Alpine eks-node runs busybox mdev (no eudev), which stops at the first matching /etc/mdev.conf rule. setup.sh appended the mulga-ebs-byid rule after the stock 'vd[a-z].* -> persistent-storage' line, so it never fired on hot-attach: only /dev/disk/by-id/virtio-<serial> was created, while the upstream aws-ebs-csi-driver resolves nvme-Amazon_Elastic_Block_Store_<volid>. Swap the stock vd* rule in place to route through mulga-ebs-byid, which now delegates to persistent-storage (preserving stock links) before minting the nvme-style link the node plugin expects. * test: EKS e2e — NAT Gateway egress for private worker nodegroup The customer-VPC worker is private; AttachIGW programs no SNAT (mirrors AWS: an IGW serves only public-IP instances), so an IGW-only fixture left the worker unable to reach the public NLB endpoint or pull public CSI images — the nodegroup never joined. createWorkerEgress now builds the real topology: an IGW-fronted public subnet hosting a NAT Gateway, with the worker subnet default-routed to it. Associating the worker route table triggers the subnet-wide SNAT egress reroute. Teardown drains the NAT Gateway before releasing the public subnet + EIP it pins.	2026-06-19 14:43:20 +10:00
Josh Siviour	2acb37342d	feat(eks): aws-load-balancer-controller addon — real ALB Ingress (L7, IRSA) (#376) ... * fix(ec2): stamp subnet AvailabilityZoneId for single-AZ ALB resolution DescribeSubnets omitted AvailabilityZoneId, so the AWS Load Balancer Controller's DescribeAvailabilityZones-by-ZoneId lookup resolved no AZ info ("cannot resolve AZ info for AZs: []") and never built the ALB model. Stamp the single zone-id on every subnet and share it with DescribeAvailabilityZones through one SingleZoneID constant so the two responses can never drift. * feat(eks): add aws-load-balancer-controller v2.8.1 addon bundle Baked manifests for the LBC managed addon (CRDs, controller Deployment/ RBAC/webhooks/Service, IngressClass alb, IRSA render markers). Two wirings make it work against the spinifex gateway: - --aws-api-endpoints carries sts= alongside elasticloadbalancing/ec2/ acm. The aws-sdk-go v1 credential bootstrap resolves the STS client from the session endpoint resolver, not AWS_ENDPOINT_URL_STS, so the web-identity AssumeRoleWithWebIdentity must be pointed at the gateway here or it falls back to real AWS STS. - --feature-gates=ALBSingleSubnet=true drops the ALB minimum subnet count from two to one so an ALB provisions in a single-AZ deployment. * feat(eks): render IRSA into managed addons + node-minted webhook cert Extend the addon-sync renderer to inject the projected-token / role-arn / gateway STS-endpoint + CA block into managed-addon pod specs via opt-in markers, and substitute cluster-name/region/vpc-id/gateway-endpoint/CA placeholders. Add cmd/eks-webhook-cert: the eks-node AMI has no openssl, so this mints (then reuses) a self-signed ECDSA-P256 serving cert for the admission webhook, cached on disk so the render is byte-stable across ticks. Wire both into the image build (manifest.conf) plus INSTALL_FILES for the LBC bundle. * feat(eks): surface cluster VPC id as EKS_VPC_ID for the LB controller Thread the control-plane VPC id through K3sServerInput.VpcID into the first-boot EKS_VPC_ID env so the in-cluster aws-load-balancer-controller can pass --aws-vpc-id to the gateway elbv2/ec2 handlers. * fix(eks): point the LB controller at the customer VPC + customer-facing gateway The aws-load-balancer-controller ran with two control-plane-scoped values that customer worker pods cannot use: - --aws-vpc-id was set from the managed control-plane VPC (cpRefs.VpcID), which the gateway's DescribeVpcs does not return for customer calls, so backend-SG auto-create failed with InvalidVpcID.NotFound. Source it from meta.ResourcesVpcConfig.VpcId, the customer cluster VPC where the workers and their subnets live and where the ALB is provisioned. - The addon render baked the mgmt gateway endpoint (EKS_GATEWAY_URL / SystemGatewayURL) into pod specs, unreachable from worker pods. Add a distinct EKS_ADDON_GATEWAY_URL (the customer-facing GatewayBaseURL, same host as the OIDC issuer) and substitute it for {{GATEWAY_ENDPOINT}} in managed addon manifests. EKS_GATEWAY_URL stays mgmt for the CP's own bootstrap, state-report and token-webhook paths. * fix(elbv2): resolve forward target group from ForwardConfig listenerActionFromSDK read only the legacy flat Action.TargetGroupArn, so a forward action carrying its target group via ForwardConfig.TargetGroups (the shape the aws-load-balancer-controller emits) left the action target empty and CreateRule/CreateListener rejected it with MissingParameter. Flatten the first target group from ForwardConfig when the flat field is absent. * fix(ec2): mirror CreateTags/DeleteTags into subnet/vpc records The generic S3 tag store and the per-resource KV records are disjoint: DescribeSubnets/DescribeVpcs read tags from their KV record, while CreateTags only wrote the generic store. Subnet tags applied after create were therefore invisible to tag-filtered describes, breaking aws-load-balancer-controller subnet auto-discovery (kubernetes.io/role/elb). Write through CreateTags/DeleteTags to the owning subnet/vpc record so tag-filtered describes observe them. Resource types the VPC service does not own are skipped; the generic store stays their record of truth. * test(daemon): pin synthetic capacity in host-dependent allocate tests TestCanAllocate_CountEdgeCases/Mixed_instance_types and TestHandleEC2DescribeInstanceTypes/NoFilterReturnsSupportedRegardlessOfAllocation called rm.allocate against live-detected host capacity. allocate gates on readMemAvailableGB (live /proc MemAvailable), so a box already running VMs fails the allocation spuriously, breaking the preflight gate locally. Pin hostVCPU/hostMemGB/reserved and nil readMemAvailableGB to a synthetic host, matching the sibling Capacity_decreases_after_allocation subtest. * fix(eks): route apiserver->pod traffic via agent tunnel (egress-selector cluster) In the managed-CP topology the apiserver VM sits in the system CP VPC with no route to the worker pod network in the unpeered customer VPC. k3s default egress-selector-mode=agent tunnels only apiserver->kubelet, so kubectl logs/exec work but apiserver->pod traffic (admission webhooks, aggregated APIs) is attempted directly and times out -- the aws-load-balancer-controller webhooks fail with context deadline exceeded, blocking Service/Ingress creation cluster-wide. Set egress-selector-mode=cluster so apiserver->pod/service traffic routes through the agent's outbound tunnel, selected by the node hosting the destination, without requiring inbound CP->worker reachability. * feat: bump aws-load-balancer-controller addon bundle to v2.11.0 v2.8.1 predates the ALBSingleSubnet feature gate, so single-AZ spinifex clusters fail Ingress reconcile (subnet resolver dedups to 1 < required 2) and never provision an ALB. v2.11.0 honors --feature-gates=ALBSingleSubnet=true. Replace the baked bundle: new CRDs (ingressclassparams + targetgroupbindings), controller manifest, and image tag v2.11.0, re-applying the spinifex deltas (IRSA SA annotation, node-minted webhook serving-cert secret, gateway-CA secret, gateway aws-api-endpoints, caBundle on every webhook clientConfig, alb IngressClass). Pick up the new ClusterRole configmaps rule. Drop v2.8.1. * fix: point LBC at the gateway via AWS_ENDPOINT_URL (sdk-v2) LBC v2.11 uses aws-sdk-go-v2, which keys --aws-api-endpoints on SDK service IDs (e.g. 'Elastic Load Balancing v2'), not the legacy lowercase names the bundle passed. Those overrides were silently ignored, so every non-STS call (ec2/elbv2/acm) went to real AWS and failed TLS against the gateway CA -- no ALB could provision. Inject AWS_ENDPOINT_URL in the shared IRSA env block (sdk-v2 global base-endpoint override; AWS_ENDPOINT_URL_STS still takes precedence for STS) and drop the dead --aws-api-endpoints flag from the controller args. * fix: synthesize taggable ARN for default listener rule Gateway DescribeRules returned the synthetic IsDefault rule with a null RuleArn. LBC read the arn-less default rule after creating a listener and issued DescribeTags with an empty resource ARN, which the gateway rejected with 400 MissingParameter, aborting reconcile. Set a deterministic RuleArn on the default rule, resolve it back to its parent listener in DescribeTags (empty tags) and DescribeRules by-arn. Also disable shield/waf/wafv2 in the LBC bundle to drop unsupported calls. * feat: inject ELB subnets into alb IngressClassParams for single-AZ ALB LBC's tag auto-discovery never threads the ALBSingleSubnet feature gate, so a single-AZ cluster (subnets sharing one AZ) dedups to 1<2 subnets and aborts Ingress reconcile. Only the explicit-subnet path honors the gate. Resolve the cluster's ELB-eligible subnets, dedup to one per AZ, and surface them as EKS_ELB_SUBNET_IDS. The addon sync renders them into the alb IngressClassParams .spec.subnets.ids so every Ingress takes the explicit path. An empty set falls back to auto-discovery, preserving the native UX. * fix: route LBC service clients to gateway via --aws-api-endpoints, link IngressClassParams Live verification showed two gaps in the LBC addon bundle: LBC installs a per-client BaseEndpoint resolver, so the global AWS_ENDPOINT_URL is ignored for the ec2/elbv2/acm clients (calls hit real AWS, failing TLS). Redirect them with --aws-api-endpoints keyed on the aws-sdk-go-v2 service IDs (EC2, "Elastic Load Balancing v2", ACM, STS) rather than the legacy lowercase names the flag never matched. Drop the misleading AWS_ENDPOINT_URL env; the web-identity STS client still resolves via AWS_ENDPOINT_URL_STS. The injected IngressClassParams was orphaned: the alb IngressClass had no .spec.parameters, so LBC fell back to subnet auto-discovery (which fails on a single-AZ cluster). Link the IngressClass to the IngressClassParams so the explicit-subnet path is taken. * fix: stamp EKS-parity providerID on worker nodes for LBC ip targets k3s registers workers with providerID k3s://<name>, which the AWS Load Balancer Controller rejects (providerID invalid for EC2 instances) when resolving nodes for ip-mode target registration. Resolve the instance-id and AZ from IMDSv2 at agent boot and set --kubelet-arg=provider-id=aws:///<az>/<id> so LBC can map the node to its EC2 instance. On any IMDS miss the agent keeps the default providerID and still starts. * fix(elbv2): honor SubnetMappings in CreateLoadBalancer LBC supplies ALB subnets via SubnetMappings, not the plain Subnets list. createLoadBalancer only read input.Subnets, so ENI allocation was skipped, VpcId/AZ stayed empty, and the HAProxy data-plane VM never launched — the ALB went active but could not serve traffic. Add flattenSubnetIDs to read both sources and reuse it in createLoadBalancer and desiredSubnetSet. * fix(vm): capture GCInterval before spawning GC goroutine TestStart_SweepsImmediatelyThenStopsOnCtxCancel mutates the package global GCInterval and restores it in a defer, racing the Start goroutine's read at time.NewTicker(GCInterval) under -race. Read GCInterval on the caller's goroutine into a local before spawning so the global access is ordered with the test's defer; the goroutine touches only the captured local.	2026-06-18 10:53:05 +10:00
Josh Siviour	bc9556d1a9	fix(eks): managed-CP worker data-plane — apiserver reachability, node-local CoreDNS, SG teardown (#368) ... * fix: persist instance TagSpecifications in RunInstance RunInstance built the ec2.Instance metadata but never read input.TagSpecifications, so DescribeInstances returned untagged instances and tag filters matched nothing. EKS node groups discover their workers by the spinifex:eks-cluster / spinifex:eks-nodegroup tags, so scale-up convergence and worker reconciliation depended on tags that were silently dropped. Apply instance-scoped TagSpecifications onto the built instance so the tags surface through DescribeInstances and tag filters. * fix: serialize nodegroup reconcile via CAS to prevent scale overshoot UpdateNodegroupConfig is dispatched on a queue group, so two overlapping requests (e.g. an SDK retry mid-flight) each read current = len(InstanceIDs) and each launched desired-current workers, overshooting the target (1 -> 5 for desired=3) and stranding untracked VMs via last-writer-wins on the unconditional PutNodegroupRecord. Make every durable nodegroup-record mutation a compare-and-swap on the KV revision. Scale-up launches one worker at a time and CAS-appends it only while the record is below the live ScalingDesired, terminating any surplus VM launched into an already-full record. Scale-down terminates before the CAS truncation so a crash or retry leaves reclaimable dead IDs rather than orphaned VMs. A concurrent regression test asserts two overlapping scale-ups converge to exactly desired under -race. * fix: return securityGroupRuleSet from AuthorizeSecurityGroup ingress/egress The AWS provider's aws_vpc_security_group_ingress_rule reads SecurityGroupRules[0].SecurityGroupRuleId on create; an empty ruleset panics the plugin (Plugin did not respond), though the rule applies server-side. Echo the created rules via sgRuleToSecurityGroupRule, the same shape DescribeSecurityGroupRules already returns. * fix: surface InsufficientInstanceCapacity from nodegroup scale, not 500 launchWorkersCAS wrapped the bare awserror code via fmt.Errorf, so the gateway ErrorHandler's exact-match against ErrorLookup missed and remapped to InternalError (500). An out-of-capacity scale now surfaces the real InsufficientInstanceCapacity code (400); opaque internal failures still wrap and stay 500. Adds awserrors.HasErrorCode for the membership test. * fix: cut EKS node boot ~50s — drop cloud-final deadlock + syslinux menu Worker/server user-data runcmd ended with `[ rc-service, local, start ]`, which runs inside cloud-final while the OpenRC `local` service is ordered after it — a deadlock broken only by OpenRC's ~50s start timeout, stalling the k3s join. Run the IMDS route script directly instead and keep `rc-update add local default` for reboot persistence. setup.sh: cut the SYSLINUX boot-menu countdown 10s->~1s (kept nonzero so the menu stays interruptible over serial). Validated live on env19 CP: boot menu 1s, no `waiting for cloud-final` stall, apiserver ready ~66s (was ~109s+). mulga-siv-332 * feat: add argocd managed addon + bake bundle into eks-node AMI The addon catalog advertised coredns and aws-load-balancer-controller, but the only baked manifest shipped in the eks-node AMI was the spinifex-noop fixture. Installing any real addon failed with 'no baked bundle for version' once the VM-side renderer found nothing on disk. Add argocd v3.0.23 as a real, demonstrable managed addon: catalog entry (no IRSA), a flattened namespaced install manifest baked into the AMI, and the manifest.conf bake line. The eks-addons-access workbook switches from coredns (redundant on K3s, never baked) to argocd. The baked bundle is generated from upstream install.yaml via kubectl kustomize with namespace: argocd plus a prepended Namespace, since upstream is namespace-agnostic and K3s auto-deploy applies with no -n flag. * fix(eks): run CoreDNS as a node-local DaemonSet for worker DNS k3s ships CoreDNS as a single-replica Deployment that tolerates the CP taint and strands itself on the control-plane node. In the managed-CP topology the worker<->CP pod overlay cannot cross the VPC boundary, so worker pods cannot reach a CP-resident CoreDNS and lose all cluster + external DNS. Skip the bundled CoreDNS (coredns.yaml.skip, staged in k3s.initd like traefik) and deploy a Mulga-owned variant: a DaemonSet (one pod per node) plus kube-dns internalTrafficPolicy=Local, so every node resolves via its own CoreDNS. Immune to the cross-VPC overlay gap and topology-agnostic (single-node and managed-CP both work). Image-side only; no handler change. * fix(eks): revoke cross-referencing SG rules before delete on teardown DeleteClusterSGs deleted the control-plane and nodegroup SGs directly, but they cross-reference each other (EnsureNodegroupSGRules), and AWS refuses to delete a security group still referenced by another SG's rule. Both deletes failed with DependencyViolation, the SGs leaked, and the VPC could not be deleted (tofu destroy hung on DeleteVpc). Resolve both SG IDs, revoke all ingress on each to break the cp<->ng cycle, then delete both. Adds RevokeSecurityGroupIngress to sgProvisioner (already on VPCServiceImpl) and revokeAllSGIngress, with a regression test that models AWS's DependencyViolation. * fix(eks): advertise NLB front-end so worker pods reach the apiserver ClusterIP In the managed-control-plane topology the apiserver advertised its Set-B node-ip into the in-cluster `kubernetes` Endpoints. That IP lives in the unpeered CP VPC and is unreachable from worker pods, so every worker pod using the apiserver ClusterIP (CoreDNS kubernetes-plugin sync, controllers, argocd redis secret-init) failed. - k3s_server_vm.go: set advertise-address to the worker-reachable NLB front-end (PrivateEndpointIP, else EndpointIP); both are already cert SANs. - nlb.go: add a second :6443 listener on the cluster NLB (same CP target group) so PrivateEndpointIP:6443 — the advertised endpoint — is served. - security_groups.go: admit :6443 on the private-endpoint SG. Validated on a live managed-CP cluster: worker CoreDNS /ready 200, all argocd pods Running, kubectl logs on worker pods working.	2026-06-16 20:27:21 +10:00
Josh Siviour	b23fa3bf60	feat(eks): managed control-plane VPC (Set B) + control-plane lifecycle hardening (#356) ... * feat(eks): VM-side managed-addon delivery transport Stage CreateAddon manifests in KV host-side; the control-plane VM pulls the staged set through the AWS gateway (signed GET /clusters/{name}/internal-addons/ {accountId}), renders the baked bundles into the K3s auto-deploy dir, and reports per-addon delivery status on a dedicated eks.addon.*.status channel that the cluster reconciler CASes onto AWS-visible AddonStatus (CREATING->ACTIVE). - gateway: ListInternalAddons route + ListStagedAddonManifests NATS action - handlers: addon status reconciler (nextAddonStatus), staging installer - eks-node AMI: eks-gateway-fetch binary + mulga-eks-addon-sync agent - agent renders idempotently (cmp before replace) so K3s auto-deploy does not re-apply every tick and clear the Addon CR .status.gvks readiness signal - agent mirrors per-tick delivery outcomes to the serial console for host-side observability * fix(eks): provision cluster NLB synchronously so the endpoint carries the real front-end IP The cluster NLB front-end IP is assigned by the async LB-VM launch (launchLBVMAsync), but EnsureClusterNLB read it back synchronously right after CreateLoadBalancer returned — losing the race and falling back to the synthetic <lb>.elb.spinifex.local DNS name. That name is unresolvable (no Route53) and is not in the apiserver serving-cert SAN, so kubectl against the cluster never connects. Split the elbv2 create: the public CreateLoadBalancer stays async (267.4 — an inline multi-minute VM boot head-of-line-blocks the gateway responder), and a new CreateLoadBalancerSync drives the data-plane launch inline and returns the LB carrying its front-end address. The EKS cluster-create worker runs off the gateway responder, so it uses the sync entry and reads the IP from the return value instead of racing persisted state. No polling. - elbv2: factor provisionLBDataPlane out of launchLBVMAsync; add CreateLoadBalancerSync - eks: ensureClusterLB uses the sync entry, sets FrontendIP from the result, drops the read-back; EnsureClusterNLB fails loud (ErrClusterNLBFrontendIPUnavailable) when no front-end IP is allocated rather than shipping the DNS-name fallback - eks: drop the DNS-name endpoint fallback; the endpoint is always the front-end IP * feat: auto-wire IGW for internet-facing EKS cluster endpoint An internet-facing cluster NLB allocates an external front-end IP and the OVN dnat_and_snat for it, but the external switch + localnet + gateway LRP that make that IP answerable on the wire are built only on IGW attach. The control-plane NLB is a system instance, so ensure an IGW on cluster build and reclaim it on teardown. EnsureClusterIGW (idempotent, reuses a customer IGW untouched) runs before EnsureClusterNLB when the endpoint is public; DeleteClusterIGW (ownership- scoped) removes only the cluster-owned IGW in purgeClusterInfra, before the VPC delete. New IGW dep wired from the daemon. * fix(eks): tear down infra on failed cluster launch to stop LB EIP leak A CreateCluster that fails during its background launch flipped the meta to FAILED but performed no teardown, relying on a later same-name create to reclaim and purge the infra. EKS cluster names are unique per create, so that reclaim never fires: the internet-facing load-balancer VM keeps running with its allocated+associated dnat_and_snat EIP indefinitely. Over repeated create/delete cycles the EIP pool fills with allocations bound to terminated instances, so a later allocation's advertised endpoint and its live OVN NAT diverge onto different pool addresses. - failClusterLaunch now runs a best-effort purgeClusterInfra on the in-memory meta (freshest refs even when the failing step was the meta write itself) before MarkClusterFailed. purgeClusterInfra is idempotent, so a later reclaim or DeleteCluster re-purge is a no-op; the FAILED meta is retained for observability. - Daemon releaseSystemInstanceEIP extracted and called from both TerminateSystemInstance and cleanupFailedSystemInstance, so a launch that fails after AssociateAddress releases the eipService allocation instead of leaking it (MarkFailed's ReleasePublicIP only knows the externalIPAM path). * feat: EKS managed control-plane VPC (AWS Set B) composed from EC2 APIs CreateCluster now auto-builds the managed control-plane VPC under the system account, the spinifex analogue of AWS EKS's hidden managed-account VPC: a public subnet for the internet-facing NLB, private subnet(s) for the k3s control-plane VM(s), and a NAT gateway giving the private CP egress for image pulls. It is composed from the real EC2 VPC-family APIs (CreateVpc/Subnet, IGW, RouteTable/Route, NatGateway), so per-subnet egress gating is wired by the existing topology subscribers off the route-table events those APIs publish — no direct OVN mutation. This resolves the mulga-siv-292 root cause: the internet-facing NLB and the private CP were crammed into one customer subnet with no IGW route, so both were treated as private and dropped by the lr_in_policy 1100 rule, killing the LB-VM return leg and lb-agent heartbeat. The CP only survived via a per-instance 1200 reroute bandaid, now removed — the topology expresses egress instead. Placement is uniform single/multi-node (CP always in the private subnet). The system LB VM heartbeats over the mgmt bridge (SystemGatewayURL) rather than the WAN. Managed-VPC teardown reclaims NAT GW + EIP, route tables, subnets, IGW, and the VPC in dependency order. Customer subnetIds (Set A / worker side) remain consumed-not-created and are deferred to the worker round. * fix(eks): bake cluster-owner account into CP-VM EKS_ACCOUNT_ID The managed control-plane VPC (Set B) launches the K3s control-plane VMs and their ENIs under the system (infra) account, but the on-VM agents — bootstrap publish, state report, token-review, and add-on fetch — must key on the cluster-OWNER (customer) account that owns the cluster meta, namespaces, and AccessEntry KV. Add ClusterAccountID to K3sServerInput and bake it into the VM's EKS_ACCOUNT_ID, decoupling it from the infra AccountID the VM launches under. Without this the VM reported state and fetched add-ons against the system account, so the cluster never observed a healthy control plane (stuck CREATING, null CA) and kubectl/add-on delivery timed out. Also refresh the e2e suite doc comment to describe the Set B topology and the data-plane subtests (kubectl + managed-addon delivery) that now run. * fix(eks): explicitly delete addon objects on unstage K3s does not delete the applied resources when an auto-deploy manifest file is removed from the server manifests dir (documented behaviour). The addon-sync GC pass relied on that, so after DeleteAddon the rendered manifest was dropped but its Namespace/ConfigMap stayed standing — the e2e AddonDelivery GC wait timed out. Copy the manifest aside, drop the original so K3s stops tracking it, then kubectl delete the objects (--wait=false to avoid finalizer stalls).	2026-06-12 22:29:24 +10:00