12 Commits

Author	SHA1	Message	Date
Josh Siviour	3e195506d3	feat(images): mkosi system-image migration — CPU eks-node/ecs-node + GPU variants (#605) ... * fix(eks): GPU node AMI declares uefi boot mode The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at firmware with zero serial console output. * fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to initialize under the closed/proprietary kernel module: dmesg shows RmInitAdapter failing with "requires use of the NVIDIA open kernel modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no separate open variant (userspace tools only), so it is unchanged. * fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons A cluster with a GPU-only nodegroup has no untainted node: the CP carries CriticalAddonsOnly (EKS parity keeps user workloads off it) and every worker carries nvidia.com/gpu. Without tolerations for both, ebs-csi-controller and aws-load-balancer-controller have nowhere to schedule and stay Pending forever. Real EKS never needs this since its control plane isn't a schedulable node, so the stock upstream manifests carry no such toleration. * fix(images): give the libguestfs appliance a working resolver The guest's /etc/resolv.conf symlinks to systemd-resolved's stub (127.0.0.53), which isn't running inside the appliance chroot, so DNS is dead until package installs need it. Point it at a real resolver before package installs, then restore the stub symlink afterward so the shipped AMI still gets its DNS from systemd-resolved at boot like a stock cloud image. * fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind A GPU's IOMMU group can include an upstream PCIe bridge when the platform lacks ACS isolation between them. vfio-pci refuses to bind non-endpoint bridge devices, so Claim/ReclaimByAddress must filter bridge-class (0x06) members out of the group before touching driver bindings, rather than failing the whole claim or leaving the bridge rebound incorrectly. * fix(ecs): GPU node AMI declares uefi boot mode The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at firmware with zero serial console output. Same defect and same fix as the eks-node-gpu entry: both images are built by the same script from the same stock Ubuntu cloudimg, so both were EFI-only and only one was corrected. * fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary kernel module. NVIDIA ships no proprietary module for Blackwell (GB202) and later, so ECS GPU container-instances could not bind those cards on any driver branch. It is the third verbatim copy of the routine and was missed when the other two were fixed. Also pins the driver branch across all three copies. The branch was resolved at build time via apt-cache search | sort -rn | head -1, which sorts on the number in the package name — but that number is a branch alias, not a driver version (nvidia-dkms-535-server installs 580.126.20), so the ordering key need not match what installs. Resolving also let identical source produce different drivers on different days. The pin lives in setup.sh rather than manifest.conf because manifest variables are not passed into the guest: the setup script is uploaded and executed by virt-customize --run, with no environment carried across. * fix(eks): tolerate CP and GPU taints in argocd addon All six argocd Deployments and its StatefulSet carried no tolerations. A cluster with a GPU-only nodegroup has no untainted node to place them on: the control plane carries CriticalAddonsOnly and every worker carries nvidia.com/gpu, so the pods stay Pending forever. Same defect the ebs-csi-controller and load-balancer-controller addons already had. Real EKS never hits it because its control plane is not a schedulable node, so the stock upstream manifests carry no such toleration and the divergence is ours to maintain. * refactor(images): extract eks-node initd logic into init-agnostic helpers The OpenRC runscripts carried their start_pre logic inline, so porting a service to systemd meant re-deriving that logic under a distro change rather than just rewriting the init wiring. Move it into /bin/sh helpers that either init system can drive, keeping eks-node on Alpine/OpenRC. k3s-prestart takes the restore-block guard, config.yaml derivation, konnectivity-agent and CoreDNS staging, and the token-webhook wait. konnectivity-server-prestart takes the CA wait and cert mint, plus the server-count derivation that used to run at parse time. Both keep every guard, bound and message as-is. Values flow back the way the GPU image already does it: the helper writes shell assignments to a /run file and the init system sources them, so command_args is built in start_pre where it used to be a top-level assignment. k3s-agent-prestart moves to eks-node and becomes the single copy. The GPU manifest claimed its helpers were reused verbatim from eks-node while hand-copying this one; it now points at the shared file, so the two cannot drift. k3s-agent.initd still sources agent.env and exports itself rather than delegating: OpenRC has no EnvironmentFile= equivalent, and the daemon inherits only what the runscript exports, so a helper subprocess sourcing it would not reach k3s. The helpers run under set -eu, which the runscripts did not. The awk over first-boot.env is guarded on the konnectivity manifest, not on the env file, so a missing env file exits awk nonzero; without || true that turned a warn-and-continue into a refusal to start k3s. Covered by a test that fails on rc=2 if the guard is dropped. * feat(images): build system images in a pinned mkosi container Move the image-build toolchain into a container so a runner needs only Docker. The toolchain cannot drift between runners, and the same build runs identically on a developer's machine. mkosi builds in a user namespace, which two of Docker's default confinements block: the seccomp profile masks CLONE_NEWUSER, and the docker-default AppArmor profile separately blocks unsharing the mount namespace. Both are relaxed; neither is a privilege grant, as capabilities stay dropped and no devices are exposed. mkosi also runs as an ordinary user rather than root, since as root it takes a path that needs real CAP_SYS_ADMIN. Verified end to end: an Ubuntu 26.04 Format=disk image builds from the Debian container and the artefact is a valid GPT disk, with the package cache on a named volume cutting a forced rebuild from ~2min to ~34s. * feat(images): publish system images from the build container Convert mkosi's raw output to a compressed qcow2, write the .sha256 sidecar the spx catalog verifier matches, and upload both to R2 using the same pinned toolchain as the build, so publishing needs nothing on the host either. The toolchain image build moves to a shared helper rather than being duplicated across the two entry points. Publishing needs no relaxed confinement, unlike the build: qemu-img and aws create no namespaces, so it keeps Docker's default seccomp and AppArmor profiles. The object path carries no version and the catalog resolves newest-wins, so an upload overwrites the live image with no way back. Copy the current object to a timestamped archive key first, making a bad publish a restore rather than an unrecoverable loss. * fix(images): build the image on the output's mount, not the cache volume mkosi assembles the image in its workspace and moves the result to the output directory. The workspace lived under the cache volume, which is a different mount from the output bind mount even when both sit on one filesystem, so the move failed EXDEV and mkosi fell back to copying — requiring the finished image to exist twice at once. That is invisible on a small image and fatal on a real one: a ~4G GPU image needed ~8G to land and died with ENOSPC after a successful build. Point the workspace at the output's own mount so the move is a rename again and the image only exists once. The package cache stays on the volume, which is the part worth persisting across runs. * feat(images): add mkosi base config and gpu-nvidia profile Establish images/ with a base mkosi.conf plus a gpu-nvidia profile, replacing the throwaway smoke-test config. The base builds a bootable UEFI disk; the profile adds the NVIDIA stack. Most of ubuntu-gpu-nvidia-setup.sh does not carry over. Its kernel-version hunt, the do_bootloader=no kernel-img.conf hack and the manual update-initramfs all exist only because that pipeline chroots into a prebuilt cloud image that already has a kernel. mkosi installs the kernel and owns the bootloader, so the version is known rather than detected, and the postinst asserts exactly one kernel instead of searching for it. The static configuration (nouveau blacklist, unattended-upgrades blacklist, cloud-init docker group) becomes files under mkosi.extra/ rather than heredocs. linux-image-generic drags in every linux-firmware split package, none of it relevant to a headless VM guest; left at the default the modules initrd embeds all of it into the UKI, which grew to ~880MB and overflowed the 512M ESP. The initrd module set is trimmed to mkosi-initrd's curated default, which is a boot fix and not only a size one. Verified in the built artefact, not just the build log: nvidia.ko.zst and the four sibling modules under /usr/lib/modules/<kver>/updates/dkms, the nouveau blacklist, and the nvidia runtime in docker's daemon.json. * fix(images): unblock guest boot on wait-online and firstboot A guest built from these images stalled twice on the way to a login prompt, both times on a unit waiting for input that never arrives in a VM. systemd-networkd-wait-online holds network-online.target until every managed link is up, and docker.service pulls that target in with Wants=, so any image carrying docker inherits the wait whether or not it wants it. A guest whose links are slow or absent ran the unit to its full 120s default with multi-user.target queued behind it. Override it to --any with a short timeout: one link up is what a single-NIC guest actually needs. systemd-firstboot separately prompts on the console for anything the image does not already carry, before multi-user.target. With no timezone baked in, boot stopped dead at "Please enter the new timezone name" and waited for a keystroke. Setting locale, keymap and timezone at build time leaves firstboot with nothing to ask. Both are baked into the base rather than delivered as first-boot drop-ins: an image built from scratch has no cloud-init to write them. Measured on wattle with an A1000 passed through: login at 42.8s, down from ~210s, and the firstboot prompt is gone. nvidia-smi in the guest reports "NVIDIA RTX A1000, 595.71.05, 8188 MiB". Also widen the images gitignore to output-*/ so a scratch build cannot leave multi-GB artefacts stageable. * feat(images): make the base image reachable — cloud-init, sshd, console An image built from scratch launched unreachable. The images this replaces start from a stock Ubuntu cloud image and inherit cloud-init, sshd, sudo, a network renderer, dbus and a serial console from it; mkosi installs only what is listed, so none of it was there and nothing said so. Guests get their SSH key and user-data from IMDS via cloud-init, so without the package a launched instance has no authorized_keys at all — no error, just no way in. It also made any cloud-init-delivered config inert, including the gpu profile's docker-group drop-in. Each addition is load-bearing on its own: cloud-init consumes IMDS: keys, user-data, hostname, network openssh-server the key needs a sshd to be installed into netplan.io cloud-init's renderer on Ubuntu; with no renderer the NIC is never configured, IMDS is never reached, and cloud-init falls back to DataSourceNone dbus systemd only Recommends it and mkosi installs no Recommends, so systemctl could not reach the system bus — and the EKS and ECS bootstraps drive guests with systemctl from user-data lsb-release cloud-init shells out to it; without it every boot reports "degraded" forever and the status stops being a usable signal The kernel also needs console=ttyS0 baked in. The launcher gives every guest a serial port and get-console-output reads it, but a kernel with no console= writes to none of it: the console stopped after the UEFI bootloader and every boot failure was invisible. ttyS0 goes last because Linux makes the final console= the controlling /dev/console. manage_etc_hosts is set because cloud-init renames the guest per launch but at its default never writes a matching hosts entry, so sudo warned on every call. datasource_list is deliberately NOT pinned: the guest identifies the platform from the SMBIOS system UUID the launcher spoofs, ds-identify narrows to Ec2 on its own, and pinning per image is the customisation this migration deletes. Verified on wattle with an A1000 passed through: ssh in as ubuntu with spinifex-key, cloud-id "aws", cloud-init status done (not degraded), no failed units, nvidia-smi reports the A1000 on 595.71.05, docker active, boot 26.7s. * feat(images): let the guest grow its root into the volume it was given mkosi sizes the disk to its content, so the image ships with a root partition that fills the disk to its last sector and ~1.4G free. Nothing in it can expand that: cloud-init's growpart module is enabled, but the growpart binary lives in cloud-guest-utils, which is not installed and is not pulled in as a Recommends by anything in the package set. systemd-repart is not an alternative — Ubuntu's systemd package carries no repart binary, unit or definitions at all. The result was a node handed a larger volume that it could only ever use the first few GiB of, with the rest allocated and unreachable. That is fatal for a worker whose job is to pull container images. gdisk is deliberately left out despite being a cloud-init Recommends: growpart resizes GPT through sfdisk, which arrives via the hard fdisk dependency, and e2fsprogs brings the resize2fs that the resizefs module runs afterwards. Verified on wattle: the gpu-nvidia image launched with a 20G root reports 20G on df -h / (was 4.4G), via "'/' resized: changed (/dev/vda2) from 4770582528 to 20936900096", with nvidia-smi still reporting the A1000 and no failed units. * feat(images): move eks-node-gpu and ecs-agent-gpu onto mkosi profiles Adds the eks-common, eks-agent and ecs profiles, so both GPU node images build from the composed base rather than a hand-authored tree, and the three verbatim copies of install_nvidia_gpu_stack() collapse into gpu-nvidia alone. Splits Docker out of gpu-nvidia into its own profile. Docker belongs to the standalone GPU AMI, whose purpose is `docker run --gpus all`; the EKS and ECS node images run k3s or containerd and reach the GPU through CDI. Fusing the two was fatal for ECS rather than merely wasteful: gpu-nvidia installed containerd.io from Docker's repository, which Conflicts/Replaces the Ubuntu containerd the ecs profile needs, so gpu-nvidia + ecs could not resolve at all. nvidia-container-toolkit stays in gpu-nvidia — every GPU image needs nvidia-ctk to generate its CDI spec. Closes three gaps the pilot left, each of which only shows up on a booted guest: /etc/modules-load.d/nvidia.conf, without which nothing loads the modules on a headless node and CDI generation finds no GPU; mulga-cdi-generate.service, which no profile shipped; and an apt-mark hold on the toolkit packages, which the unattended-upgrades blacklist does not cover for a plain `apt upgrade`. mkosi-build.sh gains --image, naming an output and expanding it to an ordered profile list. The order is load-bearing: mkosi runs each profile's postinst in the order the profiles are passed, so docker's nvidia-ctk wiring must follow gpu-nvidia. Leaving that to callers meant an argument-order mistake produced a healthy-looking image with no GPU wiring, so docker's postinst also fails loudly when gpu-nvidia is selected but nvidia-ctk is absent. Profiles reference build inputs where they already live via ExtraTrees= instead of copying them, keeping one copy of the helpers shared with the Alpine image. Go binaries are built on the host into a staging tree, since the builder container carries no Go on purpose: go.mod is the single pin for the toolchain and a second pin in the Dockerfile could disagree with it silently. * fix(images): install systemd-resolved so the guest has a DNS resolver mkosi installs only the packages named in Packages= plus their hard dependency closure, and systemd-resolved is merely a Recommends of systemd. So it was never installed: the image shipped a 0-byte /etc/resolv.conf that nothing populated, because netplan renders to systemd-networkd and networkd alone does not write resolv.conf. glibc then fell back to localhost and every lookup failed with "[::1]:53 connection refused", which reads as a network fault rather than a missing package. A node that cannot resolve cannot pull a container image. The ECS GPU node booted, registered as a container instance, then failed every task at image pull. The EKS image carries the same defect but is masked by the cloud-init bootcmd in nodegroup_userdata.go that writes resolv.conf at launch; the ECS launch path has no equivalent, so only ECS exposed it. * fix(images): name build output by image and never report a skipped build as success Two defects in the build wrapper that let the wrong bytes reach a boot test. Every composition wrote the same spinifex.raw, taken from the base config's ImageId, so an artefact carried no record of which profiles produced it and an EKS image was distinguishable from an ECS one only by which build ran last. Passing --image-id names the output after the image, and as a side effect the name lands in the guest's /usr/lib/os-release. Publishing correspondingly loses its default input: --name is the published object and --image the local artefact, and with nothing tying the two together a defaulted input meant whatever raw happened to be in the output directory could be uploaded under the name the caller asked for. mkosi also skips the build whenever the output path merely exists. It does not compare inputs, so it never notices an edited mkosi.conf or a restaged binary, prints "Use --force to rebuild" and exits 0 — a stale image indistinguishable from a fresh one. Builds are now always forced, and the run additionally asserts the expected raw was written during this invocation rather than trusting the exit status. The cache worth keeping is the package download volume, which --force does not touch. * feat: add gpu-amd mkosi profile for the ROCm AMI Ports scripts/images/ubuntu-gpu-amd-setup.sh so ubuntu-gpu-amd builds under mkosi as 'gpu-amd docker'. Without it the legacy scripts/images/ tree has to stay alive for one remaining image. Pins the ROCm repo by apt priority rather than only adding it: Ubuntu 26.04 universe ships its own vendored rocminfo, which unpinned apt preferred over AMD's, mixing release trains within one image. Uses a systemd unit for render/video group membership instead of a cloud-init runcmd, because cloud-init's config merge takes a duplicated runcmd key wholesale from one fragment and silently discards the other, which would have dropped the docker profile's own usermod. Runtime GPU verification is deferred — no AMD hardware is available. * feat(images): add eks-server, ec2-node and CPU eks-node/ecs-node mkosi images Ports the K3s server side of EKS off the legacy Alpine/OpenRC builder: nine systemd units, a templated etcd-snapshot service with daily and 15-minute timers, the addon/coredns/konnectivity assets, and the six server binaries. konnectivity-server is not an in-repo package, so it gets a pinned host-side clone-and-build rather than an entry in profile_binaries(). Splits ec2-node out of eks-common. mulga-ebs-byid mints the by-id links for any attached volume and mulga-mgmt-net configures any multi-NIC system VM, so neither belongs behind an EKS profile that a database image would have to compose to reach them. Start-at-boot moves into preset files. mkosi runs preset-all after every postinst, so a postinst enable is discarded before the image is finished, and with no policy file the fallthrough is enable — which built an image that ran a control plane and a worker at once. Installs the cloud-init datasource pin as 99- rather than 90-. cloud-init-base ships its own 90_dpkg.cfg declaring datasource_list, read_conf_d sorts the directory in reverse and the merge keeps the first source, so a 90- prefixed file lost the key silently. The postinst now asserts the merged result. * refactor(eks-node): make the role selector init-agnostic eks-node-role and k3s-first-boot drove OpenRC directly, so on the systemd image the selector enabled nothing and the node never started k3s. Both now dispatch through svc_enable/svc_start/svc_disable on a detected init, with EKS_NODE_INIT overriding detection so either branch is testable on a host that has only one. Detection prefers rc-update over systemctl: the test suite stubs rc-update on PATH, and preferring systemctl would have silently rerouted those cases to a path they were not written for. Enable and start stay two separate passes. The server path starts mulga-eks-k3s-recovery before k3s so a recovery directive applies before the apiserver comes up, and collapsing to systemctl enable --now would interleave them. The dispatch is duplicated in both scripts rather than shared: they are single-file deliverables with no library-path convention between them. Tests cover both branches across all three roles, including that server-join still gets no k3s-first-boot, and that rc-update wins when both inits are present. * fix(eks-node): enqueue role-selected units instead of blocking on them eks-node-role.service declares Before=k3s.service, and mulga-eks-k3s-recovery.service declares After=eks-node-role.service, so a blocking systemctl start deadlocks: systemd will not run the queued job while the selector is still activating, and the selector cannot finish until that job runs. Type=oneshot defaults TimeoutStartSec=infinity, so nothing breaks the tie and the node wedges mid-boot with the role written but no control plane. On a booted guest the selector sat in activating for 261s until the VM was torn down. Start with --no-block on systemd. The ordering the blocking call was implicitly providing is already declared between the units themselves, so systemd sequences them correctly on its own. OpenRC runs the script directly rather than scheduling it, so that branch keeps blocking and keeps surfacing failures through set -e. * fix: escape $ in konnectivity-server ExecStart so server-count survives systemd expands $VAR/${VAR} in Exec*= from the unit's own environment before the command runs, substituting an empty string for a name it does not know. KONN_SERVER_COUNT is not a unit variable — konnectivity-server-prestart writes it to /run/konnectivity-server-args.env, which the wrapper shell sources at runtime — so systemd rewrote the flag to `--server-count=` before /bin/sh saw it. konnectivity-server then died on flag parsing, which is why the unit logged Started and produced nothing despite --logtostderr=true and journal+console, and why the apiserver's cluster egress selector found no UDS for the life of the cluster. This is a port artefact. konnectivity-server.initd is flag-for-flag identical and shares the same prestart script, but under OpenRC the shell was the only thing reading ${KONN_SERVER_COUNT}, so the same text was correct. Escape it as $$ to defer expansion to the wrapper shell, and add lint_unit_expansions() to reject any unescaped $VAR/${VAR} in an Exec*= line of a profile's units. The linter joins line continuations first, since the offending reference sat mid-way through a wrapped multi-line ExecStart, and runs on the host before the container starts so a bad unit fails the build in seconds rather than at first boot. $(cmd) is deliberately not flagged: systemd leaves it literal. Verified on env12 with a rebuilt image: TestEKS/IRSAPod and TestEKS/EBSCSIVolume, which both previously failed on `kubectl exec ... timed out` through the egress tunnel, now pass, with zero UDS connect errors across the run. * fix: refresh the stale qcow2 mkosi-build leaves beside a fresh raw mkosi only emits the raw. A qcow2 next to it came from a previous run's hand conversion, and nothing about its name says which build it belongs to, so it survives a rebuild and silently ships the old image to a node while the raw beside it carries the new one. Observed on a rebuild that left the raw at 15:34 carrying a fix and the qcow2 at 12:45 carrying the pre-fix unit. This is the same failure the existing freshness assertion guards against, one file over, so extend that block rather than add a second check. Refresh rather than delete: the qcow2 is what gets copied to a node and imported, so removing it would push the manual conversion, and this trap, back onto the caller. Convert via a temp name and rename so an interrupted run leaves the old file rather than a half-written one that looks current. * fix: resolve the awsvpc task DHCP client instead of hardcoding udhcpc ecs-agent leases a task ENI's OVN-assigned address inside a per-task netns, which systemd-networkd does not manage, so it shells out to a DHCP client. That call was hardcoded to busybox udhcpc, which Alpine ships and Ubuntu does not. `ip netns exec` exits 1 rather than 127 when it cannot exec the binary, so the failure surfaced as an opaque "network setup failed" with the task stuck in STOPPED, and it took every awsvpc-mode workload with it: TaskAwsvpc and ServiceWithELB both failed on the mkosi image, the latter as runningCount=0 because its target group targets task ENI IPs. Resolve the client at setup time, preferring udhcpc so Alpine behaviour is unchanged and falling back to dhcpcd. Presence of the udhcpc binary is not sufficient to select it: Debian and Ubuntu's busybox ship the applet without /usr/share/udhcpc/default.script, and udhcpc then takes the lease and exits nonzero having configured nothing. Same order and same reasoning as dhcp_oneshot() in mulga-mgmt-net.sh. Both invocations stay one-shot, leaving no daemon behind in the netns. Install dhcpcd-base in base, because neither client was present in any profile. That also repairs mulga-mgmt-net.sh, whose own udhcpc/dhcpcd fallback resolved to nothing on Ubuntu and would have failed the first time a cross-host control-plane data NIC needed it — the S5 path the step-6 plan notes has never been exercised. The DHCP-client choice is injected so the tests assert a named platform's branch rather than whatever happens to be installed on the machine running them; without that the same test exercises different code on an Alpine CI box and an Ubuntu dev box. * refactor: retire the ec2-node profile, fold its helpers into the base image ec2-node carried mulga-ebs-byid and mulga-mgmt-net, and every EKS composition had to remember to list it — eks-common even guarded on its presence and aborted the build without it. ECS did not list it, so the ECS image shipped with no by-id symlinks for hot-attached volumes at all. Neither helper is answerable at build time: any image may later be attached a volume, and any image may later be launched as a control plane. Both are inert when unused (mulga-ebs-byid is udev-driven; mulga-mgmt-net exits 0 when its fw_cfg blob is absent), so the base image is where they belong. That deletes the composition rule, the guard, and the ECS gap in one move. Also drops ec2-node from the two eks image compositions in mkosi-build.sh. Verified on freshly built spinifex-ecs-node and spinifex-eks-node raws: both carry the helpers and the udev rule, mulga-ebs-byid is enabled via multi-user.target.wants and mulga-mgmt-net via cloud-init.target.wants. * fix(eks): force UseMTU=false on netplan units so DHCP renewal can't reinflate the pinned link mulga-vpc-mtu only won the boot race by ordering: netplan renders UseMTU=true (it does not inherit networkd's own false default), so a DHCP lease renewal at T1 could silently re-apply the advertised MTU to the link. usemtu_off() forces UseMTU=false via a runtime drop-in in the systemd-documented [DHCPv4] section for every rendered unit and reloads networkd, removing the dependency on boot ordering entirely. assert_pinned() then verifies the link and its routes actually match before exiting, so any regression is a loud boot-time failure instead of a silent one. Both are no-ops on the Alpine/dhcpcd image, which has no systemd-networkd. * fix(images): state UseMTU=false under both the current section and the legacy alias netplan renders the DHCP settings under [DHCP] while systemd documents [DHCPv4]. A drop-in naming only one parses cleanly and does nothing if the running systemd disagrees, which is the silent no-op the drop-in exists to prevent. * fix(ecs): install iptables on both ecs-node images, cover task-role credential fetch credendpoint.go DNATs the ECS task credential endpoint via iptables; neither the mkosi ecs profile nor the legacy Alpine manifest installed it, so the endpoint silently failed to start and any task with a task role could not fetch credentials while the instance still looked healthy. Also fixes a latent bug in the e2e task-role trust policy (Service-principal-only, which this platform's HTTPS AssumeRole path can never match) and adds a TaskRoleCredentials subtest that fetches real credentials from inside a task container, so a dead endpoint fails the suite instead of passing unnoticed. * fix(images): ship preset files for base, eks-common and ecs start-at-boot mkosi runs systemctl preset-all after every profile's postinst, discarding whatever systemctl enable a postinst performed. eks-server and eks-agent already carry preset files for this reason; base, eks-common and ecs still relied on the default-enable fallthrough, which happens to match today but is one upstream package change from reversing. Adds 10-mulga-base.preset (mulga-ebs-byid, mulga-mgmt-net), 30-mulga-eks-common.preset (mulga-eks-provider-id, mulga-vpc-mtu) and 40-mulga-ecs.preset (containerd, ecs-ca-install, ecs-agent; also disables cni-dhcp.service/socket, which containernetworking-plugins enables by default with nothing to serve under the baked host-local IPAM conflist). Removes the now-redundant systemctl enable calls from each postinst, matching the eks-server/eks-agent precedent. * fix(sts): attribute ecs-tasks.amazonaws.com to container-instance callers The HTTPS AssumeRole path passed an empty principalSource, so matchServicePrincipal denied every Service clause other than the IMDS-only EC2 one. A role carrying the trust policy AWS documents for an ECS task role was unassumable by construction. Service attribution is now derived from the authenticated SigV4 caller: an assumed-role session of the account's ecsInstanceRole — the only credential the in-guest agent holds — is attributed ecs-tasks.amazonaws.com, and only for roles in its own account. Nothing in the request body contributes, so a task holding its own task-role session cannot claim the service principal. * fix(images): remove baked SSH host keys, pin Seed= for reproducibility openssh-server's postinst bakes an identical host key pair into every build, making every guest from one build mutually MITM-able; sshd-keygen already regenerates keys on first boot (ConditionFirstBoot=yes), so the baked pair is pure non-determinism with no purpose. Also pin Seed= so systemd-repart stops assigning fresh partition UUIDs on every build. Verified on a built spinifex-ecs-node image: no ssh_host_* files ship, cloud-init still lists ssh in cloud_init_modules, and sshd-keygen.service is wired via ssh.service.wants/ (redundant regeneration path). A two-build comparison dropped content-differing files from 9 to 1 (only the UKI, whose zstd internals remain sensitive to unrelated input changes). --------- Co-authored-by: tomnewton-mulga <tom.newton@mulgadc.com>	2026-07-30 12:57:43 +10:00
Josh Siviour	4793b5ac09	fix(eks): node route MTU blackholing image pulls, and teardown wedged in DELETING (#562) ... * fix(eks-node): stop the DHCP MTU overriding the pinned link MTU Pinning eth0 to 1320 was never enough. This image runs dhcpcd, which honours DHCP option 26 and stamps the advertised MTU onto every route it installs: default via 10.32.101.1 dev eth0 proto dhcp src 10.32.101.7 \ metric 1002 mtu 1442 Linux prefers a route's MTU metric over the device MTU, so the node kept advertising an MSS derived from 1442 while the real path MTU through the NAT gateway is 1408. Small packets pass and full-MSS inbound segments — a TLS ServerHello with its certificate chain, image layers — are silently dropped. It presents as `net/http: TLS handshake timeout` on image pulls, not as anything MTU-shaped, and it only affects the host netns, so containerd pulls fail while pod traffic sized off cni0 is fine. Per-destination PMTU exceptions expiring in and out of the route cache made it look intermittent and host-specific. Three changes, innermost first: - dhcpcd.conf gets `nooption interface_mtu`, so no lease can stamp a route MTU. mulga-vpc-mtu is the single source of truth for the MTU of these nodes. - mulga-vpc-mtu re-stamps any route that already carries a different MTU, covering an image built before this fix or a dhcpcd already running. Routes with no MTU metric inherit the device MTU and are left alone. - the subnet DHCP option drops 1442 to 1408. 1442 is geneve-only and ignores the IPsec ESP overhead on the NAT egress path, so it overstates the usable MTU for every guest, not just these nodes. Also corrects the header comment, which claimed Alpine's udhcpc ignores option 26 — this image runs dhcpcd, which honours it. That wrong premise is why the route metric was never considered. Adds a stubbed-`ip` test covering the rewrite, the dev token that `ip route show dev X` omits, idempotency, and routes without an MTU metric. * fix(eks): sweep the CP VPC by identity so teardown can finish DeleteClusterCPVPC finds the managed control-plane VPC's subnets and SGs by role tag. A create that fails partway — the first control-plane server failing to launch, say — leaves resources behind that never got tagged, so the sweep cannot see them. DeleteVpc rejects while any subnet or non-default SG remains, and teardown is re-driven on a timer, so the cluster sits in DELETING forever retrying the same doomed delete every couple of minutes: eks-deleting: re-driving wedged DELETING teardown cluster=toc eks-deleting: re-drive teardown failed err="delete managed CP VPC: eks: delete cp vpc vpc-...: DependencyViolation" Nothing converges: the cluster name cannot be reused and terraform's delete wait can never pass. Observed on env19, wedged for ~11 hours until this change went in, at which point the next re-drive cleared it. After the tagged sweep, clear anything still in the VPC scoped by vpc-id: subnets, and non-default SGs (the default SG goes with the VPC cascade). Teardown had no SG surface, so CPVPCDeps gains an optional SG provisioner — nil is tolerated, since only teardown uses it. * chore: pin viperblock + predastore to the merged storage fixes viperblock f8cfd77 (#42) makes read-modify-write atomic per block, removing the lost update that silently reverted a 4096-byte block to a previous generation of itself. Confirmed as the env19 corruption cause by an A/B: the pre-fix build corrupted image layers within ~15 minutes (failed to extract layer / invalid tar header / unpigz: corrupted) while the fixed build ran 5+ hours clean, absorbing 6407 same-block RMW conflicts on the worker root disks. predastore c0a90d2 (#63) batches FSM.Restore. Without it, any predastore restart on a store whose metadata set outgrew one badger transaction fails with "Txn is too big to fit into one request" on every node at once, taking the metadata plane — and with it the AMI catalogue and cluster creation — down permanently. Both were verified in the built artifact rather than from go.mod alone: go version -m reports the new pseudo-versions, and the rmw_conflicts counter plus the batched-restore error strings are present in the binary. Pinning by go.mod while the submodule pointer says otherwise is how an earlier analysis ended up reading dead code. * test: record the corrected DHCP MTU in the kitchen-sink golden The NB golden pins the DHCP options published per subnet, including the advertised MTU. Dropping the subnet option from 1442 to 1408 changes what the reconciler writes, so the golden diverged and the live scenario test failed — the test doing exactly its job. Regenerated with UPDATE_GOLDEN=1. The diff is only mtu:1442 -> mtu:1408 on the two subnet DHCP records; nothing else in the snapshot moves.	2026-07-22 12:57:36 +10:00
Josh Siviour	e47baf321c	feat(eks): HA control-plane recovery + reconciliation, VM restart hardening, viperblock AEAD pin (#480) ... * feat(eks): reconciler in-place control-plane restart on sustained degradation Phase 1 of EKS CP recovery (siv-243). The ACTIVE-branch reconciler only recorded health; a CP VM that died (OOM/host crash) left the cluster wedged ACTIVE-but-unreachable with no recovery path. Add a CPInstanceControl surface and bounded in-place restart: once an ACTIVE cluster's CP stays unhealthy past a grace window, StartInstance is issued (backoff-spaced, capped) when the instance is stopped/error, re-mounting the same root volume so embedded etcd survives. Degradation stays AWS-faithful ACTIVE + a ClusterHealth issue rather than a non-AWS DEGRADED status enum. cpControl nil keeps the reflect-only path. Wiring of a real CPInstanceControl in the daemon and the error-state manager-eviction fix are the remaining Phase-1 integration steps; Phase 2 (etcd snapshot cadence + restore-snapshot) is tracked separately. * feat(eks): wire control-plane restart into the reconciler via instance service Threads a CPInstanceControl into the per-cluster reconciler (siv-243 Phase 1). cpControlAdapter binds the daemon instance service + account: InstanceState reads the CP's EC2 state via DescribeInstances, StartInstance restarts a stopped CP via StartStoppedInstance, which loads from the shared stopped store and re-mounts the same root volume so embedded etcd survives. Wired as EKSServiceDeps.CPControl = d.instanceService and passed through spawnReconciler. Recovers a cleanly-stopped control plane co-located with the reconciler leader. Remaining: DescribeInstances is node-local so a cross-node stopped CP needs an aggregating describe; and an error/crashed CP (not in the stopped store) needs a dedicated error-state restart path — both tracked as Phase-1 follow-ups. * feat(eks): cross-node CP describe + error-state control-plane restart The reconciler's control-plane recovery was wired to the node-local instance service: DescribeInstances only saw the leader's node and StartStoppedInstance only handled a cleanly-stopped CP from the shared KV. A crashed/OOM CP ends in error state, still resident in its owning node's manager map (not the stopped store), so recovery never reached it — the actual incident in the recovery plan. Wire EKSServiceDeps.CPControl to a NATS-backed eksCPControl. DescribeInstances fans out across every host (gateway scatter-gather) so the reconciler observes a CP on any node. RecoverInstance drives the gateway StartInstances two-step: ec2.cmd.<id> recovers a live error/running owner in place (the owner stays subscribed through the crash; vmMgr.Start relaunches error->pending, re-mounting the etcd root volume), falling back to ec2.start shared-KV rehydration for a stopped CP. The CP VM is ManagedBy=EKS, hidden from customer DescribeInstances; relax that hide so the owning account resolves its own managed CP when targeting it by explicit instance ID (bulk listings still hide it). EC2APIState already maps the internal error state to the AWS closed-enum stopped, which cpRestartable accepts. * fix(eks): reconcile CP as system account, not customer The control-plane VMs are launched under the system account (admin.SystemAccountID()), not the customer account that owns the cluster record. spawnReconciler was binding cpControlAdapter to the customer account, so InstanceState/RecoverInstance ran as the customer — which cannot see or own its own cluster's system-owned CP VMs. DescribeInstances returned zero and the reconciler logged 'control-plane instance not found', never restarting a wedged CP. Bind the adapter to the system account instead. Revert the earlier DescribeInstances managed-VM relaxation: it was based on the mistaken assumption that CP VMs are customer-owned. Visibility is resolved by the system-account adapter, so the explicit-ID escape hatch is dead code. * fix(vm): pre-create mgmt tap on disk-boot restart path The direct-boot branch pre-creates the management tap via NetworkPlumber.SetupTap (owned by the daemon euid), but the disk-boot/else branch only appended the QEMU tap,id=mgmt0 device without creating the tap. The daemon runs as a non-root user without CAP_NET_ADMIN, so QEMU can only attach a tap it already owns; on the disk-boot path QEMU tried to create /dev/net/tun itself and failed with 'Operation not permitted', failing every stopped-instance restart (StartInstances returned ServerInternal). This blocked EKS control-plane recovery: the reconciler found the stopped CP and issued the restart, but the VM never booted. Call SetupTap for the mgmt tap in the disk-boot branch, mirroring direct-boot. * feat(eks): recover all HA control-plane members, not just primary The reconciler restarted only ControlPlaneInstanceID (the primary CP member), so on an HA cluster the other server VMs stayed stopped and etcd never regained quorum (1/3) — apiserver healthz stayed fail even after recovery ran. maybeRecoverControlPlane now iterates every member in ControlPlaneNodes (falling back to the scalar ID for single-node clusters), restarting each stopped/error member per pass while preserving the grace window, attempt cap, and backoff gates. Refs mulga-siv-509 * fix(eks): keep cloud-init polling IMDS so primary CP gets its user-data The eks-node system VM's user-data is delivered only via IMDS (the NoCloud seed was retired in #411). IMDS (169.254.169.254) is reachable only once mulga-mgmt-net brings up the data NIC via DHCP and the per-tap br-imds datapath reconciles, which on the first-launched (primary) CP can lag cloud-init's stock 120s probe window. With the default max_wait cloud-init falls back to DataSourceNone, permanently losing the launch user-data (no SPINIFEX_K3S_ROLE -> no k3s server) and wedging the cluster in CREATING. Bake a cloud.cfg.d drop-in that pins DataSourceEc2 and raises max_wait to 600s so cloud-init keeps polling until the datapath is ready instead of falling through to None. Worst case a slow boot, never a lost role. * fix(eks): warm IMDS instance-role creds before first SigV4 sign On IMDS-credentials mode the on-VM SigV4 helpers (eks-token-webhook, eks-gateway-publish, mulga-eks-state-report) sign via the SDK default chain, which resolves in-VM IMDS instance-role creds. The per-tap br-imds datapath lags VM boot by minutes (observed ~264s), and the SDK's IMDS client fast-fails each probe (250ms dial), so early Retrieve calls return 'context deadline exceeded' / 'no EC2 IMDS role found'. eks-gateway-publish only retries for 30x5s=150s, shorter than the datapath warm time, so it exhausts its budget and drops the bootstrap/state publish -> cluster hangs in CREATING. The credential chain is sound (the CP role trusts ec2.amazonaws.com and AssumeRoleForInstance evaluates that principal); the failure is timing. Add gwsign.Signer.EnsureCredentials and block in eksgw.New until the instance-role creds first retrieve (2s interval, 10m ceiling aligned with cloud-init max_wait), so every helper's first signed request already holds creds instead of racing a cold datapath. Static-key mode is unaffected. * fix(eks): retry CP data-NIC DHCP until cross-host datapath is up A single ~16s DHCP one-shot on the primary data NIC lost the race against the cross-host OVN datapath (logical-port binding + per-tap reconcile on the remote placement host), stranding the NIC IP-less so the Ec2 datasource never reached IMDS and the CP member wedged. Add dhcp_acquire: retry the one-shot on a budget aligned with cloud-init's Ec2 max_wait (600s), confirming an inet address before success. Each attempt still exits leaving no daemon, so cloud-init's later ENI management is unaffected. * fix(eks): drop reconciler registry entry when Run self-exits ReconcilerRegistry.Spawn only cleaned up its holder on ctx cancel (Stop/StopAll). When a ClusterReconciler's Run returned on its own — a terminal cluster state or a lost lease — nothing cancelled the ctx, so the holder leaked and a later same-name create silently no-opped at the head of Spawn, never reconciling to ACTIVE. Thread a finished channel from RunClusterReconciler's Run goroutine back to Spawn's watcher, which now cleans up on whichever fires first: an external cancel or Run exiting. On self-exit it deletes its own holder and releases the lease, so a fresh same-name create re-spawns and reconciles. * fix(eks): run mulga-mgmt-net in boot runlevel before cloud-init-local CP user-data is delivered only via IMDS since the NoCloud seed was retired (#411), so cloud-init DataSourceEc2 must reach 169.254.169.254. mulga-mgmt-net DHCPs the data NIC that makes IMDS reachable, but it was added to the default runlevel and only declared `before cloud-init` (the network-stage service). cloud-init-local therefore ran its init-local Ec2 crawl first with no data-NIC route and burned its whole probe window (~4 min) before mulga-mgmt-net brought eth0 up, regressing CP bringup from ~1m50s to ~5m54s. Add mulga-mgmt-net to the boot runlevel in setup.sh instead of default, drop it from ENABLE_SERVICES, and order it `before cloud-init-local` so the data NIC and IMDS route are up before the init-local crawl. Keeps the IMDS-only user-data path; the raised Ec2 max_wait stays as a safety net for host br-imds datapath lag. * fix(vm): refresh LaunchTime on crash-restart so watchdog spares relaunch RestartCrashedInstance re-launched a crashed VM without refreshing Instance.LaunchTime. The pending watchdog measures now-LaunchTime against PendingWatchdogTimeout, so the stale original launch time tripped launch_timeout on the next tick and terminated the crash-restart mid-flight. Refresh LaunchTime under the state lock, mirroring the daemon-restart restore path (restore.go), so the relaunch gets a fresh timeout window. Adds a regression test asserting LaunchTime advances. * feat(eks): reconcile control-plane member count after a lost member The cluster reconciler self-heals a stopped/error control-plane VM in place but never replaced a terminated/gone member, so an HA cluster ran below haControlPlaneCount permanently. Add a member-count reconcile pass that provisions one replacement CP joining the surviving etcd quorum. - CPProvisioner + EKSServiceImpl.ProvisionReplacementCP: replay the persisted create template on a free host, the single-node analogue of the spread join branch, with rotating creds re-derived. - ControlPlaneTemplate persisted on ClusterMeta at create (per-node and rotating-cred fields cleared). SwapControlPlaneMember CAS swaps the dead node for the replacement and refreshes the [0] scalar mirrors. - maybeReplaceControlPlaneMember fires only when no member is restartable and one is truly lost, a healthy majority survives, bounded by grace, backoff and an attempt cap; placement excludes live-member hosts. - Replacement prunes the superseded member from etcd once joined: k3s-first-boot deletes the stale Node (EKS_ETCD_PRUNE_PEER_IP) so quorum width returns to N. json tags added to K3sServerInput (now persisted). * feat(eks): in-guest control-plane self-diagnosis on healthz fail When a control-plane apiserver reports unhealthy, the reconciler only saw healthz=fail with no cause, and the guest has no on-demand command channel (serial root login is password-gated, no guest agent). Make the CP self-diagnose and relay the cause through the one-way state report. - mulga-eks-state-report.sh: diagnose() collects failing /readyz?verbose subchecks, isolated /healthz/etcd reachability, and etcd-db-dir free space; on fail write a verbose block to the host-captured serial console and add a JSON-safe reason to the state payload. - ServerStateReport: add Reason (omitempty; empty from a healthy CP or an older AMI predating the field). - cluster_reconciler observe(): surface the reason in the health issue. - Tests: POSIX unit test for the agent (stubbed kubectl/df); Go tests for the reason-bearing and terse health-issue paths. * feat(eks): 15-min etcd snapshot cadence alongside nightly Nightly alone is too coarse a recovery point for a young cluster. Install the same mulga-eks-etcd-snapshot script into a second Alpine crond periodic dir (/etc/periodic/15min) so it also runs on a ~15-minute RPO window; the script picks its cadence tier from the dir it is invoked from ($0 sniff) and encodes the tier in the snapshot key (etcd-{tier}-{ts}.snap), so a predastore S3 lifecycle rule can retain the frequent rolling window shallow and the nightly copies deep. Server-role only (agents exit early); no restore-path change. Add a self-contained POSIX unit test (stub k3s/curl/logger, symlinked crond dirs to exercise the real $0 tier sniff) covering agent no-op, frequent + daily key prefixes, local-snapshot prune, and quiet no-op without creds. * feat(eks): auto-reform wedged etcd quorum via boot-time cluster-reset An HA k3s control plane does not self-heal after a simultaneous full-CP restart: embedded etcd never reforms quorum (etcd:unreachable sustained), and in-place VM restarts cannot fix it. Add a last-resort escalation that drives the documented k3s cluster-reset recovery, pull-based so the host never needs an inbound channel into the guest. Reconciler: maybeReformEtcdQuorum escalates only when every CP member is VM-running but the health issue carries etcd:unreachable, the cluster is HA, and a grace window has elapsed. It sets per-member recovery directives (seed = cluster-reset, others = wipe-rejoin), epoch-bumped and keyed by instance ID, then restarts the members. Bounded by grace/backoff/attempts; a recovered cluster clears the directives once. Enabled via WithEtcdResetRecovery alongside the existing CP control wiring. Directive plane: per-member KV directive {epoch, action, snapshot} with store helpers, an EKS service Get/Set pair over NATS-RPC, and an internal-recovery gateway route (system-cred carve-out, same as internal-addons). The CP server instance role gains eks:GetRecoveryDirective. Guest: mulga-eks-k3s-recovery is a pre-k3s oneshot (server role only) that fetches its directive through the AWS gateway (eks-gateway-fetch -resource recovery), epoch-guards one-shot application, then runs `k3s server --cluster-reset` (optionally restoring a predastore snapshot) or wipes the local etcd datastore to rejoin the reset seed. Wired into the image manifest, setup.sh perms, and the eks-node-role selector. Tests: reconciler escalation gates + recovery clear, directive store epoch increment + service validation, and a POSIX unit test for the guest agent. * fix(eks): reboot running CP members to apply etcd cluster-reset directive The etcd quorum-reformation escalation drove StartInstance on the seed and followers, but the reset gate only fires when every member is VM-running. The StartInstances path rejects a running instance with IncorrectInstanceState, so the guest never re-entered its boot sequence and the cluster-reset/wipe-rejoin directive was never applied — the wedged control plane stayed wedged. Reboot the running members (QMP system_reset) instead, re-entering the boot sequence so the on-VM recovery agent applies the directive. Adds RebootInstance to the CP control surface (interface, adapter, NATS-backed daemon impl) and switches maybeReformEtcdQuorum to it; the stopped/error in-place restart path keeps StartInstance. * fix(eks): graceful stop for etcd reset recovery The etcd quorum-reformation reset path forced running CP members through a hard QMP system_reset to re-enter the boot sequence and apply their recovery directives. A hard reset skips a clean unmount: on the next boot e2fsck cleared the recovery and k3s runlevel entries (dangling inodes), so the cluster-reset never ran and k3s was unstartable — the reset defeated recovery. Replace the reboot with a graceful StopInstance (QMP system_powerdown, clean unmount, SIGKILL fallback). The existing in-place restart path (maybeRecoverControlPlane) then boots each stopped member clean on a later tick, and the on-VM recovery agent applies its directive against an intact filesystem. Reuses existing lifecycle machinery; the stop also tolerates a wedged QMP client via the SIGKILL fallback. Swaps RebootInstance -> StopInstance across CPInstanceControl, cpInstanceController, the cpControlAdapter, and the NATS-backed eksCPControl (gateway RebootInstances -> StopInstances), plus the reset/recovery test fakes. * fix(vm): retry QMP dial and unlink stale socket on relaunch A QEMU killed with SIGKILL (crash or etcd-reset restart) leaves its qmp-<id>.sock inode behind. On relaunch the startup os.Stat probe and the QMP dial both raced that dead listener, so a single dial returned 'connection refused': StartStoppedInstance then reported ServerInternal and left the instance KV-stopped even though the guest booted fully. The wedged member stayed invisible to the reconciler's cpLive gate, stalling recovery. Unlink the stale socket before launch so the inode that appears is the fresh QEMU's, and retry the QMP connect with a short backoff to cover the window where the socket is bound but not yet listening. Handshake failures are not retried — a listener answered, so the error surfaces immediately. * feat(eks-ui): surface cluster Health.Issues on the EKS console DescribeCluster is AWS-faithful (Status stays ACTIVE, degradation lives in Health.Issues), but the cluster detail page only rendered the status badge, so a degraded control plane still showed as ACTIVE with no visible signal. Add a destructive health badge next to the status and a Health row that lists the issue messages, mirroring the addons tab. Healthy clusters read OK. * chore: rebuild ui * fix(viperblockd): stop mount-path daemon VB uploader to prevent checkpoint clobber The daemon-side VB created in the ebs.mount handler is stored in MountedVolume.VB for the whole mount, and its background chunk uploader was only stopped at daemon shutdown. That state-only VB ran DrainToBackend every 30s, overwriting the live checkpoint the nbdkit plugin VB (the data-path writer) keeps updating, leaving SeqNums inconsistent with sealed chunk ciphertext and causing transient AEAD auth failures on cloned encrypted volumes. Stop its uploader right after New, matching the other short-lived VB call sites. Also bump viperblock to the atomic chunk-ObjectID allocation fix. * build(deps): pin viperblock to landed AEAD concurrency fix Bump github.com/mulgadc/viperblock to the dev commit b08bea6 that eliminates "cipher: message authentication failed" on encrypted volumes under concurrent read-modify-write + background chunk drain. Removes the local-path replace used while the fix was in flight.	2026-07-09 09:43:32 +10:00
Josh Siviour	7bbd99c792	feat(eks): multinode control-plane + IMDS/token-webhook CP fixes (426/427/428/495/496) (#459) ... * feat(eks): multinode control-plane networking, IAM, and CoreDNS parity Integrates the EKS multinode fix set for the managed-control-plane topology so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it. EKS: - Route apiserver egress (exec/logs/webhooks) through konnectivity (apiserver-network-proxy): CP servers run konnectivity-server behind the cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod traffic rides the agent-initiated reverse tunnel instead of an unroutable CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every replica (HA-correct). - Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay flannel mesh SG rule authorized at cluster launch with ordered teardown. - Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay. - Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity. - Spread nodegroup workers across hosts; addon readiness falls back on workload rollout when gvks are absent. - elbv2: allow the proxy_protocol_v2.enabled target-group attribute. EBS: - Move volume attachment state to state.json so a live nbdkit VB cannot clobber it; route mounted-volume state updates through the live VB with a retry for in-use marks racing the mount responder; keep the boot volume in-use across stop/crash unmount. * fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM missingOrchestrationDeps gated on s.deps.IAM, but production wires only the lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only). The gate therefore reported missing=[IAM] on every node forever, so CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even though the IAM service was fully up via the provider. Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else deps.IAMProvider()) the nodegroup/role code already uses — so the protective check still rejects a node with no IAM service but passes when the provider supplies one. Add a regression test that sets only IAMProvider and asserts the gate is satisfied; the existing nil-both test still rejects. * fix(eks): let eks-token-webhook use IMDS creds when static keys absent The CP VM launches on an instance profile, so buildK3sUserData omits EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys before eksgw.New could fall back to the IMDS SDK chain, so the webhook aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver crashed on the missing --authentication-token-webhook-config-file file, stalling the cluster at CREATING until it timed out to FAILED. Drop the static-key precondition (matching the sibling eks-gateway-publish and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/ cluster gates. * fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt, which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a 169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing 169.254.169.254 and steering IMDS off the data NIC away from the per-tap br-imds datapath, where the host RST it. eks-gateway-publish then failed to sign requests and no bootstrap artifacts were ever published, so clusters sat CREATING until they timed out to FAILED. Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI) are unaffected and IMDS falls through the data NIC's default route to the br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net deletes any default route on a NIC declared NIC<n>_DEFAULT=0. * fix(eks): grant eks:WebhookTokenReview to the control-plane instance role The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway broker at POST /clusters/{name}/token-review, signing with the control-plane VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes against eks:WebhookTokenReview, but the role's inline policy granted only eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every relay and the apiserver returned 401 to all external kubectl. Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile re-asserts the inline policy on every control-plane instance-profile ensure, and the gateway evaluates authorization per request, so existing clusters converge without a rebuild once the shared role is updated.	2026-07-09 09:43:30 +10:00
Julian Sommer	8a94886b13	feat: cloud-init EC2 datasource (#411)	2026-06-25 13:48:08 +10:00
Josh Siviour	9d662f9889	feat: EKS EBS CSI driver (Viperblock) — egress, attach, node discovery, e2e (#402) ... * feat: bake aws-ebs-csi-driver addon + mdev by-id bridge Add the aws-ebs-csi-driver entry (v1.40.1) to the eks addon catalog and ship its baked manifests (RBAC, IRSA-wired controller, node DaemonSet, default gp3 StorageClass) under the eks-node image. The node plugin locates EBS volumes via /dev/disk/by-id, which busybox mdev does not populate. Add mulga-ebs-byid: an mdev rule + helper that symlinks nvme-Amazon_Elastic_Block_Store_<serial> from the virtio-blk serial, plus a boot oneshot that points the kernel hotplug helper at mdev and coldplugs volumes present at boot. * test: e2e EBS CSI volume — PVC bind + mount + reschedule survival Add a TestEKS/EBSCSIVolume subtest that installs the aws-ebs-csi-driver managed addon bound to an IRSA role, provisions a default gp3 PVC + pod, and asserts a marker written by the first pod survives a reschedule onto a fresh pod backed by the same PVC. Exercises the full data path: CreateVolume(Viperblock)/AttachVolume virtio-blk hotplug with serial set to the volume-id, mdev /dev/disk/by-id symlink, and ext4 mount. * feat: eks-node provider-id + topology/instance-type labels for EBS CSI mulga-eks-provider-id boot oneshot writes a K3s config.yaml.d drop-in from IMDS before K3s starts: kubelet provider-id plus the region, zone and instance-type node labels. The aws-ebs-csi-driver node plugin reads this metadata for NodeGetInfo (instance id from providerID, region/zone from topology labels, instance type for the volume-limit family lookup). k3s runs no AWS cloud-provider so none of it is set otherwise, which left the node plugin's metadata init failing, NodeGetInfo panicking in IsNitroInstanceType, and the CSINode carrying no topology key. Wire the service into the image manifest (INSTALL_FILES + ENABLE_SERVICES) and chmod it in setup. * test: e2e EBS CSI controller IRSA policy + failure diagnostics awsgw enforces IAM on assumed-role EC2 calls, so the controller's CreateVolume/AttachVolume need a permission policy on the IRSA role. Attach a customer-managed EC2 policy (AWS-managed ARNs are opaque and grant nothing) covering the EBS CSI action set; CreateVolume then succeeds. Add an OnFailure dump of the CSI control path (provisioner and plugin logs, pvc/pod describe, events, csinode) to the EBSCSIVolume subtest so provisioning/attach failures are diagnosable post-teardown. * feat: EBS CSI obeys control-plane taint; e2e worker nodegroup Drop the CriticalAddonsOnly/NoExecute tolerations from the EBS CSI controller + node manifests so the driver runs only on customer worker nodes, never the system control-plane VM. The e2e creates a 1-node worker nodegroup and the writer/reader pods drop their CP tolerations so the customer-owned volume attaches in the customer space. * fix: EBS CSI attach — decouple hotplug PCIe port from device name; iothread rollback virtio-blk-pci hot-plug bus was derived from the AWS device-name letter (/dev/sd[f-p]); the EBS CSI driver sends arbitrary names (/dev/xvdaa) that map to no slot, so device_add fell back to pcie.0 — which rejects hot-plug — and the attach hung. A failed attach also left the per-volume iothread object behind, so every retry died on object-add 'duplicate property'. nextHotplugEBSPort allocates the lowest free hotplug-ebs port from live query-block state, independent of the device name. delIothreadBestEffort removes the iothread on every attach rollback path. * fix: EBS CSI node discovery — un-shadow mdev by-id rule, mint nvme link Alpine eks-node runs busybox mdev (no eudev), which stops at the first matching /etc/mdev.conf rule. setup.sh appended the mulga-ebs-byid rule after the stock 'vd[a-z].* -> persistent-storage' line, so it never fired on hot-attach: only /dev/disk/by-id/virtio-<serial> was created, while the upstream aws-ebs-csi-driver resolves nvme-Amazon_Elastic_Block_Store_<volid>. Swap the stock vd* rule in place to route through mulga-ebs-byid, which now delegates to persistent-storage (preserving stock links) before minting the nvme-style link the node plugin expects. * test: EKS e2e — NAT Gateway egress for private worker nodegroup The customer-VPC worker is private; AttachIGW programs no SNAT (mirrors AWS: an IGW serves only public-IP instances), so an IGW-only fixture left the worker unable to reach the public NLB endpoint or pull public CSI images — the nodegroup never joined. createWorkerEgress now builds the real topology: an IGW-fronted public subnet hosting a NAT Gateway, with the worker subnet default-routed to it. Associating the worker route table triggers the subnet-wide SNAT egress reroute. Teardown drains the NAT Gateway before releasing the public subnet + EIP it pins.	2026-06-19 14:43:20 +10:00
Josh Siviour	bc9556d1a9	fix(eks): managed-CP worker data-plane — apiserver reachability, node-local CoreDNS, SG teardown (#368) ... * fix: persist instance TagSpecifications in RunInstance RunInstance built the ec2.Instance metadata but never read input.TagSpecifications, so DescribeInstances returned untagged instances and tag filters matched nothing. EKS node groups discover their workers by the spinifex:eks-cluster / spinifex:eks-nodegroup tags, so scale-up convergence and worker reconciliation depended on tags that were silently dropped. Apply instance-scoped TagSpecifications onto the built instance so the tags surface through DescribeInstances and tag filters. * fix: serialize nodegroup reconcile via CAS to prevent scale overshoot UpdateNodegroupConfig is dispatched on a queue group, so two overlapping requests (e.g. an SDK retry mid-flight) each read current = len(InstanceIDs) and each launched desired-current workers, overshooting the target (1 -> 5 for desired=3) and stranding untracked VMs via last-writer-wins on the unconditional PutNodegroupRecord. Make every durable nodegroup-record mutation a compare-and-swap on the KV revision. Scale-up launches one worker at a time and CAS-appends it only while the record is below the live ScalingDesired, terminating any surplus VM launched into an already-full record. Scale-down terminates before the CAS truncation so a crash or retry leaves reclaimable dead IDs rather than orphaned VMs. A concurrent regression test asserts two overlapping scale-ups converge to exactly desired under -race. * fix: return securityGroupRuleSet from AuthorizeSecurityGroup ingress/egress The AWS provider's aws_vpc_security_group_ingress_rule reads SecurityGroupRules[0].SecurityGroupRuleId on create; an empty ruleset panics the plugin (Plugin did not respond), though the rule applies server-side. Echo the created rules via sgRuleToSecurityGroupRule, the same shape DescribeSecurityGroupRules already returns. * fix: surface InsufficientInstanceCapacity from nodegroup scale, not 500 launchWorkersCAS wrapped the bare awserror code via fmt.Errorf, so the gateway ErrorHandler's exact-match against ErrorLookup missed and remapped to InternalError (500). An out-of-capacity scale now surfaces the real InsufficientInstanceCapacity code (400); opaque internal failures still wrap and stay 500. Adds awserrors.HasErrorCode for the membership test. * fix: cut EKS node boot ~50s — drop cloud-final deadlock + syslinux menu Worker/server user-data runcmd ended with `[ rc-service, local, start ]`, which runs inside cloud-final while the OpenRC `local` service is ordered after it — a deadlock broken only by OpenRC's ~50s start timeout, stalling the k3s join. Run the IMDS route script directly instead and keep `rc-update add local default` for reboot persistence. setup.sh: cut the SYSLINUX boot-menu countdown 10s->~1s (kept nonzero so the menu stays interruptible over serial). Validated live on env19 CP: boot menu 1s, no `waiting for cloud-final` stall, apiserver ready ~66s (was ~109s+). mulga-siv-332 * feat: add argocd managed addon + bake bundle into eks-node AMI The addon catalog advertised coredns and aws-load-balancer-controller, but the only baked manifest shipped in the eks-node AMI was the spinifex-noop fixture. Installing any real addon failed with 'no baked bundle for version' once the VM-side renderer found nothing on disk. Add argocd v3.0.23 as a real, demonstrable managed addon: catalog entry (no IRSA), a flattened namespaced install manifest baked into the AMI, and the manifest.conf bake line. The eks-addons-access workbook switches from coredns (redundant on K3s, never baked) to argocd. The baked bundle is generated from upstream install.yaml via kubectl kustomize with namespace: argocd plus a prepended Namespace, since upstream is namespace-agnostic and K3s auto-deploy applies with no -n flag. * fix(eks): run CoreDNS as a node-local DaemonSet for worker DNS k3s ships CoreDNS as a single-replica Deployment that tolerates the CP taint and strands itself on the control-plane node. In the managed-CP topology the worker<->CP pod overlay cannot cross the VPC boundary, so worker pods cannot reach a CP-resident CoreDNS and lose all cluster + external DNS. Skip the bundled CoreDNS (coredns.yaml.skip, staged in k3s.initd like traefik) and deploy a Mulga-owned variant: a DaemonSet (one pod per node) plus kube-dns internalTrafficPolicy=Local, so every node resolves via its own CoreDNS. Immune to the cross-VPC overlay gap and topology-agnostic (single-node and managed-CP both work). Image-side only; no handler change. * fix(eks): revoke cross-referencing SG rules before delete on teardown DeleteClusterSGs deleted the control-plane and nodegroup SGs directly, but they cross-reference each other (EnsureNodegroupSGRules), and AWS refuses to delete a security group still referenced by another SG's rule. Both deletes failed with DependencyViolation, the SGs leaked, and the VPC could not be deleted (tofu destroy hung on DeleteVpc). Resolve both SG IDs, revoke all ingress on each to break the cp<->ng cycle, then delete both. Adds RevokeSecurityGroupIngress to sgProvisioner (already on VPCServiceImpl) and revokeAllSGIngress, with a regression test that models AWS's DependencyViolation. * fix(eks): advertise NLB front-end so worker pods reach the apiserver ClusterIP In the managed-control-plane topology the apiserver advertised its Set-B node-ip into the in-cluster `kubernetes` Endpoints. That IP lives in the unpeered CP VPC and is unreachable from worker pods, so every worker pod using the apiserver ClusterIP (CoreDNS kubernetes-plugin sync, controllers, argocd redis secret-init) failed. - k3s_server_vm.go: set advertise-address to the worker-reachable NLB front-end (PrivateEndpointIP, else EndpointIP); both are already cert SANs. - nlb.go: add a second :6443 listener on the cluster NLB (same CP target group) so PrivateEndpointIP:6443 — the advertised endpoint — is served. - security_groups.go: admit :6443 on the private-endpoint SG. Validated on a live managed-CP cluster: worker CoreDNS /ready 200, all argocd pods Running, kubectl logs on worker pods working.	2026-06-16 20:27:21 +10:00
Josh Siviour	b23fa3bf60	feat(eks): managed control-plane VPC (Set B) + control-plane lifecycle hardening (#356) ... * feat(eks): VM-side managed-addon delivery transport Stage CreateAddon manifests in KV host-side; the control-plane VM pulls the staged set through the AWS gateway (signed GET /clusters/{name}/internal-addons/ {accountId}), renders the baked bundles into the K3s auto-deploy dir, and reports per-addon delivery status on a dedicated eks.addon.*.status channel that the cluster reconciler CASes onto AWS-visible AddonStatus (CREATING->ACTIVE). - gateway: ListInternalAddons route + ListStagedAddonManifests NATS action - handlers: addon status reconciler (nextAddonStatus), staging installer - eks-node AMI: eks-gateway-fetch binary + mulga-eks-addon-sync agent - agent renders idempotently (cmp before replace) so K3s auto-deploy does not re-apply every tick and clear the Addon CR .status.gvks readiness signal - agent mirrors per-tick delivery outcomes to the serial console for host-side observability * fix(eks): provision cluster NLB synchronously so the endpoint carries the real front-end IP The cluster NLB front-end IP is assigned by the async LB-VM launch (launchLBVMAsync), but EnsureClusterNLB read it back synchronously right after CreateLoadBalancer returned — losing the race and falling back to the synthetic <lb>.elb.spinifex.local DNS name. That name is unresolvable (no Route53) and is not in the apiserver serving-cert SAN, so kubectl against the cluster never connects. Split the elbv2 create: the public CreateLoadBalancer stays async (267.4 — an inline multi-minute VM boot head-of-line-blocks the gateway responder), and a new CreateLoadBalancerSync drives the data-plane launch inline and returns the LB carrying its front-end address. The EKS cluster-create worker runs off the gateway responder, so it uses the sync entry and reads the IP from the return value instead of racing persisted state. No polling. - elbv2: factor provisionLBDataPlane out of launchLBVMAsync; add CreateLoadBalancerSync - eks: ensureClusterLB uses the sync entry, sets FrontendIP from the result, drops the read-back; EnsureClusterNLB fails loud (ErrClusterNLBFrontendIPUnavailable) when no front-end IP is allocated rather than shipping the DNS-name fallback - eks: drop the DNS-name endpoint fallback; the endpoint is always the front-end IP * feat: auto-wire IGW for internet-facing EKS cluster endpoint An internet-facing cluster NLB allocates an external front-end IP and the OVN dnat_and_snat for it, but the external switch + localnet + gateway LRP that make that IP answerable on the wire are built only on IGW attach. The control-plane NLB is a system instance, so ensure an IGW on cluster build and reclaim it on teardown. EnsureClusterIGW (idempotent, reuses a customer IGW untouched) runs before EnsureClusterNLB when the endpoint is public; DeleteClusterIGW (ownership- scoped) removes only the cluster-owned IGW in purgeClusterInfra, before the VPC delete. New IGW dep wired from the daemon. * fix(eks): tear down infra on failed cluster launch to stop LB EIP leak A CreateCluster that fails during its background launch flipped the meta to FAILED but performed no teardown, relying on a later same-name create to reclaim and purge the infra. EKS cluster names are unique per create, so that reclaim never fires: the internet-facing load-balancer VM keeps running with its allocated+associated dnat_and_snat EIP indefinitely. Over repeated create/delete cycles the EIP pool fills with allocations bound to terminated instances, so a later allocation's advertised endpoint and its live OVN NAT diverge onto different pool addresses. - failClusterLaunch now runs a best-effort purgeClusterInfra on the in-memory meta (freshest refs even when the failing step was the meta write itself) before MarkClusterFailed. purgeClusterInfra is idempotent, so a later reclaim or DeleteCluster re-purge is a no-op; the FAILED meta is retained for observability. - Daemon releaseSystemInstanceEIP extracted and called from both TerminateSystemInstance and cleanupFailedSystemInstance, so a launch that fails after AssociateAddress releases the eipService allocation instead of leaking it (MarkFailed's ReleasePublicIP only knows the externalIPAM path). * feat: EKS managed control-plane VPC (AWS Set B) composed from EC2 APIs CreateCluster now auto-builds the managed control-plane VPC under the system account, the spinifex analogue of AWS EKS's hidden managed-account VPC: a public subnet for the internet-facing NLB, private subnet(s) for the k3s control-plane VM(s), and a NAT gateway giving the private CP egress for image pulls. It is composed from the real EC2 VPC-family APIs (CreateVpc/Subnet, IGW, RouteTable/Route, NatGateway), so per-subnet egress gating is wired by the existing topology subscribers off the route-table events those APIs publish — no direct OVN mutation. This resolves the mulga-siv-292 root cause: the internet-facing NLB and the private CP were crammed into one customer subnet with no IGW route, so both were treated as private and dropped by the lr_in_policy 1100 rule, killing the LB-VM return leg and lb-agent heartbeat. The CP only survived via a per-instance 1200 reroute bandaid, now removed — the topology expresses egress instead. Placement is uniform single/multi-node (CP always in the private subnet). The system LB VM heartbeats over the mgmt bridge (SystemGatewayURL) rather than the WAN. Managed-VPC teardown reclaims NAT GW + EIP, route tables, subnets, IGW, and the VPC in dependency order. Customer subnetIds (Set A / worker side) remain consumed-not-created and are deferred to the worker round. * fix(eks): bake cluster-owner account into CP-VM EKS_ACCOUNT_ID The managed control-plane VPC (Set B) launches the K3s control-plane VMs and their ENIs under the system (infra) account, but the on-VM agents — bootstrap publish, state report, token-review, and add-on fetch — must key on the cluster-OWNER (customer) account that owns the cluster meta, namespaces, and AccessEntry KV. Add ClusterAccountID to K3sServerInput and bake it into the VM's EKS_ACCOUNT_ID, decoupling it from the infra AccountID the VM launches under. Without this the VM reported state and fetched add-ons against the system account, so the cluster never observed a healthy control plane (stuck CREATING, null CA) and kubectl/add-on delivery timed out. Also refresh the e2e suite doc comment to describe the Set B topology and the data-plane subtests (kubectl + managed-addon delivery) that now run. * fix(eks): explicitly delete addon objects on unstage K3s does not delete the applied resources when an auto-deploy manifest file is removed from the server manifests dir (documented behaviour). The addon-sync GC pass relied on that, so after DeleteAddon the rendered manifest was dropped but its Namespace/ConfigMap stayed standing — the e2e AddonDelivery GC wait timed out. Copy the manifest aside, drop the original so K3s stops tracking it, then kubectl delete the objects (--wait=false to avoid finalizer stalls).	2026-06-12 22:29:24 +10:00
Josh Siviour	496424c41c	test(eks): EKS control-plane E2E suite (Sprint 6g) (#304) ... * test(eks): EKS control-plane e2e suite (Phase 1, API-level) Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw: CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone. Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted} pollers. Wire e2e-eks into the Makefile and the single-node CI suite list. * test(eks): Phase 2 kubectl get-nodes data-plane gate Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper (harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the control-plane node is Ready — the get-token webhook auth regression gate. IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up. * test(eks): tolerate doubled NotFound code + dial loopback NATS DeleteCluster detection matched the SDK constant ResourceNotFoundException, but awsgw emits ResourceNotFoundExceptionException on the wire — match the NotFound code by substring so deletion is detected immediately instead of timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which the serving cert does not SAN; rewrite the wildcard bind to loopback. Validated live: all 6 subtests green, kubectl get nodes shows the control-plane node Ready via the get-token webhook auth path. * feat: build system images with libguestfs (no host block devices) Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in build-system-image.sh with libguestfs (virt-customize / virt-resize / guestfish). Every operation runs in an isolated userspace appliance VM with its own kernel, so the build touches no host block device, no host mount, and needs no sudo — removing the failure class that wiped the host's /dev on shared CI runners (a leaked /dev bind traversed by rm -rf). Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu via virt-resize; customization (packages, binaries, files, setup script, service enable, cache cleanup) is one ordered virt-customize invocation. LIBGUESTFS_BACKEND=direct (no libvirt on runners). Wire the eks-node AMI into the single-node e2e job: the runner builds the device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are VM-only) so the eks suite's CreateCluster has an AMI to boot. * feat(eks): bootstrap + state-report via AWS gateway broker Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model: the K3s server VM publishes its bootstrap envelopes and periodic state reports via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS directly. A new gateway action (PublishInternal, POST /clusters/{name}/internal-publish) relays the payload verbatim onto the existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV persistence/encryption, and reconciler are unchanged. This removes the VM's need to reach NATS at all, so NATS can stay cluster-internal behind its bind/firewall/mTLS controls — the prior direct path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was DDIL-fragile (fire-and-forget nats pub, no ack/retry). - gateway/eks: PublishInternal broker + route; closed-set channel/kind validation - cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and mulga-eks-state-report.sh - handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL + system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA; new resolveSystemGatewayBaseURL (mgmt-bridge-first) - eks-node image: build/install eks-gateway-publish, drop nats-cli download * feat: migrate eks-token-webhook to the AWS gateway broker The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that fails, so the webhook exited at startup, the apiserver token-webhook kubeconfig was never written, and the apiserver crash-looped — the cluster never reached ACTIVE. Converge it onto the same gateway-broker model as bootstrap/state-report and ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to /clusters/{name}/token-review, and the gateway runs the verify + lookup host-side and returns the resolved identity. The VM now speaks no core NATS. - internal/eksgw: shared on-VM SigV4 client (also adopted by eks-gateway-publish). - handlers/eks/token_review.go: Authenticate decision core (moved from the webhook binary) + ResolveTokenReview host-side wiring. - gateway/eks token-review route + WebhookTokenReview handler. - webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable) not a silent deny. - initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*. * fix: build eks-node helpers with GOFIPS140; cover token-review broker eks-gateway-publish blank-imports internal/fipsboot, whose init() panics unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build omitted the flag, so the binary paniced at first boot right after k3s came up — k3s-first-boot (set -eu) died before publishing the bootstrap token and the cluster never reached ACTIVE. Build both eks-node helpers with GOFIPS140=v1.0.0, matching cmd/lb-agent. Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview gateway handler (verify responder + account KV) to cover the new broker path. * fix: export first-boot env so eks-gateway-publish child sees it k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/ first-boot.env but never exported the vars, so the eks-gateway-publish child process inherited an empty environment and exited '--account-id is required'. The control-plane VM came up healthy but never published its bootstrap envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a / set +a so the KEY=value lines export to child processes. * fix: make awsgw start --host override actually bind The 'awsgw start --host' override printed 'Overwriting awsgw host' but the assignment was commented out, so AWSGW always bound the spinifex.toml host (a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW listens on all interfaces incl. br-mgmt; the serving cert already SANs every local interface IP, and WAN clients still reach it. * test(eks): install kubectl on e2e VM so KubectlGetNodes runs The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker -> STS verify path — the only end-to-end exercise of the token-review broker. The e2e binaries run on the VM, where kubectl was absent, so harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a checksum-verified kubectl (pinned to the k3s server version) on the VM, gated on e2e-eks.	2026-06-07 20:42:25 +10:00
Josh Siviour	a22bc2f716	feat(eks): 6d nodegroup provisioning + NATS-gated cluster health (#300) ... * feat(eks): nodegroup provisioning + K3s agent join (6d) CreateNodegroup launches N customer-owned worker EC2 from the unified eks-node AMI in agent role; each joins the cluster via K3S_URL + K3S_TOKEN baked into cloud-init (first-boot.env SPINIFEX_K3S_ROLE=agent + agent.env). Workers are customer-owned and DescribeInstances-visible (no ManagedBy, no mgmt NIC) via the normal RunInstances path, matching real EKS. K3S_URL targets the control-plane ENI's in-VPC private IP on :6443 — the cluster NLB DNS endpoint is not resolvable in-VPC yet; switch to the NLB endpoint on :443 once authoritative in-VPC DNS lands. K3S_TOKEN is the decrypted cluster node-token persisted at server bootstrap. Describe/List/Update(scaling)/Delete round out CRUD. Cluster<->nodegroup SG ingress rules (apiserver 6443, flannel VXLAN 8472/udp, kubelet 10250, intra-nodegroup) authorized idempotently. UpdateNodegroupVersion deferred. * feat: gate EKS cluster health via NATS state reports The K3s apiserver binds the VPC node-ip, unreachable from the host-side spinifex daemon, so the reconciler cannot probe /healthz directly. Add a control-plane state reporter that publishes {healthz, node_count} over the management NATS bus the daemon already shares. - eks-node server role runs mulga-eks-state-report as an OpenRC service, publishing eks.state.{account}.{cluster}.server every 30s. - ClusterReconciler subscribes to that subject, gates CREATING -> ACTIVE on a healthy report, and records Ready node_count into ClusterMeta. - node_count counts Ready nodes only; k3s retains terminated workers as NotReady until pruned, so a raw count never drops on scale-down.	2026-06-06 00:52:18 +10:00
Josh Siviour	f9c08130ba	fix(eks): embedded etcd datastore + Alpine IMDS-route fix + CP-VM visibility (#299) ... * fix: switch EKS datastore to embedded etcd + fix Alpine cloud-init IMDS route Flip the K3s control-plane datastore from SQLite/kine back to embedded etcd (cluster-init: true, etcd-expose-metrics: true). The earlier abandonment premise — that etcd's per-commit fsync stalls past the apiserver's 5s deadline on the viperblock-backed root volume — was wrong: etcd inits fine on an unchanged viperblock (apiserver ready 25-30s, clusters reach ACTIVE). The real blocker was a network-config regression, fixed here. Alpine's cloud-init eni renderer crashes (KeyError: 'gateway') on the gateway-less IMDS on-link route, which failed the whole network-config render so the static mgmt NIC never came up (no NATS bootstrap, no /healthz). Fixes: - generateNetworkConfig: add emitIMDSRoute bool; selectNetworkConfigForFamily passes false for Alpine so the eni renderer never sees the route. - Deliver the IMDS route out-of-band via a persistent /etc/local.d script + OpenRC local service. Resolve the VPC NIC via its default route at runtime (ip route show default), not a hardcoded name: the persistent-net rename to vpc0 doesn't apply on the Alpine AMI, so the NIC is eth0 at first boot. - EKS path bakes the same route into buildK3sUserData's own write_files/runcmd; the generic template Alpine block fires only when no #cloud-config user-data, avoiding a duplicate write_files key (yaml.safe_load keeps last, silently dropping a block). - eks-node image: config.yaml.skel adds cluster-init; comments updated. * fix: surface CreateCluster step failures at ERROR + map subnet faults; add auth-mode/delete test cases CreateCluster wrapped its step failures in bare fmt.Errorf that the request surface collapses to an opaque ServerInternal ("contact AWS re:Post"), so the real cause never reached the operator. Add logCreateErr to log each step's precise cause at ERROR before returning, and map a subnet-resolve failure (caller-supplied parameter) to InvalidParameterValue instead of a retryable ServerInternal the SDK pointlessly retries. (mulga-siv-165.4) Tests (mulga-siv-165.15): reject authenticationMode=API_AND_CONFIG_MAP (only the CONFIG_MAP value was covered); DeleteCluster on a nonexistent cluster via the full-deps path returns ResourceNotFound (the shim path short-circuits to ServiceUnavailable before the meta lookup). * fix: hide EKS control-plane VMs from customer DescribeInstances EKS K3s control-plane VMs are owned by the customer account (their ENI lives in the customer VPC for the NLB target), so unlike LB microVMs (owned by the system account) they leaked into customer ec2:DescribeInstances. The spinifex:managed-by tag only filtered the UI Nodes page, not the EC2 API. Add a caller-aware exclusion: skip any ManagedBy-tagged VM unless the caller is root. Preserves operator visibility, matches LB behaviour, and leaves future customer-owned worker nodes (no ManagedBy) visible. * docs: correct stale SQLite datastore comments in eks-node image (now embedded etcd)	2026-06-05 17:55:31 +10:00
Josh Siviour	f14fb64562	feat(eks): unify eks-server + eks-agent into one eks-node AMI (#288) ... * feat(eks): unify eks-server + eks-agent into one eks-node AMI Collapse the two K3s AMIs into a single spinifex-eks-node image whose role (server vs agent) is chosen per-instance at first boot, halving build time, storage, and CVE-rotation surface. - scripts/images/eks-node/: superset manifest + merged setup.sh; all role services baked but only crond + the new eks-node-role selector enabled. - eks-node-role first-boot selector reads SPINIFEX_K3S_ROLE, rc-update adds + starts the matching services (server: k3s + eks-token-webhook + k3s-first-boot publisher; agent: k3s-agent), records /etc/spinifex-eks/role, self-disables. Paths are env-overridable; eks-node-role_test.sh covers all five cases. - mulga-eks-etcd-snapshot.sh early-exits on agent nodes. - k3s_server_vm.go seeds SPINIFEX_K3S_ROLE=server into first-boot.env. - Makefile/dev-install: build-eks-node-image / import-eks-node-image replace the server+agent targets. - Delete scripts/images/eks-server + eks-agent. The launcher resolves the AMI by the spinifex:managed-by=eks tag, so running clusters (which hold the resolved AMI ID) are unaffected. AMI bake + cluster boot are operational integration steps, not yet run. * fix(eks): order eks-node-role after cloud-config/cloud-final The first-boot role selector reads /etc/spinifex-eks/first-boot.env, which cloud-init delivers via write_files in the cloud-config/cloud-final stages. The selector only depended on net+cgroups, so on a fast boot it could start before write_files landed and read a missing role env. Add an after-ordering on cloud-config + cloud-final so the env file is always present first.	2026-06-04 12:28:56 +10:00