mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Josh Siviour
9f349e86ca
* feat(eks): expose NVIDIA GPUs to pods via CDI device plugin GPU nodegroups now label workers nvidia.com/gpu.present=true and taint them nvidia.com/gpu=present:NoSchedule (via a k3s config.yaml.d drop-in, since k3s has no K3S_NODE_TAINT env), and stage a hidden nvidia-device-plugin addon that addon-sync renders into the K3s auto-deploy dir. The device plugin runs with --device-list-strategy=cdi-cri and is gated to GPU nodes by nodeSelector + toleration, advertising nvidia.com/gpu allocatable so pods land on GPU nodes and get the device injected via the CDI spec baked by mulga-cdi-generate. Relies on containerd 2.0.5 default enable_cdi=true (no config fork). * fix(vm): scale QMP greeting deadline for VFIO passthrough guests A GPU passthrough guest (-device vfio-pci + large -m) must lock and DMA-map its entire guest RAM through the IOMMU before QEMU's QMP monitor answers, which exceeds the fixed 30s greeting deadline and gets the VM SIGKILLed at launch. Parameterize the greeting timeout (NewQMPClientWithGreetingTimeout) and scale it to 180s for VMs with GPU attachments; plain VMs keep the 30s default and reconnects against an already-running QEMU keep it too. * fix(eks): register GPU workers reliably and expose GPUs to pods Three fixes let a GPU nodegroup worker boot, register, and advertise its GPU: - k3s-agent.service gains Delegate=yes (plus KillMode/TasksMax/limits and overlay/br_netfilter modprobe): k3s embeds containerd+kubelet in-process, so on cgroup-v2 the kubelet cannot create /kubepods without a delegated subtree and ContainerManager.Start fails before Node registration. - The GPU node label rides a k3s config drop-in (20-gpu.yaml node-label + node-taint) instead of K3S_NODE_LABEL, which k3s ignores; without the label the nvidia-device-plugin nodeSelector never matches. - Raise the nodegroup ready timeout to 20m for the ~7min GPU first boot. Adds the GPU e2e (VPC + NAT-GW egress harness) and a slog-restore unit test. * fix(eks): install ecr-credential-provider on the GPU node AMI The GPU worker's kubelet is started (from nodegroup userdata) with --image-credential-provider-config + --image-credential-provider-bin-dir= /usr/local/bin, but the eks-node-gpu AMI never built or installed the ecr-credential-provider binary. kubelet treats a referenced-but-missing provider binary as a fatal RegisterCredentialProviderPlugins error, so the node crash-loops and never registers (while the Alpine eks-node worker, which ships the binary, joins fine). Build + install it in the GPU manifest; the server-only binaries stay omitted since GPU nodes are agent-only. Also tee k3s-agent stderr (and its embedded kubelet klog) to the serial console: Ubuntu logs only to journald, which is off-box-invisible on a worker in an isolated customer VPC, so a kubelet that fails to register left no trace. * fix(eks): give the nvidia-device-plugin its own GPU access The CDI-mode device plugin (--device-list-strategy=cdi-cri) must run nvml.Init and generate the CDI spec in its own container, but the DS had no GPU wiring, so nvml.Init crash-looped and nvidia.com/gpu was never advertised. Run the DS under the k3s-auto-configured nvidia runtimeClass with NVIDIA_VISIBLE_DEVICES/CAPABILITIES=all, the host root mounted read-only at /driver-root so it can locate the driver libs + ld.so.cache, and /var/run/cdi mounted writable so the generated spec is visible to containerd. No enable_cdi drop-in is needed on k3s v1.32.5 / containerd 2.0.5. GPU workload pods still need only resources.limits nvidia.com/gpu. Live-validated end-to-end on the single-GPU host (RTX A1000): baked control-plane AMI, GPU nodegroup, nvidia.com/gpu allocatable, nvidia-smi pod, zero manual intervention. * test(ec2): drop obsolete restoreSlogDefault test Rebased onto dev, which dropped the restoreSlogDefault workaround in #506 (viperblock no longer mutates the global slog). The test referenced the now-deleted helper and no longer compiles. Real GPU/registration diff is independently covered (diff coverage 95.3%).
53 lines
3.3 KiB
Plaintext
53 lines
3.3 KiB
Plaintext
IMAGE_NAME="eks-node-gpu"
|
|
IMAGE_DESCRIPTION="Ubuntu EKS K3s GPU worker node — NVIDIA server driver (DKMS pre-built) + nvidia-container-toolkit; agent-only (GPU nodes never run the EKS control plane)"
|
|
DISTRO="ubuntu"
|
|
UBUNTU_VERSION="26.04"
|
|
UBUNTU_CODENAME="resolute"
|
|
|
|
# 16G — headroom for kernel headers, NVIDIA driver + DKMS build artefacts, and
|
|
# nvidia-container-toolkit, mirroring scripts/images/ubuntu-gpu-nvidia.conf.
|
|
IMAGE_SIZE="16G"
|
|
|
|
# K3s bundles its own containerd, so no containerd/runc package here. The
|
|
# NVIDIA server driver + nvidia-utils come from the stock Ubuntu repos and are
|
|
# installed in setup.sh (kernel-version-pinned DKMS build, like the ubuntu-gpu
|
|
# recipe). nvidia-container-toolkit lives in its own NVIDIA-hosted apt repo,
|
|
# which is not enabled on the base cloud image, so it is also added+installed
|
|
# in setup.sh rather than here (see scripts/images/ubuntu-gpu-nvidia-setup.sh).
|
|
APT_PACKAGES="ca-certificates curl jq iptables conntrack cgroup-tools initramfs-tools"
|
|
|
|
# ecr-credential-provider is the ONLY server binary a GPU worker needs: the
|
|
# nodegroup userdata points kubelet at it (image-credential-provider-config +
|
|
# bin-dir=/usr/local/bin) for ECR pulls, and kubelet treats a referenced-but-
|
|
# missing provider binary as a FATAL init error (RegisterCredentialProviderPlugins),
|
|
# so the node never registers. The Alpine eks-node ships it; the GPU AMI must too.
|
|
# Server-only binaries (token-webhook, gateway-*, konnectivity, k3s server) stay
|
|
# omitted — GPU nodes are agent-only.
|
|
BUILD_COMMANDS="CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -o bin/ecr-credential-provider ./cmd/ecr-credential-provider"
|
|
INSTALL_BINARIES="bin/ecr-credential-provider:/usr/local/bin/ecr-credential-provider"
|
|
|
|
# Role-agnostic helper scripts are reused verbatim from scripts/images/eks-node/
|
|
# (only the init wiring changes: OpenRC -> systemd). k3s-agent-prestart is new —
|
|
# it reproduces the k3s-agent.initd start_pre logic (env require + IMDS
|
|
# provider-id) for the systemd unit. Server-only units (role selector, k3s
|
|
# server, eks-token-webhook, konnectivity, state-report, k3s recovery,
|
|
# mgmt-net) are intentionally not ported — GPU nodes are single-NIC workers.
|
|
INSTALL_FILES="\
|
|
scripts/images/eks-node/mulga-ebs-byid.sh:/usr/local/sbin/mulga-ebs-byid \
|
|
scripts/images/eks-node/mulga-eks-provider-id.sh:/usr/local/sbin/mulga-eks-provider-id \
|
|
scripts/images/eks-node/mulga-vpc-mtu.sh:/usr/local/sbin/mulga-vpc-mtu \
|
|
scripts/images/eks-node-gpu/k3s-agent-prestart.sh:/usr/local/sbin/k3s-agent-prestart \
|
|
scripts/images/eks-node-gpu/k3s-agent.service:/etc/systemd/system/k3s-agent.service \
|
|
scripts/images/eks-node-gpu/mulga-ebs-byid.service:/etc/systemd/system/mulga-ebs-byid.service \
|
|
scripts/images/eks-node-gpu/mulga-eks-provider-id.service:/etc/systemd/system/mulga-eks-provider-id.service \
|
|
scripts/images/eks-node-gpu/mulga-vpc-mtu.service:/etc/systemd/system/mulga-vpc-mtu.service \
|
|
scripts/images/eks-node-gpu/mulga-cdi-generate.service:/etc/systemd/system/mulga-cdi-generate.service \
|
|
scripts/images/eks-node-gpu/99-mulga-ebs-byid.rules:/etc/udev/rules.d/99-mulga-ebs-byid.rules"
|
|
|
|
ENABLE_SERVICES="k3s-agent mulga-ebs-byid mulga-eks-provider-id mulga-vpc-mtu mulga-cdi-generate"
|
|
|
|
SETUP_SCRIPT="scripts/images/eks-node-gpu/setup.sh"
|
|
|
|
AMI_NAME="spinifex-eks-node-gpu"
|
|
SYSTEM_TAG="spinifex:managed-by=eks gpu-vendor=nvidia"
|