mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 19:01:42 +00:00
Josh Siviour
9f349e86ca
* feat(eks): expose NVIDIA GPUs to pods via CDI device plugin GPU nodegroups now label workers nvidia.com/gpu.present=true and taint them nvidia.com/gpu=present:NoSchedule (via a k3s config.yaml.d drop-in, since k3s has no K3S_NODE_TAINT env), and stage a hidden nvidia-device-plugin addon that addon-sync renders into the K3s auto-deploy dir. The device plugin runs with --device-list-strategy=cdi-cri and is gated to GPU nodes by nodeSelector + toleration, advertising nvidia.com/gpu allocatable so pods land on GPU nodes and get the device injected via the CDI spec baked by mulga-cdi-generate. Relies on containerd 2.0.5 default enable_cdi=true (no config fork). * fix(vm): scale QMP greeting deadline for VFIO passthrough guests A GPU passthrough guest (-device vfio-pci + large -m) must lock and DMA-map its entire guest RAM through the IOMMU before QEMU's QMP monitor answers, which exceeds the fixed 30s greeting deadline and gets the VM SIGKILLed at launch. Parameterize the greeting timeout (NewQMPClientWithGreetingTimeout) and scale it to 180s for VMs with GPU attachments; plain VMs keep the 30s default and reconnects against an already-running QEMU keep it too. * fix(eks): register GPU workers reliably and expose GPUs to pods Three fixes let a GPU nodegroup worker boot, register, and advertise its GPU: - k3s-agent.service gains Delegate=yes (plus KillMode/TasksMax/limits and overlay/br_netfilter modprobe): k3s embeds containerd+kubelet in-process, so on cgroup-v2 the kubelet cannot create /kubepods without a delegated subtree and ContainerManager.Start fails before Node registration. - The GPU node label rides a k3s config drop-in (20-gpu.yaml node-label + node-taint) instead of K3S_NODE_LABEL, which k3s ignores; without the label the nvidia-device-plugin nodeSelector never matches. - Raise the nodegroup ready timeout to 20m for the ~7min GPU first boot. Adds the GPU e2e (VPC + NAT-GW egress harness) and a slog-restore unit test. * fix(eks): install ecr-credential-provider on the GPU node AMI The GPU worker's kubelet is started (from nodegroup userdata) with --image-credential-provider-config + --image-credential-provider-bin-dir= /usr/local/bin, but the eks-node-gpu AMI never built or installed the ecr-credential-provider binary. kubelet treats a referenced-but-missing provider binary as a fatal RegisterCredentialProviderPlugins error, so the node crash-loops and never registers (while the Alpine eks-node worker, which ships the binary, joins fine). Build + install it in the GPU manifest; the server-only binaries stay omitted since GPU nodes are agent-only. Also tee k3s-agent stderr (and its embedded kubelet klog) to the serial console: Ubuntu logs only to journald, which is off-box-invisible on a worker in an isolated customer VPC, so a kubelet that fails to register left no trace. * fix(eks): give the nvidia-device-plugin its own GPU access The CDI-mode device plugin (--device-list-strategy=cdi-cri) must run nvml.Init and generate the CDI spec in its own container, but the DS had no GPU wiring, so nvml.Init crash-looped and nvidia.com/gpu was never advertised. Run the DS under the k3s-auto-configured nvidia runtimeClass with NVIDIA_VISIBLE_DEVICES/CAPABILITIES=all, the host root mounted read-only at /driver-root so it can locate the driver libs + ld.so.cache, and /var/run/cdi mounted writable so the generated spec is visible to containerd. No enable_cdi drop-in is needed on k3s v1.32.5 / containerd 2.0.5. GPU workload pods still need only resources.limits nvidia.com/gpu. Live-validated end-to-end on the single-GPU host (RTX A1000): baked control-plane AMI, GPU nodegroup, nvidia.com/gpu allocatable, nvidia-smi pod, zero manual intervention. * test(ec2): drop obsolete restoreSlogDefault test Rebased onto dev, which dropped the restoreSlogDefault workaround in #506 (viperblock no longer mutates the global slog). The test referenced the now-deleted helper and no longer compiles. Real GPU/registration diff is independently covered (diff coverage 95.3%).
85 lines
7.5 KiB
Plaintext
85 lines
7.5 KiB
Plaintext
IMAGE_NAME="eks-node"
|
|
IMAGE_DESCRIPTION="Unified EKS K3s node VM — Alpine + K3s v1.32.5+k3s1 + eks-token-webhook + konnectivity-server v0.30.3; role (server|agent) chosen at first boot"
|
|
DISTRO="alpine"
|
|
ALPINE_VERSION="3.21.7"
|
|
|
|
# 4G — server superset (K3s server binary + embedded-etcd datastore working set under
|
|
# /var/lib/rancher/k3s on the root disk). Agents use the same image; the ~10MB
|
|
# of unused server binary + webhook is dwarfed by the headroom for node images.
|
|
IMAGE_SIZE="4G"
|
|
|
|
# Server superset of both package sets (etcd-ctl is ~5MB, not worth branching).
|
|
# K3s pulls cgroups + iptables; jq builds the bootstrap-message JSON envelopes;
|
|
# etcd-client used by the snapshot script (server role only). The control plane
|
|
# reaches the host via the AWS gateway (eks-gateway-publish), not a NATS CLI.
|
|
APK_PACKAGES="ca-certificates curl jq iptables ip6tables conntrack-tools cgroup-tools etcd-ctl openrc"
|
|
|
|
# eks-token-webhook binary always installed (agents never start it).
|
|
# eks-gateway-publish relays the bootstrap envelopes + state reports to the AWS
|
|
# gateway over SigV4 HTTPS (server role only). The k3s binary is downloaded +
|
|
# SHA256-verified by setup.sh (network in chroot).
|
|
# konnectivity-server is built from the upstream apiserver-network-proxy source
|
|
# at a pinned tag (its GitHub releases ship only container images, and its go.mod
|
|
# carries replace directives that block `go install pkg@version`). The agent half
|
|
# is the matching proxy-agent container image, pinned in k3s.initd.
|
|
BUILD_COMMANDS="CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -o bin/eks-token-webhook ./cmd/eks-token-webhook && CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -o bin/eks-gateway-publish ./cmd/eks-gateway-publish && CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -o bin/eks-gateway-fetch ./cmd/eks-gateway-fetch && CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -o bin/eks-webhook-cert ./cmd/eks-webhook-cert && CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -o bin/eks-konnectivity-cert ./cmd/eks-konnectivity-cert && CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -o bin/ecr-credential-provider ./cmd/ecr-credential-provider && rm -rf /tmp/anp-build && git clone --depth 1 --branch v0.30.3 https://github.com/kubernetes-sigs/apiserver-network-proxy /tmp/anp-build && ( cd /tmp/anp-build && CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -o /tmp/anp-build/konnectivity-server ./cmd/server ) && cp /tmp/anp-build/konnectivity-server bin/konnectivity-server && rm -rf /tmp/anp-build"
|
|
|
|
INSTALL_BINARIES="bin/eks-token-webhook:/usr/local/bin/eks-token-webhook bin/eks-gateway-publish:/usr/local/bin/eks-gateway-publish bin/eks-gateway-fetch:/usr/local/bin/eks-gateway-fetch bin/eks-webhook-cert:/usr/local/bin/eks-webhook-cert bin/eks-konnectivity-cert:/usr/local/bin/eks-konnectivity-cert bin/konnectivity-server:/usr/local/bin/konnectivity-server bin/ecr-credential-provider:/usr/local/bin/ecr-credential-provider"
|
|
|
|
# All role services ship; none but the selector are enabled at bake time.
|
|
# eks-node-role picks server vs agent at first boot and enables the rest.
|
|
INSTALL_FILES="\
|
|
scripts/images/eks-node/eks-node-role.initd:/etc/init.d/eks-node-role \
|
|
scripts/images/eks-node/eks-node-role.sh:/usr/local/sbin/eks-node-role \
|
|
scripts/images/eks-node/k3s.initd:/etc/init.d/k3s \
|
|
scripts/images/eks-node/k3s-agent.initd:/etc/init.d/k3s-agent \
|
|
scripts/images/eks-node/eks-token-webhook.initd:/etc/init.d/eks-token-webhook \
|
|
scripts/images/eks-node/k3s-first-boot.initd:/etc/init.d/k3s-first-boot \
|
|
scripts/images/eks-node/k3s-first-boot.sh:/usr/local/sbin/k3s-first-boot \
|
|
scripts/images/eks-node/mulga-eks-state-report.initd:/etc/init.d/mulga-eks-state-report \
|
|
scripts/images/eks-node/mulga-eks-state-report.sh:/usr/local/sbin/mulga-eks-state-report \
|
|
scripts/images/eks-node/mulga-eks-addon-sync.initd:/etc/init.d/mulga-eks-addon-sync \
|
|
scripts/images/eks-node/mulga-eks-addon-sync.sh:/usr/local/sbin/mulga-eks-addon-sync \
|
|
scripts/images/eks-node/konnectivity-server.initd:/etc/init.d/konnectivity-server \
|
|
scripts/images/eks-node/konnectivity/konnectivity-agent.yaml:/usr/share/spinifex-eks/konnectivity/konnectivity-agent.yaml \
|
|
scripts/images/eks-node/mulga-ebs-byid.initd:/etc/init.d/mulga-ebs-byid \
|
|
scripts/images/eks-node/mulga-ebs-byid.sh:/usr/local/sbin/mulga-ebs-byid \
|
|
scripts/images/eks-node/mulga-eks-provider-id.initd:/etc/init.d/mulga-eks-provider-id \
|
|
scripts/images/eks-node/mulga-eks-provider-id.sh:/usr/local/sbin/mulga-eks-provider-id \
|
|
scripts/images/eks-node/mulga-mgmt-net.initd:/etc/init.d/mulga-mgmt-net \
|
|
scripts/images/eks-node/mulga-mgmt-net.sh:/usr/local/sbin/mulga-mgmt-net \
|
|
scripts/images/eks-node/mulga-vpc-mtu.initd:/etc/init.d/mulga-vpc-mtu \
|
|
scripts/images/eks-node/mulga-vpc-mtu.sh:/usr/local/sbin/mulga-vpc-mtu \
|
|
scripts/images/eks-node/mulga-eks-k3s-recovery.initd:/etc/init.d/mulga-eks-k3s-recovery \
|
|
scripts/images/eks-node/mulga-eks-k3s-recovery.sh:/usr/local/sbin/mulga-eks-k3s-recovery \
|
|
scripts/images/eks-node/addons/spinifex-noop/0.1.0/noop.yaml:/usr/share/spinifex-eks/addons/spinifex-noop/0.1.0/noop.yaml \
|
|
scripts/images/eks-node/addons/argocd/3.0.23/argocd.yaml:/usr/share/spinifex-eks/addons/argocd/3.0.23/argocd.yaml \
|
|
scripts/images/eks-node/addons/aws-load-balancer-controller/2.11.0/00-crds.yaml:/usr/share/spinifex-eks/addons/aws-load-balancer-controller/2.11.0/00-crds.yaml \
|
|
scripts/images/eks-node/addons/aws-load-balancer-controller/2.11.0/10-aws-load-balancer-controller.yaml:/usr/share/spinifex-eks/addons/aws-load-balancer-controller/2.11.0/10-aws-load-balancer-controller.yaml \
|
|
scripts/images/eks-node/addons/aws-load-balancer-controller/2.11.0/webhook.conf:/usr/share/spinifex-eks/addons/aws-load-balancer-controller/2.11.0/webhook.conf \
|
|
scripts/images/eks-node/addons/aws-ebs-csi-driver/1.40.1/00-rbac.yaml:/usr/share/spinifex-eks/addons/aws-ebs-csi-driver/1.40.1/00-rbac.yaml \
|
|
scripts/images/eks-node/addons/aws-ebs-csi-driver/1.40.1/10-controller.yaml:/usr/share/spinifex-eks/addons/aws-ebs-csi-driver/1.40.1/10-controller.yaml \
|
|
scripts/images/eks-node/addons/aws-ebs-csi-driver/1.40.1/20-node.yaml:/usr/share/spinifex-eks/addons/aws-ebs-csi-driver/1.40.1/20-node.yaml \
|
|
scripts/images/eks-node/addons/aws-ebs-csi-driver/1.40.1/30-storageclass.yaml:/usr/share/spinifex-eks/addons/aws-ebs-csi-driver/1.40.1/30-storageclass.yaml \
|
|
scripts/images/eks-node/addons/nvidia-device-plugin/0.17.4/10-daemonset.yaml:/usr/share/spinifex-eks/addons/nvidia-device-plugin/0.17.4/10-daemonset.yaml \
|
|
scripts/images/eks-node/coredns/coredns-mulga.yaml:/usr/share/spinifex-eks/coredns/coredns-mulga.yaml \
|
|
scripts/images/eks-node/cloud-init-ec2.cfg:/etc/cloud/cloud.cfg.d/90-spinifex-ec2-imds.cfg \
|
|
scripts/images/eks-node/mulga-eks-etcd-snapshot.sh:/etc/periodic/daily/mulga-eks-etcd-snapshot \
|
|
scripts/images/eks-node/mulga-eks-etcd-snapshot.sh:/etc/periodic/15min/mulga-eks-etcd-snapshot"
|
|
|
|
# crond (etcd snapshot), the role selector, and the role-agnostic node services
|
|
# (EBS by-id, provider-id) into the default runlevel at bake. mulga-mgmt-net is
|
|
# NOT here: setup.sh adds it to the boot runlevel before cloud-init-local, since
|
|
# it must DHCP the data NIC so the init-local Ec2 crawl reaches IMDS; a default
|
|
# runlevel entry always runs after cloud-init-local (boot) and is too late.
|
|
# cloud-init-ec2.cfg keeps a generous Ec2 max_wait as a safety net for host
|
|
# br-imds datapath lag. The selector adds the role-specific services
|
|
# (k3s/eks-token-webhook/k3s-first-boot/state-report or k3s-agent) to the
|
|
# default runlevel on first boot.
|
|
ENABLE_SERVICES="crond eks-node-role mulga-ebs-byid mulga-eks-provider-id mulga-vpc-mtu"
|
|
|
|
SETUP_SCRIPT="scripts/images/eks-node/setup.sh"
|
|
|
|
AMI_NAME="spinifex-eks-node"
|
|
SYSTEM_TAG="spinifex:managed-by=eks"
|