Files
tomnewton-mulga 6389b37433 fix(gpu): Blackwell GPU support fixes for EKS/ECS image builds and VFIO (#559)
* fix(eks): GPU node AMI declares uefi boot mode

The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but
the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under
legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at
firmware with zero serial console output.

* fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs

Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
initialize under the closed/proprietary kernel module: dmesg shows
RmInitAdapter failing with "requires use of the NVIDIA open kernel
modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to
nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia
recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no
separate open variant (userspace tools only), so it is unchanged.

* fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons

A cluster with a GPU-only nodegroup has no untainted node: the CP
carries CriticalAddonsOnly (EKS parity keeps user workloads off it)
and every worker carries nvidia.com/gpu. Without tolerations for both,
ebs-csi-controller and aws-load-balancer-controller have nowhere to
schedule and stay Pending forever. Real EKS never needs this since its
control plane isn't a schedulable node, so the stock upstream
manifests carry no such toleration.

* fix(images): give the libguestfs appliance a working resolver

The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
(127.0.0.53), which isn't running inside the appliance chroot, so DNS
is dead until package installs need it. Point it at a real resolver
before package installs, then restore the stub symlink afterward so
the shipped AMI still gets its DNS from systemd-resolved at boot like
a stock cloud image.

* fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind

A GPU's IOMMU group can include an upstream PCIe bridge when the
platform lacks ACS isolation between them. vfio-pci refuses to bind
non-endpoint bridge devices, so Claim/ReclaimByAddress must filter
bridge-class (0x06) members out of the group before touching driver
bindings, rather than failing the whole claim or leaving the bridge
rebound incorrectly.

* fix(ecs): GPU node AMI declares uefi boot mode

The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the
Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS
no OVMF/pflash firmware is attached, so the guest hangs at firmware with
zero serial console output.

Same defect and same fix as the eks-node-gpu entry: both images are built
by the same script from the same stock Ubuntu cloudimg, so both were
EFI-only and only one was corrected.

* fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch

ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary
kernel module. NVIDIA ships no proprietary module for Blackwell (GB202)
and later, so ECS GPU container-instances could not bind those cards on
any driver branch. It is the third verbatim copy of the routine and was
missed when the other two were fixed.

Also pins the driver branch across all three copies. The branch was
resolved at build time via apt-cache search | sort -rn | head -1, which
sorts on the number in the package name — but that number is a branch
alias, not a driver version (nvidia-dkms-535-server installs 580.126.20),
so the ordering key need not match what installs. Resolving also let
identical source produce different drivers on different days.

The pin lives in setup.sh rather than manifest.conf because manifest
variables are not passed into the guest: the setup script is uploaded and
executed by virt-customize --run, with no environment carried across.

* fix(eks): tolerate CP and GPU taints in argocd addon

All six argocd Deployments and its StatefulSet carried no tolerations. A
cluster with a GPU-only nodegroup has no untainted node to place them on:
the control plane carries CriticalAddonsOnly and every worker carries
nvidia.com/gpu, so the pods stay Pending forever.

Same defect the ebs-csi-controller and load-balancer-controller addons
already had. Real EKS never hits it because its control plane is not a
schedulable node, so the stock upstream manifests carry no such
toleration and the divergence is ours to maintain.

---------

Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>
2026-07-27 12:11:18 +10:00

565 lines
12 KiB
YAML

---
apiVersion: v1
kind: ServiceAccount
metadata:
name: aws-load-balancer-controller
namespace: kube-system
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: aws-load-balancer-controller
app.kubernetes.io/version: "v2.11.0"
annotations:
eks.amazonaws.com/role-arn: {{SERVICE_ACCOUNT_ROLE_ARN}}
automountServiceAccountToken: true
---
# spinifex: node-minted webhook serving cert (render_addon mints a self-signed
# cert for the webhook service SANs; cert-manager is not present on the node).
apiVersion: v1
kind: Secret
metadata:
name: aws-load-balancer-webhook-tls
namespace: kube-system
labels:
app.kubernetes.io/name: aws-load-balancer-controller
type: kubernetes.io/tls
data:
ca.crt: "{{WEBHOOK_CA_B64}}"
tls.crt: "{{WEBHOOK_TLS_CRT_B64}}"
tls.key: "{{WEBHOOK_TLS_KEY_B64}}"
---
# spinifex: the AWS gateway CA, mounted as AWS_CA_BUNDLE so the web-identity
# credential bootstrap + elbv2/ec2/acm calls trust the gateway TLS endpoint.
apiVersion: v1
kind: Secret
metadata:
name: spinifex-gateway-ca
namespace: kube-system
labels:
app.kubernetes.io/name: aws-load-balancer-controller
type: Opaque
data:
ca.pem: "{{GATEWAY_CA_PEM_B64}}"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-controller-leader-election-role
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resourceNames:
- aws-load-balancer-controller-leader
resources:
- configmaps
verbs:
- get
- update
- patch
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- create
- apiGroups:
- coordination.k8s.io
resourceNames:
- aws-load-balancer-controller-leader
resources:
- leases
verbs:
- get
- update
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-controller-role
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- delete
- get
- update
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- pods/status
verbs:
- patch
- update
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- services/status
verbs:
- patch
- update
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- get
- list
- watch
- apiGroups:
- elbv2.k8s.aws
resources:
- ingressclassparams
verbs:
- get
- list
- watch
- apiGroups:
- elbv2.k8s.aws
resources:
- targetgroupbindings
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- elbv2.k8s.aws
resources:
- targetgroupbindings/status
verbs:
- patch
- update
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- get
- list
- patch
- update
- watch
- apiGroups:
- extensions
resources:
- ingresses/status
verbs:
- patch
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- patch
- update
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- patch
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-controller-leader-election-rolebinding
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: aws-load-balancer-controller-leader-election-role
subjects:
- kind: ServiceAccount
name: aws-load-balancer-controller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-controller-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: aws-load-balancer-controller-role
subjects:
- kind: ServiceAccount
name: aws-load-balancer-controller
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-webhook-service
namespace: kube-system
spec:
ports:
- port: 443
targetPort: 9443
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/name: aws-load-balancer-controller
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-controller
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: aws-load-balancer-controller
template:
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: aws-load-balancer-controller
spec:
containers:
- args:
- --cluster-name={{CLUSTER_NAME}}
- --ingress-class=alb
- --feature-gates=ALBSingleSubnet=true
- --aws-region={{AWS_REGION}}
- --aws-vpc-id={{AWS_VPC_ID}}
- --aws-api-endpoints=EC2={{GATEWAY_ENDPOINT}},Elastic Load Balancing v2={{GATEWAY_ENDPOINT}},ACM={{GATEWAY_ENDPOINT}},STS={{GATEWAY_ENDPOINT}}
- --enable-shield=false
- --enable-waf=false
- --enable-wafv2=false
env:
{{IRSA_ENV}}
image: public.ecr.aws/eks/aws-load-balancer-controller:v2.11.0
livenessProbe:
failureThreshold: 2
httpGet:
path: /healthz
port: 61779
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 10
name: controller
ports:
- containerPort: 9443
name: webhook-server
protocol: TCP
resources:
limits:
cpu: 200m
memory: 500Mi
requests:
cpu: 100m
memory: 200Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
volumeMounts:
- mountPath: /tmp/k8s-webhook-server/serving-certs
name: cert
readOnly: true
{{IRSA_VOLUME_MOUNT}}
priorityClassName: system-cluster-critical
# A cluster with a GPU-only nodegroup has no untainted node: the CP carries
# CriticalAddonsOnly (EKS parity keeps user workloads off it) and every
# worker carries nvidia.com/gpu (nodegroup_userdata.go). Without these
# tolerations the controller has nowhere to schedule and stays Pending
# forever. Real EKS never needs this: its control plane isn't a schedulable
# node, so the stock upstream manifest carries no such toleration.
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
securityContext:
fsGroup: 1337
serviceAccountName: aws-load-balancer-controller
terminationGracePeriodSeconds: 10
volumes:
- name: cert
secret:
defaultMode: 420
secretName: aws-load-balancer-webhook-tls
{{IRSA_VOLUME}}
---
apiVersion: v1
kind: List
metadata:
name: ingress-class
items:
- apiVersion: elbv2.k8s.aws/v1beta1
kind: IngressClassParams
metadata:
name: alb
labels:
app.kubernetes.io/name: aws-load-balancer-controller
app.kubernetes.io/version: "v2.11.0"
{{ELB_INGRESS_PARAMS_SPEC}}
- apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: alb
labels:
app.kubernetes.io/name: aws-load-balancer-controller
app.kubernetes.io/version: "v2.11.0"
spec:
controller: ingress.k8s.aws/alb
parameters:
apiGroup: elbv2.k8s.aws
kind: IngressClassParams
name: alb
---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-webhook
webhooks:
- admissionReviewVersions:
- v1beta1
clientConfig:
caBundle: "{{WEBHOOK_CA_B64}}"
service:
name: aws-load-balancer-webhook-service
namespace: kube-system
path: /mutate-v1-service
failurePolicy: Fail
name: mservice.elbv2.k8s.aws
objectSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: NotIn
values:
- aws-load-balancer-controller
rules:
- apiGroups:
- ""
apiVersions:
- v1
operations:
- CREATE
resources:
- services
sideEffects: None
- admissionReviewVersions:
- v1beta1
clientConfig:
caBundle: "{{WEBHOOK_CA_B64}}"
service:
name: aws-load-balancer-webhook-service
namespace: kube-system
path: /mutate-v1-pod
failurePolicy: Fail
name: mpod.elbv2.k8s.aws
namespaceSelector:
matchExpressions:
- key: elbv2.k8s.aws/pod-readiness-gate-inject
operator: In
values:
- enabled
objectSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: NotIn
values:
- aws-load-balancer-controller
rules:
- apiGroups:
- ""
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
sideEffects: None
- admissionReviewVersions:
- v1beta1
clientConfig:
caBundle: "{{WEBHOOK_CA_B64}}"
service:
name: aws-load-balancer-webhook-service
namespace: kube-system
path: /mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding
failurePolicy: Fail
name: mtargetgroupbinding.elbv2.k8s.aws
rules:
- apiGroups:
- elbv2.k8s.aws
apiVersions:
- v1beta1
operations:
- CREATE
- UPDATE
resources:
- targetgroupbindings
sideEffects: None
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-webhook
webhooks:
- admissionReviewVersions:
- v1beta1
clientConfig:
caBundle: "{{WEBHOOK_CA_B64}}"
service:
name: aws-load-balancer-webhook-service
namespace: kube-system
path: /validate-elbv2-k8s-aws-v1beta1-ingressclassparams
failurePolicy: Fail
name: vingressclassparams.elbv2.k8s.aws
objectSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: NotIn
values:
- aws-load-balancer-controller
rules:
- apiGroups:
- elbv2.k8s.aws
apiVersions:
- v1beta1
operations:
- CREATE
- UPDATE
resources:
- ingressclassparams
sideEffects: None
- admissionReviewVersions:
- v1beta1
clientConfig:
caBundle: "{{WEBHOOK_CA_B64}}"
service:
name: aws-load-balancer-webhook-service
namespace: kube-system
path: /validate-elbv2-k8s-aws-v1beta1-targetgroupbinding
failurePolicy: Fail
name: vtargetgroupbinding.elbv2.k8s.aws
rules:
- apiGroups:
- elbv2.k8s.aws
apiVersions:
- v1beta1
operations:
- CREATE
- UPDATE
resources:
- targetgroupbindings
sideEffects: None
- admissionReviewVersions:
- v1beta1
clientConfig:
caBundle: "{{WEBHOOK_CA_B64}}"
service:
name: aws-load-balancer-webhook-service
namespace: kube-system
path: /validate-networking-v1-ingress
failurePolicy: Fail
matchPolicy: Equivalent
name: vingress.elbv2.k8s.aws
rules:
- apiGroups:
- networking.k8s.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- ingresses
sideEffects: None