gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-23 12:47:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
a4a4288d7d37eeafeba9052d0a9d2ed1fdcb7813
spinifex/scripts/images/eks-node/addons/aws-load-balancer-controller/2.11.0/10-aws-load-balancer-controller.yaml
T
Josh Siviour 2acb37342d feat(eks): aws-load-balancer-controller addon — real ALB Ingress (L7, IRSA) (#376) 
* fix(ec2): stamp subnet AvailabilityZoneId for single-AZ ALB resolution

DescribeSubnets omitted AvailabilityZoneId, so the AWS Load Balancer
Controller's DescribeAvailabilityZones-by-ZoneId lookup resolved no AZ
info ("cannot resolve AZ info for AZs: []") and never built the ALB
model. Stamp the single zone-id on every subnet and share it with
DescribeAvailabilityZones through one SingleZoneID constant so the two
responses can never drift.

* feat(eks): add aws-load-balancer-controller v2.8.1 addon bundle

Baked manifests for the LBC managed addon (CRDs, controller Deployment/
RBAC/webhooks/Service, IngressClass alb, IRSA render markers). Two
wirings make it work against the spinifex gateway:

- --aws-api-endpoints carries sts= alongside elasticloadbalancing/ec2/
  acm. The aws-sdk-go v1 credential bootstrap resolves the STS client
  from the session endpoint resolver, not AWS_ENDPOINT_URL_STS, so the
  web-identity AssumeRoleWithWebIdentity must be pointed at the gateway
  here or it falls back to real AWS STS.
- --feature-gates=ALBSingleSubnet=true drops the ALB minimum subnet
  count from two to one so an ALB provisions in a single-AZ deployment.

* feat(eks): render IRSA into managed addons + node-minted webhook cert

Extend the addon-sync renderer to inject the projected-token / role-arn /
gateway STS-endpoint + CA block into managed-addon pod specs via opt-in
markers, and substitute cluster-name/region/vpc-id/gateway-endpoint/CA
placeholders. Add cmd/eks-webhook-cert: the eks-node AMI has no openssl,
so this mints (then reuses) a self-signed ECDSA-P256 serving cert for the
admission webhook, cached on disk so the render is byte-stable across
ticks. Wire both into the image build (manifest.conf) plus INSTALL_FILES
for the LBC bundle.

* feat(eks): surface cluster VPC id as EKS_VPC_ID for the LB controller

Thread the control-plane VPC id through K3sServerInput.VpcID into the
first-boot EKS_VPC_ID env so the in-cluster aws-load-balancer-controller
can pass --aws-vpc-id to the gateway elbv2/ec2 handlers.

* fix(eks): point the LB controller at the customer VPC + customer-facing gateway

The aws-load-balancer-controller ran with two control-plane-scoped values
that customer worker pods cannot use:

- --aws-vpc-id was set from the managed control-plane VPC (cpRefs.VpcID),
  which the gateway's DescribeVpcs does not return for customer calls, so
  backend-SG auto-create failed with InvalidVpcID.NotFound. Source it from
  meta.ResourcesVpcConfig.VpcId, the customer cluster VPC where the workers
  and their subnets live and where the ALB is provisioned.

- The addon render baked the mgmt gateway endpoint (EKS_GATEWAY_URL /
  SystemGatewayURL) into pod specs, unreachable from worker pods. Add a
  distinct EKS_ADDON_GATEWAY_URL (the customer-facing GatewayBaseURL, same
  host as the OIDC issuer) and substitute it for {{GATEWAY_ENDPOINT}} in
  managed addon manifests. EKS_GATEWAY_URL stays mgmt for the CP's own
  bootstrap, state-report and token-webhook paths.

* fix(elbv2): resolve forward target group from ForwardConfig

listenerActionFromSDK read only the legacy flat Action.TargetGroupArn, so a
forward action carrying its target group via ForwardConfig.TargetGroups (the
shape the aws-load-balancer-controller emits) left the action target empty and
CreateRule/CreateListener rejected it with MissingParameter. Flatten the first
target group from ForwardConfig when the flat field is absent.

* fix(ec2): mirror CreateTags/DeleteTags into subnet/vpc records

The generic S3 tag store and the per-resource KV records are disjoint:
DescribeSubnets/DescribeVpcs read tags from their KV record, while
CreateTags only wrote the generic store. Subnet tags applied after
create were therefore invisible to tag-filtered describes, breaking
aws-load-balancer-controller subnet auto-discovery (kubernetes.io/role/elb).

Write through CreateTags/DeleteTags to the owning subnet/vpc record so
tag-filtered describes observe them. Resource types the VPC service does
not own are skipped; the generic store stays their record of truth.

* test(daemon): pin synthetic capacity in host-dependent allocate tests

TestCanAllocate_CountEdgeCases/Mixed_instance_types and
TestHandleEC2DescribeInstanceTypes/NoFilterReturnsSupportedRegardlessOfAllocation
called rm.allocate against live-detected host capacity. allocate gates on
readMemAvailableGB (live /proc MemAvailable), so a box already running VMs
fails the allocation spuriously, breaking the preflight gate locally.

Pin hostVCPU/hostMemGB/reserved and nil readMemAvailableGB to a synthetic
host, matching the sibling Capacity_decreases_after_allocation subtest.

* fix(eks): route apiserver->pod traffic via agent tunnel (egress-selector cluster)

In the managed-CP topology the apiserver VM sits in the system CP VPC with
no route to the worker pod network in the unpeered customer VPC. k3s default
egress-selector-mode=agent tunnels only apiserver->kubelet, so kubectl
logs/exec work but apiserver->pod traffic (admission webhooks, aggregated
APIs) is attempted directly and times out -- the aws-load-balancer-controller
webhooks fail with context deadline exceeded, blocking Service/Ingress
creation cluster-wide.

Set egress-selector-mode=cluster so apiserver->pod/service traffic routes
through the agent's outbound tunnel, selected by the node hosting the
destination, without requiring inbound CP->worker reachability.

* feat: bump aws-load-balancer-controller addon bundle to v2.11.0

v2.8.1 predates the ALBSingleSubnet feature gate, so single-AZ spinifex
clusters fail Ingress reconcile (subnet resolver dedups to 1 < required 2)
and never provision an ALB. v2.11.0 honors --feature-gates=ALBSingleSubnet=true.

Replace the baked bundle: new CRDs (ingressclassparams + targetgroupbindings),
controller manifest, and image tag v2.11.0, re-applying the spinifex deltas
(IRSA SA annotation, node-minted webhook serving-cert secret, gateway-CA
secret, gateway aws-api-endpoints, caBundle on every webhook clientConfig,
alb IngressClass). Pick up the new ClusterRole configmaps rule. Drop v2.8.1.

* fix: point LBC at the gateway via AWS_ENDPOINT_URL (sdk-v2)

LBC v2.11 uses aws-sdk-go-v2, which keys --aws-api-endpoints on SDK
service IDs (e.g. 'Elastic Load Balancing v2'), not the legacy lowercase
names the bundle passed. Those overrides were silently ignored, so every
non-STS call (ec2/elbv2/acm) went to real AWS and failed TLS against the
gateway CA -- no ALB could provision.

Inject AWS_ENDPOINT_URL in the shared IRSA env block (sdk-v2 global
base-endpoint override; AWS_ENDPOINT_URL_STS still takes precedence for
STS) and drop the dead --aws-api-endpoints flag from the controller args.

* fix: synthesize taggable ARN for default listener rule

Gateway DescribeRules returned the synthetic IsDefault rule with a null
RuleArn. LBC read the arn-less default rule after creating a listener and
issued DescribeTags with an empty resource ARN, which the gateway rejected
with 400 MissingParameter, aborting reconcile.

Set a deterministic RuleArn on the default rule, resolve it back to its
parent listener in DescribeTags (empty tags) and DescribeRules by-arn.
Also disable shield/waf/wafv2 in the LBC bundle to drop unsupported calls.

* feat: inject ELB subnets into alb IngressClassParams for single-AZ ALB

LBC's tag auto-discovery never threads the ALBSingleSubnet feature gate, so a
single-AZ cluster (subnets sharing one AZ) dedups to 1<2 subnets and aborts
Ingress reconcile. Only the explicit-subnet path honors the gate.

Resolve the cluster's ELB-eligible subnets, dedup to one per AZ, and surface
them as EKS_ELB_SUBNET_IDS. The addon sync renders them into the alb
IngressClassParams .spec.subnets.ids so every Ingress takes the explicit path.
An empty set falls back to auto-discovery, preserving the native UX.

* fix: route LBC service clients to gateway via --aws-api-endpoints, link IngressClassParams

Live verification showed two gaps in the LBC addon bundle:

LBC installs a per-client BaseEndpoint resolver, so the global AWS_ENDPOINT_URL
is ignored for the ec2/elbv2/acm clients (calls hit real AWS, failing TLS).
Redirect them with --aws-api-endpoints keyed on the aws-sdk-go-v2 service IDs
(EC2, "Elastic Load Balancing v2", ACM, STS) rather than the legacy lowercase
names the flag never matched. Drop the misleading AWS_ENDPOINT_URL env; the
web-identity STS client still resolves via AWS_ENDPOINT_URL_STS.

The injected IngressClassParams was orphaned: the alb IngressClass had no
.spec.parameters, so LBC fell back to subnet auto-discovery (which fails on a
single-AZ cluster). Link the IngressClass to the IngressClassParams so the
explicit-subnet path is taken.

* fix: stamp EKS-parity providerID on worker nodes for LBC ip targets

k3s registers workers with providerID k3s://<name>, which the AWS Load
Balancer Controller rejects (providerID invalid for EC2 instances) when
resolving nodes for ip-mode target registration. Resolve the instance-id and
AZ from IMDSv2 at agent boot and set --kubelet-arg=provider-id=aws:///<az>/<id>
so LBC can map the node to its EC2 instance. On any IMDS miss the agent keeps
the default providerID and still starts.

* fix(elbv2): honor SubnetMappings in CreateLoadBalancer

LBC supplies ALB subnets via SubnetMappings, not the plain Subnets list.
createLoadBalancer only read input.Subnets, so ENI allocation was skipped,
VpcId/AZ stayed empty, and the HAProxy data-plane VM never launched — the
ALB went active but could not serve traffic. Add flattenSubnetIDs to read
both sources and reuse it in createLoadBalancer and desiredSubnetSet.

* fix(vm): capture GCInterval before spawning GC goroutine

TestStart_SweepsImmediatelyThenStopsOnCtxCancel mutates the package global
GCInterval and restores it in a defer, racing the Start goroutine's read at
time.NewTicker(GCInterval) under -race. Read GCInterval on the caller's
goroutine into a local before spawning so the global access is ordered with
the test's defer; the goroutine touches only the captured local.
2026-06-18 10:53:05 +10:00

553 lines
11 KiB
YAML
Raw Blame History

---
apiVersion: v1
kind: ServiceAccount
metadata:
name: aws-load-balancer-controller
namespace: kube-system
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: aws-load-balancer-controller
app.kubernetes.io/version: "v2.11.0"
annotations:
eks.amazonaws.com/role-arn: {{SERVICE_ACCOUNT_ROLE_ARN}}
automountServiceAccountToken: true
---
# spinifex: node-minted webhook serving cert (render_addon mints a self-signed
# cert for the webhook service SANs; cert-manager is not present on the node).
apiVersion: v1
kind: Secret
metadata:
name: aws-load-balancer-webhook-tls
namespace: kube-system
labels:
app.kubernetes.io/name: aws-load-balancer-controller
type: kubernetes.io/tls
data:
ca.crt: "{{WEBHOOK_CA_B64}}"
tls.crt: "{{WEBHOOK_TLS_CRT_B64}}"
tls.key: "{{WEBHOOK_TLS_KEY_B64}}"
---
# spinifex: the AWS gateway CA, mounted as AWS_CA_BUNDLE so the web-identity
# credential bootstrap + elbv2/ec2/acm calls trust the gateway TLS endpoint.
apiVersion: v1
kind: Secret
metadata:
name: spinifex-gateway-ca
namespace: kube-system
labels:
app.kubernetes.io/name: aws-load-balancer-controller
type: Opaque
data:
ca.pem: "{{GATEWAY_CA_PEM_B64}}"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-controller-leader-election-role
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resourceNames:
- aws-load-balancer-controller-leader
resources:
- configmaps
verbs:
- get
- update
- patch
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- create
- apiGroups:
- coordination.k8s.io
resourceNames:
- aws-load-balancer-controller-leader
resources:
- leases
verbs:
- get
- update
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-controller-role
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- delete
- get
- update
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- pods/status
verbs:
- patch
- update
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- services/status
verbs:
- patch
- update
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- get
- list
- watch
- apiGroups:
- elbv2.k8s.aws
resources:
- ingressclassparams
verbs:
- get
- list
- watch
- apiGroups:
- elbv2.k8s.aws
resources:
- targetgroupbindings
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- elbv2.k8s.aws
resources:
- targetgroupbindings/status
verbs:
- patch
- update
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- get
- list
- patch
- update
- watch
- apiGroups:
- extensions
resources:
- ingresses/status
verbs:
- patch
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- patch
- update
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- patch
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-controller-leader-election-rolebinding
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: aws-load-balancer-controller-leader-election-role
subjects:
- kind: ServiceAccount
name: aws-load-balancer-controller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-controller-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: aws-load-balancer-controller-role
subjects:
- kind: ServiceAccount
name: aws-load-balancer-controller
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-webhook-service
namespace: kube-system
spec:
ports:
- port: 443
targetPort: 9443
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/name: aws-load-balancer-controller
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-controller
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: aws-load-balancer-controller
template:
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: aws-load-balancer-controller
spec:
containers:
- args:
- --cluster-name={{CLUSTER_NAME}}
- --ingress-class=alb
- --feature-gates=ALBSingleSubnet=true
- --aws-region={{AWS_REGION}}
- --aws-vpc-id={{AWS_VPC_ID}}
- --aws-api-endpoints=EC2={{GATEWAY_ENDPOINT}},Elastic Load Balancing v2={{GATEWAY_ENDPOINT}},ACM={{GATEWAY_ENDPOINT}},STS={{GATEWAY_ENDPOINT}}
- --enable-shield=false
- --enable-waf=false
- --enable-wafv2=false
env:
{{IRSA_ENV}}
image: public.ecr.aws/eks/aws-load-balancer-controller:v2.11.0
livenessProbe:
failureThreshold: 2
httpGet:
path: /healthz
port: 61779
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 10
name: controller
ports:
- containerPort: 9443
name: webhook-server
protocol: TCP
resources:
limits:
cpu: 200m
memory: 500Mi
requests:
cpu: 100m
memory: 200Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
volumeMounts:
- mountPath: /tmp/k8s-webhook-server/serving-certs
name: cert
readOnly: true
{{IRSA_VOLUME_MOUNT}}
priorityClassName: system-cluster-critical
securityContext:
fsGroup: 1337
serviceAccountName: aws-load-balancer-controller
terminationGracePeriodSeconds: 10
volumes:
- name: cert
secret:
defaultMode: 420
secretName: aws-load-balancer-webhook-tls
{{IRSA_VOLUME}}
---
apiVersion: v1
kind: List
metadata:
name: ingress-class
items:
- apiVersion: elbv2.k8s.aws/v1beta1
kind: IngressClassParams
metadata:
name: alb
labels:
app.kubernetes.io/name: aws-load-balancer-controller
app.kubernetes.io/version: "v2.11.0"
{{ELB_INGRESS_PARAMS_SPEC}}
- apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: alb
labels:
app.kubernetes.io/name: aws-load-balancer-controller
app.kubernetes.io/version: "v2.11.0"
spec:
controller: ingress.k8s.aws/alb
parameters:
apiGroup: elbv2.k8s.aws
kind: IngressClassParams
name: alb
---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-webhook
webhooks:
- admissionReviewVersions:
- v1beta1
clientConfig:
caBundle: "{{WEBHOOK_CA_B64}}"
service:
name: aws-load-balancer-webhook-service
namespace: kube-system
path: /mutate-v1-service
failurePolicy: Fail
name: mservice.elbv2.k8s.aws
objectSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: NotIn
values:
- aws-load-balancer-controller
rules:
- apiGroups:
- ""
apiVersions:
- v1
operations:
- CREATE
resources:
- services
sideEffects: None
- admissionReviewVersions:
- v1beta1
clientConfig:
caBundle: "{{WEBHOOK_CA_B64}}"
service:
name: aws-load-balancer-webhook-service
namespace: kube-system
path: /mutate-v1-pod
failurePolicy: Fail
name: mpod.elbv2.k8s.aws
namespaceSelector:
matchExpressions:
- key: elbv2.k8s.aws/pod-readiness-gate-inject
operator: In
values:
- enabled
objectSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: NotIn
values:
- aws-load-balancer-controller
rules:
- apiGroups:
- ""
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
sideEffects: None
- admissionReviewVersions:
- v1beta1
clientConfig:
caBundle: "{{WEBHOOK_CA_B64}}"
service:
name: aws-load-balancer-webhook-service
namespace: kube-system
path: /mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding
failurePolicy: Fail
name: mtargetgroupbinding.elbv2.k8s.aws
rules:
- apiGroups:
- elbv2.k8s.aws
apiVersions:
- v1beta1
operations:
- CREATE
- UPDATE
resources:
- targetgroupbindings
sideEffects: None
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-webhook
webhooks:
- admissionReviewVersions:
- v1beta1
clientConfig:
caBundle: "{{WEBHOOK_CA_B64}}"
service:
name: aws-load-balancer-webhook-service
namespace: kube-system
path: /validate-elbv2-k8s-aws-v1beta1-ingressclassparams
failurePolicy: Fail
name: vingressclassparams.elbv2.k8s.aws
objectSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: NotIn
values:
- aws-load-balancer-controller
rules:
- apiGroups:
- elbv2.k8s.aws
apiVersions:
- v1beta1
operations:
- CREATE
- UPDATE
resources:
- ingressclassparams
sideEffects: None
- admissionReviewVersions:
- v1beta1
clientConfig:
caBundle: "{{WEBHOOK_CA_B64}}"
service:
name: aws-load-balancer-webhook-service
namespace: kube-system
path: /validate-elbv2-k8s-aws-v1beta1-targetgroupbinding
failurePolicy: Fail
name: vtargetgroupbinding.elbv2.k8s.aws
rules:
- apiGroups:
- elbv2.k8s.aws
apiVersions:
- v1beta1
operations:
- CREATE
- UPDATE
resources:
- targetgroupbindings
sideEffects: None
- admissionReviewVersions:
- v1beta1
clientConfig:
caBundle: "{{WEBHOOK_CA_B64}}"
service:
name: aws-load-balancer-webhook-service
namespace: kube-system
path: /validate-networking-v1-ingress
failurePolicy: Fail
matchPolicy: Equivalent
name: vingress.elbv2.k8s.aws
rules:
- apiGroups:
- networking.k8s.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- ingresses
sideEffects: None
Reference in New Issue View Git Blame Copy Permalink