spinifex

gsadmin/spinifex

Fork 0

mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-19 10:52:04 +00:00

Commit Graph

Author	SHA1	Message	Date
Josh Siviour	6873875248	fix(eks): tolerate the CriticalAddonsOnly taint on konnectivity-agent (#634 ) On a zero-worker or GPU-only cluster no node tolerates the DaemonSet, so it has zero desired pods. The apiserver then has no tunnel to kube-system pods (including metrics-server), v1beta1.metrics.k8s.io never goes Available, and namespace GC hangs on the kubernetes finalizer for every namespace, not just ones using metrics. Add the same CriticalAddonsOnly + nvidia.com/gpu tolerations already used by every other control-plane-critical addon in this repo (aws-load-balancer-controller, ebs-csi-driver, argocd) so the agent also schedules onto the control plane.	2026-08-02 08:16:00 +10:00
Josh Siviour	f7bf921ce7	fix(eks): NLB name overflow, nodegroup readiness scoping, launch retry, GPU konnectivity toleration (#511 ) * fix(eks): keep derived ELBv2 names under 32 chars and surface CREATE_FAILED reason Long cluster names made the derived "eks-<name>[-cp\|-konn]" NLB/target-group names exceed the ELBv2 32-char limit, failing cluster creation asynchronously with no visible cause. safeELBv2Name now deterministically truncates the name and appends a stable hash of the full cluster name when needed, so the same name is derived on every call (create, lookup, delete). DescribeCluster also now surfaces ClusterMeta.StatusReason as a Health.Issues entry when a cluster is FAILED, instead of leaving callers to guess why. * fix(eks): plumb per-nodegroup Ready-node counts through state reports The on-VM state-report script now emits nodegroup_ready, a per-nodegroup Ready count keyed by the eks.amazonaws.com/nodegroup label value, alongside the existing cluster-wide node_count. ServerStateReport, the ClusterReconciler's observe step, and SetClusterHealthState now carry this map through to ClusterMeta.NodegroupNodeCounts, so per-nodegroup consumers (wired in the next commit) can gate on a nodegroup's own workers instead of the cluster total. * fix(eks): scope nodegroup Ready-gate per nodegroup and retry failed worker launches waitWorkersReady compared the cluster-wide Ready-node count against a nodegroup's baseline+desired, so in a multi-nodegroup cluster one nodegroup's Ready workers could mask another whose own workers never registered. It now reads meta.NodegroupNodeCounts[ng], scoped to nodes carrying the eks.amazonaws.com/nodegroup=<name> label. Separately, launchNodegroupInfra never retried a failed worker launch (a transient QMP timeout or host capacity pressure), leaving the nodegroup stuck CREATING with zero workers. The new launchWorkersUntilDesired wrapper relaunches only the shortfall on failure, logging each retry, until the desired count is reached or workerLaunchRetryTimeout elapses. Both fixes are covered by regression tests: a two-nodegroup scenario where only one has Ready workers, a transient-failure-then-success relaunch, and an exhausted-retry-budget CREATE_FAILED path. Not covered here: a worker that dies after the nodegroup is already ACTIVE — that needs an ongoing reconciler, which this change does not add. * fix(eks): tolerate GPU NoSchedule taint in konnectivity-agent DaemonSet A GPU-only worker is tainted nvidia.com/gpu=present:NoSchedule, so the konnectivity-agent DaemonSet (no tolerations) showed DESIRED=0 there and kubectl logs/exec against pods on that node failed "No agent available". Adds an Exists toleration for nvidia.com/gpu:NoSchedule. Live-cluster validation (DESIRED count, kubectl exec) is deferred — no cluster available to test against right now. * fix: tolerate nvidia.com/gpu taint on ebs-csi-node and coredns DaemonSets so GPU workers get storage and DNS	2026-07-12 12:11:16 +10:00
Josh Siviour	7bbd99c792	feat(eks): multinode control-plane + IMDS/token-webhook CP fixes (426/427/428/495/496) (#459 ) * feat(eks): multinode control-plane networking, IAM, and CoreDNS parity Integrates the EKS multinode fix set for the managed-control-plane topology so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it. EKS: - Route apiserver egress (exec/logs/webhooks) through konnectivity (apiserver-network-proxy): CP servers run konnectivity-server behind the cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod traffic rides the agent-initiated reverse tunnel instead of an unroutable CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every replica (HA-correct). - Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay flannel mesh SG rule authorized at cluster launch with ordered teardown. - Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay. - Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity. - Spread nodegroup workers across hosts; addon readiness falls back on workload rollout when gvks are absent. - elbv2: allow the proxy_protocol_v2.enabled target-group attribute. EBS: - Move volume attachment state to state.json so a live nbdkit VB cannot clobber it; route mounted-volume state updates through the live VB with a retry for in-use marks racing the mount responder; keep the boot volume in-use across stop/crash unmount. * fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM missingOrchestrationDeps gated on s.deps.IAM, but production wires only the lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only). The gate therefore reported missing=[IAM] on every node forever, so CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even though the IAM service was fully up via the provider. Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else deps.IAMProvider()) the nodegroup/role code already uses — so the protective check still rejects a node with no IAM service but passes when the provider supplies one. Add a regression test that sets only IAMProvider and asserts the gate is satisfied; the existing nil-both test still rejects. * fix(eks): let eks-token-webhook use IMDS creds when static keys absent The CP VM launches on an instance profile, so buildK3sUserData omits EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys before eksgw.New could fall back to the IMDS SDK chain, so the webhook aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver crashed on the missing --authentication-token-webhook-config-file file, stalling the cluster at CREATING until it timed out to FAILED. Drop the static-key precondition (matching the sibling eks-gateway-publish and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/ cluster gates. * fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt, which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a 169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing 169.254.169.254 and steering IMDS off the data NIC away from the per-tap br-imds datapath, where the host RST it. eks-gateway-publish then failed to sign requests and no bootstrap artifacts were ever published, so clusters sat CREATING until they timed out to FAILED. Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI) are unaffected and IMDS falls through the data NIC's default route to the br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net deletes any default route on a NIC declared NIC<n>_DEFAULT=0. * fix(eks): grant eks:WebhookTokenReview to the control-plane instance role The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway broker at POST /clusters/{name}/token-review, signing with the control-plane VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes against eks:WebhookTokenReview, but the role's inline policy granted only eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every relay and the apiserver returned 401 to all external kubectl. Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile re-asserts the inline policy on every control-plane instance-profile ensure, and the gateway evaluates authorization per request, so existing clusters converge without a rebuild once the shared role is updated.	2026-07-09 09:43:30 +10:00

Author

SHA1

Message

Date

Josh Siviour

6873875248

fix(eks): tolerate the CriticalAddonsOnly taint on konnectivity-agent (#634 )

On a zero-worker or GPU-only cluster no node tolerates the DaemonSet,
so it has zero desired pods. The apiserver then has no tunnel to
kube-system pods (including metrics-server), v1beta1.metrics.k8s.io
never goes Available, and namespace GC hangs on the kubernetes
finalizer for every namespace, not just ones using metrics.

Add the same CriticalAddonsOnly + nvidia.com/gpu tolerations already
used by every other control-plane-critical addon in this repo
(aws-load-balancer-controller, ebs-csi-driver, argocd) so the agent
also schedules onto the control plane.

2026-08-02 08:16:00 +10:00

Josh Siviour

f7bf921ce7

fix(eks): NLB name overflow, nodegroup readiness scoping, launch retry, GPU konnectivity toleration (#511 )

* fix(eks): keep derived ELBv2 names under 32 chars and surface CREATE_FAILED reason

Long cluster names made the derived "eks-<name>[-cp|-konn]" NLB/target-group
names exceed the ELBv2 32-char limit, failing cluster creation asynchronously
with no visible cause. safeELBv2Name now deterministically truncates the name
and appends a stable hash of the full cluster name when needed, so the same
name is derived on every call (create, lookup, delete). DescribeCluster also
now surfaces ClusterMeta.StatusReason as a Health.Issues entry when a cluster
is FAILED, instead of leaving callers to guess why.

* fix(eks): plumb per-nodegroup Ready-node counts through state reports

The on-VM state-report script now emits nodegroup_ready, a per-nodegroup Ready
count keyed by the eks.amazonaws.com/nodegroup label value, alongside the
existing cluster-wide node_count. ServerStateReport, the ClusterReconciler's
observe step, and SetClusterHealthState now carry this map through to
ClusterMeta.NodegroupNodeCounts, so per-nodegroup consumers (wired in the next
commit) can gate on a nodegroup's own workers instead of the cluster total.

* fix(eks): scope nodegroup Ready-gate per nodegroup and retry failed worker launches

waitWorkersReady compared the cluster-wide Ready-node count against a
nodegroup's baseline+desired, so in a multi-nodegroup cluster one nodegroup's
Ready workers could mask another whose own workers never registered. It now
reads meta.NodegroupNodeCounts[ng], scoped to nodes carrying the
eks.amazonaws.com/nodegroup=<name> label.

Separately, launchNodegroupInfra never retried a failed worker launch (a
transient QMP timeout or host capacity pressure), leaving the nodegroup stuck
CREATING with zero workers. The new launchWorkersUntilDesired wrapper relaunches
only the shortfall on failure, logging each retry, until the desired count is
reached or workerLaunchRetryTimeout elapses.

Both fixes are covered by regression tests: a two-nodegroup scenario where only
one has Ready workers, a transient-failure-then-success relaunch, and an
exhausted-retry-budget CREATE_FAILED path. Not covered here: a worker that dies
after the nodegroup is already ACTIVE — that needs an ongoing reconciler, which
this change does not add.

* fix(eks): tolerate GPU NoSchedule taint in konnectivity-agent DaemonSet

A GPU-only worker is tainted nvidia.com/gpu=present:NoSchedule, so the
konnectivity-agent DaemonSet (no tolerations) showed DESIRED=0 there and
kubectl logs/exec against pods on that node failed "No agent available".
Adds an Exists toleration for nvidia.com/gpu:NoSchedule.

Live-cluster validation (DESIRED count, kubectl exec) is deferred — no cluster
available to test against right now.

* fix: tolerate nvidia.com/gpu taint on ebs-csi-node and coredns DaemonSets so GPU workers get storage and DNS

2026-07-12 12:11:16 +10:00

Josh Siviour

7bbd99c792

feat(eks): multinode control-plane + IMDS/token-webhook CP fixes (426/427/428/495/496) (#459 )

* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity

Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.

EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
  (apiserver-network-proxy): CP servers run konnectivity-server behind the
  cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
  traffic rides the agent-initiated reverse tunnel instead of an unroutable
  CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
  replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
  flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
  tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
  workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.

EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
  clobber it; route mounted-volume state updates through the live VB with a
  retry for in-use marks racing the mount responder; keep the boot volume
  in-use across stop/crash unmount.

* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM

missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.

Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.

* fix(eks): let eks-token-webhook use IMDS creds when static keys absent

The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.

Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.

* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane

The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.

Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.

* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role

The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.

Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.

2026-07-09 09:43:30 +10:00

3 Commits