mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-23 04:36:40 +00:00
Josh Siviour
7bbd99c792
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity
Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.
EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
(apiserver-network-proxy): CP servers run konnectivity-server behind the
cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
traffic rides the agent-initiated reverse tunnel instead of an unroutable
CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.
EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
clobber it; route mounted-volume state updates through the live VB with a
retry for in-use marks racing the mount responder; keep the boot volume
in-use across stop/crash unmount.
* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM
missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.
Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.
* fix(eks): let eks-token-webhook use IMDS creds when static keys absent
The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.
Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.
* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane
The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.
Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.
* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role
The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.
Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
90 lines
3.3 KiB
YAML
90 lines
3.3 KiB
YAML
# Konnectivity agent DaemonSet — the worker half of the apiserver egress proxy.
|
|
# Each agent dials OUT to the konnectivity-server (fronted by the cluster NLB at
|
|
# __KONN_HOST__:8132) and, via --sync-forever, opens a reverse tunnel to EVERY
|
|
# apiserver replica (it learns the replica count from the server handshake).
|
|
# apiserver→pod/kubelet/webhook egress rides back down that tunnel, so an HA
|
|
# control plane reaches worker pod IPs with no L4-VIP fan-out gap. Staged into
|
|
# the K3s auto-deploy dir by k3s.initd, which fills the __KONN_HOST__ /
|
|
# __KONN_AGENT_IMAGE__ tokens at boot.
|
|
#
|
|
# Runs on workers only: it carries no toleration for the control-plane
|
|
# CriticalAddonsOnly=true:NoExecute taint, so it is never scheduled onto the CP
|
|
# nodes (which run the server, not an agent).
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: konnectivity-agent
|
|
namespace: kube-system
|
|
labels:
|
|
app.kubernetes.io/managed-by: spinifex-eks
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: DaemonSet
|
|
metadata:
|
|
name: konnectivity-agent
|
|
namespace: kube-system
|
|
labels:
|
|
k8s-app: konnectivity-agent
|
|
app.kubernetes.io/managed-by: spinifex-eks
|
|
spec:
|
|
selector:
|
|
matchLabels:
|
|
k8s-app: konnectivity-agent
|
|
updateStrategy:
|
|
type: RollingUpdate
|
|
template:
|
|
metadata:
|
|
labels:
|
|
k8s-app: konnectivity-agent
|
|
spec:
|
|
serviceAccountName: konnectivity-agent
|
|
priorityClassName: system-node-critical
|
|
# hostNetwork so the agent reaches the NLB over the node network and
|
|
# identifies itself to the server by the node's own IP (the address the
|
|
# apiserver dials back through for kubelet/pod egress).
|
|
hostNetwork: true
|
|
dnsPolicy: ClusterFirstWithHostNet
|
|
containers:
|
|
- name: konnectivity-agent
|
|
image: __KONN_AGENT_IMAGE__
|
|
command:
|
|
- /proxy-agent
|
|
args:
|
|
- --logtostderr=true
|
|
- --ca-cert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
|
- --proxy-server-host=__KONN_HOST__
|
|
- --proxy-server-port=8132
|
|
# proxy-agent v0.30.3 has no server-count flag — it learns the
|
|
# replica count from the server handshake and keeps re-syncing
|
|
# (sync-forever) until it holds a tunnel to every apiserver, which
|
|
# is what makes the HA control plane reachable.
|
|
- --sync-forever=true
|
|
- --service-account-token-path=/var/run/secrets/tokens/konnectivity-agent-token
|
|
- --agent-identifiers=ipv4=$(HOST_IP)
|
|
- --admin-server-port=8133
|
|
- --health-server-port=8134
|
|
env:
|
|
- name: HOST_IP
|
|
valueFrom:
|
|
fieldRef:
|
|
fieldPath: status.hostIP
|
|
livenessProbe:
|
|
httpGet:
|
|
scheme: HTTP
|
|
host: 127.0.0.1
|
|
port: 8134
|
|
path: /healthz
|
|
initialDelaySeconds: 15
|
|
timeoutSeconds: 15
|
|
volumeMounts:
|
|
- name: konnectivity-agent-token
|
|
mountPath: /var/run/secrets/tokens
|
|
readOnly: true
|
|
volumes:
|
|
- name: konnectivity-agent-token
|
|
projected:
|
|
sources:
|
|
- serviceAccountToken:
|
|
path: konnectivity-agent-token
|
|
audience: system:konnectivity-server
|