mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 10:52:04 +00:00
b23fa3bf60
* feat(eks): VM-side managed-addon delivery transport
Stage CreateAddon manifests in KV host-side; the control-plane VM pulls the
staged set through the AWS gateway (signed GET /clusters/{name}/internal-addons/
{accountId}), renders the baked bundles into the K3s auto-deploy dir, and
reports per-addon delivery status on a dedicated eks.addon.*.status channel that
the cluster reconciler CASes onto AWS-visible AddonStatus (CREATING->ACTIVE).
- gateway: ListInternalAddons route + ListStagedAddonManifests NATS action
- handlers: addon status reconciler (nextAddonStatus), staging installer
- eks-node AMI: eks-gateway-fetch binary + mulga-eks-addon-sync agent
- agent renders idempotently (cmp before replace) so K3s auto-deploy does not
re-apply every tick and clear the Addon CR .status.gvks readiness signal
- agent mirrors per-tick delivery outcomes to the serial console for host-side
observability
* fix(eks): provision cluster NLB synchronously so the endpoint carries the real front-end IP
The cluster NLB front-end IP is assigned by the async LB-VM launch
(launchLBVMAsync), but EnsureClusterNLB read it back synchronously right after
CreateLoadBalancer returned — losing the race and falling back to the synthetic
<lb>.elb.spinifex.local DNS name. That name is unresolvable (no Route53) and is
not in the apiserver serving-cert SAN, so kubectl against the cluster never
connects.
Split the elbv2 create: the public CreateLoadBalancer stays async (267.4 — an
inline multi-minute VM boot head-of-line-blocks the gateway responder), and a
new CreateLoadBalancerSync drives the data-plane launch inline and returns the
LB carrying its front-end address. The EKS cluster-create worker runs off the
gateway responder, so it uses the sync entry and reads the IP from the return
value instead of racing persisted state. No polling.
- elbv2: factor provisionLBDataPlane out of launchLBVMAsync; add CreateLoadBalancerSync
- eks: ensureClusterLB uses the sync entry, sets FrontendIP from the result,
drops the read-back; EnsureClusterNLB fails loud (ErrClusterNLBFrontendIPUnavailable)
when no front-end IP is allocated rather than shipping the DNS-name fallback
- eks: drop the DNS-name endpoint fallback; the endpoint is always the front-end IP
* feat: auto-wire IGW for internet-facing EKS cluster endpoint
An internet-facing cluster NLB allocates an external front-end IP and the
OVN dnat_and_snat for it, but the external switch + localnet + gateway LRP
that make that IP answerable on the wire are built only on IGW attach. The
control-plane NLB is a system instance, so ensure an IGW on cluster build
and reclaim it on teardown.
EnsureClusterIGW (idempotent, reuses a customer IGW untouched) runs before
EnsureClusterNLB when the endpoint is public; DeleteClusterIGW (ownership-
scoped) removes only the cluster-owned IGW in purgeClusterInfra, before the
VPC delete. New IGW dep wired from the daemon.
* fix(eks): tear down infra on failed cluster launch to stop LB EIP leak
A CreateCluster that fails during its background launch flipped the meta
to FAILED but performed no teardown, relying on a later same-name create
to reclaim and purge the infra. EKS cluster names are unique per create,
so that reclaim never fires: the internet-facing load-balancer VM keeps
running with its allocated+associated dnat_and_snat EIP indefinitely.
Over repeated create/delete cycles the EIP pool fills with allocations
bound to terminated instances, so a later allocation's advertised
endpoint and its live OVN NAT diverge onto different pool addresses.
- failClusterLaunch now runs a best-effort purgeClusterInfra on the
in-memory meta (freshest refs even when the failing step was the meta
write itself) before MarkClusterFailed. purgeClusterInfra is
idempotent, so a later reclaim or DeleteCluster re-purge is a no-op;
the FAILED meta is retained for observability.
- Daemon releaseSystemInstanceEIP extracted and called from both
TerminateSystemInstance and cleanupFailedSystemInstance, so a launch
that fails after AssociateAddress releases the eipService allocation
instead of leaking it (MarkFailed's ReleasePublicIP only knows the
externalIPAM path).
* feat: EKS managed control-plane VPC (AWS Set B) composed from EC2 APIs
CreateCluster now auto-builds the managed control-plane VPC under the
system account, the spinifex analogue of AWS EKS's hidden managed-account
VPC: a public subnet for the internet-facing NLB, private subnet(s) for the
k3s control-plane VM(s), and a NAT gateway giving the private CP egress for
image pulls. It is composed from the real EC2 VPC-family APIs
(CreateVpc/Subnet, IGW, RouteTable/Route, NatGateway), so per-subnet egress
gating is wired by the existing topology subscribers off the route-table
events those APIs publish — no direct OVN mutation.
This resolves the mulga-siv-292 root cause: the internet-facing NLB and the
private CP were crammed into one customer subnet with no IGW route, so both
were treated as private and dropped by the lr_in_policy 1100 rule, killing
the LB-VM return leg and lb-agent heartbeat. The CP only survived via a
per-instance 1200 reroute bandaid, now removed — the topology expresses
egress instead.
Placement is uniform single/multi-node (CP always in the private subnet).
The system LB VM heartbeats over the mgmt bridge (SystemGatewayURL) rather
than the WAN. Managed-VPC teardown reclaims NAT GW + EIP, route tables,
subnets, IGW, and the VPC in dependency order. Customer subnetIds (Set A /
worker side) remain consumed-not-created and are deferred to the worker
round.
* fix(eks): bake cluster-owner account into CP-VM EKS_ACCOUNT_ID
The managed control-plane VPC (Set B) launches the K3s control-plane VMs and
their ENIs under the system (infra) account, but the on-VM agents — bootstrap
publish, state report, token-review, and add-on fetch — must key on the
cluster-OWNER (customer) account that owns the cluster meta, namespaces, and
AccessEntry KV. Add ClusterAccountID to K3sServerInput and bake it into the
VM's EKS_ACCOUNT_ID, decoupling it from the infra AccountID the VM launches
under. Without this the VM reported state and fetched add-ons against the
system account, so the cluster never observed a healthy control plane (stuck
CREATING, null CA) and kubectl/add-on delivery timed out.
Also refresh the e2e suite doc comment to describe the Set B topology and the
data-plane subtests (kubectl + managed-addon delivery) that now run.
* fix(eks): explicitly delete addon objects on unstage
K3s does not delete the applied resources when an auto-deploy manifest file is
removed from the server manifests dir (documented behaviour). The addon-sync
GC pass relied on that, so after DeleteAddon the rendered manifest was dropped
but its Namespace/ConfigMap stayed standing — the e2e AddonDelivery GC wait
timed out. Copy the manifest aside, drop the original so K3s stops tracking it,
then kubectl delete the objects (--wait=false to avoid finalizer stalls).
98 lines
4.7 KiB
Bash
Executable File
98 lines
4.7 KiB
Bash
Executable File
#!/bin/sh
|
|
set -eu
|
|
|
|
# setup.sh — guest customisation for the unified eks-node AMI (server + agent).
|
|
#
|
|
# Runs inside the libguestfs appliance (via virt-customize --run) under
|
|
# build-system-image.sh after packages and binaries are installed. Downloads the
|
|
# pinned K3s binary, verifies its SHA256, drops it into /usr/local/bin, sets
|
|
# executable bits on all role init scripts + cron entries, and writes the K3s
|
|
# server config skeleton. The role (server vs agent) is selected per-instance at
|
|
# first boot by eks-node-role.
|
|
#
|
|
# Network access inside the appliance is provided by virt-customize --network.
|
|
# curl is in APK_PACKAGES.
|
|
|
|
K3S_VERSION="v1.32.5+k3s1"
|
|
K3S_URL_BASE="https://github.com/k3s-io/k3s/releases/download/${K3S_VERSION}"
|
|
|
|
# Pull the upstream signed checksums file and pin the amd64 line. A tampered
|
|
# release replacing both files would still produce a self-consistent download,
|
|
# so the SHA file URL is the trust anchor — anyone forging a Mulga AMI build
|
|
# would need to compromise the k3s-io GitHub release artefacts.
|
|
echo "[eks-node-setup] fetching k3s checksums ${K3S_VERSION}"
|
|
curl -fsSL -o /tmp/k3s-checksums.txt "${K3S_URL_BASE}/sha256sum-amd64.txt"
|
|
K3S_SHA256=$(awk '/[ \t]k3s$/{print $1; exit}' /tmp/k3s-checksums.txt)
|
|
if [ -z "${K3S_SHA256}" ]; then
|
|
echo "[eks-node-setup] could not parse k3s sha256 from upstream checksums"
|
|
cat /tmp/k3s-checksums.txt
|
|
exit 1
|
|
fi
|
|
echo "[eks-node-setup] downloading k3s ${K3S_VERSION} (sha256=${K3S_SHA256})"
|
|
curl -fsSL -o /usr/local/bin/k3s "${K3S_URL_BASE}/k3s"
|
|
echo "${K3S_SHA256} /usr/local/bin/k3s" > /tmp/k3s.sha256
|
|
if ! sha256sum -c /tmp/k3s.sha256; then
|
|
echo "[eks-node-setup] k3s SHA256 mismatch — refusing to bake AMI"
|
|
exit 1
|
|
fi
|
|
chmod 0755 /usr/local/bin/k3s
|
|
ln -sf /usr/local/bin/k3s /usr/local/bin/kubectl
|
|
ln -sf /usr/local/bin/k3s /usr/local/bin/crictl
|
|
ln -sf /usr/local/bin/k3s /usr/local/bin/ctr
|
|
|
|
# Init scripts ship as 0644 from INSTALL_FILES; OpenRC requires 0755. Every
|
|
# role's services are baked; the selector enables the right ones at first boot.
|
|
chmod 0755 /etc/init.d/eks-node-role /etc/init.d/k3s /etc/init.d/k3s-agent \
|
|
/etc/init.d/eks-token-webhook /etc/init.d/k3s-first-boot /etc/init.d/mulga-eks-state-report \
|
|
/etc/init.d/mulga-eks-addon-sync
|
|
chmod 0755 /usr/local/sbin/eks-node-role /usr/local/sbin/k3s-first-boot \
|
|
/usr/local/sbin/mulga-eks-state-report /usr/local/sbin/mulga-eks-addon-sync
|
|
chmod 0755 /etc/periodic/daily/mulga-eks-etcd-snapshot
|
|
|
|
# K3s server config — skeleton; cloud-init / first-boot fills in the
|
|
# per-cluster fields (cluster-cidr, service-cidr, token-file, etc). Agents
|
|
# ignore this file (they run `k3s agent`).
|
|
# IAM token-auth is wired here via kube-apiserver-arg: the eks-token-webhook
|
|
# service (ordered `before k3s`) writes its kubeconfig to
|
|
# /etc/spinifex-eks/token-webhook.kubeconfig before the apiserver reads it.
|
|
# This only affects bearer-token requests; anonymous and client-cert paths
|
|
# (the first-boot /readyz probe uses the admin kubeconfig) are unaffected.
|
|
mkdir -p /etc/rancher/k3s
|
|
cat > /etc/rancher/k3s/config.yaml.skel <<'EOF'
|
|
# Populated at first boot by cloud-init user-data via k3s-first-boot.sh.
|
|
# Fields documented at https://docs.k3s.io/installation/configuration.
|
|
# cluster-init selects the embedded etcd datastore. cloud-init write_files
|
|
# normally overrides this skeleton; kept consistent as the boot fallback.
|
|
cluster-init: true
|
|
disable:
|
|
- traefik
|
|
- servicelb
|
|
- local-storage
|
|
kube-apiserver-arg:
|
|
- "authentication-token-webhook-config-file=/etc/spinifex-eks/token-webhook.kubeconfig"
|
|
- "authentication-token-webhook-cache-ttl=5m"
|
|
EOF
|
|
|
|
# Shared state dir for both roles (agent.env / role marker live here).
|
|
mkdir -p /etc/spinifex-eks
|
|
|
|
# Sentinel file marker — k3s-first-boot (server role) self-disables after first
|
|
# success by checking for this path. Initial state is "not yet run".
|
|
touch /var/lib/spinifex-eks/first-boot.pending 2>/dev/null || {
|
|
mkdir -p /var/lib/spinifex-eks
|
|
touch /var/lib/spinifex-eks/first-boot.pending
|
|
}
|
|
|
|
# Bind /dev/console to the serial port so userspace boot output — OpenRC
|
|
# service starts, cloud-init, role selection, and k3s-first-boot diagnostics —
|
|
# reaches ttyS0, which the orchestrator captures to a host-side log. The stock
|
|
# Alpine cloud image lists `console=tty0` LAST in default_kernel_opts; Linux
|
|
# makes the last console= the controlling /dev/console, so userspace logs to
|
|
# the framebuffer and the serial capture sees only kernel dmesg. Reorder so
|
|
# ttyS0 is last in both the generator config and the rendered extlinux.conf.
|
|
sed -i \
|
|
's|console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0|console=tty0 console=ttyAMA0,115200n8 console=ttyS0,115200n8|' \
|
|
/etc/update-extlinux.conf /boot/extlinux.conf
|
|
|
|
echo "[eks-node-setup] done"
|