mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 19:01:42 +00:00
Josh Siviour
f9c08130ba
* fix: switch EKS datastore to embedded etcd + fix Alpine cloud-init IMDS route
Flip the K3s control-plane datastore from SQLite/kine back to embedded etcd
(cluster-init: true, etcd-expose-metrics: true). The earlier abandonment
premise — that etcd's per-commit fsync stalls past the apiserver's 5s deadline
on the viperblock-backed root volume — was wrong: etcd inits fine on an
unchanged viperblock (apiserver ready 25-30s, clusters reach ACTIVE). The real
blocker was a network-config regression, fixed here.
Alpine's cloud-init eni renderer crashes (KeyError: 'gateway') on the
gateway-less IMDS on-link route, which failed the whole network-config render
so the static mgmt NIC never came up (no NATS bootstrap, no /healthz). Fixes:
- generateNetworkConfig: add emitIMDSRoute bool; selectNetworkConfigForFamily
passes false for Alpine so the eni renderer never sees the route.
- Deliver the IMDS route out-of-band via a persistent /etc/local.d script +
OpenRC local service. Resolve the VPC NIC via its default route at runtime
(ip route show default), not a hardcoded name: the persistent-net rename to
vpc0 doesn't apply on the Alpine AMI, so the NIC is eth0 at first boot.
- EKS path bakes the same route into buildK3sUserData's own write_files/runcmd;
the generic template Alpine block fires only when no #cloud-config user-data,
avoiding a duplicate write_files key (yaml.safe_load keeps last, silently
dropping a block).
- eks-node image: config.yaml.skel adds cluster-init; comments updated.
* fix: surface CreateCluster step failures at ERROR + map subnet faults; add auth-mode/delete test cases
CreateCluster wrapped its step failures in bare fmt.Errorf that the request
surface collapses to an opaque ServerInternal ("contact AWS re:Post"), so the
real cause never reached the operator. Add logCreateErr to log each step's
precise cause at ERROR before returning, and map a subnet-resolve failure
(caller-supplied parameter) to InvalidParameterValue instead of a retryable
ServerInternal the SDK pointlessly retries. (mulga-siv-165.4)
Tests (mulga-siv-165.15): reject authenticationMode=API_AND_CONFIG_MAP (only
the CONFIG_MAP value was covered); DeleteCluster on a nonexistent cluster via
the full-deps path returns ResourceNotFound (the shim path short-circuits to
ServiceUnavailable before the meta lookup).
* fix: hide EKS control-plane VMs from customer DescribeInstances
EKS K3s control-plane VMs are owned by the customer account (their ENI
lives in the customer VPC for the NLB target), so unlike LB microVMs
(owned by the system account) they leaked into customer
ec2:DescribeInstances. The spinifex:managed-by tag only filtered the UI
Nodes page, not the EC2 API.
Add a caller-aware exclusion: skip any ManagedBy-tagged VM unless the
caller is root. Preserves operator visibility, matches LB behaviour, and
leaves future customer-owned worker nodes (no ManagedBy) visible.
* docs: correct stale SQLite datastore comments in eks-node image (now embedded etcd)
109 lines
5.2 KiB
Bash
Executable File
109 lines
5.2 KiB
Bash
Executable File
#!/bin/sh
|
|
set -eu
|
|
|
|
# setup.sh — chroot customisation for the unified eks-node AMI (server + agent).
|
|
#
|
|
# Runs inside an Alpine 3.21 chroot under build-system-image.sh after packages
|
|
# and binaries are installed. Downloads the pinned K3s binary, verifies its
|
|
# SHA256, drops it into /usr/local/bin, sets executable bits on all role init
|
|
# scripts + cron entries, and writes the K3s server config skeleton. The role
|
|
# (server vs agent) is selected per-instance at first boot by eks-node-role.
|
|
#
|
|
# Network access inside the chroot is provided by the host's /etc/resolv.conf
|
|
# (build-system-image.sh copies it in). curl is in APK_PACKAGES.
|
|
|
|
K3S_VERSION="v1.32.5+k3s1"
|
|
K3S_URL_BASE="https://github.com/k3s-io/k3s/releases/download/${K3S_VERSION}"
|
|
|
|
# Pull the upstream signed checksums file and pin the amd64 line. A tampered
|
|
# release replacing both files would still produce a self-consistent download,
|
|
# so the SHA file URL is the trust anchor — anyone forging a Mulga AMI build
|
|
# would need to compromise the k3s-io GitHub release artefacts.
|
|
echo "[eks-node-setup] fetching k3s checksums ${K3S_VERSION}"
|
|
curl -fsSL -o /tmp/k3s-checksums.txt "${K3S_URL_BASE}/sha256sum-amd64.txt"
|
|
K3S_SHA256=$(awk '/[ \t]k3s$/{print $1; exit}' /tmp/k3s-checksums.txt)
|
|
if [ -z "${K3S_SHA256}" ]; then
|
|
echo "[eks-node-setup] could not parse k3s sha256 from upstream checksums"
|
|
cat /tmp/k3s-checksums.txt
|
|
exit 1
|
|
fi
|
|
echo "[eks-node-setup] downloading k3s ${K3S_VERSION} (sha256=${K3S_SHA256})"
|
|
curl -fsSL -o /usr/local/bin/k3s "${K3S_URL_BASE}/k3s"
|
|
echo "${K3S_SHA256} /usr/local/bin/k3s" > /tmp/k3s.sha256
|
|
if ! sha256sum -c /tmp/k3s.sha256; then
|
|
echo "[eks-node-setup] k3s SHA256 mismatch — refusing to bake AMI"
|
|
exit 1
|
|
fi
|
|
chmod 0755 /usr/local/bin/k3s
|
|
ln -sf /usr/local/bin/k3s /usr/local/bin/kubectl
|
|
ln -sf /usr/local/bin/k3s /usr/local/bin/crictl
|
|
ln -sf /usr/local/bin/k3s /usr/local/bin/ctr
|
|
|
|
# nats-cli is not in Alpine main/community — pull the upstream release binary.
|
|
# Used by k3s-first-boot.sh (one-shot publishes) and mulga-eks-state-report.sh
|
|
# (periodic publishes).
|
|
NATS_CLI_VERSION="0.4.0"
|
|
NATS_CLI_URL="https://github.com/nats-io/natscli/releases/download/v${NATS_CLI_VERSION}/nats-${NATS_CLI_VERSION}-linux-amd64.zip"
|
|
echo "[eks-node-setup] downloading nats-cli ${NATS_CLI_VERSION}"
|
|
curl -fsSL -o /tmp/nats-cli.zip "${NATS_CLI_URL}"
|
|
apk add --no-cache unzip
|
|
unzip -q -d /tmp/nats-cli /tmp/nats-cli.zip
|
|
install -m 0755 /tmp/nats-cli/nats-${NATS_CLI_VERSION}-linux-amd64/nats /usr/local/bin/nats
|
|
rm -rf /tmp/nats-cli /tmp/nats-cli.zip
|
|
apk del --no-cache unzip
|
|
|
|
# Init scripts ship as 0644 from INSTALL_FILES; OpenRC requires 0755. Every
|
|
# role's services are baked; the selector enables the right ones at first boot.
|
|
chmod 0755 /etc/init.d/eks-node-role /etc/init.d/k3s /etc/init.d/k3s-agent \
|
|
/etc/init.d/eks-token-webhook /etc/init.d/k3s-first-boot
|
|
chmod 0755 /usr/local/sbin/eks-node-role /usr/local/sbin/k3s-first-boot
|
|
chmod 0755 /etc/periodic/15min/mulga-eks-state-report
|
|
chmod 0755 /etc/periodic/daily/mulga-eks-etcd-snapshot
|
|
|
|
# K3s server config — skeleton; cloud-init / first-boot fills in the
|
|
# per-cluster fields (cluster-cidr, service-cidr, token-file, etc). Agents
|
|
# ignore this file (they run `k3s agent`).
|
|
# IAM token-auth is wired here via kube-apiserver-arg: the eks-token-webhook
|
|
# service (ordered `before k3s`) writes its kubeconfig to
|
|
# /etc/spinifex-eks/token-webhook.kubeconfig before the apiserver reads it.
|
|
# This only affects bearer-token requests; anonymous and client-cert paths
|
|
# (the first-boot /readyz probe uses the admin kubeconfig) are unaffected.
|
|
mkdir -p /etc/rancher/k3s
|
|
cat > /etc/rancher/k3s/config.yaml.skel <<'EOF'
|
|
# Populated at first boot by cloud-init user-data via k3s-first-boot.sh.
|
|
# Fields documented at https://docs.k3s.io/installation/configuration.
|
|
# cluster-init selects the embedded etcd datastore. cloud-init write_files
|
|
# normally overrides this skeleton; kept consistent as the boot fallback.
|
|
cluster-init: true
|
|
disable:
|
|
- traefik
|
|
- servicelb
|
|
- local-storage
|
|
kube-apiserver-arg:
|
|
- "authentication-token-webhook-config-file=/etc/spinifex-eks/token-webhook.kubeconfig"
|
|
- "authentication-token-webhook-cache-ttl=5m"
|
|
EOF
|
|
|
|
# Shared state dir for both roles (agent.env / role marker live here).
|
|
mkdir -p /etc/spinifex-eks
|
|
|
|
# Sentinel file marker — k3s-first-boot (server role) self-disables after first
|
|
# success by checking for this path. Initial state is "not yet run".
|
|
touch /var/lib/spinifex-eks/first-boot.pending 2>/dev/null || {
|
|
mkdir -p /var/lib/spinifex-eks
|
|
touch /var/lib/spinifex-eks/first-boot.pending
|
|
}
|
|
|
|
# Bind /dev/console to the serial port so userspace boot output — OpenRC
|
|
# service starts, cloud-init, role selection, and k3s-first-boot diagnostics —
|
|
# reaches ttyS0, which the orchestrator captures to a host-side log. The stock
|
|
# Alpine cloud image lists `console=tty0` LAST in default_kernel_opts; Linux
|
|
# makes the last console= the controlling /dev/console, so userspace logs to
|
|
# the framebuffer and the serial capture sees only kernel dmesg. Reorder so
|
|
# ttyS0 is last in both the generator config and the rendered extlinux.conf.
|
|
sed -i \
|
|
's|console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0|console=tty0 console=ttyAMA0,115200n8 console=ttyS0,115200n8|' \
|
|
/etc/update-extlinux.conf /boot/extlinux.conf
|
|
|
|
echo "[eks-node-setup] done"
|