mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-21 19:56:33 +00:00
test(eks): EKS control-plane E2E suite (Sprint 6g) (#304)
* test(eks): EKS control-plane e2e suite (Phase 1, API-level)
Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw:
CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact
assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone.
Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted}
pollers. Wire e2e-eks into the Makefile and the single-node CI suite list.
* test(eks): Phase 2 kubectl get-nodes data-plane gate
Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane
VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper
(harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points
kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB
DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the
control-plane node is Ready — the get-token webhook auth regression gate.
IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up.
* test(eks): tolerate doubled NotFound code + dial loopback NATS
DeleteCluster detection matched the SDK constant ResourceNotFoundException,
but awsgw emits ResourceNotFoundExceptionException on the wire — match the
NotFound code by substring so deletion is detected immediately instead of
timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which
the serving cert does not SAN; rewrite the wildcard bind to loopback.
Validated live: all 6 subtests green, kubectl get nodes shows the
control-plane node Ready via the get-token webhook auth path.
* feat: build system images with libguestfs (no host block devices)
Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in
build-system-image.sh with libguestfs (virt-customize / virt-resize /
guestfish). Every operation runs in an isolated userspace appliance VM with
its own kernel, so the build touches no host block device, no host mount, and
needs no sudo — removing the failure class that wiped the host's /dev on shared
CI runners (a leaked /dev bind traversed by rm -rf).
Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu
via virt-resize; customization (packages, binaries, files, setup script,
service enable, cache cleanup) is one ordered virt-customize invocation.
LIBGUESTFS_BACKEND=direct (no libvirt on runners).
Wire the eks-node AMI into the single-node e2e job: the runner builds the
device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are
VM-only) so the eks suite's CreateCluster has an AMI to boot.
* feat(eks): bootstrap + state-report via AWS gateway broker
Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model:
the K3s server VM publishes its bootstrap envelopes and periodic state reports
via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS
directly. A new gateway action (PublishInternal, POST
/clusters/{name}/internal-publish) relays the payload verbatim onto the
existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV
persistence/encryption, and reconciler are unchanged.
This removes the VM's need to reach NATS at all, so NATS can stay
cluster-internal behind its bind/firewall/mTLS controls — the prior direct
path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was
DDIL-fragile (fire-and-forget nats pub, no ack/retry).
- gateway/eks: PublishInternal broker + route; closed-set channel/kind validation
- cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into
the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and
mulga-eks-state-report.sh
- handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL +
system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA;
new resolveSystemGatewayBaseURL (mgmt-bridge-first)
- eks-node image: build/install eks-gateway-publish, drop nats-cli download
* feat: migrate eks-token-webhook to the AWS gateway broker
The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry
KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that
fails, so the webhook exited at startup, the apiserver token-webhook
kubeconfig was never written, and the apiserver crash-looped — the cluster
never reached ACTIVE.
Converge it onto the same gateway-broker model as bootstrap/state-report and
ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to
/clusters/{name}/token-review, and the gateway runs the verify + lookup
host-side and returns the resolved identity. The VM now speaks no core NATS.
- internal/eksgw: shared on-VM SigV4 client (also adopted by
eks-gateway-publish).
- handlers/eks/token_review.go: Authenticate decision core (moved from the
webhook binary) + ResolveTokenReview host-side wiring.
- gateway/eks token-review route + WebhookTokenReview handler.
- webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable)
not a silent deny.
- initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*.
* fix: build eks-node helpers with GOFIPS140; cover token-review broker
eks-gateway-publish blank-imports internal/fipsboot, whose init() panics
unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build
omitted the flag, so the binary paniced at first boot right after k3s came
up — k3s-first-boot (set -eu) died before publishing the bootstrap token and
the cluster never reached ACTIVE. Build both eks-node helpers with
GOFIPS140=v1.0.0, matching cmd/lb-agent.
Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview
gateway handler (verify responder + account KV) to cover the new broker path.
* fix: export first-boot env so eks-gateway-publish child sees it
k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/
first-boot.env but never exported the vars, so the eks-gateway-publish child
process inherited an empty environment and exited '--account-id is required'.
The control-plane VM came up healthy but never published its bootstrap
envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a /
set +a so the KEY=value lines export to child processes.
* fix: make awsgw start --host override actually bind
The 'awsgw start --host' override printed 'Overwriting awsgw host' but the
assignment was commented out, so AWSGW always bound the spinifex.toml host
(a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control
plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with
no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the
cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW
listens on all interfaces incl. br-mgmt; the serving cert already SANs every
local interface IP, and WAN clients still reach it.
* test(eks): install kubectl on e2e VM so KubectlGetNodes runs
The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker
-> STS verify path — the only end-to-end exercise of the token-review broker.
The e2e binaries run on the VM, where kubectl was absent, so
harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a
checksum-verified kubectl (pinned to the k3s server version) on the VM,
gated on e2e-eks.
This commit is contained in:
Josh Siviour
@@ -18,9 +18,9 @@ on:
|
||||
description: "PR head SHA to report status on"
|
||||
required: false
|
||||
suites:
|
||||
description: "Space-separated Make targets to run (e.g. 'e2e-single e2e-multinode e2e-cert e2e-lb')"
|
||||
description: "Space-separated Make targets to run (e.g. 'e2e-single e2e-multinode e2e-cert e2e-lb e2e-eks')"
|
||||
required: false
|
||||
default: "e2e-single e2e-multinode e2e-cert e2e-lb"
|
||||
default: "e2e-single e2e-multinode e2e-cert e2e-lb e2e-eks"
|
||||
test_run:
|
||||
description: "Optional regex for -test.run (e.g. '^TestMultinodeIPSec$'). Empty = run all tests in suite."
|
||||
required: false
|
||||
@@ -32,7 +32,7 @@ permissions:
|
||||
|
||||
env:
|
||||
GIT_BRANCH: ${{ github.ref_name }}
|
||||
SUITES: ${{ inputs.suites || 'e2e-single e2e-multinode e2e-cert e2e-lb' }}
|
||||
SUITES: ${{ inputs.suites || 'e2e-single e2e-multinode e2e-cert e2e-lb e2e-eks' }}
|
||||
TEST_RUN: ${{ inputs.test_run || '' }}
|
||||
|
||||
jobs:
|
||||
@@ -42,7 +42,7 @@ jobs:
|
||||
# suites='e2e-reboot' only). Push triggers and the default SUITES list
|
||||
# both include single/cert/lb so the inputs.suites null-check keeps the
|
||||
# job running on plain PR pushes.
|
||||
if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-lb')
|
||||
if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-lb') || contains(github.event.inputs.suites, 'e2e-eks')
|
||||
runs-on: [ self-hosted, ci-single ]
|
||||
timeout-minutes: 45
|
||||
env:
|
||||
@@ -154,6 +154,82 @@ jobs:
|
||||
echo "wan_ip=${WAN_IP}" >> "$GITHUB_OUTPUT"
|
||||
echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# The eks suite's CreateCluster boots a control-plane VM from the
|
||||
# eks-node AMI, so that AMI must exist in the VM's image store before the
|
||||
# suite runs. build-system-image.sh customizes the image with libguestfs
|
||||
# (an isolated userspace appliance) — it touches no host block device, so
|
||||
# it is safe on the shared runner. The build needs the spinifex source +
|
||||
# network (runner only); the import needs predastore/NATS (VM only), so
|
||||
# build on the runner then ship the raw and import on the VM.
|
||||
- name: Build eks-node AMI (libguestfs, runner)
|
||||
if: contains(env.SUITES, 'e2e-eks')
|
||||
working-directory: spinifex
|
||||
run: |
|
||||
set -euo pipefail
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
banner "build eks-node AMI (libguestfs — no host devices)"
|
||||
sudo apt-get update -qq
|
||||
sudo apt-get install -y --no-install-recommends libguestfs-tools qemu-utils
|
||||
# Debian/Ubuntu ship /boot/vmlinuz-* mode 0600; the libguestfs
|
||||
# appliance needs to read a host kernel. direct backend = no libvirt.
|
||||
export LIBGUESTFS_BACKEND=direct
|
||||
sudo chmod 0644 /boot/vmlinuz-* || true
|
||||
group_open "make build-eks-node-image"
|
||||
make build-eks-node-image
|
||||
group_close
|
||||
ok "raw: $(du -h /tmp/eks-node-image-build/eks-node-alpine.raw | cut -f1)"
|
||||
|
||||
- name: Import eks-node AMI on VM
|
||||
if: contains(env.SUITES, 'e2e-eks')
|
||||
run: |
|
||||
set -euo pipefail
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
# Pull import args straight from the manifest so they never drift.
|
||||
eval "$(grep -E '^(ALPINE_VERSION|AMI_NAME|SYSTEM_TAG)=' \
|
||||
spinifex/scripts/images/eks-node/manifest.conf)"
|
||||
SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
|
||||
VM=tf-user@${{ steps.vm.outputs.wan_ip }}
|
||||
banner "import eks-node AMI to VM image store"
|
||||
scp "${SSH_OPTS[@]}" \
|
||||
/tmp/eks-node-image-build/eks-node-alpine.raw \
|
||||
"$VM":/tmp/eks-node-alpine.raw
|
||||
ssh "${SSH_OPTS[@]}" "$VM" \
|
||||
"spx admin images import --file /tmp/eks-node-alpine.raw \
|
||||
--distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
|
||||
--boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}"
|
||||
ok "eks-node AMI imported"
|
||||
|
||||
# The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
|
||||
# `aws eks get-token` → kubectl → apiserver → token-webhook → gateway
|
||||
# broker → STS verify path. The e2e binaries run ON the VM, so kubectl
|
||||
# must be on the VM's PATH or harness.NewKubectl t.Skips the subtest.
|
||||
# Pin to the k3s server version (setup.sh K3S_VERSION) — kubectl is
|
||||
# ±1-minor skew tolerant; matching is simplest. Checksum-verified.
|
||||
- name: Install kubectl on VM
|
||||
if: contains(env.SUITES, 'e2e-eks')
|
||||
run: |
|
||||
set -euo pipefail
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
|
||||
VM=tf-user@${{ steps.vm.outputs.wan_ip }}
|
||||
banner "install kubectl on VM (eks data-plane subtest)"
|
||||
ssh "${SSH_OPTS[@]}" "$VM" 'bash -s' <<'EOF'
|
||||
set -euo pipefail
|
||||
KVER=v1.32.5
|
||||
case "$(uname -m)" in
|
||||
x86_64) ARCH=amd64 ;;
|
||||
aarch64) ARCH=arm64 ;;
|
||||
*) echo "unsupported arch $(uname -m)" >&2; exit 1 ;;
|
||||
esac
|
||||
cd /tmp
|
||||
curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl"
|
||||
curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl.sha256"
|
||||
echo "$(cat kubectl.sha256) kubectl" | sha256sum -c -
|
||||
sudo install -m 0755 kubectl /usr/local/bin/kubectl
|
||||
kubectl version --client
|
||||
EOF
|
||||
ok "kubectl installed on VM"
|
||||
|
||||
- name: Build E2E test binaries on runner
|
||||
working-directory: spinifex/tests/e2e
|
||||
run: |
|
||||
|
||||
Reference in New Issue
Block a user