2 Commits

Author	SHA1	Message	Date
Josh Siviour	496424c41c	test(eks): EKS control-plane E2E suite (Sprint 6g) (#304) ... * test(eks): EKS control-plane e2e suite (Phase 1, API-level) Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw: CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone. Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted} pollers. Wire e2e-eks into the Makefile and the single-node CI suite list. * test(eks): Phase 2 kubectl get-nodes data-plane gate Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper (harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the control-plane node is Ready — the get-token webhook auth regression gate. IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up. * test(eks): tolerate doubled NotFound code + dial loopback NATS DeleteCluster detection matched the SDK constant ResourceNotFoundException, but awsgw emits ResourceNotFoundExceptionException on the wire — match the NotFound code by substring so deletion is detected immediately instead of timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which the serving cert does not SAN; rewrite the wildcard bind to loopback. Validated live: all 6 subtests green, kubectl get nodes shows the control-plane node Ready via the get-token webhook auth path. * feat: build system images with libguestfs (no host block devices) Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in build-system-image.sh with libguestfs (virt-customize / virt-resize / guestfish). Every operation runs in an isolated userspace appliance VM with its own kernel, so the build touches no host block device, no host mount, and needs no sudo — removing the failure class that wiped the host's /dev on shared CI runners (a leaked /dev bind traversed by rm -rf). Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu via virt-resize; customization (packages, binaries, files, setup script, service enable, cache cleanup) is one ordered virt-customize invocation. LIBGUESTFS_BACKEND=direct (no libvirt on runners). Wire the eks-node AMI into the single-node e2e job: the runner builds the device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are VM-only) so the eks suite's CreateCluster has an AMI to boot. * feat(eks): bootstrap + state-report via AWS gateway broker Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model: the K3s server VM publishes its bootstrap envelopes and periodic state reports via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS directly. A new gateway action (PublishInternal, POST /clusters/{name}/internal-publish) relays the payload verbatim onto the existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV persistence/encryption, and reconciler are unchanged. This removes the VM's need to reach NATS at all, so NATS can stay cluster-internal behind its bind/firewall/mTLS controls — the prior direct path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was DDIL-fragile (fire-and-forget nats pub, no ack/retry). - gateway/eks: PublishInternal broker + route; closed-set channel/kind validation - cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and mulga-eks-state-report.sh - handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL + system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA; new resolveSystemGatewayBaseURL (mgmt-bridge-first) - eks-node image: build/install eks-gateway-publish, drop nats-cli download * feat: migrate eks-token-webhook to the AWS gateway broker The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that fails, so the webhook exited at startup, the apiserver token-webhook kubeconfig was never written, and the apiserver crash-looped — the cluster never reached ACTIVE. Converge it onto the same gateway-broker model as bootstrap/state-report and ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to /clusters/{name}/token-review, and the gateway runs the verify + lookup host-side and returns the resolved identity. The VM now speaks no core NATS. - internal/eksgw: shared on-VM SigV4 client (also adopted by eks-gateway-publish). - handlers/eks/token_review.go: Authenticate decision core (moved from the webhook binary) + ResolveTokenReview host-side wiring. - gateway/eks token-review route + WebhookTokenReview handler. - webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable) not a silent deny. - initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*. * fix: build eks-node helpers with GOFIPS140; cover token-review broker eks-gateway-publish blank-imports internal/fipsboot, whose init() panics unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build omitted the flag, so the binary paniced at first boot right after k3s came up — k3s-first-boot (set -eu) died before publishing the bootstrap token and the cluster never reached ACTIVE. Build both eks-node helpers with GOFIPS140=v1.0.0, matching cmd/lb-agent. Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview gateway handler (verify responder + account KV) to cover the new broker path. * fix: export first-boot env so eks-gateway-publish child sees it k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/ first-boot.env but never exported the vars, so the eks-gateway-publish child process inherited an empty environment and exited '--account-id is required'. The control-plane VM came up healthy but never published its bootstrap envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a / set +a so the KEY=value lines export to child processes. * fix: make awsgw start --host override actually bind The 'awsgw start --host' override printed 'Overwriting awsgw host' but the assignment was commented out, so AWSGW always bound the spinifex.toml host (a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW listens on all interfaces incl. br-mgmt; the serving cert already SANs every local interface IP, and WAN clients still reach it. * test(eks): install kubectl on e2e VM so KubectlGetNodes runs The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker -> STS verify path — the only end-to-end exercise of the token-review broker. The e2e binaries run on the VM, where kubectl was absent, so harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a checksum-verified kubectl (pinned to the k3s server version) on the VM, gated on e2e-eks.	2026-06-07 20:42:25 +10:00
Josh Siviour	bfacca7789	feat(eks): 6c token webhook + AccessEntry (IAM auth → kubectl) (#284) ... * feat(eks): 6c AccessEntry CRUD + access-policy association Replace the nine notImpl() AccessEntry/AccessPolicy stubs with real KV-backed handlers (Q9, API auth mode): - AccessEntry CRUD persisted at clusters/{name}/access-entries/{hash}; the principal ARN is hashed into the key (ARNs carry ':', illegal in NATS KV keys) while the record keeps the plaintext ARN. - AssociateAccessPolicy/Disassociate/ListAssociated against the four supported AWS-managed policies; unsupported policy ARN or bad access scope -> InvalidParameterValue. - ListAccessPolicies returns the static supported catalogue. - Cluster-existence gate -> ResourceNotFoundException; duplicate create -> ResourceInUseException; node-type entries rejected until 6d. Updates the daemon routing test for the now-implemented handlers. * feat(eks): plumb caller principal ARN + bootstrap-creator-admin Thread the caller's resolved IAM principal ARN from the gateway to the EKS control plane via a new X-Principal-ARN NATS header, so CreateCluster can mint the bootstrapClusterCreatorAdminPermissions AccessEntry (Q9): - utils.NATSRequest takes optional extra headers; PrincipalARNHeader + PrincipalARNFromMsg added. - Gateway resolves the caller ARN best-effort (buildCallerARN) and passes it down the EKS route table; only CreateCluster consumes it. - CreateCluster signature gains callerPrincipalARN; default-true bootstrap flag mints a system:masters entry keyed by the caller's exact ARN (the token webhook resolves the same ARN from STS). '' degrades gracefully. - Dedicated daemon handler reads both headers. * feat(eks): IAM token webhook + STS verify-over-NATS + apiserver wiring The eks-token-webhook (baked into the eks-server AMI) authenticates kubectl requests that present an `aws eks get-token` bearer token: decode the presigned STS GetCallerIdentity URL, verify it, resolve the principal ARN to its cluster AccessEntry, and answer the apiserver TokenReview with the mapped username + groups. STS is hosted in-process in the awsgw service and is not otherwise on the bus, so awsgw now serves an `eks.VerifyToken` NATS subject that wraps VerifyPresignedGetCallerIdentity (the cluster name is forwarded so STS enforces the x-k8s-aws-id anti-replay pin). The webhook reads the AccessEntry KV directly over its own NATS connection. Apiserver wiring: the k3s config skeleton sets authentication-token-webhook-config-file; the webhook writes its own self-signed serving cert + the apiserver kubeconfig at startup (ordered `before k3s`, with a bounded wait in the k3s start_pre so the apiserver never loads a missing file). * fix(eks): omit empty spec from token webhook TokenReview response * fix(sts): presigned GetCallerIdentity must hash empty body, not UNSIGNED-PAYLOAD VerifyPresignedGetCallerIdentity built the SigV4 canonical request with "UNSIGNED-PAYLOAD" as the payload hash. That is an S3-only convention; botocore and the aws-sdk presign non-S3 requests (sts:GetCallerIdentity, the shape `aws eks get-token` produces) with the SHA256 of the empty string. Every real get-token therefore failed signature verification with InvalidIdentityToken, breaking the entire EKS IAM token-auth flow end to end. Use the empty-string SHA256. The existing hand-rolled presign tests masked the bug by signing with the same wrong constant; they now sign correctly, and a new test presigns with the real aws-sdk-go v4 signer (the same canonicalisation the CLI uses) so this can't regress silently. Verified live: a real `aws eks get-token --cluster-name dev1` now resolves to the caller principal over the awsgw eks.VerifyToken surface; an otherwise-valid token presented for a different cluster is still rejected (anti-replay intact). * fix(eks): match access-entry routes on escaped path so slashed principal ARNs resolve IAM principal ARNs (user/admin, role/foo) embed a '/' that the AWS CLI percent-encodes in the request path. The dispatcher matched r.URL.Path (decoded), so %2F became a literal '/' and split the ARN across the ([^/]+) capture, dropping every describe/update/delete-access-entry and associate/disassociate/list-access-policy call to InvalidActionException. Match r.URL.EscapedPath() so the encoded ARN stays one segment, then url.PathUnescape each captured param before it reaches the handler. * test(eks): unit-cover access-entry scope validation, AWS converters, store guards	2026-06-04 11:25:56 +10:00