225 Commits

Author	SHA1	Message	Date
Josh Siviour	da33e574d4	fix: OVN RAFT multi-node bring-up (northd member list, ExecStop, add-nat deadline) (#466) ... * fix: repoint DB-node ovn-northd at RAFT member list; extend add-nat deadline Two related fixes for multi-node OVN clusters: - setup-ovn.sh (mulga-siv-504): in the clustered --management branch, point ovn-northd's client NB/SB connections at the full RAFT member list derived from OVN_REMOTE, so the active northd follows the leader instead of dialing its local member only. Without this, northd stops advancing SB_Global.nb_cfg once leadership moves off the node, wedging the ovn-nbctl --wait=hv barrier. Guarded on a real member list so the standalone single-DB-node path is unchanged. - utils/nats.go (mulga-js-206): raise the add-nat request deadline from 10s to 45s so the client no longer rolls back before the server-side --wait=hv flows barrier (30s) completes. * fix: stop ovn-northd restart from tearing down split NB/SB ovsdb servers The packaged ovn-northd.service ExecStop runs 'ovn-ctl stop_northd' without --ovn-manage-ovsdb=no, so restarting northd to apply OVN_CTL_OPTS also stops the separately-managed NB/SB ovsdb-server units. On a clean clustered bring-up this races the freshly-started DBs and kills them, so setup-ovn's wait for the NB socket times out and the RAFT create/join aborts. Install an ovn-northd.service drop-in overriding ExecStop with --ovn-manage-ovsdb=no so northd never touches the DB lifecycle (mulga-siv-504).	2026-07-09 09:43:30 +10:00
Josh Siviour	7bbd99c792	feat(eks): multinode control-plane + IMDS/token-webhook CP fixes (426/427/428/495/496) (#459) ... * feat(eks): multinode control-plane networking, IAM, and CoreDNS parity Integrates the EKS multinode fix set for the managed-control-plane topology so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it. EKS: - Route apiserver egress (exec/logs/webhooks) through konnectivity (apiserver-network-proxy): CP servers run konnectivity-server behind the cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod traffic rides the agent-initiated reverse tunnel instead of an unroutable CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every replica (HA-correct). - Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay flannel mesh SG rule authorized at cluster launch with ordered teardown. - Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay. - Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity. - Spread nodegroup workers across hosts; addon readiness falls back on workload rollout when gvks are absent. - elbv2: allow the proxy_protocol_v2.enabled target-group attribute. EBS: - Move volume attachment state to state.json so a live nbdkit VB cannot clobber it; route mounted-volume state updates through the live VB with a retry for in-use marks racing the mount responder; keep the boot volume in-use across stop/crash unmount. * fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM missingOrchestrationDeps gated on s.deps.IAM, but production wires only the lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only). The gate therefore reported missing=[IAM] on every node forever, so CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even though the IAM service was fully up via the provider. Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else deps.IAMProvider()) the nodegroup/role code already uses — so the protective check still rejects a node with no IAM service but passes when the provider supplies one. Add a regression test that sets only IAMProvider and asserts the gate is satisfied; the existing nil-both test still rejects. * fix(eks): let eks-token-webhook use IMDS creds when static keys absent The CP VM launches on an instance profile, so buildK3sUserData omits EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys before eksgw.New could fall back to the IMDS SDK chain, so the webhook aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver crashed on the missing --authentication-token-webhook-config-file file, stalling the cluster at CREATING until it timed out to FAILED. Drop the static-key precondition (matching the sibling eks-gateway-publish and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/ cluster gates. * fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt, which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a 169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing 169.254.169.254 and steering IMDS off the data NIC away from the per-tap br-imds datapath, where the host RST it. eks-gateway-publish then failed to sign requests and no bootstrap artifacts were ever published, so clusters sat CREATING until they timed out to FAILED. Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI) are unaffected and IMDS falls through the data NIC's default route to the br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net deletes any default route on a NIC declared NIC<n>_DEFAULT=0. * fix(eks): grant eks:WebhookTokenReview to the control-plane instance role The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway broker at POST /clusters/{name}/token-review, signing with the control-plane VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes against eks:WebhookTokenReview, but the role's inline policy granted only eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every relay and the apiserver returned 401 to all external kubectl. Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile re-asserts the inline policy on every control-plane instance-profile ensure, and the gateway evaluates authorization per request, so existing clusters converge without a rebuild once the shared role is updated.	2026-07-09 09:43:30 +10:00
Josh Siviour	0823bf8c16	feat: OVN OVSDB RAFT clustering for multinode Spinifex (epic mulga-siv-486) (#458) ... * feat: add OVSDB RAFT cluster primitives to setup-ovn.sh Add --db-cluster-local-addr / --db-cluster-remote-addr flags so a management node can run the OVN NB/SB databases as a native OVSDB RAFT cluster instead of a single non-clustered ovn-central. When a local addr is given, write a shared OVN_CTL_OPTS to /etc/default/ovn-central (NB RAFT 6643 / SB RAFT 6644) and restart the per-DB units directly, since the ovn-central aggregator is ExecStart=/bin/true and won't restart its children. Empty remote addr creates the cluster (init node); a set remote addr joins it. NB/SB client set-connection runs only on the create node (RAFT replicates it cluster-wide). The single-node --management path is unchanged when no cluster addrs are supplied, so dev boxes and existing single-node clusters keep working. --ovn-remote already passes through verbatim, so a comma-separated SB endpoint list works for compute nodes; documented in usage. * feat: OVN client multi-endpoint failover for RAFT clusters Accept a comma-separated OVN NB/SB endpoint list so vpcd can target a clustered DB. config.ParseEndpoints splits the list (trimming whitespace, dropping empties); live.go Connect passes one libovsdb WithEndpoint per endpoint, so the client tries them in order, fails over on disconnect, and follows RAFT leader redirects. Single-endpoint strings still yield one endpoint, so single-node dev is unchanged. ovn-sbctl --db= already accepts the raw comma list, so the SB CLI consumers (discoverChassis, gateway_claim) need no change. * feat: write clustered OVN NB/SB endpoints across the RAFT quorum spx admin init/join now emit comma-separated OVN NB/SB endpoint lists for the first three quorum nodes (formation.BuildOVNDBAddrs) instead of loopback or a single leader address, so vpcd fails over across the cluster. firstboot defers OVN bring-up to the provisioning controller when it owns formation (SkipFormation): a standalone single-node .db would otherwise block the controller's clustered create/join. * test: add multinode OVN RAFT cluster + failover e2e; document clustered topology ovn_raft_test.go asserts NB/SB cluster/status reports a 3-server quorum with one leader on every DB node, then stops the NB leader's ovn-central and proves an NB write still lands via a surviving endpoint. Docs: vpc-networking gains the multi-endpoint vpcd config and a RAFT cluster-status debugging section; network-connections lists the 6643/6644 RAFT ports.	2026-07-09 09:43:29 +10:00
Josh Siviour	2d4f0fa94c	feat: IMDS instance-role creds for lb-agent + EKS server VMs (siv-231.6) (#450) ... * feat: IMDS-capable gateway signer for lb-agent + EKS clients Add internal/gwsign, a SigV4 signer that draws credentials from either fixed static keys or the AWS SDK default credential chain. In a guest VM the chain falls back to the in-VM IMDS endpoint, yielding scoped, auto-rotating instance-role credentials; a session token is signed as X-Amz-Security-Token so the gateway can verify temporary (ASIA) creds. Wire it into the lb-agent (lbagent.Agent) and the EKS on-VM client (internal/eksgw). When static keys are absent the clients now sign with IMDS credentials instead of failing; when present, behaviour is unchanged (static path). This is the client-side foundation for dropping long-lived secrets from VM user-data. Credentials are retrieved per request so rotated IMDS creds take effect without an agent restart. * feat: IMDS instance-role creds for EKS server + LB VMs (Phase B/C) Attach a Spinifex-managed system instance profile to the k3s control-plane VM and the elbv2 LB VM so both authenticate to the gateway with scoped, rotating IMDS instance-role credentials instead of a long-lived static secret baked into user-data. - Add a shared handlers/iam.EnsureSystemInstanceProfile that find-or-creates a role (EC2 trust) plus its instance profile and asserts an inline permission policy. The gateway enforces per-action authz on assumed-role principals, so the roles carry exactly the actions each VM calls: - EKS server: eks:PublishInternal, eks:ListInternalAddons. - LB agent: elasticloadbalancing:LBAgentHeartbeat, GetLBConfig. - Plumb IamInstanceProfileArn through sysinstance.SystemInstanceInput and both daemon launch paths (BootAMI via PrepareRunInstances->RunInstance; direct-boot microVM via RunInstance), which persist it so the in-VM IMDS endpoint serves the role. - Drop the baked secrets when a profile is attached: omit EKS_ACCESS_KEY/ EKS_SECRET_KEY from the k3s first-boot env and LB_ACCESS_KEY/LB_SECRET_KEY from the lb-agent fw_cfg env; relax k3s server validation to accept profile-only; drop the EKS key requires from the first-boot and state-report scripts. - Both verticals fall back to baked static creds when IAM is unwired, so an absent master key never blocks a launch. * fix: resolve IMDS instance-role creds under the instance account for system VMs Three boot-path defects kept the LB VM's lb-agent from authenticating to the gateway with IMDS-served instance-role creds; each produced a silent empty IMDS role list and a heartbeat failing with "unexpected empty EC2 IMDS role list". Found and fixed against env19. - Build the system IAM ensurer lazily on the first system-VM launch instead of eagerly at daemon startup, where it raced the NATS KV backend and fell back to baked static creds permanently. ELBv2 and the EKS deps now hold an IAMProvider resolved at launch time. - Create the LB instance profile in the system account the LB VM runs under (utils.GlobalAccountID), not the LB-owner account. IMDS resolves the profile under the instance's account, so a customer-account role was invisible to it. - Record the attached instance's owning account on the ENI (InstanceOwnerId, mirroring AWS Attachment.InstanceOwnerId) and resolve the instance and every IAM/STS lookup under it. The LB ENI lives in the customer account while the system LB VM and its role live in the system account; keying the whole IMDS chain off the ENI account left the instance invisible and the role lookup missing. VPC leaves (subnet/VPC/SG CIDR) keep using the ENI account. A fresh LB on env19 then authenticated end-to-end: IMDS serves a non-empty role list plus ASIA creds, and the gateway accepts the heartbeat as an assumed-role principal with no static keys in user-data. * test(e2e): assert LB VM authenticates via IMDS instance-role creds The LB suite's targets-healthy assertion already depends on the lb-agent authenticating to the gateway, and with static keys dropped from user-data that auth is IMDS-only. Add an explicit check that a recent gateway authentication for the LB used an assumed-role (ASIA) principal, so a regression to the instance-account IMDS resolution fails with a clear message instead of a mysterious datapath timeout. The LB VM is a system-account instance invisible to the customer AWS API, so the assertion observes the gateway journal on the cluster nodes; it is time-bounded to avoid the stale-console-log false-fail and skips when no node is SSH-reachable.	2026-07-09 09:43:29 +10:00
Josh Siviour	a7241e01f9	feat: run EKS apiserver with anonymous-auth=false (CIS) (#447) ... Flip the k3s server apiserver to anonymous-auth=false. The only consumer of the anonymous path was mulga-eks-state-report's /healthz probe; switch it to an authenticated kubectl get --raw=/healthz through the node admin kubeconfig (an unauthenticated curl now returns 401). The NLB target group uses a TCP health check, and the host-side reconciler rides the authenticated NATS state-report, so neither depends on the anonymous apiserver path.	2026-06-30 09:21:07 +10:00
Josh Siviour	471e95dac2	feat(ecs): service controller + ELBv2 target-group registration (Sprint 4f) (#420) ... * feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs") client the ecs-agent will use to reach the AWS gateway over HTTPS instead of connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal. * feat(ecs): SubmitTaskStateChange gateway action → recordTaskState Wires the AWS-API SubmitTaskStateChange action end to end (gateway action → ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a gateway-routed agent reports task state without touching the NATS bus. The handler maps the SDK input onto the same bus.TaskState shape and converges on recordTaskState; the account is authoritative from the SigV4 caller, so an instance cannot report state for another account's task. * feat(ecs): durable assign KV inbox + PollAssignments gateway action Scheduler writes task assigns into a per-instance KV inbox (clusters/{cluster}/assignments/{instance}/{task}) instead of a fire-and-forget core-NATS publish, and a new PollAssignments gateway action lets a gateway-routed agent drain its inbox over TLS+SigV4. At-least-once: an unacked assign is re-delivered on the next poll; the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions reclaim stale inbox entries. The Poll response carries bus.Assign with an RFC3339 time, so it is encoded with encoding/json (RawJSONActions) rather than the jsonutil epoch-second marshaler. * feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway The agent no longer connects to NATS. It registers, heartbeats, reports task state, and receives assignments entirely through the AWS gateway over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus host-internal and closing the trust-boundary gap (a tenant container instance must not hold a shared bus token). - controlPlane seam over ecsgw.Client replaces the NATS publisher: Register (RegisterContainerInstance), SubmitTaskState (SubmitTaskStateChange), PollAssignments. - Heartbeat folds into a periodic idempotent re-register (refreshes LastSeen); bus.Heartbeat is dropped from the agent. - subscribeAssign (NATS subscribe) becomes a poll loop: drain the inbox, dispatch each assign once, ack accepted taskIDs on the next poll (at-least-once). - config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added, ECS_NATS_URL removed; cloud-init initd seeds the IAM creds. Tests move from the NATS publisher fake to a controlPlane fake. * feat(ecs): cloud-init cred-seed + ecs-node bring-up script ECS container instances are plain EC2 booted from the spinifex-ecs-node AMI (AWS-faithful: no ECS-specific launch API). This renders the cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the instance. The agent registers, heartbeats, polls assignments, and reports task state entirely through the AWS gateway over TLS+SigV4 — never NATS. The guest dials the customer-facing gateway address (network advertise IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL split. Seeds the system key for now; scoped per-instance creds are a follow-up (4g task-role endpoint). * fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring The 'Merge branch dev' resolution dropped code where Sprint 4d (#418, awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch the same paths: - recordTaskState STOPPED branch: reclaimTaskENI was clobbered by reclaimAssignInbox; both are needed (release the task ENI AND drain the KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce. - ecs-agent Agent: the netns field, its init, and the setupTaskNetns/ teardownTaskNetns wrappers were lost when the controlPlane Agent struct won the merge, leaving taskrun.go calling undefined methods. - eni_test assign assertion read the assign over NATS; #419 delivers it via the durable KV inbox, so it now polls via PollAssignments. - taskrun_test netns cases used the removed historyPub fake; switched to fakeCP to match the controlPlane seam. * feat(ecs): service controller + ELBv2 target-group registration (Sprint 4f) Land the ECS Service controller on top of the 4e scheduler/RunTask flow: - CreateService/UpdateService/DeleteService/DescribeServices/ListServices. REPLICA only; DAEMON and serviceRegistries rejected in v1 (Q15). - StartTask (explicit placement) and StopTask. - Leader-driven replica reconciliation: a 10s scheduler tick converges each service's running task count on desiredCount, launching the shortfall via RunTask (tasks tagged group=service:{name}) and force-stopping the surplus. - ELBv2 target-group registration: the scheduler registers a task's awsvpc ENI IP on the RUNNING transition and deregisters on STOPPED via the existing elbv2.RegisterTargets/DeregisterTargets NATS handlers (Q8, single-writer). - forceStopTask consolidates the control-plane stop (STOPPED state, capacity + ENI release, inbox drain, TG deregister); reused by StopTask, scale-in, and the heartbeat reaper (the reaper now deregisters targets per Q13). - Wire the 7 new actions through the gateway action map, NATS adapter, and daemon subscriptions; add ServiceNotFoundException. * fix(ecs-agent): anonymous pull for non-ECR images + normalize image refs The agent minted an ECR authorization token via IMDS for every image, so public docker.io refs failed with 'IMDS returned empty role name' on hosts without an instance role. Gate ECR auth on .dkr.ecr. refs and pull all other registries anonymously. Bare image strings (nginx:alpine) were handed to containerd unmodified, which parsed them as dummy://nginx:alpine and rejected the pull. Normalize refs to their fully-qualified form (docker.io/library/nginx:alpine, :latest default) before Pull and GetImage. Adds StartTask/StopTask service-controller coverage. * fix: address ECS 4f live-verification findings ecs-node-bringup.sh defaulted GATEWAY_IP to the mgmt bridge (10.15.8.1), which a VPC guest cannot route to — the agent could not reach the gateway. Default to the host's WAN-bridge IP, which guests reach by NATing out through the WAN. A target group not forwarded to by any listener reported its targets stuck in 'initial' forever (no health checker advances a detached TG). DescribeTargetHealth now reports such targets as 'unused' with reason Target.NotInUse, matching AWS.	2026-06-25 21:06:00 +10:00
Josh Siviour	de64d6c63d	feat(ecs): agent control-plane over the AWS gateway (drop direct NATS) (#419) ... * feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs") client the ecs-agent will use to reach the AWS gateway over HTTPS instead of connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal. * feat(ecs): SubmitTaskStateChange gateway action → recordTaskState Wires the AWS-API SubmitTaskStateChange action end to end (gateway action → ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a gateway-routed agent reports task state without touching the NATS bus. The handler maps the SDK input onto the same bus.TaskState shape and converges on recordTaskState; the account is authoritative from the SigV4 caller, so an instance cannot report state for another account's task. * feat(ecs): durable assign KV inbox + PollAssignments gateway action Scheduler writes task assigns into a per-instance KV inbox (clusters/{cluster}/assignments/{instance}/{task}) instead of a fire-and-forget core-NATS publish, and a new PollAssignments gateway action lets a gateway-routed agent drain its inbox over TLS+SigV4. At-least-once: an unacked assign is re-delivered on the next poll; the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions reclaim stale inbox entries. The Poll response carries bus.Assign with an RFC3339 time, so it is encoded with encoding/json (RawJSONActions) rather than the jsonutil epoch-second marshaler. * feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway The agent no longer connects to NATS. It registers, heartbeats, reports task state, and receives assignments entirely through the AWS gateway over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus host-internal and closing the trust-boundary gap (a tenant container instance must not hold a shared bus token). - controlPlane seam over ecsgw.Client replaces the NATS publisher: Register (RegisterContainerInstance), SubmitTaskState (SubmitTaskStateChange), PollAssignments. - Heartbeat folds into a periodic idempotent re-register (refreshes LastSeen); bus.Heartbeat is dropped from the agent. - subscribeAssign (NATS subscribe) becomes a poll loop: drain the inbox, dispatch each assign once, ack accepted taskIDs on the next poll (at-least-once). - config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added, ECS_NATS_URL removed; cloud-init initd seeds the IAM creds. Tests move from the NATS publisher fake to a controlPlane fake. * feat(ecs): cloud-init cred-seed + ecs-node bring-up script ECS container instances are plain EC2 booted from the spinifex-ecs-node AMI (AWS-faithful: no ECS-specific launch API). This renders the cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the instance. The agent registers, heartbeats, polls assignments, and reports task state entirely through the AWS gateway over TLS+SigV4 — never NATS. The guest dials the customer-facing gateway address (network advertise IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL split. Seeds the system key for now; scoped per-instance creds are a follow-up (4g task-role endpoint). * fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring The 'Merge branch dev' resolution dropped code where Sprint 4d (#418, awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch the same paths: - recordTaskState STOPPED branch: reclaimTaskENI was clobbered by reclaimAssignInbox; both are needed (release the task ENI AND drain the KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce. - ecs-agent Agent: the netns field, its init, and the setupTaskNetns/ teardownTaskNetns wrappers were lost when the controlPlane Agent struct won the merge, leaving taskrun.go calling undefined methods. - eni_test assign assertion read the assign over NATS; #419 delivers it via the durable KV inbox, so it now polls via PollAssignments. - taskrun_test netns cases used the removed historyPub fake; switched to fakeCP to match the controlPlane seam.	2026-06-25 18:42:24 +10:00
Josh Siviour	9c76df3373	feat(ecs): awsvpc mode — ENI-per-task allocation + guest datapath (Sprint 4d) (#418) ... * feat(ecs): awsvpc ENI-per-task control plane (Sprint 4d) Scheduler owns the task-ENI lifecycle for awsvpc mode: allocate from the task subnet on placement, hot-plug onto the container instance, plumb ENI identity through the assign payload + task record, and detach+delete on STOPPED (graceful + reaper, idempotent leak guard). bridge/host unchanged. * feat(ecs-agent): awsvpc guest-side task netns from hot-plugged ENI The agent builds a per-task network namespace for awsvpc tasks: it finds the hot-plugged ENI NIC by MAC, moves it into /var/run/netns/ecs-<taskID>, and DHCPs the OVN-assigned ENI IP. The container then joins that netns via the OCI network namespace path; bridge/host tasks keep the host (VM) netns as before. * build(ecs-node): add iproute2 for awsvpc task netns The ecs-agent builds awsvpc task network namespaces with `ip netns`, which busybox's `ip` applet does not implement. Install iproute2 in the ecs-node AMI; udhcpc (for the in-netns ENI DHCP lease) is already provided by busybox.	2026-06-25 16:01:14 +10:00
Julian Sommer	8a94886b13	feat: cloud-init EC2 datasource (#411)	2026-06-25 13:48:08 +10:00
Josh Siviour	24bd50a53c	feat: ecs-node AMI (alpine + containerd + ecs-agent baked) (#415) ... Adds the spinifex-ecs-node AMI variant: alpine 3.21.7 + containerd + runc + cni-plugins + ca-certificates with the ecs-agent binary baked to /usr/local/bin and three OpenRC services wired in dependency order (ecs-ca-install -> containerd -> ecs-agent). - manifest.conf drives build-system-image.sh (no host mounts). - ecs-ca-install.initd: first-boot oneshot installing the cloud-init delivered gateway CA into the system trust; no-op when absent. - ecs-agent.initd: sources /etc/spinifex-ecs/agent.env, backgrounds the agent after net + containerd. - 10-bridge.conflist: CNI bridge for task networking (Sprint 4d). - setup.sh: exec bits, /opt/cni/bin symlink, content-store dirs, ttyS0 console reorder + fast boot menu. - Makefile: build-ecs-node-image / import-ecs-node-image targets.	2026-06-24 21:54:35 +10:00
Julian Sommer	ea03efe415	fix: reset-dev-env regex	2026-06-24 16:48:06 +10:00
Julian Sommer	f45c97215a	chore: bump total coverage requirement	2026-06-24 09:32:31 +10:00
Julian Sommer	e11437cec9	feat(imds): per-tap responder datapath (#408)	2026-06-23 16:44:53 +10:00
Josh Siviour	5025a37006	feat: Containers — internal-ECR pulls, DNS-free registry, EKS hardening + docs (#409) ... * docs: add Containers section with EKS and ECR service pages New docs/containers/{eks,ecr} pages covering each service via AWS CLI, the Spinifex console, and Terraform, plus the requisite VPC/IGW/NAT/SG/IAM setup. EKS page embeds console screenshots. Cross-links the EKS Terraform workbooks. * docs(eks): rework Terraform workbooks into a capability ladder Rework the three EKS workbooks into a coherent ladder and publish them to the Containers section. All tiers are single- or three-node capable via node_desired_size and serve a shared Spinifex-themed demo app from ECR, replacing nginxdemos/hello. - demo-app: new std-lib Go web app (themed page reporting pod/node/cluster/ region + hit counter; optional /data persistence), distroless image. - eks-quickstart: node_desired_size var, ECR repo + themed app over NodePort. - eks-https-ingress (was eks-addons-access): HTTPS via the aws-load-balancer- controller addon + ACM, public/private VPC with NAT, managed-ingress=false. - eks-gitops-argocd (was eks-https-ingress): Argo CD addon syncs the app from a git repo, aws-ebs-csi-driver PVC (Viperblock-backed) for state, access entry RBAC, HTTPS Ingress carried over. Pending live e2e verification (image build/push, apply, ALB + PVC bind). * fix(ecr): implement ListTagsForResource (empty tags) for client parity The AWS provider calls ecr:ListTagsForResource on every repository read, which returned 501 NotImplementedException and broke 'tofu apply' of any workbook that creates an aws_ecr_repository. Implement it to return an empty tag set (resource tagging stays a deferred feature) so reads succeed. Drop the tags block from the three EKS workbooks' ECR repos so the provider doesn't call the still-stubbed TagResource on create. * feat(eks): pull from internal ECR via kubelet credential provider Worker nodes could not pull custom images from the internal Spinifex ECR registry: the eks-node had no DNS for the registry host, no containerd CA trust, no port rewrite to :9999, and no pull credentials. Only public images worked. Add cmd/ecr-credential-provider, a kubelet exec credential provider that fetches the node IAM-role credentials from IMDS, calls GetAuthorizationToken against the gateway ECR endpoint, and returns the minted AWS:<jwt> as Basic auth for the registry host. Thread the gateway URL, account, region, and internal suffix into the agent userdata, which now writes the gateway CA, a registries.yaml (CA trust + :9999 endpoint), the kubelet credential-provider config and config.yaml.d drop-in, and an /etc/hosts entry mapping the registry host to the gateway IP. Ship the provider binary in the eks-node AMI. Also remove a stray paragraph from the admin-cli README. * fix: dial EKS ECR registry mirror by gateway IP cloud-init manage_etc_hosts rewrites /etc/hosts from the alpine template on every boot, wiping the registry-host mapping appended via bootcmd, so worker nodes could not resolve the internal ECR host and pulls failed with no such host. Point the k3s registries.yaml mirror endpoint and tls config at the gateway IP instead (the gateway cert carries an IP SAN), keeping the pod image ref port-less and dropping the DNS leg until northstar lands. Fall back to the hostname endpoint when the gateway is not an IP. * fix: attach node-role instance profile to EKS workers for ECR EKS workers launched with no IAM instance profile, so IMDS served no role and the ECR credential provider could not mint a registry token, leaving internal ECR pulls failing with 401. Real EKS implicitly creates an instance profile for the node role; do the same here. Wire a KV-backed IAM service into the EKS deps and, at worker launch, find-or-create an instance profile named for the node role, attach the role, and pass it to RunInstances so IMDS exposes the role to the credential provider. * feat: use mulga logo in EKS demo app * feat: add ACM certificate tag support (List/Add/RemoveTags) Implements ListTagsForCertificate, AddTagsToCertificate and RemoveTagsFromCertificate across the store, service, NATS, daemon and gateway layers. Import now persists tags. Fixes terraform aws_acm_certificate InvalidActionException on ListTagsForCertificate. * test: cover ACM tag gateway and NATS wrapper paths * feat: support EKS nodegroup tags (Tag/Untag/ListTags) Extends EKS tag ops to back nodegroup ARNs alongside cluster ARNs. Nodegroup create-time tags persist on the record and echo via DescribeNodegroup, giving the terraform-aws provider a clean default_tags round-trip. Fixes aws_eks_node_group TagResource 501 NotImplementedException. * fix: provision EKS ALB ingress via node credentials The aws-load-balancer-controller addon could not provision an ALB. Three fixes: - TLS: the controller's per-service ec2/elbv2/acm SDK clients install their own BaseEndpoint resolver and build an HTTP client that bypasses AWS_CA_BUNDLE, falling back to the Go system trust pool and rejecting the gateway cert as 'unknown authority'. Add SSL_CERT_FILE so the gateway CA joins the system pool. - Credentials: an empty AWS_ROLE_ARN beside the web-identity token file half-arms the SDK web-identity provider, which then signs anonymously (MissingAuthenticationToken). Inject the IRSA env only when a service-account role ARN is staged; otherwise omit it so the default chain falls back to the node instance-profile credentials served over IMDS. - Authorization: the node role lacked the ec2/elbv2 actions the controller needs. The eks-https-ingress workbook now attaches an AWSLoadBalancerController managed policy to the node role (CreatePolicy + AttachRolePolicy). * fix: add force_delete to EKS workbook demo ECR repos tofu destroy of an EKS workbook left the demo ECR repository behind: it still holds the pushed demo image, so DeleteRepository (force=false) returns RepositoryNotEmptyException. Add force_delete = true to each workbook's aws_ecr_repository.demo so teardown removes the repo with its images. * fix: force-delete EKS control-plane ENIs on teardown TerminateK3sServerVM deleted the CP ENI with a plain force=false DeleteNetworkInterface. eniIsLiveAttachment reports InUse whenever the record's Status/InstanceId/AttachmentId are set, so when the K3s VM is gone but the ENI record's attachment fields were never cleared the delete returned InvalidNetworkInterface.InUse permanently and EKSDeletingReaper re-drove the same non-idempotent call every 90s, wedging the cluster in DELETING. Teardown owns its CP ENIs and has already terminated the VM, so the attachment is authoritatively dead. Replace the InUse-retry budget loop with detach-then-delete (detachAndDeleteServerENI): DetachENI clears the stale attachment fields, then DeleteNetworkInterface passes the in-use guard. Both calls tolerate an already-gone ENI (NotFound) so a race with the async instance-terminate cascade resolves to idempotent success. purgeClusterInfra's private-endpoint ENI path reuses the same helper. * fix: detach+delete worker ENIs on EKS nodegroup teardown A terminated nodegroup worker could leave its primary ENI in spinifex-vpc-enis as in-use/attached, and a dangling ENI throws DependencyViolation on subnet/VPC delete, blocking tofu destroy. Two gaps: TerminateWorkerInstances did a local-only vmMgr.Get and silently skipped any worker owned by another node, stranding both the VM and its ENI on multi-node clusters. Route the terminate per-instance via ec2.cmd.<id> (mirroring the gateway TerminateInstances path) so the owning daemon runs the full teardown — detach + force-delete the primary ENI and clear its record. No responder / NotFound payload is treated as already-gone so a retried DeleteNodegroup stays idempotent. deleteInstanceENI (stopped-instance terminate) issued a force=false DeleteNetworkInterface with no prior detach: a stale in-use attachment made it fail InvalidNetworkInterface.InUse, leaving the ENI record behind after the instance was already written to the terminated bucket. Detach first, then delete, tolerating NotFound. * fix: heal volume State drift on detach/terminate teardown A detach/terminate could leave a volume's State empty instead of 'available'. DeleteVolume gated on State=='available' exactly, so it returned VolumeInUse for an unattached volume nothing was using, leaving it undeletable and blocking stack teardown. DescribeVolumes cosmetically rendered an empty State as 'available', masking the drift. - DeleteVolume now treats an empty State with no attachment as deletable (an attached volume still returns VolumeInUse). - DescribeVolumes derives an empty State from the attachment (attached -> in-use, unattached -> available) instead of blindly rendering available, so an empty-but-attached volume is no longer hidden. - UpdateVolumeState normalizes an unattached volume with an empty state to available, so a detach writeback that omits the state cannot strand the volume in drift. * fix: break all SG cross-references on EKS teardown DeleteClusterSGs broke the cp<->nodegroup cycle too narrowly. It revoked only ingress (IpPermissions), never egress (IpPermissionsEgress), so a cross-reference via an egress rule kept the SG pinned on DependencyViolation. It also resolved only the two named cluster SGs, so any OTHER SG in the VPC referencing them — the default SG, the LBC/ALB SG, the private-endpoint SG, user/TF SGs — was never revoked and pinned the referenced SG on delete, leaving the VPC stuck. Enumerate every non-default SG in the VPC and revoke BOTH ingress and egress on each before deleting, breaking any cycle order-independently, then delete only the cluster-owned SGs (referrer SGs are deleted by their own teardown). The default SG is left untouched. Since DeleteClusterSGs runs against both the customer VPC and the managed control-plane VPC, the broadened sweep covers both teardown paths. * feat: support TargetGroup ProtocolVersion in elbv2 CreateTargetGroup now stores ProtocolVersion (defaulting HTTP/HTTPS target groups to HTTP1) and DescribeTargetGroups/Create output return it. Without this the aws-load-balancer-controller sees a perpetual immutable-field mismatch (desired HTTP1 vs empty) and recreates the target group every reconcile, looping on DuplicateTargetGroupName once a shared ALB IngressGroup reconciles more than one target group. NLB protocols (TCP/UDP/TLS) keep it omitted to match AWS. * fix: omit IRSA env from ebs-csi addon when no role staged The ebs-csi controller manifest hardcoded AWS_ROLE_ARN={{SERVICE_ACCOUNT_ROLE_ARN}} plus the web-identity token volume. With no service-account role staged the placeholder rendered empty, half-arming the SDK web-identity provider, which panics 'role ARN is not set' and CrashLoops the controller, leaving PVCs Pending. Convert to the conditional {{IRSA_ENV}}/{{IRSA_VOLUME_MOUNT}}/{{IRSA_VOLUME}} markers like the LBC bundle so the IRSA env and token volume are omitted when no role is staged, falling back to node instance-profile credentials over IMDS. The EBS-specific AWS_EC2_ENDPOINT/AWS_ENDPOINT_URL_EC2 stay in both modes. Requires an eks-node AMI rebake to take effect. * docs: route gitops-argocd workbook through a shared ALB and expose ArgoCD UI Fold the demo app and ArgoCD ingresses onto one ALB via an LBC IngressGroup with host routing (app./argocd. on a wildcard cert) instead of one ALB per ingress, add a NodePort + HTTPS Ingress for the ArgoCD UI, and document the access steps. Add the node-role LBC and EBS-CSI IAM policies the controllers need (Spinifex wires controller creds at the node level, not IRSA) and an ignore_changes block for the cluster's access_config and vpc_config security_group_ids, which Spinifex DescribeCluster does not echo back, so the provider stops trying to replace the cluster on every plan. * fix: stop addon-sync substituting IRSA markers named in comments The ebs-csi controller manifest's header comment spelled out the literal {{IRSA_ENV}} / {{IRSA_VOLUME_MOUNT}} / {{IRSA_VOLUME}} marker tokens. addon-sync matched each marker by substring, so the comment lines were replaced with the env/mount blocks, injecting a bare YAML sequence between two documents. K3s' auto-deploy then rejected the whole file (ParseManifestFailed: cannot unmarshal array into unstructured), so the ebs-csi Addon CR never applied and the EKS addon hung in CREATING. Reword the comment to not contain the literal tokens, and anchor the marker substitution to lines that are exactly the marker (whitespace aside) so a future comment mentioning a marker can't retrigger it. * fix: disable K3s local-storage on EKS clusters in parity mode In AWS LB Controller parity mode (BuiltinIngress=false) the K3s server disabled traefik and servicelb but left local-storage enabled. Its local-path provisioner ships a StorageClass marked default, which races the EBS CSI ebs-gp3 default StorageClass — two defaults, so dynamic PVCs bind nondeterministically. EKS has no local-path, so disable local-storage too and let ebs-gp3 be the sole default. The built-in-ingress path is unchanged. * fix: re-encrypt HTTPS target groups in haproxy ALB backends An ALB target group with backend-protocol HTTPS (e.g. argocd-server, which serves TLS on its own port) rendered a plaintext haproxy server line. HAProxy's own check then spoke HTTP to a TLS port, failed, and marked the server DOWN, returning 503 — even though the lb-agent's separate HTTPS health probe reported the target healthy, so describe-target-health masked the fault. Emit check-ssl ssl verify none on the server line for HTTPS target groups so both the check and proxied traffic re-encrypt to the backend. * fix: reconnect wedged QMP client on next command A timed-out QMP decode under host contention left the cached client's shared json.Decoder mid-message, wedging every later command and breaking EBS AttachVolume/DetachVolume for the VM's lifetime. Mark the client Dead on encode/decode error and redial the retained socket path (re-running the qmp_capabilities handshake in place) on the next send. * fix: round-trip security-group rule descriptions SG ingress/egress rules dropped the per-rule Description on store and describe. The AWS Load Balancer Controller tags the worker-node SG rules it manages with a UserIdGroupPair description and identifies them by that value to revoke them on teardown. Without the round-trip the controller cannot match its rules, never revokes them, and the backend security group stays referenced — DeleteSecurityGroup returns DependencyViolation forever, hanging the ingress finalizer and tofu destroy with orphaned security groups. Persist Description on SGRule (excluded from sgRuleKey so it is metadata, not identity) and render it back through describe. * fix: reclaim customer-VPC nodegroup SGs on managed-CP-VPC teardown In the managed control-plane VPC topology, launchNodegroupInfra creates the cluster control-plane and nodegroup security groups in the customer VPC for worker<->control-plane networking, but purgeClusterInfra only reclaimed the security groups in the managed CP VPC. The customer-VPC SGs orphaned, cross-referenced each other, and pinned the customer VPC with DependencyViolation, hanging tofu destroy of the VPC. Reclaim them in the managed-CP-VPC branch too, matching the legacy customer-VPC branch. * fix: reclaim ALB EIP when system VM already gone on terminate TerminateSystemInstance only released an internet-facing ALB's Elastic IP on the local path, where the VM is still present in the manager. When the VM was already gone (health-reaped, crashed, or raced by ENI teardown), it took the remote/no-owner path and returned idempotent success without reconciling the EIP, leaking the wan-pool address still associated to a deleted ENI. Add EIPServiceImpl.ReleaseAddressByInstanceID, an idempotent account-agnostic scan that disassociates then releases any EIP recorded against an instance, and call it as a backstop in TerminateSystemInstance for both branches. * fix: cascade-prune demo PVC via Argo CD finalizer in eks-gitops workbook The demo Application carried no resources finalizer, so tofu destroy removed the Application CR without pruning the synced PVC. The PVC survived, the cluster was destroyed, and the Delete-reclaim PV's EBS volume orphaned. Add the resources-finalizer.argocd.argoproj.io finalizer so deleting the Application cascades to the PVC and ebs-csi reaps the volume before cluster teardown. * feat: raw-IP access + documented Argo CD creds in eks-gitops workbook The shared ALB host-routed everything, so the raw ALB IP 404'd in a browser without an /etc/hosts entry. Make the app the hostless catch-all on :443 and add a hostless Argo CD listener on :8443, so https://<alb-ip>/ serves the app and https://<alb-ip>:8443/ serves the UI with no DNS. Surface the admin-password command as a tofu output and document the credential + IP URLs in the README. * fix: reclaim ALB EIP on the owning-node terminate path The EIP backstop lived only in TerminateSystemInstance, but the ALB delete path goes serveSystemTerminateInstance -> terminateSystemInstanceLocal directly, bypassing it, and releaseSystemInstanceEIP relied on the VM's cached allocation which is often empty. Extract reclaimSystemInstanceEIP (authoritative KV reconcile via ReleaseAddressByInstanceID) and call it from terminateSystemInstanceLocal as well as the no-owner backstop, so an internet-facing ALB's EIP is released on every terminate route. * chore(eks): drop coredns from the managed addon catalog Cluster DNS ships as the baked node-local coredns-mulga DaemonSet, not as a managed addon — there is no bundled coredns manifest, so CreateAddon coredns would stage but fail VM-side render. Remove the phantom catalog entry so DescribeAddonVersions and CreateAddon no longer advertise it; repoint the addon tests that used coredns as their sample to argocd. * feat(ui): group EKS and ECR under a Containers nav section Fold the separate EKS and ECR sidebar groups into one Containers group with EKS (clusters) and ECR (repositories) entries. * fix(ui): satisfy strict null checks in ECR component tests Index mock.calls via optional chaining and narrow the getAllByRole result before clicking, clearing the TS2532/TS2345 errors that failed pnpm fix. * refactor(eks): hide internal fixture addons from the public catalog Add an AddonSpec.Hidden flag and mark spinifex-noop hidden: it stays creatable (lookupAddon finds it, so the delivery-transport e2e still drives it) but no longer surfaces in the unfiltered DescribeAddonVersions listing. * refactor(eks): remove K3s built-in ingress; LBC-only Drop the BuiltinIngress option and its managed-ingress tag opt-in. traefik, servicelb, and local-storage are now always disabled on EKS nodes (AWS parity — Service type=LoadBalancer and Ingress are the Load Balancer Controller's job). Removes the EKS_DEFER_TRAEFIK userdata flag, the k3s.initd traefik .skip staging, and the maybe_enable_traefik deferred-enable loop in the state reporter. * fix(ui): clear lint debt in VPC-wizard mutations and CIDR validation Parallelize the independent subnet-create and route-table-association loops in the create-VPC mutation with Promise.all (each task records its resource in the rollback list before any further call), and switch the CIDR-validation regex to named capture groups. Clears the no-await-in-loop and prefer-named-capture-group lint errors. * fix: carry gateway port in ECR registry endpoints; rename mulga.internal -> spinifex.internal The ECR registry is served on the gateway port (:9999), but proxyEndpoint and repositoryUri were emitted port-less, so docker login/tag/push dialed :443 and failed. Derive the port from the awsgw listen host and append it to the registry host (omitted for 443). Also rename the default internal DNS suffix. * docs: drop verbose ECR :9999 host note * docs: fix literal-asterisk emphasis (single-* not rendered) * docs: brace-wrap ${REGISTRY}:latest (zsh :l modifier ate the tag); document Docker CA trust * feat: default ECR registry URIs to the gateway host (no-DNS interim) The registry authenticates by token, not hostname, so the per-account <acct>.dkr.ecr name is cosmetic. Advertise the gateway's reachable bind host:port (already a cert SAN) in repositoryUri/proxyEndpoint so docker login/tag/push need no DNS; the parity host still works server-side where DNS exists. * fix: serve ECR registry on a DNS-free gateway host end to end Advertise the gateway's AdvertiseIP as the ECR registry host when the AWSGW bind is unspecified (0.0.0.0/::), so repository_url resolves to a reachable host:9999 carried in the server cert SANs rather than the unresolvable per-account parity name. Docker push then dials it directly with no DNS. Mirror the same host on EKS workers: the registries.yaml mirror keys and the credential provider matchImages now cover the port-less parity name, the port-ful parity name, and the gateway IP:9999, all redirecting to the dialed gateway endpoint. A port-ful pull from the API-returned repository_url matches the mirror instead of bypassing it into a failed DNS lookup. Net: docker push host, pod image host, and the API's repository_url are the same value, with no DNS or /etc/hosts. * docs: resolve eks-https Ingress ALB DNS name to its public IP (no DNS) * docs: resolve eks-gitops Ingress ALB DNS name to its public IP (no DNS) * docs: remind to run aws eks update-kubeconfig before kubectl in EKS workbooks	2026-06-23 15:29:14 +10:00
Josh Siviour	9d662f9889	feat: EKS EBS CSI driver (Viperblock) — egress, attach, node discovery, e2e (#402) ... * feat: bake aws-ebs-csi-driver addon + mdev by-id bridge Add the aws-ebs-csi-driver entry (v1.40.1) to the eks addon catalog and ship its baked manifests (RBAC, IRSA-wired controller, node DaemonSet, default gp3 StorageClass) under the eks-node image. The node plugin locates EBS volumes via /dev/disk/by-id, which busybox mdev does not populate. Add mulga-ebs-byid: an mdev rule + helper that symlinks nvme-Amazon_Elastic_Block_Store_<serial> from the virtio-blk serial, plus a boot oneshot that points the kernel hotplug helper at mdev and coldplugs volumes present at boot. * test: e2e EBS CSI volume — PVC bind + mount + reschedule survival Add a TestEKS/EBSCSIVolume subtest that installs the aws-ebs-csi-driver managed addon bound to an IRSA role, provisions a default gp3 PVC + pod, and asserts a marker written by the first pod survives a reschedule onto a fresh pod backed by the same PVC. Exercises the full data path: CreateVolume(Viperblock)/AttachVolume virtio-blk hotplug with serial set to the volume-id, mdev /dev/disk/by-id symlink, and ext4 mount. * feat: eks-node provider-id + topology/instance-type labels for EBS CSI mulga-eks-provider-id boot oneshot writes a K3s config.yaml.d drop-in from IMDS before K3s starts: kubelet provider-id plus the region, zone and instance-type node labels. The aws-ebs-csi-driver node plugin reads this metadata for NodeGetInfo (instance id from providerID, region/zone from topology labels, instance type for the volume-limit family lookup). k3s runs no AWS cloud-provider so none of it is set otherwise, which left the node plugin's metadata init failing, NodeGetInfo panicking in IsNitroInstanceType, and the CSINode carrying no topology key. Wire the service into the image manifest (INSTALL_FILES + ENABLE_SERVICES) and chmod it in setup. * test: e2e EBS CSI controller IRSA policy + failure diagnostics awsgw enforces IAM on assumed-role EC2 calls, so the controller's CreateVolume/AttachVolume need a permission policy on the IRSA role. Attach a customer-managed EC2 policy (AWS-managed ARNs are opaque and grant nothing) covering the EBS CSI action set; CreateVolume then succeeds. Add an OnFailure dump of the CSI control path (provisioner and plugin logs, pvc/pod describe, events, csinode) to the EBSCSIVolume subtest so provisioning/attach failures are diagnosable post-teardown. * feat: EBS CSI obeys control-plane taint; e2e worker nodegroup Drop the CriticalAddonsOnly/NoExecute tolerations from the EBS CSI controller + node manifests so the driver runs only on customer worker nodes, never the system control-plane VM. The e2e creates a 1-node worker nodegroup and the writer/reader pods drop their CP tolerations so the customer-owned volume attaches in the customer space. * fix: EBS CSI attach — decouple hotplug PCIe port from device name; iothread rollback virtio-blk-pci hot-plug bus was derived from the AWS device-name letter (/dev/sd[f-p]); the EBS CSI driver sends arbitrary names (/dev/xvdaa) that map to no slot, so device_add fell back to pcie.0 — which rejects hot-plug — and the attach hung. A failed attach also left the per-volume iothread object behind, so every retry died on object-add 'duplicate property'. nextHotplugEBSPort allocates the lowest free hotplug-ebs port from live query-block state, independent of the device name. delIothreadBestEffort removes the iothread on every attach rollback path. * fix: EBS CSI node discovery — un-shadow mdev by-id rule, mint nvme link Alpine eks-node runs busybox mdev (no eudev), which stops at the first matching /etc/mdev.conf rule. setup.sh appended the mulga-ebs-byid rule after the stock 'vd[a-z].* -> persistent-storage' line, so it never fired on hot-attach: only /dev/disk/by-id/virtio-<serial> was created, while the upstream aws-ebs-csi-driver resolves nvme-Amazon_Elastic_Block_Store_<volid>. Swap the stock vd* rule in place to route through mulga-ebs-byid, which now delegates to persistent-storage (preserving stock links) before minting the nvme-style link the node plugin expects. * test: EKS e2e — NAT Gateway egress for private worker nodegroup The customer-VPC worker is private; AttachIGW programs no SNAT (mirrors AWS: an IGW serves only public-IP instances), so an IGW-only fixture left the worker unable to reach the public NLB endpoint or pull public CSI images — the nodegroup never joined. createWorkerEgress now builds the real topology: an IGW-fronted public subnet hosting a NAT Gateway, with the worker subnet default-routed to it. Associating the worker route table triggers the subnet-wide SNAT egress reroute. Teardown drains the NAT Gateway before releasing the public subnet + EIP it pins.	2026-06-19 14:43:20 +10:00
Josh Siviour	2acb37342d	feat(eks): aws-load-balancer-controller addon — real ALB Ingress (L7, IRSA) (#376) ... * fix(ec2): stamp subnet AvailabilityZoneId for single-AZ ALB resolution DescribeSubnets omitted AvailabilityZoneId, so the AWS Load Balancer Controller's DescribeAvailabilityZones-by-ZoneId lookup resolved no AZ info ("cannot resolve AZ info for AZs: []") and never built the ALB model. Stamp the single zone-id on every subnet and share it with DescribeAvailabilityZones through one SingleZoneID constant so the two responses can never drift. * feat(eks): add aws-load-balancer-controller v2.8.1 addon bundle Baked manifests for the LBC managed addon (CRDs, controller Deployment/ RBAC/webhooks/Service, IngressClass alb, IRSA render markers). Two wirings make it work against the spinifex gateway: - --aws-api-endpoints carries sts= alongside elasticloadbalancing/ec2/ acm. The aws-sdk-go v1 credential bootstrap resolves the STS client from the session endpoint resolver, not AWS_ENDPOINT_URL_STS, so the web-identity AssumeRoleWithWebIdentity must be pointed at the gateway here or it falls back to real AWS STS. - --feature-gates=ALBSingleSubnet=true drops the ALB minimum subnet count from two to one so an ALB provisions in a single-AZ deployment. * feat(eks): render IRSA into managed addons + node-minted webhook cert Extend the addon-sync renderer to inject the projected-token / role-arn / gateway STS-endpoint + CA block into managed-addon pod specs via opt-in markers, and substitute cluster-name/region/vpc-id/gateway-endpoint/CA placeholders. Add cmd/eks-webhook-cert: the eks-node AMI has no openssl, so this mints (then reuses) a self-signed ECDSA-P256 serving cert for the admission webhook, cached on disk so the render is byte-stable across ticks. Wire both into the image build (manifest.conf) plus INSTALL_FILES for the LBC bundle. * feat(eks): surface cluster VPC id as EKS_VPC_ID for the LB controller Thread the control-plane VPC id through K3sServerInput.VpcID into the first-boot EKS_VPC_ID env so the in-cluster aws-load-balancer-controller can pass --aws-vpc-id to the gateway elbv2/ec2 handlers. * fix(eks): point the LB controller at the customer VPC + customer-facing gateway The aws-load-balancer-controller ran with two control-plane-scoped values that customer worker pods cannot use: - --aws-vpc-id was set from the managed control-plane VPC (cpRefs.VpcID), which the gateway's DescribeVpcs does not return for customer calls, so backend-SG auto-create failed with InvalidVpcID.NotFound. Source it from meta.ResourcesVpcConfig.VpcId, the customer cluster VPC where the workers and their subnets live and where the ALB is provisioned. - The addon render baked the mgmt gateway endpoint (EKS_GATEWAY_URL / SystemGatewayURL) into pod specs, unreachable from worker pods. Add a distinct EKS_ADDON_GATEWAY_URL (the customer-facing GatewayBaseURL, same host as the OIDC issuer) and substitute it for {{GATEWAY_ENDPOINT}} in managed addon manifests. EKS_GATEWAY_URL stays mgmt for the CP's own bootstrap, state-report and token-webhook paths. * fix(elbv2): resolve forward target group from ForwardConfig listenerActionFromSDK read only the legacy flat Action.TargetGroupArn, so a forward action carrying its target group via ForwardConfig.TargetGroups (the shape the aws-load-balancer-controller emits) left the action target empty and CreateRule/CreateListener rejected it with MissingParameter. Flatten the first target group from ForwardConfig when the flat field is absent. * fix(ec2): mirror CreateTags/DeleteTags into subnet/vpc records The generic S3 tag store and the per-resource KV records are disjoint: DescribeSubnets/DescribeVpcs read tags from their KV record, while CreateTags only wrote the generic store. Subnet tags applied after create were therefore invisible to tag-filtered describes, breaking aws-load-balancer-controller subnet auto-discovery (kubernetes.io/role/elb). Write through CreateTags/DeleteTags to the owning subnet/vpc record so tag-filtered describes observe them. Resource types the VPC service does not own are skipped; the generic store stays their record of truth. * test(daemon): pin synthetic capacity in host-dependent allocate tests TestCanAllocate_CountEdgeCases/Mixed_instance_types and TestHandleEC2DescribeInstanceTypes/NoFilterReturnsSupportedRegardlessOfAllocation called rm.allocate against live-detected host capacity. allocate gates on readMemAvailableGB (live /proc MemAvailable), so a box already running VMs fails the allocation spuriously, breaking the preflight gate locally. Pin hostVCPU/hostMemGB/reserved and nil readMemAvailableGB to a synthetic host, matching the sibling Capacity_decreases_after_allocation subtest. * fix(eks): route apiserver->pod traffic via agent tunnel (egress-selector cluster) In the managed-CP topology the apiserver VM sits in the system CP VPC with no route to the worker pod network in the unpeered customer VPC. k3s default egress-selector-mode=agent tunnels only apiserver->kubelet, so kubectl logs/exec work but apiserver->pod traffic (admission webhooks, aggregated APIs) is attempted directly and times out -- the aws-load-balancer-controller webhooks fail with context deadline exceeded, blocking Service/Ingress creation cluster-wide. Set egress-selector-mode=cluster so apiserver->pod/service traffic routes through the agent's outbound tunnel, selected by the node hosting the destination, without requiring inbound CP->worker reachability. * feat: bump aws-load-balancer-controller addon bundle to v2.11.0 v2.8.1 predates the ALBSingleSubnet feature gate, so single-AZ spinifex clusters fail Ingress reconcile (subnet resolver dedups to 1 < required 2) and never provision an ALB. v2.11.0 honors --feature-gates=ALBSingleSubnet=true. Replace the baked bundle: new CRDs (ingressclassparams + targetgroupbindings), controller manifest, and image tag v2.11.0, re-applying the spinifex deltas (IRSA SA annotation, node-minted webhook serving-cert secret, gateway-CA secret, gateway aws-api-endpoints, caBundle on every webhook clientConfig, alb IngressClass). Pick up the new ClusterRole configmaps rule. Drop v2.8.1. * fix: point LBC at the gateway via AWS_ENDPOINT_URL (sdk-v2) LBC v2.11 uses aws-sdk-go-v2, which keys --aws-api-endpoints on SDK service IDs (e.g. 'Elastic Load Balancing v2'), not the legacy lowercase names the bundle passed. Those overrides were silently ignored, so every non-STS call (ec2/elbv2/acm) went to real AWS and failed TLS against the gateway CA -- no ALB could provision. Inject AWS_ENDPOINT_URL in the shared IRSA env block (sdk-v2 global base-endpoint override; AWS_ENDPOINT_URL_STS still takes precedence for STS) and drop the dead --aws-api-endpoints flag from the controller args. * fix: synthesize taggable ARN for default listener rule Gateway DescribeRules returned the synthetic IsDefault rule with a null RuleArn. LBC read the arn-less default rule after creating a listener and issued DescribeTags with an empty resource ARN, which the gateway rejected with 400 MissingParameter, aborting reconcile. Set a deterministic RuleArn on the default rule, resolve it back to its parent listener in DescribeTags (empty tags) and DescribeRules by-arn. Also disable shield/waf/wafv2 in the LBC bundle to drop unsupported calls. * feat: inject ELB subnets into alb IngressClassParams for single-AZ ALB LBC's tag auto-discovery never threads the ALBSingleSubnet feature gate, so a single-AZ cluster (subnets sharing one AZ) dedups to 1<2 subnets and aborts Ingress reconcile. Only the explicit-subnet path honors the gate. Resolve the cluster's ELB-eligible subnets, dedup to one per AZ, and surface them as EKS_ELB_SUBNET_IDS. The addon sync renders them into the alb IngressClassParams .spec.subnets.ids so every Ingress takes the explicit path. An empty set falls back to auto-discovery, preserving the native UX. * fix: route LBC service clients to gateway via --aws-api-endpoints, link IngressClassParams Live verification showed two gaps in the LBC addon bundle: LBC installs a per-client BaseEndpoint resolver, so the global AWS_ENDPOINT_URL is ignored for the ec2/elbv2/acm clients (calls hit real AWS, failing TLS). Redirect them with --aws-api-endpoints keyed on the aws-sdk-go-v2 service IDs (EC2, "Elastic Load Balancing v2", ACM, STS) rather than the legacy lowercase names the flag never matched. Drop the misleading AWS_ENDPOINT_URL env; the web-identity STS client still resolves via AWS_ENDPOINT_URL_STS. The injected IngressClassParams was orphaned: the alb IngressClass had no .spec.parameters, so LBC fell back to subnet auto-discovery (which fails on a single-AZ cluster). Link the IngressClass to the IngressClassParams so the explicit-subnet path is taken. * fix: stamp EKS-parity providerID on worker nodes for LBC ip targets k3s registers workers with providerID k3s://<name>, which the AWS Load Balancer Controller rejects (providerID invalid for EC2 instances) when resolving nodes for ip-mode target registration. Resolve the instance-id and AZ from IMDSv2 at agent boot and set --kubelet-arg=provider-id=aws:///<az>/<id> so LBC can map the node to its EC2 instance. On any IMDS miss the agent keeps the default providerID and still starts. * fix(elbv2): honor SubnetMappings in CreateLoadBalancer LBC supplies ALB subnets via SubnetMappings, not the plain Subnets list. createLoadBalancer only read input.Subnets, so ENI allocation was skipped, VpcId/AZ stayed empty, and the HAProxy data-plane VM never launched — the ALB went active but could not serve traffic. Add flattenSubnetIDs to read both sources and reuse it in createLoadBalancer and desiredSubnetSet. * fix(vm): capture GCInterval before spawning GC goroutine TestStart_SweepsImmediatelyThenStopsOnCtxCancel mutates the package global GCInterval and restores it in a defer, racing the Start goroutine's read at time.NewTicker(GCInterval) under -race. Read GCInterval on the caller's goroutine into a local before spawning so the global access is ordered with the test's defer; the goroutine touches only the captured local.	2026-06-18 10:53:05 +10:00
Josh Siviour	bc9556d1a9	fix(eks): managed-CP worker data-plane — apiserver reachability, node-local CoreDNS, SG teardown (#368) ... * fix: persist instance TagSpecifications in RunInstance RunInstance built the ec2.Instance metadata but never read input.TagSpecifications, so DescribeInstances returned untagged instances and tag filters matched nothing. EKS node groups discover their workers by the spinifex:eks-cluster / spinifex:eks-nodegroup tags, so scale-up convergence and worker reconciliation depended on tags that were silently dropped. Apply instance-scoped TagSpecifications onto the built instance so the tags surface through DescribeInstances and tag filters. * fix: serialize nodegroup reconcile via CAS to prevent scale overshoot UpdateNodegroupConfig is dispatched on a queue group, so two overlapping requests (e.g. an SDK retry mid-flight) each read current = len(InstanceIDs) and each launched desired-current workers, overshooting the target (1 -> 5 for desired=3) and stranding untracked VMs via last-writer-wins on the unconditional PutNodegroupRecord. Make every durable nodegroup-record mutation a compare-and-swap on the KV revision. Scale-up launches one worker at a time and CAS-appends it only while the record is below the live ScalingDesired, terminating any surplus VM launched into an already-full record. Scale-down terminates before the CAS truncation so a crash or retry leaves reclaimable dead IDs rather than orphaned VMs. A concurrent regression test asserts two overlapping scale-ups converge to exactly desired under -race. * fix: return securityGroupRuleSet from AuthorizeSecurityGroup ingress/egress The AWS provider's aws_vpc_security_group_ingress_rule reads SecurityGroupRules[0].SecurityGroupRuleId on create; an empty ruleset panics the plugin (Plugin did not respond), though the rule applies server-side. Echo the created rules via sgRuleToSecurityGroupRule, the same shape DescribeSecurityGroupRules already returns. * fix: surface InsufficientInstanceCapacity from nodegroup scale, not 500 launchWorkersCAS wrapped the bare awserror code via fmt.Errorf, so the gateway ErrorHandler's exact-match against ErrorLookup missed and remapped to InternalError (500). An out-of-capacity scale now surfaces the real InsufficientInstanceCapacity code (400); opaque internal failures still wrap and stay 500. Adds awserrors.HasErrorCode for the membership test. * fix: cut EKS node boot ~50s — drop cloud-final deadlock + syslinux menu Worker/server user-data runcmd ended with `[ rc-service, local, start ]`, which runs inside cloud-final while the OpenRC `local` service is ordered after it — a deadlock broken only by OpenRC's ~50s start timeout, stalling the k3s join. Run the IMDS route script directly instead and keep `rc-update add local default` for reboot persistence. setup.sh: cut the SYSLINUX boot-menu countdown 10s->~1s (kept nonzero so the menu stays interruptible over serial). Validated live on env19 CP: boot menu 1s, no `waiting for cloud-final` stall, apiserver ready ~66s (was ~109s+). mulga-siv-332 * feat: add argocd managed addon + bake bundle into eks-node AMI The addon catalog advertised coredns and aws-load-balancer-controller, but the only baked manifest shipped in the eks-node AMI was the spinifex-noop fixture. Installing any real addon failed with 'no baked bundle for version' once the VM-side renderer found nothing on disk. Add argocd v3.0.23 as a real, demonstrable managed addon: catalog entry (no IRSA), a flattened namespaced install manifest baked into the AMI, and the manifest.conf bake line. The eks-addons-access workbook switches from coredns (redundant on K3s, never baked) to argocd. The baked bundle is generated from upstream install.yaml via kubectl kustomize with namespace: argocd plus a prepended Namespace, since upstream is namespace-agnostic and K3s auto-deploy applies with no -n flag. * fix(eks): run CoreDNS as a node-local DaemonSet for worker DNS k3s ships CoreDNS as a single-replica Deployment that tolerates the CP taint and strands itself on the control-plane node. In the managed-CP topology the worker<->CP pod overlay cannot cross the VPC boundary, so worker pods cannot reach a CP-resident CoreDNS and lose all cluster + external DNS. Skip the bundled CoreDNS (coredns.yaml.skip, staged in k3s.initd like traefik) and deploy a Mulga-owned variant: a DaemonSet (one pod per node) plus kube-dns internalTrafficPolicy=Local, so every node resolves via its own CoreDNS. Immune to the cross-VPC overlay gap and topology-agnostic (single-node and managed-CP both work). Image-side only; no handler change. * fix(eks): revoke cross-referencing SG rules before delete on teardown DeleteClusterSGs deleted the control-plane and nodegroup SGs directly, but they cross-reference each other (EnsureNodegroupSGRules), and AWS refuses to delete a security group still referenced by another SG's rule. Both deletes failed with DependencyViolation, the SGs leaked, and the VPC could not be deleted (tofu destroy hung on DeleteVpc). Resolve both SG IDs, revoke all ingress on each to break the cp<->ng cycle, then delete both. Adds RevokeSecurityGroupIngress to sgProvisioner (already on VPCServiceImpl) and revokeAllSGIngress, with a regression test that models AWS's DependencyViolation. * fix(eks): advertise NLB front-end so worker pods reach the apiserver ClusterIP In the managed-control-plane topology the apiserver advertised its Set-B node-ip into the in-cluster `kubernetes` Endpoints. That IP lives in the unpeered CP VPC and is unreachable from worker pods, so every worker pod using the apiserver ClusterIP (CoreDNS kubernetes-plugin sync, controllers, argocd redis secret-init) failed. - k3s_server_vm.go: set advertise-address to the worker-reachable NLB front-end (PrivateEndpointIP, else EndpointIP); both are already cert SANs. - nlb.go: add a second :6443 listener on the cluster NLB (same CP target group) so PrivateEndpointIP:6443 — the advertised endpoint — is served. - security_groups.go: admit :6443 on the private-endpoint SG. Validated on a live managed-CP cluster: worker CoreDNS /ready 200, all argocd pods Running, kubectl logs on worker pods working.	2026-06-16 20:27:21 +10:00
Josh Siviour	c22a6f4ffc	fix: install chrony on host via curl|bash setup path (#366) ... The host clock needs an NTP daemon; without one it drifts and SigV4 request signatures fall outside the skew window, kicking the web console back to /login. Mirrors the ISO rootfs (iso-builder packages.list) so both host-install paths ship the same software. Guest VMs track the host via kvm-clock. Bug: mulga-siv-335	2026-06-16 16:03:34 +10:00
Josh Siviour	17c83cb3ad	fix: build microVM rootfs via buildx to stop overlay2 export leak (#358) ... docker export of a container leaks its overlay mount on Docker's overlay2 driver — the mount survives docker rm, so the layer dirs are never reaped except by a daemon restart. The lb-agent microVM build ran docker export on every e2e job across ~9 runners, accumulating ~69G of orphaned overlay2 layers on the CI host (disk hit 94%). Build the rootfs with `docker buildx --output type=local` from a small generated Dockerfile instead, which writes the image filesystem to a directory and releases all mounts — nothing leaks. Keep the podman path (its store is unaffected). Tighten the e2e cleanup image prune to dangling-only so the alpine base stays cached instead of re-pulling.	2026-06-15 10:50:10 +10:00
Bryn Mailer	f6f3956952	feat: enable viperblock at-rest encryption by default on fresh installs (#323) ... * feat: enable viperblock at-rest encryption by default on fresh installs Fresh installs left nodes.<node>.viperblock.encryption_key_file empty, which viperblockd treats as legacy cleartext mode. The encryption machinery existed but the install path never generated a key or set the field. Provision a cluster-wide viperblock master key in all three fresh-install flows and render its path into spinifex.toml: - admin init (single node): generate the key locally. - admin init --multi-node (leader): generate once, save locally, and distribute via the formation server (mirrors IAM master key distribution). - admin join (joiner): receive the shared key from the leader rather than minting its own, so volumes sealed on any node open everywhere. Lenient on absence — an older leader returns no key, so the joiner warns and stays cleartext instead of failing the join. The key file is omitted from the template when unset, so upgrades with no existing key keep running cleartext. * fix: make viperblock encryption key group-readable for both readers The key is loaded by two services per node — viperblockd (spinifex-viperblock) and the awsgw EC2 handlers (spinifex-gw), both in the spinifex group — so it must be group-readable, not owner-only like the per-node predastore key. The previous commit generated the key but never registered its path with the deployment ownership machinery, leaving /etc/spinifex/viperblock as root:root 0700 (non-traversable) and the key as root:root 0600. The daemon failed at startup with "stat ... permission denied". Mirror master.key: setup.sh creates /etc/spinifex/viperblock (0750), and SetServiceOwnership chowns the dir to spinifex-viperblock and promotes the key to root:spinifex 0640. * fix: read viperblock config.json through StateBody for encrypted volumes The metadata layer parsed config.json with json.Unmarshal/NewDecoder directly, which fails on encrypted volumes/AMIs now that viperblock wraps VBState in a valid-JSON envelope ({"v":1,"payload":...,"authtag":...}). DescribeImages silently skipped encrypted AMIs (count=0 → "no AMIs available"), bricking the whole launch path. Route every config.json read through viperblock.StateBody, which unwraps the envelope (no key needed — the metadata is authenticated, not encrypted) and is a no-op for unencrypted volumes: - image: DescribeImages, getVolumeConfig, amiNameExists, GetAMIConfig - volume: getVolumeConfigAndEncryption, and mergeVolumeConfig (so the refuse-to-merge-encrypted guard sees the real VBState instead of clobbering the envelope with a tag-less blob) - admin: readAMIConfig, readVolumeConfig	2026-06-15 09:31:23 +10:00
Josh Siviour	b23fa3bf60	feat(eks): managed control-plane VPC (Set B) + control-plane lifecycle hardening (#356) ... * feat(eks): VM-side managed-addon delivery transport Stage CreateAddon manifests in KV host-side; the control-plane VM pulls the staged set through the AWS gateway (signed GET /clusters/{name}/internal-addons/ {accountId}), renders the baked bundles into the K3s auto-deploy dir, and reports per-addon delivery status on a dedicated eks.addon.*.status channel that the cluster reconciler CASes onto AWS-visible AddonStatus (CREATING->ACTIVE). - gateway: ListInternalAddons route + ListStagedAddonManifests NATS action - handlers: addon status reconciler (nextAddonStatus), staging installer - eks-node AMI: eks-gateway-fetch binary + mulga-eks-addon-sync agent - agent renders idempotently (cmp before replace) so K3s auto-deploy does not re-apply every tick and clear the Addon CR .status.gvks readiness signal - agent mirrors per-tick delivery outcomes to the serial console for host-side observability * fix(eks): provision cluster NLB synchronously so the endpoint carries the real front-end IP The cluster NLB front-end IP is assigned by the async LB-VM launch (launchLBVMAsync), but EnsureClusterNLB read it back synchronously right after CreateLoadBalancer returned — losing the race and falling back to the synthetic <lb>.elb.spinifex.local DNS name. That name is unresolvable (no Route53) and is not in the apiserver serving-cert SAN, so kubectl against the cluster never connects. Split the elbv2 create: the public CreateLoadBalancer stays async (267.4 — an inline multi-minute VM boot head-of-line-blocks the gateway responder), and a new CreateLoadBalancerSync drives the data-plane launch inline and returns the LB carrying its front-end address. The EKS cluster-create worker runs off the gateway responder, so it uses the sync entry and reads the IP from the return value instead of racing persisted state. No polling. - elbv2: factor provisionLBDataPlane out of launchLBVMAsync; add CreateLoadBalancerSync - eks: ensureClusterLB uses the sync entry, sets FrontendIP from the result, drops the read-back; EnsureClusterNLB fails loud (ErrClusterNLBFrontendIPUnavailable) when no front-end IP is allocated rather than shipping the DNS-name fallback - eks: drop the DNS-name endpoint fallback; the endpoint is always the front-end IP * feat: auto-wire IGW for internet-facing EKS cluster endpoint An internet-facing cluster NLB allocates an external front-end IP and the OVN dnat_and_snat for it, but the external switch + localnet + gateway LRP that make that IP answerable on the wire are built only on IGW attach. The control-plane NLB is a system instance, so ensure an IGW on cluster build and reclaim it on teardown. EnsureClusterIGW (idempotent, reuses a customer IGW untouched) runs before EnsureClusterNLB when the endpoint is public; DeleteClusterIGW (ownership- scoped) removes only the cluster-owned IGW in purgeClusterInfra, before the VPC delete. New IGW dep wired from the daemon. * fix(eks): tear down infra on failed cluster launch to stop LB EIP leak A CreateCluster that fails during its background launch flipped the meta to FAILED but performed no teardown, relying on a later same-name create to reclaim and purge the infra. EKS cluster names are unique per create, so that reclaim never fires: the internet-facing load-balancer VM keeps running with its allocated+associated dnat_and_snat EIP indefinitely. Over repeated create/delete cycles the EIP pool fills with allocations bound to terminated instances, so a later allocation's advertised endpoint and its live OVN NAT diverge onto different pool addresses. - failClusterLaunch now runs a best-effort purgeClusterInfra on the in-memory meta (freshest refs even when the failing step was the meta write itself) before MarkClusterFailed. purgeClusterInfra is idempotent, so a later reclaim or DeleteCluster re-purge is a no-op; the FAILED meta is retained for observability. - Daemon releaseSystemInstanceEIP extracted and called from both TerminateSystemInstance and cleanupFailedSystemInstance, so a launch that fails after AssociateAddress releases the eipService allocation instead of leaking it (MarkFailed's ReleasePublicIP only knows the externalIPAM path). * feat: EKS managed control-plane VPC (AWS Set B) composed from EC2 APIs CreateCluster now auto-builds the managed control-plane VPC under the system account, the spinifex analogue of AWS EKS's hidden managed-account VPC: a public subnet for the internet-facing NLB, private subnet(s) for the k3s control-plane VM(s), and a NAT gateway giving the private CP egress for image pulls. It is composed from the real EC2 VPC-family APIs (CreateVpc/Subnet, IGW, RouteTable/Route, NatGateway), so per-subnet egress gating is wired by the existing topology subscribers off the route-table events those APIs publish — no direct OVN mutation. This resolves the mulga-siv-292 root cause: the internet-facing NLB and the private CP were crammed into one customer subnet with no IGW route, so both were treated as private and dropped by the lr_in_policy 1100 rule, killing the LB-VM return leg and lb-agent heartbeat. The CP only survived via a per-instance 1200 reroute bandaid, now removed — the topology expresses egress instead. Placement is uniform single/multi-node (CP always in the private subnet). The system LB VM heartbeats over the mgmt bridge (SystemGatewayURL) rather than the WAN. Managed-VPC teardown reclaims NAT GW + EIP, route tables, subnets, IGW, and the VPC in dependency order. Customer subnetIds (Set A / worker side) remain consumed-not-created and are deferred to the worker round. * fix(eks): bake cluster-owner account into CP-VM EKS_ACCOUNT_ID The managed control-plane VPC (Set B) launches the K3s control-plane VMs and their ENIs under the system (infra) account, but the on-VM agents — bootstrap publish, state report, token-review, and add-on fetch — must key on the cluster-OWNER (customer) account that owns the cluster meta, namespaces, and AccessEntry KV. Add ClusterAccountID to K3sServerInput and bake it into the VM's EKS_ACCOUNT_ID, decoupling it from the infra AccountID the VM launches under. Without this the VM reported state and fetched add-ons against the system account, so the cluster never observed a healthy control plane (stuck CREATING, null CA) and kubectl/add-on delivery timed out. Also refresh the e2e suite doc comment to describe the Set B topology and the data-plane subtests (kubectl + managed-addon delivery) that now run. * fix(eks): explicitly delete addon objects on unstage K3s does not delete the applied resources when an auto-deploy manifest file is removed from the server manifests dir (documented behaviour). The addon-sync GC pass relied on that, so after DeleteAddon the rendered manifest was dropped but its Namespace/ConfigMap stayed standing — the e2e AddonDelivery GC wait timed out. Copy the manifest aside, drop the original so K3s stops tracking it, then kubectl delete the objects (--wait=false to avoid finalizer stalls).	2026-06-12 22:29:24 +10:00
Julian Sommer	19472c377a	chore: delete legacy scripts	2026-06-12 15:17:48 +10:00
Julian Sommer	4adecc0794	chore: delete old scripts	2026-06-12 14:52:18 +10:00
Julian Sommer	ef86d09a42	chore: delete legacy scripts/iac	2026-06-12 14:51:38 +10:00
Josh Siviour	74b91b5cc5	feat(eks): HA control plane — 3-node etcd quorum spread across distinct hosts (#335) ... * feat(eks): node-targetable control-plane VM (HA spread transport prereq) Groundwork for HA control-plane placement (mulga-siv-231.7.1): make the k3s control-plane VM placeable on a specific Spinifex host. - Add a sys.medium instance type (2 vCPU / 4 GB) and boot the control-plane VM as it instead of t3.medium. A sys.* type keeps the CP VM out of customer DescribeInstanceTypes and makes the daemon register the node-targeted system.LaunchInstance.{type}.{nodeID} subject (only sys.* types get it). - Add K3sServerInput.TargetNodeID and a LaunchSystemInstanceOnNode method on the launcher: empty/local node launches in-process (unchanged single-CP path); any other node dispatches over the node-targeted subject, where the receiving daemon runs the launch and owns the VM. The spread orchestrator (reserve N hosts, launch per host, verify, rollback, <3-host fallback) lands separately; CreateCluster still launches locally (TargetNodeID empty), so behaviour is unchanged. * feat(eks): HA control-plane spread orchestrator Place N=3 k3s server VMs across distinct Spinifex hosts via a per-cluster spread placement group, with all-or-nothing rollback and single-CP fallback. - placeControlPlane: capacity fan-out (HostScheduler) -> >=3-host gate -> ensure spread group (system account) -> ReserveSpreadNodes(Min=Max=3) -> parallel per-node LaunchK3sServerVM(TargetNodeID) -> node-vms placement verify -> FinalizeSpreadInstances. Any launch/verify/finalize failure terminates launched VMs, releases the reservation, drops the group. - <3 schedulable hosts or a lost reservation race falls back to a single local control-plane VM (prior behaviour). - ClusterMeta gains ControlPlaneNodes[] + ControlPlaneSpreadGroup; [0] is mirrored into the scalar ControlPlane* fields for back-compat. DeleteCluster tears down every CP VM and the spread group. - Wire PlacementGroup + Scheduler into EKSServiceDeps + orchestration-deps gate. * fix(eks): size HA control-plane spread on raw host headroom SchedulableHosts matched the CP type by name in node.status's per-type capacity list, but that list omits sys.* types (system types are hidden from customers). The CP VM is sys.medium, so the match never hit and SchedulableHosts returned 0 on every node — placeControlPlane always fell back to a single CP, making the >=3-host spread unreachable in production. System VMs consume host vCPU/memory like any guest, so size the fit against each node's raw schedulable headroom (Total - Reserved - Alloc) and the type's footprint instead. Adds instancetypes.SpecForSystemType for the sys.* vCPU/memory spec and a pure nodeFitsSystemInstance helper. * feat(eks): k3s server-join role for HA control-plane quorum Servers 2..N now boot in join mode: server: https://<first-server>:6443 + a shared cluster token, without cluster-init, so they join the first server's embedded-etcd quorum instead of each cluster-initing an independent datastore. - K3sServerInput gains JoinToken (shared cluster secret, set on every server) and ServerURL (set => join server, no cluster-init, server-join role). - buildK3sUserData branches cluster-init vs server:, emits token:, and selects SPINIFEX_K3S_ROLE server vs server-join. - launchControlPlaneSpread launches nodes[0] first (cluster-init), derives the join URL from its ENI IP, then launches the rest in parallel as join servers; a first-server failure skips the joins and rolls back. - CreateCluster mints a per-cluster crypto-random token (GenerateK3sClusterToken) threaded into the placement template. - eks-node-role.sh adds a server-join case: token-webhook + k3s + state-report, but not k3s-first-boot (the first server already publishes the bootstrap artifacts; a join re-publish only races the bootstrap bus). * fix: open etcd-peer/kubelet ingress on EKS control-plane SG for HA join Join servers (231.7.2) register over the supervisor port 6443 but their embedded etcd could not peer with the quorum: the control-plane SG only admitted 6443 from the VPC CIDR, omitting etcd client+peer 2379-2380 and kubelet 10250 between control-plane nodes. A single cluster-init server never exercises those ports, so the gap stayed hidden until multi-server HA — the join registered but its etcd never replicated the raft log, leaving the node perpetually NotReady. Add EnsureControlPlaneHAIngress with self-referencing (cpSG->cpSG) rules for etcd 2379-2380 and kubelet 10250, called from CreateCluster after EnsureControlPlaneIngress. Verified live on a 3-node cluster: join servers reach Ready and form a 3-member etcd quorum that survives loss of the original first server. * feat: register all control-plane ENI IPs as EKS cluster NLB targets Register every HA control-plane server ENI IP in the cluster NLB target group, not just the primary. The TG already health-checks :6443, so a dead member drains and kubectl/apiserver traffic fails over to surviving etcd-quorum members. Add RegisterClusterTargets batch helper; the single RegisterClusterTarget now delegates to it.	2026-06-10 12:25:47 +10:00
Josh Siviour	7bcd365549	ci(eks-ami): build on self-hosted ci-single runner (#326) ... * ci(eks-ami): build on self-hosted ci-single-1 runner GitHub-hosted ubuntu-24.04 cannot run the libguestfs appliance network: passt fails (PID file denied under its AppArmor confine / privilege drop) and the SLIRP fallback has no outbound, so virt-customize --network dies. The self-hosted ci-single-1 runner has /dev/kvm and real networking, so the existing libguestfs build works unmodified. * fix(build-system-image): add --no-cache, force fresh build on publish The <10min cached-raw skip avoids duplicate work for concurrent e2e jobs, but on the persistent publish runner it could republish a stale image from a prior or concurrent run. Add --no-cache to bypass the skip and have publish-system-image.sh pass it on every build. * ci(eks-ami): target ci-single label (no ci-single-1 label exists) The self-hosted runners are named ci-single/-2/-3 but all share the single label 'ci-single'; runs-on matches labels, not names, so ci-single-1 never schedules. * ci(eks-ami): disable Go cache on self-hosted runner setup-go's cache-save (Post Setup Go) re-uploaded the Go cache to the Actions backend on every run, taking ~2.5min vs the 36s build. The self-hosted runner already keeps the cache on local disk, so cache:false.	2026-06-09 16:58:04 +10:00
Josh Siviour	2ea3e22db6	feat(systemd): cgroup-confine spinifex to a resource-limited slice + host swap (#324) ... * feat(systemd): bound spinifex to a resource-limited slice Add a spinifex.slice parent ceiling (MemoryHigh=80%, MemoryMax=90%, MemorySwapMax=0) with a spinifex-system.slice child holding the seven control/infra services, so a runaway hits the slice instead of the host. A system.slice drop-in (MemoryMin=1G, CPUWeight=300) reserves RAM and CPU for sshd/journald/the operator, keeping the box reachable when spinifex is maxed. The drop-in is generated in setup.sh install_systemd() rather than shipped as a staged file, since the packaging globs flatten the systemd/ dir; this mirrors the existing sshd-keygen drop-in pattern. The two .slice files flow through every staging path unchanged. Within the single effective cgroup, a slice MemoryMax OOM picks the highest oom_score victim — guest QEMU (+500) before any service (-500) — so a slice OOM degrades by killing one guest while the daemon, storage tier, NATS, and the host all survive. * feat(setup): provision 8G host swapfile + vm.swappiness=10 Add a setup_swap stage that creates an 8G /swapfile, enables it, persists it to fstab, and writes vm.swappiness=10 so the kernel only swaps under real pressure. Backs spinifex.slice MemorySwapMax so the slice can spill to swap for graceful degradation instead of OOMing a guest. Idempotent; skipped under ISO_BUILD (firstboot provisions swap on ISO hosts). Reverses the prior swap=0 assumption.	2026-06-09 15:21:13 +10:00
Josh Siviour	4ef19fc488	feat(ui): EKS console (#310)	2026-06-09 10:57:50 +10:00
Josh Siviour	467e4de715	feat(eks): publish eks-node AMI to R2 + spx catalogue entry (#313) ... * feat(eks): publish eks-node AMI to R2 + spx catalogue entry Add a spinifex-eks-node catalogue entry so operators install the EKS node image with a single command: spx admin images import --name spinifex-eks-node The entry pulls the Alpine+K3s image from iso.mulgadc.com, verifies its sha256, and stamps spinifex:managed-by=eks — the tag the control-plane and node-group launchers resolve the boot AMI by, so no extra config is needed. Add scripts/publish-system-image.sh (+ make publish-eks-node-image) to convert the built raw to a compressed qcow2, emit a checksum sidecar, and upload both to the R2 bucket backing iso.mulgadc.com/system-ami via the S3 API. * ci(eks): add manual workflow to build + publish eks-node AMI to R2 Build the eks-node AMI on the runner (libguestfs) and publish via publish-system-image.sh using the existing CF_R2_* secrets. Manual workflow_dispatch with a dry-run input. Fix the publish script's default bucket to spinifex-iso (the bucket iso.mulgadc.com is served from).	2026-06-08 09:48:59 +10:00
Josh Siviour	44d0548fd4	ci: make microVM depmod best-effort, drop apt kmod install (#308) ... build-microvm-image required depmod (kmod) as a mandatory tool to generate modules.dep. Self-hosted CI runners don't ship kmod and can't reliably apt-install it, so the inline 'apt-get install kmod' steps added to e2e/e2e-nightly/e2e-ddil were unreliable. modules.dep is an optimisation, not a boot requirement: the guest init's load_mod() (build/microvm/init.sh) falls back to insmod-by-path when modprobe can't resolve a module. So: - Drop depmod from the mandatory-tool preflight. - Run depmod only when present; still assert a non-empty modules.dep in that case to keep catching the trixie-kmod empty-dep bug. - Warn and skip when depmod is absent (guest uses insmod fallback). - Remove the apt-get kmod install steps from all three workflows. Gold-image bake still installs kmod (provision.sh), so production images get a proper modules.dep; only kmod-less CI runners take the fallback.	2026-06-07 22:10:56 +10:00
Josh Siviour	527308c342	fix(baremetal): empty modules.dep + capacity-aware single-node e2e (#305) ... * feat(elbv2): localize internal-LB heartbeat failures for baremetal debug Add diagnostics so a stuck-in-provisioning internal LB can be pinned to one of three points on the heartbeat path: - lb-agent failure log now carries the gateway target + region, so the microVM serial console shows whether the heartbeat hit a transport error (never reached AWSGW) or an HTTP-level rejection. - AWSGW LBAgentHeartbeat logs receipt (debug) and warns on the previously-silent not-found/account-mismatch reject path, so the journal distinguishes 'heartbeat arrived but cannot transition' from 'heartbeat never arrived'. * test(e2e/lb): fail fast on stuck internal LB + probe fw_cfg on microvm boot - lb suite: gate the remaining internal subtests on Internal_ALB. The internal LBs share one mgmt-return datapath and fixture, so if the first never reaches active the rest time out identically — skip them instead of burning ~5m per subtest (~20m saved on a known-bad run). - microvm init.sh: widen the fw_cfg netcfg poll window and, on the not-available path, dump DMI / loaded modules / qemu_fw_cfg by_name listing / dmesg fw_cfg to the serial console before init exits, so a boot panic is diagnosable from the captured console log. * fix(microvm): insmod fallback for fw_cfg + fail build on bad modules.dep Baremetal-built microVM images shipped an unusable modules.dep (the build host's depmod errors were swallowed), so busybox modprobe in the guest init silently no-oped, qemu_fw_cfg never loaded, /sys/firmware/qemu_fw_cfg was absent, init could not read opt/spinifex/netcfg, and the LB microVM panicked before lb-agent ran — leaving internal LBs stuck in provisioning. init.sh: load_mod() tries modprobe, then falls back to insmod by explicit path (.ko/.ko.gz) which needs no modules.dep, so the guest boots regardless of which host built the image. build-microvm-image.sh: require depmod, drop the error swallow, and assert modules.dep is non-empty after depmod so a boot-broken image fails the build loudly instead of silently. * fix(microvm): decompress kernel modules before depmod Debian trixie's kmod 34 depmod silently emits an empty modules.dep when fed Alpine's gzip-compressed (.ko.gz) modules — reproduced: 928 modules in, 0-byte modules.dep out. The guest's busybox modprobe then no-ops, qemu_fw_cfg never loads, and internal LB microVMs panic before lb-agent runs (stuck provisioning on bare metal; GitHub-hosted runners ship a kmod that reads .gz, so normal CI was unaffected). Decompress kept modules to plain .ko after the keep-list strip, before depmod. Host depmod then produces a usable map regardless of its gzip support, and the guest loads modules without a decompressor. * feat(e2e): capacity-aware scheduling for dense single-node runs Denominate daemon schedulable capacity in physical cores instead of logical threads, and auto-serialise e2e launches when the cluster is at capacity, so a single hyperthreaded node stops over-packing guests. Daemon: NewResourceManager sizes hostVCPU from physicalCoreCount() (distinct physical-id:core-id pairs in /proc/cpuinfo) rather than runtime.NumCPU(). Falls back to logical CPUs when topology is absent (non-Linux, container, ARM cpuinfo without core id). On a hyperthreaded box this halves the over-pack: with the unchanged 2 vCPU reserve the i5-12600H schedules 10 vCPU (12 cores - 2) instead of 14, keeping the boot-spike CPU within real cores. Harness: capacityRetryEC2 decorates the EC2 client and, on InsufficientInstanceCapacity, waits and retries RunInstances instead of failing — turning the daemon admission gate into back-pressure so concurrent subtests queue. AWSClient.EC2 is now the wrapped ec2iface.EC2API; EC2Conf keeps the raw *ec2.EC2 for .Config access. Adds unit tests for parsePhysicalCores (hyperthreaded/hybrid/dual-socket/ fallback) and the retry decorator (retry-then-succeed, non-capacity passthrough, deadline). * fix(daemon): SPINIFEX_HOST_VCPU override so small CI hosts can start Denominating capacity by physical cores starves tiny hosts: a 2-physical- core CI runner leaves 0 schedulable after the default 2-vCPU reserve, so NewResourceManager refuses to start and every daemon-constructing unit test fails. The previous runtime.NumCPU() count (4 logical on the same runner) incidentally cleared the reserve floor. Add a SPINIFEX_HOST_VCPU escape hatch (symmetric with the existing SPINIFEX_RESERVED_* overrides) that pins the detected core count for hosts whose /proc/cpuinfo topology misreports, and pin it to 4 in the daemon TestMain so unit tests are deterministic regardless of runner topology. Reserve stays at the default 2.	2026-06-07 21:34:39 +10:00
Josh Siviour	496424c41c	test(eks): EKS control-plane E2E suite (Sprint 6g) (#304) ... * test(eks): EKS control-plane e2e suite (Phase 1, API-level) Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw: CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone. Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted} pollers. Wire e2e-eks into the Makefile and the single-node CI suite list. * test(eks): Phase 2 kubectl get-nodes data-plane gate Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper (harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the control-plane node is Ready — the get-token webhook auth regression gate. IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up. * test(eks): tolerate doubled NotFound code + dial loopback NATS DeleteCluster detection matched the SDK constant ResourceNotFoundException, but awsgw emits ResourceNotFoundExceptionException on the wire — match the NotFound code by substring so deletion is detected immediately instead of timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which the serving cert does not SAN; rewrite the wildcard bind to loopback. Validated live: all 6 subtests green, kubectl get nodes shows the control-plane node Ready via the get-token webhook auth path. * feat: build system images with libguestfs (no host block devices) Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in build-system-image.sh with libguestfs (virt-customize / virt-resize / guestfish). Every operation runs in an isolated userspace appliance VM with its own kernel, so the build touches no host block device, no host mount, and needs no sudo — removing the failure class that wiped the host's /dev on shared CI runners (a leaked /dev bind traversed by rm -rf). Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu via virt-resize; customization (packages, binaries, files, setup script, service enable, cache cleanup) is one ordered virt-customize invocation. LIBGUESTFS_BACKEND=direct (no libvirt on runners). Wire the eks-node AMI into the single-node e2e job: the runner builds the device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are VM-only) so the eks suite's CreateCluster has an AMI to boot. * feat(eks): bootstrap + state-report via AWS gateway broker Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model: the K3s server VM publishes its bootstrap envelopes and periodic state reports via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS directly. A new gateway action (PublishInternal, POST /clusters/{name}/internal-publish) relays the payload verbatim onto the existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV persistence/encryption, and reconciler are unchanged. This removes the VM's need to reach NATS at all, so NATS can stay cluster-internal behind its bind/firewall/mTLS controls — the prior direct path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was DDIL-fragile (fire-and-forget nats pub, no ack/retry). - gateway/eks: PublishInternal broker + route; closed-set channel/kind validation - cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and mulga-eks-state-report.sh - handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL + system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA; new resolveSystemGatewayBaseURL (mgmt-bridge-first) - eks-node image: build/install eks-gateway-publish, drop nats-cli download * feat: migrate eks-token-webhook to the AWS gateway broker The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that fails, so the webhook exited at startup, the apiserver token-webhook kubeconfig was never written, and the apiserver crash-looped — the cluster never reached ACTIVE. Converge it onto the same gateway-broker model as bootstrap/state-report and ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to /clusters/{name}/token-review, and the gateway runs the verify + lookup host-side and returns the resolved identity. The VM now speaks no core NATS. - internal/eksgw: shared on-VM SigV4 client (also adopted by eks-gateway-publish). - handlers/eks/token_review.go: Authenticate decision core (moved from the webhook binary) + ResolveTokenReview host-side wiring. - gateway/eks token-review route + WebhookTokenReview handler. - webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable) not a silent deny. - initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*. * fix: build eks-node helpers with GOFIPS140; cover token-review broker eks-gateway-publish blank-imports internal/fipsboot, whose init() panics unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build omitted the flag, so the binary paniced at first boot right after k3s came up — k3s-first-boot (set -eu) died before publishing the bootstrap token and the cluster never reached ACTIVE. Build both eks-node helpers with GOFIPS140=v1.0.0, matching cmd/lb-agent. Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview gateway handler (verify responder + account KV) to cover the new broker path. * fix: export first-boot env so eks-gateway-publish child sees it k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/ first-boot.env but never exported the vars, so the eks-gateway-publish child process inherited an empty environment and exited '--account-id is required'. The control-plane VM came up healthy but never published its bootstrap envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a / set +a so the KEY=value lines export to child processes. * fix: make awsgw start --host override actually bind The 'awsgw start --host' override printed 'Overwriting awsgw host' but the assignment was commented out, so AWSGW always bound the spinifex.toml host (a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW listens on all interfaces incl. br-mgmt; the serving cert already SANs every local interface IP, and WAN clients still reach it. * test(eks): install kubectl on e2e VM so KubectlGetNodes runs The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker -> STS verify path — the only end-to-end exercise of the token-review broker. The e2e binaries run on the VM, where kubectl was absent, so harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a checksum-verified kubectl (pinned to the k3s server version) on the VM, gated on e2e-eks.	2026-06-07 20:42:25 +10:00
Josh Siviour	5aaf8c23d2	fix(eks): make aws eks get-token → kubectl auth work end-to-end (Sprint 6f) (#303) ... * feat(eks): webhook observability — console log sink + per-request TokenReview decision * fix(eks): wire token-webhook authenticator into generated k3s apiserver config The generated /etc/rancher/k3s/config.yaml overrides the AMI-baked config.yaml.skel, but its kube-apiserver-arg list omitted the authentication-token-webhook-config-file arg. The running apiserver therefore never loaded the token webhook, so bearer tokens minted by 'aws eks get-token' 401'd without the webhook ever being invoked. * fix(eks): echo TokenReview audiences so api-audiences apiserver accepts the principal With --api-audiences set, kube-apiserver sends spec.audiences in the TokenReview and discards an authenticated=true response that does not confirm a matching audience. The webhook returned an empty status.audiences, so a valid IAM principal surfaced as a 401. Echo the request's audiences on success. * fix(eks): pin token-webhook version v1 so apiserver accepts the response kube-apiserver --authentication-token-webhook-version defaults to v1beta1, but the eks-token-webhook responds with an authentication.k8s.io/v1 TokenReview. The apiserver decodes the response against the expected GVK and rejects the mismatch, surfacing a valid principal as a 401. Pin v1 to match the webhook.	2026-06-06 22:11:50 +10:00
Josh Siviour	a22bc2f716	feat(eks): 6d nodegroup provisioning + NATS-gated cluster health (#300) ... * feat(eks): nodegroup provisioning + K3s agent join (6d) CreateNodegroup launches N customer-owned worker EC2 from the unified eks-node AMI in agent role; each joins the cluster via K3S_URL + K3S_TOKEN baked into cloud-init (first-boot.env SPINIFEX_K3S_ROLE=agent + agent.env). Workers are customer-owned and DescribeInstances-visible (no ManagedBy, no mgmt NIC) via the normal RunInstances path, matching real EKS. K3S_URL targets the control-plane ENI's in-VPC private IP on :6443 — the cluster NLB DNS endpoint is not resolvable in-VPC yet; switch to the NLB endpoint on :443 once authoritative in-VPC DNS lands. K3S_TOKEN is the decrypted cluster node-token persisted at server bootstrap. Describe/List/Update(scaling)/Delete round out CRUD. Cluster<->nodegroup SG ingress rules (apiserver 6443, flannel VXLAN 8472/udp, kubelet 10250, intra-nodegroup) authorized idempotently. UpdateNodegroupVersion deferred. * feat: gate EKS cluster health via NATS state reports The K3s apiserver binds the VPC node-ip, unreachable from the host-side spinifex daemon, so the reconciler cannot probe /healthz directly. Add a control-plane state reporter that publishes {healthz, node_count} over the management NATS bus the daemon already shares. - eks-node server role runs mulga-eks-state-report as an OpenRC service, publishing eks.state.{account}.{cluster}.server every 30s. - ClusterReconciler subscribes to that subject, gates CREATING -> ACTIVE on a healthy report, and records Ready node_count into ClusterMeta. - node_count counts Ready nodes only; k3s retains terminated workers as NotReady until pruned, so a raw count never drops on scale-down.	2026-06-06 00:52:18 +10:00
joshsiv-mulga	da26b1db69	fix(ci): stop image-build /dev bind from wiping host /dev ... A killed ubuntu system-image build can leave `mount --bind /dev` under its build dir. The e2e Clean Workspace step then ran `sudo rm -rf "$GITHUB_WORKSPACE"/*` as root, traversed the leaked bind, and deleted the host's /dev (incl. /dev/null), wedging the runner. - e2e.yml: umount anything under the workspace (deepest-first) before deleting, and pass `rm --one-file-system` so rm can never cross a mount boundary into host /dev. Hard backstop, applied to all 3 Clean Workspace steps. - build-system-image.sh: recursive lazy unmount in cleanup() detaches the chroot binds even when busy, and mark the build subtree rprivate so the /dev bind can't propagate into the host namespace.	2026-06-05 22:52:49 +10:00
Josh Siviour	f9c08130ba	fix(eks): embedded etcd datastore + Alpine IMDS-route fix + CP-VM visibility (#299) ... * fix: switch EKS datastore to embedded etcd + fix Alpine cloud-init IMDS route Flip the K3s control-plane datastore from SQLite/kine back to embedded etcd (cluster-init: true, etcd-expose-metrics: true). The earlier abandonment premise — that etcd's per-commit fsync stalls past the apiserver's 5s deadline on the viperblock-backed root volume — was wrong: etcd inits fine on an unchanged viperblock (apiserver ready 25-30s, clusters reach ACTIVE). The real blocker was a network-config regression, fixed here. Alpine's cloud-init eni renderer crashes (KeyError: 'gateway') on the gateway-less IMDS on-link route, which failed the whole network-config render so the static mgmt NIC never came up (no NATS bootstrap, no /healthz). Fixes: - generateNetworkConfig: add emitIMDSRoute bool; selectNetworkConfigForFamily passes false for Alpine so the eni renderer never sees the route. - Deliver the IMDS route out-of-band via a persistent /etc/local.d script + OpenRC local service. Resolve the VPC NIC via its default route at runtime (ip route show default), not a hardcoded name: the persistent-net rename to vpc0 doesn't apply on the Alpine AMI, so the NIC is eth0 at first boot. - EKS path bakes the same route into buildK3sUserData's own write_files/runcmd; the generic template Alpine block fires only when no #cloud-config user-data, avoiding a duplicate write_files key (yaml.safe_load keeps last, silently dropping a block). - eks-node image: config.yaml.skel adds cluster-init; comments updated. * fix: surface CreateCluster step failures at ERROR + map subnet faults; add auth-mode/delete test cases CreateCluster wrapped its step failures in bare fmt.Errorf that the request surface collapses to an opaque ServerInternal ("contact AWS re:Post"), so the real cause never reached the operator. Add logCreateErr to log each step's precise cause at ERROR before returning, and map a subnet-resolve failure (caller-supplied parameter) to InvalidParameterValue instead of a retryable ServerInternal the SDK pointlessly retries. (mulga-siv-165.4) Tests (mulga-siv-165.15): reject authenticationMode=API_AND_CONFIG_MAP (only the CONFIG_MAP value was covered); DeleteCluster on a nonexistent cluster via the full-deps path returns ResourceNotFound (the shim path short-circuits to ServiceUnavailable before the meta lookup). * fix: hide EKS control-plane VMs from customer DescribeInstances EKS K3s control-plane VMs are owned by the customer account (their ENI lives in the customer VPC for the NLB target), so unlike LB microVMs (owned by the system account) they leaked into customer ec2:DescribeInstances. The spinifex:managed-by tag only filtered the UI Nodes page, not the EC2 API. Add a caller-aware exclusion: skip any ManagedBy-tagged VM unless the caller is root. Preserves operator visibility, matches LB behaviour, and leaves future customer-owned worker nodes (no ManagedBy) visible. * docs: correct stale SQLite datastore comments in eks-node image (now embedded etcd)	2026-06-05 17:55:31 +10:00
joshsiv-mulga	8e38faaede	fix(elbv2): make NLB nginx health-check pathway robust in CI ... E2E NLB suites timed out at "0/2 healthy" while ALB suites on the same target instances went healthy in ~5s. The host daemon journal shows the NLB agent reporting an empty server list (no probe results at all), which happens when the lb-agent never enters nginx mode — i.e. `nginx -t`/start fails closed inside the LB microVM, so the agent never probes and reports nothing. The data plane itself is verified working; this is purely the active-health-check pathway failing to converge in CI. - lbagent: pre-create nginx runtime prefix dirs (pid dir + /var/lib/nginx, /var/lib/nginx/tmp, /var/log/nginx) before `nginx -t`, mirroring reloadHAProxy's socket-dir pre-create. A stripped Alpine initramfs may lack these, dead-ending the whole NLB health path. - build-microvm-image.sh: assert the nginx binary and ngx_stream_module.so are present in the image and fail the build loudly, instead of surfacing as an opaque e2e health timeout. - e2e: capture the LB VM serial console to the artifact bundle on NLB health-wait failure. The agent's engine activation and any `reload nginx:` error land on the guest console, which the host journal never sees.	2026-06-05 08:14:18 +10:00
Josh Siviour	554c5d96c7	feat(elbv2): NLB L4 data plane on nginx stream + UI parity (#295) ... * feat(elbv2): NLB L4 data plane on nginx stream (mulga-siv-952) Network load balancers render an nginx `stream` config (TCP/UDP/TLS/ TCP_UDP) instead of HAProxy, which cannot load-balance UDP in any release. The gateway tags GetLBConfig with Engine=nginx; the LB agent dispatches to the nginx engine (config/pid/cert paths + reload) and skips HAProxy stats polling. ALB stays on HAProxy. nginx is added to the microvm image. * docs(elbv2): note NLB runs nginx stream, ALB runs HAProxy (mulga-siv-952) * fix(elbv2): load nginx stream module for NLB data plane Alpine's `nginx` apk is built without the stream module — it ships separately as `nginx-mod-stream`. Without it, the generated NLB config fails `nginx -t` with [emerg] unknown directive "stream", so the agent never applies it. Add the package to the microvm image and emit a `load_module .../ngx_stream_module.so;` directive at the top of the generated config. * fix(elbv2-ui): allow single-subnet LB + full NLB protocols in listener edit Create-LB form required min 2 subnets while the CLI/backend accept 1; relax to min(1) to match. Listener add/edit only offered HTTP/HTTPS — add the NLB protocol set (TCP, UDP, TLS, TCP_UDP), driven by LB type, with cert/ssl-policy fields gated on secure protocols (HTTPS/TLS). * feat(elbv2): active target health checks for nginx NLBs nginx stream has no active upstream probing, so NLB targets never advanced past initial — DescribeTargetHealth stayed initial forever and WaitForTargetsHealthy timed out for every NLB. GetLBConfig now delivers per-target HealthTarget specs (daemon-computed server name, probe address, protocol/path) for network LBs; the lb-agent probes each target every tick (TCP dial or HTTP/HTTPS GET) and reports UP/DOWN via the existing heartbeat path, so the daemon's threshold hysteresis advances targets to healthy. ALBs are unchanged (HAProxy stats). Adds an Internal_NLB_UDP e2e suite gating on WaitForTargetsHealthy plus a UDP datagram round-trip through the VIP.	2026-06-05 00:11:51 +10:00
Josh Siviour	f14fb64562	feat(eks): unify eks-server + eks-agent into one eks-node AMI (#288) ... * feat(eks): unify eks-server + eks-agent into one eks-node AMI Collapse the two K3s AMIs into a single spinifex-eks-node image whose role (server vs agent) is chosen per-instance at first boot, halving build time, storage, and CVE-rotation surface. - scripts/images/eks-node/: superset manifest + merged setup.sh; all role services baked but only crond + the new eks-node-role selector enabled. - eks-node-role first-boot selector reads SPINIFEX_K3S_ROLE, rc-update adds + starts the matching services (server: k3s + eks-token-webhook + k3s-first-boot publisher; agent: k3s-agent), records /etc/spinifex-eks/role, self-disables. Paths are env-overridable; eks-node-role_test.sh covers all five cases. - mulga-eks-etcd-snapshot.sh early-exits on agent nodes. - k3s_server_vm.go seeds SPINIFEX_K3S_ROLE=server into first-boot.env. - Makefile/dev-install: build-eks-node-image / import-eks-node-image replace the server+agent targets. - Delete scripts/images/eks-server + eks-agent. The launcher resolves the AMI by the spinifex:managed-by=eks tag, so running clusters (which hold the resolved AMI ID) are unaffected. AMI bake + cluster boot are operational integration steps, not yet run. * fix(eks): order eks-node-role after cloud-config/cloud-final The first-boot role selector reads /etc/spinifex-eks/first-boot.env, which cloud-init delivers via write_files in the cloud-config/cloud-final stages. The selector only depended on net+cgroups, so on a fast boot it could start before write_files landed and read a missing role env. Add an after-ordering on cloud-config + cloud-final so the env file is always present first.	2026-06-04 12:28:56 +10:00
Josh Siviour	bfacca7789	feat(eks): 6c token webhook + AccessEntry (IAM auth → kubectl) (#284) ... * feat(eks): 6c AccessEntry CRUD + access-policy association Replace the nine notImpl() AccessEntry/AccessPolicy stubs with real KV-backed handlers (Q9, API auth mode): - AccessEntry CRUD persisted at clusters/{name}/access-entries/{hash}; the principal ARN is hashed into the key (ARNs carry ':', illegal in NATS KV keys) while the record keeps the plaintext ARN. - AssociateAccessPolicy/Disassociate/ListAssociated against the four supported AWS-managed policies; unsupported policy ARN or bad access scope -> InvalidParameterValue. - ListAccessPolicies returns the static supported catalogue. - Cluster-existence gate -> ResourceNotFoundException; duplicate create -> ResourceInUseException; node-type entries rejected until 6d. Updates the daemon routing test for the now-implemented handlers. * feat(eks): plumb caller principal ARN + bootstrap-creator-admin Thread the caller's resolved IAM principal ARN from the gateway to the EKS control plane via a new X-Principal-ARN NATS header, so CreateCluster can mint the bootstrapClusterCreatorAdminPermissions AccessEntry (Q9): - utils.NATSRequest takes optional extra headers; PrincipalARNHeader + PrincipalARNFromMsg added. - Gateway resolves the caller ARN best-effort (buildCallerARN) and passes it down the EKS route table; only CreateCluster consumes it. - CreateCluster signature gains callerPrincipalARN; default-true bootstrap flag mints a system:masters entry keyed by the caller's exact ARN (the token webhook resolves the same ARN from STS). '' degrades gracefully. - Dedicated daemon handler reads both headers. * feat(eks): IAM token webhook + STS verify-over-NATS + apiserver wiring The eks-token-webhook (baked into the eks-server AMI) authenticates kubectl requests that present an `aws eks get-token` bearer token: decode the presigned STS GetCallerIdentity URL, verify it, resolve the principal ARN to its cluster AccessEntry, and answer the apiserver TokenReview with the mapped username + groups. STS is hosted in-process in the awsgw service and is not otherwise on the bus, so awsgw now serves an `eks.VerifyToken` NATS subject that wraps VerifyPresignedGetCallerIdentity (the cluster name is forwarded so STS enforces the x-k8s-aws-id anti-replay pin). The webhook reads the AccessEntry KV directly over its own NATS connection. Apiserver wiring: the k3s config skeleton sets authentication-token-webhook-config-file; the webhook writes its own self-signed serving cert + the apiserver kubeconfig at startup (ordered `before k3s`, with a bounded wait in the k3s start_pre so the apiserver never loads a missing file). * fix(eks): omit empty spec from token webhook TokenReview response * fix(sts): presigned GetCallerIdentity must hash empty body, not UNSIGNED-PAYLOAD VerifyPresignedGetCallerIdentity built the SigV4 canonical request with "UNSIGNED-PAYLOAD" as the payload hash. That is an S3-only convention; botocore and the aws-sdk presign non-S3 requests (sts:GetCallerIdentity, the shape `aws eks get-token` produces) with the SHA256 of the empty string. Every real get-token therefore failed signature verification with InvalidIdentityToken, breaking the entire EKS IAM token-auth flow end to end. Use the empty-string SHA256. The existing hand-rolled presign tests masked the bug by signing with the same wrong constant; they now sign correctly, and a new test presigns with the real aws-sdk-go v4 signer (the same canonicalisation the CLI uses) so this can't regress silently. Verified live: a real `aws eks get-token --cluster-name dev1` now resolves to the caller principal over the awsgw eks.VerifyToken surface; an otherwise-valid token presented for a different cluster is still rejected (anti-replay intact). * fix(eks): match access-entry routes on escaped path so slashed principal ARNs resolve IAM principal ARNs (user/admin, role/foo) embed a '/' that the AWS CLI percent-encodes in the request path. The dispatcher matched r.URL.Path (decoded), so %2F became a literal '/' and split the ARN across the ([^/]+) capture, dropping every describe/update/delete-access-entry and associate/disassociate/list-access-policy call to InvalidActionException. Match r.URL.EscapedPath() so the encoded ARN stays one segment, then url.PathUnescape each captured param before it reaches the handler. * test(eks): unit-cover access-entry scope validation, AWS converters, store guards	2026-06-04 11:25:56 +10:00
Josh Siviour	b0227e7eeb	chore(ansible): remove ansible tree, uplift to mulga/ansible/ (#287) ... Ansible orchestrates the whole stack (spinifex + predastore + viperblock + OVN + tofu hosts) and its spec/beads live in mulga. Moving it to top-level mulga/ansible/ ends the per-change submodule-pointer-bump friction. - git rm -r scripts/ansible/ - Makefile: drop ansible-dev-* / ansible-cluster-bootstrap targets + .PHONY The tree lands at mulga/ansible/ (flat copy, Option B). History pivot: tag feat/uplift-ansible-snapshot-pre-move marks the pre-move HEAD. Plan: docs/development/improvements/ansible-uplift-to-mulga.md (in mulga).	2026-06-04 11:23:08 +10:00
Josh Siviour	bbff004b45	feat(eks): 6b create-cluster → ACTIVE (single-node SQLite + mgmt-IP NLB stop-gap) (#280) ... * feat(eks): cluster_state KV helpers — ClusterMeta + status CAS + prefix sweep Stage 1 of Sprint 6b CreateCluster lifecycle. Introduces the first read/write helpers against the eks-account-{accountID} bucket so later stages (oidc keypair, NLB wiring, cluster reconciler) can persist state without re-inventing CAS/sweep logic. - ClusterMeta struct holds the DescribeCluster source of truth (status, endpoint, OIDC issuer, CA, VPC config, control-plane VM + ENI, NLB ARN). - ClusterStatus enum mirrors the AWS EKS string set verbatim. - PutClusterMeta is unconditional (CreateCluster path); GetClusterMeta returns ErrClusterNotFound on key absence so the service boundary can map to ResourceNotFoundException. - SetClusterStatus is revision-checked with bounded CAS retries so the reconciler and CreateCluster body cannot wedge each other. - DeleteClusterPrefix sweeps every clusters/{name}/* key; called from DeleteCluster after the OIDC private key is zeroized. Tests cover round-trip, missing/corrupt blob, idempotent status set, DELETING transition from every prior state, concurrent revision bump recovery, sibling-cluster preservation, empty-bucket no-op. Plan: docs/development/feature/eks-6b-create-cluster.md * feat(eks): per-cluster ECDSA-P256 OIDC keypair generation + zeroize Stage 2 of Sprint 6b. Lays the at-rest primitives the CreateCluster body will call before launching the K3s server VM. Pure crypto helpers — no daemon, no launcher, no NATS bus. - GenerateClusterOIDCKeypair mints an ECDSA-P256 keypair per cluster (matches eks-v1.md Q8), PEM-marshals the private as PKCS8, envelope-encrypts it via handlers_iam.EncryptSecret (AES-256-GCM, AwsGW masterKey), writes the ciphertext to OIDCSigningKeyKey, and publishes the public JWKS document to OIDCJWKSKey. Returns the JWKS bytes so CreateCluster can hand them straight to user-data without a follow-up KV read. - JWKS shape mirrors what handlers_sts.JWKS already decodes for IRSA verification (RFC 7517 EC P-256, kid = base64url SHA-256 of SPKI DER, alg ES256, use sig). Types are defined locally to avoid a handlers_eks → handlers_sts → handlers_eks import cycle; JSON wire compat is the contract. - LoadClusterOIDCPrivateKey is the recovery-path read for future stages (reconciler restart, restore-snapshot tooling). Validates PKCS8 parses to an ECDSA P-256 *PrivateKey and rejects anything else. - ZeroizeClusterOIDCKey overwrites the encrypted blob with zeros of the same length, then deletes. DeleteCluster calls this *first* so a crash mid-teardown leaves no recoverable key material. Idempotent on missing key — supports retry semantics. Tests cover JWKS shape (RFC 7517 EC P-256 with 32-byte SEC1 coordinates), per-cluster key uniqueness, round-trip private-key decrypt matches JWKS public, wrong-master-key fails, missing-key ErrClusterNotFound mapping, zeroize idempotency, JWKS survives zeroize (carries no secret material), padCoord left-pad correctness. Plan: docs/development/feature/eks-6b-create-cluster.md * feat(eks): cluster SG + NLB ensure/delete helpers Stage 3 of Sprint 6b CreateCluster lifecycle. Adds the customer-account SG provisioning + per-cluster NLB plumbing the CreateCluster body will call between OIDC keypair (Stage 2) and K3s VM launch (Stage 4). Pure interface-call helpers — no daemon, no KV, no NATS bus. - EnsureClusterSGs creates (or returns) eks-cluster-{name}-control-plane-sg and eks-cluster-{name}-nodegroup-sg in the customer VPC, tagged spinifex:managed-by=eks + spinifex:eks-cluster=<name> + spinifex:eks-role. Idempotent on group-name + vpc-id pair. DeleteClusterSGs is best-effort with continue-on-error sweep. - EnsureClusterNLB provisions the eks-<name> NLB (internal scheme), the eks-<name>-cp target group (TCP, port 6443, target-type ip per eks-v1.md Q13), and the TCP :443 forward listener. Returns ClusterNLB {LoadBalancerArn, TargetGroupArn, ListenerArn, DNSName} so CreateCluster can persist every ARN in ClusterMeta. Idempotent on cluster name; resource-name length validated against the AWS 32-char cap so a too-long cluster name fails with a clear error instead of a generic InvalidParameterValue downstream. - RegisterClusterTarget + DeregisterClusterTarget bind the K3s server VM ENI IP to the TG on :6443. Stage 4 calls Register once the VM has its ENI IP; DeleteCluster calls Deregister before VM termination. - DeleteClusterNLB tears down NLB + TG. DeleteLoadBalancer already cascades listener cleanup inside the ELBv2 impl. LB sweep error takes precedence so DeleteCluster surfaces the more visible failure first while the TG sweep still runs. Narrow sgProvisioner + nlbProvisioner interfaces in-package give tests hand-rolled recorder fakes; the production VPCService + ELBv2Service satisfy them structurally so no adapter is needed at Stage 6 wire-up. Tests cover fresh-creates with right names/tags/ports, idempotent reuse, mixed-existence (control-plane exists, nodegroup created), name-too-long rejection, empty-inputs rejection, missing-resource no-op on delete, and first-error-surfaced sweep continuation on delete. Plan: docs/development/feature/eks-6b-create-cluster.md * feat(ec2): RunInstances honors pre-created NetworkInterfaceId Closes the AWS-parity gap where RunInstances always auto-created a primary ENI even when the caller specified NetworkInterfaces[0].NetworkInterfaceId. The EKS Sprint 6b CreateCluster launcher (Stage 4) needs to pre-create the K3s server VM's ENI so the ENI can be tagged/owned distinctly from the launcher's instance; the existing path silently created a second ENI and left the caller's unattached. Changes: - ENICreator interface gains GetENI(accountID, eniID) → *ENIInfo. ENIInfo is a translated struct (SubnetID, VpcID, PrivateIp, MAC, Status, SecurityGroupIDs) so handlers/ec2/instance stays free of an import on handlers/ec2/vpc.ENIRecord. - daemon/vpc_adapter.go gains daemonENICreator.GetENI translating VPCServiceImpl.GetENIRecord into ENIInfo. - PrepareRunInstances inspects NetworkInterfaces[0].NetworkInterfaceId after the existing subnet/SG lift block. If non-empty, the loop dispatches to a new attachPreCreatedENI helper: - Looks up the ENI under accountID. - Rejects with InvalidNetworkInterface.InUse if Status == "in-use". - Calls AttachENI to mark the ENI in-use and bind it to the new VM. - Populates instance.ENIId / ENIMac and ec2.Instance network fields (PrivateIp, SubnetId, VpcId, SecurityGroups, NetworkInterfaces[0]). - Skips the auto-create + public-IP-auto-assign paths; the caller is expected to manage public IP out-of-band. - Auto-create path is left untouched for every other code path: callers that pass SubnetId without a NetworkInterfaceId still get a fresh ENI with the existing semantics. Tests cover the four shapes: happy path (auto-create not called, attach called once, instance populated), in-use ENI rejected with the canonical AWS error code, lookup error surfaces verbatim, attach error surfaces verbatim. * feat(eks): K3s control-plane VM launcher with cloud-init user-data LaunchK3sServerVM resolves the eks-server AMI by name, pre-creates the customer-account control-plane ENI in the supplied subnet with the control-plane SG, renders cloud-init user-data (first-boot env, OIDC private key, K3s config.yaml with tls-san + service-account args), then calls RunInstances with NetworkInterfaces[0].NetworkInterfaceId pointing at the pre-created ENI. RunInstances failures roll back the ENI so the caller does not leak a customer-account resource. TerminateK3sServerVM tears the instance and ENI down for DeleteCluster; missing identifiers are a no-op so retries stay safe. ENI + VM both live in the customer account (deviation from plan §7 which specified a SystemAccount-owned VM); cross-account ENI attach is deferred to a follow-up bead. * feat(eks): NATS bootstrap subscriber + cluster reconciler + leader lease NATSBootstrap subscribes to the four single-shot subjects the K3s server VM publishes (k3s-bootstrap-token, k3s-admin-kubeconfig, k3s-oidc-jwks, k3s-ca) on eks.bus.{accountID}.{clusterName}.{kind}. Token and admin kubeconfig are envelope-encrypted via handlers_iam.EncryptSecret and persisted to KV. The published JWKS is cross-checked against the controller-generated keypair by kid and rejected on mismatch. The CA lands on meta.CertificateAuthorityB64 via the new SetClusterCertificateAuthority CAS updater. ClusterReconciler holds a leader lease on spinifex-eks-leader/{accountID}/ {clusterName} (60s TTL, 20s refresh) and drives: - CREATING -> ACTIVE once the four bootstrap KV artifacts are present and NLB /healthz responds 200. - ACTIVE -> probe /healthz on the configured interval, warn on failure. - DELETING / FAILED -> exit (DeleteCluster body owns teardown). Loss of the leader lease (CAS conflict or different holderID) returns ErrReconcilerLeaseLost so the caller can re-acquire from another node. Tested with embedded JetStream + a stub HTTPDoer; race-clean. * feat(eks): OIDC issuer URL builder + reconciler registry ClusterOIDCIssuer renders the {gatewayBaseURL}/oidc/eks/{region}/{accountID}/ {clusterName} URL kube-apiserver advertises as service-account-issuer and that STS / external IRSA validators dereference. Validates URL safety on the path segments so a hostile cluster name cannot inject scheme/path fragments. ReconcilerRegistry tracks per-(accountID, clusterName) reconciler goroutines with idempotent Spawn (drop duplicate spawn calls), Stop (cancel one + wait for release), and StopAll (graceful daemon shutdown). RunClusterReconciler is the production SpawnReconcilerFn: constructs a ClusterReconciler, acquires the leader lease, runs Run in a goroutine. Both helpers consumed by the upcoming service_impl CreateCluster body. * feat(eks): CreateCluster/DescribeCluster/ListClusters/DeleteCluster bodies Wires the EKSServiceImpl orchestration deps (VPC SG/K3s/Subnet, NLB, Instance launcher, AMI resolver, master key, gateway base URL, region, holder ID, reconciler registry) and implements the four lifecycle methods: - CreateCluster: validate → CREATING meta → SGs → NLB → OIDC issuer + keypair → K3s server VM (cloud-init carries OIDC private PEM) → register ENI IP with NLB target group → spawn NATS bootstrap subscriber + per-cluster reconciler via the registry. - DescribeCluster: KV meta read → eks.Cluster shape, with ErrClusterNotFound translated to ResourceNotFoundException at the service boundary. - ListClusters: KV key-prefix scan + sorted maxResults/nextToken pagination. - DeleteCluster: SetClusterStatus DELETING → zeroize OIDC key → stop reconciler via registry → deregister NLB target + delete NLB → terminate K3s VM → delete SGs → delete cluster KV prefix. Daemon wiring: - buildEKSServiceDeps loads MasterKey from BaseDir/config/master.key best-effort, resolves the gateway base URL via the same precedence the LB-agent block applies later, and passes VPCService + ELBv2Service + ImageService directly while adapting InstanceServiceImpl via daemonEKSInstanceLauncher (RunInstances passes through; Terminate- Instances loops vmMgr.Get + StopOrTerminateInstance in-process). - SubnetVPCResolver is satisfied by daemonEKSSubnetResolver wrapping VPCService.GetSubnet. - Startup spawns reconcilers for every persisted CREATING/ACTIVE cluster via SpawnRegisteredReconcilers; shutdown calls eksService.Shutdown so every per-cluster goroutine releases its lease before exit. Test surface: - service_impl_test: TestEKSServiceImpl_ClusterLifecycleShimMode covers the four lifecycle methods when constructed via the back-compat shim (CreateCluster/DeleteCluster → ServerInternal, DescribeCluster → ResourceNotFound on empty bucket, ListClusters → empty list). - daemon_handlers_eks_test: drop the four lifecycle subjects from the NotImplemented routing loop; handler-side coverage now lives in handlers/eks. - awserrors: add ResourceNotFound (404) for the DescribeCluster path. * fix(eks): load master.key from config dir not service data dir eks_deps read master.key from {BaseDir}/config/master.key, but the daemon systemd unit sets SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex (a per-service data dir). admin init writes master.key to the shared config dir, so the daemon looked in a path that only resolved when a {BaseDir}/config symlink existed — awsgw has one, the daemon's nested data dir does not. Result: MasterKey=nil, depsReadyForOrchestration() false, CreateCluster/DeleteCluster collapse to ServerInternal. master.key is a config artifact written next to spinifex.toml. Read it from filepath.Dir(d.configPath), matching how the daemon already resolves predastore.toml and TLS certs. No symlink or provisioning change needed. * fix(eks): register + resolve eks-server AMI via tag, add import --ami-name CreateCluster's K3s launch failed 'eks: AMI eks-server not found' on every clean install. Two defects: 1. spx admin images import had no way to name a locally-built appliance AMI: --name is a catalog lookup (rejects non-catalog values) and the AMI name was hardcoded to ami-{distro}-{version}-{arch}. build-system-image.sh ran 'import --name $AMI_NAME' for eks-server/eks-agent/GPU images, which all failed 'Image name not found in available images'. 2. The launcher resolved DescribeImages by name=='eks-server', a name the pipeline could never produce. Add --ami-name to override the registered AMI name on --file imports (fixes eks-server, eks-agent, and GPU appliance imports). Resolve the EKS AMI by the spinifex:managed-by=eks tag the pipeline already stamps, newest-wins on multiple matches. Tag-based resolution survives the planned server+agent → single EKS AMI unify untouched (role chosen per-instance at launch). build-system-image.sh now passes --ami-name; Makefile gains import-eks-server-image (IMPORT=1) and dev-install.sh imports the AMI when DEV_INSTALL_EKS=1. * fix(eks): give K3s server VM NATS token + CA so bootstrap can publish The K3s control-plane VM was launched with NATSURL=ConnectedUrl() (loopback, unreachable from the VM) and no auth material, while spinifex NATS requires a shared token over TLS. The VM could not connect to NATS, so it never published its four one-shot bootstrap messages and the cluster stayed in CREATING. Hand the VM the daemon's shared NATS token + CA PEM and a VM-reachable NATS URL (mirrors the ELBv2 LB-agent shared-credential pattern; spinifex has no nkeys hierarchy to scope per-principal creds against). - K3sServerInput: drop dead NATSCredsContent; add NATSToken + NATSCACert, required by validateK3sServerInput. user-data writes SPINIFEX_NATS_TOKEN, SPINIFEX_NATS_CA + the CA file; first-boot.env tightened to 0600. - EKSServiceDeps + CreateCluster: pass NATSURL/NATSToken/NATSCACert. - buildEKSServiceDeps: read CA from config.NATS.CACert; extract resolveGatewayHost(); add resolveNATSURL() (same reachable host, NATS port). - k3s-first-boot.sh: build nats CLI args with --token + --tlsca. * fix(eks): retry transient bootstrap persists, fail cluster on permanent error The NATS bootstrap subscriber tore down on the first persist error and the cluster sat in CREATING forever. Core-NATS delivers a publication once, so a transient KV blip during persist was unrecoverable and a malformed payload (bad JWKS kid, non-base64 CA, empty envelope) silently stranded the cluster. - persistWithRetry rides out transient persist failures in-handler (3 attempts, 200ms backoff) instead of unsubscribing and waiting for a re-publish that never comes. - errBootstrapPermanent classifies validation failures (malformed envelope, empty fields, JWKS kid mismatch, non-base64 CA) as non-retryable so they return immediately. - spawnBootstrap marks the cluster FAILED (with reason) when Run exits with a non-context error, so DescribeCluster surfaces the cause and DeleteCluster can reclaim resources. - MarkClusterFailed transitions to FAILED+StatusReason only from CREATING, so a late bootstrap error cannot clobber a concurrent delete or healthy cluster. - casUpdateMeta extracts the revision-checked RMW loop shared by the meta setters. * fix(eks): resume NATS bootstrap for CREATING clusters on daemon restart SpawnRegisteredReconcilers re-spawned only the reconciler on boot, never the bootstrap subscriber. A daemon restart while a cluster was CREATING (K3s VM still booting) lost the subscriptions; the VM published its four one-shot core-NATS messages to nobody and the cluster hung in CREATING forever. - BootstrapPendingKinds reports which artifacts (token, kubeconfig, CA) are still missing from KV. JWKS is excluded: it persists no artifact and re-waiting on a once-published subject would deadlock — its cross-check is best-effort across a restart. - RunForKinds subscribes to a subject subset (nil = all four); Run delegates to it. subjectHandlers maps each kind to its handler and rejects unknown kinds. - SpawnRegisteredReconcilers resumes bootstrap for CREATING clusters, waiting only on the missing artifacts, so a mid-CREATING restart still drives the cluster to ACTIVE. * fix(eks): DeleteCluster must not wipe KV or report success on teardown failure DeleteCluster ran zeroize, NLB teardown, and VM termination as warn-and-continue then unconditionally swept the per-cluster KV prefix and returned success. A failed NLB or VM teardown leaked billable resources with no owning record (the meta holding their ARNs/IDs was erased) and told the operator the cluster was gone. Zeroize — a stated security guarantee of no recoverable key material — had its failure ignored too. - Zeroize, NLB delete, and VM terminate are now blocking: their failures accumulate (errors.Join) and abort the call before the KV sweep, leaving the cluster DELETING and returning the error so a retry can still reach the ARNs. - KV prefix sweep runs only after NLB + VM teardown confirm. - SG cleanup stays best-effort (plan step 5) — a leaked SG strands no owning record — and target deregister stays best-effort since DeleteClusterNLB tears down the whole NLB + target group. * fix(eks): persist NLB + control-plane ARNs mid-CreateCluster so failures don't leak CreateCluster set meta.NLBArn / target-group / instance / ENI in memory and persisted them only in the final PutClusterMeta, after LaunchK3sServerVM. Any failure between EnsureClusterNLB and that persist left the NLB + target group + listener + backing LB VM created but unrecorded; markFailed did no rollback and DeleteCluster keys teardown off the persisted ARNs, so the resources leaked. - Persist meta immediately after EnsureClusterNLB (records NLB/TG ARNs) and again after LaunchK3sServerVM (records instance + ENI), before the next fallible step. - A failed create now leaves a FAILED meta carrying the resource references, so the DeleteCluster teardown path (165.9) reclaims every NLB/TG/listener/VM. - Tests: NLB-arn-persisted-before-later-failure, failed-create-then-delete reclaims NLB, and a CreateCluster happy path. Shared eksServiceFixture extracted from the delete-cluster fixture. * fix(sts): parse path-based EKS OIDC issuer so IRSA accepts 6b cluster tokens ParseEKSIssuerURL required an `oidc.eks.` host prefix and an exactly-2-segment path (AWS hosted-EKS shape), but ClusterOIDCIssuer emits — and awsgw serves — https://{gateway-host}/oidc/eks/{region}/{accountID}/{clusterName}. Every 6b cluster's iss claim was rejected with "issuer host must start with oidc.eks", so AssumeRoleWithWebIdentity would fail all IRSA tokens once 6e serves them. Reconcile on the path-based shape, which is what is actually reachable on bare-metal/on-prem (no oidc.eks.* vhost or DNS exists). Host trust comes from the IAM OIDC-provider registration (the full issuer string is the registry key) and the JWKS signature check, not a host pattern — so the parser drops the host prefix rule and instead pins accountID/clusterName at fixed positions in a strict 5-segment /oidc/eks/{region}/{accountID}/{clusterName} path. A crafted iss with extra or missing segments still fails closed. Adds a cross-component round-trip test asserting a ClusterOIDCIssuer-built iss parses back to the same (accountID, clusterName). * fix(eks): reclaim FAILED clusters on retry + cluster-scoped exists/not-found errors A failed CreateCluster left a FAILED ClusterMeta that poisoned retries: the preflight returned ErrorELBv2TargetGroupInUse ("The target group is currently in use...") for a cluster-exists condition, and because that maps to a retryable code the SDK's auto-retry surfaced ResourceInUseException instead of the original failure. FAILED clusters were also a dead state — visible to neither a clean recreate nor a clear lifecycle. - Add EKS-specific ResourceInUseException / ResourceNotFoundException codes with cluster-scoped messages; CreateCluster's exists check and DescribeCluster / DeleteCluster not-found now use them instead of the ELBv2 / EC2 codes. - CreateCluster reclaims a FAILED cluster: it purges the prior attempt's recorded resources and clears KV before recreating, so a retry (including the SDK auto-retry) is idempotent rather than blocked. Teardown failure surfaces rather than recreating on top of orphans. - Extract purgeClusterInfra, shared by DeleteCluster and the reclaim path (blocking zeroize/NLB/VM, best-effort SG, KV sweep only after teardown confirms). - FAILED clusters remain observable via Describe (status=FAILED) and List — no state where create blocks but describe/list show nothing. * fix(eks): fail CREATING clusters that exceed create timeout A per-cluster reconciler that never observes a ready control plane (bootstrap artifacts missing or healthz unreachable) previously stayed CREATING forever. Add a configurable create timeout (default 15m): once meta.CreatedAt is older than the deadline and the cluster is still not ready, the reconciler marks it FAILED with a reason, exiting the loop. Within the deadline it stays CREATING. * fix(eks): surface ACTIVE cluster health in describe-cluster The reconciler only slog.Warn'd a failing /healthz probe for an ACTIVE cluster, so describe-cluster kept reporting a dead control plane as healthy. Add HealthIssue + LastHealthProbe to ClusterMeta, write them from the ACTIVE reconcile branch (recording the failure reason, clearing on recovery), and map a non-empty HealthIssue to a ClusterHealth ClusterUnreachable issue in describe-cluster. * test(eks): add plan-mandated CreateCluster/DeleteCluster service cases service_impl_test only covered shim mode. Add the four cases the plan's Testing section requires: CONFIG_MAP auth mode rejected (and API accepted), missing subnetIds rejected, DescribeCluster 404 with full deps wired (past the shim short-circuit), and the DeleteCluster zeroize-before-teardown ordering proven by forcing a teardown failure and asserting the OIDC key is already wiped. * refactor(eks): return OIDC private-key PEM from generator, drop re-read CreateCluster discarded GenerateClusterOIDCKeypair's return then did a second KV get + AES-GCM decrypt via loadOIDCPrivateKeyPEM to re-derive the same PEM the generator already had in hand. Return the plaintext PEM (alongside the JWKS) from the generator and use it directly; remove loadOIDCPrivateKeyPEM and the now -unused crypto/x509 + encoding/pem imports in service_impl. * fix(eks): make CreateCluster/DeleteCluster failures diagnosable Unmet orchestration deps and a missing eks-server AMI both collapsed to a bare ServerInternal with no request-time log, leaving the client a 'contact AWS re:Post' message and zero cause. Enumerate the specific missing deps and log them at ERROR, returning ServiceUnavailable (an operator/config fault, not an unrecoverable internal one). Add an ErrEKSServerAMINotFound sentinel so a missing control-plane AMI is logged and surfaced as ServiceUnavailable too. * fix(eks): serialize REST-JSON responses via jsonutil so the AWS SDK can parse them WriteJSONResponse marshaled aws-sdk-go *Output structs with encoding/json, which ignores their locationName tags and emits Go PascalCase keys. EKS clients parse restjson on the lowercase locationName, so every field of list/describe/create/delete responses was silently dropped (empty endpoint, null identity.oidc.issuer). Marshal via jsonutil.BuildJSON — the same marshaler the SDK uses for restjson bodies — which honors locationName. Adds wire-shape tests asserting lowercase keys present and PascalCase absent. * feat(eks): launch K3s server VM via boot-agnostic system-instance path The EKS K3s control-plane VM launched through the plain tenant RunInstances path, so it had only its VPC ENI and no management-bridge NIC -- no route to the daemon's NATS endpoint. It never published its bootstrap subjects and the cluster stayed CREATING. Generalize LaunchSystemInstance to a boot-agnostic contract supporting both system-instance boot styles: - BootDirect: direct-boot microVM (fw_cfg blobs) -- ELBv2 LB agent - BootAMI: AMI boot + cloud-init seed (q35) -- EKS K3s server VM The launch contract moves to a neutral handlers/sysinstance package so EKS can launch system VMs without an eks->elbv2 dependency; handlers/elbv2 re-exports the types as aliases. The BootAMI branch mirrors the daemon's Prepare -> Insert -> Launch split and allocates the mgmt NIC on the VM record between Insert and Launch, so the cloud-init network-config renders a static mgmt0 interface. The EKS NATSURL is now sourced from the mgmt-bridge IP (the on-link address the VM's mgmt NIC can reach), falling back to the gateway-host resolution when no mgmt bridge exists. Verified: K3s VM now boots from the eks-server AMI under q35 with a mgmt0 NIC (10.15.8.x), reachable from the host over br-mgmt, with NATS on-link at the bridge IP. No AMI rebuild required. * fix(eks): make DeleteCluster K3s teardown idempotent so it can't wedge TerminateK3sServerVM treated every DeleteNetworkInterface error -- and every TerminateSystemInstance error -- as blocking. Because instance terminate is async, the immediate ENI delete races it: the first attempt fails InvalidNetworkInterface.InUse, and once the instance-terminate cascade releases the ENI every retry fails InvalidNetworkInterfaceID.NotFound. Both joined into purgeClusterInfra's teardownErrs gate, which short-circuits before the SG sweep and KV prefix delete -- so once the ENI was gone the cluster was stuck DELETING permanently, leaking its control-plane + nodegroup SGs and KV meta. Classify the teardown errors instead of blanket-propagating: - ENI InvalidNetworkInterfaceID.NotFound / InvalidNetworkInterface.NotFound -> idempotent success (the cascade or a prior retry already deleted it); must not block the downstream SG + KV sweep. - ENI InvalidNetworkInterface.InUse (VM still terminating) and any other error -> retryable; surfaced so the cluster stays DELETING and the reconciler retries once the instance releases the ENI. - Instance already gone -> idempotent. TerminateSystemInstance now returns a sysinstance.ErrSystemInstanceNotFound sentinel that the EKS path tolerates via errors.Is, so a retried delete after the VM drained completes. Add awserrors.IsErrorCode(err, code) as the shared delete-error classifier (substring match through %w wrapping) and route the lone ad-hoc strings.Contains guard in daemon/vm_adapters.go through it. Verified: delete-cluster on three clusters left wedged DELETING / FAILED (ENI + VM already gone) all complete -- clusters erased, SGs + ENIs reclaimed, no leak. * fix(eks): gate ACTIVE on verified JWKS cross-check, not pre-seeded key The controller pre-seeds OIDCJWKSKey at create time, so its presence proved nothing about the running cluster's actual signing key. A cluster could reach ACTIVE — and start signing IRSA tokens — before the K3s VM's published JWKS was ever cross-checked against the generated keypair. Add an OIDCJWKSVerifiedKey marker written by persistJWKS only after assertJWKSMatch passes. The reconciler's ACTIVE gate and BootstrapPendingKinds now key off the marker, not the pre-seeded JWKS. assertJWKSMatch is map-based and also rejects kty mismatch. * fix(eks): collapse duplicated gateway-host switch into resolveGatewayHost startCluster reimplemented the 5-case gateway-host precedence inline, duplicating resolveGatewayHost (used by the OIDC issuer and EKS NATS URL). If one drifted, the OIDC issuer host embedded in cluster tokens would diverge from the host LB VMs use to reach the daemon -> IRSA iss mismatch and cluster-wide token rejection. startCluster now calls resolveGatewayHost and derives the multi-node mgmt host route (mgmtRouteVia) from the resolved host. resolveGatewayHost carries the canonical case enumeration; a table test pins all five branches plus the case-1 skip guards and the no-host fallthrough. * fix(eks): make eks-server VM boot diagnosable + fix first-boot publisher crash Three issues found while diagnosing why K3s clusters never leave CREATING: 1. setup.sh: the stock Alpine cloud image lists `console=tty0` LAST in default_kernel_opts, so Linux binds /dev/console to the framebuffer and the host-side serial capture sees only kernel dmesg — OpenRC, cloud-init, and k3s-first-boot output were invisible. Reorder so console=ttyS0 is last. 2. k3s-first-boot.initd: command_background was "no" while output_log/error_log were set; start-stop-daemon rejects --stdout/--stderr without --background ("Failed to start k3s-first-boot"), so the bootstrap publisher never ran and the cluster wedged in CREATING. Background it (the publisher waits up to 5min on K3s /healthz and must not block the runlevel anyway) and route its output to the serial console, which the orchestrator captures (the VM has no login). 3. k3s-first-boot.sh: on the /healthz timeout, dump the tail of /var/log/k3s.log to the console so the apiserver's own failure is visible host-side. * fix(eks): seed K3s apiserver a public OIDC key, not the private key buildK3sUserData pointed both kube-apiserver service-account args at the same private-key file. --service-account-key-file requires a PUBLIC key — k8s ParsePublicKeysPEM rejects a private-key PEM with "data does not contain any valid RSA or ECDSA public keys" and the apiserver crash-loops, so K3s never reaches /healthz, first-boot never publishes, and the cluster wedges in CREATING. Add PublicKeyPEMFromPrivate (PKIX public PEM from the PKCS#8 private key) and seed it to a separate /etc/rancher/k3s/oidc-signing-key.pub.pem. Point service-account-key-file at the public file; keep service-account-signing-key-file on the private file. Runtime-verified: the apiserver crash is gone and K3s now starts and schedules system pods. * feat(eks): egress-only SNAT + DNS for K3s control-plane VM The K3s server VM (a hidden system instance) could not pull system-pod images: it had no internet egress and no working resolver, so coredns/traefik/metrics-server never started and /healthz timed out. Egress: give the single control-plane VM egress-only internet, confined to its /32, with no inbound exposure and no customer-visible resource. - policy/route.go: SystemInstanceEgressPriority (1200, above the drop gate) + AddSystemInstanceEgress/Delete — a reroute scoped to ip4.src == <vmIP>/32 so peers in a shared subnet are unaffected. - policy/nat.go: AddSystemInstanceSNAT/Delete — a plain snat (not dnat_and_snat, so no inbound path), idempotent on its pool external IP. - external/igw.go: L5 EnsureSystemInstanceEgress/Remove resolves the gateway nexthop and attaches the reroute + snat (mirrors the NAT-GW egress path; ADR-0006 S1/S4/S5 preserved). - subscribers: vpc.add/delete-system-egress topics + handlers. - handlers/eks: allocate a hidden (ManagedBy=eks) pool address on CreateCluster and publish add; release + publish delete on teardown. EIP dep wired through EKSServiceDeps + the daemon. DNS: the eks-server AMI is Alpine/dhcpcd and /etc/resolv.conf is a dangling symlink, so DHCP-supplied resolvers are never persisted. Write a real /etc/resolv.conf from the K3s VM cloud-config via bootcmd (not write_files — a write_files entry follows the dead symlink, fails, and aborts the whole block, dropping first-boot.env + the k3s config). * feat: EKS 6b single-node SQLite pivot + bootstrap-envelope fixes Switch the K3s server VM from embedded etcd to embedded SQLite (kine) for the single-node v1, so CreateCluster reaches a healthy control plane. etcd's per-commit fsync stalls past the apiserver deadline on viperblock's deferred-durability block path; SQLite tolerates it. - k3s server: omit cluster-init (selects SQLite/kine), add anonymous-auth=true so the reconciler's anonymous /healthz NLB probe returns 200. - first-boot readiness gate: anonymous /healthz (401 under CIS anonymous-auth =false) -> authenticated 'kubectl get --raw /readyz'. - first-boot publish: emit a JSON BootstrapEnvelope for all four subjects (token, admin-kubeconfig, oidc-jwks, ca); jwks and ca were absent before. - nats pub: --stdin -> --force-stdin (nats CLI 0.4.0). - manifest: add jq (builds the bootstrap-message JSON envelopes). * feat: EKS reconciler probes control-plane mgmt IP until DNS lands The host-side cluster reconciler cannot resolve or route to the VPC-internal NLB DNS endpoint (no authoritative DNS / Eclipso yet), so CreateCluster timed out in CREATING and was marked FAILED even though the control plane was healthy. Probe the control-plane VM's br-mgmt NIC address directly — the one apiserver endpoint reachable from the daemon. Capture the mgmt IP from the launch result (SystemInstanceOutput.MgmtIP -> K3sServerOutput.MgmtIP -> ClusterMeta. ControlPlaneMgmtIP) and build the reconciler health URL as https://<mgmtIP>:6443/healthz. The probe client already sets InsecureSkipVerify, so the mgmt IP need not be in the apiserver cert SAN. Workaround until authoritative DNS + a host-reachable NLB front-end land, at which point spawnReconciler reverts to the NLB endpoint.	2026-06-03 22:29:31 +10:00
joshsiv-mulga	3de1d6d7c5	fix(microvm): build initramfs device nodes via fakeroot, drop sudo mknod ... sudo mknod created real /dev/console and /dev/null char-device nodes on the host under $CHROOT_DIR. Real device nodes on the host filesystem are a footgun: a stale-image cleanup, tmp reaper, or an empty $CHROOT_DIR (which would redirect the recursive /dev removal at the host root) could clobber host /dev/null and force a reboot, breaking CI servers. Write the two static nodes the kernel needs into the cpio archive inside a fakeroot session instead. mknod/cpio see faked device entries and emit real char-device records into the archive, while the host filesystem never gets an actual node. init mounts devtmpfs over /dev at runtime for everything else. Also guard $CHROOT_DIR against empty/unexpected values before any destructive removal, and add fakeroot to the required-tools check.	2026-06-03 11:04:51 +10:00
Julian Sommer	044d35737e	feat: remove bash e2e for e2e-go	2026-06-03 10:32:30 +10:00
Julian Sommer	9508aaacce	chore: bump go 1.26.4	2026-06-03 08:59:25 +10:00
joshsiv-mulga	41f840ee85	fix(setup): grant spinifex-vpcd sudo for ovn-appctl (siv-167) ... The W2 inject-garp fix landed in 72a6a59d adds a shellout to `ovn-appctl -t ovn-controller inject-garp <port>` via the L0 execRunner, which prefixes `sudo` when not running as root. spinifex-vpcd runs as user spinifex-vpcd, so every emission was hitting `sudo: a password is required` and silently returning a WARN that nobody read until W5 diag capture surfaced the journal. Net: W2 has been a complete no-op since it shipped. Every recycled-EIP scenario has continued to leave stale upstream ARP. Evidence: diag bundle from run 26813235295 — 12/12 GARP attempts denied by sudo, 0 successful emissions; pcap shows runner ARP for recycled EIP .232 stuck on the prior VPC's gw chassisredirect MAC (02:75:d9:ef:d8:4b) while OVN NB has handed .232 to a new VPC with a different gw MAC; pcap also shows runner sending 174 SYNs to recycled EIP .233 with 0 SYN+ACKs received. Fix: add /usr/bin/ovn-appctl to the spinifex-vpcd NOPASSWD grant. Trivial; ovn-appctl is already trusted via the existing NB/SB read-write grants. Follow-ups (separate change): tighten host.InjectGARP error classification so sudo-deny is loud (Error or fatal) rather than silently warn-and-continue; add a boot-time self-check that verifies sudo permissions for every host-tool the daemon will shell out to.	2026-06-02 21:00:20 +10:00
joshsiv-mulga	2574956f08	ansible cluster-bootstrap	2026-06-02 15:38:56 +10:00
Josh Siviour	8883984f3f	feat(eks): eks-server + eks-agent AMI manifests (#273) ... * feat(eks): eks-server + eks-agent AMI manifests + builder INSTALL_FILES Manifest-driven AMI build for the two EKS image variants (eks-v1.md Q12). Lifted forward from cs-eks-6b so the K3s boot path can be smoke-tested in isolation before CreateCluster orchestration lands. Builder: - build-system-image.sh gains optional INSTALL_FILES (src:dst pairs, mode 0644, parent dirs auto-created). Symmetric with INSTALL_BINARIES. - ENABLE_SERVICES moves after INSTALL_FILES + SETUP_SCRIPT so init scripts the manifest references are on disk before rc-update enables them. eks-server image (Alpine 3.21.7, q35, 4G): - K3s v1.32.5+k3s1 (downloaded at bake time, SHA256 verified against the upstream sha256sum-amd64.txt trust anchor). - nats-cli v0.4.0 (not in Alpine main/community — pulled from upstream release zip; unzip installed + purged in same setup step). - OpenRC init scripts: k3s, eks-token-webhook, k3s-first-boot. - Cron periodics under /etc/periodic/{15min,daily}/ for state-report + etcd-snapshot. - k3s-first-boot publishes node-token + admin kubeconfig over NATS on first successful K3s healthz, then self-disables (sentinel + rc-update del). - config.yaml.skel without webhook-token-auth — wiring it before the real webhook lands (cs-eks-6c) would block anonymous probes against /healthz. Orchestrator drops the live config + restarts k3s. eks-agent image (Alpine 3.21.7, q35, 2G): - Same pinned K3s release for server/agent lock-step. - k3s-agent OpenRC service refuses to start without K3S_URL+K3S_TOKEN — fail-closed pre-orchestration. eks-token-webhook stub (cmd/eks-token-webhook/main.go): - 503 + Retry-After:30 on every path. Statically linked. Lets the AMI build succeed and kube-apiserver boot cleanly. Replaced wholesale by cs-eks-6c. Init system divergence from spec wording: Alpine ships OpenRC, not systemd. Unit files use .initd extensions and cron periodics instead of .service / .timer. Bead-level intent (auto-enable on boot, periodic execution) preserved. Smoke-tested: - eks-server: builds 300M qcow2; qemu q35 boot reaches K3s apiserver on 6443 within ~120s; HTTPS probes return valid k8s Status JSON (the 401 is K3s v1.32 default RBAC behaviour against anonymous, not a regression). - eks-agent: builds 280M qcow2; qemu boot mounts root cleanly. * feat(eks): make targets for eks-server + eks-agent AMI builds - build-system-image gains subdir-manifest fallback: tries scripts/images/$(IMAGE).conf first, then scripts/images/$(IMAGE)/manifest.conf so the new EKS layout works alongside the existing flat lb manifest. - build-eks-server-image / build-eks-agent-image — explicit convenience targets that delegate to the generic one.	2026-06-01 16:24:49 +10:00
tomnewton-mulga	9d8597317a	Feat/mig capable (#271) ... * feat(gpu): add NVIDIA MIG multi-instance GPU support Partitions MIG-capable GPUs (A100, A30, H100) into isolated slices exposed as mdev devices instead of whole-GPU VFIO passthrough. VMs attach via -device vfio-pci,sysfsdev=<mdev-path> using the GPUAttachment discriminated union (PCIAddress for whole-GPU, MdevPath for MIG). Key changes: - gpu: new MIGProfile/MIGInstance types, IsMIGCapable model map, mig.go ops (EnableMIGMode, DisableMIGMode, CreateInstances, ListInstances, DestroyAllInstances, enrichMdevPaths) - gpu.Manager: Claim() returns (*GPUDevice, *MIGInstance, error); AddMIGInstances, ReclaimByMdev for restart recovery; MarkMIGFailed per-slice - daemon: buildGPUPool helper partitions devices into whole-GPU vs MIG pools; MIGProfile config field selects profile per node or per GPU override - vm.VM: GPUAttachments []gpu.GPUAttachment replaces GPUPCIAddresses []string - admin CLI: spx admin gpu mig status/enable/disable subcommands with admin.SetMIGProfile for spinifex.toml writes - docs: COMMANDS.md updated with three new MIG management command rows * test(gpu): unit tests for MIG parsers and pool manager Cover all pure parsing functions (parseMIGMemory, isCapacityExhaustedError, extractProfileFields/GIFields, parseMIGProfiles, parseMIGInstances, parseCreatedGIID/CIID) with realistic nvidia-smi fixture output. Manager MIG tests cover AddMIGInstances, Claim/Release on mdev-backed entries, MarkMIGFailed, ReclaimByMdev, and interaction with whole-GPU MarkFailed — all using fake mdev directories in t.TempDir() so no GPU hardware is required. * feat(gpu): dynamic MIG profile-per-request with on-demand GI/CI lifecycle Replaces static config-driven MIG pre-carving with a dynamic model where the MIG profile is specified per RunInstances call via mig.<profile> instance type names (e.g. mig.1g.10gb) that mirror nvidia-smi nomenclature directly. - Manager.Claim(instanceID, profileName): fast path claims a pre-carved slice; slow path dequeues a free MIG GPU, releases the lock, calls nvidia-smi mig -cgi/-cci to carve all slices, then reclaims the lock to commit and return the first slice - Manager.Release: last release on a dynamically-carved GPU calls DestroyAllInstances and returns the GPU to freeMIGGPUs, allowing a different profile on the next claim - AddMIGGPU(dev): registers a fresh MIG GPU for dynamic allocation; AddMIGInstances (restart recovery) keeps its no-destroy semantics - GenerateMIGTypes: produces mig.<profile> InstanceTypeInfo entries from gpu.ListProfiles at daemon startup; IsMIGType/MIGProfileFromType let callers inspect these names - buildGPUPool: calls ListInstances for recovery, AddMIGGPU for fresh GPUs, ListProfiles to collect MIG profile specs for type generation; removes setupMIGInstances and migProfileForDevice - GPUClaimer.Claim gains profileName param; daemonGPUClaimer, service_impl call sites, and fakeGPUClaimer all updated * fix(gpu): correct mdev parent detection and make --profile optional for mig enable enrichMdevPaths was using os.Readlink on "parent_bus_0" (a non-existent sysfs attribute) and then on ".." (a directory, not a symlink) — both would fail on real NVIDIA hardware, breaking CreateInstances and ListInstances on daemon restart recovery. Fix: read the mdev UUID's own symlink target, which always encodes the parent GPU's PCI address in its path (e.g. .../0000:01:00.0/<uuid>). Make basePath injectable in enrichMdevPaths so the fix is covered by a new set of unit tests using temp-dir symlinks (TestEnrichMdevPaths_*). Remove the MarkFlagRequired("profile") from "spx admin gpu mig enable": Phase 2 is fully dynamic — the daemon carves slices on demand at RunInstances time. Without --profile, the command just enables MIG mode and destroys any pre-carved slices so the daemon starts with a free GPU. With --profile, behaviour is unchanged (pre-carve a static pool). * chore: writeups for smci and arcee * test: add coverage for MIG exec, manager slow-path, instancetypes and admin - Add mig_exec_test.go with fake nvidia-smi harness covering Enable/Disable MIGMode, IsMIGModeEnabled, DestroyAllInstances, ListProfiles, CreateInstances, ListInstances - Extend manager_mig_test.go with slow-path Claim/Release tests using fake nvidia-smi - Add generate_test.go coverage for IsMIGType, MIGProfileFromType, GenerateMIGTypes - Add admin_test.go coverage for SetMIGProfile (create, update, idempotent, error paths) - Make mdevBasePath a var to allow sysfs injection in tests - Add json tags to GPUAttachment fields to satisfy musttag linter Diff coverage: 75.6% (threshold: 60%). * fixed gpu passthrough doc * Revert "fixed gpu passthrough doc" This reverts commit 1e20f5ab13. * feat: Added RTX 6000 GPU MIG support and fixed docs * feat: expose GPU/MIG inventory in GetNodes and GetVMs admin API Add Manager.Snapshot() for lock-safe pool inspection. Extend NodeStatusResponse with GPUs []GPUInfo (per-physical-GPU inventory with per-slice detail for MIG) and VMInfo with GPU *VMGPUInfo (model, VRAM, profile, mdev path for the attached GPU). Populate both in handleNodeStatus and handleNodeVMs via buildGPUInventory, buildPoolLookup, and resolveVMGPU helpers. * feat: surface GPU info on EC2 instance detail page (mulga-new-03) Admin users see a GPU card (model, VRAM, attachment type, MIG profile/mdev path or PCI address) populated from the GetVMs admin API. Adds formatVRAMMiB helper and VMGPUInfo/GPUInfo/GPUSliceInfo types to the frontend. * feat: GPU columns + inventory card on Nodes page (mulga-new-04) Adds GPU column to VMsTable (model/VRAM or MIG profile) and ResourceTable (alloc/total GPUs). Section 4 GPU Inventory card shows per-node GPU state with per-slice MIG drill-down. Also adds total_gpus/alloc_gpus to NodeInfo type. * feat: grouped GPU-aware instance type picker in Run Instances (mulga-new-05) Splits the flat instance type dropdown into Standard and GPU-Accelerated groups. GPU items show total VRAM alongside the type name. Selecting a GPU type surfaces an info line with GPU name and VRAM. * testing image builder scripts * fixed nvidia build script * more descriptive errors for image build script * more fixes * more fixes * fix: surface PCI address in whole-GPU VMGPUInfo and rebuild frontend dist Add PCIAddress to VMGPUInfo, populate it in resolveVMGPU for whole-GPU passthrough, and fix the instance detail page which was incorrectly showing instance type in the PCI address field. Rebuilt frontend dist to include GPU UI changes (inventory card, resource columns, GPU-aware instance type picker). * fix: improve kernel version detection in nvidia image build script Fall back to /lib/modules when /boot/vmlinuz-* is absent, and add diagnostic output to help debug kernel detection failures. * fixing nvidia image build script * fix ami name issue * correct kernel versions * another fix * image build works, added docker and nvidia toolkit for QoL * fix: add gpg * fixing amd image builder * add universe * another fix * nvm * fix docker user group issues * make rocm land where it is supposed to * fix docker groups * feat: add Ubuntu 26.04 GPU AMI catalog entries and fix build script parted detection Add ubuntu-26.04-nvidia-gpu-x86_64 and ubuntu-26.04-amd-gpu-x86_64 to AvailableImages, hosted on iso.mulgadc.com/system-ami/ with SHA256 checksums. Fix build-system-image.sh to detect parted in /usr/sbin when not on PATH. Suppress gosec false positives in DHCP manager (cancel stored in leaseLoop) and promisc ioctl (standard syscall uintptr pattern). * Updated passthrough doc regarding GPU images * fix: remove orphaned nolint directives (golangci-lint v2.12.2 no longer flags G115/G118) * chore: remove stale spinifex@192.168.3.107 directory Scripts were accidentally committed with a node hostname as the path prefix during initial testing. Correct copies live in scripts/images/.	2026-06-01 13:48:13 +10:00
Julian Sommer	fbd5e3a607	fix: reconciler emits configured dns_server instead of hardcoded 8.8.8.8 (#265)	2026-05-29 12:23:03 +10:00