21 Commits

Author	SHA1	Message	Date
Josh Siviour	9d662f9889	feat: EKS EBS CSI driver (Viperblock) — egress, attach, node discovery, e2e (#402) ... * feat: bake aws-ebs-csi-driver addon + mdev by-id bridge Add the aws-ebs-csi-driver entry (v1.40.1) to the eks addon catalog and ship its baked manifests (RBAC, IRSA-wired controller, node DaemonSet, default gp3 StorageClass) under the eks-node image. The node plugin locates EBS volumes via /dev/disk/by-id, which busybox mdev does not populate. Add mulga-ebs-byid: an mdev rule + helper that symlinks nvme-Amazon_Elastic_Block_Store_<serial> from the virtio-blk serial, plus a boot oneshot that points the kernel hotplug helper at mdev and coldplugs volumes present at boot. * test: e2e EBS CSI volume — PVC bind + mount + reschedule survival Add a TestEKS/EBSCSIVolume subtest that installs the aws-ebs-csi-driver managed addon bound to an IRSA role, provisions a default gp3 PVC + pod, and asserts a marker written by the first pod survives a reschedule onto a fresh pod backed by the same PVC. Exercises the full data path: CreateVolume(Viperblock)/AttachVolume virtio-blk hotplug with serial set to the volume-id, mdev /dev/disk/by-id symlink, and ext4 mount. * feat: eks-node provider-id + topology/instance-type labels for EBS CSI mulga-eks-provider-id boot oneshot writes a K3s config.yaml.d drop-in from IMDS before K3s starts: kubelet provider-id plus the region, zone and instance-type node labels. The aws-ebs-csi-driver node plugin reads this metadata for NodeGetInfo (instance id from providerID, region/zone from topology labels, instance type for the volume-limit family lookup). k3s runs no AWS cloud-provider so none of it is set otherwise, which left the node plugin's metadata init failing, NodeGetInfo panicking in IsNitroInstanceType, and the CSINode carrying no topology key. Wire the service into the image manifest (INSTALL_FILES + ENABLE_SERVICES) and chmod it in setup. * test: e2e EBS CSI controller IRSA policy + failure diagnostics awsgw enforces IAM on assumed-role EC2 calls, so the controller's CreateVolume/AttachVolume need a permission policy on the IRSA role. Attach a customer-managed EC2 policy (AWS-managed ARNs are opaque and grant nothing) covering the EBS CSI action set; CreateVolume then succeeds. Add an OnFailure dump of the CSI control path (provisioner and plugin logs, pvc/pod describe, events, csinode) to the EBSCSIVolume subtest so provisioning/attach failures are diagnosable post-teardown. * feat: EBS CSI obeys control-plane taint; e2e worker nodegroup Drop the CriticalAddonsOnly/NoExecute tolerations from the EBS CSI controller + node manifests so the driver runs only on customer worker nodes, never the system control-plane VM. The e2e creates a 1-node worker nodegroup and the writer/reader pods drop their CP tolerations so the customer-owned volume attaches in the customer space. * fix: EBS CSI attach — decouple hotplug PCIe port from device name; iothread rollback virtio-blk-pci hot-plug bus was derived from the AWS device-name letter (/dev/sd[f-p]); the EBS CSI driver sends arbitrary names (/dev/xvdaa) that map to no slot, so device_add fell back to pcie.0 — which rejects hot-plug — and the attach hung. A failed attach also left the per-volume iothread object behind, so every retry died on object-add 'duplicate property'. nextHotplugEBSPort allocates the lowest free hotplug-ebs port from live query-block state, independent of the device name. delIothreadBestEffort removes the iothread on every attach rollback path. * fix: EBS CSI node discovery — un-shadow mdev by-id rule, mint nvme link Alpine eks-node runs busybox mdev (no eudev), which stops at the first matching /etc/mdev.conf rule. setup.sh appended the mulga-ebs-byid rule after the stock 'vd[a-z].* -> persistent-storage' line, so it never fired on hot-attach: only /dev/disk/by-id/virtio-<serial> was created, while the upstream aws-ebs-csi-driver resolves nvme-Amazon_Elastic_Block_Store_<volid>. Swap the stock vd* rule in place to route through mulga-ebs-byid, which now delegates to persistent-storage (preserving stock links) before minting the nvme-style link the node plugin expects. * test: EKS e2e — NAT Gateway egress for private worker nodegroup The customer-VPC worker is private; AttachIGW programs no SNAT (mirrors AWS: an IGW serves only public-IP instances), so an IGW-only fixture left the worker unable to reach the public NLB endpoint or pull public CSI images — the nodegroup never joined. createWorkerEgress now builds the real topology: an IGW-fronted public subnet hosting a NAT Gateway, with the worker subnet default-routed to it. Associating the worker route table triggers the subnet-wide SNAT egress reroute. Teardown drains the NAT Gateway before releasing the public subnet + EIP it pins.	2026-06-19 14:43:20 +10:00
Josh Siviour	2acb37342d	feat(eks): aws-load-balancer-controller addon — real ALB Ingress (L7, IRSA) (#376) ... * fix(ec2): stamp subnet AvailabilityZoneId for single-AZ ALB resolution DescribeSubnets omitted AvailabilityZoneId, so the AWS Load Balancer Controller's DescribeAvailabilityZones-by-ZoneId lookup resolved no AZ info ("cannot resolve AZ info for AZs: []") and never built the ALB model. Stamp the single zone-id on every subnet and share it with DescribeAvailabilityZones through one SingleZoneID constant so the two responses can never drift. * feat(eks): add aws-load-balancer-controller v2.8.1 addon bundle Baked manifests for the LBC managed addon (CRDs, controller Deployment/ RBAC/webhooks/Service, IngressClass alb, IRSA render markers). Two wirings make it work against the spinifex gateway: - --aws-api-endpoints carries sts= alongside elasticloadbalancing/ec2/ acm. The aws-sdk-go v1 credential bootstrap resolves the STS client from the session endpoint resolver, not AWS_ENDPOINT_URL_STS, so the web-identity AssumeRoleWithWebIdentity must be pointed at the gateway here or it falls back to real AWS STS. - --feature-gates=ALBSingleSubnet=true drops the ALB minimum subnet count from two to one so an ALB provisions in a single-AZ deployment. * feat(eks): render IRSA into managed addons + node-minted webhook cert Extend the addon-sync renderer to inject the projected-token / role-arn / gateway STS-endpoint + CA block into managed-addon pod specs via opt-in markers, and substitute cluster-name/region/vpc-id/gateway-endpoint/CA placeholders. Add cmd/eks-webhook-cert: the eks-node AMI has no openssl, so this mints (then reuses) a self-signed ECDSA-P256 serving cert for the admission webhook, cached on disk so the render is byte-stable across ticks. Wire both into the image build (manifest.conf) plus INSTALL_FILES for the LBC bundle. * feat(eks): surface cluster VPC id as EKS_VPC_ID for the LB controller Thread the control-plane VPC id through K3sServerInput.VpcID into the first-boot EKS_VPC_ID env so the in-cluster aws-load-balancer-controller can pass --aws-vpc-id to the gateway elbv2/ec2 handlers. * fix(eks): point the LB controller at the customer VPC + customer-facing gateway The aws-load-balancer-controller ran with two control-plane-scoped values that customer worker pods cannot use: - --aws-vpc-id was set from the managed control-plane VPC (cpRefs.VpcID), which the gateway's DescribeVpcs does not return for customer calls, so backend-SG auto-create failed with InvalidVpcID.NotFound. Source it from meta.ResourcesVpcConfig.VpcId, the customer cluster VPC where the workers and their subnets live and where the ALB is provisioned. - The addon render baked the mgmt gateway endpoint (EKS_GATEWAY_URL / SystemGatewayURL) into pod specs, unreachable from worker pods. Add a distinct EKS_ADDON_GATEWAY_URL (the customer-facing GatewayBaseURL, same host as the OIDC issuer) and substitute it for {{GATEWAY_ENDPOINT}} in managed addon manifests. EKS_GATEWAY_URL stays mgmt for the CP's own bootstrap, state-report and token-webhook paths. * fix(elbv2): resolve forward target group from ForwardConfig listenerActionFromSDK read only the legacy flat Action.TargetGroupArn, so a forward action carrying its target group via ForwardConfig.TargetGroups (the shape the aws-load-balancer-controller emits) left the action target empty and CreateRule/CreateListener rejected it with MissingParameter. Flatten the first target group from ForwardConfig when the flat field is absent. * fix(ec2): mirror CreateTags/DeleteTags into subnet/vpc records The generic S3 tag store and the per-resource KV records are disjoint: DescribeSubnets/DescribeVpcs read tags from their KV record, while CreateTags only wrote the generic store. Subnet tags applied after create were therefore invisible to tag-filtered describes, breaking aws-load-balancer-controller subnet auto-discovery (kubernetes.io/role/elb). Write through CreateTags/DeleteTags to the owning subnet/vpc record so tag-filtered describes observe them. Resource types the VPC service does not own are skipped; the generic store stays their record of truth. * test(daemon): pin synthetic capacity in host-dependent allocate tests TestCanAllocate_CountEdgeCases/Mixed_instance_types and TestHandleEC2DescribeInstanceTypes/NoFilterReturnsSupportedRegardlessOfAllocation called rm.allocate against live-detected host capacity. allocate gates on readMemAvailableGB (live /proc MemAvailable), so a box already running VMs fails the allocation spuriously, breaking the preflight gate locally. Pin hostVCPU/hostMemGB/reserved and nil readMemAvailableGB to a synthetic host, matching the sibling Capacity_decreases_after_allocation subtest. * fix(eks): route apiserver->pod traffic via agent tunnel (egress-selector cluster) In the managed-CP topology the apiserver VM sits in the system CP VPC with no route to the worker pod network in the unpeered customer VPC. k3s default egress-selector-mode=agent tunnels only apiserver->kubelet, so kubectl logs/exec work but apiserver->pod traffic (admission webhooks, aggregated APIs) is attempted directly and times out -- the aws-load-balancer-controller webhooks fail with context deadline exceeded, blocking Service/Ingress creation cluster-wide. Set egress-selector-mode=cluster so apiserver->pod/service traffic routes through the agent's outbound tunnel, selected by the node hosting the destination, without requiring inbound CP->worker reachability. * feat: bump aws-load-balancer-controller addon bundle to v2.11.0 v2.8.1 predates the ALBSingleSubnet feature gate, so single-AZ spinifex clusters fail Ingress reconcile (subnet resolver dedups to 1 < required 2) and never provision an ALB. v2.11.0 honors --feature-gates=ALBSingleSubnet=true. Replace the baked bundle: new CRDs (ingressclassparams + targetgroupbindings), controller manifest, and image tag v2.11.0, re-applying the spinifex deltas (IRSA SA annotation, node-minted webhook serving-cert secret, gateway-CA secret, gateway aws-api-endpoints, caBundle on every webhook clientConfig, alb IngressClass). Pick up the new ClusterRole configmaps rule. Drop v2.8.1. * fix: point LBC at the gateway via AWS_ENDPOINT_URL (sdk-v2) LBC v2.11 uses aws-sdk-go-v2, which keys --aws-api-endpoints on SDK service IDs (e.g. 'Elastic Load Balancing v2'), not the legacy lowercase names the bundle passed. Those overrides were silently ignored, so every non-STS call (ec2/elbv2/acm) went to real AWS and failed TLS against the gateway CA -- no ALB could provision. Inject AWS_ENDPOINT_URL in the shared IRSA env block (sdk-v2 global base-endpoint override; AWS_ENDPOINT_URL_STS still takes precedence for STS) and drop the dead --aws-api-endpoints flag from the controller args. * fix: synthesize taggable ARN for default listener rule Gateway DescribeRules returned the synthetic IsDefault rule with a null RuleArn. LBC read the arn-less default rule after creating a listener and issued DescribeTags with an empty resource ARN, which the gateway rejected with 400 MissingParameter, aborting reconcile. Set a deterministic RuleArn on the default rule, resolve it back to its parent listener in DescribeTags (empty tags) and DescribeRules by-arn. Also disable shield/waf/wafv2 in the LBC bundle to drop unsupported calls. * feat: inject ELB subnets into alb IngressClassParams for single-AZ ALB LBC's tag auto-discovery never threads the ALBSingleSubnet feature gate, so a single-AZ cluster (subnets sharing one AZ) dedups to 1<2 subnets and aborts Ingress reconcile. Only the explicit-subnet path honors the gate. Resolve the cluster's ELB-eligible subnets, dedup to one per AZ, and surface them as EKS_ELB_SUBNET_IDS. The addon sync renders them into the alb IngressClassParams .spec.subnets.ids so every Ingress takes the explicit path. An empty set falls back to auto-discovery, preserving the native UX. * fix: route LBC service clients to gateway via --aws-api-endpoints, link IngressClassParams Live verification showed two gaps in the LBC addon bundle: LBC installs a per-client BaseEndpoint resolver, so the global AWS_ENDPOINT_URL is ignored for the ec2/elbv2/acm clients (calls hit real AWS, failing TLS). Redirect them with --aws-api-endpoints keyed on the aws-sdk-go-v2 service IDs (EC2, "Elastic Load Balancing v2", ACM, STS) rather than the legacy lowercase names the flag never matched. Drop the misleading AWS_ENDPOINT_URL env; the web-identity STS client still resolves via AWS_ENDPOINT_URL_STS. The injected IngressClassParams was orphaned: the alb IngressClass had no .spec.parameters, so LBC fell back to subnet auto-discovery (which fails on a single-AZ cluster). Link the IngressClass to the IngressClassParams so the explicit-subnet path is taken. * fix: stamp EKS-parity providerID on worker nodes for LBC ip targets k3s registers workers with providerID k3s://<name>, which the AWS Load Balancer Controller rejects (providerID invalid for EC2 instances) when resolving nodes for ip-mode target registration. Resolve the instance-id and AZ from IMDSv2 at agent boot and set --kubelet-arg=provider-id=aws:///<az>/<id> so LBC can map the node to its EC2 instance. On any IMDS miss the agent keeps the default providerID and still starts. * fix(elbv2): honor SubnetMappings in CreateLoadBalancer LBC supplies ALB subnets via SubnetMappings, not the plain Subnets list. createLoadBalancer only read input.Subnets, so ENI allocation was skipped, VpcId/AZ stayed empty, and the HAProxy data-plane VM never launched — the ALB went active but could not serve traffic. Add flattenSubnetIDs to read both sources and reuse it in createLoadBalancer and desiredSubnetSet. * fix(vm): capture GCInterval before spawning GC goroutine TestStart_SweepsImmediatelyThenStopsOnCtxCancel mutates the package global GCInterval and restores it in a defer, racing the Start goroutine's read at time.NewTicker(GCInterval) under -race. Read GCInterval on the caller's goroutine into a local before spawning so the global access is ordered with the test's defer; the goroutine touches only the captured local.	2026-06-18 10:53:05 +10:00
Josh Siviour	bc9556d1a9	fix(eks): managed-CP worker data-plane — apiserver reachability, node-local CoreDNS, SG teardown (#368) ... * fix: persist instance TagSpecifications in RunInstance RunInstance built the ec2.Instance metadata but never read input.TagSpecifications, so DescribeInstances returned untagged instances and tag filters matched nothing. EKS node groups discover their workers by the spinifex:eks-cluster / spinifex:eks-nodegroup tags, so scale-up convergence and worker reconciliation depended on tags that were silently dropped. Apply instance-scoped TagSpecifications onto the built instance so the tags surface through DescribeInstances and tag filters. * fix: serialize nodegroup reconcile via CAS to prevent scale overshoot UpdateNodegroupConfig is dispatched on a queue group, so two overlapping requests (e.g. an SDK retry mid-flight) each read current = len(InstanceIDs) and each launched desired-current workers, overshooting the target (1 -> 5 for desired=3) and stranding untracked VMs via last-writer-wins on the unconditional PutNodegroupRecord. Make every durable nodegroup-record mutation a compare-and-swap on the KV revision. Scale-up launches one worker at a time and CAS-appends it only while the record is below the live ScalingDesired, terminating any surplus VM launched into an already-full record. Scale-down terminates before the CAS truncation so a crash or retry leaves reclaimable dead IDs rather than orphaned VMs. A concurrent regression test asserts two overlapping scale-ups converge to exactly desired under -race. * fix: return securityGroupRuleSet from AuthorizeSecurityGroup ingress/egress The AWS provider's aws_vpc_security_group_ingress_rule reads SecurityGroupRules[0].SecurityGroupRuleId on create; an empty ruleset panics the plugin (Plugin did not respond), though the rule applies server-side. Echo the created rules via sgRuleToSecurityGroupRule, the same shape DescribeSecurityGroupRules already returns. * fix: surface InsufficientInstanceCapacity from nodegroup scale, not 500 launchWorkersCAS wrapped the bare awserror code via fmt.Errorf, so the gateway ErrorHandler's exact-match against ErrorLookup missed and remapped to InternalError (500). An out-of-capacity scale now surfaces the real InsufficientInstanceCapacity code (400); opaque internal failures still wrap and stay 500. Adds awserrors.HasErrorCode for the membership test. * fix: cut EKS node boot ~50s — drop cloud-final deadlock + syslinux menu Worker/server user-data runcmd ended with `[ rc-service, local, start ]`, which runs inside cloud-final while the OpenRC `local` service is ordered after it — a deadlock broken only by OpenRC's ~50s start timeout, stalling the k3s join. Run the IMDS route script directly instead and keep `rc-update add local default` for reboot persistence. setup.sh: cut the SYSLINUX boot-menu countdown 10s->~1s (kept nonzero so the menu stays interruptible over serial). Validated live on env19 CP: boot menu 1s, no `waiting for cloud-final` stall, apiserver ready ~66s (was ~109s+). mulga-siv-332 * feat: add argocd managed addon + bake bundle into eks-node AMI The addon catalog advertised coredns and aws-load-balancer-controller, but the only baked manifest shipped in the eks-node AMI was the spinifex-noop fixture. Installing any real addon failed with 'no baked bundle for version' once the VM-side renderer found nothing on disk. Add argocd v3.0.23 as a real, demonstrable managed addon: catalog entry (no IRSA), a flattened namespaced install manifest baked into the AMI, and the manifest.conf bake line. The eks-addons-access workbook switches from coredns (redundant on K3s, never baked) to argocd. The baked bundle is generated from upstream install.yaml via kubectl kustomize with namespace: argocd plus a prepended Namespace, since upstream is namespace-agnostic and K3s auto-deploy applies with no -n flag. * fix(eks): run CoreDNS as a node-local DaemonSet for worker DNS k3s ships CoreDNS as a single-replica Deployment that tolerates the CP taint and strands itself on the control-plane node. In the managed-CP topology the worker<->CP pod overlay cannot cross the VPC boundary, so worker pods cannot reach a CP-resident CoreDNS and lose all cluster + external DNS. Skip the bundled CoreDNS (coredns.yaml.skip, staged in k3s.initd like traefik) and deploy a Mulga-owned variant: a DaemonSet (one pod per node) plus kube-dns internalTrafficPolicy=Local, so every node resolves via its own CoreDNS. Immune to the cross-VPC overlay gap and topology-agnostic (single-node and managed-CP both work). Image-side only; no handler change. * fix(eks): revoke cross-referencing SG rules before delete on teardown DeleteClusterSGs deleted the control-plane and nodegroup SGs directly, but they cross-reference each other (EnsureNodegroupSGRules), and AWS refuses to delete a security group still referenced by another SG's rule. Both deletes failed with DependencyViolation, the SGs leaked, and the VPC could not be deleted (tofu destroy hung on DeleteVpc). Resolve both SG IDs, revoke all ingress on each to break the cp<->ng cycle, then delete both. Adds RevokeSecurityGroupIngress to sgProvisioner (already on VPCServiceImpl) and revokeAllSGIngress, with a regression test that models AWS's DependencyViolation. * fix(eks): advertise NLB front-end so worker pods reach the apiserver ClusterIP In the managed-control-plane topology the apiserver advertised its Set-B node-ip into the in-cluster `kubernetes` Endpoints. That IP lives in the unpeered CP VPC and is unreachable from worker pods, so every worker pod using the apiserver ClusterIP (CoreDNS kubernetes-plugin sync, controllers, argocd redis secret-init) failed. - k3s_server_vm.go: set advertise-address to the worker-reachable NLB front-end (PrivateEndpointIP, else EndpointIP); both are already cert SANs. - nlb.go: add a second :6443 listener on the cluster NLB (same CP target group) so PrivateEndpointIP:6443 — the advertised endpoint — is served. - security_groups.go: admit :6443 on the private-endpoint SG. Validated on a live managed-CP cluster: worker CoreDNS /ready 200, all argocd pods Running, kubectl logs on worker pods working.	2026-06-16 20:27:21 +10:00
Josh Siviour	b23fa3bf60	feat(eks): managed control-plane VPC (Set B) + control-plane lifecycle hardening (#356) ... * feat(eks): VM-side managed-addon delivery transport Stage CreateAddon manifests in KV host-side; the control-plane VM pulls the staged set through the AWS gateway (signed GET /clusters/{name}/internal-addons/ {accountId}), renders the baked bundles into the K3s auto-deploy dir, and reports per-addon delivery status on a dedicated eks.addon.*.status channel that the cluster reconciler CASes onto AWS-visible AddonStatus (CREATING->ACTIVE). - gateway: ListInternalAddons route + ListStagedAddonManifests NATS action - handlers: addon status reconciler (nextAddonStatus), staging installer - eks-node AMI: eks-gateway-fetch binary + mulga-eks-addon-sync agent - agent renders idempotently (cmp before replace) so K3s auto-deploy does not re-apply every tick and clear the Addon CR .status.gvks readiness signal - agent mirrors per-tick delivery outcomes to the serial console for host-side observability * fix(eks): provision cluster NLB synchronously so the endpoint carries the real front-end IP The cluster NLB front-end IP is assigned by the async LB-VM launch (launchLBVMAsync), but EnsureClusterNLB read it back synchronously right after CreateLoadBalancer returned — losing the race and falling back to the synthetic <lb>.elb.spinifex.local DNS name. That name is unresolvable (no Route53) and is not in the apiserver serving-cert SAN, so kubectl against the cluster never connects. Split the elbv2 create: the public CreateLoadBalancer stays async (267.4 — an inline multi-minute VM boot head-of-line-blocks the gateway responder), and a new CreateLoadBalancerSync drives the data-plane launch inline and returns the LB carrying its front-end address. The EKS cluster-create worker runs off the gateway responder, so it uses the sync entry and reads the IP from the return value instead of racing persisted state. No polling. - elbv2: factor provisionLBDataPlane out of launchLBVMAsync; add CreateLoadBalancerSync - eks: ensureClusterLB uses the sync entry, sets FrontendIP from the result, drops the read-back; EnsureClusterNLB fails loud (ErrClusterNLBFrontendIPUnavailable) when no front-end IP is allocated rather than shipping the DNS-name fallback - eks: drop the DNS-name endpoint fallback; the endpoint is always the front-end IP * feat: auto-wire IGW for internet-facing EKS cluster endpoint An internet-facing cluster NLB allocates an external front-end IP and the OVN dnat_and_snat for it, but the external switch + localnet + gateway LRP that make that IP answerable on the wire are built only on IGW attach. The control-plane NLB is a system instance, so ensure an IGW on cluster build and reclaim it on teardown. EnsureClusterIGW (idempotent, reuses a customer IGW untouched) runs before EnsureClusterNLB when the endpoint is public; DeleteClusterIGW (ownership- scoped) removes only the cluster-owned IGW in purgeClusterInfra, before the VPC delete. New IGW dep wired from the daemon. * fix(eks): tear down infra on failed cluster launch to stop LB EIP leak A CreateCluster that fails during its background launch flipped the meta to FAILED but performed no teardown, relying on a later same-name create to reclaim and purge the infra. EKS cluster names are unique per create, so that reclaim never fires: the internet-facing load-balancer VM keeps running with its allocated+associated dnat_and_snat EIP indefinitely. Over repeated create/delete cycles the EIP pool fills with allocations bound to terminated instances, so a later allocation's advertised endpoint and its live OVN NAT diverge onto different pool addresses. - failClusterLaunch now runs a best-effort purgeClusterInfra on the in-memory meta (freshest refs even when the failing step was the meta write itself) before MarkClusterFailed. purgeClusterInfra is idempotent, so a later reclaim or DeleteCluster re-purge is a no-op; the FAILED meta is retained for observability. - Daemon releaseSystemInstanceEIP extracted and called from both TerminateSystemInstance and cleanupFailedSystemInstance, so a launch that fails after AssociateAddress releases the eipService allocation instead of leaking it (MarkFailed's ReleasePublicIP only knows the externalIPAM path). * feat: EKS managed control-plane VPC (AWS Set B) composed from EC2 APIs CreateCluster now auto-builds the managed control-plane VPC under the system account, the spinifex analogue of AWS EKS's hidden managed-account VPC: a public subnet for the internet-facing NLB, private subnet(s) for the k3s control-plane VM(s), and a NAT gateway giving the private CP egress for image pulls. It is composed from the real EC2 VPC-family APIs (CreateVpc/Subnet, IGW, RouteTable/Route, NatGateway), so per-subnet egress gating is wired by the existing topology subscribers off the route-table events those APIs publish — no direct OVN mutation. This resolves the mulga-siv-292 root cause: the internet-facing NLB and the private CP were crammed into one customer subnet with no IGW route, so both were treated as private and dropped by the lr_in_policy 1100 rule, killing the LB-VM return leg and lb-agent heartbeat. The CP only survived via a per-instance 1200 reroute bandaid, now removed — the topology expresses egress instead. Placement is uniform single/multi-node (CP always in the private subnet). The system LB VM heartbeats over the mgmt bridge (SystemGatewayURL) rather than the WAN. Managed-VPC teardown reclaims NAT GW + EIP, route tables, subnets, IGW, and the VPC in dependency order. Customer subnetIds (Set A / worker side) remain consumed-not-created and are deferred to the worker round. * fix(eks): bake cluster-owner account into CP-VM EKS_ACCOUNT_ID The managed control-plane VPC (Set B) launches the K3s control-plane VMs and their ENIs under the system (infra) account, but the on-VM agents — bootstrap publish, state report, token-review, and add-on fetch — must key on the cluster-OWNER (customer) account that owns the cluster meta, namespaces, and AccessEntry KV. Add ClusterAccountID to K3sServerInput and bake it into the VM's EKS_ACCOUNT_ID, decoupling it from the infra AccountID the VM launches under. Without this the VM reported state and fetched add-ons against the system account, so the cluster never observed a healthy control plane (stuck CREATING, null CA) and kubectl/add-on delivery timed out. Also refresh the e2e suite doc comment to describe the Set B topology and the data-plane subtests (kubectl + managed-addon delivery) that now run. * fix(eks): explicitly delete addon objects on unstage K3s does not delete the applied resources when an auto-deploy manifest file is removed from the server manifests dir (documented behaviour). The addon-sync GC pass relied on that, so after DeleteAddon the rendered manifest was dropped but its Namespace/ConfigMap stayed standing — the e2e AddonDelivery GC wait timed out. Copy the manifest aside, drop the original so K3s stops tracking it, then kubectl delete the objects (--wait=false to avoid finalizer stalls).	2026-06-12 22:29:24 +10:00
Josh Siviour	74b91b5cc5	feat(eks): HA control plane — 3-node etcd quorum spread across distinct hosts (#335) ... * feat(eks): node-targetable control-plane VM (HA spread transport prereq) Groundwork for HA control-plane placement (mulga-siv-231.7.1): make the k3s control-plane VM placeable on a specific Spinifex host. - Add a sys.medium instance type (2 vCPU / 4 GB) and boot the control-plane VM as it instead of t3.medium. A sys.* type keeps the CP VM out of customer DescribeInstanceTypes and makes the daemon register the node-targeted system.LaunchInstance.{type}.{nodeID} subject (only sys.* types get it). - Add K3sServerInput.TargetNodeID and a LaunchSystemInstanceOnNode method on the launcher: empty/local node launches in-process (unchanged single-CP path); any other node dispatches over the node-targeted subject, where the receiving daemon runs the launch and owns the VM. The spread orchestrator (reserve N hosts, launch per host, verify, rollback, <3-host fallback) lands separately; CreateCluster still launches locally (TargetNodeID empty), so behaviour is unchanged. * feat(eks): HA control-plane spread orchestrator Place N=3 k3s server VMs across distinct Spinifex hosts via a per-cluster spread placement group, with all-or-nothing rollback and single-CP fallback. - placeControlPlane: capacity fan-out (HostScheduler) -> >=3-host gate -> ensure spread group (system account) -> ReserveSpreadNodes(Min=Max=3) -> parallel per-node LaunchK3sServerVM(TargetNodeID) -> node-vms placement verify -> FinalizeSpreadInstances. Any launch/verify/finalize failure terminates launched VMs, releases the reservation, drops the group. - <3 schedulable hosts or a lost reservation race falls back to a single local control-plane VM (prior behaviour). - ClusterMeta gains ControlPlaneNodes[] + ControlPlaneSpreadGroup; [0] is mirrored into the scalar ControlPlane* fields for back-compat. DeleteCluster tears down every CP VM and the spread group. - Wire PlacementGroup + Scheduler into EKSServiceDeps + orchestration-deps gate. * fix(eks): size HA control-plane spread on raw host headroom SchedulableHosts matched the CP type by name in node.status's per-type capacity list, but that list omits sys.* types (system types are hidden from customers). The CP VM is sys.medium, so the match never hit and SchedulableHosts returned 0 on every node — placeControlPlane always fell back to a single CP, making the >=3-host spread unreachable in production. System VMs consume host vCPU/memory like any guest, so size the fit against each node's raw schedulable headroom (Total - Reserved - Alloc) and the type's footprint instead. Adds instancetypes.SpecForSystemType for the sys.* vCPU/memory spec and a pure nodeFitsSystemInstance helper. * feat(eks): k3s server-join role for HA control-plane quorum Servers 2..N now boot in join mode: server: https://<first-server>:6443 + a shared cluster token, without cluster-init, so they join the first server's embedded-etcd quorum instead of each cluster-initing an independent datastore. - K3sServerInput gains JoinToken (shared cluster secret, set on every server) and ServerURL (set => join server, no cluster-init, server-join role). - buildK3sUserData branches cluster-init vs server:, emits token:, and selects SPINIFEX_K3S_ROLE server vs server-join. - launchControlPlaneSpread launches nodes[0] first (cluster-init), derives the join URL from its ENI IP, then launches the rest in parallel as join servers; a first-server failure skips the joins and rolls back. - CreateCluster mints a per-cluster crypto-random token (GenerateK3sClusterToken) threaded into the placement template. - eks-node-role.sh adds a server-join case: token-webhook + k3s + state-report, but not k3s-first-boot (the first server already publishes the bootstrap artifacts; a join re-publish only races the bootstrap bus). * fix: open etcd-peer/kubelet ingress on EKS control-plane SG for HA join Join servers (231.7.2) register over the supervisor port 6443 but their embedded etcd could not peer with the quorum: the control-plane SG only admitted 6443 from the VPC CIDR, omitting etcd client+peer 2379-2380 and kubelet 10250 between control-plane nodes. A single cluster-init server never exercises those ports, so the gap stayed hidden until multi-server HA — the join registered but its etcd never replicated the raft log, leaving the node perpetually NotReady. Add EnsureControlPlaneHAIngress with self-referencing (cpSG->cpSG) rules for etcd 2379-2380 and kubelet 10250, called from CreateCluster after EnsureControlPlaneIngress. Verified live on a 3-node cluster: join servers reach Ready and form a 3-member etcd quorum that survives loss of the original first server. * feat: register all control-plane ENI IPs as EKS cluster NLB targets Register every HA control-plane server ENI IP in the cluster NLB target group, not just the primary. The TG already health-checks :6443, so a dead member drains and kubectl/apiserver traffic fails over to surviving etcd-quorum members. Add RegisterClusterTargets batch helper; the single RegisterClusterTarget now delegates to it.	2026-06-10 12:25:47 +10:00
Josh Siviour	4ef19fc488	feat(ui): EKS console (#310)	2026-06-09 10:57:50 +10:00
Josh Siviour	496424c41c	test(eks): EKS control-plane E2E suite (Sprint 6g) (#304) ... * test(eks): EKS control-plane e2e suite (Phase 1, API-level) Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw: CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone. Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted} pollers. Wire e2e-eks into the Makefile and the single-node CI suite list. * test(eks): Phase 2 kubectl get-nodes data-plane gate Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper (harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the control-plane node is Ready — the get-token webhook auth regression gate. IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up. * test(eks): tolerate doubled NotFound code + dial loopback NATS DeleteCluster detection matched the SDK constant ResourceNotFoundException, but awsgw emits ResourceNotFoundExceptionException on the wire — match the NotFound code by substring so deletion is detected immediately instead of timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which the serving cert does not SAN; rewrite the wildcard bind to loopback. Validated live: all 6 subtests green, kubectl get nodes shows the control-plane node Ready via the get-token webhook auth path. * feat: build system images with libguestfs (no host block devices) Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in build-system-image.sh with libguestfs (virt-customize / virt-resize / guestfish). Every operation runs in an isolated userspace appliance VM with its own kernel, so the build touches no host block device, no host mount, and needs no sudo — removing the failure class that wiped the host's /dev on shared CI runners (a leaked /dev bind traversed by rm -rf). Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu via virt-resize; customization (packages, binaries, files, setup script, service enable, cache cleanup) is one ordered virt-customize invocation. LIBGUESTFS_BACKEND=direct (no libvirt on runners). Wire the eks-node AMI into the single-node e2e job: the runner builds the device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are VM-only) so the eks suite's CreateCluster has an AMI to boot. * feat(eks): bootstrap + state-report via AWS gateway broker Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model: the K3s server VM publishes its bootstrap envelopes and periodic state reports via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS directly. A new gateway action (PublishInternal, POST /clusters/{name}/internal-publish) relays the payload verbatim onto the existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV persistence/encryption, and reconciler are unchanged. This removes the VM's need to reach NATS at all, so NATS can stay cluster-internal behind its bind/firewall/mTLS controls — the prior direct path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was DDIL-fragile (fire-and-forget nats pub, no ack/retry). - gateway/eks: PublishInternal broker + route; closed-set channel/kind validation - cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and mulga-eks-state-report.sh - handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL + system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA; new resolveSystemGatewayBaseURL (mgmt-bridge-first) - eks-node image: build/install eks-gateway-publish, drop nats-cli download * feat: migrate eks-token-webhook to the AWS gateway broker The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that fails, so the webhook exited at startup, the apiserver token-webhook kubeconfig was never written, and the apiserver crash-looped — the cluster never reached ACTIVE. Converge it onto the same gateway-broker model as bootstrap/state-report and ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to /clusters/{name}/token-review, and the gateway runs the verify + lookup host-side and returns the resolved identity. The VM now speaks no core NATS. - internal/eksgw: shared on-VM SigV4 client (also adopted by eks-gateway-publish). - handlers/eks/token_review.go: Authenticate decision core (moved from the webhook binary) + ResolveTokenReview host-side wiring. - gateway/eks token-review route + WebhookTokenReview handler. - webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable) not a silent deny. - initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*. * fix: build eks-node helpers with GOFIPS140; cover token-review broker eks-gateway-publish blank-imports internal/fipsboot, whose init() panics unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build omitted the flag, so the binary paniced at first boot right after k3s came up — k3s-first-boot (set -eu) died before publishing the bootstrap token and the cluster never reached ACTIVE. Build both eks-node helpers with GOFIPS140=v1.0.0, matching cmd/lb-agent. Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview gateway handler (verify responder + account KV) to cover the new broker path. * fix: export first-boot env so eks-gateway-publish child sees it k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/ first-boot.env but never exported the vars, so the eks-gateway-publish child process inherited an empty environment and exited '--account-id is required'. The control-plane VM came up healthy but never published its bootstrap envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a / set +a so the KEY=value lines export to child processes. * fix: make awsgw start --host override actually bind The 'awsgw start --host' override printed 'Overwriting awsgw host' but the assignment was commented out, so AWSGW always bound the spinifex.toml host (a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW listens on all interfaces incl. br-mgmt; the serving cert already SANs every local interface IP, and WAN clients still reach it. * test(eks): install kubectl on e2e VM so KubectlGetNodes runs The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker -> STS verify path — the only end-to-end exercise of the token-review broker. The e2e binaries run on the VM, where kubectl was absent, so harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a checksum-verified kubectl (pinned to the k3s server version) on the VM, gated on e2e-eks.	2026-06-07 20:42:25 +10:00
Josh Siviour	5aaf8c23d2	fix(eks): make aws eks get-token → kubectl auth work end-to-end (Sprint 6f) (#303) ... * feat(eks): webhook observability — console log sink + per-request TokenReview decision * fix(eks): wire token-webhook authenticator into generated k3s apiserver config The generated /etc/rancher/k3s/config.yaml overrides the AMI-baked config.yaml.skel, but its kube-apiserver-arg list omitted the authentication-token-webhook-config-file arg. The running apiserver therefore never loaded the token webhook, so bearer tokens minted by 'aws eks get-token' 401'd without the webhook ever being invoked. * fix(eks): echo TokenReview audiences so api-audiences apiserver accepts the principal With --api-audiences set, kube-apiserver sends spec.audiences in the TokenReview and discards an authenticated=true response that does not confirm a matching audience. The webhook returned an empty status.audiences, so a valid IAM principal surfaced as a 401. Echo the request's audiences on success. * fix(eks): pin token-webhook version v1 so apiserver accepts the response kube-apiserver --authentication-token-webhook-version defaults to v1beta1, but the eks-token-webhook responds with an authentication.k8s.io/v1 TokenReview. The apiserver decodes the response against the expected GVK and rejects the mismatch, surfacing a valid principal as a 401. Pin v1 to match the webhook.	2026-06-06 22:11:50 +10:00
Josh Siviour	a22bc2f716	feat(eks): 6d nodegroup provisioning + NATS-gated cluster health (#300) ... * feat(eks): nodegroup provisioning + K3s agent join (6d) CreateNodegroup launches N customer-owned worker EC2 from the unified eks-node AMI in agent role; each joins the cluster via K3S_URL + K3S_TOKEN baked into cloud-init (first-boot.env SPINIFEX_K3S_ROLE=agent + agent.env). Workers are customer-owned and DescribeInstances-visible (no ManagedBy, no mgmt NIC) via the normal RunInstances path, matching real EKS. K3S_URL targets the control-plane ENI's in-VPC private IP on :6443 — the cluster NLB DNS endpoint is not resolvable in-VPC yet; switch to the NLB endpoint on :443 once authoritative in-VPC DNS lands. K3S_TOKEN is the decrypted cluster node-token persisted at server bootstrap. Describe/List/Update(scaling)/Delete round out CRUD. Cluster<->nodegroup SG ingress rules (apiserver 6443, flannel VXLAN 8472/udp, kubelet 10250, intra-nodegroup) authorized idempotently. UpdateNodegroupVersion deferred. * feat: gate EKS cluster health via NATS state reports The K3s apiserver binds the VPC node-ip, unreachable from the host-side spinifex daemon, so the reconciler cannot probe /healthz directly. Add a control-plane state reporter that publishes {healthz, node_count} over the management NATS bus the daemon already shares. - eks-node server role runs mulga-eks-state-report as an OpenRC service, publishing eks.state.{account}.{cluster}.server every 30s. - ClusterReconciler subscribes to that subject, gates CREATING -> ACTIVE on a healthy report, and records Ready node_count into ClusterMeta. - node_count counts Ready nodes only; k3s retains terminated workers as NotReady until pruned, so a raw count never drops on scale-down.	2026-06-06 00:52:18 +10:00
Josh Siviour	f9c08130ba	fix(eks): embedded etcd datastore + Alpine IMDS-route fix + CP-VM visibility (#299) ... * fix: switch EKS datastore to embedded etcd + fix Alpine cloud-init IMDS route Flip the K3s control-plane datastore from SQLite/kine back to embedded etcd (cluster-init: true, etcd-expose-metrics: true). The earlier abandonment premise — that etcd's per-commit fsync stalls past the apiserver's 5s deadline on the viperblock-backed root volume — was wrong: etcd inits fine on an unchanged viperblock (apiserver ready 25-30s, clusters reach ACTIVE). The real blocker was a network-config regression, fixed here. Alpine's cloud-init eni renderer crashes (KeyError: 'gateway') on the gateway-less IMDS on-link route, which failed the whole network-config render so the static mgmt NIC never came up (no NATS bootstrap, no /healthz). Fixes: - generateNetworkConfig: add emitIMDSRoute bool; selectNetworkConfigForFamily passes false for Alpine so the eni renderer never sees the route. - Deliver the IMDS route out-of-band via a persistent /etc/local.d script + OpenRC local service. Resolve the VPC NIC via its default route at runtime (ip route show default), not a hardcoded name: the persistent-net rename to vpc0 doesn't apply on the Alpine AMI, so the NIC is eth0 at first boot. - EKS path bakes the same route into buildK3sUserData's own write_files/runcmd; the generic template Alpine block fires only when no #cloud-config user-data, avoiding a duplicate write_files key (yaml.safe_load keeps last, silently dropping a block). - eks-node image: config.yaml.skel adds cluster-init; comments updated. * fix: surface CreateCluster step failures at ERROR + map subnet faults; add auth-mode/delete test cases CreateCluster wrapped its step failures in bare fmt.Errorf that the request surface collapses to an opaque ServerInternal ("contact AWS re:Post"), so the real cause never reached the operator. Add logCreateErr to log each step's precise cause at ERROR before returning, and map a subnet-resolve failure (caller-supplied parameter) to InvalidParameterValue instead of a retryable ServerInternal the SDK pointlessly retries. (mulga-siv-165.4) Tests (mulga-siv-165.15): reject authenticationMode=API_AND_CONFIG_MAP (only the CONFIG_MAP value was covered); DeleteCluster on a nonexistent cluster via the full-deps path returns ResourceNotFound (the shim path short-circuits to ServiceUnavailable before the meta lookup). * fix: hide EKS control-plane VMs from customer DescribeInstances EKS K3s control-plane VMs are owned by the customer account (their ENI lives in the customer VPC for the NLB target), so unlike LB microVMs (owned by the system account) they leaked into customer ec2:DescribeInstances. The spinifex:managed-by tag only filtered the UI Nodes page, not the EC2 API. Add a caller-aware exclusion: skip any ManagedBy-tagged VM unless the caller is root. Preserves operator visibility, matches LB behaviour, and leaves future customer-owned worker nodes (no ManagedBy) visible. * docs: correct stale SQLite datastore comments in eks-node image (now embedded etcd)	2026-06-05 17:55:31 +10:00
Josh Siviour	f14fb64562	feat(eks): unify eks-server + eks-agent into one eks-node AMI (#288) ... * feat(eks): unify eks-server + eks-agent into one eks-node AMI Collapse the two K3s AMIs into a single spinifex-eks-node image whose role (server vs agent) is chosen per-instance at first boot, halving build time, storage, and CVE-rotation surface. - scripts/images/eks-node/: superset manifest + merged setup.sh; all role services baked but only crond + the new eks-node-role selector enabled. - eks-node-role first-boot selector reads SPINIFEX_K3S_ROLE, rc-update adds + starts the matching services (server: k3s + eks-token-webhook + k3s-first-boot publisher; agent: k3s-agent), records /etc/spinifex-eks/role, self-disables. Paths are env-overridable; eks-node-role_test.sh covers all five cases. - mulga-eks-etcd-snapshot.sh early-exits on agent nodes. - k3s_server_vm.go seeds SPINIFEX_K3S_ROLE=server into first-boot.env. - Makefile/dev-install: build-eks-node-image / import-eks-node-image replace the server+agent targets. - Delete scripts/images/eks-server + eks-agent. The launcher resolves the AMI by the spinifex:managed-by=eks tag, so running clusters (which hold the resolved AMI ID) are unaffected. AMI bake + cluster boot are operational integration steps, not yet run. * fix(eks): order eks-node-role after cloud-config/cloud-final The first-boot role selector reads /etc/spinifex-eks/first-boot.env, which cloud-init delivers via write_files in the cloud-config/cloud-final stages. The selector only depended on net+cgroups, so on a fast boot it could start before write_files landed and read a missing role env. Add an after-ordering on cloud-config + cloud-final so the env file is always present first.	2026-06-04 12:28:56 +10:00
Josh Siviour	bfacca7789	feat(eks): 6c token webhook + AccessEntry (IAM auth → kubectl) (#284) ... * feat(eks): 6c AccessEntry CRUD + access-policy association Replace the nine notImpl() AccessEntry/AccessPolicy stubs with real KV-backed handlers (Q9, API auth mode): - AccessEntry CRUD persisted at clusters/{name}/access-entries/{hash}; the principal ARN is hashed into the key (ARNs carry ':', illegal in NATS KV keys) while the record keeps the plaintext ARN. - AssociateAccessPolicy/Disassociate/ListAssociated against the four supported AWS-managed policies; unsupported policy ARN or bad access scope -> InvalidParameterValue. - ListAccessPolicies returns the static supported catalogue. - Cluster-existence gate -> ResourceNotFoundException; duplicate create -> ResourceInUseException; node-type entries rejected until 6d. Updates the daemon routing test for the now-implemented handlers. * feat(eks): plumb caller principal ARN + bootstrap-creator-admin Thread the caller's resolved IAM principal ARN from the gateway to the EKS control plane via a new X-Principal-ARN NATS header, so CreateCluster can mint the bootstrapClusterCreatorAdminPermissions AccessEntry (Q9): - utils.NATSRequest takes optional extra headers; PrincipalARNHeader + PrincipalARNFromMsg added. - Gateway resolves the caller ARN best-effort (buildCallerARN) and passes it down the EKS route table; only CreateCluster consumes it. - CreateCluster signature gains callerPrincipalARN; default-true bootstrap flag mints a system:masters entry keyed by the caller's exact ARN (the token webhook resolves the same ARN from STS). '' degrades gracefully. - Dedicated daemon handler reads both headers. * feat(eks): IAM token webhook + STS verify-over-NATS + apiserver wiring The eks-token-webhook (baked into the eks-server AMI) authenticates kubectl requests that present an `aws eks get-token` bearer token: decode the presigned STS GetCallerIdentity URL, verify it, resolve the principal ARN to its cluster AccessEntry, and answer the apiserver TokenReview with the mapped username + groups. STS is hosted in-process in the awsgw service and is not otherwise on the bus, so awsgw now serves an `eks.VerifyToken` NATS subject that wraps VerifyPresignedGetCallerIdentity (the cluster name is forwarded so STS enforces the x-k8s-aws-id anti-replay pin). The webhook reads the AccessEntry KV directly over its own NATS connection. Apiserver wiring: the k3s config skeleton sets authentication-token-webhook-config-file; the webhook writes its own self-signed serving cert + the apiserver kubeconfig at startup (ordered `before k3s`, with a bounded wait in the k3s start_pre so the apiserver never loads a missing file). * fix(eks): omit empty spec from token webhook TokenReview response * fix(sts): presigned GetCallerIdentity must hash empty body, not UNSIGNED-PAYLOAD VerifyPresignedGetCallerIdentity built the SigV4 canonical request with "UNSIGNED-PAYLOAD" as the payload hash. That is an S3-only convention; botocore and the aws-sdk presign non-S3 requests (sts:GetCallerIdentity, the shape `aws eks get-token` produces) with the SHA256 of the empty string. Every real get-token therefore failed signature verification with InvalidIdentityToken, breaking the entire EKS IAM token-auth flow end to end. Use the empty-string SHA256. The existing hand-rolled presign tests masked the bug by signing with the same wrong constant; they now sign correctly, and a new test presigns with the real aws-sdk-go v4 signer (the same canonicalisation the CLI uses) so this can't regress silently. Verified live: a real `aws eks get-token --cluster-name dev1` now resolves to the caller principal over the awsgw eks.VerifyToken surface; an otherwise-valid token presented for a different cluster is still rejected (anti-replay intact). * fix(eks): match access-entry routes on escaped path so slashed principal ARNs resolve IAM principal ARNs (user/admin, role/foo) embed a '/' that the AWS CLI percent-encodes in the request path. The dispatcher matched r.URL.Path (decoded), so %2F became a literal '/' and split the ARN across the ([^/]+) capture, dropping every describe/update/delete-access-entry and associate/disassociate/list-access-policy call to InvalidActionException. Match r.URL.EscapedPath() so the encoded ARN stays one segment, then url.PathUnescape each captured param before it reaches the handler. * test(eks): unit-cover access-entry scope validation, AWS converters, store guards	2026-06-04 11:25:56 +10:00
Josh Siviour	bbff004b45	feat(eks): 6b create-cluster → ACTIVE (single-node SQLite + mgmt-IP NLB stop-gap) (#280) ... * feat(eks): cluster_state KV helpers — ClusterMeta + status CAS + prefix sweep Stage 1 of Sprint 6b CreateCluster lifecycle. Introduces the first read/write helpers against the eks-account-{accountID} bucket so later stages (oidc keypair, NLB wiring, cluster reconciler) can persist state without re-inventing CAS/sweep logic. - ClusterMeta struct holds the DescribeCluster source of truth (status, endpoint, OIDC issuer, CA, VPC config, control-plane VM + ENI, NLB ARN). - ClusterStatus enum mirrors the AWS EKS string set verbatim. - PutClusterMeta is unconditional (CreateCluster path); GetClusterMeta returns ErrClusterNotFound on key absence so the service boundary can map to ResourceNotFoundException. - SetClusterStatus is revision-checked with bounded CAS retries so the reconciler and CreateCluster body cannot wedge each other. - DeleteClusterPrefix sweeps every clusters/{name}/* key; called from DeleteCluster after the OIDC private key is zeroized. Tests cover round-trip, missing/corrupt blob, idempotent status set, DELETING transition from every prior state, concurrent revision bump recovery, sibling-cluster preservation, empty-bucket no-op. Plan: docs/development/feature/eks-6b-create-cluster.md * feat(eks): per-cluster ECDSA-P256 OIDC keypair generation + zeroize Stage 2 of Sprint 6b. Lays the at-rest primitives the CreateCluster body will call before launching the K3s server VM. Pure crypto helpers — no daemon, no launcher, no NATS bus. - GenerateClusterOIDCKeypair mints an ECDSA-P256 keypair per cluster (matches eks-v1.md Q8), PEM-marshals the private as PKCS8, envelope-encrypts it via handlers_iam.EncryptSecret (AES-256-GCM, AwsGW masterKey), writes the ciphertext to OIDCSigningKeyKey, and publishes the public JWKS document to OIDCJWKSKey. Returns the JWKS bytes so CreateCluster can hand them straight to user-data without a follow-up KV read. - JWKS shape mirrors what handlers_sts.JWKS already decodes for IRSA verification (RFC 7517 EC P-256, kid = base64url SHA-256 of SPKI DER, alg ES256, use sig). Types are defined locally to avoid a handlers_eks → handlers_sts → handlers_eks import cycle; JSON wire compat is the contract. - LoadClusterOIDCPrivateKey is the recovery-path read for future stages (reconciler restart, restore-snapshot tooling). Validates PKCS8 parses to an ECDSA P-256 *PrivateKey and rejects anything else. - ZeroizeClusterOIDCKey overwrites the encrypted blob with zeros of the same length, then deletes. DeleteCluster calls this *first* so a crash mid-teardown leaves no recoverable key material. Idempotent on missing key — supports retry semantics. Tests cover JWKS shape (RFC 7517 EC P-256 with 32-byte SEC1 coordinates), per-cluster key uniqueness, round-trip private-key decrypt matches JWKS public, wrong-master-key fails, missing-key ErrClusterNotFound mapping, zeroize idempotency, JWKS survives zeroize (carries no secret material), padCoord left-pad correctness. Plan: docs/development/feature/eks-6b-create-cluster.md * feat(eks): cluster SG + NLB ensure/delete helpers Stage 3 of Sprint 6b CreateCluster lifecycle. Adds the customer-account SG provisioning + per-cluster NLB plumbing the CreateCluster body will call between OIDC keypair (Stage 2) and K3s VM launch (Stage 4). Pure interface-call helpers — no daemon, no KV, no NATS bus. - EnsureClusterSGs creates (or returns) eks-cluster-{name}-control-plane-sg and eks-cluster-{name}-nodegroup-sg in the customer VPC, tagged spinifex:managed-by=eks + spinifex:eks-cluster=<name> + spinifex:eks-role. Idempotent on group-name + vpc-id pair. DeleteClusterSGs is best-effort with continue-on-error sweep. - EnsureClusterNLB provisions the eks-<name> NLB (internal scheme), the eks-<name>-cp target group (TCP, port 6443, target-type ip per eks-v1.md Q13), and the TCP :443 forward listener. Returns ClusterNLB {LoadBalancerArn, TargetGroupArn, ListenerArn, DNSName} so CreateCluster can persist every ARN in ClusterMeta. Idempotent on cluster name; resource-name length validated against the AWS 32-char cap so a too-long cluster name fails with a clear error instead of a generic InvalidParameterValue downstream. - RegisterClusterTarget + DeregisterClusterTarget bind the K3s server VM ENI IP to the TG on :6443. Stage 4 calls Register once the VM has its ENI IP; DeleteCluster calls Deregister before VM termination. - DeleteClusterNLB tears down NLB + TG. DeleteLoadBalancer already cascades listener cleanup inside the ELBv2 impl. LB sweep error takes precedence so DeleteCluster surfaces the more visible failure first while the TG sweep still runs. Narrow sgProvisioner + nlbProvisioner interfaces in-package give tests hand-rolled recorder fakes; the production VPCService + ELBv2Service satisfy them structurally so no adapter is needed at Stage 6 wire-up. Tests cover fresh-creates with right names/tags/ports, idempotent reuse, mixed-existence (control-plane exists, nodegroup created), name-too-long rejection, empty-inputs rejection, missing-resource no-op on delete, and first-error-surfaced sweep continuation on delete. Plan: docs/development/feature/eks-6b-create-cluster.md * feat(ec2): RunInstances honors pre-created NetworkInterfaceId Closes the AWS-parity gap where RunInstances always auto-created a primary ENI even when the caller specified NetworkInterfaces[0].NetworkInterfaceId. The EKS Sprint 6b CreateCluster launcher (Stage 4) needs to pre-create the K3s server VM's ENI so the ENI can be tagged/owned distinctly from the launcher's instance; the existing path silently created a second ENI and left the caller's unattached. Changes: - ENICreator interface gains GetENI(accountID, eniID) → *ENIInfo. ENIInfo is a translated struct (SubnetID, VpcID, PrivateIp, MAC, Status, SecurityGroupIDs) so handlers/ec2/instance stays free of an import on handlers/ec2/vpc.ENIRecord. - daemon/vpc_adapter.go gains daemonENICreator.GetENI translating VPCServiceImpl.GetENIRecord into ENIInfo. - PrepareRunInstances inspects NetworkInterfaces[0].NetworkInterfaceId after the existing subnet/SG lift block. If non-empty, the loop dispatches to a new attachPreCreatedENI helper: - Looks up the ENI under accountID. - Rejects with InvalidNetworkInterface.InUse if Status == "in-use". - Calls AttachENI to mark the ENI in-use and bind it to the new VM. - Populates instance.ENIId / ENIMac and ec2.Instance network fields (PrivateIp, SubnetId, VpcId, SecurityGroups, NetworkInterfaces[0]). - Skips the auto-create + public-IP-auto-assign paths; the caller is expected to manage public IP out-of-band. - Auto-create path is left untouched for every other code path: callers that pass SubnetId without a NetworkInterfaceId still get a fresh ENI with the existing semantics. Tests cover the four shapes: happy path (auto-create not called, attach called once, instance populated), in-use ENI rejected with the canonical AWS error code, lookup error surfaces verbatim, attach error surfaces verbatim. * feat(eks): K3s control-plane VM launcher with cloud-init user-data LaunchK3sServerVM resolves the eks-server AMI by name, pre-creates the customer-account control-plane ENI in the supplied subnet with the control-plane SG, renders cloud-init user-data (first-boot env, OIDC private key, K3s config.yaml with tls-san + service-account args), then calls RunInstances with NetworkInterfaces[0].NetworkInterfaceId pointing at the pre-created ENI. RunInstances failures roll back the ENI so the caller does not leak a customer-account resource. TerminateK3sServerVM tears the instance and ENI down for DeleteCluster; missing identifiers are a no-op so retries stay safe. ENI + VM both live in the customer account (deviation from plan §7 which specified a SystemAccount-owned VM); cross-account ENI attach is deferred to a follow-up bead. * feat(eks): NATS bootstrap subscriber + cluster reconciler + leader lease NATSBootstrap subscribes to the four single-shot subjects the K3s server VM publishes (k3s-bootstrap-token, k3s-admin-kubeconfig, k3s-oidc-jwks, k3s-ca) on eks.bus.{accountID}.{clusterName}.{kind}. Token and admin kubeconfig are envelope-encrypted via handlers_iam.EncryptSecret and persisted to KV. The published JWKS is cross-checked against the controller-generated keypair by kid and rejected on mismatch. The CA lands on meta.CertificateAuthorityB64 via the new SetClusterCertificateAuthority CAS updater. ClusterReconciler holds a leader lease on spinifex-eks-leader/{accountID}/ {clusterName} (60s TTL, 20s refresh) and drives: - CREATING -> ACTIVE once the four bootstrap KV artifacts are present and NLB /healthz responds 200. - ACTIVE -> probe /healthz on the configured interval, warn on failure. - DELETING / FAILED -> exit (DeleteCluster body owns teardown). Loss of the leader lease (CAS conflict or different holderID) returns ErrReconcilerLeaseLost so the caller can re-acquire from another node. Tested with embedded JetStream + a stub HTTPDoer; race-clean. * feat(eks): OIDC issuer URL builder + reconciler registry ClusterOIDCIssuer renders the {gatewayBaseURL}/oidc/eks/{region}/{accountID}/ {clusterName} URL kube-apiserver advertises as service-account-issuer and that STS / external IRSA validators dereference. Validates URL safety on the path segments so a hostile cluster name cannot inject scheme/path fragments. ReconcilerRegistry tracks per-(accountID, clusterName) reconciler goroutines with idempotent Spawn (drop duplicate spawn calls), Stop (cancel one + wait for release), and StopAll (graceful daemon shutdown). RunClusterReconciler is the production SpawnReconcilerFn: constructs a ClusterReconciler, acquires the leader lease, runs Run in a goroutine. Both helpers consumed by the upcoming service_impl CreateCluster body. * feat(eks): CreateCluster/DescribeCluster/ListClusters/DeleteCluster bodies Wires the EKSServiceImpl orchestration deps (VPC SG/K3s/Subnet, NLB, Instance launcher, AMI resolver, master key, gateway base URL, region, holder ID, reconciler registry) and implements the four lifecycle methods: - CreateCluster: validate → CREATING meta → SGs → NLB → OIDC issuer + keypair → K3s server VM (cloud-init carries OIDC private PEM) → register ENI IP with NLB target group → spawn NATS bootstrap subscriber + per-cluster reconciler via the registry. - DescribeCluster: KV meta read → eks.Cluster shape, with ErrClusterNotFound translated to ResourceNotFoundException at the service boundary. - ListClusters: KV key-prefix scan + sorted maxResults/nextToken pagination. - DeleteCluster: SetClusterStatus DELETING → zeroize OIDC key → stop reconciler via registry → deregister NLB target + delete NLB → terminate K3s VM → delete SGs → delete cluster KV prefix. Daemon wiring: - buildEKSServiceDeps loads MasterKey from BaseDir/config/master.key best-effort, resolves the gateway base URL via the same precedence the LB-agent block applies later, and passes VPCService + ELBv2Service + ImageService directly while adapting InstanceServiceImpl via daemonEKSInstanceLauncher (RunInstances passes through; Terminate- Instances loops vmMgr.Get + StopOrTerminateInstance in-process). - SubnetVPCResolver is satisfied by daemonEKSSubnetResolver wrapping VPCService.GetSubnet. - Startup spawns reconcilers for every persisted CREATING/ACTIVE cluster via SpawnRegisteredReconcilers; shutdown calls eksService.Shutdown so every per-cluster goroutine releases its lease before exit. Test surface: - service_impl_test: TestEKSServiceImpl_ClusterLifecycleShimMode covers the four lifecycle methods when constructed via the back-compat shim (CreateCluster/DeleteCluster → ServerInternal, DescribeCluster → ResourceNotFound on empty bucket, ListClusters → empty list). - daemon_handlers_eks_test: drop the four lifecycle subjects from the NotImplemented routing loop; handler-side coverage now lives in handlers/eks. - awserrors: add ResourceNotFound (404) for the DescribeCluster path. * fix(eks): load master.key from config dir not service data dir eks_deps read master.key from {BaseDir}/config/master.key, but the daemon systemd unit sets SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex (a per-service data dir). admin init writes master.key to the shared config dir, so the daemon looked in a path that only resolved when a {BaseDir}/config symlink existed — awsgw has one, the daemon's nested data dir does not. Result: MasterKey=nil, depsReadyForOrchestration() false, CreateCluster/DeleteCluster collapse to ServerInternal. master.key is a config artifact written next to spinifex.toml. Read it from filepath.Dir(d.configPath), matching how the daemon already resolves predastore.toml and TLS certs. No symlink or provisioning change needed. * fix(eks): register + resolve eks-server AMI via tag, add import --ami-name CreateCluster's K3s launch failed 'eks: AMI eks-server not found' on every clean install. Two defects: 1. spx admin images import had no way to name a locally-built appliance AMI: --name is a catalog lookup (rejects non-catalog values) and the AMI name was hardcoded to ami-{distro}-{version}-{arch}. build-system-image.sh ran 'import --name $AMI_NAME' for eks-server/eks-agent/GPU images, which all failed 'Image name not found in available images'. 2. The launcher resolved DescribeImages by name=='eks-server', a name the pipeline could never produce. Add --ami-name to override the registered AMI name on --file imports (fixes eks-server, eks-agent, and GPU appliance imports). Resolve the EKS AMI by the spinifex:managed-by=eks tag the pipeline already stamps, newest-wins on multiple matches. Tag-based resolution survives the planned server+agent → single EKS AMI unify untouched (role chosen per-instance at launch). build-system-image.sh now passes --ami-name; Makefile gains import-eks-server-image (IMPORT=1) and dev-install.sh imports the AMI when DEV_INSTALL_EKS=1. * fix(eks): give K3s server VM NATS token + CA so bootstrap can publish The K3s control-plane VM was launched with NATSURL=ConnectedUrl() (loopback, unreachable from the VM) and no auth material, while spinifex NATS requires a shared token over TLS. The VM could not connect to NATS, so it never published its four one-shot bootstrap messages and the cluster stayed in CREATING. Hand the VM the daemon's shared NATS token + CA PEM and a VM-reachable NATS URL (mirrors the ELBv2 LB-agent shared-credential pattern; spinifex has no nkeys hierarchy to scope per-principal creds against). - K3sServerInput: drop dead NATSCredsContent; add NATSToken + NATSCACert, required by validateK3sServerInput. user-data writes SPINIFEX_NATS_TOKEN, SPINIFEX_NATS_CA + the CA file; first-boot.env tightened to 0600. - EKSServiceDeps + CreateCluster: pass NATSURL/NATSToken/NATSCACert. - buildEKSServiceDeps: read CA from config.NATS.CACert; extract resolveGatewayHost(); add resolveNATSURL() (same reachable host, NATS port). - k3s-first-boot.sh: build nats CLI args with --token + --tlsca. * fix(eks): retry transient bootstrap persists, fail cluster on permanent error The NATS bootstrap subscriber tore down on the first persist error and the cluster sat in CREATING forever. Core-NATS delivers a publication once, so a transient KV blip during persist was unrecoverable and a malformed payload (bad JWKS kid, non-base64 CA, empty envelope) silently stranded the cluster. - persistWithRetry rides out transient persist failures in-handler (3 attempts, 200ms backoff) instead of unsubscribing and waiting for a re-publish that never comes. - errBootstrapPermanent classifies validation failures (malformed envelope, empty fields, JWKS kid mismatch, non-base64 CA) as non-retryable so they return immediately. - spawnBootstrap marks the cluster FAILED (with reason) when Run exits with a non-context error, so DescribeCluster surfaces the cause and DeleteCluster can reclaim resources. - MarkClusterFailed transitions to FAILED+StatusReason only from CREATING, so a late bootstrap error cannot clobber a concurrent delete or healthy cluster. - casUpdateMeta extracts the revision-checked RMW loop shared by the meta setters. * fix(eks): resume NATS bootstrap for CREATING clusters on daemon restart SpawnRegisteredReconcilers re-spawned only the reconciler on boot, never the bootstrap subscriber. A daemon restart while a cluster was CREATING (K3s VM still booting) lost the subscriptions; the VM published its four one-shot core-NATS messages to nobody and the cluster hung in CREATING forever. - BootstrapPendingKinds reports which artifacts (token, kubeconfig, CA) are still missing from KV. JWKS is excluded: it persists no artifact and re-waiting on a once-published subject would deadlock — its cross-check is best-effort across a restart. - RunForKinds subscribes to a subject subset (nil = all four); Run delegates to it. subjectHandlers maps each kind to its handler and rejects unknown kinds. - SpawnRegisteredReconcilers resumes bootstrap for CREATING clusters, waiting only on the missing artifacts, so a mid-CREATING restart still drives the cluster to ACTIVE. * fix(eks): DeleteCluster must not wipe KV or report success on teardown failure DeleteCluster ran zeroize, NLB teardown, and VM termination as warn-and-continue then unconditionally swept the per-cluster KV prefix and returned success. A failed NLB or VM teardown leaked billable resources with no owning record (the meta holding their ARNs/IDs was erased) and told the operator the cluster was gone. Zeroize — a stated security guarantee of no recoverable key material — had its failure ignored too. - Zeroize, NLB delete, and VM terminate are now blocking: their failures accumulate (errors.Join) and abort the call before the KV sweep, leaving the cluster DELETING and returning the error so a retry can still reach the ARNs. - KV prefix sweep runs only after NLB + VM teardown confirm. - SG cleanup stays best-effort (plan step 5) — a leaked SG strands no owning record — and target deregister stays best-effort since DeleteClusterNLB tears down the whole NLB + target group. * fix(eks): persist NLB + control-plane ARNs mid-CreateCluster so failures don't leak CreateCluster set meta.NLBArn / target-group / instance / ENI in memory and persisted them only in the final PutClusterMeta, after LaunchK3sServerVM. Any failure between EnsureClusterNLB and that persist left the NLB + target group + listener + backing LB VM created but unrecorded; markFailed did no rollback and DeleteCluster keys teardown off the persisted ARNs, so the resources leaked. - Persist meta immediately after EnsureClusterNLB (records NLB/TG ARNs) and again after LaunchK3sServerVM (records instance + ENI), before the next fallible step. - A failed create now leaves a FAILED meta carrying the resource references, so the DeleteCluster teardown path (165.9) reclaims every NLB/TG/listener/VM. - Tests: NLB-arn-persisted-before-later-failure, failed-create-then-delete reclaims NLB, and a CreateCluster happy path. Shared eksServiceFixture extracted from the delete-cluster fixture. * fix(sts): parse path-based EKS OIDC issuer so IRSA accepts 6b cluster tokens ParseEKSIssuerURL required an `oidc.eks.` host prefix and an exactly-2-segment path (AWS hosted-EKS shape), but ClusterOIDCIssuer emits — and awsgw serves — https://{gateway-host}/oidc/eks/{region}/{accountID}/{clusterName}. Every 6b cluster's iss claim was rejected with "issuer host must start with oidc.eks", so AssumeRoleWithWebIdentity would fail all IRSA tokens once 6e serves them. Reconcile on the path-based shape, which is what is actually reachable on bare-metal/on-prem (no oidc.eks.* vhost or DNS exists). Host trust comes from the IAM OIDC-provider registration (the full issuer string is the registry key) and the JWKS signature check, not a host pattern — so the parser drops the host prefix rule and instead pins accountID/clusterName at fixed positions in a strict 5-segment /oidc/eks/{region}/{accountID}/{clusterName} path. A crafted iss with extra or missing segments still fails closed. Adds a cross-component round-trip test asserting a ClusterOIDCIssuer-built iss parses back to the same (accountID, clusterName). * fix(eks): reclaim FAILED clusters on retry + cluster-scoped exists/not-found errors A failed CreateCluster left a FAILED ClusterMeta that poisoned retries: the preflight returned ErrorELBv2TargetGroupInUse ("The target group is currently in use...") for a cluster-exists condition, and because that maps to a retryable code the SDK's auto-retry surfaced ResourceInUseException instead of the original failure. FAILED clusters were also a dead state — visible to neither a clean recreate nor a clear lifecycle. - Add EKS-specific ResourceInUseException / ResourceNotFoundException codes with cluster-scoped messages; CreateCluster's exists check and DescribeCluster / DeleteCluster not-found now use them instead of the ELBv2 / EC2 codes. - CreateCluster reclaims a FAILED cluster: it purges the prior attempt's recorded resources and clears KV before recreating, so a retry (including the SDK auto-retry) is idempotent rather than blocked. Teardown failure surfaces rather than recreating on top of orphans. - Extract purgeClusterInfra, shared by DeleteCluster and the reclaim path (blocking zeroize/NLB/VM, best-effort SG, KV sweep only after teardown confirms). - FAILED clusters remain observable via Describe (status=FAILED) and List — no state where create blocks but describe/list show nothing. * fix(eks): fail CREATING clusters that exceed create timeout A per-cluster reconciler that never observes a ready control plane (bootstrap artifacts missing or healthz unreachable) previously stayed CREATING forever. Add a configurable create timeout (default 15m): once meta.CreatedAt is older than the deadline and the cluster is still not ready, the reconciler marks it FAILED with a reason, exiting the loop. Within the deadline it stays CREATING. * fix(eks): surface ACTIVE cluster health in describe-cluster The reconciler only slog.Warn'd a failing /healthz probe for an ACTIVE cluster, so describe-cluster kept reporting a dead control plane as healthy. Add HealthIssue + LastHealthProbe to ClusterMeta, write them from the ACTIVE reconcile branch (recording the failure reason, clearing on recovery), and map a non-empty HealthIssue to a ClusterHealth ClusterUnreachable issue in describe-cluster. * test(eks): add plan-mandated CreateCluster/DeleteCluster service cases service_impl_test only covered shim mode. Add the four cases the plan's Testing section requires: CONFIG_MAP auth mode rejected (and API accepted), missing subnetIds rejected, DescribeCluster 404 with full deps wired (past the shim short-circuit), and the DeleteCluster zeroize-before-teardown ordering proven by forcing a teardown failure and asserting the OIDC key is already wiped. * refactor(eks): return OIDC private-key PEM from generator, drop re-read CreateCluster discarded GenerateClusterOIDCKeypair's return then did a second KV get + AES-GCM decrypt via loadOIDCPrivateKeyPEM to re-derive the same PEM the generator already had in hand. Return the plaintext PEM (alongside the JWKS) from the generator and use it directly; remove loadOIDCPrivateKeyPEM and the now -unused crypto/x509 + encoding/pem imports in service_impl. * fix(eks): make CreateCluster/DeleteCluster failures diagnosable Unmet orchestration deps and a missing eks-server AMI both collapsed to a bare ServerInternal with no request-time log, leaving the client a 'contact AWS re:Post' message and zero cause. Enumerate the specific missing deps and log them at ERROR, returning ServiceUnavailable (an operator/config fault, not an unrecoverable internal one). Add an ErrEKSServerAMINotFound sentinel so a missing control-plane AMI is logged and surfaced as ServiceUnavailable too. * fix(eks): serialize REST-JSON responses via jsonutil so the AWS SDK can parse them WriteJSONResponse marshaled aws-sdk-go *Output structs with encoding/json, which ignores their locationName tags and emits Go PascalCase keys. EKS clients parse restjson on the lowercase locationName, so every field of list/describe/create/delete responses was silently dropped (empty endpoint, null identity.oidc.issuer). Marshal via jsonutil.BuildJSON — the same marshaler the SDK uses for restjson bodies — which honors locationName. Adds wire-shape tests asserting lowercase keys present and PascalCase absent. * feat(eks): launch K3s server VM via boot-agnostic system-instance path The EKS K3s control-plane VM launched through the plain tenant RunInstances path, so it had only its VPC ENI and no management-bridge NIC -- no route to the daemon's NATS endpoint. It never published its bootstrap subjects and the cluster stayed CREATING. Generalize LaunchSystemInstance to a boot-agnostic contract supporting both system-instance boot styles: - BootDirect: direct-boot microVM (fw_cfg blobs) -- ELBv2 LB agent - BootAMI: AMI boot + cloud-init seed (q35) -- EKS K3s server VM The launch contract moves to a neutral handlers/sysinstance package so EKS can launch system VMs without an eks->elbv2 dependency; handlers/elbv2 re-exports the types as aliases. The BootAMI branch mirrors the daemon's Prepare -> Insert -> Launch split and allocates the mgmt NIC on the VM record between Insert and Launch, so the cloud-init network-config renders a static mgmt0 interface. The EKS NATSURL is now sourced from the mgmt-bridge IP (the on-link address the VM's mgmt NIC can reach), falling back to the gateway-host resolution when no mgmt bridge exists. Verified: K3s VM now boots from the eks-server AMI under q35 with a mgmt0 NIC (10.15.8.x), reachable from the host over br-mgmt, with NATS on-link at the bridge IP. No AMI rebuild required. * fix(eks): make DeleteCluster K3s teardown idempotent so it can't wedge TerminateK3sServerVM treated every DeleteNetworkInterface error -- and every TerminateSystemInstance error -- as blocking. Because instance terminate is async, the immediate ENI delete races it: the first attempt fails InvalidNetworkInterface.InUse, and once the instance-terminate cascade releases the ENI every retry fails InvalidNetworkInterfaceID.NotFound. Both joined into purgeClusterInfra's teardownErrs gate, which short-circuits before the SG sweep and KV prefix delete -- so once the ENI was gone the cluster was stuck DELETING permanently, leaking its control-plane + nodegroup SGs and KV meta. Classify the teardown errors instead of blanket-propagating: - ENI InvalidNetworkInterfaceID.NotFound / InvalidNetworkInterface.NotFound -> idempotent success (the cascade or a prior retry already deleted it); must not block the downstream SG + KV sweep. - ENI InvalidNetworkInterface.InUse (VM still terminating) and any other error -> retryable; surfaced so the cluster stays DELETING and the reconciler retries once the instance releases the ENI. - Instance already gone -> idempotent. TerminateSystemInstance now returns a sysinstance.ErrSystemInstanceNotFound sentinel that the EKS path tolerates via errors.Is, so a retried delete after the VM drained completes. Add awserrors.IsErrorCode(err, code) as the shared delete-error classifier (substring match through %w wrapping) and route the lone ad-hoc strings.Contains guard in daemon/vm_adapters.go through it. Verified: delete-cluster on three clusters left wedged DELETING / FAILED (ENI + VM already gone) all complete -- clusters erased, SGs + ENIs reclaimed, no leak. * fix(eks): gate ACTIVE on verified JWKS cross-check, not pre-seeded key The controller pre-seeds OIDCJWKSKey at create time, so its presence proved nothing about the running cluster's actual signing key. A cluster could reach ACTIVE — and start signing IRSA tokens — before the K3s VM's published JWKS was ever cross-checked against the generated keypair. Add an OIDCJWKSVerifiedKey marker written by persistJWKS only after assertJWKSMatch passes. The reconciler's ACTIVE gate and BootstrapPendingKinds now key off the marker, not the pre-seeded JWKS. assertJWKSMatch is map-based and also rejects kty mismatch. * fix(eks): collapse duplicated gateway-host switch into resolveGatewayHost startCluster reimplemented the 5-case gateway-host precedence inline, duplicating resolveGatewayHost (used by the OIDC issuer and EKS NATS URL). If one drifted, the OIDC issuer host embedded in cluster tokens would diverge from the host LB VMs use to reach the daemon -> IRSA iss mismatch and cluster-wide token rejection. startCluster now calls resolveGatewayHost and derives the multi-node mgmt host route (mgmtRouteVia) from the resolved host. resolveGatewayHost carries the canonical case enumeration; a table test pins all five branches plus the case-1 skip guards and the no-host fallthrough. * fix(eks): make eks-server VM boot diagnosable + fix first-boot publisher crash Three issues found while diagnosing why K3s clusters never leave CREATING: 1. setup.sh: the stock Alpine cloud image lists `console=tty0` LAST in default_kernel_opts, so Linux binds /dev/console to the framebuffer and the host-side serial capture sees only kernel dmesg — OpenRC, cloud-init, and k3s-first-boot output were invisible. Reorder so console=ttyS0 is last. 2. k3s-first-boot.initd: command_background was "no" while output_log/error_log were set; start-stop-daemon rejects --stdout/--stderr without --background ("Failed to start k3s-first-boot"), so the bootstrap publisher never ran and the cluster wedged in CREATING. Background it (the publisher waits up to 5min on K3s /healthz and must not block the runlevel anyway) and route its output to the serial console, which the orchestrator captures (the VM has no login). 3. k3s-first-boot.sh: on the /healthz timeout, dump the tail of /var/log/k3s.log to the console so the apiserver's own failure is visible host-side. * fix(eks): seed K3s apiserver a public OIDC key, not the private key buildK3sUserData pointed both kube-apiserver service-account args at the same private-key file. --service-account-key-file requires a PUBLIC key — k8s ParsePublicKeysPEM rejects a private-key PEM with "data does not contain any valid RSA or ECDSA public keys" and the apiserver crash-loops, so K3s never reaches /healthz, first-boot never publishes, and the cluster wedges in CREATING. Add PublicKeyPEMFromPrivate (PKIX public PEM from the PKCS#8 private key) and seed it to a separate /etc/rancher/k3s/oidc-signing-key.pub.pem. Point service-account-key-file at the public file; keep service-account-signing-key-file on the private file. Runtime-verified: the apiserver crash is gone and K3s now starts and schedules system pods. * feat(eks): egress-only SNAT + DNS for K3s control-plane VM The K3s server VM (a hidden system instance) could not pull system-pod images: it had no internet egress and no working resolver, so coredns/traefik/metrics-server never started and /healthz timed out. Egress: give the single control-plane VM egress-only internet, confined to its /32, with no inbound exposure and no customer-visible resource. - policy/route.go: SystemInstanceEgressPriority (1200, above the drop gate) + AddSystemInstanceEgress/Delete — a reroute scoped to ip4.src == <vmIP>/32 so peers in a shared subnet are unaffected. - policy/nat.go: AddSystemInstanceSNAT/Delete — a plain snat (not dnat_and_snat, so no inbound path), idempotent on its pool external IP. - external/igw.go: L5 EnsureSystemInstanceEgress/Remove resolves the gateway nexthop and attaches the reroute + snat (mirrors the NAT-GW egress path; ADR-0006 S1/S4/S5 preserved). - subscribers: vpc.add/delete-system-egress topics + handlers. - handlers/eks: allocate a hidden (ManagedBy=eks) pool address on CreateCluster and publish add; release + publish delete on teardown. EIP dep wired through EKSServiceDeps + the daemon. DNS: the eks-server AMI is Alpine/dhcpcd and /etc/resolv.conf is a dangling symlink, so DHCP-supplied resolvers are never persisted. Write a real /etc/resolv.conf from the K3s VM cloud-config via bootcmd (not write_files — a write_files entry follows the dead symlink, fails, and aborts the whole block, dropping first-boot.env + the k3s config). * feat: EKS 6b single-node SQLite pivot + bootstrap-envelope fixes Switch the K3s server VM from embedded etcd to embedded SQLite (kine) for the single-node v1, so CreateCluster reaches a healthy control plane. etcd's per-commit fsync stalls past the apiserver deadline on viperblock's deferred-durability block path; SQLite tolerates it. - k3s server: omit cluster-init (selects SQLite/kine), add anonymous-auth=true so the reconciler's anonymous /healthz NLB probe returns 200. - first-boot readiness gate: anonymous /healthz (401 under CIS anonymous-auth =false) -> authenticated 'kubectl get --raw /readyz'. - first-boot publish: emit a JSON BootstrapEnvelope for all four subjects (token, admin-kubeconfig, oidc-jwks, ca); jwks and ca were absent before. - nats pub: --stdin -> --force-stdin (nats CLI 0.4.0). - manifest: add jq (builds the bootstrap-message JSON envelopes). * feat: EKS reconciler probes control-plane mgmt IP until DNS lands The host-side cluster reconciler cannot resolve or route to the VPC-internal NLB DNS endpoint (no authoritative DNS / Eclipso yet), so CreateCluster timed out in CREATING and was marked FAILED even though the control plane was healthy. Probe the control-plane VM's br-mgmt NIC address directly — the one apiserver endpoint reachable from the daemon. Capture the mgmt IP from the launch result (SystemInstanceOutput.MgmtIP -> K3sServerOutput.MgmtIP -> ClusterMeta. ControlPlaneMgmtIP) and build the reconciler health URL as https://<mgmtIP>:6443/healthz. The probe client already sets InsecureSkipVerify, so the mgmt IP need not be in the apiserver cert SAN. Workaround until authoritative DNS + a host-reachable NLB front-end land, at which point spawnReconciler reverts to the NLB endpoint.	2026-06-03 22:29:31 +10:00
Josh Siviour	8883984f3f	feat(eks): eks-server + eks-agent AMI manifests (#273) ... * feat(eks): eks-server + eks-agent AMI manifests + builder INSTALL_FILES Manifest-driven AMI build for the two EKS image variants (eks-v1.md Q12). Lifted forward from cs-eks-6b so the K3s boot path can be smoke-tested in isolation before CreateCluster orchestration lands. Builder: - build-system-image.sh gains optional INSTALL_FILES (src:dst pairs, mode 0644, parent dirs auto-created). Symmetric with INSTALL_BINARIES. - ENABLE_SERVICES moves after INSTALL_FILES + SETUP_SCRIPT so init scripts the manifest references are on disk before rc-update enables them. eks-server image (Alpine 3.21.7, q35, 4G): - K3s v1.32.5+k3s1 (downloaded at bake time, SHA256 verified against the upstream sha256sum-amd64.txt trust anchor). - nats-cli v0.4.0 (not in Alpine main/community — pulled from upstream release zip; unzip installed + purged in same setup step). - OpenRC init scripts: k3s, eks-token-webhook, k3s-first-boot. - Cron periodics under /etc/periodic/{15min,daily}/ for state-report + etcd-snapshot. - k3s-first-boot publishes node-token + admin kubeconfig over NATS on first successful K3s healthz, then self-disables (sentinel + rc-update del). - config.yaml.skel without webhook-token-auth — wiring it before the real webhook lands (cs-eks-6c) would block anonymous probes against /healthz. Orchestrator drops the live config + restarts k3s. eks-agent image (Alpine 3.21.7, q35, 2G): - Same pinned K3s release for server/agent lock-step. - k3s-agent OpenRC service refuses to start without K3S_URL+K3S_TOKEN — fail-closed pre-orchestration. eks-token-webhook stub (cmd/eks-token-webhook/main.go): - 503 + Retry-After:30 on every path. Statically linked. Lets the AMI build succeed and kube-apiserver boot cleanly. Replaced wholesale by cs-eks-6c. Init system divergence from spec wording: Alpine ships OpenRC, not systemd. Unit files use .initd extensions and cron periodics instead of .service / .timer. Bead-level intent (auto-enable on boot, periodic execution) preserved. Smoke-tested: - eks-server: builds 300M qcow2; qemu q35 boot reaches K3s apiserver on 6443 within ~120s; HTTPS probes return valid k8s Status JSON (the 401 is K3s v1.32 default RBAC behaviour against anonymous, not a regression). - eks-agent: builds 280M qcow2; qemu boot mounts root cleanly. * feat(eks): make targets for eks-server + eks-agent AMI builds - build-system-image gains subdir-manifest fallback: tries scripts/images/$(IMAGE).conf first, then scripts/images/$(IMAGE)/manifest.conf so the new EKS layout works alongside the existing flat lb manifest. - build-eks-server-image / build-eks-agent-image — explicit convenience targets that delegate to the generic one.	2026-06-01 16:24:49 +10:00
tomnewton-mulga	9d8597317a	Feat/mig capable (#271) ... * feat(gpu): add NVIDIA MIG multi-instance GPU support Partitions MIG-capable GPUs (A100, A30, H100) into isolated slices exposed as mdev devices instead of whole-GPU VFIO passthrough. VMs attach via -device vfio-pci,sysfsdev=<mdev-path> using the GPUAttachment discriminated union (PCIAddress for whole-GPU, MdevPath for MIG). Key changes: - gpu: new MIGProfile/MIGInstance types, IsMIGCapable model map, mig.go ops (EnableMIGMode, DisableMIGMode, CreateInstances, ListInstances, DestroyAllInstances, enrichMdevPaths) - gpu.Manager: Claim() returns (*GPUDevice, *MIGInstance, error); AddMIGInstances, ReclaimByMdev for restart recovery; MarkMIGFailed per-slice - daemon: buildGPUPool helper partitions devices into whole-GPU vs MIG pools; MIGProfile config field selects profile per node or per GPU override - vm.VM: GPUAttachments []gpu.GPUAttachment replaces GPUPCIAddresses []string - admin CLI: spx admin gpu mig status/enable/disable subcommands with admin.SetMIGProfile for spinifex.toml writes - docs: COMMANDS.md updated with three new MIG management command rows * test(gpu): unit tests for MIG parsers and pool manager Cover all pure parsing functions (parseMIGMemory, isCapacityExhaustedError, extractProfileFields/GIFields, parseMIGProfiles, parseMIGInstances, parseCreatedGIID/CIID) with realistic nvidia-smi fixture output. Manager MIG tests cover AddMIGInstances, Claim/Release on mdev-backed entries, MarkMIGFailed, ReclaimByMdev, and interaction with whole-GPU MarkFailed — all using fake mdev directories in t.TempDir() so no GPU hardware is required. * feat(gpu): dynamic MIG profile-per-request with on-demand GI/CI lifecycle Replaces static config-driven MIG pre-carving with a dynamic model where the MIG profile is specified per RunInstances call via mig.<profile> instance type names (e.g. mig.1g.10gb) that mirror nvidia-smi nomenclature directly. - Manager.Claim(instanceID, profileName): fast path claims a pre-carved slice; slow path dequeues a free MIG GPU, releases the lock, calls nvidia-smi mig -cgi/-cci to carve all slices, then reclaims the lock to commit and return the first slice - Manager.Release: last release on a dynamically-carved GPU calls DestroyAllInstances and returns the GPU to freeMIGGPUs, allowing a different profile on the next claim - AddMIGGPU(dev): registers a fresh MIG GPU for dynamic allocation; AddMIGInstances (restart recovery) keeps its no-destroy semantics - GenerateMIGTypes: produces mig.<profile> InstanceTypeInfo entries from gpu.ListProfiles at daemon startup; IsMIGType/MIGProfileFromType let callers inspect these names - buildGPUPool: calls ListInstances for recovery, AddMIGGPU for fresh GPUs, ListProfiles to collect MIG profile specs for type generation; removes setupMIGInstances and migProfileForDevice - GPUClaimer.Claim gains profileName param; daemonGPUClaimer, service_impl call sites, and fakeGPUClaimer all updated * fix(gpu): correct mdev parent detection and make --profile optional for mig enable enrichMdevPaths was using os.Readlink on "parent_bus_0" (a non-existent sysfs attribute) and then on ".." (a directory, not a symlink) — both would fail on real NVIDIA hardware, breaking CreateInstances and ListInstances on daemon restart recovery. Fix: read the mdev UUID's own symlink target, which always encodes the parent GPU's PCI address in its path (e.g. .../0000:01:00.0/<uuid>). Make basePath injectable in enrichMdevPaths so the fix is covered by a new set of unit tests using temp-dir symlinks (TestEnrichMdevPaths_*). Remove the MarkFlagRequired("profile") from "spx admin gpu mig enable": Phase 2 is fully dynamic — the daemon carves slices on demand at RunInstances time. Without --profile, the command just enables MIG mode and destroys any pre-carved slices so the daemon starts with a free GPU. With --profile, behaviour is unchanged (pre-carve a static pool). * chore: writeups for smci and arcee * test: add coverage for MIG exec, manager slow-path, instancetypes and admin - Add mig_exec_test.go with fake nvidia-smi harness covering Enable/Disable MIGMode, IsMIGModeEnabled, DestroyAllInstances, ListProfiles, CreateInstances, ListInstances - Extend manager_mig_test.go with slow-path Claim/Release tests using fake nvidia-smi - Add generate_test.go coverage for IsMIGType, MIGProfileFromType, GenerateMIGTypes - Add admin_test.go coverage for SetMIGProfile (create, update, idempotent, error paths) - Make mdevBasePath a var to allow sysfs injection in tests - Add json tags to GPUAttachment fields to satisfy musttag linter Diff coverage: 75.6% (threshold: 60%). * fixed gpu passthrough doc * Revert "fixed gpu passthrough doc" This reverts commit 1e20f5ab13. * feat: Added RTX 6000 GPU MIG support and fixed docs * feat: expose GPU/MIG inventory in GetNodes and GetVMs admin API Add Manager.Snapshot() for lock-safe pool inspection. Extend NodeStatusResponse with GPUs []GPUInfo (per-physical-GPU inventory with per-slice detail for MIG) and VMInfo with GPU *VMGPUInfo (model, VRAM, profile, mdev path for the attached GPU). Populate both in handleNodeStatus and handleNodeVMs via buildGPUInventory, buildPoolLookup, and resolveVMGPU helpers. * feat: surface GPU info on EC2 instance detail page (mulga-new-03) Admin users see a GPU card (model, VRAM, attachment type, MIG profile/mdev path or PCI address) populated from the GetVMs admin API. Adds formatVRAMMiB helper and VMGPUInfo/GPUInfo/GPUSliceInfo types to the frontend. * feat: GPU columns + inventory card on Nodes page (mulga-new-04) Adds GPU column to VMsTable (model/VRAM or MIG profile) and ResourceTable (alloc/total GPUs). Section 4 GPU Inventory card shows per-node GPU state with per-slice MIG drill-down. Also adds total_gpus/alloc_gpus to NodeInfo type. * feat: grouped GPU-aware instance type picker in Run Instances (mulga-new-05) Splits the flat instance type dropdown into Standard and GPU-Accelerated groups. GPU items show total VRAM alongside the type name. Selecting a GPU type surfaces an info line with GPU name and VRAM. * testing image builder scripts * fixed nvidia build script * more descriptive errors for image build script * more fixes * more fixes * fix: surface PCI address in whole-GPU VMGPUInfo and rebuild frontend dist Add PCIAddress to VMGPUInfo, populate it in resolveVMGPU for whole-GPU passthrough, and fix the instance detail page which was incorrectly showing instance type in the PCI address field. Rebuilt frontend dist to include GPU UI changes (inventory card, resource columns, GPU-aware instance type picker). * fix: improve kernel version detection in nvidia image build script Fall back to /lib/modules when /boot/vmlinuz-* is absent, and add diagnostic output to help debug kernel detection failures. * fixing nvidia image build script * fix ami name issue * correct kernel versions * another fix * image build works, added docker and nvidia toolkit for QoL * fix: add gpg * fixing amd image builder * add universe * another fix * nvm * fix docker user group issues * make rocm land where it is supposed to * fix docker groups * feat: add Ubuntu 26.04 GPU AMI catalog entries and fix build script parted detection Add ubuntu-26.04-nvidia-gpu-x86_64 and ubuntu-26.04-amd-gpu-x86_64 to AvailableImages, hosted on iso.mulgadc.com/system-ami/ with SHA256 checksums. Fix build-system-image.sh to detect parted in /usr/sbin when not on PATH. Suppress gosec false positives in DHCP manager (cancel stored in leaseLoop) and promisc ioctl (standard syscall uintptr pattern). * Updated passthrough doc regarding GPU images * fix: remove orphaned nolint directives (golangci-lint v2.12.2 no longer flags G115/G118) * chore: remove stale spinifex@192.168.3.107 directory Scripts were accidentally committed with a node hostname as the path prefix during initial testing. Correct copies live in scripts/images/.	2026-06-01 13:48:13 +10:00
joshsiv-mulga	dcf74b84e0	refactor: strip legacy lb-alpine AMI build path post-microvm-merge ... ELBv2 LB instances now boot direct via microvm kernel+initramfs (the lb-alpine-3.21.6 system AMI is no longer used). Remove the build/import scaffolding: - delete scripts/images/lb.conf + lb-setup.sh (system-image bake config) - remove lb-alpine catalog entry from spinifex/utils/images.go - drop LB_IMG download + BM_LB_ALPINE_IMAGE env from e2e-baremetal.yml Tracks mulga-siv-81.	2026-05-19 22:02:00 +10:00
tomnewton-mulga	43fa6eae59	feat: Updated GPU-Passthrough - AMD support, GPU images, /demo folder (#217) ... * feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * installer grub image fixes * made gpu-passthrough more user facing * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * Fix: AMD GPU edge cases * script for running 8x instances on SMCI computer * fix: increase disk space on instances * feat: multi-GPU passthrough support for g7e.12xlarge (2× MI350X) - vm.VM: GPUPCIAddress string → GPUPCIAddresses []string for multi-GPU - instancetypes: add g7e.12xlarge (2 GPUs), GPUCountForType(), AMDmi350x model - instancetypes/generate: use GPUCountForType for GpuInfo.Count and TotalGpuMemoryInMiB - gpu/manager: Release() loops all pool entries for instanceID (not just first) - vm/lifecycle: loop over GPUPCIAddresses, emit gpu0/gpu1/... QEMU devices - daemon/handlers: claim N GPUs in loop for both RunInstances and StartStoppedInstance - daemon/vm_adapters: ReclaimByAddress loops GPUPCIAddresses; ReleaseGPU checks len > 0 - scripts/launch-gpu-fleet: fixed fleet — 6× g7e.4xlarge + 1× g7e.12xlarge * feat: per-instance-type disk size in GPU fleet (300 GB single, 600 GB dual) * fix: multi-gpu instance types * fix: kill QEMU synchronously when QMP handshake fails to free VFIO device When startQEMU succeeds but newQMPClientWithHandshake times out, the instance has not yet been inserted into the manager map so vmMgr.Delete in the rollback path is a no-op. QEMU keeps running and holds the VFIO group open. The GPU pool entry gets freed, the next Claim returns the same PCI address, and every subsequent QEMU fails with 'device or resource busy'. Fix: SIGKILL QEMU and wait for it to exit before returning the QMP error, ensuring the VFIO device is released before the caller recycles the GPU pool slot. * fix: quarantine GPU after QEMU launch failure to prevent re-assignment cascade When QEMU crashes during startup (e.g. pci_irq_handler assertion on a specific GPU), the pool Release frees that address and the next NATS request immediately re-claims the same broken device. Add MarkFailed() to gpu.Manager which sets Available=false for a PCI address after a launch failure, preventing re-assignment until the daemon restarts. Call MarkFailed for each GPU address in both RunInstances and StartStoppedInstance rollback paths after the Release succeeds. * fixed instance fleet launch script * remove known hosts and stagger provisioning * improvements to launch script * more upgrades * single instance gpu instance provisioning * fixed single instance provision * smci computer demo dashboard * fixes to launch script * updated smci dashboard_server * fixed demo dashboard * updated dashboard.py * feat: GPU Passthrough fixes to passthrough process - builds proper AMD or NVIDA ready images, generalised launch script * feat: gpu-passthrough updating image builder to include gpu-ready ubuntu images * fixing image builder process * fixed image builder to remove /tmp/ file before import to free space * fix: Updated COMMANDS.md to reflect new GPU management admin commands * Update docs, update internal launch script to work with new SGs, test tweaks * Updated Docs, fixed internal launch-gpu.sh script, test tweaks * fxing CI race due to GPU load	2026-05-13 11:07:13 +10:00
tomnewton-mulga	28ca689130	feat: VFIO GPU passthrough (#188) ... * feat: add GPU discovery package (mulga-992) Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group member lookup, and a PCI device ID model table. Vendor-specific enrichment via nvidia-smi is attempted opportunistically and falls back to the built-in table when unavailable. All discovery and IOMMU logic is covered by unit tests using mock sysfs trees. * feat: add VFIO bind/unbind and GPU pool manager (mulga-993) Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device driver switching, and gpu.Manager which owns the claim/release lifecycle for the GPU pool including full IOMMU group binding, rollback on partial failure, and marking entries unavailable when rebind fails after VM termination. * feat: add G5 instance type generator with GPU model support (mulga-994) - Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup - Add generateGPUTypes to emit g5.* types with GpuInfo populated - Add IsGPUType helper for consumers filtering GPU-capable types - Update DetectAndGenerate to accept gpuModels []GPUModel parameter * feat: integrate GPU passthrough into daemon lifecycle (mulga-995) - Add gpu_passthrough config flag; discover GPUs and build manager at startup - Gate g5 instance types on GPU pool availability via canAllocateCount - Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device - Release GPU (VFIO unbind, rebind to host driver) in stopInstance - Re-claim GPU on daemon restart when QEMU process is still alive - Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS) - Fix Claim to use stored OriginalDriver when device is already vfio-pci bound * feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997) Extends build-system-image.sh to support Ubuntu alongside Alpine. Key differences handled: partition table (parted resizepart p1), bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs rc-update. DISTRO defaults to "alpine" for backward compatibility. Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script that installs nvidia-headless-550-server with DKMS pre-built against a pinned linux-image-generic kernel, blacklists nouveau, and prevents unattended-upgrades from invalidating the pre-built module. * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * feat: spx admin gpu status/enable/disable commands (mulga-1000) - admin.SetGPUPassthrough: Go implementation of the idempotent TOML write (replaces the inline Python heredoc in spx-test-gpu.sh) - spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough activation, and advertised g5 instance types via NATS NodeStatusResponse - spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP, polls until daemon confirms GPUPassthrough=true - spx admin gpu disable: refuses while GPU instances are running, otherwise writes TOML, sends SIGHUP, polls until confirmed - spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable * fixed admin * fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter the passthrough pool. enrichFromLspci() uses the system PCI ID database via lspci -vmm to populate the model name for any card not in the hardcoded table, without needing nvidia-smi or driver tools. * fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of file permissions. QEMU subprocesses also need char-vfio rw access to open /dev/vfio/N for the passed-through device. * fixing gpu bining permissions * working state of gpu passthrough on phyiscal box * installer grub image fixes * made gpu-passthrough more user facing * fixed banner * fixed banner * banner fix * gpu passthrough docs * preflight tweak * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * fixes after running /review-branch * feat: re-wire GPU claim and device injection after vm.Manager refactor GPU passthrough claim (VFIO bind) and QEMU device injection were lost when dev's vm.Manager refactor (#185/#186) moved StartInstance/ CreateQMPClient into vm/lifecycle.go. Re-add both in their correct new homes: - daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both RunInstances and StartStoppedInstance handlers; release on failure - vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag derived from instance.GPUXVGAEnabled when GPUPCIAddress is set * fix: Create spinifex user ~/.ssh and keys in firstboot --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>	2026-05-06 09:52:23 +10:00
Josh Siviour	13eae4d40b	fix(elbv2): tag-based LB AMI lookup with hard error on miss (#137) ... Replace fragile image-name substring match (-lb-/-alb-) with a DescribeImages tag filter on spinifex:managed-by=elbv2. Remove the silent fallback that picks the first image in the store when no LB image exists — CreateLoadBalancer now returns ServerInternal with a remediation hint instead of persisting a record in StateFailed. - daemon: tag filter + error return from the resolver closure - service_impl: resolver signature is func() (string, error); caches only successful results so the next call retries after an error - CreateLoadBalancer gate now requires launcher + networking; AMI is resolved inside the gate and failures short-circuit before persist - admin images import: add --name+--file combo (catalog metadata with a local file source) and a --tag key=value flag for explicit tagging of file-only imports - build-system-image.sh + lb.conf: propagate SYSTEM_TAG via --tag so locally built system AMIs get the right tag even when the catalog is bypassed Closes mulga-969	2026-04-14 12:52:28 +10:00
Josh Siviour	ab95566795	feat: initial ELBv2 implementation (#122) ... * feat: add ALB target health reporting via HAProxy stats Agent-side: periodically query HAProxy stats socket and publish target health status to NATS (elbv2.alb.<id>.health topic). Control-plane: healthChecker subscribes to health reports from all ALB agents and updates target health state in the KV store. Wire into ELBv2ServiceImpl lifecycle and DeregisterTargets cleanup. E2E: remove health check skip — targets must now become healthy. * feat: allocate public IP for internet-facing ALB VMs in daemon Add Scheme field to SystemInstanceInput so the daemon knows whether an ALB is internet-facing or internal. For internet-facing, allocate a public IP via externalIPAM and publish vpc.add-nat for the OVN dnat_and_snat rule. Internal ALBs skip public IP assignment entirely. * fix: release auto-assigned public IPs on DeleteNetworkInterface DeleteNetworkInterface previously leaked public IPs and NAT rules. Now checks if the ENI has a public IP, verifies it's not EIP-owned, and releases via externalIPAM + publishes vpc.delete-nat. Daemon injects externalIPAM and EIP KV into VPC service at init time. Tests cover: auto-assigned IP release with NAT event, EIP-owned IP skipped, and no-op when externalIPAM is not configured. * feat: enforce multi-AZ for internet-facing ALBs in pool mode (mulga-770) In CreateLoadBalancer, when DevNetworking is false (pool mode), require 2+ subnets in different AZs for internet-facing scheme. Dev mode and internal scheme skip the validation. * feat: ALB scheme unit tests + DNS internal prefix + public IP in AZ response (mulga-771) Add 9 unit tests covering scheme-dependent ALB behavior: - DNS name uses internal- prefix for internal scheme (AWS-compatible) - Internet-facing ALBs include public IP in AvailabilityZones response - Internal ALBs have no public IP in AZ response - Launcher receives correct scheme, delete cleans up VM + ENIs Also implements the code changes these tests exercise: - AvailZoneInfo.PublicIP field populated from launcher output - lbRecordToSDK populates LoadBalancerAddresses for internet-facing * feat: add custom AMI for ALB to images list * feat: add internet-facing ALB data plane E2E test (mulga-772) New E2E script that validates internet-facing ALBs get public IPs and are reachable from a peer cluster node (external validation). Skips in dev mode. * fix: set ENDPOINT to runner node IP in ALB E2E workflow * fix: correct paths filter to reference alb-e2e.yml * fix: alb ami name and workflow step to import the image * fix: spx path * fix: remove ALB AMI import from workflow and added to bootstrap.sh * fix: use two subnets in different AZs for internet-facing ALB E2E Multi-AZ enforcement (mulga-770) requires internet-facing ALBs to have subnets in 2+ AZs in pool mode. Discover AZs and create a subnet in each. * refactor: remove multi-AZ enforcement for internet-facing ALBs Team decision: ALBs must work with single-AZ and single-node clusters. Removes the multi-AZ validation, pool mode helper, associated tests, and error constant. E2E script reverted to single subnet. * fix: use correct account ID for ALB ENI public IP assignment The ENI is created under the caller's account but LaunchSystemInstance was looking it up with GlobalAccountID (system account), causing the UpdateENIPublicIP call to fail with "ENI not found". Pass the caller's account ID via SystemInstanceInput.AccountID. * fix: add debug logging for ALB public IP assignment failure * fix: expand debug logging to include peer node and more keywords * fix: lazy system AMI discovery for ALB VMs The daemon was caching the system AMI at startup, but the ALB image is imported after services start (bootstrap step 9). Replace SetSystemAMI with SetSystemAMIFunc so the AMI is resolved at request time when CreateLoadBalancer actually needs it. * fix: resolve ALB VM instance type dynamically from resource manager The instance type was hardcoded to t3.nano which may not exist on all clusters. Now the daemon provides a resolver that picks the smallest available instance type from the resource manager. * push interim changes * refactor: rewrite ALB agent from NATS client to HTTP server Replace NATS subscriptions with an HTTP server on :8405 exposing POST /config, GET /ping, and GET /health. The daemon will push config and poll health over HTTP through the VPC network, removing the agent's direct NATS dependency. - agent.go: net/http server replaces nats.Conn/Subscriptions - health.go: remove push-based reportHealth; /health handler calls queryHAProxyStats synchronously on each request - cmd/alb-agent: --nats flag replaced with --listen - tests: rewritten with httptest instead of NATS test server - health_checker_test: inline topic strings (HealthTopic removed) * refactor: replace NATS with HTTP client for daemon-to-ALB agent comms Daemon now pushes config via POST and polls ping via GET to the ALB agent's HTTP server on the VPC IP, eliminating the agent's NATS dependency from the daemon side. - service_impl: pushConfig uses http.Post, waitForAgentReady uses http.Get, both target http://<vpcIP>:8405 - albVMUserData: drop natsURL param (agent no longer needs NATS) - types: add VPCIP field to LoadBalancerRecord - daemon: remove SetNATSURL call and 10.0.2.2 translation block - arn_test: update user data tests for new signature * refactor: rewrite ALB health checker from NATS subscription to HTTP polling Health checker now polls each active ALB's GET /health endpoint on a 10s ticker instead of subscribing to a NATS wildcard topic. Removes the NATS connection dependency from the health checker. - health_checker: replace nc/sub with httpClient + stopCh polling loop - newHealthChecker: drop nc param, add agentURLFn for testability - service_impl: update newHealthChecker call site - tests: rewrite with httptest, test pollAll/pollOne paths * refactor: remove NATS wiring from ALB image and make hostfwd dev-only - alb-setup.sh: remove --nats flag, use --listen=:8405; remove placeholder HAProxy frontend on 8405 (agent owns the port now); update description and comments - service_impl: HostfwdPorts (80/443) now only set when DevNetworking is enabled — production ALBs are reached via VPC IP directly * feat: system credentials file + daemon wiring for ALB agent SigV4 auth Write config/system-credentials.json during admin init/join with the system access key. Daemon reads it at startup, resolves the gateway URL from external IPAM (or 10.0.2.2 in dev mode), and wires both into the ELBv2 service. albVMUserData now injects ALB_GATEWAY_URL, ALB_ACCESS_KEY, and ALB_SECRET_KEY into cloud-init for ALB VMs. * feat: ALB agent gateway actions + daemon handlers (heartbeat, config) Add ALBAgentHeartbeat and GetALBConfig actions to the gateway so the ALB agent can communicate via the AWS gateway instead of direct HTTP. - Add agent_types.go with input/output types for both actions - Add ALBAgentHeartbeat: processes heartbeat, transitions provisioning→active on first contact, updates LastHeartbeat, processes health report - Add GetALBConfig: returns pre-computed HAProxy config text and hash - Replace pushConfig (HTTP POST to agent) with updateStoredConfig (stores config + SHA256 hash on LB record for agent pull) - Remove waitForAgentReady polling goroutine (heartbeat handles transition) - Remove HTTP client from ELBv2ServiceImpl (no longer needed) - Strip health checker HTTP polling (pollLoop/pollAll/pollOne) — health reports now arrive directly via ALBAgentHeartbeat - Add ConfigText, ConfigHash, LastHeartbeat fields to LoadBalancerRecord - Wire NATS subscriptions + daemon handlers for both new actions - Add gateway handler files and elbv2Actions map entries - Comprehensive tests for heartbeat, config, and updateStoredConfig * feat: rewrite ALB agent as AWS SDK HTTP client (Bead 3r) Replace the HTTP server agent with a poll-based client that communicates with the gateway using SigV4-signed requests. The agent now sends ALBAgentHeartbeat every 5s with health reports and fetches config via GetALBConfig when the config hash changes. * feat: add VPC service route for ALB agent gateway access ALB VMs need to reach the gateway for heartbeat/config polling, but VPCs without an IGW have no external routing. Instead of requiring an IGW, the daemon now publishes a vpc.ensure-service-route event before launching ALB VMs. vpcd handles this by creating minimal external connectivity (external switch, localnet port, gateway router port, chassis bindings) if not already present, then adds a /32 static route for the gateway IP. Refactors handleIGWAttach to share the external connectivity setup via ensureExternalConnectivity helper. * Revert "feat: add VPC service route for ALB agent gateway access" This reverts commit d4220d8806. * feat: add management NIC infrastructure for system instance control plane Add config, IP allocator, TAP helpers, and VM struct fields for the management NIC that gives system instances (ALBs etc.) L3 access to the host control plane via br-mgmt, independent of VPC routing. - DaemonConfig.MgmtBridge (default "br-mgmt") - MgmtIPAllocator: thread-safe static IP allocation (.10-.249), rebuild from VMs on restart - generateMgmtMAC (02:a0:00 prefix), MgmtTapName, GetBridgeIPv4 - SetupMgmtTapDevice / CleanupMgmtTapDevice (plain Linux bridge TAP) - VM struct: MgmtMAC, MgmtIP, MgmtTap fields * feat: wire management NIC into QEMU launch and cloud-init config Add QEMU NIC injection for mgmt TAP (virtio-net-pci on br-mgmt), extend generateNetworkConfig to emit mgmt0 stanza with static IP, and wire mgmt MAC/IP allocation into LaunchSystemInstance. Includes mgmt TAP cleanup on instance stop and failed launch cleanup. * test: add coverage for ELBv2 service lifecycle and setter methods * fix: move br-mgmt detection before wireALBAgentConfig so gateway URL uses management bridge IP * fix: generate per-interface cloud-init config when eniMAC present without devMAC The guard in generateNetworkConfig required both eniMAC and devMAC to produce per-interface config, falling back to wildcard DHCP otherwise. System instances (ALBs) have eniMAC + mgmtMAC but no devMAC (DevNetworking is independent), so the mgmt NIC stanza was never emitted. The wildcard config does DHCP on all NICs, but br-mgmt has no DHCP server, leaving the management NIC unconfigured and the agent unable to reach the gateway. * fix: add wildcard support to ENI description filter + add ca-certificates to ALB image - ENI DescribeNetworkInterfaces description filter did exact string match, but AWS supports * wildcards. E2E test uses "ELB *" which didn't match "ELB app/my-alb/lb-123". Added matchFilter helper for wildcard matching. - ALB image was missing ca-certificates package, so cloud-init ca_certs module couldn't add the Spinifex CA to the system trust store. Agent TLS handshakes failed with "certificate signed by unknown authority". - Fixed data race in TestDeleteLoadBalancer_TerminatesVM_WithPublicIP where async TerminateSystemInstance goroutine wrote to mock fields while the test read them. * fix: detach ENI before delete on termination + defer ENI cleanup to stopInstance DeleteLoadBalancer was trying to delete ENIs synchronously, but they were still attached (in-use) to the running VM. The delete was rejected with InvalidNetworkInterface.InUse. Fix: DeleteLoadBalancer only deletes ENIs when no VM was launched (InstanceID empty). When a VM exists, ENI cleanup is deferred to stopInstance which runs after QEMU exits. stopInstance now calls DetachENI before DeleteNetworkInterface to clear the in-use status. E2E test now polls for ENI cleanup (up to 30s) since VM termination is async. * fix: detach then delete ENIs in DeleteLoadBalancer ENIs are attached (in-use) to the ALB VM, so DeleteNetworkInterface rejects them. Fix: call DetachENI first to clear in-use status, then delete. stopInstance tolerates already-deleted ENIs. * test: add internal ALB dataplane E2E test Add run-elbv2-dataplane-internal-e2e.sh replacing the generic dataplane test. Internet-facing coverage is already provided by run-elbv2-dataplane-internet-facing-e2e.sh. The internal test validates: - ALB reaches active via mgmt NIC heartbeat (no IGW dependency) - Scheme=internal, no public IP, internal- DNS prefix - Health checks pass via VPC L2 (OVN logical switch) - Round-robin traffic from a VPC client to the ALB's private IP - ENI cleanup on ALB deletion Uses a client VM inside the VPC that curls the ALB private IP and serves results via HTTP on its public IP for host-side verification. * feat: add NLB constants and data model changes (mulga-838) * feat: type-aware validation in CreateLoadBalancer (mulga-839) Accept Type=network for NLB, reject SecurityGroups for NLB, set CrossZoneEnabled defaults (false for NLB, true for ALB), and use /net/ ARN path segment for network load balancers. * feat: type-aware validation in CreateTargetGroup (mulga-840) Accept TCP/UDP/TLS/TCP_UDP protocols and apply NLB health check defaults (TCP protocol, no path/matcher, 10s timeout, 3/3 thresholds). Reject invalid protocols with InvalidParameterValue. * feat: type-aware validation in CreateListener (mulga-841) * feat: add NLB listener and protocol compatibility unit tests Cover the remaining Phase 1 test gaps: NLB CreateListener with all L4 protocols, protocol rejection for wrong LB type, listener-to-target-group protocol compatibility validation, and default protocol rejection on NLB. * feat: add TCP HAProxy config generation for NLB Add TCP mode template and generateNLBConfig function for Network Load Balancer support. GenerateHAProxyConfig now branches on LB type, producing mode-tcp configs for NLBs with tcp-check or httpchk health checks depending on the target group's health check protocol. * refactor: rename albagent to lbagent for NLB generalization Rename the ALB-specific agent package and binary to generic LB names in preparation for NLB support. Both ALB and NLB VMs use the same agent — it just applies whatever HAProxy config the control plane generates. - Rename albagent/ → lbagent/, cmd/alb-agent/ → cmd/lb-agent/ - Rename types: ALBAgentHeartbeat* → LBAgentHeartbeat*, GetALBConfig* → GetLBConfig* - Rename NATS topics: elbv2.ALBAgentHeartbeat → elbv2.LBAgentHeartbeat - Rename cloud-init user data: albVMUserData → lbVMUserData with LB_* env vars - Rename Makefile targets: build-alb-agent → build-lb-agent - Update gateway action map, daemon subscriptions, service interface - Fix pre-existing unused matchFilter function lint error - Fix awserrors test count for InvalidConfigurationRequest * feat: add sys.micro system instance type for LB VMs (mulga-845) Add sys.micro (1 vCPU, 128MB) as an internal-only instance type for system VMs (load balancers, NAT gateways). Registered in the type map for allocation but excluded from DescribeInstanceTypes, NATS subscriptions, and resource stats. Daemon SetSystemInstanceTypeFunc now returns sys.micro instead of scanning for smallest guest type. * feat: LB image shrink and rename (mulga-846) Rename alb.conf → lb.conf, alb-setup.sh → lb-setup.sh with IMAGE_SIZE reduced from 512M to 192M. Update all references to use lb-agent binary and LB_* env vars. Update COMMANDS.md for NLB support. * chore: rename file * feat: NLB E2E test and bug fixes (mulga-847) Add full NLB data plane E2E test (33 assertions, 9 phases): VPC/subnet setup, TCP target group with health checks, NLB creation (type=network), TCP listener, agent heartbeat, target health verification, TCP traffic round-robin, target deregistration, and full cleanup verification. Bug fixes found during E2E testing: - Fix HAProxy stats socket path mismatch: config used alb-/nlb- prefix but agent expects lb- prefix, preventing health reports - Fix system AMI lookup to match new -lb- image name, falling back to legacy -alb- name - Bump LB image size from 192M to 256M (Alpine base is 200M) * test: validate NLB internal scheme (mulga-857) NLB with scheme=internal works out of the box — shared code paths with ALB handle it correctly. Add unit test and E2E coverage: - Unit test: NLB internal creates with no public IP, internal- DNS prefix - E2E phases 10-14: internal NLB lifecycle with VPC client TCP traffic - E2E passed 56/56 with perfect round-robin (20/20 requests) * feat: consolidate LB e2e tests into run-lb-e2e.sh (mulga-854) Replace 3 separate LB dataplane scripts with a single consolidated test that covers all 4 variants (ALB/NLB × internet-facing/internal). - Share one VPC, subnet, and 2 dual-purpose app instances across all suites - ALB + NLB provision in parallel within each phase (internet-facing / internal) - One client VM tests both internal LBs simultaneously - Replace sleep 100 with LB health check gates - Delete standalone alb-e2e.yml, integrate into e2e.yml for both single and multi-node * chore: disable LB E2E tests in CI until dev env issue resolved * fix: read LB agent credentials from spinifex.toml instead of system-credentials.json The daemon couldn't read system-credentials.json due to systemd hardening (file owned by root:root 0600, not in ReadOnlyPaths). Use the same system credentials already loaded from spinifex.toml via the predastore config section — these are identical keys written by admin init. * feat: automate LB image build and import (mulga-861) - dev-install.sh: build lb-agent and run build-system-image.sh --import after services start so the LB image is available out of the box - build-system-image.sh: add --quiet flag to suppress build output, remove hardcoded --config path (uses auto-detected default) - images.go: rename alb-alpine → lb-alpine to match current naming * update go version * add LB e2e tests * fix: clean LB image build dir and wait for nbd device readiness Previous CI runs can leave stale state in /tmp/lb-image-build/ causing e2fsck failures. Clean the build dir before each CI build. Also add a readiness check after qemu-nbd connect to avoid race conditions. * fix: serialize nbd access with flock for concurrent CI builds Single-node and multi-node CI jobs share the same host and both build the LB system image using /dev/nbd0. Use flock to serialize the nbd connect/mount/customize/disconnect section so concurrent builds wait rather than collide. * fix: move flock to cover entire image build, not just nbd section Concurrent CI jobs share BUILD_DIR and /dev/nbd0. The lock must be acquired before downloading/copying the image, not just before nbd connect, otherwise the second job can clobber the first job's files. * fix: skip LB image build if output is fresh from concurrent CI job When concurrent CI jobs build the same image, the second job now reuses the first's output if it's less than 10 minutes old, avoiding a duplicate download + build cycle. * fix: filter AMI discovery by name in E2E tests With the LB Alpine image now imported alongside Ubuntu, the unfiltered describe-images call picked up the wrong AMI. Filter by the expected image name so tests use the Ubuntu image for instance lifecycle. * fix: filter AMI discovery by name in multi-node E2E tests Same fix as run-e2e.sh — with the LB Alpine image now imported, the unfiltered describe-images picks the wrong AMI. * fix: use awsgw bind IP as LB agent gateway URL fallback When br-mgmt is not configured and dev_networking is off (production with external pool), the gateway URL was falling through to the external IPAM pool's GatewayIP — the WAN router, not the Spinifex node. LB VMs couldn't reach the awsgw, so heartbeats never arrived and ALBs stayed stuck in provisioning. Use the awsgw bind IP instead — LB VMs on the OVN overlay can reach the host node's WAN IP where the gateway listens. * fix: capture StateReason and daemon logs when LB app instances fail Instances with user-data terminate immediately in CI. Add describe-instances StateReason query and daemon log dump to diagnose whether it's volume_preparation_failed or launch_failed. * fix: prevent orphaned AMI from breaking LB E2E and avoid stuck provisioning Three fixes for CI E2E failures: 1. run-e2e.sh: Remove custom AMI from predastore after deleting its backing snapshot. Previously the AMI remained discoverable but unusable, causing run-lb-e2e.sh to pick it and fail with NoSuchKey on the deleted snapshot. 2. run-lb-e2e.sh: Prefer ubuntu AMI by name for app instances instead of picking any non-alpine image. Also improve debug log dump to use journalctl (binary-install CI uses systemd, not log files). 3. CreateLoadBalancer: Only set provisioning state when a system VM was actually launched. Previously a failed launch still set provisioning, leaving the LB stuck forever with no agent to send heartbeats. * fix: use AWSGW bind IP as LB agent gateway URL in multi-node In multi-node deployments, the AWSGW binds to the WAN IP (e.g. 192.168.1.26:9999), but wireLBAgentConfig was using the br-mgmt IP as the gateway URL. Since the AWSGW doesn't listen on br-mgmt, the LB agent's heartbeat was refused, leaving internet-facing ALBs stuck in provisioning. Internet-facing LBs now use the AWSGW bind IP directly — the VM reaches it via VPC/external networking. Internal LBs get a host route on the mgmt NIC so they can reach the AWSGW bind IP through br-mgmt. * fix: add mgmt route via bootcmd for multi-node internal LBs Alpine cloud-init does not support v2 routes under ethernets — adding a routes: block to the network-config breaks all networking in the VM. Instead, add the host route via a bootcmd in the LB user data. This runs early in cloud-init and adds `ip route add {awsgw}/32 via {br-mgmt}` so internal LB agents can reach the AWSGW through the management NIC when the gateway binds to a specific WAN IP. Also moves the networking service from boot to default runlevel in the LB Alpine image to prevent a race with cloud-init. Verified locally: internal ALB transitions to active with AWSGW bound to WAN IP (simulating multi-node). * add LB enable flag to e2e * test: add elbv2 tests to meet diff coverage threshold Cover health checker edge cases (invalid JSON, empty servers, zero thresholds, port-0 fallback) and service_impl paths (mgmt route cloud-init, protocol validation, missing target group). * fix: swallowed health report error + needless marshal round-trip in LBAgentHeartbeat Add handleHealthReportDirect(report) to healthChecker so LBAgentHeartbeat can pass the struct directly instead of marshaling to JSON and immediately unmarshaling. Promote handleHealthReport unmarshal error from Debug to Warn so failures are visible in logs. * fix: validate credentials before launching LB VM + set failed state on launch error (mulga-867) Guard lbVMUserData with empty-value check for gatewayURL, systemAccessKey, and systemSecretKey — blank env vars previously caused agent auth-loop with no daemon-side log. Set LB state to failed (not active) when VM launch fails so the API reflects the broken data-plane instead of black-holing traffic. * fix: data race in getSystemAMI/getSystemInstanceType lazy caching Use sync.Once to guard lazy resolution of systemAMI and systemInstanceType fields. Concurrent CreateLoadBalancer calls previously raced on unsynchronized read+write of these cached values. The setter methods reset the Once so re-resolution works when the func is set after initial construction. Also fix pre-existing nosprintfhostport lint in daemon.go. * fix: use utils.PublishEvent for NAT events + return errors from isEIPOwned publishNATEvent in eni.go and daemon was copy-pasted boilerplate — now delegates to utils.PublishEvent (matching the EIP service pattern). isEIPOwned previously swallowed all KV errors and returned false, which could release an EIP-owned public IP if the store was temporarily unavailable. Now returns (bool, error) so the caller skips release on KV failure instead of assuming no EIP ownership. * fix: narrow health checker to only scan target groups for reporting LB handleHealthReportDirect was calling ListTargetGroups() on every heartbeat (every 5s per LB agent), scanning all TGs. Now uses report.LBID to follow LB → listeners → TG ARNs and fetch only the attached target groups via direct key lookups. Falls back to full scan if LBID is missing. * fix: pass region through to LB agent SigV4 signing (mulga-871) LB agent hardcoded us-east-1 as the SigV4 signing region. Thread region from ELBv2 service config → cloud-init LB_REGION env var → lb-agent --region flag → lbagent.New() → signer.Sign() call. * fix: add account-level authorization to LBAgentHeartbeat and GetLBConfig Both endpoints accepted any accountID without verifying the LB belongs to that account. Now verify accountID matches the LB's AccountID, with an exception for the system account (000000000000) which the ALB agent uses for heartbeats and config fetches. * fix: upgrade silent failure paths from Debug to Warn/Error - healthChecker target group list failure: Debug → Warn - HAProxy stats socket error in agent tick(): Debug → Warn - ENI delete EIP-owned skip: Debug → Info (operational visibility) - wireLBAgentConfig: warn when no gateway host is resolved - detectMgmtIP: warn when bridge exists but IP is empty - stale mgmt TAP cleanup: log errors instead of discarding * refactor: deduplicate MAC hash function into utils.HashMAC Three identical hash-to-MAC implementations (generateDevMAC, generateMgmtMAC, generateENIMac) now delegate to a shared utils.HashMAC(prefix, id) helper. * fix: guard LBAgentHeartbeat KV writes to state transitions or stale timestamps Previously the full LoadBalancerRecord (including ConfigText) was written to JetStream KV on every heartbeat (every 5s). Now only persists on state transitions (provisioning→active) or when LastHeartbeat is older than 60s. * fix: fail ALB launch when public IP allocation fails for internet-facing scheme Previously, IPAM allocation failure was logged as a warning and the ALB VM continued launching without a public IP. The ALB reached active state but was unreachable from the internet. Now the launch returns an error and cleans up the partially-created instance, causing the ALB to enter the failed state. * fix: retry system AMI resolution when image not yet imported sync.Once cached empty results permanently — if the daemon started before the system image was imported, all CreateLoadBalancer calls silently skipped VM creation. Replace sync.Once with a mutex that only caches non-empty results, allowing retry on subsequent calls. Same fix applied to getSystemInstanceType for consistency. * fix: propagate updateStoredConfig errors to API callers (mulga-884) updateStoredConfig and updateStoredConfigForTargetGroup now return errors instead of silently swallowing them. Callers (CreateListener, DeleteListener, RegisterTargets, DeregisterTargets) return ErrorServerInternal when config persistence fails, preventing the API from reporting success while HAProxy serves stale configuration. * test: add unit tests for wireLBAgentConfig, isEIPOwned, toHealthReport (mulga-885) * fix: parse HAProxy templates once at package level (mulga-875) generateALBConfig and generateNLBConfig were calling template.New().Parse() on every invocation. Move to package-level template.Must for one-time parsing. * fix: fail launch for internal-scheme ALBs when mgmt NIC setup fails (mulga-886) When management TAP or IP allocation fails, internal-scheme ALBs now return an error instead of silently continuing without a management NIC. Without the mgmt NIC, the LB agent cannot reach the gateway and the ALB gets stuck in provisioning forever. Internet-facing ALBs retain the existing degrade-and-continue behavior. * refactor: extract lbArnPathSegment helper for ALB/NLB ARN construction (mulga-877) The app/net path segment derivation was duplicated in buildLBArn, buildListenerArn, and inline in CreateLoadBalancer. Extracted into a single lbArnPathSegment helper. * refactor: use aws.StringValue in toHealthReport (mulga-878) Replace manual nil-check-then-deref with aws.StringValue for cleaner pointer-to-value conversion in the heartbeat health report builder. * refactor: use testutil.StartTestJetStream in health_checker_test.go (mulga-876) Replace hand-rolled setupTestNATS (~20 lines) with the shared testutil.StartTestJetStream helper. Simplify return to just *Store since no caller needed the *nats.Conn. * fix: address PR #122 review comments (mulga-887) - Extract NATEvent named type from anonymous struct in publishNATEvent - Remove unused build-lb-image.sh thin wrapper - Remove SPX_ENABLE_LB env var from e2e workflow (unused) - Remove no-op healthChecker start/stop methods - Export simple ELBv2 fields, remove trivial setter methods - Delete old e2e scripts replaced by consolidated run-lb-e2e.sh * fix: check specific error type for ENI deletion on termination (mulga-888) ENI deletion errors were all silently downgraded to Debug level, masking genuine failures (KV issues, permissions, ENI stuck in-use) that cause orphaned ENIs and leaked IPAM addresses. Now only NotFound is logged at Debug; all other errors log at Error. * fix: propagate non-ENODEV errors from GetBridgeIPv4 (mulga-890) GetBridgeIPv4 returned ("", nil) for ALL InterfaceByName failures, not just missing devices. Permission errors on /sys/class/net/ were silently treated as "bridge absent". Now only "no such network interface" returns nil; other errors propagate to the caller. * fix: remove backend address from proxy error response (mulga-892) The reverse proxy error handler included the internal backend host:port in the XML error body sent to clients. The address is already logged server-side; the client now sees a generic message. * fix: release mutex before KV store I/O in health checker (mulga-893) handleHealthReportDirect held hc.mu across PutTargetGroup calls (JetStream KV Put — network I/O), serializing concurrent heartbeats from independent LBs. Now collects changed TGs under the lock and persists them after releasing it. * fix: warn when system credentials missing in wireLBAgentConfig (mulga-894) fix: log DetachENI errors instead of discarding with blank identifier (mulga-895) * fix: detect veth bridge mode and use centralized NAT for it The veth-pair bridge topology (Linux bridge ↔ veth ↔ OVS bridge) was misclassified as direct bridge mode, causing OVN to use distributed NAT. Distributed NAT hairpin routing fails through the Linux bridge intermediary, breaking host-to-local-LB connectivity. Add BridgeModeVeth constant, detect veth-wan-ovs interface in detectBridgeMode(), and introduce useCentralizedNAT() to gate NAT mode and nat-addresses=router for both macvlan and veth modes. isMacvlanMode() is preserved for macvlan-specific behavior (MAC alignment). * fix: add retry loop for peer NLB reachability test With centralized NAT, ARP resolution for NAT external IPs relies on gratuitous ARPs from the gateway chassis. These take time to propagate to peer nodes. The peer→NLB test had a single 3s attempt while all other connectivity tests use retry loops. * fix: use request-reply for vpc.add-nat to prevent ARP propagation races publishNATEvent was fire-and-forget for vpc.add-nat, meaning the daemon continued before vpcd committed the OVN NAT rule. This caused intermittent NLB connectivity failures: the LB was reported active before OVN had the DNAT rule and sent GARPs for the public IP. Switch vpc.add-nat to request-reply (RequestEvent) so the daemon waits for OVN confirmation. Also increase nc timeout from 2-3s to 5s in E2E tests for more reliable TCP probing during ARP resolution. * fix: clean up stale NAT rules before adding new ones for reused IPs When a public IP is returned to the external pool and quickly reallocated (e.g. instance terminated, then LB created), the fire-and-forget vpc.delete-nat may not have been processed yet. handleAddNAT now removes any existing dnat_and_snat rule for the same external IP before creating the new one, preventing duplicate/conflicting OVN NAT rules. This was causing intermittent NLB connectivity failures in multi-node E2E where NLB public IPs collided with recently-freed instance IPs. * fix: clean up stale NAT rules across all VPC routers on IP reuse The previous fix (0a93081) only removed stale dnat_and_snat rules from the target VPC's router. When a public IP is freed from one VPC and reallocated to a different VPC before the fire-and-forget vpc.delete-nat is processed, the stale rule on the old VPC's router caused duplicate OVN ARP responses — traffic was black-holed when the host resolved to the wrong router's MAC. DeleteAllNATsByExternalIP now searches every logical router for stale NAT rules matching the external IP and removes them in one transaction. Also adds host connectivity diagnostics (ARP, routing, OVN NAT state) to run-lb-e2e.sh so the next CI failure surfaces the root cause of the persistent NLB-from-host issue. * fix: run LB inet tests from actual hosting node, not hardcoded node1 The host/peer connectivity tests were hardcoded: "host" = node1 (where the test script runs), "peer" = node2. But LB VMs land on whichever node picks up the NATS queue message, so the tests were randomly testing local vs remote connectivity depending on placement luck. Now the test: 1. Accepts --nodes <ip1> <ip2> ... (all cluster node IPs) 2. After LB creation, discovers which node hosts each LB VM by checking for the ENI's tap device 3. Runs "local" tests from the hosting node and "remote" tests from a different node Also updates e2e.yml to pass --nodes instead of --peer.	2026-04-10 10:27:20 +10:00
joshsiv-mulga	3129c9eb70	feat: generic system image builder + lint fixes (mulga-766) ... Extract build-alb-image.sh into manifest-driven build-system-image.sh: - alb.conf manifest + alb-setup.sh for ALB-specific config - build-alb-image.sh kept as thin wrapper for backward compat - Makefile targets: build-system-image, build-alb-image - Hardened error handling: curl --fail, e2fsck/resize2fs checks, static-link enforcement, rc-update validation, nbd device guard Also fixes 12 lint issues: errcheck, errchkjson, gosec, gofmt, modernize.	2026-03-26 22:30:19 +10:00