12 Commits

Author	SHA1	Message	Date
Ben Duncan	7163f7d62d	feat(oci): run Spinifex on Oracle Cloud, and fix the public-address and NAT gateway lifecycle (#1108) ... * feat: ship uninstall-spx.sh and fix the OCI bring-up path Adds scripts/uninstall-spx.sh, the inverse of setup.sh: it removes the software where node-reset.sh clears state and leaves the node installed. An operator who installed with `curl ... | bash` had no teardown path at all. State teardown is delegated to node-reset.sh rather than reimplemented, and setup.sh now ships both to /usr/local/share/spinifex so they are reachable without a checkout. It refuses on a multi-node cluster and --yes does not override that: a flag meaning "don't ask me" must not also mean "ignore a safety property". /var/lib/spinifex and every apt package are kept unless asked for, and both opt-ins confirm separately even under --yes. Proved by a full uninstall/reinstall round trip on dev-oci, which found two bugs neither prod nor dev-prod can reach, because both are ISO-installed: node-reset.sh exited 2, silently, at its last step and after the wipe. setup.sh creates the spinifex *group* and the spinifex-* service users but never a *user* named spinifex, so `getent passwd spinifex` exits 2 and pipefail carried it into set -e. The uninstaller broke openvswitch-switch on its first run by removing the socket-perms helper while leaving the drop-in referencing it, failing the unit at step EXEC with status 203. Fixed on both sides: the uninstaller removes drop-ins it wrote into other units' namespaces and restarts anything it failed, and setup-ovn.sh writes ExecStartPost=- so a missing helper can never stop another package's daemon. setup-ovn.sh also learns the shape OCI needs: when --wan-bridge names a Linux bridge it links it to OVS with a veth pair instead of enslaving the NIC, so the VNIC keeps its MAC identity and netplan keeps owning the addresses. It now fails on OVS datapath errors instead of reporting success, and the --management demotion guard keys on /etc/default/ovn-central rather than on database presence, which the ovn-central package creates on every node. Tests: scripts/uninstall-spx_test.sh, wired into make test-build-scripts along with shellcheck over the four scripts that ship to every node. * feat: allocate external IPs from the Oracle Cloud API An OCI VNIC drops any packet whose source is not a registered private IP object on it, so an external pool cannot be range math there — the address has to be created through the provider before it exists at all. Add source = "oci" and the allocator behind it. One allocation is two OCI objects: a secondary private IP that rides the wire, and a RESERVED public IP attached to it that the internet reaches. Allocate returns the public address because that is what AWS semantics are about, and records the pair so the host plumbing and release path can find the private half. The OCI objects are created before the binding is written, and dropped after it on release — both orders chosen so a crash leaves a discoverable leak rather than a record that lies. Reconcile squares the two on startup and is required, not optional: a leaked reserved public IP bills and holds one of the 50 per-region slots with nothing referencing it. It never touches an object without our display-name prefix, because the reference host carries operator-created secondaries on the same VNIC. Resolve the VNIC from an interface name by MAC through IMDS rather than by Oracle's VNIC ordering, which is not a documented contract. Generalise ExternalIPAM.EnableDHCP into InstallAllocator so the OCI wiring is not a second single-purpose installer. * fix: refuse source = "oci" outside routed mode, and cover the SDK path An OCI VNIC accepts exactly one MAC, its own, and delivers every inbound frame to it regardless of which of its addresses the packet is for. Measured on the reference host: an internet source reaches 10.200.0.144 as router-MAC to VNIC-MAC, the public address never appears on the wire at all, and every off-subnet destination resolves to the same next hop. Pool mode puts a per-VPC gateway LRP MAC and a per-rule external NAT MAC on the uplink and ARPs for the address, which policy.go:50 justifies with "the Linux bridge learns the per-rule external MAC on the veth port like any other MAC". An OCI VNIC does not learn MACs; it accepts one. So a pool-mode guest's egress is dropped and its ingress never reaches OVN. Routed mode keeps the address on the host, which already wears the VNIC's MAC. Pinning every LRP to that MAC does not rescue pool mode: they share one external logical switch and OVN resolves inbound frames by an L2 lookup on it, so duplicate MACs work for one VPC and silently misroute past that. The refusal names the MAC reason, because a refusal that only says "use nat mode" reads as arbitrary and gets worked around. Also test the SDK request shapes against a local endpoint. Two of them are wrong in ways reading does not catch: pagination stopping early reports a partial VNIC to reconcile, which then drops live bindings, and a nil privateIpId on detach is omitted from the body entirely, so OCI reads the update as a no-op, returns 200, and leaves the address attached and billing. * fix: install host firewall rules where a distro's catch-all reject cannot precede them Oracle's Ubuntu cloud image ships a catch-all REJECT in both INPUT and FORWARD. Two Spinifex datapaths were silently dead behind it on any such host. EnsureNATEgressRules probed with -C and appended with -A, so its ACCEPT rules landed after the REJECT and never matched: 24113 packets on the reject, 0 on both accepts, and guests had no egress. The comment named the case it did not handle, anticipating a FORWARD policy of DROP — which an append does survive — rather than an explicit REJECT rule, which it cannot. Insert at the head, and drain existing copies first, because a rule an earlier build appended still satisfies -C and would otherwise stay stranded across an upgrade. Nothing opened INPUT for the IMDS endpoints at all. The responder binds .254 and .253 per instance on the ime- devices, so guest requests are locally delivered and meet INPUT. Requests arrived and no reply ever left; cloud-init spent 490s timing out and reported the endpoint as disabled. With one wildcarded accept the same image boots in 18.6s and finds its datasource. The natuplink e2e check had the same blind spot the code did, asserting -C and therefore presence but never position. It now asserts the accepts precede the first reject. * feat: OCI address pairs, and an IMDS remap so the host keeps its own metadata An OCI external address is two objects, not one: a RESERVED public IP that OCI 1:1-NATs upstream and never puts on the wire, and a secondary private IP on the VNIC that the wire actually carries. Every AWS-facing record holds the public half; every datapath object must hold the private one. The mapping goes in policy.natManager and nowhere else. AddEIP, DeleteEIP, PruneOrphanEIPs and the HostEIPBinder all converge there, and reconcile rebuilds the rule set from AWS-facing intent on every pass — so a correctly mapped rule installed further upstream is swept on the next reconcile and replaced with one matching an address that never arrives. The seam is a DatapathResolver defaulting to identity, so every non-OCI deployment is unchanged. A resolve failure is fatal: falling back to the public address would install a rule that reads correctly in ovn-nbctl and blackholes. ocinet.Lookup is the OCI side — read-only, KV-backed, no client and no credentials, because vpcd is a different process from the daemon that owns the allocator and the bindings record is all either would consult. Two live bugs the in-memory test store was hiding, both found by making the fake report ErrNotFound the way the real store does: - KVStore.Mutate could not create, so every pool failed its own first allocation after OCI had already created and billed the pair. - Reconcile treated a missing record as a failure, aborting the one pass that exists to find an object created before its binding was written. Guest IMDS must stay on 169.254.169.254, which is also where a cloud host's own metadata service lives — the first guest launch made it a local address and took the host's metadata and DNS with it. The datapath is OpenFlow steered, so the guest-facing addresses stay put and only the host-side endpoint address moves, with a PREROUTING DNAT on ime-+ translating and conntrack un-NATing the reply. Off by default; identity when unset. The IMDS firewall rules move to vpcd. They are wildcarded over every endpoint rather than per-launch work, and spinifex-daemon holds no iptables grant and no CAP_NET_ADMIN by design, so attempting them aborted every launch. * fix: name the real constructor in the IMDS bind-address error, and mark the NAT datapath test in-package The error string pointed at DefaultHostBindAddrs, which does not exist — a caller following it would have looked for a symbol that was never written. nat_datapath_test.go reuses nat_test.go's in-package fixtures, so it carries the //test:in-package marker the package check asks for. * docs: index what each e2e suite proves Written because a datapath comment was read as an end-to-end behavioural claim and the suite that already answered it was found only after the wrong answer had been given. TestVPCEgressPaths/NATBaseline is the authority on private subnet egress, and nothing pointed at it. * fix: source-route an EIP's reply out the VNIC that holds it An EIP's reply carries the EIP as its source. On a host that pins its uplink with a policy rule, the reply took the main table instead and left by the wrong interface — which a cloud VNIC drops, because the address is registered on a different one. Ingress reached the guest and nothing came back. Every bind now mirrors the uplink's rule onto the EIP. The selector is the rule rather than the interface: both OCI VNICs sit in one subnet, so "the interface owning the EIP's subnet" names two of them, and two candidate rules is a refusal rather than a guess. A host with no source routing gets no rule and no change. * docs: name the setup-ovn recipe routed mode actually needs on OCI The OCI guide told operators to run `--wan-bridge=br-wan`, which is mutually exclusive with `--nat-uplink` and deletes the spx-nat transit veth routed mode runs on. The node then refuses to bring the uplink up, with an error naming the flag the guide should have used. Several messages also claimed routed mode is outbound-only with no public IPs. That stopped being true when the routed EIP path landed, and it reads as "what you are about to do cannot work" during the one init the OCI path documents. * feat: let routed NAT form a multi-node cluster, and carry every pool through formation Lifts the --nodes >= 2 refusal on --external-mode=nat. The refusal carried no technical justification -- it was a v1 scope decision -- but routed NAT is the only datapath a cloud VNIC can carry, so "routed NAT is single-node" had become "Spinifex on a cloud host is single-node". It is not fixed by lifting it. Measured on three OCI nodes: the cluster forms, all chassis register and guest-to-guest traffic runs at 7.1-7.7 Gbit/s, but every node claims every EIP, nodes disagree on the VPC's gateway LRP, and only the gateway chassis reaches its own guests. init now says so on stderr rather than refusing, because the refusal made the defects unreachable and therefore unfixable. Distributing the EIP datapath is the actual fix. Lifting it exposed a second defect that stops formation outright. NetworkConfig held nine flattened Pool* scalars and the init node set PoolName = "wan" unconditionally; routed NAT renders two pools and the one that matters -- nat-transit, carrying the gateway-LRP range -- was dropped. Each joiner got a single pool named "wan" holding the transit gateway and the detected WAN prefix, and the leader re-rendered its own config from the same struct, so all three nodes were wrong. Pools is now []admin.PoolData in render order, and PoolData gains json tags because it is on the wire now rather than a template input. * feat(install-node): add --external-mode so a cluster can be formed in routed NAT install-node.sh could not express routed NAT at all. It required --external-pool/--external-gateway/--external-prefix-len and passed --external-mode=pool to every spx admin init unconditionally, so the only supported way to form a routed-NAT cluster was by hand. --external-mode nat refuses the three pool flags rather than ignoring them, and threads --nat-uplink into setup-ovn.sh -- without it the WAN interface check rejects a physical NIC during formation, which is correct in pool mode and wrong in routed mode, where nothing is bridged. * fix(ocinet): stop each node deleting its peers' OCI addresses on startup Reconcile compares the binding record against the private IPs on one VNIC. The record is one key per pool and therefore cluster-wide, so on more than one node every peer's binding looks exactly like a binding whose objects OCI has lost. Each node dropped them, and direction 1 on the owning node then collected the now-unclaimed objects as leaks -- deleting the reserved public IP and private IP of a running instance. Observed on the three-node OCI cluster: node2's startup pass logged "collected a leaked address" for an address it had allocated ten minutes earlier and "dropped a stale binding" for one living on node3's VNIC. The damage did not stop at the OCI objects. With the binding gone, Lookup.DatapathIP returns the public address unchanged, so the OVN dnat_and_snat row carries the public address instead of the private half. Guest egress then leaves the VNIC sourced from a reserved public IP, which OCI drops because it enforces the registered source per VNIC -- and EnsureEIPSourceRoute no-ops too, since a public address shares no subnet with any host address. One destroyed binding takes out both directions. Binding.VNICID already exists for exactly this ("how a node that comes back from the dead tells mine from moved on without me") and Reconcile never consulted it. A binding is now only dropped when it names this node's VNIC; one naming no VNIC cannot be attributed and is left alone rather than guessed at. * fix(host): find the EIP's source-routing table when the uplink is held as a /32 uplinkSourceRuleFor identified the interface owning an EIP's subnet by finding a host address configured with a prefix that contains it. On OCI both VNICs sit in one subnet, so br-wan has to hold its address as a /32 -- a /23 there puts the subnet's on-link route in the main table at metric 0 and takes the whole subnet off the primary VNIC. A /32 contains nothing but itself, so no rule was ever found, no source rule was installed, and the reply left by the primary VNIC where OCI dropped it for carrying an unregistered source. The address is not the only record of who owns the subnet. The rule names a table, and that table holds the on-link route. Consulted as a fallback, only when the address prefixes matched nobody, so a host that already resolved this the direct way keeps its existing answer -- prod and dev-prod are unaffected. Only scope-link routes count. A default route covers every address and would make every source-routing table an equally good answer. * docs(oci): correct the multi-node story and name the oci_vnic_id constraint The nat-mode warning said EIPs and default egress were both broken on more than one node. Default egress is fine: delete a guest's dnat_and_snat row and it reaches 1.1.1.1 in 2.3 ms from a guest on any of three nodes. Only the public-IP path was broken, and the three defects behind it are now fixed. What remains is not a bug but the shape of the datapath: routed NAT centralises the EIP path on the VPC's gateway chassis, ext-shared is a localnet so the other nodes cannot reach it, and OCI drops any frame whose source is not registered on the VNIC it leaves by. So every node's pool must pin oci_vnic_id to the gateway chassis's VNIC rather than resolving its own. Say so in §5.3, where an operator writing the pool will read it, and in §9, where the cost is that a gateway change kills every public IP until the pools are repointed. The single-node cap itself is gone from the doc: a three-node cluster has been formed and measured, and the warning proceeds rather than refusing. * feat(nat): distribute the routed-NAT EIP path so it works on more than one node Routed NAT processed every EIP on the VPC's gateway chassis, and ext-shared is a localnet, so the transit segment a /32 route pointed into existed only on that one node. Measured on three nodes: all three claimed all three EIPs, the nodes disagreed on the gateway LRP, and only guests on the gateway chassis were reachable. The mechanism to fix it is not new. Pool mode already distributes EIPs by putting external_mac and logical_port on the dnat_and_snat row and letting OVN's ARP responder answer for the external IP on the external segment. Routed mode has an external segment too, the spx-nat veth pair, and it being per node is the distribution: only the node running the instance can answer ARP on it. So a routed rule is now distributed on the same terms as a pool one, and its host route becomes on-link on the transit veth instead of via a gateway LRP that may be on another chassis. bindHostEIP stops demanding that address for a distributed rule, which is what previously failed the bind on every node but one. vpcd plumbs only the ENIs local OVS reports as its own, asked of OVS rather than the Southbound DB because the question is about this host. The gateway LRP keeps its transit-pool address. Default egress routes through it and works across three nodes today, so making it link-local as pool mode does would break a working path to fix one distribution has already fixed. Pool and static are unmoved, and a test asserts that against the same input rather than leaving it to inspection. A rule with no MAC stays centralised, so a cluster part-way through an upgrade is never in a state neither shape handles. Beads: mulga-opyl7, mulga-la0pe * fix(nat): make routed-EIP host plumbing work on every node Three guests on three OCI nodes, one guest per node: only one was reachable. Two defects, both invisible on a single node. The transit veth took a random MAC per node, but OVN holds one MAC binding for the transit nexthop across the whole cluster — so every chassis addressed its egress to whichever node seeded the binding, and the other kernels dropped it at L2. The pair is node-local, so one locally-administered address everywhere is unambiguous; setup-ovn.sh pins it at creation and in the netdev unit, and EnsureUplinkPort repairs a veth made before that. Host routes and proxy-ARP also rode the leader-gated reconcile pass, so a node that never won the lease never plumbed its own guests' EIPs. BindHostEIPs does that half on its own loop on every node; Bind already leaves alone an EIP whose ENI is elsewhere, so two nodes never contend. An unresolvable address skips the sweep rather than shrinking it. Pool and static are untouched: the new pass is a no-op outside routed mode, and a test asserts it. Also adds --teardown and --dry-run to setup-ovn.sh, for re-running it on a node that is already tangled. It deliberately spares the NB/SB databases, /etc/spinifex, the system-id and the packages, and refuses to del-br a Linux bridge. * fix(nat): assert the transit MAC where vpcd actually runs it EnsureUplinkPort is never called in production — host.Wiring is constructed nowhere — so the repair landed in a path the deployed daemon does not take, and three nodes came back with three different MACs. vpcd re-asserts it beside the NAT egress rules instead, which is the routed-NAT branch that does run on every start. * fix(vpcd): make uplink-mode detection say what it saw, and check it detectBridgeMode logged only the branch it took, so a node detected as veth when it was wired for nat looked identical to one that was right. It now reports both probes every time, errors when a node carries both uplink veths (set up twice, and --teardown is the way out), and errors when external_mode and the wiring disagree — the datapath follows the wiring, so the config silently loses that argument. An explicitly configured bridge_mode gets the same cross-check, since nothing else ever compared it against the wiring. The vpc-networking doc now opens nat with "start with pool unless one of these applies" and gains a section on nat across more than one node; the admin warning for --external-mode=nat on two or more nodes says what was verified by hand and what is still uncovered. * test(nat): prove routed NAT delivers each node's own guests, on a cluster natuplink ran single-node only, which is exactly where the two defects that broke three nodes are invisible. It now admits ModeMultinode, and phase 9 takes a second shape there: a guest per node in a spread placement group, and for each one — its public IP plumbed by the node running it and by no other node, the route on-link with proxy-ARP answered there, a dnat_and_snat row carrying external_mac and logical_port, a TCP handshake in from another node, and the guest's own console verdict for egress. Every node's transit veth is checked against the one cluster-wide MAC before any guest boots. Single-node phase 9 gains the same row and on-link assertions, so the shape is held on both topologies. Two gates come from what three OCI nodes actually do: every node holds a gateway_chassis row for every VPC (the active one is the highest priority) and every node installs the VPC ingress route, so only phase 7 needs gating — it runs where the gateway port and the guest are both local, and logs which one is missing otherwise. Cell 20 is real-multi x nat with its own nightly step, leaving the permutation cells' suite set alone. * test(nat): run the VPC egress paths in nat mode, not pool mode only The single suite gated TestVPCEgressPaths and TestWANEgressSMTPBlock on external_mode = "pool", with a comment saying nat clusters have no EIP/NAT-GW support. They have had EIPs since the routed path landed, so the gate now asks whether the cluster can allocate public addresses at all — pool, or nat with a non-transit pool configured. The nat-single cell runs TestVPCEgressPaths beside natuplink, which is the only nightly coverage of a private subnet's isolation and its NAT gateway in routed mode. natuplink never creates a private subnet. * docs(config): list nat among external_mode values * fix(ci): renumber the multi-node nat cell off the bare-metal job Cell 20 is already the bare-metal single-node job, gated on the same cells input but declared in the workflow rather than e2e-cells.json. A matrix entry with the same number runs two unrelated jobs under one name, and naming 20 in a dispatch selects both. * feat(images): add Oracle Linux 10.1 and 9.8, and pin their catalog digests Oracle publishes its qcow2 checksums inline on an HTML page and ships no sums file, so the four new entries carry the expected digest in the catalog rather than a URL to fetch one from. Images gains ChecksumDigest for that, and the catalog import path verifies against it and says which of the two sources it used, so a recorded digest never reads as a fetched sums file. Pinning is safe here and only here: each URL names an immutable build (b291, b293, b178, b182) and never moves. aarch64 has a -kvm-cloud- build alongside the plain -kvm- one; the cloud build is the cloud-init image and is the one that matches x86_64's -kvm-. TestImageCatalogReachable HEADs every URL in the catalog from the general single-node suite. These artifacts live on hosts nobody here controls, so an upstream that retires a build breaks `spx admin images import` for a customer and nothing else in the suite would notice. It also asserts that every entry sets exactly one checksum source, and that no pinned entry points at a "latest" URL whose bytes would change under it. * fix(formation): carry bridge_mode onto the leader and every joiner applyNetworkConfig copied external_mode, external_interface and the pools out of the formation NetworkConfig but never bridge_mode, so a multi-node cluster wrote a spinifex.toml without the key while single-node init wrote it. Every node then fell back to autodetecting a mode that init had pinned. Nightly cell 30 caught it: TestNATUplink failed at the config phase on a spinifex.toml that did not contain bridge_mode = "nat". Deriving it from the mode rather than adding a NetworkConfig field keeps the leader and the joiners on the same rule single-node init already uses. * feat(ui): show how to ssh into an instance, and which user to use An instance detail page gave the addresses and the key pair name but never the login user, which is the one part of the command nobody can derive. It belongs to the AMI, not to Spinifex: nothing in the launch path creates or renames an account, so stock cloud-init applies the distro's own default_user. Oracle Linux is cloud-user, Debian is debian, Ubuntu is ubuntu, and guessing from the distro name gets two of those three wrong. The card resolves the user from the AMI's name and description, and an unrecognised image yields a placeholder rather than a guess, because a wrong username in a copy-paste command is worse than an honest gap. A private-only instance is called out as reachable from inside the VPC only, rather than presented as a command that will simply time out. * fix(nat): plumb host state for a NAT gateway's public address AddNATGateway installed the OVN snat row and stopped there. On a routed-NAT node the host is what delivers a public address, so an egressing private guest left with a source the host held no route back to, and the masquerade rule — which matches the transit 100.127.0.0/24 alone — never saw the packet. The gateway's address needs the same /32 route into OVN, proxy-ARP and FORWARD accepts an EIP gets. hostBindSpecs unions floatingIPSpecs with one deduped spec per NAT gateway address and feeds BindHostEIPs. It is kept apart from floatingIPSpecs deliberately: pruneOrphanEIPs and addLive read that set as the complete list of addresses allowed to hold a dnat_and_snat, and a gateway holds an snat, so widening it would stop the sweep reclaiming a stale row on an address later reused as a gateway's. AddNATGateway also binds directly, on both the create and idempotent-skip paths, so a fresh gateway is not dark for up to 30s waiting on the host EIP pass — and so a reboot that keeps the OVN row still replays the volatile half. Documents NAT gateways in nat mode in vpc-networking, and says at the head of that section that nat is a supported mode covered by nightly cells 19 and 30. * docs(oci): document Oracle Linux as a guest image, and that its user is cloud-user Four catalog entries, all UEFI. The login user is the one that catches people: opc is the default on the images Oracle publishes into OCI itself, while these are the generic KVM cloud images from yum.oracle.com, whose cloud-init default_user is cloud-user. Verified on 10.1 and 9.8 — opc, oracle and ec2-user are all refused. Also records why these four pin a checksum digest inline (Oracle ships no sums file, and each URL names an immutable build), why arm64 takes the -kvm-cloud- build, and that SELinux ships enforcing. * fix(reconcile): stop reading an unreadable intent as an empty one A wedged JetStream KV returned no keys for the ENI bucket while writes were still landing. loadPorts mapped that to an empty port map, pruneOrphanPorts read every guest LSP as an orphan against it, and every guest on the node lost DHCP and IMDS while the API kept answering and the instances stayed "running". The existing re-read guard did not help: it re-reads through the same path. Three defences, because no single one covers every shape of the fault. A failed kv.Get now fails the pass instead of dropping the record. Dropping one VPC narrows localVPCs, which filters out every ENI in it, which the sweep reads as permission to delete every guest port in that VPC — one failed read, same outcome, and it needs no exotic KV state. Unmarshal failures stay skips: that is a poison record rather than an unhealthy store, and failing on it would wedge reconcile forever. The port sweep now asks local OVS which taps are plugged in here and never deletes a port that has one. A running guest's port cannot be an orphan whatever intent says, and OVS is the one liveness signal that does not come through the KV read path the sweep is otherwise entirely driven by. Wired from vpcd; unwired callers fall back to the guard below. An unreadable OVS skips the sweep rather than deciding on intent alone. Each sweep also refuses outright when the keep set it would delete against holds nothing at all — no ENIs, no SGs, or no ports and no external IPs. That is the case local taps cannot cover, because the drift sweep is leader-gated and lists NB cluster-wide, so the leader holds no tap for another node's guest. The cost is real: a genuinely orphaned row now survives until one live record of its class exists. Leaking a stale row is the recoverable half. Three existing prune tests drove their fixtures to a wholly empty intent, which is now the refusal case; they keep one live record of the class so they still prove the prune. The Get-failure path has no unit test — there is no jetstream.JetStream fake in the tree and an embedded server cannot be made to list a key it will then refuse. * docs(oci): separate the two address quotas, and the principal that can read them The regional reserved-public-IP ceiling is tenancy-wide and does not grow with node count, so a cluster hits it before the per-VNIC 64, which does. Records the commands that read both, and that a detached reserved public IP still bills. Reading limits or usage needs a tenancy-level grant, which cannot be compartment-scoped the way the node's policy is. That is a second, read-only principal kept off every node, not an addition to SpinifexOperators. * docs(oci): give quotas their own troubleshooting section, and say we enforce none Spinifex hardcodes no address ceiling: it attempts the allocation and maps OCI's own refusal to InsufficientAddressCapacity. So a quota is raised with Oracle and nothing here is reconfigured or restarted — which is what an operator needs to know before it bites, not after. Separates the two quotas by how they scale: the regional reserved-public-IP default is tenancy-wide and raisable by ticket, the per-VNIC 64 is per node and is not. * fix(oci): follow a guest to its new node when its EIP moves OCI delivers a public address to whichever VNIC holds the private IP behind it, and nothing ever moved that private IP. Binding.VNICID was written once at allocate time, as the VNIC of whichever queue-group daemon happened to serve the AWS request, and never updated — so an address landed on the right node only by chance, and a guest that stopped and started elsewhere went dark while every OVN row, host route and firewall rule on the new node was correct. ClaimLocalAddresses closes the gap from the node that should hold the address. Local OVS decides ownership, the same authority the host EIP binder already trusts for the same question, and UpdatePrivateIp with a new VnicId does the move in one server-side call. The private IP OCID survives it, so the reserved public IP attached to it follows without being touched and the customer's address never changes. A loop per node, not leader-gated: the question is about this host's guests, and a node that never wins the reconcile lease would otherwise never claim the addresses of the instances it is running. * fix(oci): record which ENI an EIP is associated with An EIP is allocated bare and attached by a later AssociateAddress, so the OCI binding written at allocate time named no ENI and nothing ever filled it in. The address then stayed on whichever node happened to serve the allocation: the affinity pass could follow an auto-assigned address, which is allocated with its ENI, but never an EIP — the case customers actually use. BindOwner is on an optional OwnerTracker interface rather than Allocator, because only OCI has a notion of an address being delivered to one node. A static or DHCP address is on the wire everywhere its node can reach, those allocators do not implement it, and ExternalIPAM treats a pool that does not as a success with nothing to do. The same gap made the release owner guard inert for EIPs: it compares against a binding ENI that was always empty, so a stale teardown could free an address already handed to another instance. * fix(ec2): report the address an instance is actually reachable on Three faults kept DescribeInstances answering with the address launch assigned, whatever happened afterwards. The launch-time stamp lives on the stored ec2.Instance, and the stopped path only declined to add an address rather than removing one, so a stopped instance went on advertising an address it had released and that may already belong to someone else. The projection now clears it, and the NIC association with it, working on its copy so a restart can still report the address. The running path filled the public address only when it was nil, which the stamp never is, so an EIP could not be reported however correctly it was recorded. The runtime address now wins. And nothing told the rest of the system about an association: AssociateAddress wrote neither the ENI record, which is what IMDS and the network reconcile read, nor the instance record, which is what a describe answers from. The ENI write is direct; the instance write is a fan-out, because the queue-group worker that serves AssociateAddress is rarely the node running the instance. * fix(net): take the host EIP plumbing off a node its guest has left Bind refuses an EIP whose ENI is on another chassis; the prune beside it kept anything still wanted cluster-wide. A guest that moved therefore left the old node holding the /32 route into OVN and the proxy-ARP entry on the uplink, with nothing that would ever remove them. On a shared segment that node goes on answering ARP for an address another node is now serving. The two halves now ask the same question by the same authority, in the loop that already runs per node. An unreadable local OVS is not a yes: tearing down a live address on a guess would fire everywhere at once, so it keeps the binding and the next pass tries again. A binder with no chassis test behaves exactly as before, and a centralised EIP is exempt because it hairpins through the gateway chassis and every node plumbs it legitimately. * fix(net): refuse to free an Elastic IP on behalf of a departing interface Releasing an instance's public address on teardown releases it by address, scoped to the ENI holding it. Since an association now writes the EIP onto the instance record, that address is the EIP — so terminating an instance gave a customer's Elastic IP back to the pool while describe-addresses still reported it as theirs, free to be handed to another guest. The existing ownership guard cannot catch this. It only fires when the stored allocation already names an ENI, and an EIP is allocated bare and associated later, so the stored ENI is empty and the guard passes straight through. The distinction that holds is who is asking. ReleaseAddress is the address's own owner and names no interface; instance teardown names the interface it is tearing down. So a release that names an owner may free an auto-assigned address and never an EIP, which is enforced in each of the three allocators. DHCP needed the scope carried on the wire, since it releases by address and resolves the lease afterwards. Absent on an older caller, which reads as false and keeps the previous behaviour. Worst on the two providers where it is not recoverable: on DHCP the lease is the only thing holding the address and the server promises nothing about handing it back, and on OCI the address is a reserved public IP object that the release destroys outright. * test(e2e): prove an Elastic IP survives a stop and start Instance teardown releases the public address by value, and since the association writes the EIP onto the instance record, that release aims straight at the customer's Elastic IP. The allocator guards now refuse it, but nothing end to end proved the address comes back on a restart. The step goes inside EIPFlip, which already runs last on the shared public guest for the same reason a stop is safe there: the address the guest returns on is the allocation's, not the pool's. It runs on the static cells and the DHCP ones alike, and on DHCP it doubles as the only outside view of the lease still renewing while the guest is down. * feat: release an auto-assigned public address on stop, take a new one on start AWS lends an instance a public address for as long as it runs and gives it a different one on the next start. Spinifex kept the address across a stop, which holds a pool slot for a stopped instance and, on a DHCP pool, keeps renewing a lease nothing is using. A stop now offers the address back through the instance cleaner, which refuses anything an Elastic IP is behind — the EIP association record answers that, the same question an ENI delete already asks — and records on the VM that the start owes it a new one. A host drain keeps the address: Restore relaunches the guest on the next boot, and changing its address under it is an outage nobody asked for. The start allocates before the launch, so an exhausted pool refuses the start rather than booting a guest the customer cannot reach, adopts an Elastic IP associated while the instance was stopped instead of assigning a second address beside it, and hands the address back if the launch then fails. e2e: a StopStartAddress stage on the public guest asserts a stopped instance reports no address and comes back on a different one from a static pool, and a StopStartKeepsTheEIP step asserts the opposite for an Elastic IP. * docs: restructure the OCI guide around two deployment paths, with diagrams The guide had grown into a numbered wall with no shape: no frontmatter, no contents, and a reader with an OCI account had no way to tell where to start. It now follows the multi-node install template — overview, prerequisites, then the deployment path — and splits that path in two: Terraform for one node or three, which is the preferred route, and a manual single-node walkthrough for an instance that already exists. Both meet at one shared configuration section rather than repeating it. Two ASCII diagrams carry what the prose could not: one OCI instance with the whole AWS surface inside it, and three nodes with the same surface distributed, showing which plane each VNIC carries and how an address follows a guest that moves. The limits section was materially out of date and said so in the places that mattered most. Multi-node is no longer experimental, public addresses are no longer pinned to one node — each allocates on its own VNIC and the affinity pass moves the private half to wherever the guest is — and an auto-assigned address is now returned to OCI on stop, so a stopped fleet stops paying for addresses. What changed is recorded rather than quietly edited. The Terraform configuration moves in from the monorepo as scripts/terraform/oci-spx, so the guide points at something in this repository. Also marks the two new lifecycle test files in-package, with the reason each one needs to be: both substitute unexported dependencies. * docs: rework the OCI guide — SVG diagrams, Terraform variables, iSCSI and br-wan Replaces the two ASCII diagrams with SVGs in the house style, one for a single node and one for three. Renames the two deployment sections to Terraform deployment and Manual deployment, documents what the Terraform builds per node and every variable worth setting, and explains that the data volume is an iSCSI attachment rather than a disk. The manual path now covers the iSCSI login, the fstab entry and the netplan source routing that Terraform writes, and states what br-wan is for on a routed-NAT node: it names the VNIC for the OCI allocator and holds the registered addresses, but is never bridged into OVS. Opens with what Spinifex is and why OCI is worth running it on. * fix: make gateway convergence work on a multi-node routed-NAT cluster Three defects each stopped a multi-node cluster from ever attaching an internet gateway, and all three produced the same uninformative symptom: `terraform apply` sitting on `aws_internet_gateway.igw: Still creating...` forever while every service reported healthy. ovn-northd was given only its local OVSDB socket as a client connection. A RAFT follower does not forward writes -- it answers "not cluster leader, trying another server" -- so with one entry in the list, northd stops translating Northbound intent into Southbound flows the moment database leadership moves to another node. Measured on the OCI cluster: SB_Global.nb_cfg seventeen transactions behind NB_Global.nb_cfg, with no chassisredirect port for any VPC created after leadership moved. setup-ovn.sh wrote the member list only when the caller passed --ovn-remote, which only install-node.sh did; it now derives the list from the live RAFT membership otherwise, and says so when it finds a single member, because that node needs a second pass once the others join. The gateway datapath probe ran wherever the reconcile lease landed rather than on the chassis owning the port. Both probes leave from the local uplink, and under routed NAT the transit subnet exists identically on every node, so a remote probe reported a healthy gateway as dead and aimed a repair at a datapath the node does not own. GatewayPortLocal gates it. On OCI the probe then ARPed the tenant-visible public IP, which the cloud NATs and which appears on no interface anywhere. The reconciler now maps the address through the same DatapathResolver the NAT rules are built from. Also adds DescribeVpcEndpoints, which the AWS provider calls on every VPC delete to sweep GuardDuty-managed endpoints; an InvalidAction there failed the destroy and stranded the VPC. The OCI guide gains a Terraform section recording which shipped workbooks were validated against a three-node OCI cluster and which are still open. * docs: record the validated Terraform workbook results for the three-node OCI cluster Six of the seven workbooks run against the OCI cluster pass, including the three the section exists to prove: RDS, ECS and EKS. Records nginx-alb and the clean eks-quickstart re-run, and names the two lifecycle gaps that bit during the run so an operator recognises them rather than reading them as a broken workbook. * fix: route the NAT gateway's SNAT source through the datapath resolver AddNATGateway was the one NAT path that did not map its external address to the one that rides the wire. Where the public half of an allocation is NAT'd by the cloud it is on no interface anywhere, so every private-subnet guest egressed with a source that was dropped, while the gateway reported available and DNS kept working because the VPC resolver is host-local. The unmapped address also failed the host EIP sweep, which works in resolved addresses, so the bind was pruned as foreign on the next pass and reinstalled on the one after. The public half now survives as an external ID instead. Also restructures the OCI guide around numbered steps through to running Terraform, and moves the cloud-init and metadata sections into troubleshooting where they are read when something is wrong rather than followed when it is not. * fix: place a NAT gateway's OCI address on the node that forwards for it OCI drops a packet whose source is not a registered private IP on the VNIC it leaves from, so a NAT gateway's SNAT source has to be on the gateway chassis. Two independent placements decided those and nothing reconciled them: the address was created by whichever worker served AllocateAddress, while the chassis is a fixed ordering the reconcile pass asserts. On a three-node cluster that made private-subnet egress a coin flip, and the same Terraform passed or died depending on which node answered. The affinity pass already solves this for guests, by asking local OVS where the tap is. A NAT gateway has no tap, and it skipped anything with no ENI — which is exactly and only this address. The equivalent local fact is the VPC's chassisredirect binding, so ownership now has two answers rather than one, and an address that is neither a guest's nor a gateway's is still left alone. The VPC is recorded from the reconcile pass rather than from CreateNatGateway, because the answer is the gateway chassis and reconcile is what decides it. * fix: plumb a NAT gateway's host ingress only on its gateway chassis A NAT gateway's address has no ENI, so neither the bind's locality test nor the prune's could ever fire for it and every node installed its /32 route into OVN. Latent on bare metal, where pool addresses are disjoint from the node subnet; on a cloud that hands out the address from the nodes' own subnet it outranks the connected route and swallows node-to-node traffic. Give that address the same second authority the ocinet affinity pass has: the VPC's chassisredirect binding, read from the local ovn-controller. An unanswerable question still leaves the binding alone, so a single node whose gateway port is not yet claimed behaves exactly as before. The orphan EIP prune built its live set from ENI and user EIPs alone while also sweeping host ingress, so every drift pass tore down the ingress of every live NAT gateway and left it dark until the next host EIP tick. * fix: ask the gateway chassis only about a NAT gateway's address gatewayElsewhere skipped the NAT gateway marker it was named for and answered for every spec, so on a node that is not the VPC's gateway chassis every EIP — including a guest's, whose locality is its own port — was left unplumbed. Two of three nodes bound nothing. * test: assert the two private-subnet EKS workbooks in the tofu harness eks-https-ingress and eks-gitops-argocd put their workers in private subnets behind a NAT gateway, so a worker only registers Ready if its egress to the cluster endpoint and to ECR survives that path. That makes them the workbooks that catch a NAT gateway fault, and neither had an assertion, so neither could be run at all — run_workbook calls assert_<name> by construction. Both share eks-quickstart's readiness check, extracted unchanged. They also name the worker size node_instance_type rather than instance_type, and tofu rejects a -var for an undeclared variable outright, so the name is read from the config instead of kept in a list here. * test: count agents, not every node, when asserting EKS workers Ready The control plane is three K3s servers, so an unfiltered node count reaches a one-worker cluster's desired size before any worker joins — the assertion passed on exactly the failure it exists to catch, reporting "4/1 workers Ready". Filtering on the control-plane role label makes it 1/1, which is the number that says a node-group worker registered. Matters most for the two private-subnet workbooks, where the worker joining is the whole proof that egress through the NAT gateway works. * docs: record the two private-subnet EKS workbooks passing on OCI	2026-09-26 14:31:07 -05:00
Julian Sommer	19472c377a	chore: delete legacy scripts	2026-06-12 15:17:48 +10:00
Josh Siviour	f14fb64562	feat(eks): unify eks-server + eks-agent into one eks-node AMI (#288) ... * feat(eks): unify eks-server + eks-agent into one eks-node AMI Collapse the two K3s AMIs into a single spinifex-eks-node image whose role (server vs agent) is chosen per-instance at first boot, halving build time, storage, and CVE-rotation surface. - scripts/images/eks-node/: superset manifest + merged setup.sh; all role services baked but only crond + the new eks-node-role selector enabled. - eks-node-role first-boot selector reads SPINIFEX_K3S_ROLE, rc-update adds + starts the matching services (server: k3s + eks-token-webhook + k3s-first-boot publisher; agent: k3s-agent), records /etc/spinifex-eks/role, self-disables. Paths are env-overridable; eks-node-role_test.sh covers all five cases. - mulga-eks-etcd-snapshot.sh early-exits on agent nodes. - k3s_server_vm.go seeds SPINIFEX_K3S_ROLE=server into first-boot.env. - Makefile/dev-install: build-eks-node-image / import-eks-node-image replace the server+agent targets. - Delete scripts/images/eks-server + eks-agent. The launcher resolves the AMI by the spinifex:managed-by=eks tag, so running clusters (which hold the resolved AMI ID) are unaffected. AMI bake + cluster boot are operational integration steps, not yet run. * fix(eks): order eks-node-role after cloud-config/cloud-final The first-boot role selector reads /etc/spinifex-eks/first-boot.env, which cloud-init delivers via write_files in the cloud-config/cloud-final stages. The selector only depended on net+cgroups, so on a fast boot it could start before write_files landed and read a missing role env. Add an after-ordering on cloud-config + cloud-final so the env file is always present first.	2026-06-04 12:28:56 +10:00
Josh Siviour	bbff004b45	feat(eks): 6b create-cluster → ACTIVE (single-node SQLite + mgmt-IP NLB stop-gap) (#280) ... * feat(eks): cluster_state KV helpers — ClusterMeta + status CAS + prefix sweep Stage 1 of Sprint 6b CreateCluster lifecycle. Introduces the first read/write helpers against the eks-account-{accountID} bucket so later stages (oidc keypair, NLB wiring, cluster reconciler) can persist state without re-inventing CAS/sweep logic. - ClusterMeta struct holds the DescribeCluster source of truth (status, endpoint, OIDC issuer, CA, VPC config, control-plane VM + ENI, NLB ARN). - ClusterStatus enum mirrors the AWS EKS string set verbatim. - PutClusterMeta is unconditional (CreateCluster path); GetClusterMeta returns ErrClusterNotFound on key absence so the service boundary can map to ResourceNotFoundException. - SetClusterStatus is revision-checked with bounded CAS retries so the reconciler and CreateCluster body cannot wedge each other. - DeleteClusterPrefix sweeps every clusters/{name}/* key; called from DeleteCluster after the OIDC private key is zeroized. Tests cover round-trip, missing/corrupt blob, idempotent status set, DELETING transition from every prior state, concurrent revision bump recovery, sibling-cluster preservation, empty-bucket no-op. Plan: docs/development/feature/eks-6b-create-cluster.md * feat(eks): per-cluster ECDSA-P256 OIDC keypair generation + zeroize Stage 2 of Sprint 6b. Lays the at-rest primitives the CreateCluster body will call before launching the K3s server VM. Pure crypto helpers — no daemon, no launcher, no NATS bus. - GenerateClusterOIDCKeypair mints an ECDSA-P256 keypair per cluster (matches eks-v1.md Q8), PEM-marshals the private as PKCS8, envelope-encrypts it via handlers_iam.EncryptSecret (AES-256-GCM, AwsGW masterKey), writes the ciphertext to OIDCSigningKeyKey, and publishes the public JWKS document to OIDCJWKSKey. Returns the JWKS bytes so CreateCluster can hand them straight to user-data without a follow-up KV read. - JWKS shape mirrors what handlers_sts.JWKS already decodes for IRSA verification (RFC 7517 EC P-256, kid = base64url SHA-256 of SPKI DER, alg ES256, use sig). Types are defined locally to avoid a handlers_eks → handlers_sts → handlers_eks import cycle; JSON wire compat is the contract. - LoadClusterOIDCPrivateKey is the recovery-path read for future stages (reconciler restart, restore-snapshot tooling). Validates PKCS8 parses to an ECDSA P-256 *PrivateKey and rejects anything else. - ZeroizeClusterOIDCKey overwrites the encrypted blob with zeros of the same length, then deletes. DeleteCluster calls this *first* so a crash mid-teardown leaves no recoverable key material. Idempotent on missing key — supports retry semantics. Tests cover JWKS shape (RFC 7517 EC P-256 with 32-byte SEC1 coordinates), per-cluster key uniqueness, round-trip private-key decrypt matches JWKS public, wrong-master-key fails, missing-key ErrClusterNotFound mapping, zeroize idempotency, JWKS survives zeroize (carries no secret material), padCoord left-pad correctness. Plan: docs/development/feature/eks-6b-create-cluster.md * feat(eks): cluster SG + NLB ensure/delete helpers Stage 3 of Sprint 6b CreateCluster lifecycle. Adds the customer-account SG provisioning + per-cluster NLB plumbing the CreateCluster body will call between OIDC keypair (Stage 2) and K3s VM launch (Stage 4). Pure interface-call helpers — no daemon, no KV, no NATS bus. - EnsureClusterSGs creates (or returns) eks-cluster-{name}-control-plane-sg and eks-cluster-{name}-nodegroup-sg in the customer VPC, tagged spinifex:managed-by=eks + spinifex:eks-cluster=<name> + spinifex:eks-role. Idempotent on group-name + vpc-id pair. DeleteClusterSGs is best-effort with continue-on-error sweep. - EnsureClusterNLB provisions the eks-<name> NLB (internal scheme), the eks-<name>-cp target group (TCP, port 6443, target-type ip per eks-v1.md Q13), and the TCP :443 forward listener. Returns ClusterNLB {LoadBalancerArn, TargetGroupArn, ListenerArn, DNSName} so CreateCluster can persist every ARN in ClusterMeta. Idempotent on cluster name; resource-name length validated against the AWS 32-char cap so a too-long cluster name fails with a clear error instead of a generic InvalidParameterValue downstream. - RegisterClusterTarget + DeregisterClusterTarget bind the K3s server VM ENI IP to the TG on :6443. Stage 4 calls Register once the VM has its ENI IP; DeleteCluster calls Deregister before VM termination. - DeleteClusterNLB tears down NLB + TG. DeleteLoadBalancer already cascades listener cleanup inside the ELBv2 impl. LB sweep error takes precedence so DeleteCluster surfaces the more visible failure first while the TG sweep still runs. Narrow sgProvisioner + nlbProvisioner interfaces in-package give tests hand-rolled recorder fakes; the production VPCService + ELBv2Service satisfy them structurally so no adapter is needed at Stage 6 wire-up. Tests cover fresh-creates with right names/tags/ports, idempotent reuse, mixed-existence (control-plane exists, nodegroup created), name-too-long rejection, empty-inputs rejection, missing-resource no-op on delete, and first-error-surfaced sweep continuation on delete. Plan: docs/development/feature/eks-6b-create-cluster.md * feat(ec2): RunInstances honors pre-created NetworkInterfaceId Closes the AWS-parity gap where RunInstances always auto-created a primary ENI even when the caller specified NetworkInterfaces[0].NetworkInterfaceId. The EKS Sprint 6b CreateCluster launcher (Stage 4) needs to pre-create the K3s server VM's ENI so the ENI can be tagged/owned distinctly from the launcher's instance; the existing path silently created a second ENI and left the caller's unattached. Changes: - ENICreator interface gains GetENI(accountID, eniID) → *ENIInfo. ENIInfo is a translated struct (SubnetID, VpcID, PrivateIp, MAC, Status, SecurityGroupIDs) so handlers/ec2/instance stays free of an import on handlers/ec2/vpc.ENIRecord. - daemon/vpc_adapter.go gains daemonENICreator.GetENI translating VPCServiceImpl.GetENIRecord into ENIInfo. - PrepareRunInstances inspects NetworkInterfaces[0].NetworkInterfaceId after the existing subnet/SG lift block. If non-empty, the loop dispatches to a new attachPreCreatedENI helper: - Looks up the ENI under accountID. - Rejects with InvalidNetworkInterface.InUse if Status == "in-use". - Calls AttachENI to mark the ENI in-use and bind it to the new VM. - Populates instance.ENIId / ENIMac and ec2.Instance network fields (PrivateIp, SubnetId, VpcId, SecurityGroups, NetworkInterfaces[0]). - Skips the auto-create + public-IP-auto-assign paths; the caller is expected to manage public IP out-of-band. - Auto-create path is left untouched for every other code path: callers that pass SubnetId without a NetworkInterfaceId still get a fresh ENI with the existing semantics. Tests cover the four shapes: happy path (auto-create not called, attach called once, instance populated), in-use ENI rejected with the canonical AWS error code, lookup error surfaces verbatim, attach error surfaces verbatim. * feat(eks): K3s control-plane VM launcher with cloud-init user-data LaunchK3sServerVM resolves the eks-server AMI by name, pre-creates the customer-account control-plane ENI in the supplied subnet with the control-plane SG, renders cloud-init user-data (first-boot env, OIDC private key, K3s config.yaml with tls-san + service-account args), then calls RunInstances with NetworkInterfaces[0].NetworkInterfaceId pointing at the pre-created ENI. RunInstances failures roll back the ENI so the caller does not leak a customer-account resource. TerminateK3sServerVM tears the instance and ENI down for DeleteCluster; missing identifiers are a no-op so retries stay safe. ENI + VM both live in the customer account (deviation from plan §7 which specified a SystemAccount-owned VM); cross-account ENI attach is deferred to a follow-up bead. * feat(eks): NATS bootstrap subscriber + cluster reconciler + leader lease NATSBootstrap subscribes to the four single-shot subjects the K3s server VM publishes (k3s-bootstrap-token, k3s-admin-kubeconfig, k3s-oidc-jwks, k3s-ca) on eks.bus.{accountID}.{clusterName}.{kind}. Token and admin kubeconfig are envelope-encrypted via handlers_iam.EncryptSecret and persisted to KV. The published JWKS is cross-checked against the controller-generated keypair by kid and rejected on mismatch. The CA lands on meta.CertificateAuthorityB64 via the new SetClusterCertificateAuthority CAS updater. ClusterReconciler holds a leader lease on spinifex-eks-leader/{accountID}/ {clusterName} (60s TTL, 20s refresh) and drives: - CREATING -> ACTIVE once the four bootstrap KV artifacts are present and NLB /healthz responds 200. - ACTIVE -> probe /healthz on the configured interval, warn on failure. - DELETING / FAILED -> exit (DeleteCluster body owns teardown). Loss of the leader lease (CAS conflict or different holderID) returns ErrReconcilerLeaseLost so the caller can re-acquire from another node. Tested with embedded JetStream + a stub HTTPDoer; race-clean. * feat(eks): OIDC issuer URL builder + reconciler registry ClusterOIDCIssuer renders the {gatewayBaseURL}/oidc/eks/{region}/{accountID}/ {clusterName} URL kube-apiserver advertises as service-account-issuer and that STS / external IRSA validators dereference. Validates URL safety on the path segments so a hostile cluster name cannot inject scheme/path fragments. ReconcilerRegistry tracks per-(accountID, clusterName) reconciler goroutines with idempotent Spawn (drop duplicate spawn calls), Stop (cancel one + wait for release), and StopAll (graceful daemon shutdown). RunClusterReconciler is the production SpawnReconcilerFn: constructs a ClusterReconciler, acquires the leader lease, runs Run in a goroutine. Both helpers consumed by the upcoming service_impl CreateCluster body. * feat(eks): CreateCluster/DescribeCluster/ListClusters/DeleteCluster bodies Wires the EKSServiceImpl orchestration deps (VPC SG/K3s/Subnet, NLB, Instance launcher, AMI resolver, master key, gateway base URL, region, holder ID, reconciler registry) and implements the four lifecycle methods: - CreateCluster: validate → CREATING meta → SGs → NLB → OIDC issuer + keypair → K3s server VM (cloud-init carries OIDC private PEM) → register ENI IP with NLB target group → spawn NATS bootstrap subscriber + per-cluster reconciler via the registry. - DescribeCluster: KV meta read → eks.Cluster shape, with ErrClusterNotFound translated to ResourceNotFoundException at the service boundary. - ListClusters: KV key-prefix scan + sorted maxResults/nextToken pagination. - DeleteCluster: SetClusterStatus DELETING → zeroize OIDC key → stop reconciler via registry → deregister NLB target + delete NLB → terminate K3s VM → delete SGs → delete cluster KV prefix. Daemon wiring: - buildEKSServiceDeps loads MasterKey from BaseDir/config/master.key best-effort, resolves the gateway base URL via the same precedence the LB-agent block applies later, and passes VPCService + ELBv2Service + ImageService directly while adapting InstanceServiceImpl via daemonEKSInstanceLauncher (RunInstances passes through; Terminate- Instances loops vmMgr.Get + StopOrTerminateInstance in-process). - SubnetVPCResolver is satisfied by daemonEKSSubnetResolver wrapping VPCService.GetSubnet. - Startup spawns reconcilers for every persisted CREATING/ACTIVE cluster via SpawnRegisteredReconcilers; shutdown calls eksService.Shutdown so every per-cluster goroutine releases its lease before exit. Test surface: - service_impl_test: TestEKSServiceImpl_ClusterLifecycleShimMode covers the four lifecycle methods when constructed via the back-compat shim (CreateCluster/DeleteCluster → ServerInternal, DescribeCluster → ResourceNotFound on empty bucket, ListClusters → empty list). - daemon_handlers_eks_test: drop the four lifecycle subjects from the NotImplemented routing loop; handler-side coverage now lives in handlers/eks. - awserrors: add ResourceNotFound (404) for the DescribeCluster path. * fix(eks): load master.key from config dir not service data dir eks_deps read master.key from {BaseDir}/config/master.key, but the daemon systemd unit sets SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex (a per-service data dir). admin init writes master.key to the shared config dir, so the daemon looked in a path that only resolved when a {BaseDir}/config symlink existed — awsgw has one, the daemon's nested data dir does not. Result: MasterKey=nil, depsReadyForOrchestration() false, CreateCluster/DeleteCluster collapse to ServerInternal. master.key is a config artifact written next to spinifex.toml. Read it from filepath.Dir(d.configPath), matching how the daemon already resolves predastore.toml and TLS certs. No symlink or provisioning change needed. * fix(eks): register + resolve eks-server AMI via tag, add import --ami-name CreateCluster's K3s launch failed 'eks: AMI eks-server not found' on every clean install. Two defects: 1. spx admin images import had no way to name a locally-built appliance AMI: --name is a catalog lookup (rejects non-catalog values) and the AMI name was hardcoded to ami-{distro}-{version}-{arch}. build-system-image.sh ran 'import --name $AMI_NAME' for eks-server/eks-agent/GPU images, which all failed 'Image name not found in available images'. 2. The launcher resolved DescribeImages by name=='eks-server', a name the pipeline could never produce. Add --ami-name to override the registered AMI name on --file imports (fixes eks-server, eks-agent, and GPU appliance imports). Resolve the EKS AMI by the spinifex:managed-by=eks tag the pipeline already stamps, newest-wins on multiple matches. Tag-based resolution survives the planned server+agent → single EKS AMI unify untouched (role chosen per-instance at launch). build-system-image.sh now passes --ami-name; Makefile gains import-eks-server-image (IMPORT=1) and dev-install.sh imports the AMI when DEV_INSTALL_EKS=1. * fix(eks): give K3s server VM NATS token + CA so bootstrap can publish The K3s control-plane VM was launched with NATSURL=ConnectedUrl() (loopback, unreachable from the VM) and no auth material, while spinifex NATS requires a shared token over TLS. The VM could not connect to NATS, so it never published its four one-shot bootstrap messages and the cluster stayed in CREATING. Hand the VM the daemon's shared NATS token + CA PEM and a VM-reachable NATS URL (mirrors the ELBv2 LB-agent shared-credential pattern; spinifex has no nkeys hierarchy to scope per-principal creds against). - K3sServerInput: drop dead NATSCredsContent; add NATSToken + NATSCACert, required by validateK3sServerInput. user-data writes SPINIFEX_NATS_TOKEN, SPINIFEX_NATS_CA + the CA file; first-boot.env tightened to 0600. - EKSServiceDeps + CreateCluster: pass NATSURL/NATSToken/NATSCACert. - buildEKSServiceDeps: read CA from config.NATS.CACert; extract resolveGatewayHost(); add resolveNATSURL() (same reachable host, NATS port). - k3s-first-boot.sh: build nats CLI args with --token + --tlsca. * fix(eks): retry transient bootstrap persists, fail cluster on permanent error The NATS bootstrap subscriber tore down on the first persist error and the cluster sat in CREATING forever. Core-NATS delivers a publication once, so a transient KV blip during persist was unrecoverable and a malformed payload (bad JWKS kid, non-base64 CA, empty envelope) silently stranded the cluster. - persistWithRetry rides out transient persist failures in-handler (3 attempts, 200ms backoff) instead of unsubscribing and waiting for a re-publish that never comes. - errBootstrapPermanent classifies validation failures (malformed envelope, empty fields, JWKS kid mismatch, non-base64 CA) as non-retryable so they return immediately. - spawnBootstrap marks the cluster FAILED (with reason) when Run exits with a non-context error, so DescribeCluster surfaces the cause and DeleteCluster can reclaim resources. - MarkClusterFailed transitions to FAILED+StatusReason only from CREATING, so a late bootstrap error cannot clobber a concurrent delete or healthy cluster. - casUpdateMeta extracts the revision-checked RMW loop shared by the meta setters. * fix(eks): resume NATS bootstrap for CREATING clusters on daemon restart SpawnRegisteredReconcilers re-spawned only the reconciler on boot, never the bootstrap subscriber. A daemon restart while a cluster was CREATING (K3s VM still booting) lost the subscriptions; the VM published its four one-shot core-NATS messages to nobody and the cluster hung in CREATING forever. - BootstrapPendingKinds reports which artifacts (token, kubeconfig, CA) are still missing from KV. JWKS is excluded: it persists no artifact and re-waiting on a once-published subject would deadlock — its cross-check is best-effort across a restart. - RunForKinds subscribes to a subject subset (nil = all four); Run delegates to it. subjectHandlers maps each kind to its handler and rejects unknown kinds. - SpawnRegisteredReconcilers resumes bootstrap for CREATING clusters, waiting only on the missing artifacts, so a mid-CREATING restart still drives the cluster to ACTIVE. * fix(eks): DeleteCluster must not wipe KV or report success on teardown failure DeleteCluster ran zeroize, NLB teardown, and VM termination as warn-and-continue then unconditionally swept the per-cluster KV prefix and returned success. A failed NLB or VM teardown leaked billable resources with no owning record (the meta holding their ARNs/IDs was erased) and told the operator the cluster was gone. Zeroize — a stated security guarantee of no recoverable key material — had its failure ignored too. - Zeroize, NLB delete, and VM terminate are now blocking: their failures accumulate (errors.Join) and abort the call before the KV sweep, leaving the cluster DELETING and returning the error so a retry can still reach the ARNs. - KV prefix sweep runs only after NLB + VM teardown confirm. - SG cleanup stays best-effort (plan step 5) — a leaked SG strands no owning record — and target deregister stays best-effort since DeleteClusterNLB tears down the whole NLB + target group. * fix(eks): persist NLB + control-plane ARNs mid-CreateCluster so failures don't leak CreateCluster set meta.NLBArn / target-group / instance / ENI in memory and persisted them only in the final PutClusterMeta, after LaunchK3sServerVM. Any failure between EnsureClusterNLB and that persist left the NLB + target group + listener + backing LB VM created but unrecorded; markFailed did no rollback and DeleteCluster keys teardown off the persisted ARNs, so the resources leaked. - Persist meta immediately after EnsureClusterNLB (records NLB/TG ARNs) and again after LaunchK3sServerVM (records instance + ENI), before the next fallible step. - A failed create now leaves a FAILED meta carrying the resource references, so the DeleteCluster teardown path (165.9) reclaims every NLB/TG/listener/VM. - Tests: NLB-arn-persisted-before-later-failure, failed-create-then-delete reclaims NLB, and a CreateCluster happy path. Shared eksServiceFixture extracted from the delete-cluster fixture. * fix(sts): parse path-based EKS OIDC issuer so IRSA accepts 6b cluster tokens ParseEKSIssuerURL required an `oidc.eks.` host prefix and an exactly-2-segment path (AWS hosted-EKS shape), but ClusterOIDCIssuer emits — and awsgw serves — https://{gateway-host}/oidc/eks/{region}/{accountID}/{clusterName}. Every 6b cluster's iss claim was rejected with "issuer host must start with oidc.eks", so AssumeRoleWithWebIdentity would fail all IRSA tokens once 6e serves them. Reconcile on the path-based shape, which is what is actually reachable on bare-metal/on-prem (no oidc.eks.* vhost or DNS exists). Host trust comes from the IAM OIDC-provider registration (the full issuer string is the registry key) and the JWKS signature check, not a host pattern — so the parser drops the host prefix rule and instead pins accountID/clusterName at fixed positions in a strict 5-segment /oidc/eks/{region}/{accountID}/{clusterName} path. A crafted iss with extra or missing segments still fails closed. Adds a cross-component round-trip test asserting a ClusterOIDCIssuer-built iss parses back to the same (accountID, clusterName). * fix(eks): reclaim FAILED clusters on retry + cluster-scoped exists/not-found errors A failed CreateCluster left a FAILED ClusterMeta that poisoned retries: the preflight returned ErrorELBv2TargetGroupInUse ("The target group is currently in use...") for a cluster-exists condition, and because that maps to a retryable code the SDK's auto-retry surfaced ResourceInUseException instead of the original failure. FAILED clusters were also a dead state — visible to neither a clean recreate nor a clear lifecycle. - Add EKS-specific ResourceInUseException / ResourceNotFoundException codes with cluster-scoped messages; CreateCluster's exists check and DescribeCluster / DeleteCluster not-found now use them instead of the ELBv2 / EC2 codes. - CreateCluster reclaims a FAILED cluster: it purges the prior attempt's recorded resources and clears KV before recreating, so a retry (including the SDK auto-retry) is idempotent rather than blocked. Teardown failure surfaces rather than recreating on top of orphans. - Extract purgeClusterInfra, shared by DeleteCluster and the reclaim path (blocking zeroize/NLB/VM, best-effort SG, KV sweep only after teardown confirms). - FAILED clusters remain observable via Describe (status=FAILED) and List — no state where create blocks but describe/list show nothing. * fix(eks): fail CREATING clusters that exceed create timeout A per-cluster reconciler that never observes a ready control plane (bootstrap artifacts missing or healthz unreachable) previously stayed CREATING forever. Add a configurable create timeout (default 15m): once meta.CreatedAt is older than the deadline and the cluster is still not ready, the reconciler marks it FAILED with a reason, exiting the loop. Within the deadline it stays CREATING. * fix(eks): surface ACTIVE cluster health in describe-cluster The reconciler only slog.Warn'd a failing /healthz probe for an ACTIVE cluster, so describe-cluster kept reporting a dead control plane as healthy. Add HealthIssue + LastHealthProbe to ClusterMeta, write them from the ACTIVE reconcile branch (recording the failure reason, clearing on recovery), and map a non-empty HealthIssue to a ClusterHealth ClusterUnreachable issue in describe-cluster. * test(eks): add plan-mandated CreateCluster/DeleteCluster service cases service_impl_test only covered shim mode. Add the four cases the plan's Testing section requires: CONFIG_MAP auth mode rejected (and API accepted), missing subnetIds rejected, DescribeCluster 404 with full deps wired (past the shim short-circuit), and the DeleteCluster zeroize-before-teardown ordering proven by forcing a teardown failure and asserting the OIDC key is already wiped. * refactor(eks): return OIDC private-key PEM from generator, drop re-read CreateCluster discarded GenerateClusterOIDCKeypair's return then did a second KV get + AES-GCM decrypt via loadOIDCPrivateKeyPEM to re-derive the same PEM the generator already had in hand. Return the plaintext PEM (alongside the JWKS) from the generator and use it directly; remove loadOIDCPrivateKeyPEM and the now -unused crypto/x509 + encoding/pem imports in service_impl. * fix(eks): make CreateCluster/DeleteCluster failures diagnosable Unmet orchestration deps and a missing eks-server AMI both collapsed to a bare ServerInternal with no request-time log, leaving the client a 'contact AWS re:Post' message and zero cause. Enumerate the specific missing deps and log them at ERROR, returning ServiceUnavailable (an operator/config fault, not an unrecoverable internal one). Add an ErrEKSServerAMINotFound sentinel so a missing control-plane AMI is logged and surfaced as ServiceUnavailable too. * fix(eks): serialize REST-JSON responses via jsonutil so the AWS SDK can parse them WriteJSONResponse marshaled aws-sdk-go *Output structs with encoding/json, which ignores their locationName tags and emits Go PascalCase keys. EKS clients parse restjson on the lowercase locationName, so every field of list/describe/create/delete responses was silently dropped (empty endpoint, null identity.oidc.issuer). Marshal via jsonutil.BuildJSON — the same marshaler the SDK uses for restjson bodies — which honors locationName. Adds wire-shape tests asserting lowercase keys present and PascalCase absent. * feat(eks): launch K3s server VM via boot-agnostic system-instance path The EKS K3s control-plane VM launched through the plain tenant RunInstances path, so it had only its VPC ENI and no management-bridge NIC -- no route to the daemon's NATS endpoint. It never published its bootstrap subjects and the cluster stayed CREATING. Generalize LaunchSystemInstance to a boot-agnostic contract supporting both system-instance boot styles: - BootDirect: direct-boot microVM (fw_cfg blobs) -- ELBv2 LB agent - BootAMI: AMI boot + cloud-init seed (q35) -- EKS K3s server VM The launch contract moves to a neutral handlers/sysinstance package so EKS can launch system VMs without an eks->elbv2 dependency; handlers/elbv2 re-exports the types as aliases. The BootAMI branch mirrors the daemon's Prepare -> Insert -> Launch split and allocates the mgmt NIC on the VM record between Insert and Launch, so the cloud-init network-config renders a static mgmt0 interface. The EKS NATSURL is now sourced from the mgmt-bridge IP (the on-link address the VM's mgmt NIC can reach), falling back to the gateway-host resolution when no mgmt bridge exists. Verified: K3s VM now boots from the eks-server AMI under q35 with a mgmt0 NIC (10.15.8.x), reachable from the host over br-mgmt, with NATS on-link at the bridge IP. No AMI rebuild required. * fix(eks): make DeleteCluster K3s teardown idempotent so it can't wedge TerminateK3sServerVM treated every DeleteNetworkInterface error -- and every TerminateSystemInstance error -- as blocking. Because instance terminate is async, the immediate ENI delete races it: the first attempt fails InvalidNetworkInterface.InUse, and once the instance-terminate cascade releases the ENI every retry fails InvalidNetworkInterfaceID.NotFound. Both joined into purgeClusterInfra's teardownErrs gate, which short-circuits before the SG sweep and KV prefix delete -- so once the ENI was gone the cluster was stuck DELETING permanently, leaking its control-plane + nodegroup SGs and KV meta. Classify the teardown errors instead of blanket-propagating: - ENI InvalidNetworkInterfaceID.NotFound / InvalidNetworkInterface.NotFound -> idempotent success (the cascade or a prior retry already deleted it); must not block the downstream SG + KV sweep. - ENI InvalidNetworkInterface.InUse (VM still terminating) and any other error -> retryable; surfaced so the cluster stays DELETING and the reconciler retries once the instance releases the ENI. - Instance already gone -> idempotent. TerminateSystemInstance now returns a sysinstance.ErrSystemInstanceNotFound sentinel that the EKS path tolerates via errors.Is, so a retried delete after the VM drained completes. Add awserrors.IsErrorCode(err, code) as the shared delete-error classifier (substring match through %w wrapping) and route the lone ad-hoc strings.Contains guard in daemon/vm_adapters.go through it. Verified: delete-cluster on three clusters left wedged DELETING / FAILED (ENI + VM already gone) all complete -- clusters erased, SGs + ENIs reclaimed, no leak. * fix(eks): gate ACTIVE on verified JWKS cross-check, not pre-seeded key The controller pre-seeds OIDCJWKSKey at create time, so its presence proved nothing about the running cluster's actual signing key. A cluster could reach ACTIVE — and start signing IRSA tokens — before the K3s VM's published JWKS was ever cross-checked against the generated keypair. Add an OIDCJWKSVerifiedKey marker written by persistJWKS only after assertJWKSMatch passes. The reconciler's ACTIVE gate and BootstrapPendingKinds now key off the marker, not the pre-seeded JWKS. assertJWKSMatch is map-based and also rejects kty mismatch. * fix(eks): collapse duplicated gateway-host switch into resolveGatewayHost startCluster reimplemented the 5-case gateway-host precedence inline, duplicating resolveGatewayHost (used by the OIDC issuer and EKS NATS URL). If one drifted, the OIDC issuer host embedded in cluster tokens would diverge from the host LB VMs use to reach the daemon -> IRSA iss mismatch and cluster-wide token rejection. startCluster now calls resolveGatewayHost and derives the multi-node mgmt host route (mgmtRouteVia) from the resolved host. resolveGatewayHost carries the canonical case enumeration; a table test pins all five branches plus the case-1 skip guards and the no-host fallthrough. * fix(eks): make eks-server VM boot diagnosable + fix first-boot publisher crash Three issues found while diagnosing why K3s clusters never leave CREATING: 1. setup.sh: the stock Alpine cloud image lists `console=tty0` LAST in default_kernel_opts, so Linux binds /dev/console to the framebuffer and the host-side serial capture sees only kernel dmesg — OpenRC, cloud-init, and k3s-first-boot output were invisible. Reorder so console=ttyS0 is last. 2. k3s-first-boot.initd: command_background was "no" while output_log/error_log were set; start-stop-daemon rejects --stdout/--stderr without --background ("Failed to start k3s-first-boot"), so the bootstrap publisher never ran and the cluster wedged in CREATING. Background it (the publisher waits up to 5min on K3s /healthz and must not block the runlevel anyway) and route its output to the serial console, which the orchestrator captures (the VM has no login). 3. k3s-first-boot.sh: on the /healthz timeout, dump the tail of /var/log/k3s.log to the console so the apiserver's own failure is visible host-side. * fix(eks): seed K3s apiserver a public OIDC key, not the private key buildK3sUserData pointed both kube-apiserver service-account args at the same private-key file. --service-account-key-file requires a PUBLIC key — k8s ParsePublicKeysPEM rejects a private-key PEM with "data does not contain any valid RSA or ECDSA public keys" and the apiserver crash-loops, so K3s never reaches /healthz, first-boot never publishes, and the cluster wedges in CREATING. Add PublicKeyPEMFromPrivate (PKIX public PEM from the PKCS#8 private key) and seed it to a separate /etc/rancher/k3s/oidc-signing-key.pub.pem. Point service-account-key-file at the public file; keep service-account-signing-key-file on the private file. Runtime-verified: the apiserver crash is gone and K3s now starts and schedules system pods. * feat(eks): egress-only SNAT + DNS for K3s control-plane VM The K3s server VM (a hidden system instance) could not pull system-pod images: it had no internet egress and no working resolver, so coredns/traefik/metrics-server never started and /healthz timed out. Egress: give the single control-plane VM egress-only internet, confined to its /32, with no inbound exposure and no customer-visible resource. - policy/route.go: SystemInstanceEgressPriority (1200, above the drop gate) + AddSystemInstanceEgress/Delete — a reroute scoped to ip4.src == <vmIP>/32 so peers in a shared subnet are unaffected. - policy/nat.go: AddSystemInstanceSNAT/Delete — a plain snat (not dnat_and_snat, so no inbound path), idempotent on its pool external IP. - external/igw.go: L5 EnsureSystemInstanceEgress/Remove resolves the gateway nexthop and attaches the reroute + snat (mirrors the NAT-GW egress path; ADR-0006 S1/S4/S5 preserved). - subscribers: vpc.add/delete-system-egress topics + handlers. - handlers/eks: allocate a hidden (ManagedBy=eks) pool address on CreateCluster and publish add; release + publish delete on teardown. EIP dep wired through EKSServiceDeps + the daemon. DNS: the eks-server AMI is Alpine/dhcpcd and /etc/resolv.conf is a dangling symlink, so DHCP-supplied resolvers are never persisted. Write a real /etc/resolv.conf from the K3s VM cloud-config via bootcmd (not write_files — a write_files entry follows the dead symlink, fails, and aborts the whole block, dropping first-boot.env + the k3s config). * feat: EKS 6b single-node SQLite pivot + bootstrap-envelope fixes Switch the K3s server VM from embedded etcd to embedded SQLite (kine) for the single-node v1, so CreateCluster reaches a healthy control plane. etcd's per-commit fsync stalls past the apiserver deadline on viperblock's deferred-durability block path; SQLite tolerates it. - k3s server: omit cluster-init (selects SQLite/kine), add anonymous-auth=true so the reconciler's anonymous /healthz NLB probe returns 200. - first-boot readiness gate: anonymous /healthz (401 under CIS anonymous-auth =false) -> authenticated 'kubectl get --raw /readyz'. - first-boot publish: emit a JSON BootstrapEnvelope for all four subjects (token, admin-kubeconfig, oidc-jwks, ca); jwks and ca were absent before. - nats pub: --stdin -> --force-stdin (nats CLI 0.4.0). - manifest: add jq (builds the bootstrap-message JSON envelopes). * feat: EKS reconciler probes control-plane mgmt IP until DNS lands The host-side cluster reconciler cannot resolve or route to the VPC-internal NLB DNS endpoint (no authoritative DNS / Eclipso yet), so CreateCluster timed out in CREATING and was marked FAILED even though the control plane was healthy. Probe the control-plane VM's br-mgmt NIC address directly — the one apiserver endpoint reachable from the daemon. Capture the mgmt IP from the launch result (SystemInstanceOutput.MgmtIP -> K3sServerOutput.MgmtIP -> ClusterMeta. ControlPlaneMgmtIP) and build the reconciler health URL as https://<mgmtIP>:6443/healthz. The probe client already sets InsecureSkipVerify, so the mgmt IP need not be in the apiserver cert SAN. Workaround until authoritative DNS + a host-reachable NLB front-end land, at which point spawnReconciler reverts to the NLB endpoint.	2026-06-03 22:29:31 +10:00
Julian Sommer	b2dde0a66d	feat: spx admin images remove (#239)	2026-05-20 16:20:09 +10:00
Josh Siviour	4ae2a4ac53	feat(elbv2): QEMU microvm direct-boot for LB system instances (#228) ... * feat(vm): direct-boot foundations + config/input extensions + image build pipeline vm package (mulga-siv-62): - Add KernelImage, Initrd, KernelCmdline, FwCfg/FwCfgEntry to vm.Config - Emit -kernel/-initrd/-append/-fw_cfg args when fields set - Relax drive invariant: accept kernel image in place of drives - Add devices.go: IsMMIO predicate, NetDevice/BlkDevice/RngDevice helpers - Refactor lifecycle.go + network_helpers.go call sites to use helpers - shutdown.go: os.Remove fw_cfg tmpfiles after QEMU exits - Add devices_test.go, microvm_test.go; extend vm_test.go config/input extensions (mulga-siv-63): - Add MicrovmConfig{ELBv2Enabled, ImagePath} to DaemonConfig - ImagePath falls back to /usr/share/spinifex/microvm when empty - Add commented [microvm] section to spinifex.toml template - Add DirectBoot bool + NICs []NICConfig to SystemInstanceInput image build pipeline (mulga-siv-64): - scripts/build-microvm-image.sh: Alpine chroot build producing vmlinuz + initramfs.cpio.gz - build/microvm/init.sh: fw_cfg-driven PID 1 init (mounts, modprobe, NIC config, lb-agent-env, ca-cert) - build/microvm/inittab: BusyBox init with lb-agent respawn - make build-microvm-image target - setup.sh: install microvm/* to /usr/share/spinifex/microvm/ - .gitignore: exclude built vmlinuz + initramfs.cpio.gz artifacts * feat(daemon): direct-boot launch path for microvm system instances When SystemInstanceInput.DirectBoot=true: - Skip GetAMIConfig and GenerateVolumes - Build vm.Config directly (no buildBaseVMConfig, no PCIe root ports) - MachineType=microvm,x-option-roms=off,pic=off,pit=off,rtc=on,acpi=on,isa-serial=on - Resolve kernel/initrd from Microvm.ImagePath config field - Serialise NICs to three fw_cfg tmpfiles (netcfg, lb-agent-env, ca-cert) - Validate exactly one NICConfig.IsDefault=true before writing netcfg blob - Shutdown path cleans up tmpfiles via existing Config.FwCfg iteration PC-machine path (DirectBoot=false) unchanged. Closes mulga-siv-65 * feat(elbv2): wire microvm.elbv2_enabled flag into ELBv2 service When ELBv2Enabled=true: - Drop lbVMUserData cloud-config; emit KEY=value LBAgentEnv blob instead - Set SystemInstanceInput.DirectBoot=true; omit ImageID - Populate NICs []NICConfig (index 0=primary ENI IsDefault, 1=mgmt, 2+=extra ENIs) - Validate exactly one IsDefault NIC at service layer When ELBv2Enabled=false: existing PC-machine path unchanged. Adds buildLBAgentEnv, buildMicrovmNICs, subnetCIDRForIP, subnetGatewayIP helpers. Tests in arn_test.go cover both flag-on and flag-off paths. Closes mulga-siv-66 * test(daemon,vm): add unit tests for direct-boot helpers to meet diff coverage threshold * feat(daemon): Phase 1 perf instrumentation + e2e runner microvm flag - Log t1→t2 boot timing via slog on the direct-boot path - measureQEMURSSMiB helper reads /proc/<pid>/status (tested) - Gate constants: phase1BootGateMs=500, phase1RSSGateMiB=25 - run-lb-e2e.sh: --microvm flag, dual-path CI annotation, density test note Closes mulga-siv-67 * fix(build): remove busybox-initscripts — not in Alpine 3.20, not needed * fix(microvm): end-to-end fixes for direct-boot LB path Serial output was silent (x-option-roms=off suppresses BIOS serial init); fixed by removing it from the machine string and using -serial chardev:console0 to attach to the machine-created ISA serial device instead of adding a duplicate isa-serial device (which would create ttyS1 while kernel uses ttyS0). virtio-net interfaces were invisible to the kernel because QEMU's auto-kernel-cmdline does not append virtio_mmio.device= params when -append is used. Fixed by generating explicit virtio_mmio.device=0x200@0xfebNNNNN:N params for each NIC in KernelCmdline, and modprobing virtio_mmio + virtio_net in init.sh. sys.micro bumped to 512 MiB — 128 MiB caused kernel OOM during initramfs decompression (~68 MiB uncompressed). BusyBox init ::respawn: entries do not source or export env files; fixed inittab to use set -a; . /etc/conf.d/lb-agent; set +a before exec. wireLBAgentConfig was documented as reading the CA cert but never set elbv2Service.CACert; lb-agent TLS handshakes to AWSGW failed with x509: certificate signed by unknown authority. Fixed by reading d.config.NATS.CACert (same local CA that signs the AWSGW server cert) and assigning to elbv2Service.CACert so writeFwCfgBlobs writes a non-empty cacert blob into fw_cfg. Adds vm_test TestExecute_MicrovmFileChardev and updates lifecycle_test to assert SerialSocket is intentionally empty on direct-boot VMs. * fix(build): auto-detect docker/podman when apk unavailable for microvm image build * fix(build): create /dev/console and /dev/null as proper device nodes in initramfs tar extraction without root silently demotes device files to 0-byte regular files (mknod requires CAP_MKNOD). The kernel opens /dev/console when wiring init's stdio to the serial console; a regular file there causes all init output to be silently discarded to RAM — the VM appears stuck at 'Run /init as init process' with no subsequent console output and lb-agent never heartbeats. Fix: after rootfs population, wipe /dev and recreate console (c 5 1) and null (c 1 3) via sudo mknod. devtmpfs provides all other devices at runtime once init.sh mounts it. * fix(microvm): fix mgmt NIC routing for all LB schemes Two bugs prevented lb-agent from reaching the AWSGW via the mgmt interface: 1. RouteDst/RouteVia swapped — resolveMgmtRoute returns (gateway, target) but buildMicrovmNICs assigned them as (Dst=gateway, Via=target), producing 'ip route add 10.15.8.1 via 192.168.1.33' instead of the correct 'ip route add 192.168.1.33 via 10.15.8.1'. 2. NIC[1] MAC and CIDR never filled — the daemon allocates mgmt IP/MAC after buildMicrovmNICs returns, but didn't inject them into input.NICs[1] before calling writeFwCfgBlobs, leaving NIC1_MAC= empty in the netcfg blob and skipping interface setup in init.sh entirely. Also extends mgmt routing to internet-facing scheme in the microvm path: VPC SNAT is not reliable across all VPC topologies for control-plane heartbeats; mgmt is always a direct L2 path to the host. * feat(microvm): Phase 2 kernel stripping + boot optimisation - cpu: switch to host,-pmu (disable PMU — not needed in microvm guest) - machine type: make isa-serial configurable via microvm.isa_serial config field; default true (on) preserves console log path - build: strip unused kernel modules (USB, IDE, ATA, GPU, media, input, HID, bluetooth, staging, sound, wireless) from initramfs before cpio - build: add Phase 2 artifact size gate — fail if vmlinuz + initramfs.cpio.gz exceeds MICROVM_ARTIFACT_MAX_MiB (default 50 MiB); override via env var during development - perf: add phase2BootGateMs=300 and phase2ArtifactGateMiB=50 constants with canary tests to catch unintentional gate changes - tests: cover buildMicrovmNICs nil-VPC and extra-ENI paths - tests: cover microvmMachineType nil/true/false branches * fix(build): use find-based virtio+fw_cfg module keep-list in initramfs strip The directory-only keep-list (drivers/virtio, drivers/firmware) missed virtio_net.ko which lives in drivers/net/, virtio_blk in drivers/block/, and virtio-rng in drivers/char/hw_random/. Switch to find-based approach: collect all virtio*.ko* and *fw_cfg*.ko* regardless of subdirectory, then restore them into their original relative paths under kernel/. Artifacts now: 42 MiB (vmlinuz 11 MiB + initramfs 31 MiB), gate passes. * fix(build): keep failover modules in initramfs strip virtio_net depends on net_failover/failover for live-migration support. Stripping them left virtio_net with unresolved symbols (net_failover_create/destroy), so the module failed to load and the guest came up with only lo. init.sh then skipped every NIC ("no interface found for MAC ..."), leaving lb-agent with no network path to AWSGW and the LB stuck in provisioning. Add *failover*.ko* to the keep-list. Image size unchanged at 42 MiB (under the 50 MiB Phase 2 gate). ELBv2 e2e now reaches active and 2/2 targets healthy for both ALB and NLB internal LBs. * fix(daemon): restore slog default after viperblock.New mutates it viperblock.New unconditionally calls vb.SetDebug, which calls slog.SetDefault with a TextHandler at LevelError — globally silencing every slog.Info and slog.Warn in the caller process for the rest of its life. Discovered while validating microvm Phase 2 (mulga-siv-68): the daemon stopped emitting Info logs the moment the first customer VM with volumes launched, hiding direct-boot timing, LaunchSystemInstance completed, and CreateLoadBalancer completed lines that we need for boot-time tuning. Workaround in newViperblock: re-install an Info-level TextHandler on os.Stderr after each viperblock.New call. Also drop the three explicit vb.SetDebug(false) calls in prepareRootVolume / prepareEFIVolume / prepareCloudInitVolume — they were no-ops (viperblock.New already runs SetDebug internally) and each one re-triggered the global mutation we are trying to undo. Proper fix is in viperblock itself — tracked as mulga-siv-70: SetDebug should not call slog.SetDefault. Until then, this restore call is the shortest path to getting the daemon's observability back so we can measure t1_to_t2_ms and tune Phase 2. Preflight note: golangci-lint not available on this dev host, so the project preflight target cannot run end-to-end. Verified with go vet ./spinifex/handlers/ec2/instance/... (clean) and go test ./spinifex/handlers/ec2/instance/... (pass). * perf(microvm): hit Phase 2 boot/memory gates t1→t2 hypervisor: 3224 ms → 264 ms (12×). 128 MiB guest now boots cleanly without OOM, reverting the temporary 512 MiB bump. Root causes: 1. build-microvm-image.sh staged the entire chroot into the cpio without dropping /boot, embedding a duplicate vmlinuz, Alpine's stock initramfs-virt, System.map, and a "boot → ." symlink loop (~25 MiB). Decompression peak exceeded 128 MiB. Strip /boot after the kernel is copied out, before the cpio. Initramfs drops from 31 MiB → 11 MiB (53 MiB → 28 MiB uncompressed); total artifact 42 MiB → 22 MiB. 2. vm/lifecycle.go burned 3 s of fixed sleep on every direct-boot launch: 2 s nbdkit pre-exec wait (no nbdkit on direct-boot) and 1 s post-fork settle (QEMU binds QMP within ~10 ms when there is no firmware, ROM, or block layer). Both extracted to directBoot-aware helpers — nbdkitPreExecWait collapses to 0, and qemuStartSettleWait to 50 ms. 3. KernelCmdline only carried `console=ttyS0 virtio_mmio.device=…`. Added boot-time perf flags: quiet loglevel=3 (silence printk over 115200-baud isa-serial), mitigations=off (KVM guest on trusted host, single-tenant LB workload), tsc=reliable, no_timer_check, reboot=t, i8042.{nopnp,noaux,nokbd} (microvm has no PS/2). 4. init.sh dropped modprobe virtio_mmio (built-in: CONFIG_VIRTIO_MMIO=y) and tightened polling: fw_cfg/iface waits go from 10 × 200 ms (up to 2 s) to 5 × 20 ms (up to 100 ms). Phase 2 gates: - t1→t2 ≤ 300 ms ✅ 264 ms / 281 ms - artifact ≤ 50 MiB ✅ 22 MiB (gate already in build script) - ELBv2 e2e green ✅ 32/33 (pre-existing client-VM failure, unrelated to microvm path) * build(microvm): wire build into deploy + distro + release pipeline Adds install-microvm Makefile target (with grouped-target staleness detection on vmlinuz/initramfs.cpio.gz), wires it into dev-install.sh and reset-dev-env.sh, makes 'make deploy' reinstall artifacts when present, and makes 'make distro-amd64' stage build/microvm/* into the release tarball staging dir. arm64 distro stays unchanged (no aarch64 microvm artifact yet — Phase 4). Release workflow gains a setup-go step and a 'make build-microvm-image' step before 'make distro-amd64' on the amd64 runner so the resulting tarball includes the kernel + initramfs without a manual copy. Precursor for the Phase 3 PC-machine LB removal (mulga-siv-69). Closes mulga-siv-71. * build(ansible): include microvm artifacts in build role tarball Adds a build-microvm-image task to the ansible build role and stages build/microvm/* into the role's release tarball so the install role's setup.sh run picks them up via INSTALL_SPINIFEX_TARBALL. Skipped gracefully when the build script is absent (older branches without direct-boot LB wiring). * feat(config): enable microvm direct-boot LB path by default Uncomments the [nodes.<node>.daemon.microvm] section in the generated spinifex.toml template and sets elbv2_enabled = true so all new clusters bring up ELBv2 LBs via the direct-boot microvm path. Operators can opt back to the legacy PC-machine path by flipping the flag to false in the rendered config. Required for end-to-end CI on this branch to exercise the direct-boot path. Existing clusters' configs are unchanged (the template only affects fresh 'spx admin init' renders). * fix(microvm): bootstrap apk.static when no host rootfs tool exists Self-hosted CI runners and the GitHub-hosted ubuntu-24.04 image often have none of apk, docker, or podman installed. apk-tools is not in the Debian 13 (trixie) repositories. Without one of these the rootfs build fails up front with 'no rootfs build tool available'. When all three are absent, fetch apk-tools-static from the configured Alpine mirror, extract it to a tempdir, and prepend it to PATH so the existing native-apk strategy below picks it up transparently. Requires only curl + tar, both of which are present on every CI runner that can already check out source. No-op when apk, docker, or podman is already present. Unblocks https://github.com/mulgadc/spinifex/actions/runs/25988830244 * revert: drop apk.static bootstrap in build-microvm-image Reverts 0bd76b7. Self-hosted runners will get docker.io installed out of band; existing Strategy 2 (docker run alpine:3.20) handles the rootfs build without host sudo or unsigned package install. * docs(workbook): drop lb-alpine import for microvm direct-boot LB LBs now launch via direct-boot QEMU microvm using kernel + initramfs bundled at /usr/share/spinifex/microvm/, so the legacy lb-alpine system AMI import is no longer required. * fix(daemon): own tap devices by daemon user so QEMU can attach without CAP_NET_ADMIN When the daemon runs non-root (production: User=spinifex-daemon), child QEMU processes inherit that uid but lack CAP_NET_ADMIN. The kernel's tun attach check then requires the tap owner uid to match the caller's euid. Without the `user` flag, `ip tuntap add` defaults to root:root and qemu's TUNSETIFF fails with EPERM ("could not configure /dev/net/tun: Operation not permitted"). Surfaced by the e2e-nightly cell-17 microvm LB launch on the ci-pseudo runner where the kernel enforces strictly; legacy EC2 launches on ci-single runners happened to attach successfully despite the same code path. Adding `user $current group $gid` to `ip tuntap add` ensures the tap is owned by whatever uid the daemon (and its qemu children) run as. Root daemons skip the flag — they have CAP_NET_ADMIN ambient. * fix(e2e): redirect lb probe chatter to stderr so stalls are visible test_http_from / test_tcp_from echoed progress to stdout, which the callers captured into *_OK vars via $(...). Result: 6+ minute retry loops ran silently in CI, and the "$ok" comparison never matched "true" because the captured value contained the full multi-line chatter. Reachability pass/fail counters were also lost in the subshell. Redirect helper output to stderr, leave only the true/false flag on stdout, and move pass/fail to the caller so they actually count. * fix(microvm): don't /32 AdvertiseIP via mgmt for internet-facing single-node buildMicrovmNICs overrode resolveMgmtRoute's deliberate empty return for the single-node internet-facing case by falling back to (MgmtBridgeIP, AdvertiseIP). That installed a /32 route to AdvertiseIP via the mgmt NIC inside the LB microvm. On single-node where the AWSGW host IP equals the WAN client IP, the /32 also catches reply traffic for inbound HTTP/TCP to the LB's floating IP — replies leave via the mgmt NIC with src=data-NIC IP, the host kernel drops the asymmetric flow, and same-chassis ingress hangs. resolveMgmtRoute already encodes the correct policy: - multi-node any scheme → explicit MgmtRoute{Gateway,Target} - single-node internal → fallback to (MgmtBridgeIP, AdvertiseIP) - single-node internet-facing → empty (reach AWSGW via VPC SNAT) Honor it. Matches the legacy lbVMUserData path, whose own comment flagged the stolen-WAN-return-path symptom this introduced. Test covers all three branches. * feat(ansible): dev-deploy playbook for in-place artifact swap Fills the gap between ansible-dev-install (skips when already installed) and ansible-dev-reset (full teardown + reinstall + smoketest). Mirrors Makefile `deploy` target: rebuild + replace binaries and microvm artifacts on a running node, restart spinifex.target, wait for awsgw. Skips setup.sh (no user/dir/systemd-unit changes) and smoketest so iteration is fast. Use ansible-dev-reset when changing systemd units, helper scripts, logrotate, or setup.sh itself. Adds: - roles/deploy/tasks/main.yml — install spx + nbdkit plugin + microvm artifacts, with plugindir auto-detection via `nbdkit --dump-config`. - playbooks/dev-deploy.yml — chains build + deploy roles. - Makefile ansible-dev-deploy target. - README playbook table + "when to use which" guidance. * chore(microvm): remove legacy LB Alpine image build pipeline Phase 3 microvm removal — strip the PC-machine LB image build artifacts that direct-boot no longer needs. The microvm initramfs already carries lb-agent + HAProxy via build-microvm-image, so the lb.conf manifest, lb-setup.sh chroot script, build-lb-image target, dev-install / reset-dev-env import steps, and ansible LB-image tasks all become dead. build-system-image stays — other manifests (e.g. GPU images) still use it. * refactor(microvm): remove legacy PC-machine LB boot path Phase 3 — direct-boot becomes the only ELBv2 LB launch path. - Drop SystemInstanceInput.DirectBoot / ImageID / UserData (every LB caller set DirectBoot=true; legacy cloud-init userdata is gone). - Drop ELBv2ServiceImpl.lbVMUserData, systemAMI* fields, SetSystemAMIFunc, getSystemAMI, and the lazy-AMI wiring in daemon.startCluster. - Collapse CreateLoadBalancer to a single direct-boot launch input. - Collapse daemon.LaunchSystemInstance: drop AMI validation, ImageID on RunInstancesInput, UserData base64 encode, GenerateVolumes branch. buildDirectBootConfig hardcodes /usr/share/spinifex/microvm and isa-serial=on (microvmMachineType no longer takes a flag). - Drop MicrovmConfig from config.DaemonConfig and the [daemon.microvm] section from the spinifex.toml template. - Prune dead tests (lbVMUserData_*, PCMachine_Unchanged, AMI resolver paths) and refresh stale comments that still mentioned lbVMUserData. ARM hosts have no LB launch path until Phase 4 adds direct-boot on virt — CreateLoadBalancer on aarch64 fails naturally at QEMU exec. * docs: drop lb-alpine catalog entry from cluster setup guide LB system instances no longer use an AMI — direct-boot microvm artifacts ship in the spinifex package. * feat(ui): remove LB image import banner + LB-AMI gating Direct-boot microvm artifacts ship in the spinifex package, so the LB system AMI no longer exists — the "Load balancer image not imported" banner and the Create-button disable on the LB list / wizard pages can never fire. Strip: - lb-image-missing-banner.tsx component - hasLbImage / isLbImage / LB_IMAGE_NAME / LB_MANAGED_BY_VALUE from system-managed.ts (isSystemManagedImage stays — still used by the EC2 Images pages to hide system AMIs) - Banner gating + ec2ImagesQueryOptions imports from the LB list and wizard pages - The corresponding tests that asserted the banner appears * revert: restore lb.conf + lb-setup.sh for mulga install pipeline Phase 3 microvm removal (commit 6313d74) deleted these too early — the mulga install/iso artifact builders (scripts/tofu-cluster/build-install -artifacts.sh, scripts/tofu-iso/build-iso-artifacts.sh) still invoke scripts/build-system-image.sh scripts/images/lb.conf, breaking e2e with "ERROR: Manifest not found: scripts/images/lb.conf". Restore both files verbatim so the legacy pipeline keeps building until the mulga-side migration to microvm-only artifacts lands post-merge.	2026-05-18 15:48:04 +10:00
tomnewton-mulga	28ca689130	feat: VFIO GPU passthrough (#188) ... * feat: add GPU discovery package (mulga-992) Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group member lookup, and a PCI device ID model table. Vendor-specific enrichment via nvidia-smi is attempted opportunistically and falls back to the built-in table when unavailable. All discovery and IOMMU logic is covered by unit tests using mock sysfs trees. * feat: add VFIO bind/unbind and GPU pool manager (mulga-993) Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device driver switching, and gpu.Manager which owns the claim/release lifecycle for the GPU pool including full IOMMU group binding, rollback on partial failure, and marking entries unavailable when rebind fails after VM termination. * feat: add G5 instance type generator with GPU model support (mulga-994) - Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup - Add generateGPUTypes to emit g5.* types with GpuInfo populated - Add IsGPUType helper for consumers filtering GPU-capable types - Update DetectAndGenerate to accept gpuModels []GPUModel parameter * feat: integrate GPU passthrough into daemon lifecycle (mulga-995) - Add gpu_passthrough config flag; discover GPUs and build manager at startup - Gate g5 instance types on GPU pool availability via canAllocateCount - Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device - Release GPU (VFIO unbind, rebind to host driver) in stopInstance - Re-claim GPU on daemon restart when QEMU process is still alive - Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS) - Fix Claim to use stored OriginalDriver when device is already vfio-pci bound * feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997) Extends build-system-image.sh to support Ubuntu alongside Alpine. Key differences handled: partition table (parted resizepart p1), bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs rc-update. DISTRO defaults to "alpine" for backward compatibility. Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script that installs nvidia-headless-550-server with DKMS pre-built against a pinned linux-image-generic kernel, blacklists nouveau, and prevents unattended-upgrades from invalidating the pre-built module. * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * feat: spx admin gpu status/enable/disable commands (mulga-1000) - admin.SetGPUPassthrough: Go implementation of the idempotent TOML write (replaces the inline Python heredoc in spx-test-gpu.sh) - spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough activation, and advertised g5 instance types via NATS NodeStatusResponse - spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP, polls until daemon confirms GPUPassthrough=true - spx admin gpu disable: refuses while GPU instances are running, otherwise writes TOML, sends SIGHUP, polls until confirmed - spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable * fixed admin * fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter the passthrough pool. enrichFromLspci() uses the system PCI ID database via lspci -vmm to populate the model name for any card not in the hardcoded table, without needing nvidia-smi or driver tools. * fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of file permissions. QEMU subprocesses also need char-vfio rw access to open /dev/vfio/N for the passed-through device. * fixing gpu bining permissions * working state of gpu passthrough on phyiscal box * installer grub image fixes * made gpu-passthrough more user facing * fixed banner * fixed banner * banner fix * gpu passthrough docs * preflight tweak * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * fixes after running /review-branch * feat: re-wire GPU claim and device injection after vm.Manager refactor GPU passthrough claim (VFIO bind) and QEMU device injection were lost when dev's vm.Manager refactor (#185/#186) moved StartInstance/ CreateQMPClient into vm/lifecycle.go. Re-add both in their correct new homes: - daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both RunInstances and StartStoppedInstance handlers; release on failure - vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag derived from instance.GPUXVGAEnabled when GPUPCIAddress is set * fix: Create spinifex user ~/.ssh and keys in firstboot --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>	2026-05-06 09:52:23 +10:00
Ben Duncan	36425a48eb	refactor: rewrite reset-dev-env.sh against production layout ... Reset now targets /etc/spinifex, /var/lib/spinifex, /var/log/spinifex, /run/spinifex (the privilege-separated layout installed by setup.sh) and drives re-install from a source build via dev-install.sh instead of the legacy in-tree $HOME/spinifex + start-dev.sh flow. Capture region, AZ, external_mode, pool range, gateway, and prefix from the existing spinifex.toml before the wipe so the reset restores the current topology rather than applying defaults. Fresh boxes fall back to ap-southeast-2 and a pool derived from the WAN subnet. Drop the macvlan branch entirely — bridge-only from here on. Operators on single-NIC hosts must run reset from the console (documented in the script header). dev-install.sh gains DEV_INSTALL_SKIP_INIT so reset can own the post- setup.sh steps (setup-ovn, spx admin init with preserved args, CA trust, service start, LB image import). Also honors SPINIFEX_REGION / SPINIFEX_AZ for non-reset callers, and scopes systemctl reset-failed to spinifex-* so unrelated failed units aren't silently cleared. Closes mulga-983.	2026-04-20 11:38:10 +10:00
Julian Sommer	1ad03bb518	feat(gateway): per-IP auth failure rate limiting (#138)	2026-04-15 09:56:04 +10:00
Julian Sommer	0547bdd1fe	feat: loadbalancer frontend (#136)	2026-04-14 09:24:14 +10:00
Josh Siviour	ab95566795	feat: initial ELBv2 implementation (#122) ... * feat: add ALB target health reporting via HAProxy stats Agent-side: periodically query HAProxy stats socket and publish target health status to NATS (elbv2.alb.<id>.health topic). Control-plane: healthChecker subscribes to health reports from all ALB agents and updates target health state in the KV store. Wire into ELBv2ServiceImpl lifecycle and DeregisterTargets cleanup. E2E: remove health check skip — targets must now become healthy. * feat: allocate public IP for internet-facing ALB VMs in daemon Add Scheme field to SystemInstanceInput so the daemon knows whether an ALB is internet-facing or internal. For internet-facing, allocate a public IP via externalIPAM and publish vpc.add-nat for the OVN dnat_and_snat rule. Internal ALBs skip public IP assignment entirely. * fix: release auto-assigned public IPs on DeleteNetworkInterface DeleteNetworkInterface previously leaked public IPs and NAT rules. Now checks if the ENI has a public IP, verifies it's not EIP-owned, and releases via externalIPAM + publishes vpc.delete-nat. Daemon injects externalIPAM and EIP KV into VPC service at init time. Tests cover: auto-assigned IP release with NAT event, EIP-owned IP skipped, and no-op when externalIPAM is not configured. * feat: enforce multi-AZ for internet-facing ALBs in pool mode (mulga-770) In CreateLoadBalancer, when DevNetworking is false (pool mode), require 2+ subnets in different AZs for internet-facing scheme. Dev mode and internal scheme skip the validation. * feat: ALB scheme unit tests + DNS internal prefix + public IP in AZ response (mulga-771) Add 9 unit tests covering scheme-dependent ALB behavior: - DNS name uses internal- prefix for internal scheme (AWS-compatible) - Internet-facing ALBs include public IP in AvailabilityZones response - Internal ALBs have no public IP in AZ response - Launcher receives correct scheme, delete cleans up VM + ENIs Also implements the code changes these tests exercise: - AvailZoneInfo.PublicIP field populated from launcher output - lbRecordToSDK populates LoadBalancerAddresses for internet-facing * feat: add custom AMI for ALB to images list * feat: add internet-facing ALB data plane E2E test (mulga-772) New E2E script that validates internet-facing ALBs get public IPs and are reachable from a peer cluster node (external validation). Skips in dev mode. * fix: set ENDPOINT to runner node IP in ALB E2E workflow * fix: correct paths filter to reference alb-e2e.yml * fix: alb ami name and workflow step to import the image * fix: spx path * fix: remove ALB AMI import from workflow and added to bootstrap.sh * fix: use two subnets in different AZs for internet-facing ALB E2E Multi-AZ enforcement (mulga-770) requires internet-facing ALBs to have subnets in 2+ AZs in pool mode. Discover AZs and create a subnet in each. * refactor: remove multi-AZ enforcement for internet-facing ALBs Team decision: ALBs must work with single-AZ and single-node clusters. Removes the multi-AZ validation, pool mode helper, associated tests, and error constant. E2E script reverted to single subnet. * fix: use correct account ID for ALB ENI public IP assignment The ENI is created under the caller's account but LaunchSystemInstance was looking it up with GlobalAccountID (system account), causing the UpdateENIPublicIP call to fail with "ENI not found". Pass the caller's account ID via SystemInstanceInput.AccountID. * fix: add debug logging for ALB public IP assignment failure * fix: expand debug logging to include peer node and more keywords * fix: lazy system AMI discovery for ALB VMs The daemon was caching the system AMI at startup, but the ALB image is imported after services start (bootstrap step 9). Replace SetSystemAMI with SetSystemAMIFunc so the AMI is resolved at request time when CreateLoadBalancer actually needs it. * fix: resolve ALB VM instance type dynamically from resource manager The instance type was hardcoded to t3.nano which may not exist on all clusters. Now the daemon provides a resolver that picks the smallest available instance type from the resource manager. * push interim changes * refactor: rewrite ALB agent from NATS client to HTTP server Replace NATS subscriptions with an HTTP server on :8405 exposing POST /config, GET /ping, and GET /health. The daemon will push config and poll health over HTTP through the VPC network, removing the agent's direct NATS dependency. - agent.go: net/http server replaces nats.Conn/Subscriptions - health.go: remove push-based reportHealth; /health handler calls queryHAProxyStats synchronously on each request - cmd/alb-agent: --nats flag replaced with --listen - tests: rewritten with httptest instead of NATS test server - health_checker_test: inline topic strings (HealthTopic removed) * refactor: replace NATS with HTTP client for daemon-to-ALB agent comms Daemon now pushes config via POST and polls ping via GET to the ALB agent's HTTP server on the VPC IP, eliminating the agent's NATS dependency from the daemon side. - service_impl: pushConfig uses http.Post, waitForAgentReady uses http.Get, both target http://<vpcIP>:8405 - albVMUserData: drop natsURL param (agent no longer needs NATS) - types: add VPCIP field to LoadBalancerRecord - daemon: remove SetNATSURL call and 10.0.2.2 translation block - arn_test: update user data tests for new signature * refactor: rewrite ALB health checker from NATS subscription to HTTP polling Health checker now polls each active ALB's GET /health endpoint on a 10s ticker instead of subscribing to a NATS wildcard topic. Removes the NATS connection dependency from the health checker. - health_checker: replace nc/sub with httpClient + stopCh polling loop - newHealthChecker: drop nc param, add agentURLFn for testability - service_impl: update newHealthChecker call site - tests: rewrite with httptest, test pollAll/pollOne paths * refactor: remove NATS wiring from ALB image and make hostfwd dev-only - alb-setup.sh: remove --nats flag, use --listen=:8405; remove placeholder HAProxy frontend on 8405 (agent owns the port now); update description and comments - service_impl: HostfwdPorts (80/443) now only set when DevNetworking is enabled — production ALBs are reached via VPC IP directly * feat: system credentials file + daemon wiring for ALB agent SigV4 auth Write config/system-credentials.json during admin init/join with the system access key. Daemon reads it at startup, resolves the gateway URL from external IPAM (or 10.0.2.2 in dev mode), and wires both into the ELBv2 service. albVMUserData now injects ALB_GATEWAY_URL, ALB_ACCESS_KEY, and ALB_SECRET_KEY into cloud-init for ALB VMs. * feat: ALB agent gateway actions + daemon handlers (heartbeat, config) Add ALBAgentHeartbeat and GetALBConfig actions to the gateway so the ALB agent can communicate via the AWS gateway instead of direct HTTP. - Add agent_types.go with input/output types for both actions - Add ALBAgentHeartbeat: processes heartbeat, transitions provisioning→active on first contact, updates LastHeartbeat, processes health report - Add GetALBConfig: returns pre-computed HAProxy config text and hash - Replace pushConfig (HTTP POST to agent) with updateStoredConfig (stores config + SHA256 hash on LB record for agent pull) - Remove waitForAgentReady polling goroutine (heartbeat handles transition) - Remove HTTP client from ELBv2ServiceImpl (no longer needed) - Strip health checker HTTP polling (pollLoop/pollAll/pollOne) — health reports now arrive directly via ALBAgentHeartbeat - Add ConfigText, ConfigHash, LastHeartbeat fields to LoadBalancerRecord - Wire NATS subscriptions + daemon handlers for both new actions - Add gateway handler files and elbv2Actions map entries - Comprehensive tests for heartbeat, config, and updateStoredConfig * feat: rewrite ALB agent as AWS SDK HTTP client (Bead 3r) Replace the HTTP server agent with a poll-based client that communicates with the gateway using SigV4-signed requests. The agent now sends ALBAgentHeartbeat every 5s with health reports and fetches config via GetALBConfig when the config hash changes. * feat: add VPC service route for ALB agent gateway access ALB VMs need to reach the gateway for heartbeat/config polling, but VPCs without an IGW have no external routing. Instead of requiring an IGW, the daemon now publishes a vpc.ensure-service-route event before launching ALB VMs. vpcd handles this by creating minimal external connectivity (external switch, localnet port, gateway router port, chassis bindings) if not already present, then adds a /32 static route for the gateway IP. Refactors handleIGWAttach to share the external connectivity setup via ensureExternalConnectivity helper. * Revert "feat: add VPC service route for ALB agent gateway access" This reverts commit d4220d8806. * feat: add management NIC infrastructure for system instance control plane Add config, IP allocator, TAP helpers, and VM struct fields for the management NIC that gives system instances (ALBs etc.) L3 access to the host control plane via br-mgmt, independent of VPC routing. - DaemonConfig.MgmtBridge (default "br-mgmt") - MgmtIPAllocator: thread-safe static IP allocation (.10-.249), rebuild from VMs on restart - generateMgmtMAC (02:a0:00 prefix), MgmtTapName, GetBridgeIPv4 - SetupMgmtTapDevice / CleanupMgmtTapDevice (plain Linux bridge TAP) - VM struct: MgmtMAC, MgmtIP, MgmtTap fields * feat: wire management NIC into QEMU launch and cloud-init config Add QEMU NIC injection for mgmt TAP (virtio-net-pci on br-mgmt), extend generateNetworkConfig to emit mgmt0 stanza with static IP, and wire mgmt MAC/IP allocation into LaunchSystemInstance. Includes mgmt TAP cleanup on instance stop and failed launch cleanup. * test: add coverage for ELBv2 service lifecycle and setter methods * fix: move br-mgmt detection before wireALBAgentConfig so gateway URL uses management bridge IP * fix: generate per-interface cloud-init config when eniMAC present without devMAC The guard in generateNetworkConfig required both eniMAC and devMAC to produce per-interface config, falling back to wildcard DHCP otherwise. System instances (ALBs) have eniMAC + mgmtMAC but no devMAC (DevNetworking is independent), so the mgmt NIC stanza was never emitted. The wildcard config does DHCP on all NICs, but br-mgmt has no DHCP server, leaving the management NIC unconfigured and the agent unable to reach the gateway. * fix: add wildcard support to ENI description filter + add ca-certificates to ALB image - ENI DescribeNetworkInterfaces description filter did exact string match, but AWS supports * wildcards. E2E test uses "ELB *" which didn't match "ELB app/my-alb/lb-123". Added matchFilter helper for wildcard matching. - ALB image was missing ca-certificates package, so cloud-init ca_certs module couldn't add the Spinifex CA to the system trust store. Agent TLS handshakes failed with "certificate signed by unknown authority". - Fixed data race in TestDeleteLoadBalancer_TerminatesVM_WithPublicIP where async TerminateSystemInstance goroutine wrote to mock fields while the test read them. * fix: detach ENI before delete on termination + defer ENI cleanup to stopInstance DeleteLoadBalancer was trying to delete ENIs synchronously, but they were still attached (in-use) to the running VM. The delete was rejected with InvalidNetworkInterface.InUse. Fix: DeleteLoadBalancer only deletes ENIs when no VM was launched (InstanceID empty). When a VM exists, ENI cleanup is deferred to stopInstance which runs after QEMU exits. stopInstance now calls DetachENI before DeleteNetworkInterface to clear the in-use status. E2E test now polls for ENI cleanup (up to 30s) since VM termination is async. * fix: detach then delete ENIs in DeleteLoadBalancer ENIs are attached (in-use) to the ALB VM, so DeleteNetworkInterface rejects them. Fix: call DetachENI first to clear in-use status, then delete. stopInstance tolerates already-deleted ENIs. * test: add internal ALB dataplane E2E test Add run-elbv2-dataplane-internal-e2e.sh replacing the generic dataplane test. Internet-facing coverage is already provided by run-elbv2-dataplane-internet-facing-e2e.sh. The internal test validates: - ALB reaches active via mgmt NIC heartbeat (no IGW dependency) - Scheme=internal, no public IP, internal- DNS prefix - Health checks pass via VPC L2 (OVN logical switch) - Round-robin traffic from a VPC client to the ALB's private IP - ENI cleanup on ALB deletion Uses a client VM inside the VPC that curls the ALB private IP and serves results via HTTP on its public IP for host-side verification. * feat: add NLB constants and data model changes (mulga-838) * feat: type-aware validation in CreateLoadBalancer (mulga-839) Accept Type=network for NLB, reject SecurityGroups for NLB, set CrossZoneEnabled defaults (false for NLB, true for ALB), and use /net/ ARN path segment for network load balancers. * feat: type-aware validation in CreateTargetGroup (mulga-840) Accept TCP/UDP/TLS/TCP_UDP protocols and apply NLB health check defaults (TCP protocol, no path/matcher, 10s timeout, 3/3 thresholds). Reject invalid protocols with InvalidParameterValue. * feat: type-aware validation in CreateListener (mulga-841) * feat: add NLB listener and protocol compatibility unit tests Cover the remaining Phase 1 test gaps: NLB CreateListener with all L4 protocols, protocol rejection for wrong LB type, listener-to-target-group protocol compatibility validation, and default protocol rejection on NLB. * feat: add TCP HAProxy config generation for NLB Add TCP mode template and generateNLBConfig function for Network Load Balancer support. GenerateHAProxyConfig now branches on LB type, producing mode-tcp configs for NLBs with tcp-check or httpchk health checks depending on the target group's health check protocol. * refactor: rename albagent to lbagent for NLB generalization Rename the ALB-specific agent package and binary to generic LB names in preparation for NLB support. Both ALB and NLB VMs use the same agent — it just applies whatever HAProxy config the control plane generates. - Rename albagent/ → lbagent/, cmd/alb-agent/ → cmd/lb-agent/ - Rename types: ALBAgentHeartbeat* → LBAgentHeartbeat*, GetALBConfig* → GetLBConfig* - Rename NATS topics: elbv2.ALBAgentHeartbeat → elbv2.LBAgentHeartbeat - Rename cloud-init user data: albVMUserData → lbVMUserData with LB_* env vars - Rename Makefile targets: build-alb-agent → build-lb-agent - Update gateway action map, daemon subscriptions, service interface - Fix pre-existing unused matchFilter function lint error - Fix awserrors test count for InvalidConfigurationRequest * feat: add sys.micro system instance type for LB VMs (mulga-845) Add sys.micro (1 vCPU, 128MB) as an internal-only instance type for system VMs (load balancers, NAT gateways). Registered in the type map for allocation but excluded from DescribeInstanceTypes, NATS subscriptions, and resource stats. Daemon SetSystemInstanceTypeFunc now returns sys.micro instead of scanning for smallest guest type. * feat: LB image shrink and rename (mulga-846) Rename alb.conf → lb.conf, alb-setup.sh → lb-setup.sh with IMAGE_SIZE reduced from 512M to 192M. Update all references to use lb-agent binary and LB_* env vars. Update COMMANDS.md for NLB support. * chore: rename file * feat: NLB E2E test and bug fixes (mulga-847) Add full NLB data plane E2E test (33 assertions, 9 phases): VPC/subnet setup, TCP target group with health checks, NLB creation (type=network), TCP listener, agent heartbeat, target health verification, TCP traffic round-robin, target deregistration, and full cleanup verification. Bug fixes found during E2E testing: - Fix HAProxy stats socket path mismatch: config used alb-/nlb- prefix but agent expects lb- prefix, preventing health reports - Fix system AMI lookup to match new -lb- image name, falling back to legacy -alb- name - Bump LB image size from 192M to 256M (Alpine base is 200M) * test: validate NLB internal scheme (mulga-857) NLB with scheme=internal works out of the box — shared code paths with ALB handle it correctly. Add unit test and E2E coverage: - Unit test: NLB internal creates with no public IP, internal- DNS prefix - E2E phases 10-14: internal NLB lifecycle with VPC client TCP traffic - E2E passed 56/56 with perfect round-robin (20/20 requests) * feat: consolidate LB e2e tests into run-lb-e2e.sh (mulga-854) Replace 3 separate LB dataplane scripts with a single consolidated test that covers all 4 variants (ALB/NLB × internet-facing/internal). - Share one VPC, subnet, and 2 dual-purpose app instances across all suites - ALB + NLB provision in parallel within each phase (internet-facing / internal) - One client VM tests both internal LBs simultaneously - Replace sleep 100 with LB health check gates - Delete standalone alb-e2e.yml, integrate into e2e.yml for both single and multi-node * chore: disable LB E2E tests in CI until dev env issue resolved * fix: read LB agent credentials from spinifex.toml instead of system-credentials.json The daemon couldn't read system-credentials.json due to systemd hardening (file owned by root:root 0600, not in ReadOnlyPaths). Use the same system credentials already loaded from spinifex.toml via the predastore config section — these are identical keys written by admin init. * feat: automate LB image build and import (mulga-861) - dev-install.sh: build lb-agent and run build-system-image.sh --import after services start so the LB image is available out of the box - build-system-image.sh: add --quiet flag to suppress build output, remove hardcoded --config path (uses auto-detected default) - images.go: rename alb-alpine → lb-alpine to match current naming * update go version * add LB e2e tests * fix: clean LB image build dir and wait for nbd device readiness Previous CI runs can leave stale state in /tmp/lb-image-build/ causing e2fsck failures. Clean the build dir before each CI build. Also add a readiness check after qemu-nbd connect to avoid race conditions. * fix: serialize nbd access with flock for concurrent CI builds Single-node and multi-node CI jobs share the same host and both build the LB system image using /dev/nbd0. Use flock to serialize the nbd connect/mount/customize/disconnect section so concurrent builds wait rather than collide. * fix: move flock to cover entire image build, not just nbd section Concurrent CI jobs share BUILD_DIR and /dev/nbd0. The lock must be acquired before downloading/copying the image, not just before nbd connect, otherwise the second job can clobber the first job's files. * fix: skip LB image build if output is fresh from concurrent CI job When concurrent CI jobs build the same image, the second job now reuses the first's output if it's less than 10 minutes old, avoiding a duplicate download + build cycle. * fix: filter AMI discovery by name in E2E tests With the LB Alpine image now imported alongside Ubuntu, the unfiltered describe-images call picked up the wrong AMI. Filter by the expected image name so tests use the Ubuntu image for instance lifecycle. * fix: filter AMI discovery by name in multi-node E2E tests Same fix as run-e2e.sh — with the LB Alpine image now imported, the unfiltered describe-images picks the wrong AMI. * fix: use awsgw bind IP as LB agent gateway URL fallback When br-mgmt is not configured and dev_networking is off (production with external pool), the gateway URL was falling through to the external IPAM pool's GatewayIP — the WAN router, not the Spinifex node. LB VMs couldn't reach the awsgw, so heartbeats never arrived and ALBs stayed stuck in provisioning. Use the awsgw bind IP instead — LB VMs on the OVN overlay can reach the host node's WAN IP where the gateway listens. * fix: capture StateReason and daemon logs when LB app instances fail Instances with user-data terminate immediately in CI. Add describe-instances StateReason query and daemon log dump to diagnose whether it's volume_preparation_failed or launch_failed. * fix: prevent orphaned AMI from breaking LB E2E and avoid stuck provisioning Three fixes for CI E2E failures: 1. run-e2e.sh: Remove custom AMI from predastore after deleting its backing snapshot. Previously the AMI remained discoverable but unusable, causing run-lb-e2e.sh to pick it and fail with NoSuchKey on the deleted snapshot. 2. run-lb-e2e.sh: Prefer ubuntu AMI by name for app instances instead of picking any non-alpine image. Also improve debug log dump to use journalctl (binary-install CI uses systemd, not log files). 3. CreateLoadBalancer: Only set provisioning state when a system VM was actually launched. Previously a failed launch still set provisioning, leaving the LB stuck forever with no agent to send heartbeats. * fix: use AWSGW bind IP as LB agent gateway URL in multi-node In multi-node deployments, the AWSGW binds to the WAN IP (e.g. 192.168.1.26:9999), but wireLBAgentConfig was using the br-mgmt IP as the gateway URL. Since the AWSGW doesn't listen on br-mgmt, the LB agent's heartbeat was refused, leaving internet-facing ALBs stuck in provisioning. Internet-facing LBs now use the AWSGW bind IP directly — the VM reaches it via VPC/external networking. Internal LBs get a host route on the mgmt NIC so they can reach the AWSGW bind IP through br-mgmt. * fix: add mgmt route via bootcmd for multi-node internal LBs Alpine cloud-init does not support v2 routes under ethernets — adding a routes: block to the network-config breaks all networking in the VM. Instead, add the host route via a bootcmd in the LB user data. This runs early in cloud-init and adds `ip route add {awsgw}/32 via {br-mgmt}` so internal LB agents can reach the AWSGW through the management NIC when the gateway binds to a specific WAN IP. Also moves the networking service from boot to default runlevel in the LB Alpine image to prevent a race with cloud-init. Verified locally: internal ALB transitions to active with AWSGW bound to WAN IP (simulating multi-node). * add LB enable flag to e2e * test: add elbv2 tests to meet diff coverage threshold Cover health checker edge cases (invalid JSON, empty servers, zero thresholds, port-0 fallback) and service_impl paths (mgmt route cloud-init, protocol validation, missing target group). * fix: swallowed health report error + needless marshal round-trip in LBAgentHeartbeat Add handleHealthReportDirect(report) to healthChecker so LBAgentHeartbeat can pass the struct directly instead of marshaling to JSON and immediately unmarshaling. Promote handleHealthReport unmarshal error from Debug to Warn so failures are visible in logs. * fix: validate credentials before launching LB VM + set failed state on launch error (mulga-867) Guard lbVMUserData with empty-value check for gatewayURL, systemAccessKey, and systemSecretKey — blank env vars previously caused agent auth-loop with no daemon-side log. Set LB state to failed (not active) when VM launch fails so the API reflects the broken data-plane instead of black-holing traffic. * fix: data race in getSystemAMI/getSystemInstanceType lazy caching Use sync.Once to guard lazy resolution of systemAMI and systemInstanceType fields. Concurrent CreateLoadBalancer calls previously raced on unsynchronized read+write of these cached values. The setter methods reset the Once so re-resolution works when the func is set after initial construction. Also fix pre-existing nosprintfhostport lint in daemon.go. * fix: use utils.PublishEvent for NAT events + return errors from isEIPOwned publishNATEvent in eni.go and daemon was copy-pasted boilerplate — now delegates to utils.PublishEvent (matching the EIP service pattern). isEIPOwned previously swallowed all KV errors and returned false, which could release an EIP-owned public IP if the store was temporarily unavailable. Now returns (bool, error) so the caller skips release on KV failure instead of assuming no EIP ownership. * fix: narrow health checker to only scan target groups for reporting LB handleHealthReportDirect was calling ListTargetGroups() on every heartbeat (every 5s per LB agent), scanning all TGs. Now uses report.LBID to follow LB → listeners → TG ARNs and fetch only the attached target groups via direct key lookups. Falls back to full scan if LBID is missing. * fix: pass region through to LB agent SigV4 signing (mulga-871) LB agent hardcoded us-east-1 as the SigV4 signing region. Thread region from ELBv2 service config → cloud-init LB_REGION env var → lb-agent --region flag → lbagent.New() → signer.Sign() call. * fix: add account-level authorization to LBAgentHeartbeat and GetLBConfig Both endpoints accepted any accountID without verifying the LB belongs to that account. Now verify accountID matches the LB's AccountID, with an exception for the system account (000000000000) which the ALB agent uses for heartbeats and config fetches. * fix: upgrade silent failure paths from Debug to Warn/Error - healthChecker target group list failure: Debug → Warn - HAProxy stats socket error in agent tick(): Debug → Warn - ENI delete EIP-owned skip: Debug → Info (operational visibility) - wireLBAgentConfig: warn when no gateway host is resolved - detectMgmtIP: warn when bridge exists but IP is empty - stale mgmt TAP cleanup: log errors instead of discarding * refactor: deduplicate MAC hash function into utils.HashMAC Three identical hash-to-MAC implementations (generateDevMAC, generateMgmtMAC, generateENIMac) now delegate to a shared utils.HashMAC(prefix, id) helper. * fix: guard LBAgentHeartbeat KV writes to state transitions or stale timestamps Previously the full LoadBalancerRecord (including ConfigText) was written to JetStream KV on every heartbeat (every 5s). Now only persists on state transitions (provisioning→active) or when LastHeartbeat is older than 60s. * fix: fail ALB launch when public IP allocation fails for internet-facing scheme Previously, IPAM allocation failure was logged as a warning and the ALB VM continued launching without a public IP. The ALB reached active state but was unreachable from the internet. Now the launch returns an error and cleans up the partially-created instance, causing the ALB to enter the failed state. * fix: retry system AMI resolution when image not yet imported sync.Once cached empty results permanently — if the daemon started before the system image was imported, all CreateLoadBalancer calls silently skipped VM creation. Replace sync.Once with a mutex that only caches non-empty results, allowing retry on subsequent calls. Same fix applied to getSystemInstanceType for consistency. * fix: propagate updateStoredConfig errors to API callers (mulga-884) updateStoredConfig and updateStoredConfigForTargetGroup now return errors instead of silently swallowing them. Callers (CreateListener, DeleteListener, RegisterTargets, DeregisterTargets) return ErrorServerInternal when config persistence fails, preventing the API from reporting success while HAProxy serves stale configuration. * test: add unit tests for wireLBAgentConfig, isEIPOwned, toHealthReport (mulga-885) * fix: parse HAProxy templates once at package level (mulga-875) generateALBConfig and generateNLBConfig were calling template.New().Parse() on every invocation. Move to package-level template.Must for one-time parsing. * fix: fail launch for internal-scheme ALBs when mgmt NIC setup fails (mulga-886) When management TAP or IP allocation fails, internal-scheme ALBs now return an error instead of silently continuing without a management NIC. Without the mgmt NIC, the LB agent cannot reach the gateway and the ALB gets stuck in provisioning forever. Internet-facing ALBs retain the existing degrade-and-continue behavior. * refactor: extract lbArnPathSegment helper for ALB/NLB ARN construction (mulga-877) The app/net path segment derivation was duplicated in buildLBArn, buildListenerArn, and inline in CreateLoadBalancer. Extracted into a single lbArnPathSegment helper. * refactor: use aws.StringValue in toHealthReport (mulga-878) Replace manual nil-check-then-deref with aws.StringValue for cleaner pointer-to-value conversion in the heartbeat health report builder. * refactor: use testutil.StartTestJetStream in health_checker_test.go (mulga-876) Replace hand-rolled setupTestNATS (~20 lines) with the shared testutil.StartTestJetStream helper. Simplify return to just *Store since no caller needed the *nats.Conn. * fix: address PR #122 review comments (mulga-887) - Extract NATEvent named type from anonymous struct in publishNATEvent - Remove unused build-lb-image.sh thin wrapper - Remove SPX_ENABLE_LB env var from e2e workflow (unused) - Remove no-op healthChecker start/stop methods - Export simple ELBv2 fields, remove trivial setter methods - Delete old e2e scripts replaced by consolidated run-lb-e2e.sh * fix: check specific error type for ENI deletion on termination (mulga-888) ENI deletion errors were all silently downgraded to Debug level, masking genuine failures (KV issues, permissions, ENI stuck in-use) that cause orphaned ENIs and leaked IPAM addresses. Now only NotFound is logged at Debug; all other errors log at Error. * fix: propagate non-ENODEV errors from GetBridgeIPv4 (mulga-890) GetBridgeIPv4 returned ("", nil) for ALL InterfaceByName failures, not just missing devices. Permission errors on /sys/class/net/ were silently treated as "bridge absent". Now only "no such network interface" returns nil; other errors propagate to the caller. * fix: remove backend address from proxy error response (mulga-892) The reverse proxy error handler included the internal backend host:port in the XML error body sent to clients. The address is already logged server-side; the client now sees a generic message. * fix: release mutex before KV store I/O in health checker (mulga-893) handleHealthReportDirect held hc.mu across PutTargetGroup calls (JetStream KV Put — network I/O), serializing concurrent heartbeats from independent LBs. Now collects changed TGs under the lock and persists them after releasing it. * fix: warn when system credentials missing in wireLBAgentConfig (mulga-894) fix: log DetachENI errors instead of discarding with blank identifier (mulga-895) * fix: detect veth bridge mode and use centralized NAT for it The veth-pair bridge topology (Linux bridge ↔ veth ↔ OVS bridge) was misclassified as direct bridge mode, causing OVN to use distributed NAT. Distributed NAT hairpin routing fails through the Linux bridge intermediary, breaking host-to-local-LB connectivity. Add BridgeModeVeth constant, detect veth-wan-ovs interface in detectBridgeMode(), and introduce useCentralizedNAT() to gate NAT mode and nat-addresses=router for both macvlan and veth modes. isMacvlanMode() is preserved for macvlan-specific behavior (MAC alignment). * fix: add retry loop for peer NLB reachability test With centralized NAT, ARP resolution for NAT external IPs relies on gratuitous ARPs from the gateway chassis. These take time to propagate to peer nodes. The peer→NLB test had a single 3s attempt while all other connectivity tests use retry loops. * fix: use request-reply for vpc.add-nat to prevent ARP propagation races publishNATEvent was fire-and-forget for vpc.add-nat, meaning the daemon continued before vpcd committed the OVN NAT rule. This caused intermittent NLB connectivity failures: the LB was reported active before OVN had the DNAT rule and sent GARPs for the public IP. Switch vpc.add-nat to request-reply (RequestEvent) so the daemon waits for OVN confirmation. Also increase nc timeout from 2-3s to 5s in E2E tests for more reliable TCP probing during ARP resolution. * fix: clean up stale NAT rules before adding new ones for reused IPs When a public IP is returned to the external pool and quickly reallocated (e.g. instance terminated, then LB created), the fire-and-forget vpc.delete-nat may not have been processed yet. handleAddNAT now removes any existing dnat_and_snat rule for the same external IP before creating the new one, preventing duplicate/conflicting OVN NAT rules. This was causing intermittent NLB connectivity failures in multi-node E2E where NLB public IPs collided with recently-freed instance IPs. * fix: clean up stale NAT rules across all VPC routers on IP reuse The previous fix (0a93081) only removed stale dnat_and_snat rules from the target VPC's router. When a public IP is freed from one VPC and reallocated to a different VPC before the fire-and-forget vpc.delete-nat is processed, the stale rule on the old VPC's router caused duplicate OVN ARP responses — traffic was black-holed when the host resolved to the wrong router's MAC. DeleteAllNATsByExternalIP now searches every logical router for stale NAT rules matching the external IP and removes them in one transaction. Also adds host connectivity diagnostics (ARP, routing, OVN NAT state) to run-lb-e2e.sh so the next CI failure surfaces the root cause of the persistent NLB-from-host issue. * fix: run LB inet tests from actual hosting node, not hardcoded node1 The host/peer connectivity tests were hardcoded: "host" = node1 (where the test script runs), "peer" = node2. But LB VMs land on whichever node picks up the NATS queue message, so the tests were randomly testing local vs remote connectivity depending on placement luck. Now the test: 1. Accepts --nodes <ip1> <ip2> ... (all cluster node IPs) 2. After LB creation, discovers which node hosts each LB VM by checking for the ENI's tap device 3. Runs "local" tests from the hosting node and "remote" tests from a different node Also updates e2e.yml to pass --nodes instead of --peer.	2026-04-10 10:27:20 +10:00
Julian Sommer	a56a2baf1f	feat: privilege separation (#114)	2026-04-07 16:22:19 +10:00