18 Commits

Author	SHA1	Message	Date
Ben Duncan	7163f7d62d	feat(oci): run Spinifex on Oracle Cloud, and fix the public-address and NAT gateway lifecycle (#1108) ... * feat: ship uninstall-spx.sh and fix the OCI bring-up path Adds scripts/uninstall-spx.sh, the inverse of setup.sh: it removes the software where node-reset.sh clears state and leaves the node installed. An operator who installed with `curl ... | bash` had no teardown path at all. State teardown is delegated to node-reset.sh rather than reimplemented, and setup.sh now ships both to /usr/local/share/spinifex so they are reachable without a checkout. It refuses on a multi-node cluster and --yes does not override that: a flag meaning "don't ask me" must not also mean "ignore a safety property". /var/lib/spinifex and every apt package are kept unless asked for, and both opt-ins confirm separately even under --yes. Proved by a full uninstall/reinstall round trip on dev-oci, which found two bugs neither prod nor dev-prod can reach, because both are ISO-installed: node-reset.sh exited 2, silently, at its last step and after the wipe. setup.sh creates the spinifex *group* and the spinifex-* service users but never a *user* named spinifex, so `getent passwd spinifex` exits 2 and pipefail carried it into set -e. The uninstaller broke openvswitch-switch on its first run by removing the socket-perms helper while leaving the drop-in referencing it, failing the unit at step EXEC with status 203. Fixed on both sides: the uninstaller removes drop-ins it wrote into other units' namespaces and restarts anything it failed, and setup-ovn.sh writes ExecStartPost=- so a missing helper can never stop another package's daemon. setup-ovn.sh also learns the shape OCI needs: when --wan-bridge names a Linux bridge it links it to OVS with a veth pair instead of enslaving the NIC, so the VNIC keeps its MAC identity and netplan keeps owning the addresses. It now fails on OVS datapath errors instead of reporting success, and the --management demotion guard keys on /etc/default/ovn-central rather than on database presence, which the ovn-central package creates on every node. Tests: scripts/uninstall-spx_test.sh, wired into make test-build-scripts along with shellcheck over the four scripts that ship to every node. * feat: allocate external IPs from the Oracle Cloud API An OCI VNIC drops any packet whose source is not a registered private IP object on it, so an external pool cannot be range math there — the address has to be created through the provider before it exists at all. Add source = "oci" and the allocator behind it. One allocation is two OCI objects: a secondary private IP that rides the wire, and a RESERVED public IP attached to it that the internet reaches. Allocate returns the public address because that is what AWS semantics are about, and records the pair so the host plumbing and release path can find the private half. The OCI objects are created before the binding is written, and dropped after it on release — both orders chosen so a crash leaves a discoverable leak rather than a record that lies. Reconcile squares the two on startup and is required, not optional: a leaked reserved public IP bills and holds one of the 50 per-region slots with nothing referencing it. It never touches an object without our display-name prefix, because the reference host carries operator-created secondaries on the same VNIC. Resolve the VNIC from an interface name by MAC through IMDS rather than by Oracle's VNIC ordering, which is not a documented contract. Generalise ExternalIPAM.EnableDHCP into InstallAllocator so the OCI wiring is not a second single-purpose installer. * fix: refuse source = "oci" outside routed mode, and cover the SDK path An OCI VNIC accepts exactly one MAC, its own, and delivers every inbound frame to it regardless of which of its addresses the packet is for. Measured on the reference host: an internet source reaches 10.200.0.144 as router-MAC to VNIC-MAC, the public address never appears on the wire at all, and every off-subnet destination resolves to the same next hop. Pool mode puts a per-VPC gateway LRP MAC and a per-rule external NAT MAC on the uplink and ARPs for the address, which policy.go:50 justifies with "the Linux bridge learns the per-rule external MAC on the veth port like any other MAC". An OCI VNIC does not learn MACs; it accepts one. So a pool-mode guest's egress is dropped and its ingress never reaches OVN. Routed mode keeps the address on the host, which already wears the VNIC's MAC. Pinning every LRP to that MAC does not rescue pool mode: they share one external logical switch and OVN resolves inbound frames by an L2 lookup on it, so duplicate MACs work for one VPC and silently misroute past that. The refusal names the MAC reason, because a refusal that only says "use nat mode" reads as arbitrary and gets worked around. Also test the SDK request shapes against a local endpoint. Two of them are wrong in ways reading does not catch: pagination stopping early reports a partial VNIC to reconcile, which then drops live bindings, and a nil privateIpId on detach is omitted from the body entirely, so OCI reads the update as a no-op, returns 200, and leaves the address attached and billing. * fix: install host firewall rules where a distro's catch-all reject cannot precede them Oracle's Ubuntu cloud image ships a catch-all REJECT in both INPUT and FORWARD. Two Spinifex datapaths were silently dead behind it on any such host. EnsureNATEgressRules probed with -C and appended with -A, so its ACCEPT rules landed after the REJECT and never matched: 24113 packets on the reject, 0 on both accepts, and guests had no egress. The comment named the case it did not handle, anticipating a FORWARD policy of DROP — which an append does survive — rather than an explicit REJECT rule, which it cannot. Insert at the head, and drain existing copies first, because a rule an earlier build appended still satisfies -C and would otherwise stay stranded across an upgrade. Nothing opened INPUT for the IMDS endpoints at all. The responder binds .254 and .253 per instance on the ime- devices, so guest requests are locally delivered and meet INPUT. Requests arrived and no reply ever left; cloud-init spent 490s timing out and reported the endpoint as disabled. With one wildcarded accept the same image boots in 18.6s and finds its datasource. The natuplink e2e check had the same blind spot the code did, asserting -C and therefore presence but never position. It now asserts the accepts precede the first reject. * feat: OCI address pairs, and an IMDS remap so the host keeps its own metadata An OCI external address is two objects, not one: a RESERVED public IP that OCI 1:1-NATs upstream and never puts on the wire, and a secondary private IP on the VNIC that the wire actually carries. Every AWS-facing record holds the public half; every datapath object must hold the private one. The mapping goes in policy.natManager and nowhere else. AddEIP, DeleteEIP, PruneOrphanEIPs and the HostEIPBinder all converge there, and reconcile rebuilds the rule set from AWS-facing intent on every pass — so a correctly mapped rule installed further upstream is swept on the next reconcile and replaced with one matching an address that never arrives. The seam is a DatapathResolver defaulting to identity, so every non-OCI deployment is unchanged. A resolve failure is fatal: falling back to the public address would install a rule that reads correctly in ovn-nbctl and blackholes. ocinet.Lookup is the OCI side — read-only, KV-backed, no client and no credentials, because vpcd is a different process from the daemon that owns the allocator and the bindings record is all either would consult. Two live bugs the in-memory test store was hiding, both found by making the fake report ErrNotFound the way the real store does: - KVStore.Mutate could not create, so every pool failed its own first allocation after OCI had already created and billed the pair. - Reconcile treated a missing record as a failure, aborting the one pass that exists to find an object created before its binding was written. Guest IMDS must stay on 169.254.169.254, which is also where a cloud host's own metadata service lives — the first guest launch made it a local address and took the host's metadata and DNS with it. The datapath is OpenFlow steered, so the guest-facing addresses stay put and only the host-side endpoint address moves, with a PREROUTING DNAT on ime-+ translating and conntrack un-NATing the reply. Off by default; identity when unset. The IMDS firewall rules move to vpcd. They are wildcarded over every endpoint rather than per-launch work, and spinifex-daemon holds no iptables grant and no CAP_NET_ADMIN by design, so attempting them aborted every launch. * fix: name the real constructor in the IMDS bind-address error, and mark the NAT datapath test in-package The error string pointed at DefaultHostBindAddrs, which does not exist — a caller following it would have looked for a symbol that was never written. nat_datapath_test.go reuses nat_test.go's in-package fixtures, so it carries the //test:in-package marker the package check asks for. * docs: index what each e2e suite proves Written because a datapath comment was read as an end-to-end behavioural claim and the suite that already answered it was found only after the wrong answer had been given. TestVPCEgressPaths/NATBaseline is the authority on private subnet egress, and nothing pointed at it. * fix: source-route an EIP's reply out the VNIC that holds it An EIP's reply carries the EIP as its source. On a host that pins its uplink with a policy rule, the reply took the main table instead and left by the wrong interface — which a cloud VNIC drops, because the address is registered on a different one. Ingress reached the guest and nothing came back. Every bind now mirrors the uplink's rule onto the EIP. The selector is the rule rather than the interface: both OCI VNICs sit in one subnet, so "the interface owning the EIP's subnet" names two of them, and two candidate rules is a refusal rather than a guess. A host with no source routing gets no rule and no change. * docs: name the setup-ovn recipe routed mode actually needs on OCI The OCI guide told operators to run `--wan-bridge=br-wan`, which is mutually exclusive with `--nat-uplink` and deletes the spx-nat transit veth routed mode runs on. The node then refuses to bring the uplink up, with an error naming the flag the guide should have used. Several messages also claimed routed mode is outbound-only with no public IPs. That stopped being true when the routed EIP path landed, and it reads as "what you are about to do cannot work" during the one init the OCI path documents. * feat: let routed NAT form a multi-node cluster, and carry every pool through formation Lifts the --nodes >= 2 refusal on --external-mode=nat. The refusal carried no technical justification -- it was a v1 scope decision -- but routed NAT is the only datapath a cloud VNIC can carry, so "routed NAT is single-node" had become "Spinifex on a cloud host is single-node". It is not fixed by lifting it. Measured on three OCI nodes: the cluster forms, all chassis register and guest-to-guest traffic runs at 7.1-7.7 Gbit/s, but every node claims every EIP, nodes disagree on the VPC's gateway LRP, and only the gateway chassis reaches its own guests. init now says so on stderr rather than refusing, because the refusal made the defects unreachable and therefore unfixable. Distributing the EIP datapath is the actual fix. Lifting it exposed a second defect that stops formation outright. NetworkConfig held nine flattened Pool* scalars and the init node set PoolName = "wan" unconditionally; routed NAT renders two pools and the one that matters -- nat-transit, carrying the gateway-LRP range -- was dropped. Each joiner got a single pool named "wan" holding the transit gateway and the detected WAN prefix, and the leader re-rendered its own config from the same struct, so all three nodes were wrong. Pools is now []admin.PoolData in render order, and PoolData gains json tags because it is on the wire now rather than a template input. * feat(install-node): add --external-mode so a cluster can be formed in routed NAT install-node.sh could not express routed NAT at all. It required --external-pool/--external-gateway/--external-prefix-len and passed --external-mode=pool to every spx admin init unconditionally, so the only supported way to form a routed-NAT cluster was by hand. --external-mode nat refuses the three pool flags rather than ignoring them, and threads --nat-uplink into setup-ovn.sh -- without it the WAN interface check rejects a physical NIC during formation, which is correct in pool mode and wrong in routed mode, where nothing is bridged. * fix(ocinet): stop each node deleting its peers' OCI addresses on startup Reconcile compares the binding record against the private IPs on one VNIC. The record is one key per pool and therefore cluster-wide, so on more than one node every peer's binding looks exactly like a binding whose objects OCI has lost. Each node dropped them, and direction 1 on the owning node then collected the now-unclaimed objects as leaks -- deleting the reserved public IP and private IP of a running instance. Observed on the three-node OCI cluster: node2's startup pass logged "collected a leaked address" for an address it had allocated ten minutes earlier and "dropped a stale binding" for one living on node3's VNIC. The damage did not stop at the OCI objects. With the binding gone, Lookup.DatapathIP returns the public address unchanged, so the OVN dnat_and_snat row carries the public address instead of the private half. Guest egress then leaves the VNIC sourced from a reserved public IP, which OCI drops because it enforces the registered source per VNIC -- and EnsureEIPSourceRoute no-ops too, since a public address shares no subnet with any host address. One destroyed binding takes out both directions. Binding.VNICID already exists for exactly this ("how a node that comes back from the dead tells mine from moved on without me") and Reconcile never consulted it. A binding is now only dropped when it names this node's VNIC; one naming no VNIC cannot be attributed and is left alone rather than guessed at. * fix(host): find the EIP's source-routing table when the uplink is held as a /32 uplinkSourceRuleFor identified the interface owning an EIP's subnet by finding a host address configured with a prefix that contains it. On OCI both VNICs sit in one subnet, so br-wan has to hold its address as a /32 -- a /23 there puts the subnet's on-link route in the main table at metric 0 and takes the whole subnet off the primary VNIC. A /32 contains nothing but itself, so no rule was ever found, no source rule was installed, and the reply left by the primary VNIC where OCI dropped it for carrying an unregistered source. The address is not the only record of who owns the subnet. The rule names a table, and that table holds the on-link route. Consulted as a fallback, only when the address prefixes matched nobody, so a host that already resolved this the direct way keeps its existing answer -- prod and dev-prod are unaffected. Only scope-link routes count. A default route covers every address and would make every source-routing table an equally good answer. * docs(oci): correct the multi-node story and name the oci_vnic_id constraint The nat-mode warning said EIPs and default egress were both broken on more than one node. Default egress is fine: delete a guest's dnat_and_snat row and it reaches 1.1.1.1 in 2.3 ms from a guest on any of three nodes. Only the public-IP path was broken, and the three defects behind it are now fixed. What remains is not a bug but the shape of the datapath: routed NAT centralises the EIP path on the VPC's gateway chassis, ext-shared is a localnet so the other nodes cannot reach it, and OCI drops any frame whose source is not registered on the VNIC it leaves by. So every node's pool must pin oci_vnic_id to the gateway chassis's VNIC rather than resolving its own. Say so in §5.3, where an operator writing the pool will read it, and in §9, where the cost is that a gateway change kills every public IP until the pools are repointed. The single-node cap itself is gone from the doc: a three-node cluster has been formed and measured, and the warning proceeds rather than refusing. * feat(nat): distribute the routed-NAT EIP path so it works on more than one node Routed NAT processed every EIP on the VPC's gateway chassis, and ext-shared is a localnet, so the transit segment a /32 route pointed into existed only on that one node. Measured on three nodes: all three claimed all three EIPs, the nodes disagreed on the gateway LRP, and only guests on the gateway chassis were reachable. The mechanism to fix it is not new. Pool mode already distributes EIPs by putting external_mac and logical_port on the dnat_and_snat row and letting OVN's ARP responder answer for the external IP on the external segment. Routed mode has an external segment too, the spx-nat veth pair, and it being per node is the distribution: only the node running the instance can answer ARP on it. So a routed rule is now distributed on the same terms as a pool one, and its host route becomes on-link on the transit veth instead of via a gateway LRP that may be on another chassis. bindHostEIP stops demanding that address for a distributed rule, which is what previously failed the bind on every node but one. vpcd plumbs only the ENIs local OVS reports as its own, asked of OVS rather than the Southbound DB because the question is about this host. The gateway LRP keeps its transit-pool address. Default egress routes through it and works across three nodes today, so making it link-local as pool mode does would break a working path to fix one distribution has already fixed. Pool and static are unmoved, and a test asserts that against the same input rather than leaving it to inspection. A rule with no MAC stays centralised, so a cluster part-way through an upgrade is never in a state neither shape handles. Beads: mulga-opyl7, mulga-la0pe * fix(nat): make routed-EIP host plumbing work on every node Three guests on three OCI nodes, one guest per node: only one was reachable. Two defects, both invisible on a single node. The transit veth took a random MAC per node, but OVN holds one MAC binding for the transit nexthop across the whole cluster — so every chassis addressed its egress to whichever node seeded the binding, and the other kernels dropped it at L2. The pair is node-local, so one locally-administered address everywhere is unambiguous; setup-ovn.sh pins it at creation and in the netdev unit, and EnsureUplinkPort repairs a veth made before that. Host routes and proxy-ARP also rode the leader-gated reconcile pass, so a node that never won the lease never plumbed its own guests' EIPs. BindHostEIPs does that half on its own loop on every node; Bind already leaves alone an EIP whose ENI is elsewhere, so two nodes never contend. An unresolvable address skips the sweep rather than shrinking it. Pool and static are untouched: the new pass is a no-op outside routed mode, and a test asserts it. Also adds --teardown and --dry-run to setup-ovn.sh, for re-running it on a node that is already tangled. It deliberately spares the NB/SB databases, /etc/spinifex, the system-id and the packages, and refuses to del-br a Linux bridge. * fix(nat): assert the transit MAC where vpcd actually runs it EnsureUplinkPort is never called in production — host.Wiring is constructed nowhere — so the repair landed in a path the deployed daemon does not take, and three nodes came back with three different MACs. vpcd re-asserts it beside the NAT egress rules instead, which is the routed-NAT branch that does run on every start. * fix(vpcd): make uplink-mode detection say what it saw, and check it detectBridgeMode logged only the branch it took, so a node detected as veth when it was wired for nat looked identical to one that was right. It now reports both probes every time, errors when a node carries both uplink veths (set up twice, and --teardown is the way out), and errors when external_mode and the wiring disagree — the datapath follows the wiring, so the config silently loses that argument. An explicitly configured bridge_mode gets the same cross-check, since nothing else ever compared it against the wiring. The vpc-networking doc now opens nat with "start with pool unless one of these applies" and gains a section on nat across more than one node; the admin warning for --external-mode=nat on two or more nodes says what was verified by hand and what is still uncovered. * test(nat): prove routed NAT delivers each node's own guests, on a cluster natuplink ran single-node only, which is exactly where the two defects that broke three nodes are invisible. It now admits ModeMultinode, and phase 9 takes a second shape there: a guest per node in a spread placement group, and for each one — its public IP plumbed by the node running it and by no other node, the route on-link with proxy-ARP answered there, a dnat_and_snat row carrying external_mac and logical_port, a TCP handshake in from another node, and the guest's own console verdict for egress. Every node's transit veth is checked against the one cluster-wide MAC before any guest boots. Single-node phase 9 gains the same row and on-link assertions, so the shape is held on both topologies. Two gates come from what three OCI nodes actually do: every node holds a gateway_chassis row for every VPC (the active one is the highest priority) and every node installs the VPC ingress route, so only phase 7 needs gating — it runs where the gateway port and the guest are both local, and logs which one is missing otherwise. Cell 20 is real-multi x nat with its own nightly step, leaving the permutation cells' suite set alone. * test(nat): run the VPC egress paths in nat mode, not pool mode only The single suite gated TestVPCEgressPaths and TestWANEgressSMTPBlock on external_mode = "pool", with a comment saying nat clusters have no EIP/NAT-GW support. They have had EIPs since the routed path landed, so the gate now asks whether the cluster can allocate public addresses at all — pool, or nat with a non-transit pool configured. The nat-single cell runs TestVPCEgressPaths beside natuplink, which is the only nightly coverage of a private subnet's isolation and its NAT gateway in routed mode. natuplink never creates a private subnet. * docs(config): list nat among external_mode values * fix(ci): renumber the multi-node nat cell off the bare-metal job Cell 20 is already the bare-metal single-node job, gated on the same cells input but declared in the workflow rather than e2e-cells.json. A matrix entry with the same number runs two unrelated jobs under one name, and naming 20 in a dispatch selects both. * feat(images): add Oracle Linux 10.1 and 9.8, and pin their catalog digests Oracle publishes its qcow2 checksums inline on an HTML page and ships no sums file, so the four new entries carry the expected digest in the catalog rather than a URL to fetch one from. Images gains ChecksumDigest for that, and the catalog import path verifies against it and says which of the two sources it used, so a recorded digest never reads as a fetched sums file. Pinning is safe here and only here: each URL names an immutable build (b291, b293, b178, b182) and never moves. aarch64 has a -kvm-cloud- build alongside the plain -kvm- one; the cloud build is the cloud-init image and is the one that matches x86_64's -kvm-. TestImageCatalogReachable HEADs every URL in the catalog from the general single-node suite. These artifacts live on hosts nobody here controls, so an upstream that retires a build breaks `spx admin images import` for a customer and nothing else in the suite would notice. It also asserts that every entry sets exactly one checksum source, and that no pinned entry points at a "latest" URL whose bytes would change under it. * fix(formation): carry bridge_mode onto the leader and every joiner applyNetworkConfig copied external_mode, external_interface and the pools out of the formation NetworkConfig but never bridge_mode, so a multi-node cluster wrote a spinifex.toml without the key while single-node init wrote it. Every node then fell back to autodetecting a mode that init had pinned. Nightly cell 30 caught it: TestNATUplink failed at the config phase on a spinifex.toml that did not contain bridge_mode = "nat". Deriving it from the mode rather than adding a NetworkConfig field keeps the leader and the joiners on the same rule single-node init already uses. * feat(ui): show how to ssh into an instance, and which user to use An instance detail page gave the addresses and the key pair name but never the login user, which is the one part of the command nobody can derive. It belongs to the AMI, not to Spinifex: nothing in the launch path creates or renames an account, so stock cloud-init applies the distro's own default_user. Oracle Linux is cloud-user, Debian is debian, Ubuntu is ubuntu, and guessing from the distro name gets two of those three wrong. The card resolves the user from the AMI's name and description, and an unrecognised image yields a placeholder rather than a guess, because a wrong username in a copy-paste command is worse than an honest gap. A private-only instance is called out as reachable from inside the VPC only, rather than presented as a command that will simply time out. * fix(nat): plumb host state for a NAT gateway's public address AddNATGateway installed the OVN snat row and stopped there. On a routed-NAT node the host is what delivers a public address, so an egressing private guest left with a source the host held no route back to, and the masquerade rule — which matches the transit 100.127.0.0/24 alone — never saw the packet. The gateway's address needs the same /32 route into OVN, proxy-ARP and FORWARD accepts an EIP gets. hostBindSpecs unions floatingIPSpecs with one deduped spec per NAT gateway address and feeds BindHostEIPs. It is kept apart from floatingIPSpecs deliberately: pruneOrphanEIPs and addLive read that set as the complete list of addresses allowed to hold a dnat_and_snat, and a gateway holds an snat, so widening it would stop the sweep reclaiming a stale row on an address later reused as a gateway's. AddNATGateway also binds directly, on both the create and idempotent-skip paths, so a fresh gateway is not dark for up to 30s waiting on the host EIP pass — and so a reboot that keeps the OVN row still replays the volatile half. Documents NAT gateways in nat mode in vpc-networking, and says at the head of that section that nat is a supported mode covered by nightly cells 19 and 30. * docs(oci): document Oracle Linux as a guest image, and that its user is cloud-user Four catalog entries, all UEFI. The login user is the one that catches people: opc is the default on the images Oracle publishes into OCI itself, while these are the generic KVM cloud images from yum.oracle.com, whose cloud-init default_user is cloud-user. Verified on 10.1 and 9.8 — opc, oracle and ec2-user are all refused. Also records why these four pin a checksum digest inline (Oracle ships no sums file, and each URL names an immutable build), why arm64 takes the -kvm-cloud- build, and that SELinux ships enforcing. * fix(reconcile): stop reading an unreadable intent as an empty one A wedged JetStream KV returned no keys for the ENI bucket while writes were still landing. loadPorts mapped that to an empty port map, pruneOrphanPorts read every guest LSP as an orphan against it, and every guest on the node lost DHCP and IMDS while the API kept answering and the instances stayed "running". The existing re-read guard did not help: it re-reads through the same path. Three defences, because no single one covers every shape of the fault. A failed kv.Get now fails the pass instead of dropping the record. Dropping one VPC narrows localVPCs, which filters out every ENI in it, which the sweep reads as permission to delete every guest port in that VPC — one failed read, same outcome, and it needs no exotic KV state. Unmarshal failures stay skips: that is a poison record rather than an unhealthy store, and failing on it would wedge reconcile forever. The port sweep now asks local OVS which taps are plugged in here and never deletes a port that has one. A running guest's port cannot be an orphan whatever intent says, and OVS is the one liveness signal that does not come through the KV read path the sweep is otherwise entirely driven by. Wired from vpcd; unwired callers fall back to the guard below. An unreadable OVS skips the sweep rather than deciding on intent alone. Each sweep also refuses outright when the keep set it would delete against holds nothing at all — no ENIs, no SGs, or no ports and no external IPs. That is the case local taps cannot cover, because the drift sweep is leader-gated and lists NB cluster-wide, so the leader holds no tap for another node's guest. The cost is real: a genuinely orphaned row now survives until one live record of its class exists. Leaking a stale row is the recoverable half. Three existing prune tests drove their fixtures to a wholly empty intent, which is now the refusal case; they keep one live record of the class so they still prove the prune. The Get-failure path has no unit test — there is no jetstream.JetStream fake in the tree and an embedded server cannot be made to list a key it will then refuse. * docs(oci): separate the two address quotas, and the principal that can read them The regional reserved-public-IP ceiling is tenancy-wide and does not grow with node count, so a cluster hits it before the per-VNIC 64, which does. Records the commands that read both, and that a detached reserved public IP still bills. Reading limits or usage needs a tenancy-level grant, which cannot be compartment-scoped the way the node's policy is. That is a second, read-only principal kept off every node, not an addition to SpinifexOperators. * docs(oci): give quotas their own troubleshooting section, and say we enforce none Spinifex hardcodes no address ceiling: it attempts the allocation and maps OCI's own refusal to InsufficientAddressCapacity. So a quota is raised with Oracle and nothing here is reconfigured or restarted — which is what an operator needs to know before it bites, not after. Separates the two quotas by how they scale: the regional reserved-public-IP default is tenancy-wide and raisable by ticket, the per-VNIC 64 is per node and is not. * fix(oci): follow a guest to its new node when its EIP moves OCI delivers a public address to whichever VNIC holds the private IP behind it, and nothing ever moved that private IP. Binding.VNICID was written once at allocate time, as the VNIC of whichever queue-group daemon happened to serve the AWS request, and never updated — so an address landed on the right node only by chance, and a guest that stopped and started elsewhere went dark while every OVN row, host route and firewall rule on the new node was correct. ClaimLocalAddresses closes the gap from the node that should hold the address. Local OVS decides ownership, the same authority the host EIP binder already trusts for the same question, and UpdatePrivateIp with a new VnicId does the move in one server-side call. The private IP OCID survives it, so the reserved public IP attached to it follows without being touched and the customer's address never changes. A loop per node, not leader-gated: the question is about this host's guests, and a node that never wins the reconcile lease would otherwise never claim the addresses of the instances it is running. * fix(oci): record which ENI an EIP is associated with An EIP is allocated bare and attached by a later AssociateAddress, so the OCI binding written at allocate time named no ENI and nothing ever filled it in. The address then stayed on whichever node happened to serve the allocation: the affinity pass could follow an auto-assigned address, which is allocated with its ENI, but never an EIP — the case customers actually use. BindOwner is on an optional OwnerTracker interface rather than Allocator, because only OCI has a notion of an address being delivered to one node. A static or DHCP address is on the wire everywhere its node can reach, those allocators do not implement it, and ExternalIPAM treats a pool that does not as a success with nothing to do. The same gap made the release owner guard inert for EIPs: it compares against a binding ENI that was always empty, so a stale teardown could free an address already handed to another instance. * fix(ec2): report the address an instance is actually reachable on Three faults kept DescribeInstances answering with the address launch assigned, whatever happened afterwards. The launch-time stamp lives on the stored ec2.Instance, and the stopped path only declined to add an address rather than removing one, so a stopped instance went on advertising an address it had released and that may already belong to someone else. The projection now clears it, and the NIC association with it, working on its copy so a restart can still report the address. The running path filled the public address only when it was nil, which the stamp never is, so an EIP could not be reported however correctly it was recorded. The runtime address now wins. And nothing told the rest of the system about an association: AssociateAddress wrote neither the ENI record, which is what IMDS and the network reconcile read, nor the instance record, which is what a describe answers from. The ENI write is direct; the instance write is a fan-out, because the queue-group worker that serves AssociateAddress is rarely the node running the instance. * fix(net): take the host EIP plumbing off a node its guest has left Bind refuses an EIP whose ENI is on another chassis; the prune beside it kept anything still wanted cluster-wide. A guest that moved therefore left the old node holding the /32 route into OVN and the proxy-ARP entry on the uplink, with nothing that would ever remove them. On a shared segment that node goes on answering ARP for an address another node is now serving. The two halves now ask the same question by the same authority, in the loop that already runs per node. An unreadable local OVS is not a yes: tearing down a live address on a guess would fire everywhere at once, so it keeps the binding and the next pass tries again. A binder with no chassis test behaves exactly as before, and a centralised EIP is exempt because it hairpins through the gateway chassis and every node plumbs it legitimately. * fix(net): refuse to free an Elastic IP on behalf of a departing interface Releasing an instance's public address on teardown releases it by address, scoped to the ENI holding it. Since an association now writes the EIP onto the instance record, that address is the EIP — so terminating an instance gave a customer's Elastic IP back to the pool while describe-addresses still reported it as theirs, free to be handed to another guest. The existing ownership guard cannot catch this. It only fires when the stored allocation already names an ENI, and an EIP is allocated bare and associated later, so the stored ENI is empty and the guard passes straight through. The distinction that holds is who is asking. ReleaseAddress is the address's own owner and names no interface; instance teardown names the interface it is tearing down. So a release that names an owner may free an auto-assigned address and never an EIP, which is enforced in each of the three allocators. DHCP needed the scope carried on the wire, since it releases by address and resolves the lease afterwards. Absent on an older caller, which reads as false and keeps the previous behaviour. Worst on the two providers where it is not recoverable: on DHCP the lease is the only thing holding the address and the server promises nothing about handing it back, and on OCI the address is a reserved public IP object that the release destroys outright. * test(e2e): prove an Elastic IP survives a stop and start Instance teardown releases the public address by value, and since the association writes the EIP onto the instance record, that release aims straight at the customer's Elastic IP. The allocator guards now refuse it, but nothing end to end proved the address comes back on a restart. The step goes inside EIPFlip, which already runs last on the shared public guest for the same reason a stop is safe there: the address the guest returns on is the allocation's, not the pool's. It runs on the static cells and the DHCP ones alike, and on DHCP it doubles as the only outside view of the lease still renewing while the guest is down. * feat: release an auto-assigned public address on stop, take a new one on start AWS lends an instance a public address for as long as it runs and gives it a different one on the next start. Spinifex kept the address across a stop, which holds a pool slot for a stopped instance and, on a DHCP pool, keeps renewing a lease nothing is using. A stop now offers the address back through the instance cleaner, which refuses anything an Elastic IP is behind — the EIP association record answers that, the same question an ENI delete already asks — and records on the VM that the start owes it a new one. A host drain keeps the address: Restore relaunches the guest on the next boot, and changing its address under it is an outage nobody asked for. The start allocates before the launch, so an exhausted pool refuses the start rather than booting a guest the customer cannot reach, adopts an Elastic IP associated while the instance was stopped instead of assigning a second address beside it, and hands the address back if the launch then fails. e2e: a StopStartAddress stage on the public guest asserts a stopped instance reports no address and comes back on a different one from a static pool, and a StopStartKeepsTheEIP step asserts the opposite for an Elastic IP. * docs: restructure the OCI guide around two deployment paths, with diagrams The guide had grown into a numbered wall with no shape: no frontmatter, no contents, and a reader with an OCI account had no way to tell where to start. It now follows the multi-node install template — overview, prerequisites, then the deployment path — and splits that path in two: Terraform for one node or three, which is the preferred route, and a manual single-node walkthrough for an instance that already exists. Both meet at one shared configuration section rather than repeating it. Two ASCII diagrams carry what the prose could not: one OCI instance with the whole AWS surface inside it, and three nodes with the same surface distributed, showing which plane each VNIC carries and how an address follows a guest that moves. The limits section was materially out of date and said so in the places that mattered most. Multi-node is no longer experimental, public addresses are no longer pinned to one node — each allocates on its own VNIC and the affinity pass moves the private half to wherever the guest is — and an auto-assigned address is now returned to OCI on stop, so a stopped fleet stops paying for addresses. What changed is recorded rather than quietly edited. The Terraform configuration moves in from the monorepo as scripts/terraform/oci-spx, so the guide points at something in this repository. Also marks the two new lifecycle test files in-package, with the reason each one needs to be: both substitute unexported dependencies. * docs: rework the OCI guide — SVG diagrams, Terraform variables, iSCSI and br-wan Replaces the two ASCII diagrams with SVGs in the house style, one for a single node and one for three. Renames the two deployment sections to Terraform deployment and Manual deployment, documents what the Terraform builds per node and every variable worth setting, and explains that the data volume is an iSCSI attachment rather than a disk. The manual path now covers the iSCSI login, the fstab entry and the netplan source routing that Terraform writes, and states what br-wan is for on a routed-NAT node: it names the VNIC for the OCI allocator and holds the registered addresses, but is never bridged into OVS. Opens with what Spinifex is and why OCI is worth running it on. * fix: make gateway convergence work on a multi-node routed-NAT cluster Three defects each stopped a multi-node cluster from ever attaching an internet gateway, and all three produced the same uninformative symptom: `terraform apply` sitting on `aws_internet_gateway.igw: Still creating...` forever while every service reported healthy. ovn-northd was given only its local OVSDB socket as a client connection. A RAFT follower does not forward writes -- it answers "not cluster leader, trying another server" -- so with one entry in the list, northd stops translating Northbound intent into Southbound flows the moment database leadership moves to another node. Measured on the OCI cluster: SB_Global.nb_cfg seventeen transactions behind NB_Global.nb_cfg, with no chassisredirect port for any VPC created after leadership moved. setup-ovn.sh wrote the member list only when the caller passed --ovn-remote, which only install-node.sh did; it now derives the list from the live RAFT membership otherwise, and says so when it finds a single member, because that node needs a second pass once the others join. The gateway datapath probe ran wherever the reconcile lease landed rather than on the chassis owning the port. Both probes leave from the local uplink, and under routed NAT the transit subnet exists identically on every node, so a remote probe reported a healthy gateway as dead and aimed a repair at a datapath the node does not own. GatewayPortLocal gates it. On OCI the probe then ARPed the tenant-visible public IP, which the cloud NATs and which appears on no interface anywhere. The reconciler now maps the address through the same DatapathResolver the NAT rules are built from. Also adds DescribeVpcEndpoints, which the AWS provider calls on every VPC delete to sweep GuardDuty-managed endpoints; an InvalidAction there failed the destroy and stranded the VPC. The OCI guide gains a Terraform section recording which shipped workbooks were validated against a three-node OCI cluster and which are still open. * docs: record the validated Terraform workbook results for the three-node OCI cluster Six of the seven workbooks run against the OCI cluster pass, including the three the section exists to prove: RDS, ECS and EKS. Records nginx-alb and the clean eks-quickstart re-run, and names the two lifecycle gaps that bit during the run so an operator recognises them rather than reading them as a broken workbook. * fix: route the NAT gateway's SNAT source through the datapath resolver AddNATGateway was the one NAT path that did not map its external address to the one that rides the wire. Where the public half of an allocation is NAT'd by the cloud it is on no interface anywhere, so every private-subnet guest egressed with a source that was dropped, while the gateway reported available and DNS kept working because the VPC resolver is host-local. The unmapped address also failed the host EIP sweep, which works in resolved addresses, so the bind was pruned as foreign on the next pass and reinstalled on the one after. The public half now survives as an external ID instead. Also restructures the OCI guide around numbered steps through to running Terraform, and moves the cloud-init and metadata sections into troubleshooting where they are read when something is wrong rather than followed when it is not. * fix: place a NAT gateway's OCI address on the node that forwards for it OCI drops a packet whose source is not a registered private IP on the VNIC it leaves from, so a NAT gateway's SNAT source has to be on the gateway chassis. Two independent placements decided those and nothing reconciled them: the address was created by whichever worker served AllocateAddress, while the chassis is a fixed ordering the reconcile pass asserts. On a three-node cluster that made private-subnet egress a coin flip, and the same Terraform passed or died depending on which node answered. The affinity pass already solves this for guests, by asking local OVS where the tap is. A NAT gateway has no tap, and it skipped anything with no ENI — which is exactly and only this address. The equivalent local fact is the VPC's chassisredirect binding, so ownership now has two answers rather than one, and an address that is neither a guest's nor a gateway's is still left alone. The VPC is recorded from the reconcile pass rather than from CreateNatGateway, because the answer is the gateway chassis and reconcile is what decides it. * fix: plumb a NAT gateway's host ingress only on its gateway chassis A NAT gateway's address has no ENI, so neither the bind's locality test nor the prune's could ever fire for it and every node installed its /32 route into OVN. Latent on bare metal, where pool addresses are disjoint from the node subnet; on a cloud that hands out the address from the nodes' own subnet it outranks the connected route and swallows node-to-node traffic. Give that address the same second authority the ocinet affinity pass has: the VPC's chassisredirect binding, read from the local ovn-controller. An unanswerable question still leaves the binding alone, so a single node whose gateway port is not yet claimed behaves exactly as before. The orphan EIP prune built its live set from ENI and user EIPs alone while also sweeping host ingress, so every drift pass tore down the ingress of every live NAT gateway and left it dark until the next host EIP tick. * fix: ask the gateway chassis only about a NAT gateway's address gatewayElsewhere skipped the NAT gateway marker it was named for and answered for every spec, so on a node that is not the VPC's gateway chassis every EIP — including a guest's, whose locality is its own port — was left unplumbed. Two of three nodes bound nothing. * test: assert the two private-subnet EKS workbooks in the tofu harness eks-https-ingress and eks-gitops-argocd put their workers in private subnets behind a NAT gateway, so a worker only registers Ready if its egress to the cluster endpoint and to ECR survives that path. That makes them the workbooks that catch a NAT gateway fault, and neither had an assertion, so neither could be run at all — run_workbook calls assert_<name> by construction. Both share eks-quickstart's readiness check, extracted unchanged. They also name the worker size node_instance_type rather than instance_type, and tofu rejects a -var for an undeclared variable outright, so the name is read from the config instead of kept in a list here. * test: count agents, not every node, when asserting EKS workers Ready The control plane is three K3s servers, so an unfiltered node count reaches a one-worker cluster's desired size before any worker joins — the assertion passed on exactly the failure it exists to catch, reporting "4/1 workers Ready". Filtering on the control-plane role label makes it 1/1, which is the number that says a node-group worker registered. Matters most for the two private-subnet workbooks, where the worker joining is the whole proof that egress through the NAT gateway works. * docs: record the two private-subnet EKS workbooks passing on OCI	2026-09-26 14:31:07 -05:00
Josh Siviour	e54b79b7bf	fix: harden ochre ingest progress, job cancel, and retrieve error typing (#893) ... Persist ingestion progress and bound per-document work with a timeout so a job on a dead backend fails fast and visibly instead of sitting silently IN_PROGRESS; add StopIngestionJob so a stuck job can be cancelled; map a backend-down Retrieve/RetrieveAndGenerate error to 503 instead of an opaque 500.	2026-08-26 23:01:57 +10:00
Josh Siviour	bc8ad05a0f	feat: Ochre Phase 5 knowledge bases (bedrock-agent KB/data-source + RAG) (#792) ... * feat: ochre-vector list-jobs subject Adds ochre.vector.listJobs to the daemon-side VectorService surface (both the in-process and NATS-forwarding implementations) so a caller can list every ingestion job for an account rather than looking one up by id. Widens jobGetter to expose JobStore.List, registers the subject on the daemon's post-connect subscription table, and updates the CLI test double to satisfy the widened VectorService interface. Also carries the jackc/puddle/v2 indirect dependency go.sum was missing. * feat: bedrock-agent knowledge-base + data-source CRUD Adds the emulated AWS bedrock-agent control-plane gateway service: KnowledgeBase/DataSource CRUD, StartIngestionJob/GetIngestionJob/ ListIngestionJobs, backed by new gateway-owned KBStore/DataSourceStore JetStream KV persistence (mirroring registry.go's CAS idiom) and the existing ochre-vector daemon surface for the bound vector index. One knowledge base binds exactly one vector index. embeddingModelArn maps to EmbeddingModel/Dimension; storageConfiguration and roleArn are accepted and stored verbatim but never acted on. CreateDataSource maps S3 + fixed-size chunking config onto a SourceSpec, converting AWS's maxTokens/overlapPercentage into internal ChunkSize/ChunkOverlap. DeleteKnowledgeBase cascades to every data source under it before dropping the bound index. Wires BedrockAgentKB/BedrockAgentDataSources/ BedrockAgentVector into the gateway and registers bedrock-agent in the supported-services and dispatch tables. Runtime (Retrieve/RetrieveAndGenerate) is out of scope here. * test: cover bedrock-agent routing and KB/DataSource store edges Adds end-to-end HTTP tests for BedrockAgent_Request's route table and per-route handler closures (all eleven operations, plus unknown-route/ missing-account/nil-dependency error paths), KBStore/DataSourceStore PurgeAll and no-JetStream-client error paths, and ListJobs coverage (vectorService, NATS round trip, per-account scoping) in the existing ochre-vector API test suite. * feat: bedrock-agent-runtime retrieve + retrieve-and-generate Adds the emulated bedrock-agent-runtime data plane: Retrieve resolves a knowledge base to its bound vector index and runs a similarity query; RetrieveAndGenerate runs the same retrieval, renders a RAG prompt (default or caller-supplied template), and calls Converse in-process for generation, returning citations built from the retrieved chunks and a session id. Recovers the full RetrievalFilter operator set from the raw request body since the vendored aws-sdk-go RetrievalFilter type only carries andAll/orAll, and fixes GetService never routing real signed bedrock-agent traffic (only bedrock-runtime and bedrock had discriminator rules; bedrock-agent's own knowledgebases paths had none). * fix: bedrock-agent KB delete idempotency + ingestion-job ownership CreateKnowledgeBase: a rollback DeleteIndex failure after a kb.Create claim failure is now logged at Error level with the orphaned index id instead of a bare Warn, while the caller still sees kb.Create's own error unchanged. DeleteKnowledgeBase: a bound index that DeleteIndex reports already gone (ErrIndexNotFound) no longer aborts before the KB record itself is removed, so a KB never dangles pointing at an index that no longer exists. Any other DeleteIndex error still aborts the cascade. GetIngestionJob: now verifies the addressed dataSourceId owns the job (Bucket+Prefix match against the data source's own SourceSpec, same best-effort check ListIngestionJobs already used), not just the knowledge base's bound index -- closing a path where a job could be read through the wrong data source id. * fix: default bedrock-agent-runtime retrieve to 5 results (AWS parity) * fix: thread DataSourceID through ingestion jobs JobRecord and IngestRequest now carry an exact DataSourceID instead of relying on Bucket+Prefix as a best-effort proxy. StartIngestionJob stamps it from the DataSource record; GetIngestionJob and ListIngestionJobs filter on it exactly, treating a job with no DataSourceID as belonging to no bedrock-agent data source. * fix: purge KB/DataSource records on appliance teardown Appliance.Teardown purged the index registry and job store but left the gateway-owned KB/DataSource KV records dangling, pointing at a deleted index. WithKBStores attaches those stores the same way WithStores does for registry/jobs, built before Connect so teardown still works on a broken appliance, and Teardown purges them too.	2026-08-15 12:18:59 +10:00
Josh Siviour	d52245001e	feat: Ochre vector store (tenant pgvector, embeddings, ingestion, operator teardown) (#788) ... * feat: add /v1/embeddings gateway adapter * feat: Ochre vector store pgvector backend, KV index registry, orchestration Adds handlers_ochrevector: the VectorBackend interface + pgx/v5 impl with per-account schema/role isolation enforced via SET LOCAL ROLE + search_path inside every transaction, the KV index registry (CREATING/READY/DELETING/STALE), and the create/delete/list/reconcile orchestration tying them together with crash-safe rollback and re-entrant recovery of stuck records. * feat: singleton appliance-ensure state machine for ochre vector store Adds the encrypted control-plane credential store and re-entrant Ensure state machine for the platform pgvector appliance (D2): a fixed-key KV claim mirroring Registry.Reserve elects exactly one winner to launch the appliance via an injected ApplianceLauncher, losers wait or resume a stale crashed provision, and the master password is generated once, stored only AES-256-GCM encrypted, and decrypted solely inside Connect to build a pgxBackend. * feat: Ochre vector store async ingestion (chunker, job store, worker) Adds Stage 3: a deterministic recursive chunk splitter, a per-account ingestion job-record store mirroring the index registry, the transactional per-document replace on the backend, and the IngestService worker that lists a predastore source, chunks and embeds each object, and replaces its rows idempotently. * feat: Ochre vector store similarity query and metadata filter grammar Adds VectorBackend.Query (cosine kNN, k default/cap, iterative HNSW scan under a filter) and a Bedrock-KB-shaped metadata Filter AST that compiles to parameterized SQL over jsonb, closing off arbitrary SQL by allowlisting jsonb keys and always binding values as params. Also fixes RunJob reporting READY on a job that ingested nothing regardless of failure cause, and stamps SourceSpec.Metadata onto ingested rows so the new filters have data to match against. * feat: Ochre vector store request/response API, VectorService, and CLI Adds the daemon-facing VectorService interface (index CRUD, ingest, describe job, query), its server implementation over the existing registry/backend/embedder, a NATS-forwarding client, and the `spx admin ochre vector` operator CLI. Query resolves and embeds against each index's own pinned model; ingest stamps the same model/dimension onto every source spec server-side. accountID is always taken from the caller context, never the request payload. Daemon subscription wiring and manifest registration are a later change; this is pure request/response plumbing with no construction into daemon.go or awsgw.go. * feat: wire Ochre vector store VectorService into the daemon Constructs the platform Postgres appliance and VectorService behind a new OchreVector config section that defaults off, so existing daemons are unaffected. Adds a real RDS-backed ApplianceLauncher that provisions the appliance over rds.CreateDBInstance/DescribeDBInstances, registers the six ochre.vector.* NATS subjects mirroring the bedrock endpoint service pattern, and degrades to a disabled (nil) service on any boot-time appliance failure rather than blocking daemon startup. * fix: fix Ochre vector store boot ordering and self-register subjects startOchreVector ran before subscribeAll, so its appliance launch fired rds.CreateDBInstance/DescribeDBInstances requests with no responder yet registered, timing out the launch on a first-enable boot. It also ran synchronously, blocking daemon startup up to the appliance launch timeout. Move it to a background goroutine started after subscribeAll returns, and have it register its own ochre.vector.* subjects once the VectorService is actually constructed, via a new registerNatsSubs helper shared with subscribeAll's boot-time table. Guard d.natsSubscriptions with d.mu, since this background registration can now race the shutdown path's unsubscribe-all sweep. * feat: bake pgvector into the shared rds-postgres image * fix: reach the ochre appliance over a routed daemon port on its ENI IP The daemon dialled the RDS appliance over rec.Endpoint, a vanity spx3.net hostname the control-plane host can neither resolve nor route to, so a healthy appliance was never reachable and the vector store disabled itself. Resolve the appliance's customer-ENI private IP read-only via ec2.DescribeNetworkInterfaces filtered on the spinifex:rds-db-instance tag, dial that IP for the DSN instead of the hostname, and give the daemon its own routed ENI in the appliance's subnet bound to an OVS host port so the IP is reachable. Retry appliance startup with bounded backoff instead of hard-disabling on the first transient. * fix: install pgvector at bootstrap for the Ochre vector appliance The daemon connects to the appliance as its NOSUPERUSER master, and the image's pgvector is not a trusted extension, so CREATE EXTENSION vector failed with permission denied. rds-init now installs it via the bootstrap superuser, gated on the appliance's fixed master name so tenant instances are unaffected. * fix: grant the Ochre appliance master CREATE on its database The daemon's EnsureAccount runs CREATE SCHEMA per account as the NOSUPERUSER master, which does not own the postgres database, so bootstrap now grants it CREATE on the database alongside the pgvector install. Both are gated to the appliance master, so tenant instances are unaffected. * fix: make the Ochre appliance's pgvector reachable under per-account isolation The daemon connects as a NOSUPERUSER master and runs every account operation under SET ROLE with a search_path pinned to the account schema. Two gaps blocked that against a real non-superuser Postgres: the master could not SET ROLE into the roles it creates, and the account role could not see the vector type. Bootstrap now grants the appliance master createrole_self_grant and installs pgvector into a read-only extensions schema granted to PUBLIC; the backend carries that schema on each account search_path so the vector type resolves without weakening per-account write isolation. All appliance-scoped. * fix: prune stale VPC host port before installing the daemon appliance port A replaced Ochre appliance is handed the same subnet IP, so a new daemon ENI mints a fresh vhp- port at an address a superseded port still holds. The old port survives at the OS/OVS layer even after its ENI record is wiped, so two interfaces claim one address and the kernel can route to the dead one (EHOSTUNREACH). installVPCHostPort now prunes any other vhp- port holding the target address before installing. * fix: gate VM restore's stale-socket kill on backing-store readiness On daemon start, a live VM whose NBD sockets fail to dial was force-killed as an orphan. While predastore/viperblock is still coming up after a restart those sockets are transiently unreachable, so a healthy VM (the Ochre appliance was the observed victim) was destroyed and its RDS record flipped to failed. The kill now only fires when the backing store is confirmed up; otherwise the live QEMU is kept and QMP reconnected. Genuine orphan reaping under a healthy store is unchanged. * fix: drive ochre ingest jobs from a daemon-side sweep scheduler * feat: add operator teardown verb for the ochre platform appliance Adds Appliance.Teardown to remove the singleton platform Postgres appliance (RDS instance, host port, KV record), an admin-only NATS subject and daemon handler to drive it, and `spx admin ochre appliance teardown --confirm` to invoke it without exposing the action through the tenant awsgw surface. * fix: register ochre appliance teardown before Connect so it survives a broken appliance * fix: purge ochre index and job metadata on appliance teardown * test: mark ochre in-package test files for the external-test gate	2026-08-13 15:54:48 +10:00
Julian Sommer	5e17169661	feat(e2e): complete the RDS end-to-end suite (#608)	2026-08-03 15:38:07 +10:00
Josh Siviour	eff11b611b	fix: stop upstream DHCP leases leaking out of the external pool (#602) ... * fix(dhcp): release stranded leases and label them with the owning node Two paths dropped a lease from KV without returning it to the upstream server, so a source="dhcp" pool leaked addresses permanently. Manager.Start deleted an expired entry outright. Dropping the entry alone abandons the binding rather than returning it, and a server that ages leases by liveness holds the address for as long as anything answers ARP for it. Adoption now sends a bounded, best-effort RELEASE first, skipping leases with no stored offer/ack because nclient4 cannot rebuild those. Release-by-IP answered SUCCESS when the IP was absent from KV, telling the caller the address was freed when nothing went on the wire. It now returns an error. Every ReleaseIP call site logs and continues, so this surfaces the strand without wedging teardown. Leases were also unattributable upstream: chaddr is a derived HashMAC and option 61 carries that same MAC, so no field identified the host that took the lease. Option 12 is now "<node>-<client-id>" and option 60 is "mulga-spinifex/<node>", giving the lease table a per-lease identity and a per-host column to group on. HashMAC is untouched — encoding the node there would cut per-node entropy and reverse a documented decision. Observed live: 20 office-LAN addresses held by one node's orphaned OVN gateway LRPs, with no lease records and no renewals. * fix: rebind the datapath when a DHCP lease moves to a new address A lease re-issued on a different IP was persisted to KV with no notification of any kind. doAcquire replaced the lease after an expiry and doRenew accepted whatever a rebind answered with, while the gateway LRP's address is written once at attach and never revisited. Each such event burned one address permanently: the old one squatted by OVN's ARP responder with no renewer and no DHCPRELEASE, the new one renewed forever attached to nothing. Route both paths through applyLease, which releases the superseded address and invokes an IPChangeHook so the datapath can follow. vpcd registers a hook that moves the gateway LRP's networks and gateway_ip stamp, re-runs the routed leg so the VPC SNAT and host ingress route follow, and waits on the flows barrier. EIP, ENI-public and NATGW addresses are recorded outside vpcd in DescribeAddresses and DescribeInstances, so the hook reports those rather than repointing a datapath the control plane no longer agrees with. The release half still applies to every purpose, so the leak itself is closed either way. A failed rebind does not fail the lease loop: the new address is held regardless, and dropping it would strand both. * fix: reconcile API-visible records when their DHCP lease moves An EIP or auto-assigned public IP whose lease was re-issued elsewhere left DescribeAddresses and DescribeNetworkInterfaces advertising an address vpcd had already released. The datapath kept the old dnat_and_snat too, so the address was dead on both sides while the upstream server was free to hand it to someone else. vpcd cannot fix this alone: it holds the lease, but the records naming those addresses live daemon-side. Add vpc.dhcp.lease-changed, which vpcd requests and the daemon serves, dispatching to the service owning the moved address. RebindPublicIP and RebindENIPublicIP commit the new external IP through vpc.add-nat first — AddEIP scrubs the predecessor row keyed on the logical IP, so one commit is enough — and only then move the record, so a NAT failure never leaves a record advertising an address that does not route. Both refuse to move a record naming neither address, so a stale request cannot overwrite an allocation the lease does not own, and both are idempotent under duplicate delivery. A failed or unanswered reconcile propagates back to the lease loop rather than reading as success. vpc.dhcp.lease-expired, listed in the service manifest but never implemented, is replaced by the subject that now exists. * fix: dedupe retried AllocateAddress so one call takes one address EC2 AllocateAddress carries no ClientToken, and vpcd's DORA ladder can outlast the AWS client read timeout, so every SDK-level retry minted a fresh allocation ID and drew another upstream DHCP lease. The caller kept one EIP and the rest stayed leased with no record naming them, so nothing ever released them. Retries are now identified by the SDK's amz-sdk-invocation-id, which is stable across attempts of one call. The gateway lifts it off the request, NATSRequest forwards it as a header, and the daemon puts it back on the context, so the EIP service sees the same token every attempt. singleflight joins a retry to the call still in flight — the common case — and a TTL'd KV record answers retries that arrive after the first call completed. * fix: reclaim DHCP leases whose owner is gone and stop dropping expired ones Three leaks that all end the same way: an address stays leased upstream with nothing renewing it on purpose and nothing that will ever release it. Manager.Start deleted any lease that expired while vpcd was down, after releasing it. The resource it was taken for still existed, so the datapath kept forwarding on an address the upstream server now considers free — a duplicate-IP conflict waiting for the server to reissue it — and nothing ever re-acquired. Expired leases now get a renewal loop like any other, which re-DORAs on its first pass; the existing IP-change hook rebinds the datapath when the new address differs. The reaffirm RENEW is skipped for these, since a lease the server has already aged out will only fail it. Nothing reconciled leases against the resources that justify them, so a resource deleted out from under its lease left the address renewed forever. ReapOrphans sweeps them: vpcd asks the daemon over vpc.dhcp.owner-check whether the EIP, ENI or VPC still exists, and releases the lease when it does not. A gateway lease also has a router port to remove, so DetachIGW runs before the address goes back — releasing it while OVN still answers ARP for it invites the same conflict. Anything the daemon cannot resolve is reported unknown and kept: an unreachable daemon must never read as a fleet-wide deletion. The acquire ladder ran to ~62s, just past the 60s read timeout botocore defaults to, which is what turned one AllocateAddress into two leases. It now totals 40s with a 45s budget, and the DORA RPC 50s, so the caller hears the outcome instead of giving up and retrying. This is a mitigation, not a fix, for callers that send no retry token: botocore emits no amz-sdk-invocation-id, so the dedupe added earlier cannot see their retries at all. Fixes mulga-j19gj, mulga-9exe6, mulga-7qyzj. * feat: add spx get dhcp-leases so held external addresses are visible The upstream leases behind a source="dhcp" external pool lived in a per-AZ KV bucket with no read path, so working out which addresses were held, by what, and whether the owner still existed meant writing a NATS client or reading journals. That was the main obstacle to diagnosing every leak in this series. The command lists each lease with its purpose, VPC, address, MAC, holding node and remaining time, and resolves the owner over vpc.dhcp.owner-check so an orphan is visible as such. --orphans narrows the output to leaks; --skip-owner-check drops the lookup when only the addresses matter. Fixes mulga-655iu. * fix(vpcd): reconcile gateway ports onto their lease address at startup A gateway LRP configured from a static pool keeps its address when the pool is switched to source="dhcp", so the VPC forwards on an address nothing leases while the lease it holds goes unused. No lease event ever fires to correct it. vpcd now sweeps every gw-lrp lease on startup and rebinds the port to the address the lease actually holds. RebindGatewayIP is idempotent, so a port already in agreement costs one OVSDB read. * test: cover dhcp lease owner-check handlers and the lease CLI The owner-check path decides whether the reaper releases an address, so every inconclusive branch is asserted to report unknown rather than gone. * fix(dhcp): make the outcome of a stale lease release observable A stale RELEASE was logged only when the transport failed. A lease with no raw offer/ack was skipped in silence, and a RELEASE the server ignored looked identical to one it honoured, so an address left bound upstream could not be traced to either cause. The skip now warns, and a successful release logs the chaddr it used, which is what a server matches a RELEASE against. * fix(dhcp): take the RELEASE chaddr from the lease being released A server matches a RELEASE to a binding on chaddr, and the release path re-resolved it live. For a UseIfaceMAC lease whose interface MAC had changed since, that sends an address the binding never had: the server drops the RELEASE and says nothing, so the address stays held with no renewer. The chaddr now comes from the lease record, which carries what took the lease out, falling back to live resolution only when the record has none.	2026-07-30 13:29:19 +10:00
Julian Sommer	2569601e3d	feat: implement RDS CreateDBInstance, reconciler and endpoint DNS (#594)	2026-07-29 11:32:59 +10:00
Josh Siviour	cee73671ff	feat(bedrock): Ochre runtime MVP — Converse over self-host + Anthropic-direct (#520) ... * feat(bedrock): Ochre runtime MVP — Converse over self-host + Anthropic-direct Add the AWS-Bedrock-compatible gateway services (bedrock, bedrock-runtime) following the EKS REST-JSON pattern: ListFoundationModels / GetFoundationModel over a static connectivity-tiered catalog, and non-streaming Converse routed per modelId to one of two provider backends — self-hosted vLLM (OpenAI wire) and Anthropic-direct (Anthropic Messages wire). Per-account provider API keys are a master-key-encrypted JetStream KV secret with an optional platform default; the advertised catalog gates external models on a resolvable credential. Self-host endpoints are pinned via config (OCHRE_VLLM_ENDPOINTS) behind an EndpointResolver seam. Adds gateway/bedrock/ (catalog, provider router, Anthropic + vLLM adapters, credential store), awserrors REST-JSON codes, and the bedrock E2E suite (list/get light; Converse gated on live backends). Daemon/NATS control plane and dynamic endpoint lifecycle are deferred to Phase 3. * test(bedrock): cover dispatch, router, credentials KV + endpoint config Add unit coverage for the Ochre Phase 1 diff: Bedrock_Request / BedrockRuntime_Request end-to-end dispatch, provider router, JSON response/error writers, upstream-status mapping, the credential-store KV round-trip, the Anthropic bound-provider path, and parseBedrockEndpoints. Lifts PR diff coverage above the 60% gate.	2026-07-13 21:40:34 +10:00
Josh Siviour	e1c4f85a36	feat: routed-NAT external mode with two-tier ingress (#475) ... * feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper * feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode * feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules) * feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode * feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks * feat(scripts): add --nat-uplink routed mode to setup-ovn.sh * test(e2e): add routed-NAT single-node natuplink suite Covers the external_mode=nat lane end to end on a nat-initialized node: host transit wiring (veth, br-ext port, ip_forward, iptables egress rules), spinifex.toml contents, disabled EIP surface (empty describe-addresses, UnsupportedOperation on allocate), default subnet MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit /24, guest outbound egress proven via a serial-console probe (nat mode has no inbound path), and unique transit gateway IPs across VPCs. Suite self-gates on external_mode=nat; registered as e2e-natuplink and in SUITES_BUILD so it compiles in CI. Manifest entry stays commented until CI can provision a nat-mode environment. * feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column Adds the exempted_ext_ips column (optional strong Address_Set ref) to the nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an in-place SetNATExemptedExtIPs update on the client interface, live client, and mock. Test NB schema gains the column and covers insert-resolves-UUID, converge, clear, and missing-NAT paths. * feat(policy): exempt address set on routed SNAT/EIP rows Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR) with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip path patch the ref in place on pre-existing rows. The set is a singleton strong ref and is never deleted. Non-routed modes are unchanged. * feat(host): VPC ingress route primitives for routed-NAT mode EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the spx-nat veth. Replace/del semantics keep both idempotent; a missing route on removal is not an error. * feat(external): routed ingress leg on IGW attach/detach, reconcile-safe AttachIGW's routed leg now also installs the host VPC ingress route via RoutedIngressHooks, and re-runs on the already-attached path so the reconcile loop restores host routes (and patches legacy SNAT exempt refs) after a reboot — OVN state survives restarts but host routes do not. DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs, validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct pass nil hooks and are unchanged. * test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row pointing at spinifex_nat_exempt (containing the transit /24), and — after opening the SG from the transit net (default-closed, AWS parity) — ping and an sshd TCP handshake against the instance private IP from the host, which proves the un-SNATted return path. Phase 8 additionally waits for the second VPC's ingress route to appear on IGW attach. vpc-networking README documents the jumpbox pattern, SG note, and nat_exempt_cidrs. * feat(config,admin): public wan pool alongside nat-transit Restructure admin config generation from single-pool scalars to a Pools slice so nat mode can emit both the transit pool and an optional public wan pool. nat mode now accepts --external-pool/--external-source (plus --external-bind-bridge/--gateway-ip when a public pool is requested), validated by the shared resolvePublicPoolFlags helper also used by pool mode. Transit pool always renders first so the IGW allocator picks it. Formation wire format unchanged (nat mode is single-node only). * feat(daemon): enable EIP service and auto-assign in nat mode with public pool External IPAM (and with it the real EIP service, RunInstances public IP auto-assign, and MapPublicIpOnLaunch on default subnets) now activates whenever public pools exist: pool mode as before, plus nat mode when a wan pool rides alongside the transit segment. The transit pool is filtered out of IPAM — its addresses are gateway-LRP plumbing, never allocatable EIPs. Transit-only nat configs keep the disabled EIP stub. * feat(host): per-EIP ingress primitives for routed-NAT mode DetectUplinkFor parses ip route get to find the interface and source IP facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move). EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC gateway LRP transit IP (src-hinted so host-local traffic still DNATs), answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC, WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated. * feat(policy,vpcd): host EIP binder option on NATManager WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds after priming on both the fresh-add and idempotent-skip paths (host routes and proxy-ARP are volatile even when the OVN row survives, so reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort. The next hop comes from the new gatewayPortIP helper reading the gateway LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID key now an exported policy const shared with the external package. vpcd selects the IGW pool by name instead of index — with two pools in nat mode the gateway allocator must stay on the transit pool — and wires the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced pools (gateway unknown until lease) proxy-ARP on their bind bridge. * feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection dhcp_mac = "derived" (default) | "interface" on external pools. The interface variant leases with the bind interface's own MAC — WiFi/WWAN uplinks drop frames with foreign source MACs, so derived chaddr MACs never complete DORA there. Option 61 stays the lease discriminator. UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and persists through KV so restart re-acquires repeat it. Routers that key leases on MAC and ignore client-id would hand every lease the same IP; the manager detects an ACK matching the interface's own address or another client-id's live lease, releases the colliding lease, and hard-errors advising a static range instead. admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp pools bound to non-bridgeable uplinks (wlan/wwan/ppp). * test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs The natuplink suite now detects a public (non-transit) pool in spinifex.toml and switches lanes: Tier-1-only configs keep the EIP-disabled and no-public-IP assertions; public-pool configs assert the EIP surface is enabled, the default subnet has MapPublicIpOnLaunch=true, instances auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink, per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the EIP, reconcile replay across a vpcd restart, and teardown on disassociate. Docs: vpc-networking README gains a public-pool section for routed-NAT nodes (init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool. * fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd Routed-NAT mode shells out to iptables for the transit masquerade and per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for host-delivered EIPs. Without these sudoers entries vpcd crash-loops with "command not allowed" on nat-mode nodes. * fix(ovn): scope DeleteStaticRoute prefix match to the target router Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache match could grab another router's row and fail with a referential integrity violation, wedging IGW attach on multi-VPC nodes. Match only rows referenced by the target router, mirroring FindStaticRoute. Mock updated to the same semantics, with a regression test covering drift replacement across two routers. * fix(reconcile): always replay AttachIGW so host ingress state is re-ensured applyIGWs skipped AttachIGW whenever the gateway switch port existed in actual state, so the already-attached re-ensure path never ran and routed-NAT host ingress routes lost on reboot were never reinstalled. AttachIGW is idempotent; run it unconditionally. * fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes collide on the single host route table and the last IGW attach silently stole the route (route replace). Pass vpcID and the gateway LRP IP through RoutedIngressHooks; the bootstrap VPC always installs its route, other VPCs only install when the CIDR is free and only remove a route their own gateway IP holds. Adds host.VPCIngressRouteVia to read the current route holder. * ci(e2e): add nightly cell 19 for routed-NAT single-node suite Cell 19 bootstraps with --external-mode=nat plus the CI static pool as the public wan pool and runs the natuplink suite, covering the Tier-1 transit datapath and Tier-2 EIP ingress. The run step exits 0 when natuplink.test is absent from the tests tar so scheduled runs against branches predating the suite stay green. * fix(admin): nat-mode init summary reflects configured public pool The summary printed 'outbound-only — no public IPs/EIPs' even when --external-pool/--external-source configured a public wan pool. Print the pool range (or dhcp bind) and note EIPs are enabled. * fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites Reconcile the routed-NAT branch with dev's ctx-first service APIs after rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService interface, the vm_adapters public-IP resolver asserts the ctx-signed AssociatedPublicIPForInstance, and the default-subnet mapping test threads ctx into DescribeSubnets.	2026-07-09 09:43:32 +10:00
Josh Siviour	47c9808a96	test(e2e): add ECS end-to-end suite (awsvpc, bridge, host, ELB service) (#439) ... Drives the ECS data plane against the local awsgw: a customer VPC plus a real container instance launched from the spinifex-ecs-node AMI, then standalone tasks in all three network modes and an awsvpc service fronted by an ALB target group. Adds an ECS client to the harness, a Makefile target, and a service-interfaces suite entry. bridge/host tasks share the host netns where the agent credential endpoint owns 169.254.170.2:80, so they run a no-port command and assert placement + mode; awsvpc and the ELB service prove real :80 serving. waitTaskRunning re-checks after a settle so a transient RUNNING cannot pass.	2026-06-29 14:16:23 +10:00
Josh Siviour	f9858d4e5d	test(e2e): ECR end-to-end suite (control plane + docker/crane/skopeo) (#398) ... * test(e2e): add ECR end-to-end suite (control plane + docker/crane/skopeo) New tests/e2e/ecr/ suite plus harness support: - harness: ECR SDK client, control-plane helpers (repo CRUD, get-login-password, describe-images), and docker/crane/skopeo CLI wrappers that skip when the binary or registry-host resolution is absent. - suite: control-plane CRUD + policy + lifecycle round-trip; docker login/push/ pull; crane append/pull and skopeo copy exercising the Bearer /v2/token flow. - registered e2e-ecr in the Makefile + service manifest. Predastore capacity/load leg is a skipped placeholder pending mulga-siv-359/356. * test(e2e): carry awsgw port in ECR registry host for dev clusters Dev clusters bind awsgw on a non-standard port (e.g. 9999); production uses :443. ECRRegistryHost now appends SPINIFEX_ECR_PORT when set so docker/crane/ skopeo address the right endpoint, and RequireRegistryResolves strips the port before DNS lookup.	2026-06-19 11:34:38 +10:00
Julian Sommer	eced3f733a	feat: split IAM e2e (#372)	2026-06-17 15:06:16 +10:00
Josh Siviour	465fa1e6c7	fix: complete at-rest encryption launch + volume lifecycle (#361) ... * chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against the fix for the encrypted AMI snapshot metadata tag-verify failure (cipher: message authentication failed) on instance launch. Pairs with viperblock PR #22. * fix: forward viperblock master key to nbdkit mount path viperblockd loaded the master key but never passed it to the nbdkit plugin it spawns for volume mounts, so encrypted boot volumes opened in cleartext and failed LoadState with an encryption-state mismatch. Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's encryption_key_file= argument when set. * chore: bump viperblock to fresh-volume VolumeUUID mint fix * chore: bump viperblock to snapshot-link LoadState ordering fix * fix: route encrypted-volume config writes through viperblock keyholder config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived nonce can only be advanced by the master-key holder that owns the volume. The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to refuse encrypted volumes (rewriting would strip the tag) — bricking UpdateVolumeState (attach in-use / detach available) and ModifyVolume. Route encrypted-volume config updates through viperblockd, mirroring the ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config queue group opens the detached volume exclusively and reseals. Cleartext volumes keep the direct-S3 fast path. No crypto leaves viperblockd. mulga-siv-323 * fix: unwrap encryption envelope when snapshot handler reads config.json CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into a VBState. For encrypted volumes config.json is an at-rest envelope {payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so CreateSnapshot 500s on the size guard and the in-use scan silently drops encrypted volumes' snapshots. Unwrap via viperblock.StateBody first, mirroring the volume handler's mergeVolumeConfig path. mulga-siv-324 * test: cover viperblockd ebs.config handlers Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler (ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only size reconcile, live-VB apply, invalid JSON, apply errors, and the detached-volume open path. Clears the diff-coverage gate (68.8%). * chore: pin viperblock to dev tip (#22 merged)	2026-06-15 22:43:56 +10:00
Josh Siviour	d076a434d4	Feat/e2e targeted suites (#352) ... * test(e2e): add targeted suite selector + shadow-mode CI step select-suites maps a diff to the e2e suites it must run: changed files → services (manifest path/additional_paths) → consumer closure over depends_on vs publishes/subscribes → suites whose covers intersect. Special cases force-all (infra/harness/workflow, ≥3 services, submodule bumps) or a lone suite dir; empty diffs fall back to default suites. An AST scan of each suite's harness.Ensure* call graph derives per-test service coverage and emits an optional -test.run subset (RUN_PATTERN). Wired into e2e.yml as a non-gating 'Select Suites (shadow)' job that logs the computed selection to the run summary so predicted vs actual can be compared before the selector drives SUITES. * ci(e2e): add dispatch-only per-suite job blocks grouped by env Restructure the looped e2e flow into env-grouped provision -> matrix suites -> teardown blocks so each suite surfaces as its own node in the Actions graph instead of a single green tick: single_provision -> single_suite [single,cert,lb,eks] -> single_teardown multi_provision -> multi_suite [multinode] -> multi_teardown Provision resolves the VM/cluster and ships test binaries, emitting WAN IP + ssh key path as job outputs; each matrix leg runs one suite over SSH and uploads its own junit/analysis artifacts; teardown (if:always) reuses the persisted tofu workspace on the same runner to destroy. workflow_dispatch only -- runs alongside the gating e2e.yml for validation and posts no commit status. * test(e2e): add manifest drift lint (fixture-create + NATS subject) Bead 5 of the targeted-suite-selection plan. Two ratcheted drift guards in tests/e2e/manifest-lint, wired into make preflight and the spinifex.yml PR job: - Fixture lint: AST-flags direct resource-create AWS calls (c.EC2.RunInstances, c.ELBv2.CreateLoadBalancer, ...) in suite test files, steering them through harness.Ensure* fixtures. ~104 pre-existing call sites are captured in a checked-in baseline; the lint fails only on NEW creates. Intentional create-path tests opt out with an inline // e2e:allow-create. - Subject lint: extracts NATS subject literals from the natsSub{} registry and literal Subscribe/Publish/Request args, cross-referenced wildcard-aware against service-interfaces.yaml. Code subjects missing from the manifest fail (ratcheted); manifest subjects with no static reference warn only (many are produced dynamically). Fixed real drift surfaced on first run: declared the five spinifex.cluster.shutdown.* phase subjects under spinifex_daemon. manifest-check is now CI-enforced alongside manifest-lint (previously preflight/local-only). * feat: run-scoped e2e teardown sweep Tag every Ensure* fixture resource with e2e:run=<run-id> when SPINIFEX_E2E_RUN_ID is set, and add a teardown.test suite that sweeps everything carrying that tag at end of run — reclaiming resources leaked by a crashed suite, an interrupted run, or a persistent cluster. - harness/sweep.go: tagRunResources (EC2 CreateTags) + tagRunELB (ELBv2 AddTags) + SweepRunResources in dependency order (LB, instances, NAT, snapshots, volumes, images, SGs, subnets, key pairs). Blank run id no-ops both tagging and sweeping, so the sweep can never touch an untagged or foreign resource. - harness/fixtures.go: stamp the run tag on each created resource. - tests/e2e/teardown: thin suite that runs the sweep; skips without a cluster or run id, logs the report, never fails the run. - e2e-suites.yml: build + ship teardown.test, pass the run id into suite legs, sweep on the VM in both teardown jobs before tofu destroy.	2026-06-11 21:09:50 +10:00
Josh Siviour	38f5dd0d41	feat(acm): minimal ACM-alike certificate service (#285) ... Stand up acm as its own AWS JSON 1.1 gateway service wiring the BYO-cert import/store path ELBv2 HTTPS listeners need. ImportCertificate validates tls.X509KeyPair, parses the leaf for domain/serial/expiry/key-algorithm, mints arn:aws:acm:<region>:<acct>:certificate/<uuid>, and stores PEM (cert, chain, key) in JetStream KV. DescribeCertificate/ListCertificates/ DeleteCertificate round-trip; access is account-scoped. New gateway dispatch by X-Amz-Target (CertificateManager.<Action>); errors share the AWS JSON 1.1 envelope with EKS. Wired through gateway/acm ops, handlers/acm (service iface, store, NATS client), daemon handlers + acm.* subscriptions, and service-interfaces.yaml. Out of scope (future, same epic): RequestCertificate + DNS validation, managed renewal, Get/ExportCertificate, tagging, ACM-PCA.	2026-06-04 10:13:10 +10:00
Josh Siviour	e06d0c0865	feat(elbv2): HTTPS listener certificates + SSL policies (Phase A) (#283) ... * feat(elbv2): HTTPS listener certificates + SSL policies Add Certificates[]/SslPolicy to ListenerRecord. CreateListener and ModifyListener validate secure protocols (HTTPS/TLS) require >=1 certificate with exactly one default, non-secure protocols reject certs/policy, and default+validate the SSL policy. DescribeListeners echoes certificates and SslPolicy. New endpoints: AddListenerCertificates, RemoveListenerCertificates, DescribeListenerCertificates, DescribeSSLPolicies (gateway ops, NATS subjects, daemon handlers, service-interfaces.yaml). No TLS termination yet. * test(elbv2): cover listener-cert gateway ops + handler error paths	2026-06-04 09:51:10 +10:00
Josh Siviour	fd3dc6b680	feat(network): ADR-0006 layer redesign + IPsec/OVN hardening (#255) ... * refactor(vpcd): remove macvlan bridge mode Macvlan was a single-NIC SSH-safe hack that predated veth mode and was never selected by any active config path. Deletion is Phase 0 item 1 of the network layer redesign (mulga-siv-125.1). Removed: - BridgeModeMacvlan const + setMacvlanMAC + ifaceIsMacvlan probe - Macvlan MAC alignment block in launchService (only fired on dead path) - Macvlan branch in detectBridgeMode + matching tests - Macvlan duplicates of IGW/AddNAT centralized-NAT tests (veth versions already cover the same path) - --macvlan flag + macvlan case branch in setup-ovn.sh Refactored: - useCentralizedNAT() now == BridgeModeVeth (was != BridgeModeDirect); verifyBridgeMode already rejects empty/unknown at startup - Three IGW-pool tests now explicitly WithBridgeMode(BridgeModeVeth) (previously relied on empty-mode defaulting to centralized) Kept intentionally: - Deprecation notes in docs/compute/vpc-networking/README.md - Legacy macvlan-interface cleanup in reset-dev-env.sh + ansible teardown role (migration for operator systems with stale spx-ext-* interfaces from pre-veth installs) * feat(vpcd): resolve external CIDR at startup with bounded retry Block service start until the WAN bridge has an IPv4 address, with a 30s retry window. Phase 0 item 2 of the network layer redesign (mulga-siv-125.1). Guards the boot race where vpcd starts before systemd-networkd finishes DHCP on br-wan, or before netplan applies a static config. Skipped when ExternalMode is empty (overlay-only deployments). Forward-compatible with ADR-0006: the new resolveExternalCIDR / externalCIDRFromBridge helpers become the backing implementation of HostWiring.ExternalCIDR() once the network/host package lands in Phase 1. Until then the resolved value is only used to fail-start on missing uplink. * fix(vpcd): cover external CIDR startup paths Extract launchService's CIDR-resolve block to ensureExternalCIDRReady helper so it's unit-testable, and add direct tests for externalCIDRFromBridge (loopback + unknown-interface) to exercise the original literal body. Closes the diff-coverage gap on Phase 0 commit 15b564d which left these paths unreachable from unit tests. * refactor(network/ovn): extract L1 OVN client to network/ovn package Move OVNClient interface, LiveOVNClient, ACLSpec, ErrNATNotFound and namedUUID helper from services/vpcd to a new spinifex/network/ovn package. The mock relocates to spinifex/network/ovn/mock as a sub-package. Storage fields on the mock become exported so vpcd tests can continue to seed/inspect state across the new package boundary. services/vpcd retains thin type-alias and var-alias shims so existing callers compile unchanged; Phase 2 removes those shims as vpcd is torn down. Adds -coverpkg=./spinifex/... to test-cover so cross-package test exercise of mock methods continues to count toward coverage now that the mock lives outside vpcd. First sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md (bead mulga-siv-125.2). Pure refactor; no functional change. All existing tests pass; preflight green. * feat(network/topology): TopologyManager interface + vpcd adapter Add spinifex/network/topology with Manager interface, VPC/Subnet/Port specs, and OVN object naming helpers covering the L2 stable contract (§7 of the redesign plan). Phase 1 keeps existing in-tree OVN names (rtr-, ext-port-, sg_) and notes the §7.3 targets for a later reconciliation migration. TopologyHandler in services/vpcd grows EnsureVPC / DeleteVPC / EnsureSubnet / DeleteSubnet / EnsurePort / DeletePort / SetPortSecurityGroups methods that mirror the corresponding NATS handlers' OVN ops with Spec inputs instead of nats.Msg envelopes, satisfying topology.Manager via a compile-time check. Existing handlers are unchanged; the duplication is short-lived (Phase 2 collapses them). Smoke tests in topology_manager_test.go cover the idempotent create path, delete cascade, and declarative SG membership move. Deliverable: TopologyManager.EnsureVPC/Subnet/Port callable; all existing tests green. Second sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md (bead mulga-siv-125.2). * feat(network/host): L0 HostWiring interface + physical/veth impls Phase 1 deliverable for mulga-siv-125.2. Introduces network/host (L0 of ADR-0006): UplinkMode enum, Wiring interface, and Physical/Veth implementations with injectable Runner + InterfaceReader for testability. Not yet wired into daemon/vpcd — Phase 2 replaces daemon.SetupComputeNode and vpcd's resolveExternalCIDR with calls through this package. * feat(network/policy): L3 SecurityGroup/NAT/Route managers Introduce network/policy as the data-plane policy layer (ADR-0006 L3). Three managers, all backed by network/ovn: - SecurityGroupManager: clear+add ACL replacement on an existing port group; port group lifecycle stays with L2 (topology.Manager). - NATManager: bound to one NATMode at construction (derived from host.UplinkMode). Distributed sets logical_port/external_mac on EIP dnat_and_snat rules; centralised leaves them unset. Per-EIP, NAT GW, and IGW-default SNAT entrypoints; all deletes idempotent. - RouteManager: AddStaticRoute is read-then-decide so retries no longer duplicate rows (closing the long-standing non-idempotency bug in vpcd/topology.go's IGW default-route path). ACL priority constants and infrastructure ACL builders move from vpcd/{security,acl}.go into policy/acl.go with their semantics preserved (default-deny 900/800 logged, DHCP-allow 1050, tenant 1000). No vpcd wiring changes here — additive only; the reconciler (bead .3.2) and network/external (bead .3.4) will consume these managers. bead: mulga-siv-125.3.1 * feat(network/external): L5 IGW/EIP/NATGW managers Pure additive — new package, no topology.Manager changes, no rewiring of existing vpcd handlers. IGWManager builds the external switch + localnet + gateway LRP + default route + chassis HA, with a flows barrier callers can inject (production: ovn-nbctl --wait=hv sync). EIP/NATGW are thin L5 wrappers over policy.NATManager. GatewayIPAllocator interface lets the DHCP-backed allocator stay in vpcd until the DHCP-mgr removal bead lands; LinkLocalAllocator + StaticRangeAllocator ship here. * feat(network/reconcile): L6 intent-actual reconciler replaces 5-pass startup Build spinifex/spinifex/network/reconcile/ package per Phase 2.2 of the network layer redesign. Reads desired state from NATS KV filtered by vpc.AZ (with legacy empty-AZ matches-all rule), scans actual state from OVN NB DB, diffs and applies with VPC → Subnet → Port → SG → IGW → EIP/NATGW topological ordering. Atomic port-group join via CreateLogicalSwitchPortInGroups closes the racy gap the legacy scanENIPortMembership pass used to revive 30s later. A 5-minute drift goroutine (leader-gated via JetStream KV CAS) replaces the old 30s ReconcileSGsLoop. Adds AZ field to VPCRecord stamped at create time. Empty AZ on pre-Phase 2.2 records matches every local AZ, so no migration pass is required. Replaces and removes the 5-pass startup sequence in services/vpcd: - Reconcile + ReconcileFromKV - ReconcileSGsOnce + ReconcileSGsLoop - RetrofitAllExternalLocalnetOptions - RetrofitAllGatewayPortNetworks IGW fresh-create continues to flow through the legacy NATS handler path (handleIGWAttach) so the dhcpManager wiring stays intact until Phase 2.3; the reconciler only rebinds gateway-chassis on existing IGW topology. Closes mulga-siv-125.3.2. * refactor(network/topology): promote live Manager impl to network/topology Move the topology.Manager implementation out of services/vpcd/topology_manager.go into network/topology/manager_live.go as a standalone liveManager constructed via NewLiveManager(ovn.Client, opts...). Inputs that varied per-handler (DNS server resolution from the external pool) are now passed as functional options (WithDNSServer). TopologyHandler still satisfies topology.Manager but its methods delegate to a cached liveManager. The shim stays until the rest of mulga-siv-129 routes NATS subscribers through topology.Manager directly. Also exports SubnetGatewayCIDR + FormatDNSServerList from network/topology so the reconciler in network/reconcile can share helpers with the live manager (audit pass to follow per plan §6). Test cases ported one-for-one from services/vpcd/topology_manager_test.go to network/topology/manager_live_test.go; the vpcd-side test file is deleted. New WithDNSServer test asserts the option propagates into DHCPOptions rows. Refs mulga-siv-129 (Phase 2.6 vpcd subscriber migration, Slice 1 of N). * refactor(vpcd): collapse SG subscribers onto network/{topology,policy} managers Phase 2.6 / mulga-siv-129 slice 2. Rewrites the vpc.create-sg / vpc.update-sg / vpc.delete-sg NATS subscribers in services/vpcd as thin wrappers over topology.Manager (port group lifecycle) and policy.SecurityGroupManager (ACL programming). Deletes the ~1000 lines of duplicated ACL builder and provisioning logic — its authoritative copy lives in network/policy. Surface additions: - ovn.ErrPortGroupNotFound sentinel; mock + live GetPortGroup wrap not-found via fmt.Errorf("%w: ...") so callers can use errors.Is. - topology.Manager grows EnsureSGPortGroup / DeleteSGPortGroup (resolves the apply.go TODO from §2.1 of the redesign plan). liveManager implements them with GetPortGroup-then-Create / GetPortGroup-then-Clear+Delete idempotency. Behaviour delta: the old provisionSG rolled back the OVN port group on AddACLs failure ("all-or-nothing"). The new path lets the port group survive and relies on network/reconcile to retry EnsureSG on its next sweep. The all-or-nothing fail-fast test cases are dropped; ACL emission semantics are covered by network/policy/sg_test.go. * refactor(vpcd): migrate VPC/Subnet/ENI subscribers to topology.Manager Phase 2.6 / mulga-siv-129 slice 3. Rewrites vpc.create / vpc.delete / vpc.create-subnet / vpc.delete-subnet / vpc.create-port / vpc.delete-port / vpc.update-port-sgs NATS subscribers as thin wrappers (≤30 lines each) over topology.Manager. Subscriber bodies now: parse event → translate to spec → delegate to manager. Deletes: - vpcd.subnetGateway (and helpers_test.go + topology_test.go's TestSubnetGateway block); parity coverage lives in network/topology.TestSubnetGatewayCIDR. - vpcd.reconcilePortSGs (the live manager already exposes the same diff-then-transact via reconcilePortSGs on the L2 side). - vpcd.TopologyHandler.dnsServer; lmDNSServer (passed via topology.WithDNSServer to the live manager) is the live path. Behaviour delta: handlers now strict-parse PrivateIpAddress / MacAddress / CidrBlock at the NATS boundary and reject malformed payloads instead of degrading to "no DHCP options". The reconciler guarantees that field validity is upheld upstream of the publish. * refactor(network/reconcile): drive SG port groups via topology.Manager applySGs now calls topology.Manager.EnsureSGPortGroup / DeleteSGPortGroupByName instead of reaching directly into the OVN client. Reconcile.Config gains a Topology field (required); the production caller in services/vpcd passes the TopologyHandler shim, the reconcile package tests construct a fresh topology.NewLiveManager bound to the mock OVN client. DeleteSGPortGroupByName is added to the topology.Manager interface for the reconciler's orphan-removal path, which iterates raw OVN port-group names without a groupID round-trip. The existing DeleteSGPortGroup(groupID) delegates to it. Resolves the TODO at apply.go:137 carried over from slice 2 of mulga-siv-129. * feat(network/policy): add FlowsBarrier option to NATManager NATManager.AddEIP needs to block on `ovn-nbctl --wait=hv sync` after writing the dnat_and_snat row so the NATS reply only returns once every chassis has the rewrite flow. The legacy vpc.add-nat handler does this at the tail of handleAddNAT; the manager has lacked it since slice 2.1. NewNATManager grows a functional-options tail (Option / WithFlowsBarrier). Defaults to a no-op for tests; production wiring in services/vpcd passes waitForFlowsHV. AddEIP now invokes the barrier after AddNAT, logging at warn level on failure to match the legacy handler's _ = waitForFlowsHV() swallowing semantics. Pre-work for slice 4 of mulga-siv-129 — without the barrier the NAT EIP subscriber migration would lose the post-write flow-install gate. * refactor(vpcd): migrate NAT GW subscribers to policy.NATManager handleAddNATGateway / handleDeleteNATGateway now decode the NATS event and delegate to policy.NATManager.{AddNATGateway,DeleteNATGateway}. The manager carries identical OVN logic — same snat type, same ExternalIDs (spinifex:vpc_id + spinifex:nat_gateway_id) — so existing TestTopologyHandler_AddNATGateway / DeleteNATGateway suites stay green. Lazy-constructed NATManager added on TopologyHandler (natmOnce). Mode is derived from the bridge mode (veth → centralized, direct → distributed) and the FlowsBarrier is bound to waitForFlowsHV for parity with the legacy EIP path's tail barrier (used by the next slice, vpc.add-nat). Slice 4 of mulga-siv-129, sub-step b. * refactor(vpcd): migrate EIP NAT subscribers to policy.NATManager handleAddNAT / handleDeleteNAT now decode the NATS event and delegate to policy.NATManager.{AddEIP,DeleteEIP}. The behavioural contract is preserved end-to-end: - Idempotency-skip on a matching dnat_and_snat row (same VPC + LogicalIP, plus MAC + LogicalPort under distributed NAT) moves into AddEIP so duplicate vpc.add-nat publishes never tear down the rule the previous invocation just installed (mulga-siv-124). - Stale-rule cleanup across every router stays in AddEIP. - Post-write waitForFlowsHV barrier is now driven by the manager via the FlowsBarrier option wired in services/vpcd (slice 4a). - DeleteEIP returns nil on ErrNATNotFound, matching the handler's fire-and-forget "already absent = success" semantic. The vpcd "errors" import drops out with the inline error-comparison. Slice 4 of mulga-siv-129, sub-step c. * refactor(vpcd): migrate IGW subscribers to external.IGWManager (partial) handleIGWAttach now branches at decode time: - Centralised NAT + DHCP-sourced pool keeps using attachIGWLegacy because the gateway-LRP IP comes from vpc.dhcp.acquire, and that DHCP client is the vpcd-local manager that bead mulga-siv-125.3.3 will retire. - All other shapes (distributed NAT, static-range pool, no pool) delegate to external.IGWManager.AttachIGW. The manager mirrors every step the legacy handler ran — external switch + localnet port + gateway LRP + switch-side router peer + default route + chassis HA + waitForFlowsHV tail barrier — so observable behaviour is unchanged. handleIGWDetach drops its inline OVN walk entirely; both paths exit through external.IGWManager.DetachIGW followed by releaseGatewayLRPLease. The lease release stays in vpcd because the upstream client lives in services/vpcd/dhcp; calling it after every detach is safe because the upstream server expires unknown leases on its own. Lazy-constructed IGWManager added on TopologyHandler (igwmOnce). Pool, allocator (StaticRangeAllocator), chassis, NAT mode, NAT manager, route manager, and FlowsBarrier are all derived from the existing TopologyHandler state. The DHCP-coupled legacy branch retains its full inline OVN logic; it will disappear in slice-after-125.3.3 follow-on. Slice 4 of mulga-siv-129, sub-step d. * feat(ec2/instance): honor AssociatePublicIpAddress per-NIC override RunInstances --network-interfaces "AssociatePublicIpAddress=true|false" was silently ignored, only subnet.MapPublicIpOnLaunch gated public IP assignment. AWS contract is per-NIC override wins. Extend TestPrepareRunInstances_PublicIPAutoAssigned to table-driven 3 cases: subnet true no-override, subnet false override true, subnet true override false. * refactor(config): drop upstream-DHCP config surface (static WAN pool only) Hard-error in LoadConfig on legacy keys: [network] external_dhcp, [nodes.<n>.vpcd] dhcp_bind_bridge, and [[network.external_pools]] source = "dhcp". Add pool range validation (must fit inside the gateway/prefix_len CIDR; per-cluster non-overlap). Strip ExternalDHCP from config.NetworkConfig + formation.NetworkConfig + admin.ConfigSettings, and DhcpBindBridge from config.VPCDConfig + admin.ConfigSettings + vpcd.Config. vpcd.ExternalPoolConfig.IsDHCP() short-circuits to false; the field and downstream DHCP branches in external_ipam / topology.go remain for Slice C to delete together with dhcp_manager. cmd/spinifex/cmd/admin.go: drop useExternalDHCP path and the detectedDhcpBindBridge helper; require --external-pool when --external-mode=pool; auto-detect default no longer infers source=dhcp. service.go's wan_bridge legacy-key rejection now advises removing the key entirely (vpcd auto-detects br-wan), not renaming to dhcp_bind_bridge. Two topology_test cases that exercise the DHCP gateway-LRP path are skipped — Slice C deletes attachIGWLegacy and these tests together. * refactor(vpcd): delete dhcp_manager + DHCP gateway-LRP path (Slice C) Removes upstream-DHCP-client gateway-LRP allocation in favor of static WAN-pool allocation (StaticRangeAllocator). Part of bead mulga-siv-125.3.3. - delete services/vpcd/dhcp_manager.go + dhcp/ subpackage (~865 lines) - delete allocateGatewayLRPIPViaDHCP + releaseGatewayLRPLease paths - drop ExternalPoolConfig.Source + DhcpBindBridge + IsDHCP() fields - drop ExternalIPAllocation lease fields (Expiry/ServerID/HWAddr) and ExternalIPAMRecord.Source - drop NewExternalIPAM nc *nats.Conn argument; update all 4 call sites - rename verifyBridgeMode local dhcpBindBridge -> wanBridge; update error strings and tests Slice D (delete attachIGWLegacy + dispatcher) intentionally deferred. * refactor(vpcd): delete attachIGWLegacy + IsDHCP stub (Slice D) Closes mulga-siv-129 slice 4d. Part of bead mulga-siv-125.3.3. handleIGWAttach now unconditionally delegates to external.IGWManager. attachIGWLegacy (~207 lines, ~5 OVN write steps + chassis scheduling) deleted along with its dispatcher branch in handleIGWAttach. The ExternalPoolConfig.IsDHCP() dead stub is gone. No code paths or NATS subjects regress: the static-pool / distributed branches were already going through external.IGWManager since Slice 4d (6719dc3). * feat(ipam): Purpose field on every IPAM record (Slice B) Part of bead mulga-siv-125.3.3. Adds a Purpose enum (handlers/ec2/vpc/purpose.go) tagging every IP allocation with its semantic role (eni-primary, eni-public, eip, igw-lrp, natgw-external, …) so multi-VPC clusters can reclaim/audit by (owner, purpose). External IPAM (spinifex-external-ipam, v1→v2): - ExternalIPAllocation.Type → .Purpose; legacy values mapped via LegacyExternalTypeToPurpose (gateway→igw-lrp, auto_assign→eni-public, elastic_ip→eip). Unknown legacy values warn and become "unknown". - AllocateIP / AllocateFromPool / allocateFromPool param rename allocType→purpose. All call sites pass Purpose* constants. - migrate/002_external_ipam_purpose.go: idempotent CAS-retry migration with v1/v2-union record shape so partial migrations resume cleanly. Internal IPAM (spinifex-vpc-ipam, v1→v2): - IPAMRecord.Allocated []string → []IPEntry{IP, Purpose, OwnerID}. - AllocateIP gains purpose+ownerID args; ReleaseIP unchanged (still keyed on IP). - AllocatedIPs returns []IPEntry. - eni.go reorders eniId generation before AllocateIP so the owning resource ID lands on the new entry. - migrate/003_ipam_purpose.go: cross-bucket backfill that reads the ENI bucket via the new RunKVWithJetStream path. Orphaned IPs (no owning ENI) warn and become Purpose=unknown. Migrate framework: - KVContext gains an optional JetStream field; new Registry.RunKVWithJetStream method threads it through. RunKV signature unchanged — all other callers are unaffected. Tests: - ipam_test.go: AllocateIP signature updates; AllocatedIPs now asserts IP / Purpose / OwnerID on the returned entries. - external_ipam_test.go: replaces "auto_assign"/"elastic_ip" literals with PurposeENIPublic/PurposeEIP and asserts alloc.Purpose. - migrate/002 + 003 tests: schema-detect, idempotent rerun, no-JetStream fallback, empty bucket. * refactor(daemon): tombstone obsolete spinifex-dhcp-leases KV bucket The dhcp_manager removed in Phase 2.3 (bead mulga-siv-125.3.3) owned the spinifex-dhcp-leases KV bucket. The migrate framework only handles per-bucket schema version bumps, not bucket deletion, so the obsolete bucket would linger on upgraded clusters. Add a one-shot tombstone after initJetStream that deletes the bucket if present. DeleteKVBucketIfExists treats ErrBucketNotFound and ErrStreamNotFound as success, so the call is idempotent across restarts. * feat(config): add network.ipsec_enabled cluster flag (default true) Add IPSecEnabled to NetworkConfig in both the cluster TOML schema and the formation handshake payload. Default true so AWS-parity intra-AZ encryption is on by default. Operators can flip false in [network] for trusted single-rack lab topologies. Phase 2.5 (bead mulga-siv-125.3.5) consumes the flag from admin init (strongSwan provisioning), formation server (propagates to joiners), and daemon startup (ovs-vsctl ipsec_encapsulation toggle). * feat(admin): IPsec peer cert generator signed by cluster CA Add GenerateIPSecPeerCert + IPSecCertPaths in spinifex/admin so admin init (and the future admin add-node path) can issue per-node IKEv2 peer credentials signed by the existing cluster CA. The cert carries id-kp-ipsecIKE (RFC 4945) so strongSwan accepts it as an IKEv2 peer credential when ovs-monitor-ipsec materialises tunnel configs. Cluster CA is reused directly — no intermediate strongSwan PKI today; when intermediate CAs land that becomes a separate workstream. The cert/key pair lives under {configDir}/ipsec/peer.{pem,key}; the daemon points OVS Open_vSwitch.other_config (certificate, private_key, ca_cert) at these paths when network.ipsec_enabled is true. * feat(admin): wire IPsec peer cert into admin init + join paths Add --ipsec bool flag (default true) on adminInitCmd. Single-node init and the multi-node leader path both emit an IPsec peer cert after the existing CA+server-cert generation. The multi-node formation server propagates the IPSecEnabled flag to joining nodes via NetworkConfig; admin join honours it and emits its own peer cert during the same cert-issuance step that already produces the server cert. spinifex.toml template gains an unconditional [network] block carrying ipsec_enabled so the value is visible to operators on every install regardless of whether external networking is configured. The daemon reads it via cluster config (NetworkConfig.IPSecEnabled). Bead mulga-siv-125.3.5. * build: install openvswitch-ipsec for OVN native IPsec openvswitch-ipsec provides ovs-monitor-ipsec, which reads the Open_vSwitch.other_config cert/key/CA pointers and materialises strongSwan configs for every Geneve tunnel managed by ovn-controller. Pulls in strongswan + strongswan-pki as transitive deps. Required by bead mulga-siv-125.3.5. * feat(daemon): enable OVN native IPsec on intra-AZ Geneve at startup When network.ipsec_enabled is true (default), the daemon points the local OVS DB other_config at the per-node IPsec peer cert that admin init / admin join provisioned (configDir/ipsec/peer.{pem,key} + the cluster CA at configDir/ca.pem), then sets other_config:ipsec_encapsulation=true. ovs-monitor-ipsec (shipped by the openvswitch-ipsec package) consumes those pointers and materialises strongSwan configs for every Geneve tunnel ovn-controller programs — including OVN-IC inter-AZ tunnels when federation lands in Phase 3a. Missing cert files log a warning and leave Geneve plaintext; operators upgrading an older cluster reissue certs via spx admin to close the gap. Bead mulga-siv-125.3.5. * test(e2e): multinode IPsec wiring + xfrm SAs + ESP capture TestMultinodeIPSec asserts: - OVS DB on every node carries certificate/private_key/ca_cert pointers and other_config:ipsec_encapsulation=true (what daemon.enableOVNIPSec writes). - ip xfrm state on every node has at least one AEAD SA in AES-GCM mode (rfc4106(gcm(aes)) or gcm(aes)) within 90s. - tcpdump catches ESP (ip proto 50) on the underlay (best-effort — logged only; geneve may be idle during the 5s capture window). Skips cleanly when the cluster was bootstrapped with --ipsec=false (no ipsec_encapsulation key on first node). Read-only over SSH; runs in the parallel bucket alongside the other non-mutating multinode tests. * fix(igw): write attachment State="attached" for AWS parity Reconciler intent loader (network/reconcile/intent.go::loadIGWs) filters IGW records with `State == "attached"`, but AttachInternetGateway was writing `State = "available"`. The reconciler therefore never picked up any attached IGW — `intent_igws=0` on every pass — so the OVN external switch + gateway LRP + WAN bridge wiring built by IGWManager.AttachIGW never ran for default-VPC IGWs (whose live `vpc.igw-attach` publish at daemon bootstrap is lost — vpcd subscriber not yet wired). Net effect on single-node e2e: every SSH probe to 192.168.x public IPs timed out (TestSSHProbe, TestModifyInstanceAttribute, TestRebootInstance, TestNATGateway, TestSGToSGDatapath) because SNAT/DNAT rules on the VPC router had no path to the external network. Match AWS InternetGatewayAttachment.State semantics: - Attach: State="attached" - Detach: State="detached" Detach value is internal — recordToEC2 gates Attachments on VpcId != "", so the detached State is never surfaced to API consumers. Bead: mulga-siv-131 * fix(reconcile): call AttachIGW when IGW topology missing applyIGWs previously deferred to the NATS handler when an intent IGW had no external switch in OVN. When the vpc.igw-attach event is lost (bootstrap-time default-VPC IGW published before vpcd subscriber is ready), deferral never resolves and VM public-IP traffic dies. Call IGWManager.AttachIGW directly — it is idempotent (short-circuits when the external switch already exists), so the reconciler converges default-VPC IGWs on first pass without duplicating work on subsequent passes. Closes mulga-siv-132. * fix(test): align NewExternalIPAM call with 2-arg signature Rebase onto dev pulled in daemon_system_instance_test.go which used the legacy 3-arg NewExternalIPAM(jsNC, js, pools). The feature branch already narrowed the signature to (js, pools) via the upstream-DHCP config drop; align the test with the current signature. * fix(vpcd): EnsureSubnet must EnsureVPC first to avoid NATS race vpc.create-vpc and vpc.create-subnet are independent NATS events with no inter-event ordering guarantee. Tenant bootstrap on iam.account.created publishes both within <1ms, so the subnet handler's CreateLogicalRouterPort can race the parent VPC's CreateLogicalRouter and fail with "logical router not found". The cascade fails CreateNetworkInterface which fails RunInstances which surfaces as InsufficientInstanceCapacity to the SDK — masking the underlying topology race. handleSubnetCreate now calls EnsureVPC(VPCID-only) before EnsureSubnet. EnsureVPC short-circuits on existing LR (idempotent), so the race-winning vpc.create-vpc handler's full CIDR/VNI ExternalIDs are preserved; if subnet wins the race the LR is created with bare ExternalIDs (cosmetic) and the follow-on vpc.create-vpc short-circuits. Adds TestIntegration_SubnetBeforeVPC regression covering both orderings. Closes mulga-siv-133. * feat(ansible): require --external-pool, drop NAT, forward EXTRA_VARS spx admin init now rejects --external-mode=nat and demands --external-pool when mode=pool. The ansible init role silently skipped pool flags when the range was unset, so installs fell over inside the spx validator. Make targets also swallowed `-e` (make's env-override flag), so operators could not override pool start/end without editing defaults.yml. - vars/defaults.yml: rename nat_gateway_ip → external_gateway_ip; add spinifex_external_no_external opt-out; document the spx flag contract. - roles/init/tasks/main.yml: drop NAT branch; assert pool range upfront with a fail_msg pointing at EXTRA_VARS / defaults.yml; emit gateway, prefix-len, gateway-ip only when explicitly set (let spx auto-detect); honour --no-external. - playbooks/dev-reset.yml: rename captured var; coerce legacy external_mode=nat to pool with a warn; tighten gateway regex with \b. - Makefile: ansible-dev-* targets now forward EXTRA_VARS="key=val ..." through --extra-vars so `make` callers can override without editing defaults.yml. - scripts/ansible/README.md: document pool requirement, both invocation styles (direct -e vs make EXTRA_VARS), NAT removal. * fix(ipsec): guard ipsec_encapsulation on ovs-monitor-ipsec liveness enableOVNIPSec set OVS cert pointers + ipsec_encapsulation=true without ensuring ovs-monitor-ipsec.service was running. Without that daemon, OVS holds the cert pointers unused, strongSwan never gets IKE config, and tunnel traffic ovn-controller marks for encryption gets silently dropped. TestMultinodeIPSec fails on `node1 xfrm has no AEAD SAs`. - systemd: spinifex-daemon Wants=/After= ovs-monitor-ipsec.service - daemon: ensureOVSMonitorIPSecActive — systemctl enable --now + is-active poll; refuses to flip ipsec_encapsulation if daemon is dead - daemon: single-node short-circuit — no peers means no tunnels means the flag is at best inert and at worst load-bearing-silent-drop - ci: capture-net-diag.sh + workflow steps to snapshot OVS/OVN/xfrm state on failure, fan out across all 3 multinode nodes Refs: mulga-siv-136 * test(e2e): sticky-skip downstream SSH tests when probe fails When TestSSHProbe times out (3 min, typically because of mulga-siv-134 datapath bug), four downstream tests each re-probe SSH on the same broken datapath and time out identically — single-node job runs ~24 min instead of ~12 min, with the same error logged five times. First waitForSSHReady timeout now sets a package-level atomic flag; subsequent waitForSSHReady / trySSHReady / requireSSHHealthy calls skip immediately. Preserves the original red signal on TestSSHProbe and on any test that genuinely needed SSH (Skip ≠ Pass). Bool variant (trySSHReady) consults but does not set the flag — its caller already treats False as advisory warning, not fatal. TestSGToSGDatapath has its own EventuallyErr SSH wait (not via waitForSSHReady) and needs an explicit requireSSHHealthy(t) gate at the top of runSGToSGDatapath. TestNATGateway already calls waitForSSHReady first so the gate fires naturally. Refs: mulga-siv-137 * fix(ipsec): use Debian unit name openvswitch-ipsec.service CI artifact shows `Unit ovs-monitor-ipsec.service could not be found` on every node. Debian's openvswitch-ipsec package ships the unit as openvswitch-ipsec.service (the binary inside is ovs-monitor-ipsec, but systemd's name follows the package). The previous commit guarded on the wrong unit name, so: - spinifex-daemon.service Wants=/After= silently no-op'd (missing unit) - enableOVNIPSec's `systemctl enable --now ovs-monitor-ipsec.service` failed, daemon logged a warning and continued without flipping ipsec_encapsulation=true - TestMultinodeIPSec skipped with `node1 reports ""` because OVS DB never received the cert pointers Rename across all 5 spots (daemon, systemd unit, test, CI diag script, workflow journal capture) to match the Debian package layout. Note: the package itself still must be present on the CI VM image — that's tracked in the mulga repo (image-builder provision.sh + bootstrap-install.sh shim). Refs: mulga-siv-136 * fix(e2e): authorize default SG tcp/22 + icmp in single-node fixture admin init leaves the default SG closed (AWS parity); the Go single suite never ported the Phase-5 SG-ingress block that run-e2e.sh has had since day one, so every test runner SSH/ping probe hits the OVN port-group ACL drop. - ec2helpers: add AuthorizeICMPIngress + EnsureDefaultSGOpen helpers. - single/main_test: open default SG once per process in pkgFixOnce.Do. - witness: extend ensureDefaultSGSSHIngress to authorize ICMP too. - run-e2e.sh: append matching ICMP authorize next to tcp/22 block. mulga-siv-134 * fix(network/reconcile): key DHCP_Options lookup by subnet_id not CIDR Multi-tenant default VPCs share the same subnet CIDR (172.31.0.0/20) per account. The existing FindDHCPOptionsByCIDR gate at apply.go:107 found the first account's DHCP_Options row and skipped creation for every subsequent subnet with the same CIDR. Those subnets' ports then attached nothing (FindDHCPOptionsByExternalID misses on subnet_id), VMs got no DHCPv4 lease, and SSH/ICMP into them silently failed even though SG ACLs and NAT were correct. Switch the gate to FindDHCPOptionsByExternalID keyed on spinifex:subnet_id so each subnet owns its own row, matching the attach lookup at line 203. Verified: account 000000000001 default VPC RunInstances now produces an LSP with dhcpv4_options populated, VM cloud-init reports a 172.31.0.5/20 lease + default route via 172.31.0.1, and ping + ssh TCP handshake to the EIP succeed end-to-end. Stale DHCP_Options row for the prior dead subnet was wiped manually; reconciler recreated the correct row for the live subnet on its next pass. * fix(ipsec): enable openvswitch-ipsec at provision time, drop daemon write Provisioning now runs `systemctl enable --now openvswitch-ipsec.service` alongside openvswitch-switch and ovn-controller. The spinifex-daemon process drops its `enable --now` shell-out and only polls is-active. Sudoers gains one read-only line: spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/systemctl is-active openvswitch-ipsec.service This mirrors the existing vpcd ovn-controller is-active precedent and removes the daemon's need for systemd-write capability. Aligns with least-privilege service account pattern: provision enables services, daemon only reads status. If the unit is inactive at runtime, the daemon refuses to flip ipsec_encapsulation=true (silent-drop guard from mulga-siv-136) and surfaces a clear error pointing at scripts/setup-ovn.sh. * chore: unblock preflight on feat/network-layer-redesign Two pre-existing failures blocked preflight on the branch: - govulncheck flagged golang.org/x/net@v0.54.0 (GO-2026-5026, ASCII Punycode rejection bug in idna.ToASCII); reached via utils.DownloadFileWithProgress. Bump to v0.55.0; go mod tidy carries x/sys to v0.45.0. - TestTopologyHandler_SubnetCreate_RouterPortFails asserted subnet create fails when the parent VPC router is missing. That contract changed in aed7b2e (mulga-siv-133), which made handleSubnetCreate call EnsureVPC idempotently before EnsureSubnet to avoid a NATS publish-order race. Delete the now-impossible failure case. * refactor(network/ovn): relocate nbdb model package from services/vpcd Phase 4 / mulga-siv-125.6 Slice A. Pure import rename, no logic change. The OVN Northbound DB model (LogicalSwitch, LogicalRouter, DHCPOptions, etc.) lived under services/vpcd/nbdb/ because that was the original home of every OVN caller. After Phase 1+2 the L1 client (network/ovn) and all L3/L5 managers (network/policy, network/external, network/topology, network/reconcile) became the primary callers — 8 importers in network/* vs 2 in services/vpcd/. The directory move puts the schema model where its consumers are, and unblocks deleting services/vpcd/ entirely once the subscriber/launcher relocations land (Slices B–D). * refactor(vpcd): delete dead retrofit/reconcileIGW helpers + bridge-mode bleed Phase 4 / mulga-siv-125.6 Slice B. Production path no longer touches any of: - RetrofitAllGatewayPortNetworks / RetrofitAllExternalLocalnetOptions — the 5-pass startup leftovers that network/reconcile (Phase 2.2) made redundant. Nothing called them outside tests. - reconcileIGW + its allocateGatewayLRPIP / gwLrpRange helpers + the ipv4ToUint32 / uint32ToIPv4 utilities — superseded by external.IGWManager (Phase 2.4) which carries its own pool logic. - ensureLocalnetOptions / ensureGatewayPortNetworks / expectedGatewayPortNetwork — read-before-write retrofit shims for the same code paths. - generateMAC — only callers were reconcileIGW + its tests. useCentralizedNAT (the bridge-mode bleed flagged in plan §2.2) is replaced by a policy.NATMode value injected via WithNATMode(...). The mode is resolved once in vpcd.go from host.UplinkMode and threaded into both the topology-handler lazy managers and the reconciler stack, so "is this veth?" is no longer a question topology.go can ask. The TopologyHandler defaults to NATModeDistributed when WithNATMode is omitted, matching the previous default. Drops services/vpcd/topology.go from 1220 LOC to 693, deletes ~700 LOC of obsolete tests (TestGenerateMAC, TestEnsureLocalnetOptions_*, TestRetrofitAllExternalLocalnetOptions_FixesClearedOptions, TestReconcileIGW_*, TestGwLrpRange), and frees the nbdb + utils + net imports in topology.go. Stale "phase 4 migrates those onto the new managers" comment in vpcd.go fixed in passing — the subscribers were migrated in Phase 2.6 (mulga-siv-129); only the file-relocation parts of Phase 4 remain. Stale ensureLocalnetOptions / ensureGatewayPortNetworks references in network/ovn/{live.go,mock/mock.go} doc comments also cleaned up. * refactor(network): move NATS VPC subscribers to network/subscribers Phase 4 Slice C (mulga-siv-125.6). Promotes the NATS event handlers out of services/vpcd into a new spinifex/network/subscribers package. Subscriber holds the topology/policy/external managers directly — the lazy *Once adapter shim (topology_manager.go) is deleted along with vpcd's local event-struct copies. launchService now constructs topology.LiveManager eagerly with a DNS callback derived from external pools (preserving the unscoped-pool lookup old TopologyHandler.lmDNSServer did), builds every manager up front, then hands them to subscribers.New + Subscribe. reconcile.New receives topology.Manager directly instead of through the adapter. Bridge-mode constants and waitForFlowsHV move into vpcd.go alongside the launcher; they were left behind as topology.go's last responsibilities. Handler-layer tests (topology_test.go, security_test.go, integration_test.go) are dropped: substance is covered by network/topology, network/policy, network/external manager tests. subscribers/subscribers_test.go adds focused coverage: missing-dep rejection, topic registration smoke, and bad-JSON envelope per topic. services/vpcd is now down to the launcher + the ovn/mock compat aliases (removed in Slice D when the launcher promotes to cmd/vpcd/). * refactor(vpcd): relocate services/vpcd → spinifex/vpcd; drop dead aliases Phase 4 Slice D (mulga-siv-125.6). Final cleanup of the network-redesign package layout. services/vpcd/ disappears entirely; the launcher remains an in-process Service impl, just rehomed under spinifex/vpcd/ (no separate binary — spx vpcd start stays the cobra entrypoint). - Delete services/vpcd/mock_ovn.go + ovn.go (compat aliases now dead post Slice C; only callsite was launchService's NewLiveOVNClient, inlined to ovn.NewLiveClient). - git-rename services/vpcd/vpcd.go + vpcd_test.go to spinifex/vpcd/. - Retarget three importers (spinifex/service/service.go + service_test.go, cmd/spinifex/cmd/service.go) to the new path. - Fix stale services/vpcd / TopologyHandler comments in network/reconcile/ + network/external/types.go + spinifex/vpcd/vpcd.go. - service-interfaces.yaml vpcd path updated. - Plan doc §13/§14/§16 + §9.2/§19 updated: drop cmd/vpcd/main.go (was speculative; never built) in favour of in-process Service impl at spinifex/vpcd/vpcd.go. Plan §16 Created list now lists subscribers/. Phase 4 plan items 1+2 (remove services/vpcd, entrypoint location) satisfied. Items 3+4 remain blocked on mulga-siv-125.4 + 125.5 (bundled join token, plan archive). * fix(ovn): install openvswitch-ipsec + strongswan explicitly ovs-monitor-ipsec.service fails to start without an IKE daemon. The openvswitch-ipsec package only Recommends strongswan, so any apt config that drops Recommends (or a base image with the recommends already skipped) leaves the service in a Restart loop with 'IKE daemon is not installed'. Hoist both packages into setup-ovn.sh so provisioning is the single source of truth — the bootstrap shim in mulga is no longer needed. * test(network): pin ADR-0006 invariants to executable suite Add static-analysis + behavioural tests that bind every safety and liveness clause in ADR-0006 (S1-S9, L1-L3) to a named Go test. Failure output quotes the ADR clause verbatim and points at the offending file:line so future debugging sessions can't quietly bypass a layer contract to make a test green. Static-analysis suite (spinifex/network/invariants/): - S1 layers_test.go: parses `go list -json`; rejects upward layer imports and layers pulling from cross-cutter packages - S2 host_containment_test.go: AST-detects ovs-vsctl / ovn-nbctl / ovs-ofctl shell-out outside network/host/ - S3 nat_mode_test.go: rejects .UplinkMode() / NATModeFromUplinkMode() calls outside the init-time resolver locations - S4 names_test.go: AST-detects OVN object name construction (concat, fmt.Sprintf, strings.Join) outside topology/names.go - S5 policy_no_create_test.go: rejects CreateLogical{Switch,Router,...} and DeleteLogical{...} calls inside network/policy/ - S6 federation_isolation_test.go: rejects TransitSwitch / TransitRouterPort references and ovn-ic literals outside network/federation/ (pre-protects the layer before it lands) - S8 ipsec_test.go: rejects strongswan / xfrm / IKEv2 / libreswan / openswan imports anywhere in network/ - S9 reconciler_singleton_test.go: rejects Reconcile…Loop and Reconcile…Pass<n> function names anywhere in network/ - S7 nats_subjects_test.go: scans for `vpc.<verb>` subjects missing the {azID} segment; calls t.Skip with the gap report so every test run prints the work-to-do without breaking CI. Flip to t.Fatalf when the cluster-wide subject rename ships. Behavioural suite (spinifex/network/reconcile/liveness_test.go): - L1 reconciler converges: scaled-intent (N=1,3,8) double-Reconcile property test; asserts second pass is a no-op - L2 drift bound: structural check that DriftInterval is positive and ≤30 min support window; full fake-clock behavioural test deferred - L3 federation re-enables: t.Skip stub pointing at the missing network/federation/ package Fixes the one production S4 violation the new walker found: network/external/igw.go was constructing "gw-port-"+vpcID inline. Adds topology.GatewaySwitchPort(vpcID) helper and retargets both AttachIGW and DetachIGW callsites through L2. Plan + per-clause mechanic + phasing in docs/development/improvements/network-invariants.md. Refs: mulga-siv-140 * fix(ovn): install strongswan-charon (not strongswan metapackage) The strongswan metapackage's '\''charon-systemd | strongswan-charon'\'' alternative resolves to charon-systemd by default, which ships /usr/sbin/charon-systemd but not the legacy /usr/sbin/ipsec helper or /usr/lib/ipsec/charon binary that ovs-monitor-ipsec --ike-daemon=strongswan shells out to. Install strongswan-charon explicitly to pull strongswan-starter (legacy ipsec controller) + the matching charon binary, matching what openvswitch-ipsec.service expects. Also reset-failed + restart openvswitch-ipsec.service after enable to recover from the deb post-install starting the service before its dependencies finished unpacking. * chore: add CODEOWNERS — joshsiv-mulga gates network/ + vpcd/ Fall-through reviewer for any change is the maintainer. network/ and vpcd/ are explicitly listed so the matching branch protection rule ("Require review from Code Owners") routes those PRs to joshsiv-mulga. Those trees are governed by ADR-0006 invariant tests (spinifex/network/invariants/); CODEOWNERS adds a human review gate on top of the mechanical fence. Refs: mulga-siv-141 * chore: CODEOWNERS fallthrough → @mulgadc/mulgadc-engineers team Fallthrough reviewer should be the team, not a single maintainer, so any engineer can approve PRs outside the explicitly-gated paths. network/ and vpcd/ keep joshsiv-mulga as the required reviewer per ADR-0006 invariant ownership. Refs: mulga-siv-141 * feat(ci): add test_run workflow input for -test.run filter Pass an optional regex through to each suite's '\''go test -test.run'\'' so dispatches can target a single test (e.g. '\''^TestMultinodeIPSec$'\'') without running the full multinode suite. Threaded through single, multinode, and reboot suite invocations; empty input preserves current behaviour. * fix(daemon): flip NB_Global.ipsec=true on management node Setting Open_vSwitch.other_config:ipsec_encapsulation=true on each chassis is necessary but not sufficient for OVN native IPsec — without NB_Global.ipsec=true, ovn-controller never adds options:remote_name to the Geneve tunnel ports, so ovs-monitor-ipsec sees no IPsec-enabled tunnels and the kernel xfrm tables stay empty. After enabling ipsec_encapsulation locally, also write NB_Global.ipsec=true when the local ovnnb_db socket is present (management node only); workers skip silently — one writer is enough for a cluster-wide flag. Adds a scoped sudoers entry pinning the daemon to that exact ovn-nbctl invocation so the grant can't be reused for other NB mutations. * fix(ovn): grant charon AppArmor read on /etc/spinifex/ipsec The strongswan-charon profile only allows reads from /etc/ipsec.*, /etc/strongswan.*, and a small fixed allowlist. ovs-monitor-ipsec writes the per-tunnel strongSwan config with absolute paths to our peer cert + key under /etc/spinifex/ipsec/, so charon hits 'Permission denied' loading them (it runs as uid 0 but AppArmor still blocks) and no xfrm SAs come up. Drop a read grant for /etc/spinifex/ipsec/** into the profile's local override hook + reload. * fix(ovn): pin OVS system-id to hostname for IPsec cert identity match ovs-monitor-ipsec uses chassis-id as the IKEv2 `@<name>` peer identity in the strongSwan config it generates. Our per-node IPsec peer cert (admin.GenerateIPSecPeerCert) carries the node name as CN + dnsName SAN. With chassis-id left at the package-generated UUID, charon validates `@<UUID>` against a cert with dnsName=<hostname> — mismatch returns AUTHENTICATION_FAILED and no xfrm SAs are negotiated. Pin system-id to hostname (which equals the node name on Mulga clusters) by writing both the on-disk source of truth (/etc/openvswitch/system-id.conf, read by ovs-ctl on boot) and the running OVS DB value. Writing only one would drift on the next reboot — the original "do not override" comment was right about that hazard, just wrong about the resolution. Cross-link the invariant in admin/ipsec.go: changing the cert identity without updating setup-ovn.sh (or vice versa) breaks IKE silently. Refs: mulga-siv-136 * fix(ovn): take --node-name flag so chassis-id matches cluster identity CI hostnames are bead-themed (env20-casuarina) and don't match the cluster node name (node1/node2) baked into the IPsec peer cert dnsName SAN at `spx admin init/join --node NAME`. Pinning system-id to `hostname -s` produced env20-casuarina; charon then validated `@node1` against dnsName=env20-casuarina and IKE failed with AUTHENTICATION_FAILED. Add a --node-name=NAME flag to setup-ovn.sh, default to hostname for manual/dev invocations, and update the ansible install role to pass {{ spinifex_node_name }}. The matching bootstrap-install.sh change lives in mulga. Refs: mulga-siv-136 * refactor(daemon): relocate L0 helpers to network/host + widen ADR-0006 S2 Phase 4 Slice E of mulga-siv-125.6. Cleans up spinifex/daemon/network.go per the audit and widens the S2 invariant to forbid OVS/OVN shell-outs in spinifex/daemon/ (daemon is an L0 consumer; S2 applies). Moved to network/host/: - OVSNetworkPlumber + SetupTap/CleanupTap → host.OVSPlumber (host.NewOVSPlumber) - GetBridgeIPv4 → host.GetBridgeIPv4 - OVNHealthStatus + CheckOVNHealth → host.OVNHealth + host.HealthStatus - daemon/ipsec.go OVS/OVN ops → host.SetIPSecCertPaths, host.EnableIPSecEncapsulation, host.SetNBGlobalIPSec Moved to utils: - sudoCommand → utils.SudoCommand + utils.SetSudoCommandForTest test seam (cross-package reassign would trip the reassign linter; helper keeps the indirection inside utils). Moved to vm: - generateMgmtMAC → vm.GenerateMgmtMAC (pairs with vm.GenerateDevMAC) Deleted (dead): - SetupComputeNode (replaced by network/host/bridges.go:ensureBridges + scripts/setup-ovn.sh:572-579) - EnsureDataRoute - findInterfaceByIP - daemon/network.go + dead tests in daemon/network_test.go (duplicates of vm package tests retained; mgmt MAC test moved to vm/lifecycle_test.go) S2 invariant (network/invariants/host_containment_test.go) now walks both spinifex/network/ and spinifex/daemon/, skipping network/host/. * fix(ovn): broaden AppArmor read grant to /etc/spinifex (covers CA cert) The earlier override granted reads on /etc/spinifex/ipsec/** only, which covers the peer cert + key but NOT the cluster CA at /etc/spinifex/ca.pem. charon then can't load the CA, can't validate the peer cert chain, and surfaces the failure as `no trusted RSA public key found for '<peer>'` even though identity matching now succeeds — IKE_AUTH fails with AUTHENTICATION_FAILED and no xfrm SAs negotiate. Grant /etc/spinifex/** instead (covers ca.pem + ipsec/peer.{pem,key}) and always rewrite the override so re-runs replace the older narrower rule. Refs: mulga-siv-136 * fix(ipsec): symlink cluster CA into charon's default cacerts dir ovs-monitor-ipsec writes per-conn `cacert=/etc/spinifex/ca.pem` references in the strongSwan config, but strongSwan 6.0 only loads CAs declared in standalone `ca` sections or files placed in /etc/ipsec.d/cacerts at startup. Without this, charon authenticates the local side fine but rejects every inbound peer cert as `no trusted RSA public key found for '<peer>'`, AUTH_FAILED, and no xfrm SAs negotiate. Do the symlink in admin.GenerateIPSecPeerCert (right after writing the peer cert) — setup-ovn.sh runs before `spx admin init/join` creates /etc/spinifex/ca.pem, so the wiring can't live there. Use a stage-and- rename pattern so concurrent test invocations don't race on the symlink. /etc/ipsec.d/cacerts is on charon's default AppArmor allowlist so no profile change needed. Refs: mulga-siv-136 * chore(setup-ovn): drop apt-get install_packages block OVS/OVN/openvswitch-ipsec/strongswan-charon are baked into the CI gold image via mulga's image-builder/scripts/provision.sh (commits 116bb05, a9b3a9c). Keeping a runtime apt-get install path here pulled CI through apt-cacher-ng on every bootstrap and tripped on intermittent "Message has been manipulated" InRelease cache corruption (env21/env20 runs). Drop the install block. Gold-image rebuild is the canonical place to add new system packages. If a package is missing, downstream ovs/ovn commands fail loudly during setup-ovn.sh's later steps — clearer signal than silently re-installing through a flaky proxy. setup.sh already supports INSTALL_SPINIFEX_SKIP_APT=1 (passed by mulga's bootstrap-install.sh:237) so this removes the last runtime apt-get call on the CI provisioning path. * fix(ovn): gate system-id pin on explicit --node-name setup-ovn.sh used to fall back to `hostname -s` when --node-name wasn't passed, and then unconditionally wrote that value to /etc/openvswitch/system-id.conf + ovs-vsctl external_ids:system-id. On CI single-node, the gold image already registered a chassis with name=<UUID>; setup-ovn.sh then rewrote system-id to the bead-themed hostname (env10-banksia). vpcd's discoverChassis treats any chassis row whose name != local system-id but hostname == local hostname as a stale local chassis and skips it, so vpcd reported "no OVN chassis registered in SBDB" and refused to start, cascading into the e2e + e2e-go failures on run 26430839508. Drop the hostname fallback. If --node-name is given, pin both the on-disk file and the running OVS DB to it (callers that care about IPsec cert identity matching still get it). If no --node-name is given, leave the existing system-id alone — that's always safe whether the value came from a gold-image UUID, bootstrap-install.sh writing nodeN before this script runs, or a manual ansible value. * fix(network/reconcile): skip orphan prune on startup; drift owns GC The vpcd startup reconcile races against the daemon's EnsureDefaultVPC: the daemon writes SG KV rows + publishes vpc.create-sg on each node in parallel with vpcd starting up. The leader-gated reconcile can load intent before any SG row is visible on its local JetStream replica while subscribers on peer nodes are concurrently creating port groups from the same vpc.create-sg events. A startup orphan-port-group sweep then deletes managed sg_* port groups that intent hasn't caught up to. CI run 26434124790 trace: node2 reconcile/lock: elected holder=env20-bottlebrush node2 reconcile: starting intent_sgs=0 node2 reconcile/apply: removed orphan port group pg=sg_161c70ce23e704a2a node2 reconcile/apply: removed orphan port group pg=sg_a0d078650a3150391 node2 reconcile/apply: removed orphan port group pg=sg_54b564b374e9162bb Every subsequent RunInstances → vpc.create-port failed with "port group not found" for ~3 minutes until the 5-minute drift tick recreated the port groups. Tests give up after 60s; six multinode suites cascade-fail on InsufficientInstanceCapacity. Split the reconciler into Reconcile (full prune, drift loop) and ReconcileApplyOnly (no prune, startup). applySGs takes a pruneOrphans bool. vpcd startup calls ReconcileApplyOnly; drift loop unchanged. Legitimate orphans get garbage-collected on the next drift tick — at worst 5 minutes of lingering rows, no production impact. Add TestReconcile_ApplyOnlyKeepsOrphanPortGroup to lock the contract. * fix(ipsec): trigger charon rereadcacerts after CA symlink install strongSwan scans /etc/ipsec.d/cacerts only at charon startup. The unit openvswitch-ipsec.service is enabled at provision time and starts before admin init writes the cluster CA symlink, so charon ends up running with zero CAs loaded. Every IKE_AUTH then fails with `no trusted RSA public key found for '<peer>'` and no SAs negotiate — the failure mode exposed by TestMultinodeIPSec. Append a stroke rereadcacerts call at the end of installCAIntoCharonTrustStore so an already-running charon picks up the freshly-placed CA without a service restart. Failure is swallowed at debug — a missing/stopped charon at admin-init time is expected and a later startup will scan cacerts naturally. * refactor(network/host): move IPSec orchestrator from daemon (ADR-0006 S8) ADR-0006 S8: "IPSec is OVN-native only. No layer implements custom IKEv2, XFRM rules, or direct strongSwan management. IPSec SA lifecycle is delegated entirely to OVN native IPSec and is invisible above L0." The enableOVNIPSec orchestrator and its ensureOVSMonitorIPSecActive helper lived in daemon/ipsec.go but composed L0 primitives (SetIPSecCertPaths, EnableIPSecEncapsulation, SetNBGlobalIPSec) that already lived in network/host. The orchestrator now lives alongside its callees as host.EnableOVNIPSec, taking configPath and clusterConfig from the daemon at the call site. Helper and test-override package vars (systemctlActiveTimeout, ovnNBSocketPath) move with it; tests are translated to call EnableOVNIPSec directly rather than via the Daemon struct. Closes mulga-siv-142 * refactor(daemon): use topology.Port for OVN port names (ADR-0006 S4) ADR-0006 S4 forbids any layer above L2 from constructing OVN object names. Three daemon sites formatted "port-" + ENIId inline; route them through topology.Port instead. Closes mulga-siv-143 * refactor: use topology.Port for OVN port names in handlers + vm (ADR-0006 S4) Replace inline "port-"+eniID formatting across five sites (eip, instance x2, vpc/eni, vm/network_helpers) with topology.Port. OVSIfaceID stays as a thin wrapper to preserve the OVN binding contract comment and existing call sites. Closes mulga-siv-144 * test(network/invariants): widen scan roots to daemon/handlers/vm/testutil/utils (ADR-0006) Widen ADR-0006 invariant scan roots so drift in upstack consumers surfaces in CI. Backlog of violations in widened roots was drained by mulga-siv-142/-143/-144; this commit makes the wider scan permanent. - S2 (host_containment): add vm/ alongside network/ + daemon/ - S3 (nat_mode): add daemon/ alongside network/ + vpcd/ - S4 (names): add daemon/, handlers/, vm/ alongside network/ + vpcd/; add narrow AWS-ID exemption (vpc-cidr-assoc-) so AWS resource IDs are not misclassified as OVN object names - S6 (federation): add daemon/, handlers/ alongside network/; convert single-root walk to slice - S7 (nats_subjects): add daemon/, handlers/, testutil/, utils/ — stays t.Skip pending cluster-wide subject rename; gap report now lists 58 subjects CODEOWNERS: add /spinifex/network/invariants/ rule so widening or narrowing the scan needs explicit owner review. Closes mulga-siv-145 Refs: audit-adr-0006.md * chore(e2e): disable IPsec in pseudo multi-node test Pseudo multi-node runs three simulated nodes as unprivileged tf-user. The charon trust-store install in 904b45e writes /etc/ipsec.d/cacerts, which fails with EACCES under a non-root caller and aborts admin init. Disable --ipsec for this suite — loopback-only nodes don't exercise real Geneve encryption anyway. Real-multi-node + single-node e2e (both run admin init via sudo) continue to cover the IPsec path. * fix(ovn): atomic Ensure* primitives prevent duplicate logical entities OVN NB has no unique constraint on Name. The previous GetLogicalRouter→CreateLogicalRouter sequence in EnsureVPC was a TOCTOU race: when vpc.create on one node and the defensive vpc.create-subnet EnsureVPC call on another both observed absence, both inserted, yielding two routers per VPC. The orphan held DNAT references blocking EIP detach, and DNAT/flow installation landing on the orphan caused TestMultinodeGuestSSH SSH timeouts. Add EnsureLogicalRouter / EnsureLogicalSwitch / EnsurePortGroup to the libovsdb client. Each builds a single OVSDB transaction combining a wait-op (Until=equal, Rows=[]) gated on absence of a Name match with the insert. The server serialises the wait+insert atomically; concurrent callers see exactly one survivor and the rest fall through to a re-fetch that returns the canonical row. Switch all production callers (topology.EnsureVPC / EnsureSubnet / EnsureSGPortGroup, external.AttachIGW, reconcile.applyVPCs / applySubnets) to the Ensure* variants. Bare Create* remains available to the L1 wrapper. Add invariant test TestS4_OVNNamedCreatesUseEnsure (AST walk) to block future reintroduction of the bare Create* call sites outside network/ovn. Mock implements Ensure* under its existing mutex so the concurrency tests (50/30/30 goroutines for Router/Switch/PortGroup) prove a single UUID survives. Refs: mulga-siv-146 * fix(ovn): wait-op rows must be non-empty; flip to NotEqual predicate The Ensure* primitives sent a wait-op with `Rows: []ovsdb.Row{}` plus `Until: ==` (RFC 7047 §5.2.6 absence test). The libovsdb Operation struct tags Rows as `json:"rows,omitempty"`, so the empty slice was dropped on the wire and ovsdb-server rejected every transaction with "Required 'rows' member is missing", breaking reconciler bootstrap and every downstream create-port lookup ("logical switch X not found"). Flip the predicate to the equivalent `Until: !=` formulation with a single-row projection containing the Name we want absent: no row → projection [] != [{name}] → predicate holds row exists → projection [{name}] == [{name}] → predicate fails Same atomic guarantee, JSON now well-formed. Mock tests can't catch this — only a real ovsdb-server validates the envelope. * chore: remove generateMAC wrapper * chore: remove all bead / plan references * fix(network/policy): wait for chassis flows after AddNATGateway AddEIP fires m.barrier() to block until ovn-northd compiles NB→SB and every chassis installs the new dnat_and_snat flows; AddNATGateway returned right after the NB commit. So `aws ec2 create-route … --nat-gateway-id` would publish vpc.add-nat-gateway, the SNAT row would land in NB, and the API call would succeed before the gateway chassis had programmed the SNAT flow for the private subnet. Phase 8d's 10×5s=50s ping budget races that flow-install latency on slow CI hosts (last seen run 26455602615 single-node, identical flake on unmodified dev run 26434711010 — pre-existing, not introduced by Ensure*). Mirror AddEIP: call m.barrier() after AddNAT returns. Add unit test TestNATManager_AddNATGateway_FlowsBarrier_Fires covering the new path. * chore: remove more plan refs * chore: remove GenerateIPSecPeerCertIfEnabled * chore: remove dupliate subnetGatewayCIDR function * chore(network): trim superfluous comments across network/ tree Collapse multi-paragraph godocs to 1-2 sentences, drop WHAT-restatement comments, remove section dividers grouping <10 items, and cut prose narration in tests. ADR clause references, one-line WHY comments for non-obvious OVN/OVSDB/AWS/NATS semantics, and exported-symbol godoc are preserved. No behavioural changes. Packages: invariants, topology, host, external, ovn, reconcile, policy, subscribers. * fix(igw): restore attachment State="available" for AWS parity AWS InternetGatewayAttachment.State is "available" when an IGW is attached to a VPC, per the AWS SDK doc ("the state is available when attached to a VPC; otherwise, this value is not returned"). Commit b7505f8 wrote "attached"/"detached" to align with the reconciler filter, which inverted AWS parity. Fix the producer to write "available" again and update the consumer: - handlers/ec2/igw: AttachInternetGateway and DetachInternetGateway both persist State="available"; IGWRecord comment narrowed. - network/reconcile/intent.loadIGWs: filter accepts "available" (was "attached"); attachment is identified by VpcId != "". - tests updated: igw service_impl_test, routetable service_impl_test fixture, reconcile intent_test fixture. * Update spinifex.toml * refactor(subscribers): route EIP/NATGW through L5 wrappers (ADR-0006 §S1) Subscriber previously called policy.NATManager directly for EIP and NAT Gateway operations, bypassing the L5 wrappers (EIPManager, NATGWManager) that existed on paper. ADR-0006 §S1 requires every cross-layer call to pass through the immediate lower neighbor's typed interface; IGW already followed this pattern. Wire EIPManager and NATGWManager into the Subscriber struct, construct them in vpcd alongside IGWManager, and swap the four direct-L3 call sites (AddEIP, DeleteEIP, AddNATGateway, DeleteNATGateway) onto the L5 surface (AttachEIP, DetachEIP, AttachNATGateway, DetachNATGateway). AttachEIP now fires the flows-barrier, giving callers the same "reachable on return" guarantee IGW already had. The reconciler still calls policy.NATManager directly; that's a cross-cutter and a separate concern. * test(reboot): port 8MB WAL pad workaround from bash run-reboot-e2e.sh App VMs' user-data writes (unit file + index.html) sit in the on-host WAL until WriteWALToChunk crosses the 4MB threshold. Host reboot before that loses them — systemd then refuses to load the zero-byte unit and the responder never starts. Mirror the bash driver's dd+sync pad so the WAL flushes pre-reboot. Drop when the upstream WAL fix lands. * test(e2e): dump OVN/OVS/SG state on Phase 8d NATGW failure Phase 8d's existing failure path only prints lr-nat-list and lr-route-list. CI artifacts are not uploaded when run-e2e.sh exits nonzero, so debugging the NATGW regression requires inline diagnostics in the journal. Expand the dump to include nb/sb show, chassis bindings, port_group ACLs, br-int flows filtered to the NATGW IP/private subnet, host routes, and the private VM's own ip/route/neigh state. * fix(ovn,reconcile): scope NAT deletes to router; NATGW snat per associated subnet Two compounding bugs caused Phase 8d NATGW E2E flake (~60% CI failure): 1. ovn.DeleteNAT/DeleteNATByExternalIP searched all NAT rows globally and only mutated the named router afterwards. With AWS subnet CIDRs repeating across VPCs, (type, logicalIP/externalIP) is not globally unique — the delete picked a NAT row owned by a different router, the mutate was a no-op, and the row delete failed with "remaining reference(s)", leaving the original router's NAT orphaned. Both calls now resolve the LR first and filter NAT lookups to lr.NAT UUIDs. Mock client mirrors the scoping. 2. reconcile.loadNATGWs used the NATGW's home (public) subnet CIDR as SubnetCIDR, producing an snat for the wrong CIDR that collided with the correct event-published snat. loadNATGWs now walks route tables and emits one NATGWSpec per (NATGW, associated private subnet) pair, mirroring publishNatGatewayEventsForAssociation. Intent map keyed by natgwID|subnetCIDR so multi-subnet NATGWs produce distinct specs. Tests: - nat_test.go: cross-router same-logicalIP isolation for DeleteEIP and DeleteNATGateway - intent_test.go: NATGW snat uses associated private subnet CIDR, not the NATGW home subnet; NATGW with no associations produces no spec --------- Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>	2026-05-27 21:05:01 +10:00
Josh Siviour	050ac8f861	feat(e2e): Go E2E harness + single-node migration to Ensure*/Discover* + reboot suite (#241) ... * feat(e2e): add Go test harness + cert scenario, port run-cert-e2e.sh Lands step 1 of docs/development/improvements/e2e-go-harness.md: shared harness primitives (env, wait, artifacts, cert, tls) plus the TestCertIssuance scenario that supersedes run-cert-e2e.sh. All files gated by //go:build e2e so default builds and `make preflight` are unaffected. Bash script stays in place for one CI cycle before deletion. * feat(e2e): port run-lb-e2e.sh to TestLoadBalancer Go scenario Step 3 of docs/development/improvements/e2e-go-harness.md. Replaces 1246 lines of bash with a typed Go scenario exercising all 4 LB variants (ALB/NLB × internet-facing/internal) against a shared VPC + app instance pool. Internet-facing phase auto-skips when no peer node is configured; internal phase always runs. Adds shared harness primitives: - harness/aws.go: AWS SDK client factory wired to Spinifex AWS gateway - harness/lb.go: WaitForLBActive, WaitForTargetsHealthy, WaitForENICleanup, WaitForInstanceRunning/Terminated, InstanceENI - harness/traffic.go: HTTPRoundRobin, TCPRoundRobin, distribution asserts - harness/ssh.go: minimal peer SSH wrapper shelling out to /usr/bin/ssh (replaced by typed x/crypto/ssh client hoisted from DDIL in mulga-siv-74) User-data scripts embedded via //go:embed under lb/testdata/. Bash script stays in place for one CI cycle before deletion. * fix(e2e): log client launch params + dump daemon log on InvalidInstanceType Adds diagnostics for the LB internal client launch path. Logs the exact InstanceType, AMI, subnet, key name, and user-data size right before RunInstances, and registers a daemon-log dump on failure so we can correlate the 400 InvalidInstanceType with what spinifex saw. * refactor(e2e): serialise LB suites + share probe client across internal tests Splits TestLoadBalancer into 4 sequential subtests (ALB/NLB × internet-facing/internal). Each suite creates its own TG + LB + listener, runs its traffic test, and tears down before the next starts. Replaces the per-suite throwaway client VM with a single long-lived probe server launched at fixture setup; per-suite internal traffic tests GET /probe?ip=...&proto=... against it instead of cold-booting a fresh client. Drops peak instance count from 5 to 4 (2 app + 1 LB + 1 client), so the internal phase passes on capacity-constrained dev nodes that previously hit InvalidInstanceType (see mulga-siv-76, mulga-siv-77 for the underlying spinifex bugs). * feat(daemon): make hostReserve env-configurable for dev capacity squeeze Adds SPINIFEX_RESERVED_VCPU and SPINIFEX_RESERVED_MEM_GB env-var overrides read at daemon startup. Invalid values are logged and ignored so a typo doesn't silently widen the reserve. Unblocks the LB E2E on capacity-tight dev hosts (8 vCPU box: default 2-vCPU reserve + 3x t3.nano test load leaves 0 free for the sys.micro LB VM). Lays groundwork for the planned operator-tunable [capacity] config. bead: mulga-siv-78 * fix(e2e): revert LB to per-suite client + fail-fast on terminal LB state Reverts the shared probe-server VM that wouldn't come up reliably during the InternalFacing suites — back to the proven bash pattern of launching a fresh client VM per suite (one of ALB_IP / NLB_IP populated based on the suite kind). Adds renderClientUserData + a conditional template so each client probes only its target LB. WaitForLBActive now bails immediately on terminal LB states (failed, provisioning_failed) and emits a "still waiting" log line every 30s so hangs are visible instead of silently burning the full 5-minute timeout. * ci(e2e): add e2e-go.yml workflow + GOTESTSUM Makefile opt-in Standalone workflow runs the new Go-based E2E harness on the existing ci-single self-hosted runner — provisions a tofu VM, bootstraps spinifex, installs gotestsum, runs each requested suite via the e2e Makefile, and uploads per-test artifacts + JUnit XML + JSON streams as a build artifact. Test summary parses the JUnit and emits ::error file=...:: annotations so failures surface in the PR diff. Gated to the feat/e2e-go-harness branch + workflow_dispatch so it doesn't interfere with the existing e2e.yml while the Go conversion lands. tests/e2e/Makefile gains a GOTESTSUM=1 opt-in: when set, the e2e target runs gotestsum with --junitfile and --jsonfile rooted in ARTIFACT_DIR under a suite-tagged filename derived from the first PKG component. * feat(e2e): share probe client across both internal LB suites + SG fixes - Replaces per-suite probe client with a single shared VM launched in the fixture. Saves ~60s by avoiding a cold boot between the ALB and NLB internal suites. - Client cloud-init exposes a JSON trigger endpoint on :9090 in addition to the http.server on :80 (results dir). Each suite POSTs {proto, ip, count, outfile, http_port, tcp_port} to run a probe loop against the LB private IP and writes the result file under /tmp/httpd for the driver to fetch back. - configureDefaultSG now uses the structured IpPermissions form (top-level shortcut is silently ignored by vpcd — tracked separately) and opens port 9090 in addition to 80 + 9000. - deleteLB captures the underlying sys.micro VM id via the LB ENI before delete, then polls DescribeLoadBalancers until LoadBalancerNotFound so the next suite doesn't race the capacity reclaim. - Internal_NLB sleeps 15s up front to let the ALB sys.micro deallocate fully before its own createLB fires. DescribeLoadBalancers can report the ALB gone before the vCPU/memory allocation is actually released on capacity-tight dev hosts; without this settle, NLB's createLB trips the sys.micro reserve and lands in terminal state=failed. Still cheaper than cold-booting a per-suite client. * ci(e2e): install Go on tofu VM before running Go E2E suites The tofu-provisioned VM ships with the spx binary and bash test scripts but no go toolchain, so the prior workflow's "Install gotestsum on VM" step failed with `bash: line 2: go: command not found`. Add a curl-based Go 1.26.3 install (matching go.mod) ahead of the gotestsum install and update the test step's PATH so make e2e-* runs the freshly-installed go binary. * ci(e2e): bump gotestsum to @latest for Go 1.26 compatibility gotestsum v1.12.0 pins golang.org/x/tools v0.19.0 which fails to compile under Go 1.26 with: invalid array length -delta * delta (constant -256 of type int64) Switching to @latest pulls a newer x/tools that builds clean. * ci(e2e): ship spinifex source to VM so make -C tests/e2e can run there Single-node tofu VM only has the spx binary + bash test bundle at ~/e2e-tests; the Go e2e workflow was running 'make -C tests/e2e e2e-cert' from $HOME/Development/mulga/spinifex which doesn't exist, producing: make: *** No rule to make target 'e2e-cert'. Stop. Add a checkout of the spinifex repo on the runner and rsync it (minus .git, build/, bin/, node_modules) to the VM as ~/Development/mulga/spinifex before installing Go + gotestsum. Interim solution; mulga-siv-80 tracks replacing the rsync with shipping pre-built test binaries from the runner. * ci(e2e-go): replace rsync with tar-over-ssh; VM lacks rsync bootstrap-install.sh ships a minimal binary VM with no rsync; the ship-source and collect-artifacts steps failed with 'rsync: command not found'. Stream a tarball over ssh instead — same exclude list, no extra dependency on the VM. * fix(e2e): discover single-node IPs + awsgw bind in LoadEnv run-cert-e2e.sh single mode probes awsgw.host from spinifex.toml and adds every non-loopback global IPv4 to the node list. The Go harness defaulted to 127.0.0.1 only, so CI on a tofu VM (awsgw bound to the LAN IP, not loopback) saw 'connection refused' on every TLS check. Port the same discovery: ModeSingle now walks net.InterfaceAddrs for NodeIPs and grep-parses spinifex.toml for the awsgw bind. Both stay overridable via SPINIFEX_NODE_IPS / SPINIFEX_SERVICE_IPS. * ci(e2e-go): pre-build test binaries on runner; drop go toolchain from VM Replaces the source-tree + go + gotestsum payload that the e2e-go.yml workflow previously shipped to the tofu CI VM with self-contained test binaries built on the runner via `go test -c`. testdata for the LB suite is already //go:embed'd into the binaries, so the bundle on the VM is just `<suite>.test` files (cert: 14M, lb: 17M uncompressed). Workflow changes - actions/setup-go on runner. - New step builds binaries via `make e2e-build SUITES_BUILD=...`. - Install go-junit-report on runner; convert each VM-side test log to junit-<suite>.xml after artifact pull so "Publish test summary" still parses JUnit XML. - Drop "Install Go + gotestsum on VM" entirely (saves ~60s + network). - Drop source-shipping rsync/tar (saves ~30s + a lot of bandwidth). - VM step runs `./<suite>.test -test.v -test.timeout=30m` directly. Makefile changes - New `e2e-build` target compiles each `SUITES_BUILD` entry into `BIN_DIR/<suite>.test` via `go test -c -tags=e2e`. `clean` also removes BIN_DIR. Closes mulga-siv-80. * ci(e2e-go): disable setup-go cache; upload dominates self-hosted run time * refactor(e2e): hoist generic DDIL harness primitives to shared tests/e2e/harness Step 1.5 of docs/development/improvements/e2e-go-harness.md. Promotes Cluster/Node/SSH/DaemonClient/LocalStateRevision/ResetAllNodes/Witness into the shared package so the LB, multinode, and BM scenarios in steps 4-5 can build on the same primitives instead of duplicating SSH + cluster plumbing. Moves - ddil/harness/{cluster,ssh,daemon_client,state,reset,witness}.go -> harness/ (plus the witness testdata embed dir). - ddil/harness/{fault,netem,profiles,snapshot}.go -> ddil/fault/ as a new package; fault.PartitionNode/HealNode/KillNATS/StartNATS/ RestartDaemonOnly/ApplyNetem/ClearNetem/LinkProfile*/TakeSnapshot now take harness.Cluster / harness.Node / harness.SSH explicitly. - ddil/harness/retry.go::Run retry-with-reset extracted as harness.RetryWithReset in harness/wait.go; ddil/harness/quarantine.go keeps the DDIL-only quarantine wrapper that calls into it. - WaitForMode moved from fault.go to harness/daemon_client.go since it polls daemon state and has no fault-injection concern. - The pre-hoist harness/ssh.go (PeerSSH, subprocess ssh) is preserved as harness/peer_ssh.go so the lb_test suite keeps working unchanged; the typed x/crypto/ssh client from DDIL now owns harness/ssh.go. - New harness/util.go exports ShellQuote (was unexported shellQuote in netem.go) so both ddil/fault and reset.go/witness.go can use it. Env var rename - SPINIFEX_NODES / SPINIFEX_SSH_USER / SPINIFEX_SSH_KEY replace DDIL_NODES / DDIL_SSH_USER / DDIL_SSH_KEY. Old names still honoured with a stderr deprecation note (envWithLegacy) so the DDIL CI keeps passing until .github/workflows/e2e-ddil.yml drops the old names. DDIL_QUARANTINED stays DDIL-specific. DDIL parity preserved — scenarios still call ddilh.Run for quarantine semantics; the underlying retry-with-reset is the same code path, just shared. Closes mulga-siv-74. * fix(e2e): open default SG :22 ingress via structured form before witness launch The daemon currently drops the top-level shortcut AuthorizeSecurityGroupIngress form silently (mulga-siv-79), so the default SG starts with no inbound :22 and the witness EIP awaitBaseline tunnel times out under DDIL CI. Mirror the workaround already present in lb_test by issuing the structured IpPermissions form before LaunchWitnessVM. Idempotent on InvalidPermission.Duplicate so reruns in the same cluster are safe. * ci: render JUnit into job summary + collapsible groups in noisy steps Add a reusable composite action (.github/actions/junit-report) that parses a glob of JUnit XML files, writes a totals table + collapsible per-failure details into $GITHUB_STEP_SUMMARY, and emits one ::error annotation per failed/errored test case so failures surface in the run header instead of being buried in step logs. Apply the action to both e2e-go.yml and e2e-ddil.yml (e2e-ddil gotestsum now also emits --junitfile so the action has input). Same workflows also wrap noisy phases (tofu apply/destroy, scrub-env, tar-over-ssh, per-suite test runs, cluster-log collection) in ::group::/::endgroup:: markers so the Actions UI collapses the chatter by default and the human can expand only the failing phase. Tofu output still tees to $TOFU_LOG so the artifact bundle is unchanged. Future workflows (multinode, baremetal) can adopt the same pattern via the README in .github/actions/junit-report. * test(ddil): quarantine Scenario C until predastore §2 lands CI run 26016840819 surfaced that ScenarioC_CleanPartition silently depends on predastore-ddil-hardening §2 (write-quorum with deferred parity repair). Root cause from node3 daemon log: predastore: write udp ...->192.168.0.22:9991: sendmsg: operation not permitted predastore: putObjectViaQUIC: put failed spx QMP: BLOCK_IO_ERROR instanceId=<witness> spx: StopOrTerminateInstance: Terminating spx: Instance state transition running → shutting-down The 'sendmsg: operation not permitted' is the EPERM Linux returns when iptables drops an outbound packet — exactly the OUTPUT DROP rules PartitionNode installs. The witness VM's block I/O is backed by distributed predastore, so the same partition that severs node3↔peer NATS traffic also severs the witness disk's cross-shard QUIC writes. QEMU power-downs the guest after BLOCK_IO_ERROR, the daemon's reconciler terminates the instance, and the next AssertProgressed dials a sshd that's already gone (hence the 'handshake failed: EOF' the test reports). Quarantine by collapsing C to the same scenarioSkip stub D/E/F already use, citing predastore-ddil-hardening §2 as the dependency. Flip C back to ENABLED in the same PR that lands §2 — that's when the witness disk can survive a peer partition without QEMU halting. TEST_COVERAGE.md row C flipped ENABLED → SKIPPED; added a quarantine note explaining the predastore dependency. * ci: colored phase banners + quiet-on-success tofu output Pretty up the live Actions log without flooding it. Three changes: 1. .github/scripts/ci-fmt.sh — sourceable bash helpers (banner, ok, bad, warn, note, quiet_run) using ANSI colors that the Actions UI renders. 2. e2e-go.yml + e2e-ddil.yml — tofu phases switch back to quiet redirect into $TOFU_LOG (the previous tee'd version regressed live noise vs the original silent redirect). Each phase now prints one colored ok/✗ line; on failure the tail of $TOFU_LOG dumps inside a single collapsed group, so the live log stays scannable but the failing tail is one click away. 3. junit-report render.py — compact summary: short status icons (✅/❌) in a single column, no separate err column, single-line failure list (truncated to 120 chars) instead of <details> blocks that wrap badly on narrow screens. Per-suite test runs still wrap in ::group:: with an inline cyan banner so pass output collapses and the suite name is obvious when expanded. * test(e2e): add harness.Phase / Step / Detail for human-readable test output Test fixtures were emitting flat walls of t.Logf with every line prefixed by the same `lb_test.go:61:` (file:line of the call site that the t.Helper() chain bubbles up to) and forced 6-space indent. Hard to skim, hard to find where one phase ends and the next begins. Add three helpers in tests/e2e/harness/log.go: Phase(t, fmt, ...) — flush-left colored banner via fmt.Println. Bypasses t.Logf so the line lands without the testing framework's indent/file:line prefix. Pure visual structure; the diagnostic detail stays on t.Logf. Step(t, fmt, ...) — single-line sub-phase marker via t.Logf. Keeps file:line attribution so a triager can find the source of a progress message. Detail(t, k, v, ...) — key=value resource IDs via t.Logf. Same attribution rationale as Step — IDs are diagnostic and want to land in JUnit XML bucketed with the owning test. Why the split: Phase is purely a banner; losing JUnit attribution costs nothing because banners aren't diagnostic. Step/Detail carry triage context (instance IDs, AMI, etc.) and a failing test's JUnit testcase node should still contain them, so t.Logf attribution is kept. ANSI colors are auto-disabled when stdout isn't a TTY and GITHUB_ACTIONS isn't set, so piped local `go test -v` stays clean. Apply to lb_test setupSharedFixture — five Phase banners replace the implicit run of helper logs ("Discovering cluster capacity", "Ensuring SSH key pair", "Creating shared VPC topology", "Launching app instances", "Launching probe client") so the live CI log reads as a story rather than a flat resource dump. * ci(e2e-go): strip indent + file:line on display, propagate color env over SSH Two display bugs spotted on the previous run: 1. Phase banners came out white instead of bold-cyan because the test binary runs over SSH on the tofu VM, where neither GITHUB_ACTIONS nor a TTY is present, so harness.log.go init() disables ANSI escapes. Fix: export GITHUB_ACTIONS=true and FORCE_COLOR=1 in the remote env so the helper keeps emitting color codes. 2. Every t.Logf line still rendered as lb_test.go:61: instance type: t3.nano (4-space indent + `<file>:<line>:` prefix) because t.Helper() bubbles file:line to the call site. Fix: pipe the test binary's stdout through `sed -E 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'` so the display strips both. Tee to the artifact log happens BEFORE the sed so raw output is preserved for triage; only the live UI sees the cleaned form. * feat(e2e-harness): extend harness for single-node Go suite port Stage A of mulga-siv-89 — add the primitives the single-node run-e2e.sh port consumes: - AWSClient gains IAM + STS service handles, plus NewAWSClientWithCreds for per-tenant account-scoping in Phase 8. - awserror.go: ErrorCodeIs, AssertAWSError, ExpectError — Go equivalent of the bash expect_error helper used throughout the negative phases. - poll.go: WaitForInstance/Volume/Snapshot/Image state to any target, with per-resource default timeouts tuned from the bash sleep loops. - spxcli.go: thin shellouts for spx get nodes/vms, spx top nodes, spx admin account create with output parsing for tenant-id/access-key. - rawhttp.go: signed raw POST against the AWS gateway for the Phase 8 InvalidAction probe (gateway rejects on auth before action parsing). - ovn.go: OvnNbctl/OvnSbctl wrappers + SkipIfNoOVN for Phase 8b-e datapath checks; WaitForPortGroupMember handles propagation polls. - ec2helpers.go: InstancePublicSSHHost (public-IP-or-qemu-hostfwd), LsblkRootGiB, serial-console-access trio, DiscoverDefaultVPC, idempotent AuthorizeSSHIngress. - account.go: AccountCarousel maps tenant name -> scoped AWSClient, built atop NewAWSClientWithCreds for the Phase 8 isolation tests. Pure groundwork — no new tests, no production behaviour change. The cert + lb scenarios still compile against the extended harness. * ci(e2e-go): auto-cancel + reap orphans to prevent runner wedge A cancelled run on the single-runner ci-single host has been leaving orphan tofu/terraform procs (and stale tfstate locks) behind, which caused the next push's tofu destroy to hang for 45m+ and saturate the runner host until sshd refused new connections. - Add `concurrency: e2e-go-${ref}, cancel-in-progress: true` so a new push to feat/e2e-go-harness cancels the in-flight run instead of stacking on the same self-hosted runner. - New `Reap Orphan Procs` step at job start: SIGTERM then SIGKILL any lingering `tofu`/`terraform` from the runner user, and unlink any stale `.terraform.tfstate.lock.info`. - Harden `Teardown`: drop `set -e` (scrub-env must always run as the libvirt-level fallback), reap orphans before destroy, and wrap `tofu destroy` in `timeout 600` so a wedged provider can't pin the job to its 45m timeout. * ci(e2e-go): drop push trigger, manual dispatch only Pull the on.push branch trigger so the workflow only fires from workflow_dispatch — matches the e2e-ddil.yml Phase-1 pattern. Push trigger lifts again once the single-node Go port (mulga-siv-89) is green end-to-end and run-e2e.sh is retired. * test(e2e-single): port run-e2e.sh phases 1-4 to Go Stage B of mulga-siv-89. Lands the TestSingleNode orchestrator + Fixture that threads state between sequential phase subtests, plus phases: - Phase 1: KVM check, AWS gateway TLS reachability, daemon NATS readiness polling, region/AZ smoke. - Phase 1b: spx get nodes / top nodes / get vms (single-node only). - Phase 2: nano instance-type + arch discovery. - Phase 2b: serial-console-access default/enable/disable round-trips. - Phase 3: create-key-pair + import-key-pair lifecycle; test-key-1 PEM kept for Phase 5 instance launch (cleanup deferred to teardown phase in stage G). - Phase 4: architecture-aware AMI discovery via name filter ami-ubuntu-24.04-{x86_64,arm64}, falls back from spx import output. Phase ordering is contractual (Phase 5+ depend on AMI, key pair, AZ). detectPoolMode reads [network] external_mode from spinifex.toml so stages F/G can skip pool-only phases on dev_networking fixtures. No production code changed. cert + lb scenarios continue to compile. * test(e2e-single): port run-e2e.sh phases 5-5e to Go Stage C of mulga-siv-89 — the instance/volume/snapshot/image lifecycle that everything else hangs off: - Phase 5: run-instances (default VPC + default SG with tcp/22 opened via idempotent harness.AuthorizeSSHIngress), wait running, resolve root volume from BlockDeviceMappings. - Phase 5a-pre: spx get vms now contains the running instance. - Phase 5a: describe-instances metadata round-trip (type/key/AMI/BDMs). - Phase 5a-ii: TCP wait, ssh id, lsblk cross-check vs API root volume size (±1 GiB tolerance — lsblk rounds down), soft hostname check. - Phase 5a-iii: get-console-output round-trip on InstanceId. - Phase 5b: create-volume (10 GiB) → modify-volume (20 GiB) → attach-volume → detach-volume → delete-volume, polling each state transition through harness.WaitForVolumeState. - Phase 5b-ii: describe-volume-status on the root volume. - Phase 5c: create-snapshot → wait completed → copy-snapshot → both visible → delete original → copy survives → delete copy. - Phase 5d: AMI snapshot-backed launch verification via DescribeImages.BlockDeviceMappings[].Ebs.SnapshotId. Predastore-side config.json reads are TODO (no S3 client in harness yet). - Phase 5e: create-image --no-reboot, wait available, capture backing snapshot ID for Phase 9 cleanup. Phase 7a (attach to stopped → error) stubbed in volume_test.go for Stage D to fill once the lifecycle stop/start lands. * test(e2e-single): port run-e2e.sh phases 5f-8 to Go Stage D of the single-node Go port (mulga-siv-89). Adds four files under tests/e2e/single/ covering Phase 5f (SG egress ACL enforcement), Phase 6 (tag CRUD), Phases 7 / 7a / 7b / 7c-pre / 7c (stop, start, attach-to-stopped negative, ModifyInstanceAttribute, reboot, multi-count RunInstances), and Phase 8 negatives (18 sub-tests asserting AWS error codes via harness.ExpectError / AssertAWSError / PostAWSAction). Wired into TestSingleNode in main_test.go; phase ordering preserved because Phase 7 chain mutates the shared instance state (stopped at end of 7, .small after 7b, running after 7c-pre, untouched by 7c siblings) and Phase 8 negatives need a running instance with the AMI/key created earlier. Removes the placeholder phase7a_AttachToStoppedError stub from volume_test.go (real impl now in lifecycle_test.go). run-e2e.sh stays in tree until step-7 cleanup per the parent plan. * ci(e2e-go): wire e2e-single target + add single to build set Lets the in-progress single-node Go port (mulga-siv-89, Stages A-D landed) be exercised in CI via explicit `suites: e2e-single` dispatch input ahead of Stage G flipping the workflow default. Workflow default suites unchanged until Phase 9 teardown lands. * test(e2e-single): port run-e2e.sh IAM 1-7 + Phase 8 account scoping to Go Stage E of the single-node Go port (mulga-siv-89). Adds two files under tests/e2e/single/: - iam_test.go (903 lines): IAM Phases 1-7 — User CRUD, Access Key Lifecycle, User Authentication (via harness.NewAWSClientWithCreds), Policy CRUD (+ CreatePolicyVersion / SetDefaultPolicyVersion / DeletePolicyVersion which bash lacks), Policy Attachment & Enforcement (allow / deny / wildcard / root bypass), Policy Lifecycle (DeleteConflict while attached), and idempotent Cleanup. - account_test.go (1186 lines): Phase 8 EC2 Account Scoping — 11 Steps (Account Setup, Instance / Volume / KeyPair / Snapshot / VPC-Subnet / IGW-EIGW Scoping, Account Settings, Global Resources, Edge Cases, Cleanup). Uses harness.AccountCarousel for the multi-profile credential map and SpxAdminAccountCreate for admin account bootstrap. Fixture extended with IAM cross-phase state (IAMAdminAccount, IAM{Alice,Bob,Charlie}{KeyID,Secret}) so Phase 7 cleanup is defensive even when an intermediate IAM phase short-circuits. All 11 Phase 8 Steps share local state in the outer function; t.Cleanup mirrors Step 11 LIFO so a mid-Step failure still tears down accounts. Wired into TestSingleNode. run-e2e.sh stays in tree until step-7 cleanup per parent plan. * fix(e2e-harness): use kebab-case is-default filter for DescribeVpcs AWS EC2 DescribeVpcs filter names are kebab-case (is-default), and the daemon's filter allowlist enforces that. The harness sent the camelCase SDK field name (isDefault), so the daemon returned InvalidParameterValue 400 and DiscoverDefaultVPC fataled — leaving Phase 5 InstanceID empty and cascading 16 downstream Single-Node E2E subtests to fail. * test(e2e-single): port run-e2e.sh phases 8b-8e to Go Stage F of the single-node Go port (mulga-siv-89). Adds three files under tests/e2e/single/: - vpc_test.go (553 lines): Phase 8b (pool-mode VPC public/private subnet E2E — CreateVpc, dual subnets with MapPublicIpOnLaunch assertions, IGW attach, custom route table + 0.0.0.0/0 route, instance launch + SSH + external connectivity probe) and Phase 8c (route table validation — main RT association, local + IGW route presence, custom RT CRUD + association/disassociation, negative delete-local + delete-main assertions). - natgw_test.go (467 lines): Phase 8d (pool-mode NAT GW E2E — CreateNatGateway with EIP, private subnet route via NAT, bastion- hopped SSH into private VM, baseline egress drop / post-NAT egress succeed / post-teardown egress lost). Inline waitForNATGatewayState helper bounds the pending->available->deleted state machine via harness.EventuallyErr. - datapath_test.go (326 lines): Phase 8e (OVN-gated SG-to-SG datapath — two SGs with UserIdGroupPair ingress rule, OVN port-group membership + SB address-set wait loops, allowed traffic curl, denied traffic nc, revoke->drop, re-add->restore). Pool-mode gating via fix.PoolMode for 8b/8d; OVN gating via harness.SkipIfNoOVN for 8e. Per-phase t.Cleanup LIFO ensures teardown unwinds even on partial failure. Wired into TestSingleNode. * test(e2e-single): port run-e2e.sh phases 9, 9a, 9b to Go Stage G of the single-node Go port (mulga-siv-89) — final teardown + post-teardown verification. Adds tests/e2e/single/teardown_test.go (226 lines): - phase9_Teardown: DeleteSnapshot(CustomAMISnapID) -> DeregisterImage (CustomAMIID) -> TerminateInstances(InstanceID) + wait terminated -> DeleteKeyPair(KeyName). Each call swallows *NotFound via harness.ErrorCodeIs; unexpected errors logged as warnings so 9a/9b always run. - phase9a_VerifyTeardown: DescribeInstances must show terminated or NotFound; DescribeImages must NOT include the deregistered AMI; DescribeKeyPairs must NOT include the deleted key. Clears the consumed Fixture fields. - phase9b_FinalClusterStats: harness.SpxGetVMs must NOT list the terminated InstanceID; orphan i- rows surface as soft Logf warnings to avoid flakes when parallel suites share the node. Wired into TestSingleNode. With this, mulga-siv-89's single-node Go port (Stages A-G) is feature-complete; run-e2e.sh remains in tree until a separate cleanup bead deletes it after CI burn-in. * ci(e2e-go): unbuffer sed in test pipeline for live log streaming GNU sed defaults to block-buffered stdout when stdout is a pipe (GitHub Actions runner). Accumulates 4-64KB before flushing, so test output appeared in sporadic batches unaligned to test boundaries. -u flag forces line-buffered output so each go test -v line surfaces in the Actions UI as it is produced. Side effect: none. No process count change, no SSH session change, no test execution change. Pure output flushing on the runner host. * fix(e2e-single): persist key PEM across phases and skip unimplemented IAM ops Three bugs blocked the Single-Node Go E2E suite after the DescribeVpcs filter fix: 1. Phase 3 wrote the SSH PEM into `t.TempDir()`, which Go cleans up when the Phase 3 subtest exits. Phases 5a-ii / 5a-iii / 7b / 7c-pre then saw "Permission denied (publickey)" / "Identity file not accessible" and the cascade left the instance in a state that broke 7c-pre's `start-instances`. Move the PEM under a TestSingleNode-scoped `fix.TmpDir` so it survives until the whole test exits. 2. IAM Phase 2's `require.NotEmpty(t, fix.IAMAdminAccount, "")` was intended as a no-op (Phase 4 populates the field), but require.NotEmpty fails on empty regardless of message. Removed the bogus check. 3. The daemon does not yet implement `UpdateUser` (IAM Phase 1) or the policy-version family (IAM Phase 4): `CreatePolicyVersion`, `ListPolicyVersions`, `SetDefaultPolicyVersion`, `DeletePolicyVersion`. They return `InvalidAction` and abort the phase. Wrap each block in a t.Run sub-test with t.Skip so the surrounding phase still runs and the skips show up explicitly in the test report. * test(e2e): parallelise hot phases + tune fast-path polls Squash merge of feat/e2e-parallel-quickwins (mulga-siv-96). Four quick wins from decomposing run-e2e.sh into Go phase subtests: - Phase 8 negatives (single/negative_test.go): t.Parallel() inside each of the 18 read-only error-path sub-tests. ~5-10x speedup. - IAM Phase 4 policies (single/iam_test.go): EC2ReadOnly stays seeded sequentially (populates fix.IAMAdminAccount); the other four policies fan out as siblings under a wrapping create_policies_parallel t.Run. ~5x on the Phase 4 sub-step. - Volume + snapshot fast paths (single/{volume,snapshot}_test.go): pass harness.WithPoll(500ms) on the five state transitions that finish in <1s on local QEMU. Halves Phase 5b / 5c wall-clock. - Cert per-IP loop (cert/cert_test.go): t.Parallel() inside the 4 inner t.Run(ip, ...) blocks. Cuts cert phase by len(nodes). No harness changes, no Fixture changes, no test-execution-order changes. mulga-siv-96. * test(e2e-lb): resolve peer availability before shared fixture setup setupSharedFixture builds a VPC + 2 app VMs + probe client (minutes on capacity-tight dev hosts) and previously ran BEFORE the peer pickup + ssh.Ping. On single-node, the InternetFacing_* subtests would correctly t.Skip on peer == "" — but only after the fixture work and once subtests started, so the wait appeared to be attached to the first subtest name gotestsum printed. Reorder: pickPeer + ssh.Ping first, log "internet-facing will skip" immediately, then build the fixture for the internal subtests. No behavior change for multi-node + peer-reachable; on single-node the skip rationale surfaces before the fixture wait. * test(e2e-single): extend SSH wait budget to 3m on slow boot paths CI run 26034322018 timed out Phase5a_ii_SSH (60s budget) and Phase8b_VPCSubnetE2E (2m budget) on the TCP-port-open poll while sshd was still finishing pam/cloud-init. Bump all four wait sites to 3m as a stop-gap: - single/instance_test.go phase5aii_SSHProbe - single/lifecycle_test.go waitForSSHReady (Phase 7b/7c/7c-pre) - single/vpc_test.go phase8b_VPCSubnetE2E - single/natgw_test.go phase8d bastion SSH wait Proper fix tracked under mulga-siv-101 (hoist a harness.WaitForSSH that probes the actual handshake, not just tcp/22 reachability). * test(e2e-single): skip-gate IAM3/IAM5/IAM6 until daemon handlers land CI run 26034322018 surfaced three IAM daemon gaps: - IAM3_UserAuthentication: alice's freshly-created access key returns 403 from DescribeInstances — daemon doesn't honour scoped IAM key signatures yet. - IAM5_PolicyAttachmentEnforcement: cascades from IAM3 (alice can't sign). - IAM6_PolicyLifecycle: DetachUserPolicy returns 404 even on attachments Phase 5 confirmed — daemon attachment ledger gap. Skip-gate each phase with t.Skip + a pointer to mulga-siv-100 so the remaining IAM coverage (IAM1/IAM2/IAM4/IAM7) keeps surfacing regressions, and the suite as a whole doesn't fail on known gaps. Flip to real assertions once the daemon handlers land. * test(e2e-single): probe full SSH handshake (not just TCP port open) CI run 26036025293 hit "Connection timed out during banner exchange" on Phase5a-ii / 8b / 8d even with the 3m TCP-port-open budget: tcp/22 opens immediately, but sshd is still finishing pam/cloud-init, so the first real runSSH call afterwards races the half-open daemon. Switch waitForSSHReady to issue an actual ssh BatchMode + ConnectTimeout handshake against the target, retrying until the daemon completes the banner exchange. Drop the four TCP-only Eventually loops in favour of the upgraded helper: - instance_test.go phase5aii_SSHProbe - lifecycle_test.go waitForSSHReady (Phase 7b / 7c-pre / 7c) - vpc_test.go phase8b_VPCSubnetE2E - natgw_test.go phase8d_NATGateway (bastion) Signature: waitForSSHReady(t, host, port, keyPath). Backwards-compat alias waitForSSHHandshake exposed for the instance_test caller. Proper harness hoist tracked under mulga-siv-101. * test(e2e-harness): dual-emit Step + Detail for live CI streaming go test's testing framework buffers t.Logf output per-test and releases it only on PASS/FAIL. Long subtests (Phase8Acct_AccountScoping ~115s, Internal_NLB ~60s, Phase8b_VPCSubnetE2E 2-3m, Phase8d_NATGateway ~90s) therefore went silent in CI for the entire subtest duration, even with sed -u line-buffering in the workflow pipeline. harness.Phase already wrote to os.Stdout directly for this reason. Extend the pattern: have Step and Detail dual-emit — write the same line to os.Stdout (streams live) AND t.Logf (keeps JUnit per-test attribution). The duplicate on a failure is acceptable; both copies carry identical content. mulga-siv-103 (note: numbering skipped past siv-102 which was already filed in the parent branch). * test(e2e-harness): drop t.Logf duplicate from Step + Detail The dual-emit (Fprintln + t.Logf) caused every Step/Detail line to appear twice after long subtests: once live via stdout, once again when the testing framework released its per-test buffer at PASS/FAIL. Most visible after lb subtests (Internal_NLB ~60s) where the bulk release dumped ~dozens of duplicated lines at once. Drop the t.Logf copy. go-junit-report parses stdout between `=== RUN` and `--- PASS/FAIL` markers, so per-test attribution in JUnit XML is preserved without the duplicate. Refs: mulga-siv-103 * test(e2e-harness): stream lb-waiter progress live in CI WaitForLBActive, WaitForTargetsHealthy, WaitForENICleanup, and WaitForInstanceRunning emitted progress via t.Logf, which the testing framework buffers per-test. After the previous commit dropped the dual-emit, that meant every poll loop went silent until subtest end — visible in CI as 'nlb is active' hanging on screen for the whole duration of the subsequent traffic check. Route progress through harness.Step (direct stdout). Also emit a 'waiting for X' Step at the start of each helper so observers know what the silent stretches actually represent. Refs: mulga-siv-103 * test(e2e): dump VPC/IGW datapath diagnostics on Phase8b SSH timeout Phase8b's 3min SSH timeout on a fresh-VPC VM (CI run 26037892507) is the customer-visible surface of mulga-siv-105 — vpcd's handleIGWAttach returns before ovn-northd compiles + the gateway chassis installs flows. Two other cleanup-only bugs (siv-106 missing spinifex:cidr, siv-107 dnat_and_snat double-delete) showed up in the same triage. Keep the test hard-failing so the regression stays visible, but dump a triage bundle on timeout so each occurrence is actionable: - guest console output (last 4KB — boot + cloud-init state) - ovn-nbctl show + ovn-sbctl show - Logical_Router external_ids (confirms siv-106 reproduction) - NAT rules (confirms siv-105 + siv-107 surface) - Port_Binding chassis,up (confirms gateway port up state) Diagnostics go straight to stdout (Step/Detail rationale) so they stream live in CI rather than dumping at PASS/FAIL. Refs: mulga-siv-105, mulga-siv-106, mulga-siv-107 * ci(junit-report): drop parent t.Run containers from tally + list skips with reasons GitHub Actions test summary was counting Go's top-level Test* parent plus its failing subtest as two distinct failures (the parent is just a rollup of its children). The summary now de-dupes by dropping any test case whose name is a strict prefix of another's, so the per-file counts and the Failures block both match the actual leaf-test outcome. CI run 26037892507 was reporting '2 failed' for single when it should have been 1. Also adds a collapsible Skipped section listing each skipped leaf test with its t.Skipf call-site reason (the .go:<line>: line from the CDATA). ANSI escapes from harness.Phase banners are stripped so the reason is plain text. Reasons are useful triage signal — current run has 9 skips spread across daemon gaps (mulga-siv-100) and pool-mode gates (InternetFacing_*). Refs: mulga-siv-89 * ci(e2e-go): propagate cancel to remote .test procs + nuke VM before destroy Two related leaks made cancelled runs strand QEMU work on the libvirt host: 1. The long `ssh tf-user@VM "for SUITE in …; do ./$SUITE.test"` ran without -tt, so when the runner cancelled the step the local ssh died but sshd never sent SIGHUP through (no PTY). The remote `.test` binary orphaned and kept running until its own 30m `-test.timeout`, holding tmp dirs, NATS connections, and qcow2 file handles. 2. Teardown went straight to `tofu destroy` without first killing any stragglers on the VM, so destroy raced the .test's open file handles on the root disk — sometimes successfully, sometimes leaving the libvirt host with a half-released qcow2. Fix: - Allocate a PTY on the run-suites ssh (`-tt`) so sshd HUPs children when the local ssh dies. - Add a trap (EXIT INT TERM) in the run-suites step that ssh's back to the VM and pkill's `/spinifex-e2e-bin/*.test` belt-and-braces. - Mirror the same pkill at the top of Teardown so destroy runs against a quiesced VM. Both cleanups are scoped to the per-run wan_ip — safe to land while multiple ci-single runner agents run concurrent jobs against different tofu envs on the same host (we deliberately keep that parallelism). Refs: runner host wedged 2026-05-19 (post-cancel orphans suspected; no journal access yet to confirm). * ci(e2e-go): scope orphan reap to GITHUB_WORKSPACE so it spares sibling runners Reap Orphan Procs (job start) and the in-Teardown pkill block were using `pkill -u $UID_SELF -x tofu/terraform` plus `find $HOME -maxdepth 6 -name .terraform.tfstate.lock.info`. The single host runs 5+ self-hosted runners all as `tf-user`, so every e2e-go run was SIGKILLing in-flight tofu from sibling workflows (e2e.yml, e2e-ddil.yml, e2e-baremetal.yml, ...) and deleting their state-lock files — exactly the kind of cross-runner collateral the reap was meant to recover from. Scope by `/proc/$pid/cwd` starting with $GITHUB_WORKSPACE; each runner's workspace dir is unique on the host, so only this run's own tofu procs match. Find for stale lockfiles is likewise scoped to $GITHUB_WORKSPACE. * ci(e2e-go): bump setup-go v5->v6, upload-artifact v4->v7 GitHub now warns Node 20 actions are deprecated. e2e-go.yml lagged the rest of the repo; bump to match versions already in use: - actions/setup-go@v5 -> @v6 (matches spinifex.yml) - actions/upload-artifact@v4 -> @v7 (matches release.yml, e2e-ddil.yml) checkout@v6 and create-github-app-token@v3 are already current across all workflows. The local junit-report action is `using: composite` (shell-only), so the Node runtime change does not apply. * fix(vpcd): add flows-ready barrier after IGW attach (mulga-siv-105) handleIGWAttach wrote the external switch + gateway port + LRP + default route to OVN NB and returned immediately, before ovn-northd had compiled NB -> SB and before ovn-controller on the gateway chassis had installed the resulting flows. A subsequent RunInstances -> vpc.add-nat could complete while the datapath was still dark; the launched VM was then unreachable on its public IP for whatever flow-install latency the chassis happened to incur. Customer-visible as 'I just created a VPC, launched a VM, and can't SSH in' for ~minutes. Reproduced deterministically in CI run 26037892507 Phase8b: a VM in a fresh VPC was unreachable at 192.168.0.202:22 for 3+ minutes, while a VM at the same public IP in the warm default VPC (Phase8d) was reachable in ~40 seconds. Block at the end of handleIGWAttach on `ovn-nbctl --wait=hv sync`, which bumps NB_Global.nb_cfg and waits for every connected chassis to acknowledge the new sequence number — i.e. flows installed. Bounded by --timeout=30; overruns log a WARN and continue (a 30s OVN stall is already a deeper problem than failing VPC create would surface). waitForFlowsHV is a package var so tests can stub it; in environments without ovn-nbctl on PATH the shell-out fails fast and the WARN path returns nil, so existing IGW tests stay green without a live OVN. Refs: mulga-siv-105 * fix(vpcd): store spinifex:cidr on VPC create (mulga-siv-106) handleVPCCreate built the OVN logical router with only spinifex:vpc_id and spinifex:vni in ExternalIDs. Cleanup paths (handleIGWDetach + NAT sweeps) read external_ids['spinifex:cidr'] to scope SNAT removal to the VPC's own CIDR; when unset, they fell back to an overbroad 10.0.0.0/8 and emitted a WARN. reconcileVPC already stored the same key at startup; the live-create path drifted. Add the field to handleVPCCreate, log the cidr alongside vpc_id, extend TestTopologyHandler_VPCCreate to assert the external_id is set. Refs: mulga-siv-106 * fix(vpcd): make handleDeleteNAT idempotent on 'already absent' (mulga-siv-107) Instance termination fans out through multiple cleanup paths (instanceCleanerAdapter.ReleasePublicIP AND VPCServiceImpl.DeleteNetworkInterface), both of which publish vpc.delete-nat for the same (vpc, logical_ip). The publish is fire-and-forget (utils.PublishNATEvent uses PublishEvent for vpc.delete-nat, not RequestEvent), so callers never branch on the response. The second publisher then triggered a spurious 'ERROR vpcd: failed to delete dnat_and_snat rule ... not found' log line ms after the first delete succeeded — confusing triage signal with no actionable consequence. Add ErrNATNotFound sentinel to the OVN client interface and wrap the not-found return paths in both LiveOVNClient and MockOVNClient. handleDeleteNAT matches errors.Is(err, ErrNATNotFound) and treats that as success (logs INFO 'already absent (idempotent)'). Existing TestTopologyHandler_DeleteNAT_NotFound asserted the opposite contract; the comment claimed EIP disassociation needed the error signal, but DisassociateAddress also fire-and-forgets vpc.delete-nat, so the contract never had a real consumer. Test renamed to _Idempotent with assertSuccess. Other failure modes (router missing, OVN unreachable, transact error) still surface as ERROR + reply with the error string. Refs: mulga-siv-107 * ci(e2e-go): cluster JUnit failures + name likely root cause (#230) * ci(e2e-go): cluster JUnit failures + name likely root cause New composite action `.github/actions/e2e-analyze` runs after `go-junit-report` converts each suite log. For every `<failure>` it strips ANSI, normalises dynamic noise (EC2 IDs, UUIDs, IPs, timestamps, file:line) to a stable signature, picks the earliest non-cascade failure per suite as the likely root cause, and buckets the rest by signature. Output is appended to `$GITHUB_STEP_SUMMARY` and written to `<log-dir>/analysis.md` so it travels with the artifact bundle. Cascade markers (`must populate fix.`, `Should NOT be empty`, `Expected value not to be nil`) collapse the typical "one upstream phase fails, ten downstream `require.NotEmpty` assertions trip" pattern into a single grouped entry under the real first failure. The analyser is advisory — it always exits 0. The existing "Fail job if any suite failed" gate keeps owning exit-code semantics. Stage 1 of `docs/development/improvements/e2e-go-failure-analysis.md` (mulga-siv-102). Stage 2 (per-failure log slicing) and Stage 3 (rule-based hints via `tests/e2e/.failure-rules.yaml`, placeholder seeded here) are independent follow-ups. CI wiring: - e2e-go.yml: new "Analyze e2e failures" step between "Publish test summary" and the artifact upload, always-runs. - spinifex.yml: new `make test-actions` step so the golden-file tests under `.github/actions/e2e-analyze/testdata/` run on every PR. - Makefile: `test-actions` target; kept out of `test-cover` so CI-only tooling doesn't dilute coverage %. Golden fixtures: a synthetic cascade-vpcs case (one Phase 5 InvalidParameterValue root + 15 downstream cascades), one real single-failure case captured from run 26037892507, and a clean no-failures case. * ci(e2e-go): per-failure log bundles with time-window slicing Stage 2 of the e2e failure analyzer. For each failure surfaced by Stage 1, materialise <log-dir>/analysis/NN-<TestName>/ containing: failure.txt — summary (test, start, duration, file-site, error) journal.log — spinifex-journal.log sliced to [StartAt - 5s, StartAt + Duration + 5s] test.log — test-<suite>.log block between `=== RUN <name>` and the matching `--- FAIL` marker <copies> — any files the harness already dropped at <log-dir>/<TestName_with_/_replaced>/ Report links each entry to its bundle so a reader on the Actions page goes straight to the relevant slice. Workflow now invokes journalctl with --output=short-iso so the analyzer can parse wall-clock timestamps for the journal window. Lines without a parseable timestamp inherit the previous line's classification (keeps multi-line records and stack traces contiguous). Render output is unchanged when BundlePath is empty, so the Stage 1 golden snapshots still match. * ci(e2e-go): restore GOTOOLCHAIN=auto after setup-go v6 bump setup-go@v6 flipped the GOTOOLCHAIN default from "auto" to "local", which makes Go refuse to auto-download the toolchain version named in go.mod. The self-hosted ci-single runner ships /usr/local/go older than spinifex/go.mod's `go 1.26.3`, and build-install-artifacts.sh prepends that path so the system Go wins the resolution race. Under v5 ("auto"), Go silently fetched 1.26.3 on demand; under v6 ("local"), the Build Install Artifacts step errors out: go: spinifex/go.mod requires go >= 1.26.3 (running go 1.26.1; GOTOOLCHAIN=local) Re-set the env to "auto" so behaviour matches pre-v6 runs. * ci(e2e-go): re-export GOTOOLCHAIN=auto after setup-go Workflow-level env doesn't help: setup-go@v6 writes GOTOOLCHAIN=local to $GITHUB_ENV, which overrides workflow env in every later step. Add a dedicated step right after setup-go that appends GOTOOLCHAIN=auto to $GITHUB_ENV so the Build Install Artifacts step (which uses /usr/local/go from a stale runner path) can self-heal by fetching the toolchain named in go.mod, matching pre-v6 behaviour. * fix(vpcd): add flows-ready barrier after dnat_and_snat add handleAddNAT wrote the NAT rule to OVN NB and returned immediately, mirroring the gap siv-105 closed for handleIGWAttach but on the NAT add path. ec2d/RunInstances then returned success while ovn-northd was still compiling NB->SB and ovn-controller hadn't installed the DNAT/SNAT flows on the gateway chassis, so the freshly launched VM was unreachable on its public IP for whatever flow-install latency the chassis happened to incur. Reproduced in CI run 26072432957 Phase8b_VPCSubnetE2E: VM at 192.168.0.212 timed out at 3min while diagnostics showed Port_Binding up=true for both the VM eni and the chassis-redirect gateway. SB-side convergence was complete; the missing piece was the same `ovn-nbctl --wait=hv sync` barrier siv-105 added to IGW attach, applied at the end of handleAddNAT as well. Reuses the existing package-var `waitForFlowsHV` so tests can stub it; no new infra. Parallel unit test asserts the call site fires exactly once per add-nat. * ci(e2e-analyze): real-artifact fixes for journal slicing Three bugs surfaced after Stage 2 ran against real CI artifacts in run 26072432957: 1. Regex `[+-]\d{4}` rejected the colon-offset form (e.g. `+10:00`) that systemd's `journalctl --output=short-iso` actually emits on most versions. Added a parser that tries RFC3339 first and falls back to the no-colon `-0700` layout. Result: 0-line journal slices become populated. 2. go-junit-report's `timestamp` attr on <testsuite> records the moment the XML was produced (post-suite), not when the tests started. Computing StartAt as `that + cumulative-priors` placed every failure AFTER the journal collection ended, so windows always missed. Workflow now writes `${ARTIFACT_DIR}/test-<suite>.start` right before running each suite; analyzer prefers that file over the junit timestamp. 3. Persisted `OffsetFromSuiteStart` on each Failure so the suite-start override can recompute StartAt without re-parsing the XML. Unit tests added for the colon-offset journal format. End-to-end test (TestWriteBundles_EndToEnd) still passes with the new layout. * ci(e2e-analyze): skip parent testcases when summing suite offsets go-junit-report emits one <testcase> for the parent Go test AND each subtest, with the parent's time = sum of its children. ParseFile was summing every testcase into the suite-cumulative-offset counter, so every leaf's offset was inflated by ~the parent's total duration. Visible on run 26073817490 as Phase8b's StartAt landing 13 minutes after the suite finished collecting logs — beyond the journal window, so the per-failure journal slice was empty. Pre-compute the leaf set per suite (a name is a leaf iff no other testcase name starts with `<name>/`) and only accumulate cumul on leaves. Golden snapshots updated; the offset for the only failure in each fixture now matches the actual leaf-sum start time. * test(e2e-harness): capture IP-targeted datapath state on Phase8b SSH fail Phase 1 diagnostics for mulga-siv-111. The Phase8b SSH timeout reproduced on the e2e-failure-analysis branch with siv-105's IGW barrier AND the new add-nat barrier BOTH firing cleanly, so the residual gap is downstream of `--wait=hv sync` — most likely stale OVS conntrack, stale upstream-host ARP, or an OF flow-install miss. DumpVPCFlowDiagnostics now takes a VPCDiagnosticsOpts carrying the external IP, logical IP, and per-test artifact dir. When set, the helper greps `ovs-ofctl dump-flows br-int`, `conntrack -L`, and `ip -4 neigh show` for the external IP / logical IP and snapshots `ovn-nbctl find NAT external_ip=…`. Each capture is both printed inline (existing affordance) and written to a separate file under fix.Artifacts so the Stage 2 analyzer bundler picks it up alongside the journal + test-log slices, no analyzer changes needed. Phase8b's caller now passes pubIP / privIP / fix.Artifacts so the captures fire on the next failure; updates the t.Fatalf message to point at siv-111 instead of siv-105 (which is the now-closed parent). * fix(e2e-single): authorize SSH ingress on fresh-VPC default SG in Phase8b Phase 8b stands up a non-default VPC and runs an instance there. Its auto-created default SG denies all ingress, so the SSH probe times out even when every datapath barrier converges — OF flow captures show the SYN reaches OVN, gets DNAT'd at table 13, then drops at the ACL between tables 13 and 17. The bash driver relies on admin init having pre-baked this rule on the *default* VPC's default SG; a fresh VPC has not. Add a DescribeSecurityGroups + AuthorizeSSHIngress step between route-table setup and run-instances, using the existing harness helper. Fix surfaced via mulga-siv-111's Phase 1 diagnostics (OF flow + ACL drop counts), invalidating the H1/H2/H3 stale-state hypotheses in the bead plan doc. * test(e2e): dual-detect ubuntu image version (24.04/26.04) during gold-image transition After mulga gold image bumped to v6-fastboot (stages ubuntu-26.04 / resolute instead of ubuntu-24.04 / noble), all consumers that hardcoded 24.04 break on the new image. Make them detect at runtime so v3 and v6 envs both pass. - tests/e2e/run-e2e.sh: DescribeImages filter accepts both ami-ubuntu-{26,24}.04-<arch>, then derives IMAGE_NAME from the match. - tests/e2e/run-pseudo-multinode-e2e.sh: prefer ~/images/ubuntu-26.04.img, fall back to ubuntu-24.04.img; pass detected version to spx import. - tests/e2e/single/image_test.go: walk candidate catalog names (26.04 first, then 24.04), stop on the first one DescribeImages can resolve. - .github/workflows/e2e-baremetal.yml: download resolute (26.04) into BM_ISO_CACHE; update BM_UBUNTU_IMAGE env var path. - docs/terraform-workbooks/{bastion-private-subnet,s3-webapp,nginx-webserver}: AMI filter matches `*ubuntu-26.04*` OR `*ubuntu-24.04*` (AWS values= array is ORed). Companion changes for mulga (bootstrap scripts, baremetal, iso pipeline) land in scripts/tofu-cluster/* and scripts/{tofu-iso,baremetal,iso-builder}/* under bead mulga-siv-114. Drop the 24.04 fallbacks once all envs are on the v6 gold image and 24.04 entries are removed from spinifex/utils/images.go. * feat(e2e): add service-interfaces.yaml manifest + validator Source of truth for which spinifex services produce/consume which NATS subjects, and which e2e suites cover which services. Consumed downstream by the targeted suite selector (Bead 4) to derive subset E2E runs from a PR diff so single-service changes stop paying the full 25 min suite cost. - docs/service-interfaces.yaml: 7 services, 4 active suites (cert, lb, single, ddil). Subscribes / publishes / depends_on filled from grep of NATS subjects in source. spinifexui marked covers_no_suite so UI-only diffs skip e2e entirely. Future suites (multinode, reboot, tofu, baremetal) commented in until Go ports land. - tests/e2e/manifest-check: schema + cross-reference validator (path existence, subject regex, duplicate detection, suite covers resolves to a real service, fixtures references resolve). 8 unit tests cover the failure modes plus a load-and-validate on the real manifest. - Makefile: manifest-check target, hooked into preflight. - docs/DESIGN.md: link to the manifest under "NATS Messaging". mulga-siv-116. See docs/development/improvements/e2e-targeted-suite-selection.md. * fix(e2e): drop catalog-download import path from phase4_Image `spx admin images import --name ubuntu-26.04-x86_64` with wantErr=true was meant to detect that bootstrap-install.sh already imported the AMI and short-circuit on the duplicate exit code. After the v6 gold-image bump preloaded `~/images/ubuntu-26.04.img` and bootstrap-install.sh switched to `--file` for the import, two things broke at once: - The catalog `--name` path no longer hits a duplicate guard for the --file-imported AMI, so the import re-downloads ~857MB and either succeeds (failing the wantErr=true assertion) or wedges the job timeout while the catalog server crawls along at 16KB/s. - `--file` from the test won't work either: the test binary runs as `tf-user` over SSH, while bootstrap-install.sh placed the image at `/home/spinifex/images/...` (mode 0700 home dir). bootstrap-install.sh is already responsible for staging the AMI. The test's job is verifying DescribeImages round-trips by-name and by-ID — not re-exercising import. Trust the bootstrap-staged AMI and stop calling import. Keeps the 26.04→24.04 dual-detect for the v3 gold image transition; drop the 24.04 candidate when no envs remain on v3. Refs mulga-siv-115 (caught by the EC2 cold-boot fast-path E2E run on feat/ec2-coldboot-fastpath). * ci(e2e-go): default to e2e-single + harden status-report SHA fallback Two changes to .github/workflows/e2e-go.yml: 1. Default `suites` input now includes e2e-single so a no-input manual dispatch exercises the full single-node ladder. Single-node Go harness is greenable now (image_test fix at 10eb38a). 2. Report E2E (Go) Status job no longer gated on `inputs.sha` and falls back to `github.sha` when the input is unset. Previously a manual dispatch without a PR head SHA either skipped the status post entirely or hit `gh api repos/.../statuses/` with an empty path component (the 'invalid hash' error) — tests passed, the report step failed silently. Now status posts on every dispatch against the dispatched ref's commit when no explicit SHA given. Closes mulga-siv-112. * feat(e2e): add Ensure* fixture API with memoization + cleanup Introduces tests/e2e/harness/fixtures.go: the memoized, cleanup-aware provisioning helpers that Bead 3 (single-fixture-migration) will rewrite phase tests around. Each Ensure* helper: - takes (t *testing.T, fx *Fixture, args...) - memoizes by a key derived from args (sync.Mutex + singleflight) - registers t.Cleanup on the Fixture's parent test so memoized resources survive child-subtest lifetimes - bakes readiness polling into the helper (no sleeps in callers) Exported helpers: EnsureKeyPair, EnsureAMI, EnsureDefaultVPC, EnsureSubnet, EnsureSG, EnsureInstance, EnsureVolume, EnsureSnapshot, EnsureNATGateway, EnsureLoadBalancer. Fixture stores ec2iface.EC2API + elbv2iface.ELBV2API so unit tests can inject fakes without building a real AWSClient (TLS + env). Compile-time checks guard that the SDK concrete types satisfy the stored interfaces. Manifest fixtures: block populated — Bead 4 selector will resolve test coverage from AST-scanned Ensure* call sites via this map. Makefile: new test-harness target runs the harness unit tests under -tags=e2e (mocked, no infra). Hooked into preflight so signature drift and memo/cleanup regressions surface locally. Bead 2 of docs/development/improvements/e2e-targeted-suite-selection.md. No existing tests modified — fixture API is net-new. * feat(e2e): port run-reboot-e2e.sh to Go Adds tests/e2e/reboot/reboot_test.go — Go port of the single-node host-reboot resilience driver (cell #18). Runs from the GitHub Actions runner (not shipped to the VM like cert/lb/single) because systemctl reboot kills any in-VM process mid-step. All AWS API traffic crosses the WAN to https://WAN_IP:9999 via the SDK; SSH-level ops (reboot, ovn-nbctl, journalctl, HTTP burst) hop via harness.PeerSSH. Trust path tries to SCP the cluster CA off the VM and falls back to InsecureSkipVerify with a log message if SCP fails — the cert suite already validates real trust, so the fallback is intentional. Harness changes: - env: WANHost field sourced from SPINIFEX_WAN_IP, distinct from ServiceIPs[0] (which is the awsgw bind IP, often loopback in-VM). - aws: SPINIFEX_AWS_INSECURE=1 escape hatch when the runner has no CA on disk. Used as the fallback when CA fetch fails. - cert: SPINIFEX_CA_CERT explicit path override so the reboot test can point ResolveCACert at the SCP'd tmp file. - peer_ssh: IsReachable() probe wrapper for cleanup paths that must skip rather than fatal when the host is mid-reboot. Workflow (.github/workflows/e2e-go.yml): - New e2e_go_reboot job, opt-in via `e2e-reboot` in the SUITES dispatch input. needs: e2e_go_single to serialise on the ci-single self-hosted runner. Fetches AWS creds from the VM's ~/.aws/credentials over SSH (mirrors the cred hop already proven by e2e-ddil.yml). - e2e_go_single filters 'reboot' out of its build + run loops so the in-VM driver doesn't try to ship reboot.test to the VM and reboot it from inside. - e2e_go_status treats skipped reboot as success. Refs mulga-siv-118. Plan: docs/development/improvements/e2e-go-reboot-port.md (in the mulga parent repo). * ci(e2e-go): run reboot job in parallel with single Drop `needs: e2e_go_single` from e2e_go_reboot. Both jobs target the same [self-hosted, ci-single] label, so GitHub Actions serialises on runner allocation rather than blocking via job ordering — parallel-by- spec, queued-by-runner. Lets a reboot-only dispatch start immediately instead of waiting for an unrelated single-job pre-condition. * fix(e2e/reboot): wait for both backends to serve before post-reboot burst Phase 8.6 previously probed until ANY HTTP response from the ALB, then fired a 20-req burst. Post-reboot, only one of the two backends may be serving (the second is still bringing its responder online) while the daemon's TG cache still shows pre-reboot 'healthy' for both — so WaitForTargetsHealthy returns in ms instead of seconds, the ANY probe matches the first-up backend, and the burst lands 20/0 instead of 10/10. Probe until BOTH backends' instance_ids each appear at least once before the round-robin burst — confirms recovery has reached both, not just one. Underlying daemon-side TG cache staleness tracked separately as mulga-siv-119. This wait is the test-side guard; the daemon fix will make WaitForTargetsHealthy meaningful again post-reboot. Refs mulga-siv-118. * ci(e2e-go): skip single job when dispatch requests only reboot Add if: gate to e2e_go_single that skips the job when the workflow_dispatch suites input contains none of {e2e-single, e2e-cert, e2e-lb}. Push events and the default suites list still trigger it, so plain PR runs are unaffected. Avoids ~15min of wasted tofu provision+bootstrap+teardown when only the runner-resident reboot suite is requested. * fix(e2e/reboot): bump post-reboot LB recovery timeout to 5min Pre-reboot the test creates the LB AFTER apps are already running, so HAProxy starts probing against an already-serving responder — ~75s to healthy. Post-reboot the daemon's recovery scan launches app VMs and the ALB sys.micro VM simultaneously, so health checks start probing within seconds of app boot beginning and both targets honestly report unhealthy until the responder unit finishes starting. Cloud-init's per-instance semaphore means no shortcut path on relaunch, so guest second-boot can be slower than first boot. Bash run-reboot-e2e.sh hid this behind stale 'healthy' state read from the daemon's pre-reboot TG cache — fixed via ResetTargetHealthOnStartup (mulga-siv-119), which made the test honestly wait on real lb-agent reports. The 180s bash inherited turned out to be wishful thinking backed by the cache bug, not measured recovery time. 5min matches the actual post-reboot eventually-consistent window observed on run 26134156355. Refs mulga-siv-118. * feat(e2e/reboot): ssh into guest VMs on failure for HTTP responder diag Post-reboot failures bottom out at "0/2 healthy" with no insight into whether the responder unit died, port 80 isn't bound, or the OVN path is broken. Capture the keypair material from CreateKeyPair so OnFailure can hop host -> qemu hostfwd:22 -> guest and dump systemctl status, port listeners, curl localhost, and the responder journal per instance. * fix(e2e/reboot): probe app instances via public IP (no QEMU hostfwd) Previous hostfwd-SSH approach didn't apply: app VMs use pure OVN tap interfaces (qemu argv has no hostfwd=tcp:*:*-:22), so the regex never matched. Replace with host-side probes against the IGW-NAT'd public IP: ping, /dev/tcp connect, and curl. Combined with the qemu ps check and ovn-nbctl list nat dump, this localises post-reboot failures to one of: VM not running, OVN DNAT missing, responder down, or sys.micro-only reachability (path asymmetry). * feat(e2e/reboot): probe ALB + dump host OVN/OVS state on failure Last run showed apps unreachable from host post-reboot (ping 100% loss, tcp refused) even though OVN dnat_snat rules looked correct. Need to distinguish gateway/ext-bridge breakage from app-side breakage: add a parallel ALB-public-IP probe (works pre-reboot, so any post-reboot failure isolates the regression) plus a full host net-state dump (ip route, ip neigh, ovn-nbctl show, ovs-vsctl show, br-int port list). * feat(e2e/reboot): dump qemu serial console per app on failure /run/spinifex/console-i-<id>.log captures cloud-init, kernel boot, systemd, anything the guest emitted to ttyS0. Tail to 256KiB (~4x a typical Linux boot) so the latest boot lands in full plus the tail of the prior. Surfaces guest-side boot failures (kernel panic, fs corruption, cloud-init OOM, etc.) that are invisible to host-side network probes. * feat(e2e/reboot): SSH into guest VMs via IGW for responder root-cause Last run nailed the symptom: ALB returns 503, app public IPs answer TCP with RST (port 80 not bound), so reboot-e2e-httpd.service isn't running on relaunched VMs. To find out why (unit never installed? installed but not enabled? failed to start?), open tcp/22 ingress on the shared SG and add a guest-SSH diagnostic via the IGW NAT path. The diagnostic dumps the unit state, port listeners, /var/lib/httpd, cloud-init result.json, cloud-init logs, and the instance-id semaphore directory. * test(e2e/reboot): always run guest SSH diag in phase 8 Last run passed after adding tcp/22 SG ingress. Two possibilities to distinguish: (a) responder was running all along but blocked at the OVN/SG layer, (b) responder wasn't running and HC was satisfied by some other path. Capture in-guest unit + cloud-init state on every run so we can confirm which without needing the test to fail. * refactor(e2e/single): migrate VPC group to EnsureDefaultVPC [skip ci] Bead 3a of mulga-siv-120 (e2e-single-fixture-migration). Replaces the "discover default VPC + cache on Fixture" pattern in vpc_test.go, natgw_test.go, securitygroup_test.go with harness.EnsureDefaultVPC. - Adds Harness *harness.Fixture field on single.Fixture; constructed in TestSingleNode alongside AWSClient. Parent test for memoized Ensure* resources is TestSingleNode itself. - Removes fix.DefaultVPCID / fix.DefaultSGID / fix.DefaultSubnetID reads in the three target files. Fields stay on the struct until 3e+3h migrate the instance / datapath / umbrella sites. - Custom VPC graphs created by phase8b (10.99.0.0/16) and the bastion graph in phase8d remain inline — those are the resource lifecycle assertions, not prereqs. CI gate skipped per dev request — full single-node suite verifies when upstream main is unbroken. * refactor(e2e/single): migrate Phase 3 key pair to EnsureKeyPair [skip ci] Bead 3c of mulga-siv-120 (e2e-single-fixture-migration). Phase 3 now calls harness.EnsureKeyPair for the primary key pair (Phase 5+ prereq). The Ensure* path owns the create assertion (PEM shape), memoizes across phases, and registers TestSingleNode-scoped cleanup — replacing the former TODO(stage-G) deferred-cleanup marker. - Drops hardcoded "test-key-1" name; EnsureKeyPair uses scratch-suffixed name so parallel CI runs don't collide on the namespace-flat AWS key pair name. - fix.KeyName / fix.KeyPath still assigned for downstream Phase 5+ readers. Per 3e migration plan, those reads move to in-test EnsureKeyPair calls and the Fixture fields drop with 3h. - Import / describe / delete dance for test-key-2 stays inline — it is the lifecycle assertion for the import path, not a prereq. * refactor(e2e/single): migrate Phase 4 + 5e AMI paths to EnsureAMI [skip ci] Bead 3d of mulga-siv-120 (e2e-single-fixture-migration). - Phase 4 (image_test.go): discover catalog AMI by canonical name, then route through harness.EnsureAMI({Existing}) so the state-available wait + memoization live with the fixture. - Phase 5e (createimage_test.go): replace inline CreateImage block with harness.EnsureAMI({CreateFrom}). The custom AMI build now memoizes, registers DeregisterImage cleanup at TestSingleNode exit, and reuses the harness state poll. Harness fix: extend AMICreateSpec with NoReboot bool — Phase 5e bakes the AMI while leaving the source instance running for Phase 6/7 lifecycle assertions; Bead 2 missed this option. fix.AMIID / fix.CustomAMIID / fix.CustomAMISnapID stay on Fixture for downstream Phase 5/6/7 + Phase 9 cleanup. Those reads migrate with 3e/3h. * refactor(e2e/single): migrate Phase 5 instance lifecycle to EnsureInstance [skip ci] Bead 3e of e2e-single-fixture-migration. - phase5_LaunchInstance: provision via EnsureInstance + re-describe for BDM/root-volume extraction; default VPC via EnsureDefaultVPC. - phase8e_SGToSGDatapath: replace inline DiscoverDefaultVPC with EnsureDefaultVPC. createSG / runSGEInstance take vpcID/subnetID arguments so phase 8e no longer depends on fix.Default* fields. - runSGEInstance stays direct RunInstances: phase 8e VMs are test subjects (need fresh ENI/private-IP per run), not memoized prereqs. - harness.EnsureInstance memo key now includes InstanceType, SGID, and UserData hash — a second call with a different SG would have silently returned the first instance ID (Bead 2 bug). * refactor(e2e/single): migrate Phase 5b/5c volume + snapshot to Ensure* [skip ci] Bead 3f of e2e-single-fixture-migration. - phase5b_VolumeLifecycle: create+wait via EnsureVolume; subsequent modify/attach/detach/delete stay inline (test subject lifecycle). - phase5c_SnapshotLifecycle: create+wait via EnsureSnapshot. Drop the create-response State/Progress shape assertions — after EnsureSnapshot the snapshot is already "completed", and the describe block below re-asserts VolumeId / VolumeSize / Description. - harness.EnsureSnapshot now takes SnapshotSpec{VolumeID, Description} so callers can pin a description (snapshot_test asserts on it). Memo key includes description hash so distinct-description calls do not silently collide on the volumeID prefix. * refactor(e2e/single): isolate IAM stage E via parent-scoped cleanup [skip ci] Bead 3g of e2e-single-fixture-migration. - harness.Fixture: add RegisterCleanup helper that wraps fx.parent.Cleanup so test code can register parent-scoped teardown for resources created outside ensureOnce (IAM users / policies threaded across phase subtests). Registering on the calling subtest would tear down before the next dependent phase ran. - phaseIAM1_UserCRUD: register parent-cleanup for alice/bob/charlie. - phaseIAM4_PolicyCRUD: register parent-cleanup for the 5 test policies. - IAM stage no longer relies solely on phaseIAM7 to clean up after a mid-stage failure; phase 7 still runs and now serves as an idempotent post-cleanup assertion. No new Ensure* helper: IAM resources have no memoization story (users and policies are test subjects, secrets only returned once on create). IAM phases also have no EC2 prereqs, so EnsureDefaultVPC / EnsureInstance are not applicable here. * refactor(e2e/single): drop dead Fixture fields after Ensure* migration [skip ci] First half of bead 3h. Removes per-phase Fixture fields with zero remaining readers after bead 3a-3g: - DefaultVPCID / DefaultSGID / DefaultSubnetID: superseded by harness.EnsureDefaultVPC (3a wired phase8b/8c/8d/5f; 3e wired phase5 + phase8e). - OVNAvailable: never assigned anywhere — leftover from the bash port. Phase fields still in use (AMIID, InstanceID, IAM*, etc.) stay until the umbrella TestSingleNode is replaced with top-level Test* + package- level fixture singleton in the second half of 3h. * refactor(e2e): add process-mode Fixture + TestMain singleton infra [skip ci] Second half of bead 3h. Lands the infrastructure required to dismantle the TestSingleNode umbrella in a follow-up bead, without changing the umbrella body today. harness.Fixture changes: - NewProcessFixture(aws) builds a Fixture with no *testing.T parent. Cleanup callbacks queue on an internal LIFO drained by Close() — intended to be invoked from TestMain after m.Run(). Lets a singleton fixture span every top-level Test* in a package. - Existing NewFixture(t, aws) unchanged; cleanups still route via t.Cleanup. registerCleanup() and logf() helpers fork on parent==nil so ensureOnce and RegisterCleanup work in either mode. - TestProcessFixture_CloseRunsCleanupsLIFO covers Close ordering, one-shot semantics, and the no-op second-Close path. single/main_test.go additions: - TestMain owns process-level lifecycle: drains pkgFix.Harness.Close() and removes pkgFix.TmpDir after m.Run(). - requireSingleNodeFixture(t) returns the package-scoped singleton, building it lazily on first call. Skips when SPINIFEX_E2E is unset or mode != single; fails on construction error. Currently unreferenced — TestSingleNode keeps its own per-test Fixture for now. The helper exists so the follow-up bead (per-phase top-level Test* migration) has a stable API to call from day one. Deferred to follow-up bead: extract Phase 4 AMI-name discovery + Phase 2 InstanceType/AZ selection into harness helpers so each phase function can self-bootstrap, then promote phase subtests to top-level Test* and delete the TestSingleNode umbrella. * fix(e2e/harness): EnsureDefaultVPC filter name is-default not isDefault Daemon rejects the camelCase `isDefault` form with 400 InvalidParameterValue. AWS CLI / DiscoverDefaultVPC use `is-default`. Surfaced as Phase 5_LaunchInstance fatal in CI run 26140908505 — cascade-failed every downstream phase that needed fix.InstanceID. * fix(e2e/single): phase8 8j/8k probe fix.KeyName not hardcoded test-key-1 Bead 3c migrated phase 3 to harness.EnsureKeyPair which produces a scratch-suffixed name (e2e-key-<random>), not the literal "test-key-1" the bash port used. 8j_CreateKeyPairDuplicate / 8k_ImportKeyPairDuplicate kept the old name and so created a fresh key pair instead of triggering InvalidKeyPair.Duplicate — surfaced as the only two FAILs in CI run 26143227369 (everything else green after the EnsureDefaultVPC fix). Read fix.KeyName (set by phase3_KeyPairs) so the probe hits the staged primary key. Header comments updated to match. make preflight diff-coverage against origin/dev fails at 57.9% due to unrelated commits inherited from the recent rebase onto feat/e2e-go-harness; against the fork base the diff is 0 instrumentable lines (test-only change). * refactor(e2e/single): dismantle TestSingleNode umbrella, per-phase self-bootstrap mulga-siv-121. Each phase from run-e2e.sh now runs as a top-level Test* in tests/e2e/single, runnable in isolation via `go test -run TestX`. Highlights: * TestSingleNode umbrella removed. Phase ordering dropped — each Test* self-bootstraps prerequisites through harness.Ensure*/harness.Discover* + the new package-local need*/iamEnsure* helpers. * Fixture shrunk to {Env, AWS, Harness, Artifacts, TmpDir, PoolMode}. Every per-phase resource ID now lives on harness.Fixture via EnsureKeyPair/EnsureAMI/EnsureDefaultVPC/EnsureInstance/etc. * TestMain owns the package singleton Fixture; harness.Fixture.Close drains every cleanup at process exit. phase9 + phase9a (in-line teardown) deleted; phase9b retained as TestFinalClusterStats. * IAM Test* share state via sync.Once-gated helpers in iam_helpers_test.go (alice/bob/charlie users + keys + admin account). * New harness/discover.go: DiscoverDefaultAZ, DiscoverNanoInstanceType, DiscoverUbuntuAMI — memoized on the Fixture and routed through EnsureAMI so any Test* can resolve them without phase coupling. * New tests_test.go: descriptive top-level Test* wrappers (TestVPCSubnetE2E, TestKeyPairs, TestSnapshotLifecycle, …) that delegate to phaseN_X(t, requireSingleNodeFixture(t)). * test(e2e/reboot): SSH as ec2-user, dump unit + network-online state spinifex's cloud-init template hardcodes Username="ec2-user" (handlers/ec2/instance/service_impl.go:1218) regardless of AMI, so even on Ubuntu the SSH-injectable account is ec2-user, not ubuntu. Switch the diag SSH-in to ec2-user so we actually get in (was "Permission denied (publickey)"). Add diagnostics to distinguish the two leading post-reboot hypotheses: (a) viperblock write loss = unit file gone vs (b) network-online.target never fires = unit stuck on After=network-online.target. New dumps: unit file content + enable symlink, network-online.target status, networkctl, ip addr / route, prior-boot journal grep for the responder. * fix(e2e/single): resolve admin account ID before TestIAM7 deletes alice iamEnsureAdminAccountID probes via GetUser(alice). TestIAM7_Cleanup deleted alice/bob/charlie first, then called the helper at line 808 — NoSuchEntity if IAM7 was the first IAM test to run (no memoized account ID). Move the call above the deletion loop.	2026-05-20 21:04:13 +10:00