35 Commits

Author	SHA1	Message	Date
Ben Duncan	7163f7d62d	feat(oci): run Spinifex on Oracle Cloud, and fix the public-address and NAT gateway lifecycle (#1108) ... * feat: ship uninstall-spx.sh and fix the OCI bring-up path Adds scripts/uninstall-spx.sh, the inverse of setup.sh: it removes the software where node-reset.sh clears state and leaves the node installed. An operator who installed with `curl ... | bash` had no teardown path at all. State teardown is delegated to node-reset.sh rather than reimplemented, and setup.sh now ships both to /usr/local/share/spinifex so they are reachable without a checkout. It refuses on a multi-node cluster and --yes does not override that: a flag meaning "don't ask me" must not also mean "ignore a safety property". /var/lib/spinifex and every apt package are kept unless asked for, and both opt-ins confirm separately even under --yes. Proved by a full uninstall/reinstall round trip on dev-oci, which found two bugs neither prod nor dev-prod can reach, because both are ISO-installed: node-reset.sh exited 2, silently, at its last step and after the wipe. setup.sh creates the spinifex *group* and the spinifex-* service users but never a *user* named spinifex, so `getent passwd spinifex` exits 2 and pipefail carried it into set -e. The uninstaller broke openvswitch-switch on its first run by removing the socket-perms helper while leaving the drop-in referencing it, failing the unit at step EXEC with status 203. Fixed on both sides: the uninstaller removes drop-ins it wrote into other units' namespaces and restarts anything it failed, and setup-ovn.sh writes ExecStartPost=- so a missing helper can never stop another package's daemon. setup-ovn.sh also learns the shape OCI needs: when --wan-bridge names a Linux bridge it links it to OVS with a veth pair instead of enslaving the NIC, so the VNIC keeps its MAC identity and netplan keeps owning the addresses. It now fails on OVS datapath errors instead of reporting success, and the --management demotion guard keys on /etc/default/ovn-central rather than on database presence, which the ovn-central package creates on every node. Tests: scripts/uninstall-spx_test.sh, wired into make test-build-scripts along with shellcheck over the four scripts that ship to every node. * feat: allocate external IPs from the Oracle Cloud API An OCI VNIC drops any packet whose source is not a registered private IP object on it, so an external pool cannot be range math there — the address has to be created through the provider before it exists at all. Add source = "oci" and the allocator behind it. One allocation is two OCI objects: a secondary private IP that rides the wire, and a RESERVED public IP attached to it that the internet reaches. Allocate returns the public address because that is what AWS semantics are about, and records the pair so the host plumbing and release path can find the private half. The OCI objects are created before the binding is written, and dropped after it on release — both orders chosen so a crash leaves a discoverable leak rather than a record that lies. Reconcile squares the two on startup and is required, not optional: a leaked reserved public IP bills and holds one of the 50 per-region slots with nothing referencing it. It never touches an object without our display-name prefix, because the reference host carries operator-created secondaries on the same VNIC. Resolve the VNIC from an interface name by MAC through IMDS rather than by Oracle's VNIC ordering, which is not a documented contract. Generalise ExternalIPAM.EnableDHCP into InstallAllocator so the OCI wiring is not a second single-purpose installer. * fix: refuse source = "oci" outside routed mode, and cover the SDK path An OCI VNIC accepts exactly one MAC, its own, and delivers every inbound frame to it regardless of which of its addresses the packet is for. Measured on the reference host: an internet source reaches 10.200.0.144 as router-MAC to VNIC-MAC, the public address never appears on the wire at all, and every off-subnet destination resolves to the same next hop. Pool mode puts a per-VPC gateway LRP MAC and a per-rule external NAT MAC on the uplink and ARPs for the address, which policy.go:50 justifies with "the Linux bridge learns the per-rule external MAC on the veth port like any other MAC". An OCI VNIC does not learn MACs; it accepts one. So a pool-mode guest's egress is dropped and its ingress never reaches OVN. Routed mode keeps the address on the host, which already wears the VNIC's MAC. Pinning every LRP to that MAC does not rescue pool mode: they share one external logical switch and OVN resolves inbound frames by an L2 lookup on it, so duplicate MACs work for one VPC and silently misroute past that. The refusal names the MAC reason, because a refusal that only says "use nat mode" reads as arbitrary and gets worked around. Also test the SDK request shapes against a local endpoint. Two of them are wrong in ways reading does not catch: pagination stopping early reports a partial VNIC to reconcile, which then drops live bindings, and a nil privateIpId on detach is omitted from the body entirely, so OCI reads the update as a no-op, returns 200, and leaves the address attached and billing. * fix: install host firewall rules where a distro's catch-all reject cannot precede them Oracle's Ubuntu cloud image ships a catch-all REJECT in both INPUT and FORWARD. Two Spinifex datapaths were silently dead behind it on any such host. EnsureNATEgressRules probed with -C and appended with -A, so its ACCEPT rules landed after the REJECT and never matched: 24113 packets on the reject, 0 on both accepts, and guests had no egress. The comment named the case it did not handle, anticipating a FORWARD policy of DROP — which an append does survive — rather than an explicit REJECT rule, which it cannot. Insert at the head, and drain existing copies first, because a rule an earlier build appended still satisfies -C and would otherwise stay stranded across an upgrade. Nothing opened INPUT for the IMDS endpoints at all. The responder binds .254 and .253 per instance on the ime- devices, so guest requests are locally delivered and meet INPUT. Requests arrived and no reply ever left; cloud-init spent 490s timing out and reported the endpoint as disabled. With one wildcarded accept the same image boots in 18.6s and finds its datasource. The natuplink e2e check had the same blind spot the code did, asserting -C and therefore presence but never position. It now asserts the accepts precede the first reject. * feat: OCI address pairs, and an IMDS remap so the host keeps its own metadata An OCI external address is two objects, not one: a RESERVED public IP that OCI 1:1-NATs upstream and never puts on the wire, and a secondary private IP on the VNIC that the wire actually carries. Every AWS-facing record holds the public half; every datapath object must hold the private one. The mapping goes in policy.natManager and nowhere else. AddEIP, DeleteEIP, PruneOrphanEIPs and the HostEIPBinder all converge there, and reconcile rebuilds the rule set from AWS-facing intent on every pass — so a correctly mapped rule installed further upstream is swept on the next reconcile and replaced with one matching an address that never arrives. The seam is a DatapathResolver defaulting to identity, so every non-OCI deployment is unchanged. A resolve failure is fatal: falling back to the public address would install a rule that reads correctly in ovn-nbctl and blackholes. ocinet.Lookup is the OCI side — read-only, KV-backed, no client and no credentials, because vpcd is a different process from the daemon that owns the allocator and the bindings record is all either would consult. Two live bugs the in-memory test store was hiding, both found by making the fake report ErrNotFound the way the real store does: - KVStore.Mutate could not create, so every pool failed its own first allocation after OCI had already created and billed the pair. - Reconcile treated a missing record as a failure, aborting the one pass that exists to find an object created before its binding was written. Guest IMDS must stay on 169.254.169.254, which is also where a cloud host's own metadata service lives — the first guest launch made it a local address and took the host's metadata and DNS with it. The datapath is OpenFlow steered, so the guest-facing addresses stay put and only the host-side endpoint address moves, with a PREROUTING DNAT on ime-+ translating and conntrack un-NATing the reply. Off by default; identity when unset. The IMDS firewall rules move to vpcd. They are wildcarded over every endpoint rather than per-launch work, and spinifex-daemon holds no iptables grant and no CAP_NET_ADMIN by design, so attempting them aborted every launch. * fix: name the real constructor in the IMDS bind-address error, and mark the NAT datapath test in-package The error string pointed at DefaultHostBindAddrs, which does not exist — a caller following it would have looked for a symbol that was never written. nat_datapath_test.go reuses nat_test.go's in-package fixtures, so it carries the //test:in-package marker the package check asks for. * docs: index what each e2e suite proves Written because a datapath comment was read as an end-to-end behavioural claim and the suite that already answered it was found only after the wrong answer had been given. TestVPCEgressPaths/NATBaseline is the authority on private subnet egress, and nothing pointed at it. * fix: source-route an EIP's reply out the VNIC that holds it An EIP's reply carries the EIP as its source. On a host that pins its uplink with a policy rule, the reply took the main table instead and left by the wrong interface — which a cloud VNIC drops, because the address is registered on a different one. Ingress reached the guest and nothing came back. Every bind now mirrors the uplink's rule onto the EIP. The selector is the rule rather than the interface: both OCI VNICs sit in one subnet, so "the interface owning the EIP's subnet" names two of them, and two candidate rules is a refusal rather than a guess. A host with no source routing gets no rule and no change. * docs: name the setup-ovn recipe routed mode actually needs on OCI The OCI guide told operators to run `--wan-bridge=br-wan`, which is mutually exclusive with `--nat-uplink` and deletes the spx-nat transit veth routed mode runs on. The node then refuses to bring the uplink up, with an error naming the flag the guide should have used. Several messages also claimed routed mode is outbound-only with no public IPs. That stopped being true when the routed EIP path landed, and it reads as "what you are about to do cannot work" during the one init the OCI path documents. * feat: let routed NAT form a multi-node cluster, and carry every pool through formation Lifts the --nodes >= 2 refusal on --external-mode=nat. The refusal carried no technical justification -- it was a v1 scope decision -- but routed NAT is the only datapath a cloud VNIC can carry, so "routed NAT is single-node" had become "Spinifex on a cloud host is single-node". It is not fixed by lifting it. Measured on three OCI nodes: the cluster forms, all chassis register and guest-to-guest traffic runs at 7.1-7.7 Gbit/s, but every node claims every EIP, nodes disagree on the VPC's gateway LRP, and only the gateway chassis reaches its own guests. init now says so on stderr rather than refusing, because the refusal made the defects unreachable and therefore unfixable. Distributing the EIP datapath is the actual fix. Lifting it exposed a second defect that stops formation outright. NetworkConfig held nine flattened Pool* scalars and the init node set PoolName = "wan" unconditionally; routed NAT renders two pools and the one that matters -- nat-transit, carrying the gateway-LRP range -- was dropped. Each joiner got a single pool named "wan" holding the transit gateway and the detected WAN prefix, and the leader re-rendered its own config from the same struct, so all three nodes were wrong. Pools is now []admin.PoolData in render order, and PoolData gains json tags because it is on the wire now rather than a template input. * feat(install-node): add --external-mode so a cluster can be formed in routed NAT install-node.sh could not express routed NAT at all. It required --external-pool/--external-gateway/--external-prefix-len and passed --external-mode=pool to every spx admin init unconditionally, so the only supported way to form a routed-NAT cluster was by hand. --external-mode nat refuses the three pool flags rather than ignoring them, and threads --nat-uplink into setup-ovn.sh -- without it the WAN interface check rejects a physical NIC during formation, which is correct in pool mode and wrong in routed mode, where nothing is bridged. * fix(ocinet): stop each node deleting its peers' OCI addresses on startup Reconcile compares the binding record against the private IPs on one VNIC. The record is one key per pool and therefore cluster-wide, so on more than one node every peer's binding looks exactly like a binding whose objects OCI has lost. Each node dropped them, and direction 1 on the owning node then collected the now-unclaimed objects as leaks -- deleting the reserved public IP and private IP of a running instance. Observed on the three-node OCI cluster: node2's startup pass logged "collected a leaked address" for an address it had allocated ten minutes earlier and "dropped a stale binding" for one living on node3's VNIC. The damage did not stop at the OCI objects. With the binding gone, Lookup.DatapathIP returns the public address unchanged, so the OVN dnat_and_snat row carries the public address instead of the private half. Guest egress then leaves the VNIC sourced from a reserved public IP, which OCI drops because it enforces the registered source per VNIC -- and EnsureEIPSourceRoute no-ops too, since a public address shares no subnet with any host address. One destroyed binding takes out both directions. Binding.VNICID already exists for exactly this ("how a node that comes back from the dead tells mine from moved on without me") and Reconcile never consulted it. A binding is now only dropped when it names this node's VNIC; one naming no VNIC cannot be attributed and is left alone rather than guessed at. * fix(host): find the EIP's source-routing table when the uplink is held as a /32 uplinkSourceRuleFor identified the interface owning an EIP's subnet by finding a host address configured with a prefix that contains it. On OCI both VNICs sit in one subnet, so br-wan has to hold its address as a /32 -- a /23 there puts the subnet's on-link route in the main table at metric 0 and takes the whole subnet off the primary VNIC. A /32 contains nothing but itself, so no rule was ever found, no source rule was installed, and the reply left by the primary VNIC where OCI dropped it for carrying an unregistered source. The address is not the only record of who owns the subnet. The rule names a table, and that table holds the on-link route. Consulted as a fallback, only when the address prefixes matched nobody, so a host that already resolved this the direct way keeps its existing answer -- prod and dev-prod are unaffected. Only scope-link routes count. A default route covers every address and would make every source-routing table an equally good answer. * docs(oci): correct the multi-node story and name the oci_vnic_id constraint The nat-mode warning said EIPs and default egress were both broken on more than one node. Default egress is fine: delete a guest's dnat_and_snat row and it reaches 1.1.1.1 in 2.3 ms from a guest on any of three nodes. Only the public-IP path was broken, and the three defects behind it are now fixed. What remains is not a bug but the shape of the datapath: routed NAT centralises the EIP path on the VPC's gateway chassis, ext-shared is a localnet so the other nodes cannot reach it, and OCI drops any frame whose source is not registered on the VNIC it leaves by. So every node's pool must pin oci_vnic_id to the gateway chassis's VNIC rather than resolving its own. Say so in §5.3, where an operator writing the pool will read it, and in §9, where the cost is that a gateway change kills every public IP until the pools are repointed. The single-node cap itself is gone from the doc: a three-node cluster has been formed and measured, and the warning proceeds rather than refusing. * feat(nat): distribute the routed-NAT EIP path so it works on more than one node Routed NAT processed every EIP on the VPC's gateway chassis, and ext-shared is a localnet, so the transit segment a /32 route pointed into existed only on that one node. Measured on three nodes: all three claimed all three EIPs, the nodes disagreed on the gateway LRP, and only guests on the gateway chassis were reachable. The mechanism to fix it is not new. Pool mode already distributes EIPs by putting external_mac and logical_port on the dnat_and_snat row and letting OVN's ARP responder answer for the external IP on the external segment. Routed mode has an external segment too, the spx-nat veth pair, and it being per node is the distribution: only the node running the instance can answer ARP on it. So a routed rule is now distributed on the same terms as a pool one, and its host route becomes on-link on the transit veth instead of via a gateway LRP that may be on another chassis. bindHostEIP stops demanding that address for a distributed rule, which is what previously failed the bind on every node but one. vpcd plumbs only the ENIs local OVS reports as its own, asked of OVS rather than the Southbound DB because the question is about this host. The gateway LRP keeps its transit-pool address. Default egress routes through it and works across three nodes today, so making it link-local as pool mode does would break a working path to fix one distribution has already fixed. Pool and static are unmoved, and a test asserts that against the same input rather than leaving it to inspection. A rule with no MAC stays centralised, so a cluster part-way through an upgrade is never in a state neither shape handles. Beads: mulga-opyl7, mulga-la0pe * fix(nat): make routed-EIP host plumbing work on every node Three guests on three OCI nodes, one guest per node: only one was reachable. Two defects, both invisible on a single node. The transit veth took a random MAC per node, but OVN holds one MAC binding for the transit nexthop across the whole cluster — so every chassis addressed its egress to whichever node seeded the binding, and the other kernels dropped it at L2. The pair is node-local, so one locally-administered address everywhere is unambiguous; setup-ovn.sh pins it at creation and in the netdev unit, and EnsureUplinkPort repairs a veth made before that. Host routes and proxy-ARP also rode the leader-gated reconcile pass, so a node that never won the lease never plumbed its own guests' EIPs. BindHostEIPs does that half on its own loop on every node; Bind already leaves alone an EIP whose ENI is elsewhere, so two nodes never contend. An unresolvable address skips the sweep rather than shrinking it. Pool and static are untouched: the new pass is a no-op outside routed mode, and a test asserts it. Also adds --teardown and --dry-run to setup-ovn.sh, for re-running it on a node that is already tangled. It deliberately spares the NB/SB databases, /etc/spinifex, the system-id and the packages, and refuses to del-br a Linux bridge. * fix(nat): assert the transit MAC where vpcd actually runs it EnsureUplinkPort is never called in production — host.Wiring is constructed nowhere — so the repair landed in a path the deployed daemon does not take, and three nodes came back with three different MACs. vpcd re-asserts it beside the NAT egress rules instead, which is the routed-NAT branch that does run on every start. * fix(vpcd): make uplink-mode detection say what it saw, and check it detectBridgeMode logged only the branch it took, so a node detected as veth when it was wired for nat looked identical to one that was right. It now reports both probes every time, errors when a node carries both uplink veths (set up twice, and --teardown is the way out), and errors when external_mode and the wiring disagree — the datapath follows the wiring, so the config silently loses that argument. An explicitly configured bridge_mode gets the same cross-check, since nothing else ever compared it against the wiring. The vpc-networking doc now opens nat with "start with pool unless one of these applies" and gains a section on nat across more than one node; the admin warning for --external-mode=nat on two or more nodes says what was verified by hand and what is still uncovered. * test(nat): prove routed NAT delivers each node's own guests, on a cluster natuplink ran single-node only, which is exactly where the two defects that broke three nodes are invisible. It now admits ModeMultinode, and phase 9 takes a second shape there: a guest per node in a spread placement group, and for each one — its public IP plumbed by the node running it and by no other node, the route on-link with proxy-ARP answered there, a dnat_and_snat row carrying external_mac and logical_port, a TCP handshake in from another node, and the guest's own console verdict for egress. Every node's transit veth is checked against the one cluster-wide MAC before any guest boots. Single-node phase 9 gains the same row and on-link assertions, so the shape is held on both topologies. Two gates come from what three OCI nodes actually do: every node holds a gateway_chassis row for every VPC (the active one is the highest priority) and every node installs the VPC ingress route, so only phase 7 needs gating — it runs where the gateway port and the guest are both local, and logs which one is missing otherwise. Cell 20 is real-multi x nat with its own nightly step, leaving the permutation cells' suite set alone. * test(nat): run the VPC egress paths in nat mode, not pool mode only The single suite gated TestVPCEgressPaths and TestWANEgressSMTPBlock on external_mode = "pool", with a comment saying nat clusters have no EIP/NAT-GW support. They have had EIPs since the routed path landed, so the gate now asks whether the cluster can allocate public addresses at all — pool, or nat with a non-transit pool configured. The nat-single cell runs TestVPCEgressPaths beside natuplink, which is the only nightly coverage of a private subnet's isolation and its NAT gateway in routed mode. natuplink never creates a private subnet. * docs(config): list nat among external_mode values * fix(ci): renumber the multi-node nat cell off the bare-metal job Cell 20 is already the bare-metal single-node job, gated on the same cells input but declared in the workflow rather than e2e-cells.json. A matrix entry with the same number runs two unrelated jobs under one name, and naming 20 in a dispatch selects both. * feat(images): add Oracle Linux 10.1 and 9.8, and pin their catalog digests Oracle publishes its qcow2 checksums inline on an HTML page and ships no sums file, so the four new entries carry the expected digest in the catalog rather than a URL to fetch one from. Images gains ChecksumDigest for that, and the catalog import path verifies against it and says which of the two sources it used, so a recorded digest never reads as a fetched sums file. Pinning is safe here and only here: each URL names an immutable build (b291, b293, b178, b182) and never moves. aarch64 has a -kvm-cloud- build alongside the plain -kvm- one; the cloud build is the cloud-init image and is the one that matches x86_64's -kvm-. TestImageCatalogReachable HEADs every URL in the catalog from the general single-node suite. These artifacts live on hosts nobody here controls, so an upstream that retires a build breaks `spx admin images import` for a customer and nothing else in the suite would notice. It also asserts that every entry sets exactly one checksum source, and that no pinned entry points at a "latest" URL whose bytes would change under it. * fix(formation): carry bridge_mode onto the leader and every joiner applyNetworkConfig copied external_mode, external_interface and the pools out of the formation NetworkConfig but never bridge_mode, so a multi-node cluster wrote a spinifex.toml without the key while single-node init wrote it. Every node then fell back to autodetecting a mode that init had pinned. Nightly cell 30 caught it: TestNATUplink failed at the config phase on a spinifex.toml that did not contain bridge_mode = "nat". Deriving it from the mode rather than adding a NetworkConfig field keeps the leader and the joiners on the same rule single-node init already uses. * feat(ui): show how to ssh into an instance, and which user to use An instance detail page gave the addresses and the key pair name but never the login user, which is the one part of the command nobody can derive. It belongs to the AMI, not to Spinifex: nothing in the launch path creates or renames an account, so stock cloud-init applies the distro's own default_user. Oracle Linux is cloud-user, Debian is debian, Ubuntu is ubuntu, and guessing from the distro name gets two of those three wrong. The card resolves the user from the AMI's name and description, and an unrecognised image yields a placeholder rather than a guess, because a wrong username in a copy-paste command is worse than an honest gap. A private-only instance is called out as reachable from inside the VPC only, rather than presented as a command that will simply time out. * fix(nat): plumb host state for a NAT gateway's public address AddNATGateway installed the OVN snat row and stopped there. On a routed-NAT node the host is what delivers a public address, so an egressing private guest left with a source the host held no route back to, and the masquerade rule — which matches the transit 100.127.0.0/24 alone — never saw the packet. The gateway's address needs the same /32 route into OVN, proxy-ARP and FORWARD accepts an EIP gets. hostBindSpecs unions floatingIPSpecs with one deduped spec per NAT gateway address and feeds BindHostEIPs. It is kept apart from floatingIPSpecs deliberately: pruneOrphanEIPs and addLive read that set as the complete list of addresses allowed to hold a dnat_and_snat, and a gateway holds an snat, so widening it would stop the sweep reclaiming a stale row on an address later reused as a gateway's. AddNATGateway also binds directly, on both the create and idempotent-skip paths, so a fresh gateway is not dark for up to 30s waiting on the host EIP pass — and so a reboot that keeps the OVN row still replays the volatile half. Documents NAT gateways in nat mode in vpc-networking, and says at the head of that section that nat is a supported mode covered by nightly cells 19 and 30. * docs(oci): document Oracle Linux as a guest image, and that its user is cloud-user Four catalog entries, all UEFI. The login user is the one that catches people: opc is the default on the images Oracle publishes into OCI itself, while these are the generic KVM cloud images from yum.oracle.com, whose cloud-init default_user is cloud-user. Verified on 10.1 and 9.8 — opc, oracle and ec2-user are all refused. Also records why these four pin a checksum digest inline (Oracle ships no sums file, and each URL names an immutable build), why arm64 takes the -kvm-cloud- build, and that SELinux ships enforcing. * fix(reconcile): stop reading an unreadable intent as an empty one A wedged JetStream KV returned no keys for the ENI bucket while writes were still landing. loadPorts mapped that to an empty port map, pruneOrphanPorts read every guest LSP as an orphan against it, and every guest on the node lost DHCP and IMDS while the API kept answering and the instances stayed "running". The existing re-read guard did not help: it re-reads through the same path. Three defences, because no single one covers every shape of the fault. A failed kv.Get now fails the pass instead of dropping the record. Dropping one VPC narrows localVPCs, which filters out every ENI in it, which the sweep reads as permission to delete every guest port in that VPC — one failed read, same outcome, and it needs no exotic KV state. Unmarshal failures stay skips: that is a poison record rather than an unhealthy store, and failing on it would wedge reconcile forever. The port sweep now asks local OVS which taps are plugged in here and never deletes a port that has one. A running guest's port cannot be an orphan whatever intent says, and OVS is the one liveness signal that does not come through the KV read path the sweep is otherwise entirely driven by. Wired from vpcd; unwired callers fall back to the guard below. An unreadable OVS skips the sweep rather than deciding on intent alone. Each sweep also refuses outright when the keep set it would delete against holds nothing at all — no ENIs, no SGs, or no ports and no external IPs. That is the case local taps cannot cover, because the drift sweep is leader-gated and lists NB cluster-wide, so the leader holds no tap for another node's guest. The cost is real: a genuinely orphaned row now survives until one live record of its class exists. Leaking a stale row is the recoverable half. Three existing prune tests drove their fixtures to a wholly empty intent, which is now the refusal case; they keep one live record of the class so they still prove the prune. The Get-failure path has no unit test — there is no jetstream.JetStream fake in the tree and an embedded server cannot be made to list a key it will then refuse. * docs(oci): separate the two address quotas, and the principal that can read them The regional reserved-public-IP ceiling is tenancy-wide and does not grow with node count, so a cluster hits it before the per-VNIC 64, which does. Records the commands that read both, and that a detached reserved public IP still bills. Reading limits or usage needs a tenancy-level grant, which cannot be compartment-scoped the way the node's policy is. That is a second, read-only principal kept off every node, not an addition to SpinifexOperators. * docs(oci): give quotas their own troubleshooting section, and say we enforce none Spinifex hardcodes no address ceiling: it attempts the allocation and maps OCI's own refusal to InsufficientAddressCapacity. So a quota is raised with Oracle and nothing here is reconfigured or restarted — which is what an operator needs to know before it bites, not after. Separates the two quotas by how they scale: the regional reserved-public-IP default is tenancy-wide and raisable by ticket, the per-VNIC 64 is per node and is not. * fix(oci): follow a guest to its new node when its EIP moves OCI delivers a public address to whichever VNIC holds the private IP behind it, and nothing ever moved that private IP. Binding.VNICID was written once at allocate time, as the VNIC of whichever queue-group daemon happened to serve the AWS request, and never updated — so an address landed on the right node only by chance, and a guest that stopped and started elsewhere went dark while every OVN row, host route and firewall rule on the new node was correct. ClaimLocalAddresses closes the gap from the node that should hold the address. Local OVS decides ownership, the same authority the host EIP binder already trusts for the same question, and UpdatePrivateIp with a new VnicId does the move in one server-side call. The private IP OCID survives it, so the reserved public IP attached to it follows without being touched and the customer's address never changes. A loop per node, not leader-gated: the question is about this host's guests, and a node that never wins the reconcile lease would otherwise never claim the addresses of the instances it is running. * fix(oci): record which ENI an EIP is associated with An EIP is allocated bare and attached by a later AssociateAddress, so the OCI binding written at allocate time named no ENI and nothing ever filled it in. The address then stayed on whichever node happened to serve the allocation: the affinity pass could follow an auto-assigned address, which is allocated with its ENI, but never an EIP — the case customers actually use. BindOwner is on an optional OwnerTracker interface rather than Allocator, because only OCI has a notion of an address being delivered to one node. A static or DHCP address is on the wire everywhere its node can reach, those allocators do not implement it, and ExternalIPAM treats a pool that does not as a success with nothing to do. The same gap made the release owner guard inert for EIPs: it compares against a binding ENI that was always empty, so a stale teardown could free an address already handed to another instance. * fix(ec2): report the address an instance is actually reachable on Three faults kept DescribeInstances answering with the address launch assigned, whatever happened afterwards. The launch-time stamp lives on the stored ec2.Instance, and the stopped path only declined to add an address rather than removing one, so a stopped instance went on advertising an address it had released and that may already belong to someone else. The projection now clears it, and the NIC association with it, working on its copy so a restart can still report the address. The running path filled the public address only when it was nil, which the stamp never is, so an EIP could not be reported however correctly it was recorded. The runtime address now wins. And nothing told the rest of the system about an association: AssociateAddress wrote neither the ENI record, which is what IMDS and the network reconcile read, nor the instance record, which is what a describe answers from. The ENI write is direct; the instance write is a fan-out, because the queue-group worker that serves AssociateAddress is rarely the node running the instance. * fix(net): take the host EIP plumbing off a node its guest has left Bind refuses an EIP whose ENI is on another chassis; the prune beside it kept anything still wanted cluster-wide. A guest that moved therefore left the old node holding the /32 route into OVN and the proxy-ARP entry on the uplink, with nothing that would ever remove them. On a shared segment that node goes on answering ARP for an address another node is now serving. The two halves now ask the same question by the same authority, in the loop that already runs per node. An unreadable local OVS is not a yes: tearing down a live address on a guess would fire everywhere at once, so it keeps the binding and the next pass tries again. A binder with no chassis test behaves exactly as before, and a centralised EIP is exempt because it hairpins through the gateway chassis and every node plumbs it legitimately. * fix(net): refuse to free an Elastic IP on behalf of a departing interface Releasing an instance's public address on teardown releases it by address, scoped to the ENI holding it. Since an association now writes the EIP onto the instance record, that address is the EIP — so terminating an instance gave a customer's Elastic IP back to the pool while describe-addresses still reported it as theirs, free to be handed to another guest. The existing ownership guard cannot catch this. It only fires when the stored allocation already names an ENI, and an EIP is allocated bare and associated later, so the stored ENI is empty and the guard passes straight through. The distinction that holds is who is asking. ReleaseAddress is the address's own owner and names no interface; instance teardown names the interface it is tearing down. So a release that names an owner may free an auto-assigned address and never an EIP, which is enforced in each of the three allocators. DHCP needed the scope carried on the wire, since it releases by address and resolves the lease afterwards. Absent on an older caller, which reads as false and keeps the previous behaviour. Worst on the two providers where it is not recoverable: on DHCP the lease is the only thing holding the address and the server promises nothing about handing it back, and on OCI the address is a reserved public IP object that the release destroys outright. * test(e2e): prove an Elastic IP survives a stop and start Instance teardown releases the public address by value, and since the association writes the EIP onto the instance record, that release aims straight at the customer's Elastic IP. The allocator guards now refuse it, but nothing end to end proved the address comes back on a restart. The step goes inside EIPFlip, which already runs last on the shared public guest for the same reason a stop is safe there: the address the guest returns on is the allocation's, not the pool's. It runs on the static cells and the DHCP ones alike, and on DHCP it doubles as the only outside view of the lease still renewing while the guest is down. * feat: release an auto-assigned public address on stop, take a new one on start AWS lends an instance a public address for as long as it runs and gives it a different one on the next start. Spinifex kept the address across a stop, which holds a pool slot for a stopped instance and, on a DHCP pool, keeps renewing a lease nothing is using. A stop now offers the address back through the instance cleaner, which refuses anything an Elastic IP is behind — the EIP association record answers that, the same question an ENI delete already asks — and records on the VM that the start owes it a new one. A host drain keeps the address: Restore relaunches the guest on the next boot, and changing its address under it is an outage nobody asked for. The start allocates before the launch, so an exhausted pool refuses the start rather than booting a guest the customer cannot reach, adopts an Elastic IP associated while the instance was stopped instead of assigning a second address beside it, and hands the address back if the launch then fails. e2e: a StopStartAddress stage on the public guest asserts a stopped instance reports no address and comes back on a different one from a static pool, and a StopStartKeepsTheEIP step asserts the opposite for an Elastic IP. * docs: restructure the OCI guide around two deployment paths, with diagrams The guide had grown into a numbered wall with no shape: no frontmatter, no contents, and a reader with an OCI account had no way to tell where to start. It now follows the multi-node install template — overview, prerequisites, then the deployment path — and splits that path in two: Terraform for one node or three, which is the preferred route, and a manual single-node walkthrough for an instance that already exists. Both meet at one shared configuration section rather than repeating it. Two ASCII diagrams carry what the prose could not: one OCI instance with the whole AWS surface inside it, and three nodes with the same surface distributed, showing which plane each VNIC carries and how an address follows a guest that moves. The limits section was materially out of date and said so in the places that mattered most. Multi-node is no longer experimental, public addresses are no longer pinned to one node — each allocates on its own VNIC and the affinity pass moves the private half to wherever the guest is — and an auto-assigned address is now returned to OCI on stop, so a stopped fleet stops paying for addresses. What changed is recorded rather than quietly edited. The Terraform configuration moves in from the monorepo as scripts/terraform/oci-spx, so the guide points at something in this repository. Also marks the two new lifecycle test files in-package, with the reason each one needs to be: both substitute unexported dependencies. * docs: rework the OCI guide — SVG diagrams, Terraform variables, iSCSI and br-wan Replaces the two ASCII diagrams with SVGs in the house style, one for a single node and one for three. Renames the two deployment sections to Terraform deployment and Manual deployment, documents what the Terraform builds per node and every variable worth setting, and explains that the data volume is an iSCSI attachment rather than a disk. The manual path now covers the iSCSI login, the fstab entry and the netplan source routing that Terraform writes, and states what br-wan is for on a routed-NAT node: it names the VNIC for the OCI allocator and holds the registered addresses, but is never bridged into OVS. Opens with what Spinifex is and why OCI is worth running it on. * fix: make gateway convergence work on a multi-node routed-NAT cluster Three defects each stopped a multi-node cluster from ever attaching an internet gateway, and all three produced the same uninformative symptom: `terraform apply` sitting on `aws_internet_gateway.igw: Still creating...` forever while every service reported healthy. ovn-northd was given only its local OVSDB socket as a client connection. A RAFT follower does not forward writes -- it answers "not cluster leader, trying another server" -- so with one entry in the list, northd stops translating Northbound intent into Southbound flows the moment database leadership moves to another node. Measured on the OCI cluster: SB_Global.nb_cfg seventeen transactions behind NB_Global.nb_cfg, with no chassisredirect port for any VPC created after leadership moved. setup-ovn.sh wrote the member list only when the caller passed --ovn-remote, which only install-node.sh did; it now derives the list from the live RAFT membership otherwise, and says so when it finds a single member, because that node needs a second pass once the others join. The gateway datapath probe ran wherever the reconcile lease landed rather than on the chassis owning the port. Both probes leave from the local uplink, and under routed NAT the transit subnet exists identically on every node, so a remote probe reported a healthy gateway as dead and aimed a repair at a datapath the node does not own. GatewayPortLocal gates it. On OCI the probe then ARPed the tenant-visible public IP, which the cloud NATs and which appears on no interface anywhere. The reconciler now maps the address through the same DatapathResolver the NAT rules are built from. Also adds DescribeVpcEndpoints, which the AWS provider calls on every VPC delete to sweep GuardDuty-managed endpoints; an InvalidAction there failed the destroy and stranded the VPC. The OCI guide gains a Terraform section recording which shipped workbooks were validated against a three-node OCI cluster and which are still open. * docs: record the validated Terraform workbook results for the three-node OCI cluster Six of the seven workbooks run against the OCI cluster pass, including the three the section exists to prove: RDS, ECS and EKS. Records nginx-alb and the clean eks-quickstart re-run, and names the two lifecycle gaps that bit during the run so an operator recognises them rather than reading them as a broken workbook. * fix: route the NAT gateway's SNAT source through the datapath resolver AddNATGateway was the one NAT path that did not map its external address to the one that rides the wire. Where the public half of an allocation is NAT'd by the cloud it is on no interface anywhere, so every private-subnet guest egressed with a source that was dropped, while the gateway reported available and DNS kept working because the VPC resolver is host-local. The unmapped address also failed the host EIP sweep, which works in resolved addresses, so the bind was pruned as foreign on the next pass and reinstalled on the one after. The public half now survives as an external ID instead. Also restructures the OCI guide around numbered steps through to running Terraform, and moves the cloud-init and metadata sections into troubleshooting where they are read when something is wrong rather than followed when it is not. * fix: place a NAT gateway's OCI address on the node that forwards for it OCI drops a packet whose source is not a registered private IP on the VNIC it leaves from, so a NAT gateway's SNAT source has to be on the gateway chassis. Two independent placements decided those and nothing reconciled them: the address was created by whichever worker served AllocateAddress, while the chassis is a fixed ordering the reconcile pass asserts. On a three-node cluster that made private-subnet egress a coin flip, and the same Terraform passed or died depending on which node answered. The affinity pass already solves this for guests, by asking local OVS where the tap is. A NAT gateway has no tap, and it skipped anything with no ENI — which is exactly and only this address. The equivalent local fact is the VPC's chassisredirect binding, so ownership now has two answers rather than one, and an address that is neither a guest's nor a gateway's is still left alone. The VPC is recorded from the reconcile pass rather than from CreateNatGateway, because the answer is the gateway chassis and reconcile is what decides it. * fix: plumb a NAT gateway's host ingress only on its gateway chassis A NAT gateway's address has no ENI, so neither the bind's locality test nor the prune's could ever fire for it and every node installed its /32 route into OVN. Latent on bare metal, where pool addresses are disjoint from the node subnet; on a cloud that hands out the address from the nodes' own subnet it outranks the connected route and swallows node-to-node traffic. Give that address the same second authority the ocinet affinity pass has: the VPC's chassisredirect binding, read from the local ovn-controller. An unanswerable question still leaves the binding alone, so a single node whose gateway port is not yet claimed behaves exactly as before. The orphan EIP prune built its live set from ENI and user EIPs alone while also sweeping host ingress, so every drift pass tore down the ingress of every live NAT gateway and left it dark until the next host EIP tick. * fix: ask the gateway chassis only about a NAT gateway's address gatewayElsewhere skipped the NAT gateway marker it was named for and answered for every spec, so on a node that is not the VPC's gateway chassis every EIP — including a guest's, whose locality is its own port — was left unplumbed. Two of three nodes bound nothing. * test: assert the two private-subnet EKS workbooks in the tofu harness eks-https-ingress and eks-gitops-argocd put their workers in private subnets behind a NAT gateway, so a worker only registers Ready if its egress to the cluster endpoint and to ECR survives that path. That makes them the workbooks that catch a NAT gateway fault, and neither had an assertion, so neither could be run at all — run_workbook calls assert_<name> by construction. Both share eks-quickstart's readiness check, extracted unchanged. They also name the worker size node_instance_type rather than instance_type, and tofu rejects a -var for an undeclared variable outright, so the name is read from the config instead of kept in a list here. * test: count agents, not every node, when asserting EKS workers Ready The control plane is three K3s servers, so an unfiltered node count reaches a one-worker cluster's desired size before any worker joins — the assertion passed on exactly the failure it exists to catch, reporting "4/1 workers Ready". Filtering on the control-plane role label makes it 1/1, which is the number that says a node-group worker registered. Matters most for the two private-subnet workbooks, where the worker joining is the whole proof that egress through the NAT gateway works. * docs: record the two private-subnet EKS workbooks passing on OCI	2026-09-26 14:31:07 -05:00
Ben Duncan	225bf65a8e	feat: EBS export tunables, teardown reapers, fan-out and OVN fixes (#800) ... Merges `feat/ebs-export-tunables` (83 commits) into dev. The branch could not be rebase-merged because it carries five merges from dev, so it lands as a squash. ## What is in it - **EBS export tunables** and the nbdkit/viperblock export path they configure. - **Teardown reapers and backoff** for S3 buckets, load balancers, gateways, spot requests, certificates, endpoints, elastic IPs and network interfaces, so a dependent that keeps failing is backed off rather than retried hot. - **Retry and tolerance paths**: ECS IAM service init, ECR signing keys deleted between the list and the read, an unreadable signing key at awsgw startup, and an ECS instance-profile policy attach converging on a stale read. - **Key pair validation reads the material by name** instead of listing the account, closing the `InvalidKeyPair.NotFound` race that a single-replica LIST could produce. - **Concurrency and fan-out**: concurrent metadata document fetches, bounded metadata reads for describes, and the node-discovery timeout removed from every fan-out call. - **OVN correctness against the configured NB cluster**: the flows barrier, nexthop MAC binding, and stopping a compute node from running the package's standalone database. - **nats-js-watchdog can re-arm itself.** The escalation check ran above the probe loop while the only line that cleared the counter sat below it, so a latched watchdog could never observe a recovery. It now probes first and decides second, and says so once per latch rather than once per tick. - **CI**: 19 commits of nightly and build-pipeline work. ## Verification `make preflight` green locally. Build and Test, Integration Tests, Race Detection, and Lint and Security all green on `c2da0385`. ## Known red, not introduced here Two nightly cells fail on this branch and are pre-existing: - **cell-10** — a partial OVN IPsec enable leaves `NB_Global ipsec=true` asserted cluster-wide while two of three chassis failed their local reconcile, black-holing cross-chassis guest traffic for roughly 5.5 minutes while every health signal stays green. - **cell-19** — a total layer-3 egress blackhole below a correctly and promptly programmed NAT/EIP datapath, aggravated by a natuplink probe whose retry budget outlives the test's own wait, so it can never emit its FAIL verdict. Both are filed separately and should be weighed before a release is cut.	2026-08-17 17:42:27 -05:00
Ben Duncan	42d87d7a5a	feat: EBS export tunables, plus blob-read fault isolation for ebsmetadata (#799) ... * feat(ebs): make nbdkit thread count and per-volume read cache configurable Both numbers that set a volume's export concurrency and memory were hard-coded: nbdkit ran on its own default of 16 threads per connection because no -t was ever passed, and the read cache was a 128 MiB literal in the mount path. Neither could be changed without a recompile, and neither was visible in the logs or the process argv. Add default_threads and cache_size_mb to [nodes.<node>.ebs]. Defaults are unchanged, so a config that does not set them behaves exactly as before. CacheSizeMB is a pointer so an explicit 0 (cache disabled) stays distinguishable from an unset key, matching how ViperblockConfig treats its own toggles. -efi volumes stay uncached regardless. Validation is a range check only. Both are operator tunables applied per nbdkit process, and there is one of those per volume, so the host cost is the value times the volume count -- left for the operator to weigh rather than second-guessed here. * feat(security): arm the firewall by install path and sync the clock at firstboot The host firewall armed itself from a code default, so a curl-to-bash install onto an existing Debian or Ubuntu machine applied a default-deny INPUT policy seconds after bootstrap. Anything the operator was already running that is not on the allow list stopped accepting connections, with nothing logged and no config key to set beforehand — no template writes firewall_enabled, so every install fell through to the default. Arming is now the install path's decision. setup.sh takes --firewall=on|off and records it in /etc/spinifex/firewall/mode, defaulting on under ISO_BUILD and off everywhere else: the ISO is an appliance on hardware we define, a package install is a guest on a machine that was already doing something. firewall_enabled becomes a three-state *bool so "unset" and "explicitly false" stay distinguishable. Flipping the default to false instead would have torn down a working policy on every existing node, because false is an instruction to disable rather than an absence. Resolution is config, then mode file, then on, so a node installed before any of this keeps what it has. The ISO had the opposite problem. setup.sh is the only place that enables spinifex-firewall.service and it returns before that line under ISO_BUILD, with no systemd in the chroot to talk to. Nothing else enabled it, so after a reboot the ruleset — runtime state — was gone, and ReconcileFirewall short-circuited on an unchanged peer file and never put it back. An ISO node was firewalled until its first reboot and open permanently after. firstboot now enables the unit, and the reconcile checks the table is loaded rather than treating an unchanged peer set as proof the policy is applied. Two policy changes: 443 is held for the nginx TLS edge so that rollout needs no per-node firewall change, and the SSH port is read from sshd_config and its Include drop-ins instead of being hardcoded to 22 — a host on Port 2222 was allowed exactly one port that was not the one it uses, and ct state established kept the operator's session alive until they disconnected. 53 stays open on every node, now with a comment saying so: northstar serves public authoritative DNS. Clock: firstboot steps and waits for sync before spx admin init mints credentials, since a certificate issued against a wrong RTC carries a NotBefore that outlives the correction. Bounded to 60s and non-fatal so an air-gapped install still completes, and once is enough because rtcsync writes the corrected time back. A chrony drop-in replaces Debian's makestep 1 3, which stops stepping after three updates and then slews at roughly 7 seconds a day — a node that burned those before the network came up never visibly converged. * fix(security): let a node join a cluster whose firewall is already armed Expanding a single-node ISO install into a cluster could not succeed. Two independent blockers, both from the same root: the peer set is derived from converged cluster state, but the ports needed to reach convergence are gated on that same set. The formation port is peer-scoped, and a node dialling in to join is not a peer yet, so the handshake was dropped rather than refused and the joiner retried in silence until it timed out. It cannot simply move to the public rule: outside formation the same port carries the daemon cluster manager, whose /health and /local/* routes report node topology, service inventory and running instances unauthenticated. So it is opened for the formation window only, by spx admin init, and closed again on every exit path. The reconcile then deadlocked. Holding out for the full chassis list is right on a node with no policy loaded, but on one armed for a smaller cluster the ports a joining node needs in order to register its chassis are exactly the ones the stale set blocks, so the list can never complete. Widen the peer set on the tunnel set already loaded, and report the partial state so the caller retries sooner than the steady-state interval. A converged cluster whose Southbound DB is briefly unreadable is still left untouched. * docs(install): explain the firewall around cluster formation An ISO-installed server is a cluster of one, and its firewall admits internal cluster traffic only from members it recognises. Forming a cluster out of such servers therefore needs the firewall off first: step 3 of the multi-node guide has each server dial node 1's OVN database on the raft ports, and those are peer-scoped like everything else on the cluster plane. Neither guide said so, and the failure is a silent drop that looks like a network fault. Both guides now walk the off / form / on sequence, and the multi-node guide gains a reference section covering which ports are public, how arming differs between the ISO and the binary installer, and how to verify membership landed. Re-enable the boot-time unit whenever the daemon arms the policy. Disabling the firewall to form a cluster also disables that unit, so re-arming used to look complete while leaving the node bare after its next reboot. * fix: make the ISO-to-cluster conversion path work end to end Forming a cluster from four ISO-imaged servers surfaced a run of bugs that share a starting state CI never exercises: the ISO delivers an initialized, armed, running node, while the conversion path was written against the binary installer's unconfigured one. - install-node.sh never disarmed the host firewall, so nodes 2 and 3 could not dial the first node on the OVN raft ports and formation could not proceed. It now owns the whole lifecycle: disarm, form, re-arm, then verify the loaded policy against the expected peer set. - spx admin init left the admin access key and secret empty on the preserve path. The pair cannot be read back, and the IAM store is rebuilt by formation, so the empty strings reached every joiner's bootstrap.json and crash-looped awsgw. It is reissued instead, and IAM seeding now skips an empty admin block rather than taking the gateway down on every restart. - node-reset.sh left /etc/default/ovn-central behind. Raft membership lives in that file, not in the databases the reset deletes, so ovn-central recreated the database as a cluster member dialling a peer that had itself been wiped. - node-reset.sh also destroyed the installed firewall policy, which nothing regenerates. set-peers then failed on every attempt and the node stayed open permanently and silently. The arming mode is read before the wipe and the stage regenerated after it. - spinifex-firewall-apply rejected every verb when the policy file was missing, including disable. Turning the firewall off is the recovery path, so that left a node that could be neither armed nor disarmed. - viperblock exited 0 on a fatal startup error, so Restart=on-failure never fired. During formation its JetStream bucket has no leader yet, so the EBS provider died for good and every volume request failed with no responders. predastore and qemunbd had the same bug. - spx admin images import created image directories at 0700, so the first sudo import locked group members out of a tree deliberately made group-writable. Three of these hid behind checks that cannot succeed: ovn-sbctl show against a clustered database, and the OVN northbound and southbound waits in setup-ovn.sh. All three failed to stderr and let the run continue, so a check that could never pass was indistinguishable from one that was merely slow. The USB install guide claimed the ISO leaves Spinifex unconfigured and told the operator to run spx admin init. firstboot has always done that itself, so the guide now describes what the ISO actually produces and how to start using it. * fix: let spread placement launch across every node A spread placement group reserved four distinct nodes correctly and then failed the launch with ServerInternal, having started only two or three of them. The scheduler was never at fault; two faults downstream stopped the instances following the reservation. vpcd subscribes to vpc.add-nat with QueueSubscribe, which delivers one message at a time, and the handler holds its reply across the OVN flows barrier. The 45s caller deadline was sized against a single 30s barrier, which is right for one instance and wrong for the N a spread launch issues at once — the third request waits behind two full barriers and times out. That one subscription now overlaps its handlers under a bounded semaphore. Each barrier still syncs after its own write, so the guarantee is unchanged while N barriers collapse into roughly one. With the requests overlapping, every barrier still ran to its alarm. install-node.sh computed OVN_REMOTE after building the database nodes and gave it only to compute nodes, so the database nodes were configured without --ovn-remote. setup-ovn.sh writes ovn-northd's NB/SB connections from that list and skips it on the localhost default, and it already documents what follows: a northd dialling only its local member stops advancing nb_cfg once raft leadership moves, wedging ovn-nbctl --wait=hv. The list is knowable before any node is touched, so it is computed first and passed to the database nodes too. Nothing reported an OVN fault while this was happening. Every chassis was registered, ovn-controller and IPsec were up, and the raft cluster was healthy. The only signal was the barrier logging that it had overrun and was continuing without confirmation — which reads like slowness and was a permanent wedge. * feat: sweep block sizes in the disk benchmark and add an iperf3 network benchmark * fix: use iperf 2 with CSV output and report why a remote start failed * fix: stop pkill matching its own shell and verify the iperf server is listening * ci: run the predastore throughput benchmark at the end of the nightly * ci: boot the ISO in the nightly, single-node and three-node * ci: use the existing CI node password secret in the ISO cells * ci: report free space before the benchmark instead of failing inside warp * ci: keep the benchmarks and node disks off tmpfs * ci: export the work directories from RUNNER_TEMP rather than job env * ci: run the ISO and benchmark workloads from the nvme scratch area * perf: run guest NICs on vhost-net with multiqueue Guest-to-guest VPC throughput measured 1.68 Gbit/s single-stream against 21.8 Gbit/s host-to-host on the same underlay, and did not improve with more streams. QEMU's -netdev tap defaults vhost to off, so the whole datapath ran in the main loop thread in userspace: that thread sat at 59-65% CPU while all four vCPU threads idled at ~4%. Every tap netdev now carries vhost=on, and VPC ENIs additionally run multiqueue at min(vCPUs, 8) queue pairs. Multiqueue needs all three of the tap, the netdev and the device to agree, so TapNetDev and NetDevice derive the netdev options and the 2N+2 MSI-X vector count from one queue figure rather than leaving callers to keep them in step. The QMP hot-plug path takes the same options as cold boot, with a test comparing it against the rendered cold-boot strings so the two cannot drift. MMIO machines stay single-queue: they have no MSI-X. * ci: check out the sibling repos the go workspace needs in the ISO cells * ci: skip cell-21 workload cleanup when the workload never ran * ci: size the cell-21 node at 8 GiB and keep its serial log * ci: keep the cell-21 node diagnostics alongside the serial log * fix: let the daemon open /dev/vhost-net and delete multi_queue taps Enabling vhost=on broke instance launch two ways. DevicePolicy=closed gates the cgroup device list before capabilities are consulted, so QEMU's open of /dev/vhost-net returned EPERM despite CAP_DAC_OVERRIDE; add the device to the allow list and bump the unit version. Separately, `ip tuntap del` re-opens the device with TUNSETIFF and fails with EINVAL on a multi_queue tap, leaking it on every teardown; route deletion through `ip link del`, which is flag-agnostic. * perf: default guest NICs to a single queue Multiqueue measured a net loss. Spreading one flow across queue pairs reorders packets and TCP reads that as loss: a 4-vCPU guest fell from 1.76 to 1.50 Gbit/s, with cwnd collapsing on spurious retransmits. The host overlay funnels a node pair onto one softirq core, so the extra queues have no headroom to win back what the reordering costs. Keep the plumbing and its tests behind MultiqueueNICs so re-enabling is one line once that funnel is fixed. vhost-net stays on unconditionally; it moves the datapath off the QEMU main loop, measured at 60% of a core under load against 3% with it. * fix: raise the UDP receive buffer for predastore's QUIC transport Predastore carries blob, meta and raft over QUIC and quic-go asks for a 7MiB receive buffer. Left at the kernel default of 208KiB it silently dropped datagrams under load: the four-node cluster had shed 80k-125k each, which failed shard writes, put raft into repeated elections, and ultimately handed guests an I/O error that cost two of them their root filesystems. Adds a sysctl stage writing /etc/sysctl.d/99-spinifex-net.conf, so the setting lands in the ISO image and on every node push rather than living in a runbook. * ci: give the cell-21 workload an identity the node accepts * feat: tie the guest MTU and NIC multiqueue to ipsec_enabled ESP costs 34 bytes on top of geneve's 58, so a cluster running the overlay in plaintext can advertise 1442 instead of 1408 — worth roughly 10% of guest throughput. Derive it from network.ipsec_enabled rather than hardcoding the conservative figure. Multiqueue moves with the same flag, because its sign depends on it. Behind IPsec it costs ~12%: ESP has no L4 ports, so RSS funnels a node pair onto one core and extra queues only reorder packets, which TCP reads as loss. Without IPsec the sending host's single vhost thread saturates first and queues are worth +54%. IPsec on with multiqueue on is the worst of the four combinations, so it stops being a standalone knob and becomes vm.Deps.MultiqueueNICs. The reconciler only ever created a subnet's DHCPOptions row and never revisited it, so the create-time MTU stuck for the life of the subnet and this change would have applied to new subnets only. Worse in the other direction: turning IPsec back on would have left subnets advertising 1442 over ESP, which drops large segments with no ICMP to recover from. applySubnets now diffs and updates in place. reconcile.Config takes IPSecDisabled rather than IPSecEnabled so the zero value is the conservative MTU — a caller that forgets it is merely slower. * docs: add an install-time performance tuning guide Measured on the four-node cluster rather than inferred: the UDP receive buffer sysctl, what IPsec actually costs and why, the multiqueue sign flip, the overlay MTU budget, and the jumbo-frame check that must precede enabling it. A starter doc — several sections end in an open question rather than a recommendation. * ci: keep the cell-21 node alive for the workload step * ci: collect the cell-21 qemu log and widen its timeout * feat: derive the guest MTU from the fabric, and deepen the virtio NIC config The overlay overhead is passed on to the guest rather than hidden from it. Hiding it means running the fabric far enough above 1500 that a standard guest MTU fits in the headroom, which is what the large public clouds do — but they own the switches. On a fabric we do not control, the honest figure is the one the path can carry, and a jumbo underlay should reach the guest: 9000 gives 8942, or 8908 with IPsec. network.underlay_mtu is the lever; SubnetMTU spends it. Configured rather than probed, because a node's own NIC MTU says nothing about the switch between it and its peers. host_mtu sets VIRTIO_NET_F_MTU so a guest that never takes a DHCP lease — statically addressed, or IPv6-only — still learns the ceiling. rx/tx_queue_size raise the virtio ring from QEMU's default 256. That default is a packet-rate ceiling rather than a memory saving: a vCPU preempted for a fraction of a millisecond fills all 256 slots and the overflow is dropped. ethtool is the only way to read NIC offload, ring and channel state, so a node without it cannot be diagnosed for throughput at all. * ci: give the cell-21 workload its own guest key * fix: set the guest MTU on the tap, not just in the guest The tap was created and brought up without an MTU, so it stayed at the kernel default of 1500 while DHCP and host_mtu told the guest something else. Harmless at 1442; it silently blackholed every large frame on a jumbo underlay, with no ICMP back because the drop is at the first hop. TapSpec.MTU comes from the same guest overlay MTU the DHCP option and host_mtu already use, and is applied in the bring-up call so a size the fabric cannot carry fails the launch instead of degrading quietly. * fix: stop setting tx_queue_size, which QEMU caps at 256 for tap QEMU only honours a tx_queue_size above 256 for vhost-user and vhost-vdpa backends. On a tap it rejects the value outright rather than clamping it, so every launch died with "must be a power of 2 between 256 and 256" and the instance went straight to terminated. rx_queue_size is unaffected and stays at 1024. A test pins the absence of tx_queue_size across machine types and queue counts. * docs: lead the tuning guide with the overlay segmentation finding The guide's headline was a 5.4x that came almost entirely from turning IPsec off, and it treated MTU and the fabric as the main levers. The host reaches 92.6 Gbit/s at MTU 1500 on the same NIC, so the fabric is not the constraint and the encrypted default is not the thing to tune away from. Adds the host-versus-guest comparison up front, documents network.underlay_mtu, and makes the missing segmentation offload the first open question. * ci: stop deselected matrix cells failing in teardown * docs: quantify what disabling overlay IPsec is worth The guide described the trade without the numbers an operator needs to make it. Adds the measured 4.9x, notes that the MTU and multiqueue changes follow ipsec_enabled rather than being separate knobs, and records two transition hazards: a stale wide MTU on running guests, and RunInstances failing while the OVN flows barrier catches up with the new SAs. * ci: collect and retain the e2e benchmark and cluster logs Cells 21 and 22 both copied two paths that only ever exist on the nodes, so the benchmark JSON was never uploaded and cell-22 collected nothing at all. Both now gather what they actually produce, and keep it for 90 days. * fix: quieten spx by installing an explicit CLI logger default spx never set a process logger, so it inherited whatever a linked package left behind — until now a JSON handler at Info on stdout, interleaving library log records with command output and breaking piped use. Execute() claims the default before any command runs: text on stderr, LevelError, so commands report failures through their own stderr writes and libraries stay quiet. --verbose/-v raises to info and SPX_LOG_LEVEL sets an arbitrary level, the flag winning over the environment. Service subcommands still call initTelemetry, which replaces the handler outright. No root --debug flag: viperblock, qemunbd, nats and awsgw each define their own --debug with a service-specific meaning, and a root persistent flag would shadow inconsistently depending on the command. The guard test samples the handler at package-init time rather than inside the test, because tests in this package start services that install their own logger legitimately. * feat: private /admin/CreateAccount for self-service tenant signup Add POST /admin/<Method> to the AWS gateway: SigV4-signed, JSON in and JSON out, callable only by an IAM user in the super-admin account holding spinifex:<Method>. CreateAccount is the one method, so a signup form can provision a tenant without an operator shell. No new service, port, unit or certificate — it mounts inside the gateway's existing authenticated group and inherits its auth, throttling and tracing. Provisioning is resumable rather than transactional, which fits stores that have no transaction. Two CAS creates carry the correctness: the client token admits one in-flight attempt and makes a lost response replayable for 24 hours, and the name reservation makes duplicate account names impossible. Every step after the account exists is create-if-absent, so a retry with the same token finishes the job instead of failing. The CLI and the endpoint share one provisioning helper, which closes a gap on both at once: CreateAccount never checked names, so running `spx admin account create` twice produced two accounts that differed only by ID. The response waits for the account's default VPC over a new ec2.EnsureDefaultVpc request/reply, rather than trusting the existing fire-and-forget iam.account.created publish — credentials for an account that cannot launch anything are worse than a slower call. [signup] max_accounts bounds how many accounts may exist; absent means uncapped, which is what every cluster does today. * fix: stop the awsgw template hardcoding one deployment's console URL console_url shipped as https://console.spx3.com, so every fresh install — including an on-premise cluster that will never run signup — was configured to point new accounts at our console. It is the only deployment-specific domain in any template. It now ships empty, and the gateway warns when it provisions an account without one, since a blank sign-in link is otherwise visible only to the customer who receives it. Extract the remote client from the command body while here. The signing, request shape and error decoding are the part a Cloudflare Worker mirrors, and as a function they are testable against a TLS test server rather than only by hand against a live cluster. * refactor: drop console_url from signup and default max_accounts to 128 The gateway never read console_url — it passed the value straight through to the response, and the idempotency record froze it for 24 hours so a corrected typo still replayed stale. The client that made the request already knows where it wants to send the new tenant. max_accounts is now a pointer in the parsed config so an absent [signup] section takes the 128 default while an explicit 0 still means uncapped. * feat: run the predastore fault-injection harness as cell-24 Cell-23 asks how fast predastore is when every node is healthy; cell-24 asks whether it survives one that is not. It runs after cell-23 rather than beside it: both want a quiet single runner, and a benchmark sharing a machine with a fault injection run measures the wrong thing. The port offset keeps the two clear of each other even so, and the job clears stale nodes on entry and exit, since a run killed before its cleanup trap leaves one holding ports that fail the next run somewhere unrelated. * ci: keep the throughput benchmark's output with its results The run log only existed in the job log, so the artifact it uploaded could not be read without going back to the run that produced it. * ci: keep the fault-injection run log with its results events.txt carries the timeline and the FAIL line, but not the build output or anything Warp wrote before the harness took over, so a failed cell could only be read from the job log it was printed into. * fix: serve the cluster region to the console instead of hardcoding it The console hardcoded ap-southeast-2 as its SigV4 region in seven files, so on any other cluster the credential scope was wrong and the gateway rejected every signed request. The SPA is embedded in the binary, so a build-time value would pin one binary to one region. The UI server now serves GET /api/config with the region, no-store so a proxy fronting several clusters cannot serve one the other's region, and the SPA loads it once before render. The region comes from the serving node's config because awsgw pins signature verification to that same field. Reading the cluster-wide [aws] region instead would let a cluster whose two keys disagree pass review and still fail to authenticate. Also relabels the login card as Spinifex credentials, since the fields are not AWS ones, and hides the CA install banner unless the console is reached directly: behind a proxy with a publicly trusted certificate there is nothing for the cluster CA to fix. * fix: say once when the host firewall will not be maintained Both quiet outcomes of the reconcile — no cluster config, and the policy switched off — return nil without logging, so a node that never arms is indistinguishable in the journal from one that armed on its first pass. * test: arm the host firewall before asserting on its peer set Only the ISO install arms the policy; every other path writes off to the mode file, so the daemon disables the firewall and never writes a peer set. The test read that file as its baseline, which it could never have. It now switches the mode on for its own duration and puts it back, waits for the peer set the way install-node.sh does, and on failure reports the mode, the node count and what the daemon said rather than just the missing file. * test: move the console region test to the external cmd package It was in package cmd, which test-package-check rejects, so preflight failed on the branch. The hook follows the pattern already in export_test.go. * test: make the reboot EIP diagnostic capable of naming the drop The two ofproto/trace probes injected packets with no TTL, so OVN's router killed them at lr_in_ip_input and both traces ended early for a reason that had nothing to do with the failure under investigation. A keyword filter then stripped the table-match lines, hiding where each trace stopped. Traces now carry a TTL and are kept whole, the real datapath megaflows for the guest reply are dumped alongside them, and any drop matching the guest source is resolved from its OpenFlow cookie back to the Southbound logical flow that installed it. * fix: exempt an EIP instance from the subnet drop gate with allow, not reroute A subnet with no route of its own carries a priority-1100 drop policy, and an instance with an external IP is lifted above it by a priority-1200 policy. That policy rerouted to the pool gateway, which in distributed NAT is outside every logical router port subnet, because the gateway LRP is the link-local /30. The static default route survives that by naming an output port; a routing policy has no such field, so ovn-northd discarded all three of them and the drop gate below swallowed everything the instances sent. The effect was that every external IP answered inbound traffic and none of it came back: the SYN reached the guest, the guest replied, and the reply died in the router's policy stage. Guest-initiated egress was lost the same way. The policy only ever needed to lift the gate. The router's default route already points at the gateway, so allow leaves the routing decision intact and gives northd nothing to reject. * test: match the log the firewall quorum gate actually emits The assertion waited for "chassis encap addresses; not writing a partial peer set", which no code has ever logged. The reconcile reports a short chassis list as "OVN reports N of M chassis encap addresses" and the maintain loop carries that into its retry warning, so the wait could only ever time out. Arming the node in the previous commit got the test as far as this assertion for the first time, which is why it surfaced now. The failure also prints the journal tail and the same inputs the earlier wait reports. * fix: stop one unreadable volume document failing every volume listing A single volume document whose shards no longer reconstruct made DescribeVolumes, DescribeVolumeStatus and DescribeVolumesModifications fail for every account, because the listing returned the first fetch error rather than skipping the document. ListVolumes now skips a document it cannot read or decode and logs the skip, mirroring ListAMIs. Callers whose answer would be wrong rather than merely partial take ListVolumesStrict: the orphan scan and the snapshot in-use check both gate destructive work on a complete list. The tolerance covers fetch failures as well as decode failures. An object whose shards no longer join is as unusable as one whose bytes will not parse, and it was the fetch path that took the cluster down.	2026-08-15 14:22:49 -05:00
Josh Siviour	80faf59a43	fix(ovn): bind the NB/SB client listeners to loopback and the lan plane (#765) ... setup-ovn.sh set the OVN Northbound and Southbound client remotes with a bare set-connection ptcp:6641 / ptcp:6642. The address-free ptcp: form binds every interface, so on a multi-plane node the unauthenticated, unencrypted OVSDB control plane was reachable from the public interface. Write access to the Southbound database is control of the whole tenant network fabric. ovsdb-server takes remotes from two additive sources: the Connection table, which RAFT replicates cluster-wide, and local ovn-ctl command-line flags. That split is what the fix uses. Loopback goes in the replicated row, because 127.0.0.1 is correct on every node, so replication is harmless and local clients keep working. The lan address goes on the per-node command line via --db-nb-addr and --db-nb-create-insecure-remote, which never replicates, so each node binds its own address without pushing it to its peers. A new --lan-addr flag carries that address, falling back to --db-cluster-local-addr so clustered nodes stay correct without caller changes. The standalone branch now writes /etc/default/ovn-central too, since a two-node install still has a remote ovn-controller that needs the lan listener. The guard on --db-nb-addr is load-bearing: it defaults to 0.0.0.0, so emitting create-insecure-remote without a resolved address would recreate the bug. install-node.sh passes the lan address on both branches and warns when the lan plane has collapsed onto the public address, since the listener then lands on the public interface and a firewall is the only remaining control.	2026-08-12 10:37:01 +10:00
Julian Sommer	2e09461080	chore: cleanup code comments	2026-08-04 14:20:19 +10:00
Ben Duncan	b76aab8b50	feat(installer): three-plane networking, ZFS root, distributed NAT and multi-node install (#663)	2026-08-04 10:21:12 +10:00
Josh Siviour	628fbbc0c3	fix: group-scope the OVS/OVN sockets and retire the vpcd sudoers grants (#609) ... * fix: group-scope the OVS/OVN sockets and stop sudoing read-only probes The OVS DB socket and both sets of ctl sockets were chmod 0666 by setup-ovn.sh, with a systemd override re-applying it on every openvswitch-switch restart. Write access to db.sock is full control of the local datapath, so this handed the datapath to any account on the box. Both service users run with Group=spinifex, so the access is now 0660 root:spinifex, /var/run/ovn is 0750, and a shared helper re-applies it from ExecStartPost on openvswitch-switch and ovn-controller alike — the latter is new, and matters because the ovn-controller ctl socket name embeds the pid and is recreated on every start. That also fixes the health probe reporting ovn-controller as not_running on a healthy node. It shelled out to `sudo ovn-appctl`, which only spinifex-vpcd is granted; as spinifex-daemon sudo demanded a password and the swallowed error read as OVN being down. With the ctl socket group-owned the probe needs no privilege, so health.go moves to a non-escalating Runner rather than the alternative of granting the daemon a root-equivalent ovn-appctl rule to read a status. ListIMDSTaps ran `list-ports` plus one `get` per patch port on vpcd's 15s tick, so 1+N sudo invocations every tick, each one logging a pam_limits warning about the core rlimit the unit deliberately pins to 0. It now issues a single JSON query per pass, and the sudoers drop-in disables the PAM session for both service users, which also leaves sudo children inheriting the unit's rlimits. An absent iface-id on a patch port is now skipped with a warning instead of aborting the pass: it is the only record of the full ENI, so such a port cannot join the live set either way, and failing the pass stalled reconcile for every healthy port behind it. A transient query failure still aborts. * fix: group-own the OVS/OVN pidfiles too, and cover the ipsec ctl socket Group-owning the ctl sockets was not enough for `ovn-appctl -t ovn-controller`: it resolves the pid-named socket through the pidfile, and OVS opens that O_RDWR to test the fcntl liveness lock, so a group-readable pidfile still fails the probe with EACCES. The pidfiles now get the same 0660 root:spinifex, which also tightens them from the 0644 they ship with. openvswitch-ipsec joins the override list because setup-ovn.sh restarts it after the sweep runs, leaving its ctl socket as the one file still at the shipped mode. Verified on the dev node: every socket and pidfile is 0660 root:spinifex and survives a restart of all three units, an unprivileged user is refused, and /health reports ovn-controller ok where the sudo probe read not_running. * refactor: stop escalating the OVS/OVN socket clients Now that setup-ovn.sh group-owns the control sockets, ovs-vsctl, ovs-appctl, ovn-nbctl, ovn-sbctl, ovn-appctl and systemctl reach their daemons as the service user. utils.NeedsPrivilege carries that policy and both command builders consult it, so ip, iptables, dhcpcd, sysctl and arping still escalate while the socket clients no longer do. Their sudoers grants are removed: each took unrestricted args and all of them accept --log-file=PATH, so every one was a root-equivalent grant handed out to run a query. ovs-ofctl keeps its grant. It talks to a per-bridge <bridge>.mgmt socket that ovs-vswitchd creates when the bridge appears, including bridges spinifex creates at runtime, so the provisioning sweep cannot group-own it up front. The NB and SB database sockets join the sweep for ovn-nbctl and ovn-sbctl. vpcd carried its own sudoCommand copy that escalated unconditionally, bypassing the policy entirely. It survived only because the grants existed and broke the OVN flows-ready barrier the moment they were removed, so it now delegates to utils.SudoCommand. TestRG10_SudoOnlyThroughThePolicy fails the build if another package reintroduces a local copy. Verified on the dev node: an instance launches, IMDS is served, the flows-ready barrier completes, and no sudo denial appears anywhere in the journal. * refactor: drop every spinifex-vpcd sudoers grant for ambient capabilities vpcd held NOPASSWD grants for ip, iptables, sysctl and arping. The ip grant alone was root-equivalent: sudoers cannot constrain its arguments, so `sudo ip netns exec <ns> /bin/sh` was a root shell for the service account. The unit already grants CAP_NET_ADMIN and CAP_NET_RAW ambiently, and the kernel passes an ambient set to each child on exec, so those tools work as the service user with no escalation at all. Verified on a live node: all sixteen ip subcommands vpcd issues, plus iptables reads and writes, sysctl -w on the net.ipv4.neigh tree, and arping, all succeed under exactly the unit's ambient set. spinifex-vpcd now has no sudoers rules. NeedsPrivilege had to become capability-aware to do this. spx is one binary shared by both service units, and the daemon holds CAP_SYS_ADMIN and CAP_DAC_OVERRIDE but no CAP_NET_ADMIN, so a static tool list cannot express "vpcd runs ip directly, the daemon escalates it". The decision now reads CapAmb from /proc/self/status once and matches it against the capability each invocation needs. sysctl is resolved per key, since CAP_NET_ADMIN covers the net.* trees and nothing else. The dhcpcd grants go too: neither daemon invokes it. Only setup-ovn.sh, which runs as the provisioning user, and the in-guest ecs-agent. TestRG10_VPCDHoldsNoSudoersGrant pins both halves together, so a future grant cannot be re-added without also explaining why the capability stopped working. * chore: restore routeTree.gen.ts swept in by mistake * fix: probe ovn-controller with ovn-appctl, not ovs-appctl checkOVNController's first probe passed a bare -t ovn-controller to ovs-appctl, which resolves it against OVS_RUNDIR — /var/run/openvswitch. ovn-controller's socket and pidfile are in /var/run/ovn, so the probe could never succeed and logged "ovn-controller.pid: open: No such file or directory" on every check. Not a functional break: the next probe globs the full /var/run/ovn path and succeeds, so the function still returned nil. It cost an exec per check and made a healthy node look like it was failing an OVN liveness probe. ovn-appctl defaults OVS_RUNDIR to /var/run/ovn, so the common case now matches first. The glob and systemctl fallbacks stay for older layouts.	2026-07-30 17:33:24 +10:00
Josh Siviour	a0659ae7d0	fix: gate ovn-controller on SB RAFT readiness and force recompute (#599) ... Single-node OVN RAFT bring-up gated only on the Northbound DB socket, then restarted ovn-controller without waiting for the Southbound RAFT to elect a leader, serve clients, and for northd to write the gateway logical flows. With no join step to absorb the boot window the controller attached mid-election, programmed a partial datapath (SNAT ct-commit without output/delivery) and never reconverged, so NAT-GW egress and EIP ingress intermittently black-holed on the single-node baremetal CI host. Add an SB readiness gate before ovn-controller starts (SB reachable and, when clustered, an elected leader), and after ovn-controller restart wait for its SB connection then force inc-engine/recompute to rebuild all OpenFlow from the converged Southbound. Applies to 1/2/3-node RAFT alike.	2026-07-29 16:22:30 +10:00
Josh Siviour	e1c4f85a36	feat: routed-NAT external mode with two-tier ingress (#475) ... * feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper * feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode * feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules) * feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode * feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks * feat(scripts): add --nat-uplink routed mode to setup-ovn.sh * test(e2e): add routed-NAT single-node natuplink suite Covers the external_mode=nat lane end to end on a nat-initialized node: host transit wiring (veth, br-ext port, ip_forward, iptables egress rules), spinifex.toml contents, disabled EIP surface (empty describe-addresses, UnsupportedOperation on allocate), default subnet MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit /24, guest outbound egress proven via a serial-console probe (nat mode has no inbound path), and unique transit gateway IPs across VPCs. Suite self-gates on external_mode=nat; registered as e2e-natuplink and in SUITES_BUILD so it compiles in CI. Manifest entry stays commented until CI can provision a nat-mode environment. * feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column Adds the exempted_ext_ips column (optional strong Address_Set ref) to the nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an in-place SetNATExemptedExtIPs update on the client interface, live client, and mock. Test NB schema gains the column and covers insert-resolves-UUID, converge, clear, and missing-NAT paths. * feat(policy): exempt address set on routed SNAT/EIP rows Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR) with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip path patch the ref in place on pre-existing rows. The set is a singleton strong ref and is never deleted. Non-routed modes are unchanged. * feat(host): VPC ingress route primitives for routed-NAT mode EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the spx-nat veth. Replace/del semantics keep both idempotent; a missing route on removal is not an error. * feat(external): routed ingress leg on IGW attach/detach, reconcile-safe AttachIGW's routed leg now also installs the host VPC ingress route via RoutedIngressHooks, and re-runs on the already-attached path so the reconcile loop restores host routes (and patches legacy SNAT exempt refs) after a reboot — OVN state survives restarts but host routes do not. DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs, validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct pass nil hooks and are unchanged. * test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row pointing at spinifex_nat_exempt (containing the transit /24), and — after opening the SG from the transit net (default-closed, AWS parity) — ping and an sshd TCP handshake against the instance private IP from the host, which proves the un-SNATted return path. Phase 8 additionally waits for the second VPC's ingress route to appear on IGW attach. vpc-networking README documents the jumpbox pattern, SG note, and nat_exempt_cidrs. * feat(config,admin): public wan pool alongside nat-transit Restructure admin config generation from single-pool scalars to a Pools slice so nat mode can emit both the transit pool and an optional public wan pool. nat mode now accepts --external-pool/--external-source (plus --external-bind-bridge/--gateway-ip when a public pool is requested), validated by the shared resolvePublicPoolFlags helper also used by pool mode. Transit pool always renders first so the IGW allocator picks it. Formation wire format unchanged (nat mode is single-node only). * feat(daemon): enable EIP service and auto-assign in nat mode with public pool External IPAM (and with it the real EIP service, RunInstances public IP auto-assign, and MapPublicIpOnLaunch on default subnets) now activates whenever public pools exist: pool mode as before, plus nat mode when a wan pool rides alongside the transit segment. The transit pool is filtered out of IPAM — its addresses are gateway-LRP plumbing, never allocatable EIPs. Transit-only nat configs keep the disabled EIP stub. * feat(host): per-EIP ingress primitives for routed-NAT mode DetectUplinkFor parses ip route get to find the interface and source IP facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move). EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC gateway LRP transit IP (src-hinted so host-local traffic still DNATs), answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC, WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated. * feat(policy,vpcd): host EIP binder option on NATManager WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds after priming on both the fresh-add and idempotent-skip paths (host routes and proxy-ARP are volatile even when the OVN row survives, so reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort. The next hop comes from the new gatewayPortIP helper reading the gateway LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID key now an exported policy const shared with the external package. vpcd selects the IGW pool by name instead of index — with two pools in nat mode the gateway allocator must stay on the transit pool — and wires the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced pools (gateway unknown until lease) proxy-ARP on their bind bridge. * feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection dhcp_mac = "derived" (default) | "interface" on external pools. The interface variant leases with the bind interface's own MAC — WiFi/WWAN uplinks drop frames with foreign source MACs, so derived chaddr MACs never complete DORA there. Option 61 stays the lease discriminator. UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and persists through KV so restart re-acquires repeat it. Routers that key leases on MAC and ignore client-id would hand every lease the same IP; the manager detects an ACK matching the interface's own address or another client-id's live lease, releases the colliding lease, and hard-errors advising a static range instead. admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp pools bound to non-bridgeable uplinks (wlan/wwan/ppp). * test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs The natuplink suite now detects a public (non-transit) pool in spinifex.toml and switches lanes: Tier-1-only configs keep the EIP-disabled and no-public-IP assertions; public-pool configs assert the EIP surface is enabled, the default subnet has MapPublicIpOnLaunch=true, instances auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink, per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the EIP, reconcile replay across a vpcd restart, and teardown on disassociate. Docs: vpc-networking README gains a public-pool section for routed-NAT nodes (init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool. * fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd Routed-NAT mode shells out to iptables for the transit masquerade and per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for host-delivered EIPs. Without these sudoers entries vpcd crash-loops with "command not allowed" on nat-mode nodes. * fix(ovn): scope DeleteStaticRoute prefix match to the target router Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache match could grab another router's row and fail with a referential integrity violation, wedging IGW attach on multi-VPC nodes. Match only rows referenced by the target router, mirroring FindStaticRoute. Mock updated to the same semantics, with a regression test covering drift replacement across two routers. * fix(reconcile): always replay AttachIGW so host ingress state is re-ensured applyIGWs skipped AttachIGW whenever the gateway switch port existed in actual state, so the already-attached re-ensure path never ran and routed-NAT host ingress routes lost on reboot were never reinstalled. AttachIGW is idempotent; run it unconditionally. * fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes collide on the single host route table and the last IGW attach silently stole the route (route replace). Pass vpcID and the gateway LRP IP through RoutedIngressHooks; the bootstrap VPC always installs its route, other VPCs only install when the CIDR is free and only remove a route their own gateway IP holds. Adds host.VPCIngressRouteVia to read the current route holder. * ci(e2e): add nightly cell 19 for routed-NAT single-node suite Cell 19 bootstraps with --external-mode=nat plus the CI static pool as the public wan pool and runs the natuplink suite, covering the Tier-1 transit datapath and Tier-2 EIP ingress. The run step exits 0 when natuplink.test is absent from the tests tar so scheduled runs against branches predating the suite stay green. * fix(admin): nat-mode init summary reflects configured public pool The summary printed 'outbound-only — no public IPs/EIPs' even when --external-pool/--external-source configured a public wan pool. Print the pool range (or dhcp bind) and note EIPs are enabled. * fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites Reconcile the routed-NAT branch with dev's ctx-first service APIs after rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService interface, the vm_adapters public-IP resolver asserts the ctx-signed AssociatedPublicIPForInstance, and the default-subnet mapping test threads ctx into DescribeSubnets.	2026-07-09 09:43:32 +10:00
Josh Siviour	da33e574d4	fix: OVN RAFT multi-node bring-up (northd member list, ExecStop, add-nat deadline) (#466) ... * fix: repoint DB-node ovn-northd at RAFT member list; extend add-nat deadline Two related fixes for multi-node OVN clusters: - setup-ovn.sh (mulga-siv-504): in the clustered --management branch, point ovn-northd's client NB/SB connections at the full RAFT member list derived from OVN_REMOTE, so the active northd follows the leader instead of dialing its local member only. Without this, northd stops advancing SB_Global.nb_cfg once leadership moves off the node, wedging the ovn-nbctl --wait=hv barrier. Guarded on a real member list so the standalone single-DB-node path is unchanged. - utils/nats.go (mulga-js-206): raise the add-nat request deadline from 10s to 45s so the client no longer rolls back before the server-side --wait=hv flows barrier (30s) completes. * fix: stop ovn-northd restart from tearing down split NB/SB ovsdb servers The packaged ovn-northd.service ExecStop runs 'ovn-ctl stop_northd' without --ovn-manage-ovsdb=no, so restarting northd to apply OVN_CTL_OPTS also stops the separately-managed NB/SB ovsdb-server units. On a clean clustered bring-up this races the freshly-started DBs and kills them, so setup-ovn's wait for the NB socket times out and the RAFT create/join aborts. Install an ovn-northd.service drop-in overriding ExecStop with --ovn-manage-ovsdb=no so northd never touches the DB lifecycle (mulga-siv-504).	2026-07-09 09:43:30 +10:00
Josh Siviour	0823bf8c16	feat: OVN OVSDB RAFT clustering for multinode Spinifex (epic mulga-siv-486) (#458) ... * feat: add OVSDB RAFT cluster primitives to setup-ovn.sh Add --db-cluster-local-addr / --db-cluster-remote-addr flags so a management node can run the OVN NB/SB databases as a native OVSDB RAFT cluster instead of a single non-clustered ovn-central. When a local addr is given, write a shared OVN_CTL_OPTS to /etc/default/ovn-central (NB RAFT 6643 / SB RAFT 6644) and restart the per-DB units directly, since the ovn-central aggregator is ExecStart=/bin/true and won't restart its children. Empty remote addr creates the cluster (init node); a set remote addr joins it. NB/SB client set-connection runs only on the create node (RAFT replicates it cluster-wide). The single-node --management path is unchanged when no cluster addrs are supplied, so dev boxes and existing single-node clusters keep working. --ovn-remote already passes through verbatim, so a comma-separated SB endpoint list works for compute nodes; documented in usage. * feat: OVN client multi-endpoint failover for RAFT clusters Accept a comma-separated OVN NB/SB endpoint list so vpcd can target a clustered DB. config.ParseEndpoints splits the list (trimming whitespace, dropping empties); live.go Connect passes one libovsdb WithEndpoint per endpoint, so the client tries them in order, fails over on disconnect, and follows RAFT leader redirects. Single-endpoint strings still yield one endpoint, so single-node dev is unchanged. ovn-sbctl --db= already accepts the raw comma list, so the SB CLI consumers (discoverChassis, gateway_claim) need no change. * feat: write clustered OVN NB/SB endpoints across the RAFT quorum spx admin init/join now emit comma-separated OVN NB/SB endpoint lists for the first three quorum nodes (formation.BuildOVNDBAddrs) instead of loopback or a single leader address, so vpcd fails over across the cluster. firstboot defers OVN bring-up to the provisioning controller when it owns formation (SkipFormation): a standalone single-node .db would otherwise block the controller's clustered create/join. * test: add multinode OVN RAFT cluster + failover e2e; document clustered topology ovn_raft_test.go asserts NB/SB cluster/status reports a 3-server quorum with one leader on every DB node, then stops the NB leader's ovn-central and proves an NB write still lands via a surviving endpoint. Docs: vpc-networking gains the multi-endpoint vpcd config and a RAFT cluster-status debugging section; network-connections lists the 6643/6644 RAFT ports.	2026-07-09 09:43:29 +10:00
Josh Siviour	fd3dc6b680	feat(network): ADR-0006 layer redesign + IPsec/OVN hardening (#255) ... * refactor(vpcd): remove macvlan bridge mode Macvlan was a single-NIC SSH-safe hack that predated veth mode and was never selected by any active config path. Deletion is Phase 0 item 1 of the network layer redesign (mulga-siv-125.1). Removed: - BridgeModeMacvlan const + setMacvlanMAC + ifaceIsMacvlan probe - Macvlan MAC alignment block in launchService (only fired on dead path) - Macvlan branch in detectBridgeMode + matching tests - Macvlan duplicates of IGW/AddNAT centralized-NAT tests (veth versions already cover the same path) - --macvlan flag + macvlan case branch in setup-ovn.sh Refactored: - useCentralizedNAT() now == BridgeModeVeth (was != BridgeModeDirect); verifyBridgeMode already rejects empty/unknown at startup - Three IGW-pool tests now explicitly WithBridgeMode(BridgeModeVeth) (previously relied on empty-mode defaulting to centralized) Kept intentionally: - Deprecation notes in docs/compute/vpc-networking/README.md - Legacy macvlan-interface cleanup in reset-dev-env.sh + ansible teardown role (migration for operator systems with stale spx-ext-* interfaces from pre-veth installs) * feat(vpcd): resolve external CIDR at startup with bounded retry Block service start until the WAN bridge has an IPv4 address, with a 30s retry window. Phase 0 item 2 of the network layer redesign (mulga-siv-125.1). Guards the boot race where vpcd starts before systemd-networkd finishes DHCP on br-wan, or before netplan applies a static config. Skipped when ExternalMode is empty (overlay-only deployments). Forward-compatible with ADR-0006: the new resolveExternalCIDR / externalCIDRFromBridge helpers become the backing implementation of HostWiring.ExternalCIDR() once the network/host package lands in Phase 1. Until then the resolved value is only used to fail-start on missing uplink. * fix(vpcd): cover external CIDR startup paths Extract launchService's CIDR-resolve block to ensureExternalCIDRReady helper so it's unit-testable, and add direct tests for externalCIDRFromBridge (loopback + unknown-interface) to exercise the original literal body. Closes the diff-coverage gap on Phase 0 commit 15b564d which left these paths unreachable from unit tests. * refactor(network/ovn): extract L1 OVN client to network/ovn package Move OVNClient interface, LiveOVNClient, ACLSpec, ErrNATNotFound and namedUUID helper from services/vpcd to a new spinifex/network/ovn package. The mock relocates to spinifex/network/ovn/mock as a sub-package. Storage fields on the mock become exported so vpcd tests can continue to seed/inspect state across the new package boundary. services/vpcd retains thin type-alias and var-alias shims so existing callers compile unchanged; Phase 2 removes those shims as vpcd is torn down. Adds -coverpkg=./spinifex/... to test-cover so cross-package test exercise of mock methods continues to count toward coverage now that the mock lives outside vpcd. First sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md (bead mulga-siv-125.2). Pure refactor; no functional change. All existing tests pass; preflight green. * feat(network/topology): TopologyManager interface + vpcd adapter Add spinifex/network/topology with Manager interface, VPC/Subnet/Port specs, and OVN object naming helpers covering the L2 stable contract (§7 of the redesign plan). Phase 1 keeps existing in-tree OVN names (rtr-, ext-port-, sg_) and notes the §7.3 targets for a later reconciliation migration. TopologyHandler in services/vpcd grows EnsureVPC / DeleteVPC / EnsureSubnet / DeleteSubnet / EnsurePort / DeletePort / SetPortSecurityGroups methods that mirror the corresponding NATS handlers' OVN ops with Spec inputs instead of nats.Msg envelopes, satisfying topology.Manager via a compile-time check. Existing handlers are unchanged; the duplication is short-lived (Phase 2 collapses them). Smoke tests in topology_manager_test.go cover the idempotent create path, delete cascade, and declarative SG membership move. Deliverable: TopologyManager.EnsureVPC/Subnet/Port callable; all existing tests green. Second sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md (bead mulga-siv-125.2). * feat(network/host): L0 HostWiring interface + physical/veth impls Phase 1 deliverable for mulga-siv-125.2. Introduces network/host (L0 of ADR-0006): UplinkMode enum, Wiring interface, and Physical/Veth implementations with injectable Runner + InterfaceReader for testability. Not yet wired into daemon/vpcd — Phase 2 replaces daemon.SetupComputeNode and vpcd's resolveExternalCIDR with calls through this package. * feat(network/policy): L3 SecurityGroup/NAT/Route managers Introduce network/policy as the data-plane policy layer (ADR-0006 L3). Three managers, all backed by network/ovn: - SecurityGroupManager: clear+add ACL replacement on an existing port group; port group lifecycle stays with L2 (topology.Manager). - NATManager: bound to one NATMode at construction (derived from host.UplinkMode). Distributed sets logical_port/external_mac on EIP dnat_and_snat rules; centralised leaves them unset. Per-EIP, NAT GW, and IGW-default SNAT entrypoints; all deletes idempotent. - RouteManager: AddStaticRoute is read-then-decide so retries no longer duplicate rows (closing the long-standing non-idempotency bug in vpcd/topology.go's IGW default-route path). ACL priority constants and infrastructure ACL builders move from vpcd/{security,acl}.go into policy/acl.go with their semantics preserved (default-deny 900/800 logged, DHCP-allow 1050, tenant 1000). No vpcd wiring changes here — additive only; the reconciler (bead .3.2) and network/external (bead .3.4) will consume these managers. bead: mulga-siv-125.3.1 * feat(network/external): L5 IGW/EIP/NATGW managers Pure additive — new package, no topology.Manager changes, no rewiring of existing vpcd handlers. IGWManager builds the external switch + localnet + gateway LRP + default route + chassis HA, with a flows barrier callers can inject (production: ovn-nbctl --wait=hv sync). EIP/NATGW are thin L5 wrappers over policy.NATManager. GatewayIPAllocator interface lets the DHCP-backed allocator stay in vpcd until the DHCP-mgr removal bead lands; LinkLocalAllocator + StaticRangeAllocator ship here. * feat(network/reconcile): L6 intent-actual reconciler replaces 5-pass startup Build spinifex/spinifex/network/reconcile/ package per Phase 2.2 of the network layer redesign. Reads desired state from NATS KV filtered by vpc.AZ (with legacy empty-AZ matches-all rule), scans actual state from OVN NB DB, diffs and applies with VPC → Subnet → Port → SG → IGW → EIP/NATGW topological ordering. Atomic port-group join via CreateLogicalSwitchPortInGroups closes the racy gap the legacy scanENIPortMembership pass used to revive 30s later. A 5-minute drift goroutine (leader-gated via JetStream KV CAS) replaces the old 30s ReconcileSGsLoop. Adds AZ field to VPCRecord stamped at create time. Empty AZ on pre-Phase 2.2 records matches every local AZ, so no migration pass is required. Replaces and removes the 5-pass startup sequence in services/vpcd: - Reconcile + ReconcileFromKV - ReconcileSGsOnce + ReconcileSGsLoop - RetrofitAllExternalLocalnetOptions - RetrofitAllGatewayPortNetworks IGW fresh-create continues to flow through the legacy NATS handler path (handleIGWAttach) so the dhcpManager wiring stays intact until Phase 2.3; the reconciler only rebinds gateway-chassis on existing IGW topology. Closes mulga-siv-125.3.2. * refactor(network/topology): promote live Manager impl to network/topology Move the topology.Manager implementation out of services/vpcd/topology_manager.go into network/topology/manager_live.go as a standalone liveManager constructed via NewLiveManager(ovn.Client, opts...). Inputs that varied per-handler (DNS server resolution from the external pool) are now passed as functional options (WithDNSServer). TopologyHandler still satisfies topology.Manager but its methods delegate to a cached liveManager. The shim stays until the rest of mulga-siv-129 routes NATS subscribers through topology.Manager directly. Also exports SubnetGatewayCIDR + FormatDNSServerList from network/topology so the reconciler in network/reconcile can share helpers with the live manager (audit pass to follow per plan §6). Test cases ported one-for-one from services/vpcd/topology_manager_test.go to network/topology/manager_live_test.go; the vpcd-side test file is deleted. New WithDNSServer test asserts the option propagates into DHCPOptions rows. Refs mulga-siv-129 (Phase 2.6 vpcd subscriber migration, Slice 1 of N). * refactor(vpcd): collapse SG subscribers onto network/{topology,policy} managers Phase 2.6 / mulga-siv-129 slice 2. Rewrites the vpc.create-sg / vpc.update-sg / vpc.delete-sg NATS subscribers in services/vpcd as thin wrappers over topology.Manager (port group lifecycle) and policy.SecurityGroupManager (ACL programming). Deletes the ~1000 lines of duplicated ACL builder and provisioning logic — its authoritative copy lives in network/policy. Surface additions: - ovn.ErrPortGroupNotFound sentinel; mock + live GetPortGroup wrap not-found via fmt.Errorf("%w: ...") so callers can use errors.Is. - topology.Manager grows EnsureSGPortGroup / DeleteSGPortGroup (resolves the apply.go TODO from §2.1 of the redesign plan). liveManager implements them with GetPortGroup-then-Create / GetPortGroup-then-Clear+Delete idempotency. Behaviour delta: the old provisionSG rolled back the OVN port group on AddACLs failure ("all-or-nothing"). The new path lets the port group survive and relies on network/reconcile to retry EnsureSG on its next sweep. The all-or-nothing fail-fast test cases are dropped; ACL emission semantics are covered by network/policy/sg_test.go. * refactor(vpcd): migrate VPC/Subnet/ENI subscribers to topology.Manager Phase 2.6 / mulga-siv-129 slice 3. Rewrites vpc.create / vpc.delete / vpc.create-subnet / vpc.delete-subnet / vpc.create-port / vpc.delete-port / vpc.update-port-sgs NATS subscribers as thin wrappers (≤30 lines each) over topology.Manager. Subscriber bodies now: parse event → translate to spec → delegate to manager. Deletes: - vpcd.subnetGateway (and helpers_test.go + topology_test.go's TestSubnetGateway block); parity coverage lives in network/topology.TestSubnetGatewayCIDR. - vpcd.reconcilePortSGs (the live manager already exposes the same diff-then-transact via reconcilePortSGs on the L2 side). - vpcd.TopologyHandler.dnsServer; lmDNSServer (passed via topology.WithDNSServer to the live manager) is the live path. Behaviour delta: handlers now strict-parse PrivateIpAddress / MacAddress / CidrBlock at the NATS boundary and reject malformed payloads instead of degrading to "no DHCP options". The reconciler guarantees that field validity is upheld upstream of the publish. * refactor(network/reconcile): drive SG port groups via topology.Manager applySGs now calls topology.Manager.EnsureSGPortGroup / DeleteSGPortGroupByName instead of reaching directly into the OVN client. Reconcile.Config gains a Topology field (required); the production caller in services/vpcd passes the TopologyHandler shim, the reconcile package tests construct a fresh topology.NewLiveManager bound to the mock OVN client. DeleteSGPortGroupByName is added to the topology.Manager interface for the reconciler's orphan-removal path, which iterates raw OVN port-group names without a groupID round-trip. The existing DeleteSGPortGroup(groupID) delegates to it. Resolves the TODO at apply.go:137 carried over from slice 2 of mulga-siv-129. * feat(network/policy): add FlowsBarrier option to NATManager NATManager.AddEIP needs to block on `ovn-nbctl --wait=hv sync` after writing the dnat_and_snat row so the NATS reply only returns once every chassis has the rewrite flow. The legacy vpc.add-nat handler does this at the tail of handleAddNAT; the manager has lacked it since slice 2.1. NewNATManager grows a functional-options tail (Option / WithFlowsBarrier). Defaults to a no-op for tests; production wiring in services/vpcd passes waitForFlowsHV. AddEIP now invokes the barrier after AddNAT, logging at warn level on failure to match the legacy handler's _ = waitForFlowsHV() swallowing semantics. Pre-work for slice 4 of mulga-siv-129 — without the barrier the NAT EIP subscriber migration would lose the post-write flow-install gate. * refactor(vpcd): migrate NAT GW subscribers to policy.NATManager handleAddNATGateway / handleDeleteNATGateway now decode the NATS event and delegate to policy.NATManager.{AddNATGateway,DeleteNATGateway}. The manager carries identical OVN logic — same snat type, same ExternalIDs (spinifex:vpc_id + spinifex:nat_gateway_id) — so existing TestTopologyHandler_AddNATGateway / DeleteNATGateway suites stay green. Lazy-constructed NATManager added on TopologyHandler (natmOnce). Mode is derived from the bridge mode (veth → centralized, direct → distributed) and the FlowsBarrier is bound to waitForFlowsHV for parity with the legacy EIP path's tail barrier (used by the next slice, vpc.add-nat). Slice 4 of mulga-siv-129, sub-step b. * refactor(vpcd): migrate EIP NAT subscribers to policy.NATManager handleAddNAT / handleDeleteNAT now decode the NATS event and delegate to policy.NATManager.{AddEIP,DeleteEIP}. The behavioural contract is preserved end-to-end: - Idempotency-skip on a matching dnat_and_snat row (same VPC + LogicalIP, plus MAC + LogicalPort under distributed NAT) moves into AddEIP so duplicate vpc.add-nat publishes never tear down the rule the previous invocation just installed (mulga-siv-124). - Stale-rule cleanup across every router stays in AddEIP. - Post-write waitForFlowsHV barrier is now driven by the manager via the FlowsBarrier option wired in services/vpcd (slice 4a). - DeleteEIP returns nil on ErrNATNotFound, matching the handler's fire-and-forget "already absent = success" semantic. The vpcd "errors" import drops out with the inline error-comparison. Slice 4 of mulga-siv-129, sub-step c. * refactor(vpcd): migrate IGW subscribers to external.IGWManager (partial) handleIGWAttach now branches at decode time: - Centralised NAT + DHCP-sourced pool keeps using attachIGWLegacy because the gateway-LRP IP comes from vpc.dhcp.acquire, and that DHCP client is the vpcd-local manager that bead mulga-siv-125.3.3 will retire. - All other shapes (distributed NAT, static-range pool, no pool) delegate to external.IGWManager.AttachIGW. The manager mirrors every step the legacy handler ran — external switch + localnet port + gateway LRP + switch-side router peer + default route + chassis HA + waitForFlowsHV tail barrier — so observable behaviour is unchanged. handleIGWDetach drops its inline OVN walk entirely; both paths exit through external.IGWManager.DetachIGW followed by releaseGatewayLRPLease. The lease release stays in vpcd because the upstream client lives in services/vpcd/dhcp; calling it after every detach is safe because the upstream server expires unknown leases on its own. Lazy-constructed IGWManager added on TopologyHandler (igwmOnce). Pool, allocator (StaticRangeAllocator), chassis, NAT mode, NAT manager, route manager, and FlowsBarrier are all derived from the existing TopologyHandler state. The DHCP-coupled legacy branch retains its full inline OVN logic; it will disappear in slice-after-125.3.3 follow-on. Slice 4 of mulga-siv-129, sub-step d. * feat(ec2/instance): honor AssociatePublicIpAddress per-NIC override RunInstances --network-interfaces "AssociatePublicIpAddress=true|false" was silently ignored, only subnet.MapPublicIpOnLaunch gated public IP assignment. AWS contract is per-NIC override wins. Extend TestPrepareRunInstances_PublicIPAutoAssigned to table-driven 3 cases: subnet true no-override, subnet false override true, subnet true override false. * refactor(config): drop upstream-DHCP config surface (static WAN pool only) Hard-error in LoadConfig on legacy keys: [network] external_dhcp, [nodes.<n>.vpcd] dhcp_bind_bridge, and [[network.external_pools]] source = "dhcp". Add pool range validation (must fit inside the gateway/prefix_len CIDR; per-cluster non-overlap). Strip ExternalDHCP from config.NetworkConfig + formation.NetworkConfig + admin.ConfigSettings, and DhcpBindBridge from config.VPCDConfig + admin.ConfigSettings + vpcd.Config. vpcd.ExternalPoolConfig.IsDHCP() short-circuits to false; the field and downstream DHCP branches in external_ipam / topology.go remain for Slice C to delete together with dhcp_manager. cmd/spinifex/cmd/admin.go: drop useExternalDHCP path and the detectedDhcpBindBridge helper; require --external-pool when --external-mode=pool; auto-detect default no longer infers source=dhcp. service.go's wan_bridge legacy-key rejection now advises removing the key entirely (vpcd auto-detects br-wan), not renaming to dhcp_bind_bridge. Two topology_test cases that exercise the DHCP gateway-LRP path are skipped — Slice C deletes attachIGWLegacy and these tests together. * refactor(vpcd): delete dhcp_manager + DHCP gateway-LRP path (Slice C) Removes upstream-DHCP-client gateway-LRP allocation in favor of static WAN-pool allocation (StaticRangeAllocator). Part of bead mulga-siv-125.3.3. - delete services/vpcd/dhcp_manager.go + dhcp/ subpackage (~865 lines) - delete allocateGatewayLRPIPViaDHCP + releaseGatewayLRPLease paths - drop ExternalPoolConfig.Source + DhcpBindBridge + IsDHCP() fields - drop ExternalIPAllocation lease fields (Expiry/ServerID/HWAddr) and ExternalIPAMRecord.Source - drop NewExternalIPAM nc *nats.Conn argument; update all 4 call sites - rename verifyBridgeMode local dhcpBindBridge -> wanBridge; update error strings and tests Slice D (delete attachIGWLegacy + dispatcher) intentionally deferred. * refactor(vpcd): delete attachIGWLegacy + IsDHCP stub (Slice D) Closes mulga-siv-129 slice 4d. Part of bead mulga-siv-125.3.3. handleIGWAttach now unconditionally delegates to external.IGWManager. attachIGWLegacy (~207 lines, ~5 OVN write steps + chassis scheduling) deleted along with its dispatcher branch in handleIGWAttach. The ExternalPoolConfig.IsDHCP() dead stub is gone. No code paths or NATS subjects regress: the static-pool / distributed branches were already going through external.IGWManager since Slice 4d (6719dc3). * feat(ipam): Purpose field on every IPAM record (Slice B) Part of bead mulga-siv-125.3.3. Adds a Purpose enum (handlers/ec2/vpc/purpose.go) tagging every IP allocation with its semantic role (eni-primary, eni-public, eip, igw-lrp, natgw-external, …) so multi-VPC clusters can reclaim/audit by (owner, purpose). External IPAM (spinifex-external-ipam, v1→v2): - ExternalIPAllocation.Type → .Purpose; legacy values mapped via LegacyExternalTypeToPurpose (gateway→igw-lrp, auto_assign→eni-public, elastic_ip→eip). Unknown legacy values warn and become "unknown". - AllocateIP / AllocateFromPool / allocateFromPool param rename allocType→purpose. All call sites pass Purpose* constants. - migrate/002_external_ipam_purpose.go: idempotent CAS-retry migration with v1/v2-union record shape so partial migrations resume cleanly. Internal IPAM (spinifex-vpc-ipam, v1→v2): - IPAMRecord.Allocated []string → []IPEntry{IP, Purpose, OwnerID}. - AllocateIP gains purpose+ownerID args; ReleaseIP unchanged (still keyed on IP). - AllocatedIPs returns []IPEntry. - eni.go reorders eniId generation before AllocateIP so the owning resource ID lands on the new entry. - migrate/003_ipam_purpose.go: cross-bucket backfill that reads the ENI bucket via the new RunKVWithJetStream path. Orphaned IPs (no owning ENI) warn and become Purpose=unknown. Migrate framework: - KVContext gains an optional JetStream field; new Registry.RunKVWithJetStream method threads it through. RunKV signature unchanged — all other callers are unaffected. Tests: - ipam_test.go: AllocateIP signature updates; AllocatedIPs now asserts IP / Purpose / OwnerID on the returned entries. - external_ipam_test.go: replaces "auto_assign"/"elastic_ip" literals with PurposeENIPublic/PurposeEIP and asserts alloc.Purpose. - migrate/002 + 003 tests: schema-detect, idempotent rerun, no-JetStream fallback, empty bucket. * refactor(daemon): tombstone obsolete spinifex-dhcp-leases KV bucket The dhcp_manager removed in Phase 2.3 (bead mulga-siv-125.3.3) owned the spinifex-dhcp-leases KV bucket. The migrate framework only handles per-bucket schema version bumps, not bucket deletion, so the obsolete bucket would linger on upgraded clusters. Add a one-shot tombstone after initJetStream that deletes the bucket if present. DeleteKVBucketIfExists treats ErrBucketNotFound and ErrStreamNotFound as success, so the call is idempotent across restarts. * feat(config): add network.ipsec_enabled cluster flag (default true) Add IPSecEnabled to NetworkConfig in both the cluster TOML schema and the formation handshake payload. Default true so AWS-parity intra-AZ encryption is on by default. Operators can flip false in [network] for trusted single-rack lab topologies. Phase 2.5 (bead mulga-siv-125.3.5) consumes the flag from admin init (strongSwan provisioning), formation server (propagates to joiners), and daemon startup (ovs-vsctl ipsec_encapsulation toggle). * feat(admin): IPsec peer cert generator signed by cluster CA Add GenerateIPSecPeerCert + IPSecCertPaths in spinifex/admin so admin init (and the future admin add-node path) can issue per-node IKEv2 peer credentials signed by the existing cluster CA. The cert carries id-kp-ipsecIKE (RFC 4945) so strongSwan accepts it as an IKEv2 peer credential when ovs-monitor-ipsec materialises tunnel configs. Cluster CA is reused directly — no intermediate strongSwan PKI today; when intermediate CAs land that becomes a separate workstream. The cert/key pair lives under {configDir}/ipsec/peer.{pem,key}; the daemon points OVS Open_vSwitch.other_config (certificate, private_key, ca_cert) at these paths when network.ipsec_enabled is true. * feat(admin): wire IPsec peer cert into admin init + join paths Add --ipsec bool flag (default true) on adminInitCmd. Single-node init and the multi-node leader path both emit an IPsec peer cert after the existing CA+server-cert generation. The multi-node formation server propagates the IPSecEnabled flag to joining nodes via NetworkConfig; admin join honours it and emits its own peer cert during the same cert-issuance step that already produces the server cert. spinifex.toml template gains an unconditional [network] block carrying ipsec_enabled so the value is visible to operators on every install regardless of whether external networking is configured. The daemon reads it via cluster config (NetworkConfig.IPSecEnabled). Bead mulga-siv-125.3.5. * build: install openvswitch-ipsec for OVN native IPsec openvswitch-ipsec provides ovs-monitor-ipsec, which reads the Open_vSwitch.other_config cert/key/CA pointers and materialises strongSwan configs for every Geneve tunnel managed by ovn-controller. Pulls in strongswan + strongswan-pki as transitive deps. Required by bead mulga-siv-125.3.5. * feat(daemon): enable OVN native IPsec on intra-AZ Geneve at startup When network.ipsec_enabled is true (default), the daemon points the local OVS DB other_config at the per-node IPsec peer cert that admin init / admin join provisioned (configDir/ipsec/peer.{pem,key} + the cluster CA at configDir/ca.pem), then sets other_config:ipsec_encapsulation=true. ovs-monitor-ipsec (shipped by the openvswitch-ipsec package) consumes those pointers and materialises strongSwan configs for every Geneve tunnel ovn-controller programs — including OVN-IC inter-AZ tunnels when federation lands in Phase 3a. Missing cert files log a warning and leave Geneve plaintext; operators upgrading an older cluster reissue certs via spx admin to close the gap. Bead mulga-siv-125.3.5. * test(e2e): multinode IPsec wiring + xfrm SAs + ESP capture TestMultinodeIPSec asserts: - OVS DB on every node carries certificate/private_key/ca_cert pointers and other_config:ipsec_encapsulation=true (what daemon.enableOVNIPSec writes). - ip xfrm state on every node has at least one AEAD SA in AES-GCM mode (rfc4106(gcm(aes)) or gcm(aes)) within 90s. - tcpdump catches ESP (ip proto 50) on the underlay (best-effort — logged only; geneve may be idle during the 5s capture window). Skips cleanly when the cluster was bootstrapped with --ipsec=false (no ipsec_encapsulation key on first node). Read-only over SSH; runs in the parallel bucket alongside the other non-mutating multinode tests. * fix(igw): write attachment State="attached" for AWS parity Reconciler intent loader (network/reconcile/intent.go::loadIGWs) filters IGW records with `State == "attached"`, but AttachInternetGateway was writing `State = "available"`. The reconciler therefore never picked up any attached IGW — `intent_igws=0` on every pass — so the OVN external switch + gateway LRP + WAN bridge wiring built by IGWManager.AttachIGW never ran for default-VPC IGWs (whose live `vpc.igw-attach` publish at daemon bootstrap is lost — vpcd subscriber not yet wired). Net effect on single-node e2e: every SSH probe to 192.168.x public IPs timed out (TestSSHProbe, TestModifyInstanceAttribute, TestRebootInstance, TestNATGateway, TestSGToSGDatapath) because SNAT/DNAT rules on the VPC router had no path to the external network. Match AWS InternetGatewayAttachment.State semantics: - Attach: State="attached" - Detach: State="detached" Detach value is internal — recordToEC2 gates Attachments on VpcId != "", so the detached State is never surfaced to API consumers. Bead: mulga-siv-131 * fix(reconcile): call AttachIGW when IGW topology missing applyIGWs previously deferred to the NATS handler when an intent IGW had no external switch in OVN. When the vpc.igw-attach event is lost (bootstrap-time default-VPC IGW published before vpcd subscriber is ready), deferral never resolves and VM public-IP traffic dies. Call IGWManager.AttachIGW directly — it is idempotent (short-circuits when the external switch already exists), so the reconciler converges default-VPC IGWs on first pass without duplicating work on subsequent passes. Closes mulga-siv-132. * fix(test): align NewExternalIPAM call with 2-arg signature Rebase onto dev pulled in daemon_system_instance_test.go which used the legacy 3-arg NewExternalIPAM(jsNC, js, pools). The feature branch already narrowed the signature to (js, pools) via the upstream-DHCP config drop; align the test with the current signature. * fix(vpcd): EnsureSubnet must EnsureVPC first to avoid NATS race vpc.create-vpc and vpc.create-subnet are independent NATS events with no inter-event ordering guarantee. Tenant bootstrap on iam.account.created publishes both within <1ms, so the subnet handler's CreateLogicalRouterPort can race the parent VPC's CreateLogicalRouter and fail with "logical router not found". The cascade fails CreateNetworkInterface which fails RunInstances which surfaces as InsufficientInstanceCapacity to the SDK — masking the underlying topology race. handleSubnetCreate now calls EnsureVPC(VPCID-only) before EnsureSubnet. EnsureVPC short-circuits on existing LR (idempotent), so the race-winning vpc.create-vpc handler's full CIDR/VNI ExternalIDs are preserved; if subnet wins the race the LR is created with bare ExternalIDs (cosmetic) and the follow-on vpc.create-vpc short-circuits. Adds TestIntegration_SubnetBeforeVPC regression covering both orderings. Closes mulga-siv-133. * feat(ansible): require --external-pool, drop NAT, forward EXTRA_VARS spx admin init now rejects --external-mode=nat and demands --external-pool when mode=pool. The ansible init role silently skipped pool flags when the range was unset, so installs fell over inside the spx validator. Make targets also swallowed `-e` (make's env-override flag), so operators could not override pool start/end without editing defaults.yml. - vars/defaults.yml: rename nat_gateway_ip → external_gateway_ip; add spinifex_external_no_external opt-out; document the spx flag contract. - roles/init/tasks/main.yml: drop NAT branch; assert pool range upfront with a fail_msg pointing at EXTRA_VARS / defaults.yml; emit gateway, prefix-len, gateway-ip only when explicitly set (let spx auto-detect); honour --no-external. - playbooks/dev-reset.yml: rename captured var; coerce legacy external_mode=nat to pool with a warn; tighten gateway regex with \b. - Makefile: ansible-dev-* targets now forward EXTRA_VARS="key=val ..." through --extra-vars so `make` callers can override without editing defaults.yml. - scripts/ansible/README.md: document pool requirement, both invocation styles (direct -e vs make EXTRA_VARS), NAT removal. * fix(ipsec): guard ipsec_encapsulation on ovs-monitor-ipsec liveness enableOVNIPSec set OVS cert pointers + ipsec_encapsulation=true without ensuring ovs-monitor-ipsec.service was running. Without that daemon, OVS holds the cert pointers unused, strongSwan never gets IKE config, and tunnel traffic ovn-controller marks for encryption gets silently dropped. TestMultinodeIPSec fails on `node1 xfrm has no AEAD SAs`. - systemd: spinifex-daemon Wants=/After= ovs-monitor-ipsec.service - daemon: ensureOVSMonitorIPSecActive — systemctl enable --now + is-active poll; refuses to flip ipsec_encapsulation if daemon is dead - daemon: single-node short-circuit — no peers means no tunnels means the flag is at best inert and at worst load-bearing-silent-drop - ci: capture-net-diag.sh + workflow steps to snapshot OVS/OVN/xfrm state on failure, fan out across all 3 multinode nodes Refs: mulga-siv-136 * test(e2e): sticky-skip downstream SSH tests when probe fails When TestSSHProbe times out (3 min, typically because of mulga-siv-134 datapath bug), four downstream tests each re-probe SSH on the same broken datapath and time out identically — single-node job runs ~24 min instead of ~12 min, with the same error logged five times. First waitForSSHReady timeout now sets a package-level atomic flag; subsequent waitForSSHReady / trySSHReady / requireSSHHealthy calls skip immediately. Preserves the original red signal on TestSSHProbe and on any test that genuinely needed SSH (Skip ≠ Pass). Bool variant (trySSHReady) consults but does not set the flag — its caller already treats False as advisory warning, not fatal. TestSGToSGDatapath has its own EventuallyErr SSH wait (not via waitForSSHReady) and needs an explicit requireSSHHealthy(t) gate at the top of runSGToSGDatapath. TestNATGateway already calls waitForSSHReady first so the gate fires naturally. Refs: mulga-siv-137 * fix(ipsec): use Debian unit name openvswitch-ipsec.service CI artifact shows `Unit ovs-monitor-ipsec.service could not be found` on every node. Debian's openvswitch-ipsec package ships the unit as openvswitch-ipsec.service (the binary inside is ovs-monitor-ipsec, but systemd's name follows the package). The previous commit guarded on the wrong unit name, so: - spinifex-daemon.service Wants=/After= silently no-op'd (missing unit) - enableOVNIPSec's `systemctl enable --now ovs-monitor-ipsec.service` failed, daemon logged a warning and continued without flipping ipsec_encapsulation=true - TestMultinodeIPSec skipped with `node1 reports ""` because OVS DB never received the cert pointers Rename across all 5 spots (daemon, systemd unit, test, CI diag script, workflow journal capture) to match the Debian package layout. Note: the package itself still must be present on the CI VM image — that's tracked in the mulga repo (image-builder provision.sh + bootstrap-install.sh shim). Refs: mulga-siv-136 * fix(e2e): authorize default SG tcp/22 + icmp in single-node fixture admin init leaves the default SG closed (AWS parity); the Go single suite never ported the Phase-5 SG-ingress block that run-e2e.sh has had since day one, so every test runner SSH/ping probe hits the OVN port-group ACL drop. - ec2helpers: add AuthorizeICMPIngress + EnsureDefaultSGOpen helpers. - single/main_test: open default SG once per process in pkgFixOnce.Do. - witness: extend ensureDefaultSGSSHIngress to authorize ICMP too. - run-e2e.sh: append matching ICMP authorize next to tcp/22 block. mulga-siv-134 * fix(network/reconcile): key DHCP_Options lookup by subnet_id not CIDR Multi-tenant default VPCs share the same subnet CIDR (172.31.0.0/20) per account. The existing FindDHCPOptionsByCIDR gate at apply.go:107 found the first account's DHCP_Options row and skipped creation for every subsequent subnet with the same CIDR. Those subnets' ports then attached nothing (FindDHCPOptionsByExternalID misses on subnet_id), VMs got no DHCPv4 lease, and SSH/ICMP into them silently failed even though SG ACLs and NAT were correct. Switch the gate to FindDHCPOptionsByExternalID keyed on spinifex:subnet_id so each subnet owns its own row, matching the attach lookup at line 203. Verified: account 000000000001 default VPC RunInstances now produces an LSP with dhcpv4_options populated, VM cloud-init reports a 172.31.0.5/20 lease + default route via 172.31.0.1, and ping + ssh TCP handshake to the EIP succeed end-to-end. Stale DHCP_Options row for the prior dead subnet was wiped manually; reconciler recreated the correct row for the live subnet on its next pass. * fix(ipsec): enable openvswitch-ipsec at provision time, drop daemon write Provisioning now runs `systemctl enable --now openvswitch-ipsec.service` alongside openvswitch-switch and ovn-controller. The spinifex-daemon process drops its `enable --now` shell-out and only polls is-active. Sudoers gains one read-only line: spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/systemctl is-active openvswitch-ipsec.service This mirrors the existing vpcd ovn-controller is-active precedent and removes the daemon's need for systemd-write capability. Aligns with least-privilege service account pattern: provision enables services, daemon only reads status. If the unit is inactive at runtime, the daemon refuses to flip ipsec_encapsulation=true (silent-drop guard from mulga-siv-136) and surfaces a clear error pointing at scripts/setup-ovn.sh. * chore: unblock preflight on feat/network-layer-redesign Two pre-existing failures blocked preflight on the branch: - govulncheck flagged golang.org/x/net@v0.54.0 (GO-2026-5026, ASCII Punycode rejection bug in idna.ToASCII); reached via utils.DownloadFileWithProgress. Bump to v0.55.0; go mod tidy carries x/sys to v0.45.0. - TestTopologyHandler_SubnetCreate_RouterPortFails asserted subnet create fails when the parent VPC router is missing. That contract changed in aed7b2e (mulga-siv-133), which made handleSubnetCreate call EnsureVPC idempotently before EnsureSubnet to avoid a NATS publish-order race. Delete the now-impossible failure case. * refactor(network/ovn): relocate nbdb model package from services/vpcd Phase 4 / mulga-siv-125.6 Slice A. Pure import rename, no logic change. The OVN Northbound DB model (LogicalSwitch, LogicalRouter, DHCPOptions, etc.) lived under services/vpcd/nbdb/ because that was the original home of every OVN caller. After Phase 1+2 the L1 client (network/ovn) and all L3/L5 managers (network/policy, network/external, network/topology, network/reconcile) became the primary callers — 8 importers in network/* vs 2 in services/vpcd/. The directory move puts the schema model where its consumers are, and unblocks deleting services/vpcd/ entirely once the subscriber/launcher relocations land (Slices B–D). * refactor(vpcd): delete dead retrofit/reconcileIGW helpers + bridge-mode bleed Phase 4 / mulga-siv-125.6 Slice B. Production path no longer touches any of: - RetrofitAllGatewayPortNetworks / RetrofitAllExternalLocalnetOptions — the 5-pass startup leftovers that network/reconcile (Phase 2.2) made redundant. Nothing called them outside tests. - reconcileIGW + its allocateGatewayLRPIP / gwLrpRange helpers + the ipv4ToUint32 / uint32ToIPv4 utilities — superseded by external.IGWManager (Phase 2.4) which carries its own pool logic. - ensureLocalnetOptions / ensureGatewayPortNetworks / expectedGatewayPortNetwork — read-before-write retrofit shims for the same code paths. - generateMAC — only callers were reconcileIGW + its tests. useCentralizedNAT (the bridge-mode bleed flagged in plan §2.2) is replaced by a policy.NATMode value injected via WithNATMode(...). The mode is resolved once in vpcd.go from host.UplinkMode and threaded into both the topology-handler lazy managers and the reconciler stack, so "is this veth?" is no longer a question topology.go can ask. The TopologyHandler defaults to NATModeDistributed when WithNATMode is omitted, matching the previous default. Drops services/vpcd/topology.go from 1220 LOC to 693, deletes ~700 LOC of obsolete tests (TestGenerateMAC, TestEnsureLocalnetOptions_*, TestRetrofitAllExternalLocalnetOptions_FixesClearedOptions, TestReconcileIGW_*, TestGwLrpRange), and frees the nbdb + utils + net imports in topology.go. Stale "phase 4 migrates those onto the new managers" comment in vpcd.go fixed in passing — the subscribers were migrated in Phase 2.6 (mulga-siv-129); only the file-relocation parts of Phase 4 remain. Stale ensureLocalnetOptions / ensureGatewayPortNetworks references in network/ovn/{live.go,mock/mock.go} doc comments also cleaned up. * refactor(network): move NATS VPC subscribers to network/subscribers Phase 4 Slice C (mulga-siv-125.6). Promotes the NATS event handlers out of services/vpcd into a new spinifex/network/subscribers package. Subscriber holds the topology/policy/external managers directly — the lazy *Once adapter shim (topology_manager.go) is deleted along with vpcd's local event-struct copies. launchService now constructs topology.LiveManager eagerly with a DNS callback derived from external pools (preserving the unscoped-pool lookup old TopologyHandler.lmDNSServer did), builds every manager up front, then hands them to subscribers.New + Subscribe. reconcile.New receives topology.Manager directly instead of through the adapter. Bridge-mode constants and waitForFlowsHV move into vpcd.go alongside the launcher; they were left behind as topology.go's last responsibilities. Handler-layer tests (topology_test.go, security_test.go, integration_test.go) are dropped: substance is covered by network/topology, network/policy, network/external manager tests. subscribers/subscribers_test.go adds focused coverage: missing-dep rejection, topic registration smoke, and bad-JSON envelope per topic. services/vpcd is now down to the launcher + the ovn/mock compat aliases (removed in Slice D when the launcher promotes to cmd/vpcd/). * refactor(vpcd): relocate services/vpcd → spinifex/vpcd; drop dead aliases Phase 4 Slice D (mulga-siv-125.6). Final cleanup of the network-redesign package layout. services/vpcd/ disappears entirely; the launcher remains an in-process Service impl, just rehomed under spinifex/vpcd/ (no separate binary — spx vpcd start stays the cobra entrypoint). - Delete services/vpcd/mock_ovn.go + ovn.go (compat aliases now dead post Slice C; only callsite was launchService's NewLiveOVNClient, inlined to ovn.NewLiveClient). - git-rename services/vpcd/vpcd.go + vpcd_test.go to spinifex/vpcd/. - Retarget three importers (spinifex/service/service.go + service_test.go, cmd/spinifex/cmd/service.go) to the new path. - Fix stale services/vpcd / TopologyHandler comments in network/reconcile/ + network/external/types.go + spinifex/vpcd/vpcd.go. - service-interfaces.yaml vpcd path updated. - Plan doc §13/§14/§16 + §9.2/§19 updated: drop cmd/vpcd/main.go (was speculative; never built) in favour of in-process Service impl at spinifex/vpcd/vpcd.go. Plan §16 Created list now lists subscribers/. Phase 4 plan items 1+2 (remove services/vpcd, entrypoint location) satisfied. Items 3+4 remain blocked on mulga-siv-125.4 + 125.5 (bundled join token, plan archive). * fix(ovn): install openvswitch-ipsec + strongswan explicitly ovs-monitor-ipsec.service fails to start without an IKE daemon. The openvswitch-ipsec package only Recommends strongswan, so any apt config that drops Recommends (or a base image with the recommends already skipped) leaves the service in a Restart loop with 'IKE daemon is not installed'. Hoist both packages into setup-ovn.sh so provisioning is the single source of truth — the bootstrap shim in mulga is no longer needed. * test(network): pin ADR-0006 invariants to executable suite Add static-analysis + behavioural tests that bind every safety and liveness clause in ADR-0006 (S1-S9, L1-L3) to a named Go test. Failure output quotes the ADR clause verbatim and points at the offending file:line so future debugging sessions can't quietly bypass a layer contract to make a test green. Static-analysis suite (spinifex/network/invariants/): - S1 layers_test.go: parses `go list -json`; rejects upward layer imports and layers pulling from cross-cutter packages - S2 host_containment_test.go: AST-detects ovs-vsctl / ovn-nbctl / ovs-ofctl shell-out outside network/host/ - S3 nat_mode_test.go: rejects .UplinkMode() / NATModeFromUplinkMode() calls outside the init-time resolver locations - S4 names_test.go: AST-detects OVN object name construction (concat, fmt.Sprintf, strings.Join) outside topology/names.go - S5 policy_no_create_test.go: rejects CreateLogical{Switch,Router,...} and DeleteLogical{...} calls inside network/policy/ - S6 federation_isolation_test.go: rejects TransitSwitch / TransitRouterPort references and ovn-ic literals outside network/federation/ (pre-protects the layer before it lands) - S8 ipsec_test.go: rejects strongswan / xfrm / IKEv2 / libreswan / openswan imports anywhere in network/ - S9 reconciler_singleton_test.go: rejects Reconcile…Loop and Reconcile…Pass<n> function names anywhere in network/ - S7 nats_subjects_test.go: scans for `vpc.<verb>` subjects missing the {azID} segment; calls t.Skip with the gap report so every test run prints the work-to-do without breaking CI. Flip to t.Fatalf when the cluster-wide subject rename ships. Behavioural suite (spinifex/network/reconcile/liveness_test.go): - L1 reconciler converges: scaled-intent (N=1,3,8) double-Reconcile property test; asserts second pass is a no-op - L2 drift bound: structural check that DriftInterval is positive and ≤30 min support window; full fake-clock behavioural test deferred - L3 federation re-enables: t.Skip stub pointing at the missing network/federation/ package Fixes the one production S4 violation the new walker found: network/external/igw.go was constructing "gw-port-"+vpcID inline. Adds topology.GatewaySwitchPort(vpcID) helper and retargets both AttachIGW and DetachIGW callsites through L2. Plan + per-clause mechanic + phasing in docs/development/improvements/network-invariants.md. Refs: mulga-siv-140 * fix(ovn): install strongswan-charon (not strongswan metapackage) The strongswan metapackage's '\''charon-systemd | strongswan-charon'\'' alternative resolves to charon-systemd by default, which ships /usr/sbin/charon-systemd but not the legacy /usr/sbin/ipsec helper or /usr/lib/ipsec/charon binary that ovs-monitor-ipsec --ike-daemon=strongswan shells out to. Install strongswan-charon explicitly to pull strongswan-starter (legacy ipsec controller) + the matching charon binary, matching what openvswitch-ipsec.service expects. Also reset-failed + restart openvswitch-ipsec.service after enable to recover from the deb post-install starting the service before its dependencies finished unpacking. * chore: add CODEOWNERS — joshsiv-mulga gates network/ + vpcd/ Fall-through reviewer for any change is the maintainer. network/ and vpcd/ are explicitly listed so the matching branch protection rule ("Require review from Code Owners") routes those PRs to joshsiv-mulga. Those trees are governed by ADR-0006 invariant tests (spinifex/network/invariants/); CODEOWNERS adds a human review gate on top of the mechanical fence. Refs: mulga-siv-141 * chore: CODEOWNERS fallthrough → @mulgadc/mulgadc-engineers team Fallthrough reviewer should be the team, not a single maintainer, so any engineer can approve PRs outside the explicitly-gated paths. network/ and vpcd/ keep joshsiv-mulga as the required reviewer per ADR-0006 invariant ownership. Refs: mulga-siv-141 * feat(ci): add test_run workflow input for -test.run filter Pass an optional regex through to each suite's '\''go test -test.run'\'' so dispatches can target a single test (e.g. '\''^TestMultinodeIPSec$'\'') without running the full multinode suite. Threaded through single, multinode, and reboot suite invocations; empty input preserves current behaviour. * fix(daemon): flip NB_Global.ipsec=true on management node Setting Open_vSwitch.other_config:ipsec_encapsulation=true on each chassis is necessary but not sufficient for OVN native IPsec — without NB_Global.ipsec=true, ovn-controller never adds options:remote_name to the Geneve tunnel ports, so ovs-monitor-ipsec sees no IPsec-enabled tunnels and the kernel xfrm tables stay empty. After enabling ipsec_encapsulation locally, also write NB_Global.ipsec=true when the local ovnnb_db socket is present (management node only); workers skip silently — one writer is enough for a cluster-wide flag. Adds a scoped sudoers entry pinning the daemon to that exact ovn-nbctl invocation so the grant can't be reused for other NB mutations. * fix(ovn): grant charon AppArmor read on /etc/spinifex/ipsec The strongswan-charon profile only allows reads from /etc/ipsec.*, /etc/strongswan.*, and a small fixed allowlist. ovs-monitor-ipsec writes the per-tunnel strongSwan config with absolute paths to our peer cert + key under /etc/spinifex/ipsec/, so charon hits 'Permission denied' loading them (it runs as uid 0 but AppArmor still blocks) and no xfrm SAs come up. Drop a read grant for /etc/spinifex/ipsec/** into the profile's local override hook + reload. * fix(ovn): pin OVS system-id to hostname for IPsec cert identity match ovs-monitor-ipsec uses chassis-id as the IKEv2 `@<name>` peer identity in the strongSwan config it generates. Our per-node IPsec peer cert (admin.GenerateIPSecPeerCert) carries the node name as CN + dnsName SAN. With chassis-id left at the package-generated UUID, charon validates `@<UUID>` against a cert with dnsName=<hostname> — mismatch returns AUTHENTICATION_FAILED and no xfrm SAs are negotiated. Pin system-id to hostname (which equals the node name on Mulga clusters) by writing both the on-disk source of truth (/etc/openvswitch/system-id.conf, read by ovs-ctl on boot) and the running OVS DB value. Writing only one would drift on the next reboot — the original "do not override" comment was right about that hazard, just wrong about the resolution. Cross-link the invariant in admin/ipsec.go: changing the cert identity without updating setup-ovn.sh (or vice versa) breaks IKE silently. Refs: mulga-siv-136 * fix(ovn): take --node-name flag so chassis-id matches cluster identity CI hostnames are bead-themed (env20-casuarina) and don't match the cluster node name (node1/node2) baked into the IPsec peer cert dnsName SAN at `spx admin init/join --node NAME`. Pinning system-id to `hostname -s` produced env20-casuarina; charon then validated `@node1` against dnsName=env20-casuarina and IKE failed with AUTHENTICATION_FAILED. Add a --node-name=NAME flag to setup-ovn.sh, default to hostname for manual/dev invocations, and update the ansible install role to pass {{ spinifex_node_name }}. The matching bootstrap-install.sh change lives in mulga. Refs: mulga-siv-136 * refactor(daemon): relocate L0 helpers to network/host + widen ADR-0006 S2 Phase 4 Slice E of mulga-siv-125.6. Cleans up spinifex/daemon/network.go per the audit and widens the S2 invariant to forbid OVS/OVN shell-outs in spinifex/daemon/ (daemon is an L0 consumer; S2 applies). Moved to network/host/: - OVSNetworkPlumber + SetupTap/CleanupTap → host.OVSPlumber (host.NewOVSPlumber) - GetBridgeIPv4 → host.GetBridgeIPv4 - OVNHealthStatus + CheckOVNHealth → host.OVNHealth + host.HealthStatus - daemon/ipsec.go OVS/OVN ops → host.SetIPSecCertPaths, host.EnableIPSecEncapsulation, host.SetNBGlobalIPSec Moved to utils: - sudoCommand → utils.SudoCommand + utils.SetSudoCommandForTest test seam (cross-package reassign would trip the reassign linter; helper keeps the indirection inside utils). Moved to vm: - generateMgmtMAC → vm.GenerateMgmtMAC (pairs with vm.GenerateDevMAC) Deleted (dead): - SetupComputeNode (replaced by network/host/bridges.go:ensureBridges + scripts/setup-ovn.sh:572-579) - EnsureDataRoute - findInterfaceByIP - daemon/network.go + dead tests in daemon/network_test.go (duplicates of vm package tests retained; mgmt MAC test moved to vm/lifecycle_test.go) S2 invariant (network/invariants/host_containment_test.go) now walks both spinifex/network/ and spinifex/daemon/, skipping network/host/. * fix(ovn): broaden AppArmor read grant to /etc/spinifex (covers CA cert) The earlier override granted reads on /etc/spinifex/ipsec/** only, which covers the peer cert + key but NOT the cluster CA at /etc/spinifex/ca.pem. charon then can't load the CA, can't validate the peer cert chain, and surfaces the failure as `no trusted RSA public key found for '<peer>'` even though identity matching now succeeds — IKE_AUTH fails with AUTHENTICATION_FAILED and no xfrm SAs negotiate. Grant /etc/spinifex/** instead (covers ca.pem + ipsec/peer.{pem,key}) and always rewrite the override so re-runs replace the older narrower rule. Refs: mulga-siv-136 * fix(ipsec): symlink cluster CA into charon's default cacerts dir ovs-monitor-ipsec writes per-conn `cacert=/etc/spinifex/ca.pem` references in the strongSwan config, but strongSwan 6.0 only loads CAs declared in standalone `ca` sections or files placed in /etc/ipsec.d/cacerts at startup. Without this, charon authenticates the local side fine but rejects every inbound peer cert as `no trusted RSA public key found for '<peer>'`, AUTH_FAILED, and no xfrm SAs negotiate. Do the symlink in admin.GenerateIPSecPeerCert (right after writing the peer cert) — setup-ovn.sh runs before `spx admin init/join` creates /etc/spinifex/ca.pem, so the wiring can't live there. Use a stage-and- rename pattern so concurrent test invocations don't race on the symlink. /etc/ipsec.d/cacerts is on charon's default AppArmor allowlist so no profile change needed. Refs: mulga-siv-136 * chore(setup-ovn): drop apt-get install_packages block OVS/OVN/openvswitch-ipsec/strongswan-charon are baked into the CI gold image via mulga's image-builder/scripts/provision.sh (commits 116bb05, a9b3a9c). Keeping a runtime apt-get install path here pulled CI through apt-cacher-ng on every bootstrap and tripped on intermittent "Message has been manipulated" InRelease cache corruption (env21/env20 runs). Drop the install block. Gold-image rebuild is the canonical place to add new system packages. If a package is missing, downstream ovs/ovn commands fail loudly during setup-ovn.sh's later steps — clearer signal than silently re-installing through a flaky proxy. setup.sh already supports INSTALL_SPINIFEX_SKIP_APT=1 (passed by mulga's bootstrap-install.sh:237) so this removes the last runtime apt-get call on the CI provisioning path. * fix(ovn): gate system-id pin on explicit --node-name setup-ovn.sh used to fall back to `hostname -s` when --node-name wasn't passed, and then unconditionally wrote that value to /etc/openvswitch/system-id.conf + ovs-vsctl external_ids:system-id. On CI single-node, the gold image already registered a chassis with name=<UUID>; setup-ovn.sh then rewrote system-id to the bead-themed hostname (env10-banksia). vpcd's discoverChassis treats any chassis row whose name != local system-id but hostname == local hostname as a stale local chassis and skips it, so vpcd reported "no OVN chassis registered in SBDB" and refused to start, cascading into the e2e + e2e-go failures on run 26430839508. Drop the hostname fallback. If --node-name is given, pin both the on-disk file and the running OVS DB to it (callers that care about IPsec cert identity matching still get it). If no --node-name is given, leave the existing system-id alone — that's always safe whether the value came from a gold-image UUID, bootstrap-install.sh writing nodeN before this script runs, or a manual ansible value. * fix(network/reconcile): skip orphan prune on startup; drift owns GC The vpcd startup reconcile races against the daemon's EnsureDefaultVPC: the daemon writes SG KV rows + publishes vpc.create-sg on each node in parallel with vpcd starting up. The leader-gated reconcile can load intent before any SG row is visible on its local JetStream replica while subscribers on peer nodes are concurrently creating port groups from the same vpc.create-sg events. A startup orphan-port-group sweep then deletes managed sg_* port groups that intent hasn't caught up to. CI run 26434124790 trace: node2 reconcile/lock: elected holder=env20-bottlebrush node2 reconcile: starting intent_sgs=0 node2 reconcile/apply: removed orphan port group pg=sg_161c70ce23e704a2a node2 reconcile/apply: removed orphan port group pg=sg_a0d078650a3150391 node2 reconcile/apply: removed orphan port group pg=sg_54b564b374e9162bb Every subsequent RunInstances → vpc.create-port failed with "port group not found" for ~3 minutes until the 5-minute drift tick recreated the port groups. Tests give up after 60s; six multinode suites cascade-fail on InsufficientInstanceCapacity. Split the reconciler into Reconcile (full prune, drift loop) and ReconcileApplyOnly (no prune, startup). applySGs takes a pruneOrphans bool. vpcd startup calls ReconcileApplyOnly; drift loop unchanged. Legitimate orphans get garbage-collected on the next drift tick — at worst 5 minutes of lingering rows, no production impact. Add TestReconcile_ApplyOnlyKeepsOrphanPortGroup to lock the contract. * fix(ipsec): trigger charon rereadcacerts after CA symlink install strongSwan scans /etc/ipsec.d/cacerts only at charon startup. The unit openvswitch-ipsec.service is enabled at provision time and starts before admin init writes the cluster CA symlink, so charon ends up running with zero CAs loaded. Every IKE_AUTH then fails with `no trusted RSA public key found for '<peer>'` and no SAs negotiate — the failure mode exposed by TestMultinodeIPSec. Append a stroke rereadcacerts call at the end of installCAIntoCharonTrustStore so an already-running charon picks up the freshly-placed CA without a service restart. Failure is swallowed at debug — a missing/stopped charon at admin-init time is expected and a later startup will scan cacerts naturally. * refactor(network/host): move IPSec orchestrator from daemon (ADR-0006 S8) ADR-0006 S8: "IPSec is OVN-native only. No layer implements custom IKEv2, XFRM rules, or direct strongSwan management. IPSec SA lifecycle is delegated entirely to OVN native IPSec and is invisible above L0." The enableOVNIPSec orchestrator and its ensureOVSMonitorIPSecActive helper lived in daemon/ipsec.go but composed L0 primitives (SetIPSecCertPaths, EnableIPSecEncapsulation, SetNBGlobalIPSec) that already lived in network/host. The orchestrator now lives alongside its callees as host.EnableOVNIPSec, taking configPath and clusterConfig from the daemon at the call site. Helper and test-override package vars (systemctlActiveTimeout, ovnNBSocketPath) move with it; tests are translated to call EnableOVNIPSec directly rather than via the Daemon struct. Closes mulga-siv-142 * refactor(daemon): use topology.Port for OVN port names (ADR-0006 S4) ADR-0006 S4 forbids any layer above L2 from constructing OVN object names. Three daemon sites formatted "port-" + ENIId inline; route them through topology.Port instead. Closes mulga-siv-143 * refactor: use topology.Port for OVN port names in handlers + vm (ADR-0006 S4) Replace inline "port-"+eniID formatting across five sites (eip, instance x2, vpc/eni, vm/network_helpers) with topology.Port. OVSIfaceID stays as a thin wrapper to preserve the OVN binding contract comment and existing call sites. Closes mulga-siv-144 * test(network/invariants): widen scan roots to daemon/handlers/vm/testutil/utils (ADR-0006) Widen ADR-0006 invariant scan roots so drift in upstack consumers surfaces in CI. Backlog of violations in widened roots was drained by mulga-siv-142/-143/-144; this commit makes the wider scan permanent. - S2 (host_containment): add vm/ alongside network/ + daemon/ - S3 (nat_mode): add daemon/ alongside network/ + vpcd/ - S4 (names): add daemon/, handlers/, vm/ alongside network/ + vpcd/; add narrow AWS-ID exemption (vpc-cidr-assoc-) so AWS resource IDs are not misclassified as OVN object names - S6 (federation): add daemon/, handlers/ alongside network/; convert single-root walk to slice - S7 (nats_subjects): add daemon/, handlers/, testutil/, utils/ — stays t.Skip pending cluster-wide subject rename; gap report now lists 58 subjects CODEOWNERS: add /spinifex/network/invariants/ rule so widening or narrowing the scan needs explicit owner review. Closes mulga-siv-145 Refs: audit-adr-0006.md * chore(e2e): disable IPsec in pseudo multi-node test Pseudo multi-node runs three simulated nodes as unprivileged tf-user. The charon trust-store install in 904b45e writes /etc/ipsec.d/cacerts, which fails with EACCES under a non-root caller and aborts admin init. Disable --ipsec for this suite — loopback-only nodes don't exercise real Geneve encryption anyway. Real-multi-node + single-node e2e (both run admin init via sudo) continue to cover the IPsec path. * fix(ovn): atomic Ensure* primitives prevent duplicate logical entities OVN NB has no unique constraint on Name. The previous GetLogicalRouter→CreateLogicalRouter sequence in EnsureVPC was a TOCTOU race: when vpc.create on one node and the defensive vpc.create-subnet EnsureVPC call on another both observed absence, both inserted, yielding two routers per VPC. The orphan held DNAT references blocking EIP detach, and DNAT/flow installation landing on the orphan caused TestMultinodeGuestSSH SSH timeouts. Add EnsureLogicalRouter / EnsureLogicalSwitch / EnsurePortGroup to the libovsdb client. Each builds a single OVSDB transaction combining a wait-op (Until=equal, Rows=[]) gated on absence of a Name match with the insert. The server serialises the wait+insert atomically; concurrent callers see exactly one survivor and the rest fall through to a re-fetch that returns the canonical row. Switch all production callers (topology.EnsureVPC / EnsureSubnet / EnsureSGPortGroup, external.AttachIGW, reconcile.applyVPCs / applySubnets) to the Ensure* variants. Bare Create* remains available to the L1 wrapper. Add invariant test TestS4_OVNNamedCreatesUseEnsure (AST walk) to block future reintroduction of the bare Create* call sites outside network/ovn. Mock implements Ensure* under its existing mutex so the concurrency tests (50/30/30 goroutines for Router/Switch/PortGroup) prove a single UUID survives. Refs: mulga-siv-146 * fix(ovn): wait-op rows must be non-empty; flip to NotEqual predicate The Ensure* primitives sent a wait-op with `Rows: []ovsdb.Row{}` plus `Until: ==` (RFC 7047 §5.2.6 absence test). The libovsdb Operation struct tags Rows as `json:"rows,omitempty"`, so the empty slice was dropped on the wire and ovsdb-server rejected every transaction with "Required 'rows' member is missing", breaking reconciler bootstrap and every downstream create-port lookup ("logical switch X not found"). Flip the predicate to the equivalent `Until: !=` formulation with a single-row projection containing the Name we want absent: no row → projection [] != [{name}] → predicate holds row exists → projection [{name}] == [{name}] → predicate fails Same atomic guarantee, JSON now well-formed. Mock tests can't catch this — only a real ovsdb-server validates the envelope. * chore: remove generateMAC wrapper * chore: remove all bead / plan references * fix(network/policy): wait for chassis flows after AddNATGateway AddEIP fires m.barrier() to block until ovn-northd compiles NB→SB and every chassis installs the new dnat_and_snat flows; AddNATGateway returned right after the NB commit. So `aws ec2 create-route … --nat-gateway-id` would publish vpc.add-nat-gateway, the SNAT row would land in NB, and the API call would succeed before the gateway chassis had programmed the SNAT flow for the private subnet. Phase 8d's 10×5s=50s ping budget races that flow-install latency on slow CI hosts (last seen run 26455602615 single-node, identical flake on unmodified dev run 26434711010 — pre-existing, not introduced by Ensure*). Mirror AddEIP: call m.barrier() after AddNAT returns. Add unit test TestNATManager_AddNATGateway_FlowsBarrier_Fires covering the new path. * chore: remove more plan refs * chore: remove GenerateIPSecPeerCertIfEnabled * chore: remove dupliate subnetGatewayCIDR function * chore(network): trim superfluous comments across network/ tree Collapse multi-paragraph godocs to 1-2 sentences, drop WHAT-restatement comments, remove section dividers grouping <10 items, and cut prose narration in tests. ADR clause references, one-line WHY comments for non-obvious OVN/OVSDB/AWS/NATS semantics, and exported-symbol godoc are preserved. No behavioural changes. Packages: invariants, topology, host, external, ovn, reconcile, policy, subscribers. * fix(igw): restore attachment State="available" for AWS parity AWS InternetGatewayAttachment.State is "available" when an IGW is attached to a VPC, per the AWS SDK doc ("the state is available when attached to a VPC; otherwise, this value is not returned"). Commit b7505f8 wrote "attached"/"detached" to align with the reconciler filter, which inverted AWS parity. Fix the producer to write "available" again and update the consumer: - handlers/ec2/igw: AttachInternetGateway and DetachInternetGateway both persist State="available"; IGWRecord comment narrowed. - network/reconcile/intent.loadIGWs: filter accepts "available" (was "attached"); attachment is identified by VpcId != "". - tests updated: igw service_impl_test, routetable service_impl_test fixture, reconcile intent_test fixture. * Update spinifex.toml * refactor(subscribers): route EIP/NATGW through L5 wrappers (ADR-0006 §S1) Subscriber previously called policy.NATManager directly for EIP and NAT Gateway operations, bypassing the L5 wrappers (EIPManager, NATGWManager) that existed on paper. ADR-0006 §S1 requires every cross-layer call to pass through the immediate lower neighbor's typed interface; IGW already followed this pattern. Wire EIPManager and NATGWManager into the Subscriber struct, construct them in vpcd alongside IGWManager, and swap the four direct-L3 call sites (AddEIP, DeleteEIP, AddNATGateway, DeleteNATGateway) onto the L5 surface (AttachEIP, DetachEIP, AttachNATGateway, DetachNATGateway). AttachEIP now fires the flows-barrier, giving callers the same "reachable on return" guarantee IGW already had. The reconciler still calls policy.NATManager directly; that's a cross-cutter and a separate concern. * test(reboot): port 8MB WAL pad workaround from bash run-reboot-e2e.sh App VMs' user-data writes (unit file + index.html) sit in the on-host WAL until WriteWALToChunk crosses the 4MB threshold. Host reboot before that loses them — systemd then refuses to load the zero-byte unit and the responder never starts. Mirror the bash driver's dd+sync pad so the WAL flushes pre-reboot. Drop when the upstream WAL fix lands. * test(e2e): dump OVN/OVS/SG state on Phase 8d NATGW failure Phase 8d's existing failure path only prints lr-nat-list and lr-route-list. CI artifacts are not uploaded when run-e2e.sh exits nonzero, so debugging the NATGW regression requires inline diagnostics in the journal. Expand the dump to include nb/sb show, chassis bindings, port_group ACLs, br-int flows filtered to the NATGW IP/private subnet, host routes, and the private VM's own ip/route/neigh state. * fix(ovn,reconcile): scope NAT deletes to router; NATGW snat per associated subnet Two compounding bugs caused Phase 8d NATGW E2E flake (~60% CI failure): 1. ovn.DeleteNAT/DeleteNATByExternalIP searched all NAT rows globally and only mutated the named router afterwards. With AWS subnet CIDRs repeating across VPCs, (type, logicalIP/externalIP) is not globally unique — the delete picked a NAT row owned by a different router, the mutate was a no-op, and the row delete failed with "remaining reference(s)", leaving the original router's NAT orphaned. Both calls now resolve the LR first and filter NAT lookups to lr.NAT UUIDs. Mock client mirrors the scoping. 2. reconcile.loadNATGWs used the NATGW's home (public) subnet CIDR as SubnetCIDR, producing an snat for the wrong CIDR that collided with the correct event-published snat. loadNATGWs now walks route tables and emits one NATGWSpec per (NATGW, associated private subnet) pair, mirroring publishNatGatewayEventsForAssociation. Intent map keyed by natgwID|subnetCIDR so multi-subnet NATGWs produce distinct specs. Tests: - nat_test.go: cross-router same-logicalIP isolation for DeleteEIP and DeleteNATGateway - intent_test.go: NATGW snat uses associated private subnet CIDR, not the NATGW home subnet; NATGW with no associations produces no spec --------- Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>	2026-05-27 21:05:01 +10:00
joshsiv-mulga	8841241541	fix(setup-ovn): del-port cleanup without bridge name to avoid error on first run	2026-05-14 14:30:40 +10:00
tomnewton-mulga	9d665f9dde	bringing local networking changes onto dev branch	2026-05-01 12:13:34 +10:00
Josh Siviour	69404b38ce	fix(vpcd): public-IP boot race + DHCP DORA hardening (siv-33/34/36/37/38/39) (#177) ... * fix(installer): emit kernel cmdline to ttyS0 alongside tty0 (mulga-siv-33) GRUB_CMDLINE_LINUX in /etc/default/grub only listed console=tty0, so the installed kernel printed nothing on serial after the GRUB menu. Headless installs (CI, racks with serial-only IPMI, remote-dev SSH-to-QEMU) saw a black screen forever and had no path to debug boot failures. Append console=ttyS0,115200 — the last `console=` on the cmdline becomes the system console, so ttyS0 wins on serial-only hardware while tty0 still receives output on local displays. Plan: docs/development/bugs/iso-public-ip-boot-race.md * fix(ipam): widen vpc.dhcp.acquire retry budget to 60 s (mulga-siv-34) ObtainDHCPLease retried only 5 attempts × 1 s before returning a hard error. On a cold ISO boot, NATS comes up well before vpcd's reconcile leader-election + dhcp.Manager subscribe (typical 5–30 s, pathological ~60 s during full reconcile), so daemon's NewExternalIPAM races vpcd's dhcp.acquire subscription and fails permanently — d.externalIPAM stays nil for the daemon's lifetime, daemon_handlers_instance.go silently skips all auto-public-IP allocation, and instances launch with no public IP, no dnat_and_snat, and no NAT event ever published. Bump maxAttempts to 60 with 1 s spacing; expose both as package vars so tests can shrink the budget for fast timeout-path coverage. Pure widening of the existing retry loop — no behaviour change on the success path, only the worst-case wallclock when vpcd never comes up. The dominant cause of slow vpcd starts (mulga-siv-33: systemd-networkd not enabled in the ISO → veth pair vanishes on reboot → vpcd crashloops) is fixed alongside this. A 60 s budget fits comfortably above all observed healthy-startup times. Plan: docs/development/bugs/iso-public-ip-boot-race.md * fix(setup-ovn): declare Linux WAN bridge as NetDev so networkd-managed veth resolves Bridge= on reboot (mulga-siv-33) The veth-mode reboot persistence drop-ins added in mulga-998.b were incomplete: setup-ovn.sh wrote `15-spinifex-veth-wan.network` with `Bridge=br-wan`, but never declared br-wan itself to networkd. br-wan is created by `networking.service` (ifupdown via `/etc/network/interfaces`, generated by `install.go writeNetworkConfig`), not networkd. networkd's `Bridge=` directive requires the target to be a known NetDev — when it isn't, networkd silently logs `br-wan NetDev could not be found, ignoring assignment` and the veth ends up with no master. vpcd's bridge-mode sanity check then fails on every reboot. Fix: also write `14-spinifex-br-wan.netdev` declaring br-wan as `Kind=bridge`. networkd matches the existing kernel bridge by name+kind (the `Failed to create netdev: File exists` log is harmless), and `Bridge=br-wan` resolves on next boot. ifupdown still owns DHCP and the IP — networkd just needs the NetDev declaration to enslave the veth. The write is gated to avoid clobbering operator-managed networkd config: if any other `.netdev` under `/etc/systemd/network`, `/run/systemd/network`, or `/usr/lib/systemd/network` already declares the WAN bridge by name, the write is skipped with a log line pointing at the operator file. Cleanup hook also removes the new file when switching away from veth mode. Verified end-to-end via QEMU repro: clean install + first boot + reboot, veth-wan-br master persists across reboots, vpcd starts cleanly. * fix(vpcd/dhcp): enable AF_PACKET promisc on bridge for DORA (mulga-siv-33) Real-world DHCP servers commonly ignore the BOOTP BROADCAST flag and reply with a unicast OFFER addressed to the synthetic per-tenant chaddr (e.g. 02:f3:...). A Linux bridge floods unknown unicast to slave ports but does NOT pass it up to the bridge netdev unless IFF_PROMISC is set, so vpcd's AF_PACKET socket on br-wan misses the OFFER and DORA times out at 15s ("unable to receive an offer: context deadline exceeded"). Dev/CI didn't hit this: - CI uses spx_wan_mode=static — no DORA via vpcd. - QEMU dev DHCP servers (SLIRP/dnsmasq) honor the BROADCAST flag, so their replies are L2 broadcast which the bridge always passes up. Replace the three nclient4.New(iface, ...) call sites in Acquire/Renew/ Release with nclient4.NewWithConn(...) wrapping a custom AF_PACKET DGRAM conn that calls SetPromiscuous(true). Promisc is per-socket (per-DORA, ~15s) and refcounted by the kernel — auto-clears on close. * revert(vpcd/dhcp): drop AF_PACKET promisc wrapper, restore nclient4.New ecfbd4f swapped nclient4.New for openPromiscBridgeConn + NewWithConn to catch unicast OFFER replies that bypass the bridge netdev. On dev hosts where upstream honors the BOOTP BROADCAST flag, the wrapper regressed DORA — public-IP allocation failed where main works. Reverting until the promisc path is validated against both broadcast- and unicast-OFFER servers. * fix(setup-ovn): mark br-int/br-ext Unmanaged for systemd-networkd (mulga-siv-37) Fresh ISO install on Trixie+systemd-networkd left the OVS internal netdevs admin-down: networkd has no matching .network so it tags them as unmanaged but never brings them up. OVS dataplane works but ovn-controller flow programming and any tooling that probes link state misbehaves. Drop /etc/systemd/network/05-spinifex-ovs-internal.network with Unmanaged=yes so OVS itself owns the link state — `ovs-vsctl add-br` already runs the netdev up at create time. * fix(vpcd): allocate gateway LRP IP from pool in centralized NAT (mulga-siv-36) In veth/macvlan modes the gateway LRP carries all egress, so it must hold a WAN-subnet IP — otherwise the upstream router silently drops ARP for the WAN nexthop (RFC 826) and the default route never resolves. The hardcoded 169.254.0.1/30 was correct for distributed NAT but regressed centralized NAT (introduced under mulga-siv-26). Add gw_lrp_range_start/end to ExternalPool. In centralized mode at IGW attach the gateway LRP gets a fresh IP from that range, persisted as external_ids:spinifex:gateway_ip on the LRP for idempotent reconcile and retrofit. Direct mode keeps link-local. Per-VM EIP allocator skips the gw range; ValidatePoolConfig rejects overlap with the EIP range. Adds TestTopologyHandler_IGWAttach_CentralizedNATAllocatesGwLrpIP and extends RetrofitAllGatewayPortNetworks to migrate stale link-local LRPs to allocated pool IPs at startup. * fix(vpcd): drop link-local fallback, auto-derive gw_lrp_range (mulga-siv-36) In centralized NAT (veth/macvlan) the gateway LRP must hold a WAN-subnet IP — link-local cannot ARP-resolve the upstream nexthop, so the upstream router silently drops egress. Hardware deployment confirmed the previous fallback to 169.254.0.1/30 was always reached when gw_lrp_range was not explicitly configured. Auto-derive a 16-IP range at the top of the WAN subnet (broadcast - 16 .. broadcast - 1) when gw_lrp_range is unset; shift below RangeStart on overlap with the per-VM EIP range; clamp inside the subnet; guard the upstream gateway IP. IGW attach now errors out instead of falling back to link-local when allocation fails — surfaces misconfiguration loudly. Adds focused unit tests for gwLrpRange, ValidatePoolConfig and nextAvailableExternalIP gw-lrp skip; updates the multi-VPC and PoolWithGatewayIP regression tests. * feat(vpcd): DHCP-acquire gateway LRP IPs (mulga-siv-38) Per-VPC OVN gateway LRPs now obtain their WAN-subnet IP via vpc.dhcp.acquire when the pool is source="dhcp", instead of carving from a static gw_lrp_range. The upstream DHCP server becomes the single arbiter of LAN address space; a cluster grows by N nodes without operator-side range carving in spinifex.toml. - Consolidate ObtainDHCPLease/ReleaseDHCPLease into the dhcp package as RequestAcquire/RequestRelease so vpcd and the daemon-side ExternalIPAM share one helper. handlers/ec2/vpc keeps its existing ENI/EIP DHCP path. - TopologyHandler dispatches by pool source: DHCP pools call DHCPManager via NATS (client-id "gw-lrp-<vpcId>", idempotent on retry); static pools keep the siv-36 gw_lrp_range / auto-derive allocator. Direct mode and no-pool deployments still use link-local. - IGW detach issues vpc.dhcp.release so the upstream server reclaims the IP. Best-effort — server-side expiry is the safety net. - Tests: TestTopologyHandler_IGWAttach_DHCPGwLrpIP (3 VPCs share one DHCP pool, distinct IPs, idempotent attach, release on detach), IGWAttach_DHCPAcquireFailureAborts (no link-local fallback when DHCP errors), TestPrefixFromMask. Existing static / auto-derive tests preserved. * fix(vpcd): subscribe DHCPManager before reconcile to fix boot race (mulga-siv-38) DHCPManager.Subscribe ran AFTER ReconcileFromKV / RetrofitAllGatewayPortNetworks, so retrofit's vpc.dhcp.acquire requests for existing IGWs hit "no responders" and burned the 60-attempt × 1 s cold-boot retry budget per VPC before failing. On a dev box with one bootstrap VPC the symptom is a 60 s vpcd startup hang followed by "allocate gw LRP IP failed", and the bootstrap IGW topology missing from OVN until the next reconcile pass. Move dhcpManager.Subscribe ahead of the two reconcile passes so the subscription is live before any topology code can fan out an acquire. * fix(vpcd): also subscribe DHCPManager before bootstrap reconcile pass Previous fix moved DHCPManager.Subscribe ahead of ReconcileFromKV but missed Pass 1 (the bootstrap-config Reconcile that runs even earlier). On a dev box with a source="dhcp" pool and a bootstrap IGW, that earlier pass still hit "no responders" and burned the 60-retry cold-boot budget before the Manager came up. Move both reconcile passes after Subscribe. * fix(vpcd): RFC 2131 §4.1 DORA exponential backoff (mulga-siv-39) DHCPManager.Acquire previously did one DISCOVER and gave up after 15s because the outer ctx.WithTimeout matched the per-attempt timeout — nclient4's WithRetry(3) could never fire. Real DHCP servers (consumer routers, ISC dhcpd, dnsmasq) silently drop OFFERs under load; the spec mandates retransmission with exponential backoff, and a single lost DISCOVER was killing legitimate boots. Move retransmission into DHCPManager.acquireWithBackoff with a 4s/8s/16s/32s schedule (±1s jitter, ~60s worst case). NClient4Client becomes single-attempt; manager owns retry policy. Default acquireTimeout lifts from 15s to 90s (sum of schedule + slack). * fix(vpcd): retry NewDHCPManager on JetStream-not-ready (mulga-siv-40) NewDHCPManager creates the spinifex-dhcp-leases KV bucket via utils.GetOrCreateKVBucket → js.CreateKeyValue with no retry. After the mulga-siv-38 launchService reorder, this runs before any other JetStream user, so on multi-node cold-boot vpcd hits "JetStream system temporarily unavailable" and exits before cluster quorum forms. Pseudo-multinode CI calls start_node_services sequentially, so node2's vpcd failure aborts the script before node3 ever launches and quorum then never forms. Wrap NewDHCPManager in newDHCPManagerWithRetry (500ms→10s, 5 min cap, ctx-cancellable) mirroring daemon.initJetStream so vpcd waits as patiently as the daemon for the cluster to come up. * fix(daemon): idempotent DetachVolume QMP path (mulga-siv-41) DetachVolume's QMP sequence was not retry-safe. First call: device_del succeeds, blockdev-del fails with "Node nbd-vol-X is in use" because NBD client teardown / in-flight I/O still hold the block node. Handler bails ServerInternal "leaving volume state intact" but the guest device is actually gone. AWS-CLI retries: device_del returns DeviceNotFound, handler bails again. After 2 retries aws cli exits 254 — Single-Node Phase 8 Step 3 reproducibly fails this way. Phase 1: treat DeviceNotFound on device_del as already-detached, log and continue. Detach is now idempotent across retries. Phase 2: tryBlockdevDel polls blockdev-del up to 20×detachDelay on "in use" errors so the I/O drain can complete instead of failing the whole detach. Other QMP errors fall through immediately as before.	2026-05-01 09:48:52 +10:00
Ben Duncan	94600d7b84	OVN chassis issues, new reconsile function, reboot fixed (#165)	2026-04-27 15:48:47 +10:00
joshsiv-mulga	6b75470b0f	fix: admin-up veth-wan-ovs via networkd (mulga-998.b Fix 1 follow-up) ... Fix 1 persisted the veth pair across reboot via 15-spinifex-veth-wan.{netdev,network}, but only configured veth-wan-br. Its peer veth-wan-ovs had no .network unit, so networkd left it unmanaged and admin-DOWN after reboot. That cascades: peer goes LOWERLAYERDOWN, br-wan loses carrier, external DHCP/NAT path broken — despite vpcd correctly detecting mode=veth and the sanity check passing. OVS does not admin-up external (non-internal) ports — that's networkd's job. Pre-reboot the setup-ovn.sh 'ip link set veth-wan-ovs up' made it work, but that doesn't persist. Write a second unit 16-spinifex-veth-wan-ovs.network with ConfigureWithoutCarrier=yes so networkd flips admin state without taking ownership of bridge/address assignment (OVS retains that). Cleanup paths in the non-veth branch of setup-ovn.sh and reset-dev-env.sh updated to remove the new unit too. Verified live via 'networkctl reload': veth-wan-ovs and veth-wan-br both UP, br-wan LOWER_UP. Reboot drill per docs/development/bugs/dhcp-vpcd-issues.md 'Live drill 1' still required to confirm persistence.	2026-04-24 15:03:05 +10:00
joshsiv-mulga	abd93098fc	fix: persist veth-wan pair across reboot (mulga-998.b Fix 1) ... setup-ovn.sh's veth branch creates veth-wan-br ↔ veth-wan-ovs but nothing recreates the kernel interfaces on reboot. OVSDB remembers veth-wan-ovs as a port on br-ext but the peer is gone — the external path silently breaks and vpcd falls through to direct mode. - setup-ovn.sh: write /etc/systemd/network/15-spinifex-veth-wan.{netdev,network} in the veth branch so systemd-networkd recreates the pair on boot. Add a single pre-case guard that rips down stale veth persistence + the OVS port + the kernel veth whenever WAN_BRIDGE_MODE != veth, so mode transitions are idempotent (D17). - reset-dev-env.sh: delete the netdev/network units during cleanup so `make reset` gives a truly clean slate. - spinifex-vpcd.service: After=systemd-networkd.service openvswitch-switch.service so vpcd's sanity checks see a populated kernel + OVSDB (D5).	2026-04-24 12:37:01 +10:00
Josh Siviour	75fb15be2b	feat: spx admin init/join listen-vs-advertise split + br-mgmt OVS provisioning (#158) ... * fix: provision br-mgmt as OVS bridge + loopback guard for LB gateway Convert br-mgmt from an externally-provisioned Linux bridge to an OVS bridge owned by scripts/setup-ovn.sh, giving every deployment type (tofu-cluster, production single-node, production multi-node) a single provisioning path. setup-ovn.sh grows --mgmt-bridge / --mgmt-cidr / --mgmt-iface / --no-mgmt-bridge flags. Default creates br-mgmt with fail-mode=standalone (excluded from ovn-bridge-mappings so ovn-controller ignores it) and assigns the mgmt IP via a systemd-networkd drop-in so it survives reboot. SetupMgmtTapDevice / CleanupMgmtTapDevice switch from `ip link master` to `ovs-vsctl add-port` / `del-port` to match the new bridge type. Add a loopback guard to wireLBAgentConfig's fallback branch: on a production single-node with awsgw bound to 127.0.0.1 and no mgmt bridge, the fallback previously injected 127.0.0.1 into LB VM cloud-init — silently unreachable because the VM's loopback is its own. The guard leaves GatewayURL empty and the existing "LB agent gateway URL not configured" error fires instead. Regression covered in daemon_wire_lb_test.go. * fix: auto-detect primary IP for admin init --bind default Bare `spx admin init` used a 127.0.0.1 default that kept awsgw on loopback only, so system VMs on br-mgmt could never reach it. Auto-detect the default-route source IPv4 instead, which keeps TLS cert SANs specific (still not a wildcard). Dev-install scripts now pass --bind 127.0.0.1 explicitly to preserve loopback-only behavior on laptops. Closes mulga-siv-7. * test: refresh wire lb loopback-bind comment for siv-7 * fix: dev-install scripts should let admin init auto-detect --bind Pinning --bind to 127.0.0.1 on dev installs left awsgw loopback-only while setup-ovn.sh still provisions br-mgmt, so LB VMs hit 10.15.8.1:9999 with no listener. Drop the explicit --bind so auto-detect picks the primary IP and the siv-6 mgmtRouteVia path routes LB traffic to a real listener. * fix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template PR #144 (nats TLS) made every service listen on --bind's specific IP. For awsgw and predastore this broke localhost-reachability that the UI proxy (spinifexui/spinifexui.go:193-194) and AWS CLI spinifex profile rely on — latent since #144, surfaced by siv-7's auto-detect default on dev installs (502 from the UI). Hardcode awsgw and predastore host fields to 0.0.0.0 so both listen wildcard; loopback clients (UI proxy, spinifex AWS CLI profile, same-host daemon → predastore) keep working, LAN clients still reach them via the LAN-IP SAN on the cert. daemon.host and nats.host stay as {{.BindIP}} because peer-advertised values must not resolve to 0.0.0.0/loopback. predastore.host normalization at config.go:207 rewrites 0.0.0.0 → 127.0.0.1 for local-node consumers, so existing callers see a usable connect address. * revert: siv-7 --bind auto-detect + siv-6 loopback guard (prep mulga-siv-8) Clears the deck before layering the listen-vs-advertise split (mulga-siv-8). The plan restores --bind=0.0.0.0 as the single-node default and introduces an explicit --advertise flag for off-host dial targets; the reverted approach (auto-detect a specific IP at init, hardcode 0.0.0.0 in the template, guard against loopback in wireLBAgentConfig) becomes redundant once that lands. Reverts: - bb0ad3a fix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template - f20a47b fix: dev-install scripts should let admin init auto-detect --bind - 49b0b4e test: refresh wire lb loopback-bind comment for siv-7 - b88dfdf fix: auto-detect primary IP for admin init --bind default Also drops the loopback guard from siv-6 (74b6e2d) wireLBAgentConfig fallback and its regression test — subsumed by the GatewayURL precedence rework in mulga-siv-8.5 (br-mgmt → AdvertiseIP → error, no silent empty-string fallthrough). The br-mgmt OVS provisioning from 74b6e2d is retained. Plan: docs/development/bugs/spx-admin-init-bind.md * feat: admin.DialTarget + AdvertiseHost helpers (mulga-siv-8.2) Introduce the two small address-resolution helpers the listen-vs- advertise split needs, with full unit coverage. No callers yet — dial-site migration lands in mulga-siv-8.5. - DialTarget(listenAddr): rewrites wildcard listen addresses (0.0.0.0:N, [::]:N, :N) to their loopback equivalents so in-process clients dial a target the daemon's cert already has in its SANs. Specific IPs, loopback, hostnames, and malformed input pass through. - AdvertiseHost(listenAddr, advertiseIP): picks the host off-host clients should dial. Wildcard listen → advertiseIP; specific host → that host. Port is dropped so callers append the protocol-appropriate port. Plan: docs/development/bugs/spx-admin-init-bind.md §Proposed Changes/1 * feat: plumb AdvertiseIP through config/formation (mulga-siv-8.3) Pure data-model change. Add AdvertiseIP to: - admin.ConfigSettings (local node, populated by init). - config.Config (per-node; mapstructure "advertise"). - formation.NodeInfo (json "advertise_ip,omitempty") so the join handshake carries it between peers. buildRemoteNodes now prefers n.AdvertiseIP and falls back to n.BindIP so joined nodes from older clusters / pre-siv-8 joiners keep working. No behaviour change yet — templates, CLI flag, and dial-site migration land in siv-8.4 and siv-8.5. * feat: --advertise flag + resolveAdvertiseIP + templates (mulga-siv-8.4) Introduce the listen-vs-advertise split at the CLI boundary. admin init/join: - --bind default for init is 0.0.0.0 (was 127.0.0.1 under siv-7 revert). - New --advertise <IP> flag on both init and join; optional, auto-detected from DetectNetwork().WAN.IP when --bind is wildcard or loopback. - resolveAdvertiseIP(bindIP, advertiseFlag, detected) implements precedence: flag > specific bindIP > WAN auto-detect > 127.0.0.1 + warning. - Join resolves advertise before building the JoinRequest so peers record a reachable dial target (not 0.0.0.0). - isMultiNode: trigger on --nodes >= 2 regardless of bind literal; operator can now run "init --nodes 3" without typing a specific --bind. - Init success message prints bind / advertise / loopback three-way split. Templates: - spinifex.toml: [nodes.X] gains advertise = "{{.AdvertiseIP}}" rendered only when AdvertiseIP is set. - predastore-multinode.toml: iam.nats_url host wrapped in the same {{if eq .BindIP "0.0.0.0"}}localhost{{...}}{{end}} shim that predastore.toml already uses, so co-resident predastore dials its local NATS via loopback instead of 0.0.0.0. Tests cover the resolveAdvertiseIP matrix, the template advertise field (present + omitted), and the predastore nats_url loopback shim. Dial-site migration and GatewayURL rework land in siv-8.5. * feat: dial-site migration + GatewayURL AdvertiseIP fallback (mulga-siv-8.5) Switch every in-process dial site to admin.DialTarget so the wildcard listen address (0.0.0.0) is rewritten to loopback before Dial(): daemon → NATS and → predastore (boot, reconnect, status probe, S3 store, service manifest write), awsgw → NATS. The TLS cert already SANs 127.0.0.1, so loopback-dial a wildcard-listen service is always valid; the old 0.0.0.0 literal was tripping TLS SAN verification. Rework wireLBAgentConfig GatewayURL precedence (spinifex/daemon/daemon.go): 1. AWSGW on specific WAN IP + br-mgmt present → WAN IP (unchanged) 2. br-mgmt present, AWSGW on 0.0.0.0 → br-mgmt IP (unchanged) 3. d.config.AdvertiseIP (single-node default install) → AdvertiseIP (NEW) 4. DevNetworking shim → 10.0.2.2 (unchanged) 5. AWSGW on specific IP, no bridge → that IP (unchanged) 6. Else → log error, skip assignment Kills the pre-siv-8 silent empty-URL failure mode that made CreateLoadBalancer fail with 'missing system credentials: gatewayURL=' on every fresh single-node install. Supersedes siv-6's loopback guard. Extend SetupAWSCredentials with a wanIP parameter (reserved for a future --operator-endpoint flag) and thread AdvertiseIP through finalizeNodeSetup's three call sites (single-node init, multi-node init, join). Print 'Bind / Advertise / Loopback' on multi-node init and join completion to match the single-node summary. Plan: docs/development/bugs/spx-admin-init-bind.md §2, §3	2026-04-23 14:02:12 +10:00
Julian Sommer	6026d46ba1	fix: ovn logging	2026-04-08 13:00:06 +10:00
Julian Sommer	a56a2baf1f	feat: privilege separation (#114)	2026-04-07 16:22:19 +10:00
Julian Sommer	216842cc1b	fix: enable ovn on boot	2026-03-27 09:41:23 +10:00
Ben Duncan	4d56565c3f	feat: veth pair bridge — link Linux bridge to OVS without conversion ... setup-ovn.sh now detects whether the default route is on a Linux bridge or OVS bridge. When it's a Linux bridge (e.g. br-wan from cloud-init), instead of destructively converting it, the script creates an OVS bridge (br-ext) and links them with a veth pair. The Linux bridge keeps its IP, routes, and ports untouched — no WAN interruption. Also fixes all pre-existing lint issues (gofmt, ineffassign, errchkjson, gosec, modernize, whitespace) to bring golangci-lint to 0 issues. Tested: pool mode with static IP range on single-node (ben-banksia). VM gets public IP 192.168.1.201, SSH works, outbound internet works.	2026-03-25 10:01:37 +10:00
Ben Duncan	34b50b11fb	refactor: setup-ovn.sh — replace br-external with configurable WAN bridge ... Replace --external-bridge/--external-iface/--direct with: - --wan-bridge=NAME: OVS bridge for WAN traffic (auto-detected from default route if it's already a bridge) - --wan-iface=NAME: physical NIC to add to the WAN bridge - --macvlan: SSH-safe macvlan mode for single-NIC hosts Auto-detection logic: - If default route goes through a bridge (br-wan) → use it directly - If default route is a physical NIC → stop with guidance (never auto-move a NIC into OVS — may drop SSH) Bridge-mappings updated: external:$WAN_BRIDGE (was external:br-external). OVN logical network name stays "external" (no topology.go change needed). Encap IP now prefers br-vpc IP when available (dedicated VPC data plane). Part of mulga-739. Blocks VPC E2E work.	2026-03-24 15:46:49 +10:00
Ben Duncan	88baf5e202	feat: direct bridge mode — prefer over macvlan when WAN NIC != SSH NIC ... Auto-detect bridge topology: if a dedicated WAN NIC exists (not used for SSH), add it directly to br-external instead of creating a macvlan. Direct bridge enables distributed NAT (DNAT on VM's own chassis) and eliminates macvlan workarounds (MAC alignment, gratuitous ARPs, centralized NAT bottleneck). - topology.go: bridge-mode-aware NAT and localnet port options - vpcd.go: auto-detect bridge mode, skip MAC alignment for direct bridge - setup-ovn.sh: --direct flag for physical NIC on br-external - reset-dev-env.sh: SSH NIC detection via SSH_CONNECTION	2026-03-24 01:57:54 +00:00
Ben Duncan	daae6ca6b8	refactor: remove single-nic bridge mode, standardize on macvlan ... The single-nic IP migration path was fragile and caused connectivity issues during setup. macvlan is now the only bridge strategy — always creates spx-ext-{nic} on the WAN NIC. Removed --single-nic flag, SingleNIC config/detection fields, and updated all docs.	2026-03-23 01:12:22 +00:00
Ben Duncan	665093343c	fix: preserve DNS resolution after IP migration to br-external ... When setup-ovn.sh migrates the WAN IP from a physical NIC to br-external, systemd-resolved loses its DNS association (DNS servers are per-interface). This breaks hostname resolution for everything on the host — git, apt, curl, SSH all fail. Now captures DNS config from the original interface before migration and re-associates it with br-external via resolvectl. Falls back to 8.8.8.8 if no DNS was configured on the original interface.	2026-03-20 06:57:18 +00:00
Ben Duncan	2af3bf5287	feat: auto-detect network topology + DHCP gateway in admin init ... spx admin init now works with zero external networking flags: - Auto-detects WAN interface, gateway, prefix from ip route - Auto-detects single-NIC vs multi-NIC mode - Defaults to nat mode with DHCP for gateway IP - Prints detected topology with interface roles setup-ovn.sh gains --dhcp flag: obtains a DHCP lease on the macvlan/bridge interface for the OVN gateway SNAT address, then writes gateway_ip into spinifex.toml. Tested on VM: macvlan gets separate IP from router DHCP, host keeps its original IP. Three tiers of external networking: - Zero flags: auto nat+DHCP (VMs get outbound internet) - --external-pool=start-end: pool mode (VMs get public IPs) - --no-external: overlay only (no internet for VMs)	2026-03-20 05:45:30 +00:00
Ben Duncan	1ee7012ba9	feat: Phase 7 — single-NIC macvlan bridge strategy ... Add --single-nic flag to setup-ovn.sh. When used with --external-bridge, creates a macvlan sub-interface (spx-ext-{iface}) in bridge mode off the host's NIC instead of adding the NIC directly to br-external. The host keeps its IP — no migration, SSH-safe. Strategy selection: - Dedicated NIC: --external-iface=eth1 (add directly) - Single NIC: --external-iface=eth0 --single-nic (macvlan) - IP migration: --external-iface=eth0 (legacy, no --single-nic) Add single_nic bool field to VPCDConfig for per-node config. Different nodes in a cluster can use different strategies. Unit test verifies single_nic parsing alongside external_interface.	2026-03-20 05:15:35 +00:00
Ben Duncan	c1cdfc07c3	feat: Phase 1 completion — external IPAM, br-external setup, config template ... Add external IPAM module (CAS-based IP allocation from admin-defined pools in NATS KV). Supports multi-pool with region/AZ scoping, pool fallback logic, gateway IP reservation, and concurrent allocation via CAS retries. 12 unit tests covering allocation, release, exhaustion, CAS conflict, multi-pool, fallback, validation, and idempotent init. Add --external-bridge flag to setup-ovn.sh: creates OVS bridge br-external, adds WAN NIC, migrates host IP to bridge, and sets ovn-bridge-mappings=external:br-external for OVN localnet ports. Add commented [network] section and external_interface to spinifex.toml config template for discoverability.	2026-03-20 03:59:47 +00:00
Ben Duncan	5d93786995	refactor: rename hive to spinifex in all shell scripts ... - Update scripts/: start-dev, stop-dev, dev-setup, setup, setup-ovn, etc. - Update tests/e2e/: all E2E test scripts - Update scripts/iac/: rename hive-test.sh → spinifex-test.sh, update TF files - Update config/: rename hive.service and install script - Binary refs use spx, env vars use SPINIFEX_, paths use spinifex	2026-03-18 01:04:12 +00:00
Julian Sommer	4466de04d3	fix: ovn logrotate	2026-03-12 08:24:52 +10:00
Julian Sommer	d90bf53eaf	fix: log rotate ovn	2026-03-06 10:10:34 +10:00
Julian Sommer	48c0565d47	fix: ovn shut down on stop-dev.sh	2026-03-03 15:12:44 +10:00
Julian Sommer	8188707c91	feat: VPC networking v1 — OVN-based virtual networking for EC2 instances	2026-02-24 14:13:57 +10:00