62 Commits

Author	SHA1	Message	Date
Ben Duncan	c6841ce1f0	feat(oci): separate using an instance principal from creating one (#1157) ... * feat(oci): separate using an instance principal from creating one The allocator needs OCI credentials at runtime, and a node with none forms, passes every health check, then refuses every launch wanting a public address. Instance principal is the credential route that puts no key material on a node, but it needs a dynamic group and a policy, and those are tenancy-root resources the compartment-scoped user the rest of the configuration runs as cannot create. enable_instance_principal conflated the two, so the better credential route was only reachable by a tenancy admin. instance_principal now takes off, adopt or create: adopt selects the credential and creates nothing, because the matching rule covers the whole compartment and so names no instance to reference. setup-identity.sh owns the create case, with its own state. validate-topology.sh destroys its state at the end of every run, so holding tenancy resources there would let a teardown delete the tenancy's policy. Also package install-node.sh and smoke-test.sh, so a cluster can be formed from a curl | bash install with no checkout. smoke-test.sh is not optional: formation scp's it from its own directory for the post-formation verification. * feat(oci): take the credential handoff from an operator hook, and widen the workflow name An API-key deployment needs a credential on each node, and nothing in this repository can supply one: a key in user-data is readable from instance metadata for the life of the instance, and this host runs other people's guests. A manual install does not survive either, because a topology destroys itself on success. So the harness calls an operator-owned executable after formation and before the pool, the only window where a node has /etc/spinifex and has not started the allocator. Instance principal needs no handoff, so the hook is skipped there. The workflow is renamed for the vendors it will cover rather than the first one. Nothing is orphaned in the Actions UI: it has never been on the default branch and has never run, so there is no history attached to the old filename. * docs(oci): the API-key path needs the config file, not just the PEM oci_config_file names an ordinary OCI SDK config and nothing creates it, so a node holding only the key forms and then fails every allocation with no stated cause. * fix(ci): the runner context is not available in a job-level env block A workflow referencing it there is rejected before any job starts, which is why the rename produced a failed run attributed to the push. The reference had never been validated: a dispatch-only workflow that is never dispatched is never parsed. * fix(deps): bump containerd to v2.4.1 for GO-2026-6597 The image-pull DoS via crafted OCI index graph amplification is reachable from the ECS agent's containerd puller, so the vulnerability scan fails preflight on v2.4.0. A patch release with no API change. * fix(e2e): probe guest public addresses from outside the cloud's own network A VCN does not reflect its own public addresses back into itself, so a node asking for one gets nothing however well the datapath works. The workbook driver runs on the first node, so every assertion that curls or SSHes a guest's public address failed on OCI against a cluster that was serving those addresses correctly to the internet. The driver now takes E2E_PUBLIC_PROXY, a SOCKS5 proxy used for public addresses only, and validate-topology.sh supplies one with a remote dynamic forward on the connection that launches the driver. Egress is then the host running the validation, which is outside the VCN and is where a customer sits. Unset leaves every other platform on a direct probe, which is already correct there. Also adds --version TAG for a repeatable install, separates the workflow's auth choice from its external-pool choice so an API key can run with the allocator live, and captures allocator diagnostics before the teardown destroys them. * ci: read the Go version from go.mod in the release workflows Both pinned 1.27.0 by hand while every other workflow reads the file, so the toolchain they build with drifts from the one the module declares the day go.mod moves. release.yml was already inconsistent with itself: its second setup-go step reads spinifex/go.mod. * feat: publish the OCI topology results as tables on the run page A green tick on one job with 4,000 lines of log behind it is confidence for whoever wrote the harness and nothing to anybody else. Someone assessing whether Spinifex works on their cloud should be able to see that it formed and served guests on a single VM, on three VMs and on bare metal without opening a log. record now takes <gate> <status> [detail] and writes results.tsv beside the human block, so a gate's status is a field the script sets rather than a regex over prose that reads fine and matches wrong. record_workbooks turns the driver's own go-test-shaped lines into workbooks.tsv, and runs on the failing branch too: which four passed is most of what the fifth failing means. e2e-oci.sh --summary-only renders both tables to stdout and to GITHUB_STEP_SUMMARY, and the workflow calls it under always() before the sweep, because the sweep destroys the state directories the tables are read from. A skipped gate gets its own mark, and a topology that never ran says so: the one way a results table can do real harm is a reader taking an absent row for a passing one. * docs(oci): cut the deploy-by-hand appendix, and validate the config for real The appendix was 444 of 1014 lines restating an install the Terraform already does, which made the document read as though deploying by hand were a normal route. It is not: the configuration builds the VCN, the volumes and both VNICs per node and renders the cloud-init, in the same order every time. What is genuinely OCI-specific is four things, each of which already has a section, so the replacement names those and sends everyone else to the single-node and multi-node install guides. Verify end to end told the reader to ping a guest's public address from the host, which cannot work: a VCN does not route its own public addresses back into itself. That is the finding this branch is built on and the document was asserting the opposite. It now says where to probe from and why, and carries the allocator's `resolved the external VNIC` check, which existed only in the cut appendix and is the one line proving the credentials and compartment are right. `spx config validate` is not a command -- spx has admin, get, service, top and version -- so that line could only ever fail and its output and status were both discarded. It is replaced by parsing the TOML, which is what was wanted: the remap and the pool append are textual edits, and a broken result would otherwise surface as every service failing to start with nothing saying why. The remap now checks its own substitution count for the same reason, since re.sub with no match rewrites the file unchanged and exits 0. Publication of the run-page tables moves to one owner. GITHUB_STEP_SUMMARY is per step, so appending from the run as well as the publish step would have put two copies of every table on the run page. * fix(oci): check the credential hook before building anything The hook is a path from a CI variable, and whether it exists on the runner is knowable before Terraform runs. Checked where it was used, a wrong path cost the apply, the install and the formation first -- forty minutes and a bare-metal bill to be told a file is missing. Nothing about the check depends on the apply, so it moves to the top, where --dry-run reaches it too. * fix(oci): the summary must say what was built and what was not tested The first CI run headed its tables "Published release `latest channel`" over a build of a feature branch, which is the one error in a results table that would mislead a reader who trusted it. The header now states a tree build as a tree build, and names how the allocator authenticated. A run with no external pool says so in the header rather than leaving it to a SKIPPED row further down. Public-address allocation is the claim a green table is most likely to be read as making, and it is exactly the claim such a run cannot support. Both flags are passed to the publish step too, so the summary rendered after a failure describes the same run as the one that failed. Raise the dev prerelease prune from five to twenty. A pinned dev tag is how an external evaluator installs a specific build, and five is about a week of them -- short enough that the tag someone was handed stops resolving while they are still using it. * docs(oci): the workbook table is a snapshot, and the nightly is the current answer It is dated against a build five releases of work ago, and the nightly now runs the same workbooks on all three topologies and publishes its own table. Say which one to read rather than letting a stale table be taken as current. * ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file A workflow's displayed name is captured when GitHub first registers it and is not refreshed afterwards. Both OCI entries were registered from a run that could not be parsed -- a job-level env block referencing runner.temp -- so both list as their path, which reads as a missing name: key and is not one. The 2026-10-02 run was green on all three topologies and the entry still read as a path, with updated_at unchanged from the broken first run, so fixing the file is necessary and not sufficient. A new path is a new registration, and its first run reads a valid file. Dropping -nightly is also more honest: there is no schedule: here, because GitHub registers one only from the default branch, so a cron could never run dev and a feature branch as separate nightlies. * Revert "ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file" The rename makes the workflow undispatchable. `gh workflow run` resolves a filename against the default branch, and the raw dispatch endpoint returns 404 for a path GitHub has not registered -- checked twice, several minutes apart, so it is not an indexing delay. The old name works only because it is already registered, and an unrunnable workflow is a worse problem than a mislabelled one. The sidebar label needs a fresh registration, which a rename cannot provide from a feature branch. It corrects itself when the file reaches main, where GitHub re-reads it; the alternative is deleting the dead registrations' runs, which is an operator's call. * fix(ci): never publish the credential hook's output as an artifact The hook is the operator's own executable, and the one thing known about it is that it handles a private key. Its stdout and stderr are captured to credential-hook.log, which the artifact's *.log glob was collecting into a downloadable archive with a fourteen-day retention. Excluded rather than trusted to have been quiet. A hook that prints its key once while someone is debugging it should not thereby publish it. * fix(oci): the hairpin probe measured nothing on one node, and claimed too much With a single host, HOSTS[0] and HOSTS[-1] are the same address, so the node connected to itself -- which always succeeds. It reported "present" on vm-single and bm for that reason and not as a measurement. It now needs two nodes and says so otherwise. The claim it carried was also wrong. "A VCN does not reflect its own public addresses back into itself" was read off a workstation run whose node_client_cidr_allow_list was narrowed to one office address, which makes a node's own public address a non-permitted source: the refusal that follows is identical to an absent hairpin. With the committed 0.0.0.0/0 default the probe reports the hairpin present between two nodes, so the original reading was confounded and is retracted. None of this changes the remedy. A customer reaches a guest from outside the VCN, which is where the proxy's egress is, and probing from there is correct whichever of the two causes is in play -- 15 of 15 workbooks pass through it on all three topologies. What changes is what the documentation may assert: the guide now names both causes as indistinguishable rather than stating one as fact. * docs(oci): a reader could not run the first command, because nothing told them to clone The guide's first code block was `cd scripts/terraform/oci-spx`, into a directory no step had obtained. It also never mentioned the SSH keypair the driver refuses to start without, nor that the configuration is not on main or in v1.21.0 yet, so all three had to be inferred from the deploy failing. Restructured so the deploy comes first and reads as four steps and one command: get the configuration, choose one node or three, write the inputs, run it. The credential decision stays before step 4, because it changes an argument there and a node without one forms, looks healthy, and then refuses every launch that wants a public address. Everything else -- why OCI, sizing, the architecture, the IAM policy, quotas, the oci CLI, what Terraform builds, every variable -- moves under Reference, unchanged in substance. Adds what a successful run prints, naming `allocator ready` as the line to look for: every line above it appears on a node that cannot allocate, and that is the one failure the run does not announce. Also drops an empty verdict.txt committed from a local run. The directory the guide now tells customers to work in should not ship one run's artifacts.	2026-10-04 09:04:55 -05:00
Ben Duncan	1743b60246	feat: publish a dev channel, so cloud contacts can install dev without a release (#1154) ... install.mulgadc.com has served a dev channel since it was written -- setup.sh understands INSTALL_SPINIFEX_CHANNEL=dev and the Worker resolves it to the newest GitHub prerelease -- but nothing ever produced one, so /download/dev/amd64 returned 404. This is the producer. A separate workflow rather than conditionals in release.yml. A dev build is a different artifact, not a release with flags off: amd64 only, no ISO, no system images, no bare-metal gate, and nothing on a self-hosted runner. The sibling repositories are taken at dev rather than their default branch, because dev of spinifex against main of the others is a pairing nobody runs. The tag is the describe string plus -dev, which release.yml's own convention already treats as a prerelease, and which makes the commit recoverable from the version alone. The flag is set at creation and then the published state is asserted: a prerelease must never become the latest release, because latest is what an unqualified curl | bash installs. setup.sh gains --channel and --version so the instruction is a flag rather than an environment variable, and the OCI harness gains OCI_CHANNEL so a nightly can test the path the contacts are actually given.	2026-10-01 13:51:57 -05:00
Ben Duncan	7163f7d62d	feat(oci): run Spinifex on Oracle Cloud, and fix the public-address and NAT gateway lifecycle (#1108) ... * feat: ship uninstall-spx.sh and fix the OCI bring-up path Adds scripts/uninstall-spx.sh, the inverse of setup.sh: it removes the software where node-reset.sh clears state and leaves the node installed. An operator who installed with `curl ... | bash` had no teardown path at all. State teardown is delegated to node-reset.sh rather than reimplemented, and setup.sh now ships both to /usr/local/share/spinifex so they are reachable without a checkout. It refuses on a multi-node cluster and --yes does not override that: a flag meaning "don't ask me" must not also mean "ignore a safety property". /var/lib/spinifex and every apt package are kept unless asked for, and both opt-ins confirm separately even under --yes. Proved by a full uninstall/reinstall round trip on dev-oci, which found two bugs neither prod nor dev-prod can reach, because both are ISO-installed: node-reset.sh exited 2, silently, at its last step and after the wipe. setup.sh creates the spinifex *group* and the spinifex-* service users but never a *user* named spinifex, so `getent passwd spinifex` exits 2 and pipefail carried it into set -e. The uninstaller broke openvswitch-switch on its first run by removing the socket-perms helper while leaving the drop-in referencing it, failing the unit at step EXEC with status 203. Fixed on both sides: the uninstaller removes drop-ins it wrote into other units' namespaces and restarts anything it failed, and setup-ovn.sh writes ExecStartPost=- so a missing helper can never stop another package's daemon. setup-ovn.sh also learns the shape OCI needs: when --wan-bridge names a Linux bridge it links it to OVS with a veth pair instead of enslaving the NIC, so the VNIC keeps its MAC identity and netplan keeps owning the addresses. It now fails on OVS datapath errors instead of reporting success, and the --management demotion guard keys on /etc/default/ovn-central rather than on database presence, which the ovn-central package creates on every node. Tests: scripts/uninstall-spx_test.sh, wired into make test-build-scripts along with shellcheck over the four scripts that ship to every node. * feat: allocate external IPs from the Oracle Cloud API An OCI VNIC drops any packet whose source is not a registered private IP object on it, so an external pool cannot be range math there — the address has to be created through the provider before it exists at all. Add source = "oci" and the allocator behind it. One allocation is two OCI objects: a secondary private IP that rides the wire, and a RESERVED public IP attached to it that the internet reaches. Allocate returns the public address because that is what AWS semantics are about, and records the pair so the host plumbing and release path can find the private half. The OCI objects are created before the binding is written, and dropped after it on release — both orders chosen so a crash leaves a discoverable leak rather than a record that lies. Reconcile squares the two on startup and is required, not optional: a leaked reserved public IP bills and holds one of the 50 per-region slots with nothing referencing it. It never touches an object without our display-name prefix, because the reference host carries operator-created secondaries on the same VNIC. Resolve the VNIC from an interface name by MAC through IMDS rather than by Oracle's VNIC ordering, which is not a documented contract. Generalise ExternalIPAM.EnableDHCP into InstallAllocator so the OCI wiring is not a second single-purpose installer. * fix: refuse source = "oci" outside routed mode, and cover the SDK path An OCI VNIC accepts exactly one MAC, its own, and delivers every inbound frame to it regardless of which of its addresses the packet is for. Measured on the reference host: an internet source reaches 10.200.0.144 as router-MAC to VNIC-MAC, the public address never appears on the wire at all, and every off-subnet destination resolves to the same next hop. Pool mode puts a per-VPC gateway LRP MAC and a per-rule external NAT MAC on the uplink and ARPs for the address, which policy.go:50 justifies with "the Linux bridge learns the per-rule external MAC on the veth port like any other MAC". An OCI VNIC does not learn MACs; it accepts one. So a pool-mode guest's egress is dropped and its ingress never reaches OVN. Routed mode keeps the address on the host, which already wears the VNIC's MAC. Pinning every LRP to that MAC does not rescue pool mode: they share one external logical switch and OVN resolves inbound frames by an L2 lookup on it, so duplicate MACs work for one VPC and silently misroute past that. The refusal names the MAC reason, because a refusal that only says "use nat mode" reads as arbitrary and gets worked around. Also test the SDK request shapes against a local endpoint. Two of them are wrong in ways reading does not catch: pagination stopping early reports a partial VNIC to reconcile, which then drops live bindings, and a nil privateIpId on detach is omitted from the body entirely, so OCI reads the update as a no-op, returns 200, and leaves the address attached and billing. * fix: install host firewall rules where a distro's catch-all reject cannot precede them Oracle's Ubuntu cloud image ships a catch-all REJECT in both INPUT and FORWARD. Two Spinifex datapaths were silently dead behind it on any such host. EnsureNATEgressRules probed with -C and appended with -A, so its ACCEPT rules landed after the REJECT and never matched: 24113 packets on the reject, 0 on both accepts, and guests had no egress. The comment named the case it did not handle, anticipating a FORWARD policy of DROP — which an append does survive — rather than an explicit REJECT rule, which it cannot. Insert at the head, and drain existing copies first, because a rule an earlier build appended still satisfies -C and would otherwise stay stranded across an upgrade. Nothing opened INPUT for the IMDS endpoints at all. The responder binds .254 and .253 per instance on the ime- devices, so guest requests are locally delivered and meet INPUT. Requests arrived and no reply ever left; cloud-init spent 490s timing out and reported the endpoint as disabled. With one wildcarded accept the same image boots in 18.6s and finds its datasource. The natuplink e2e check had the same blind spot the code did, asserting -C and therefore presence but never position. It now asserts the accepts precede the first reject. * feat: OCI address pairs, and an IMDS remap so the host keeps its own metadata An OCI external address is two objects, not one: a RESERVED public IP that OCI 1:1-NATs upstream and never puts on the wire, and a secondary private IP on the VNIC that the wire actually carries. Every AWS-facing record holds the public half; every datapath object must hold the private one. The mapping goes in policy.natManager and nowhere else. AddEIP, DeleteEIP, PruneOrphanEIPs and the HostEIPBinder all converge there, and reconcile rebuilds the rule set from AWS-facing intent on every pass — so a correctly mapped rule installed further upstream is swept on the next reconcile and replaced with one matching an address that never arrives. The seam is a DatapathResolver defaulting to identity, so every non-OCI deployment is unchanged. A resolve failure is fatal: falling back to the public address would install a rule that reads correctly in ovn-nbctl and blackholes. ocinet.Lookup is the OCI side — read-only, KV-backed, no client and no credentials, because vpcd is a different process from the daemon that owns the allocator and the bindings record is all either would consult. Two live bugs the in-memory test store was hiding, both found by making the fake report ErrNotFound the way the real store does: - KVStore.Mutate could not create, so every pool failed its own first allocation after OCI had already created and billed the pair. - Reconcile treated a missing record as a failure, aborting the one pass that exists to find an object created before its binding was written. Guest IMDS must stay on 169.254.169.254, which is also where a cloud host's own metadata service lives — the first guest launch made it a local address and took the host's metadata and DNS with it. The datapath is OpenFlow steered, so the guest-facing addresses stay put and only the host-side endpoint address moves, with a PREROUTING DNAT on ime-+ translating and conntrack un-NATing the reply. Off by default; identity when unset. The IMDS firewall rules move to vpcd. They are wildcarded over every endpoint rather than per-launch work, and spinifex-daemon holds no iptables grant and no CAP_NET_ADMIN by design, so attempting them aborted every launch. * fix: name the real constructor in the IMDS bind-address error, and mark the NAT datapath test in-package The error string pointed at DefaultHostBindAddrs, which does not exist — a caller following it would have looked for a symbol that was never written. nat_datapath_test.go reuses nat_test.go's in-package fixtures, so it carries the //test:in-package marker the package check asks for. * docs: index what each e2e suite proves Written because a datapath comment was read as an end-to-end behavioural claim and the suite that already answered it was found only after the wrong answer had been given. TestVPCEgressPaths/NATBaseline is the authority on private subnet egress, and nothing pointed at it. * fix: source-route an EIP's reply out the VNIC that holds it An EIP's reply carries the EIP as its source. On a host that pins its uplink with a policy rule, the reply took the main table instead and left by the wrong interface — which a cloud VNIC drops, because the address is registered on a different one. Ingress reached the guest and nothing came back. Every bind now mirrors the uplink's rule onto the EIP. The selector is the rule rather than the interface: both OCI VNICs sit in one subnet, so "the interface owning the EIP's subnet" names two of them, and two candidate rules is a refusal rather than a guess. A host with no source routing gets no rule and no change. * docs: name the setup-ovn recipe routed mode actually needs on OCI The OCI guide told operators to run `--wan-bridge=br-wan`, which is mutually exclusive with `--nat-uplink` and deletes the spx-nat transit veth routed mode runs on. The node then refuses to bring the uplink up, with an error naming the flag the guide should have used. Several messages also claimed routed mode is outbound-only with no public IPs. That stopped being true when the routed EIP path landed, and it reads as "what you are about to do cannot work" during the one init the OCI path documents. * feat: let routed NAT form a multi-node cluster, and carry every pool through formation Lifts the --nodes >= 2 refusal on --external-mode=nat. The refusal carried no technical justification -- it was a v1 scope decision -- but routed NAT is the only datapath a cloud VNIC can carry, so "routed NAT is single-node" had become "Spinifex on a cloud host is single-node". It is not fixed by lifting it. Measured on three OCI nodes: the cluster forms, all chassis register and guest-to-guest traffic runs at 7.1-7.7 Gbit/s, but every node claims every EIP, nodes disagree on the VPC's gateway LRP, and only the gateway chassis reaches its own guests. init now says so on stderr rather than refusing, because the refusal made the defects unreachable and therefore unfixable. Distributing the EIP datapath is the actual fix. Lifting it exposed a second defect that stops formation outright. NetworkConfig held nine flattened Pool* scalars and the init node set PoolName = "wan" unconditionally; routed NAT renders two pools and the one that matters -- nat-transit, carrying the gateway-LRP range -- was dropped. Each joiner got a single pool named "wan" holding the transit gateway and the detected WAN prefix, and the leader re-rendered its own config from the same struct, so all three nodes were wrong. Pools is now []admin.PoolData in render order, and PoolData gains json tags because it is on the wire now rather than a template input. * feat(install-node): add --external-mode so a cluster can be formed in routed NAT install-node.sh could not express routed NAT at all. It required --external-pool/--external-gateway/--external-prefix-len and passed --external-mode=pool to every spx admin init unconditionally, so the only supported way to form a routed-NAT cluster was by hand. --external-mode nat refuses the three pool flags rather than ignoring them, and threads --nat-uplink into setup-ovn.sh -- without it the WAN interface check rejects a physical NIC during formation, which is correct in pool mode and wrong in routed mode, where nothing is bridged. * fix(ocinet): stop each node deleting its peers' OCI addresses on startup Reconcile compares the binding record against the private IPs on one VNIC. The record is one key per pool and therefore cluster-wide, so on more than one node every peer's binding looks exactly like a binding whose objects OCI has lost. Each node dropped them, and direction 1 on the owning node then collected the now-unclaimed objects as leaks -- deleting the reserved public IP and private IP of a running instance. Observed on the three-node OCI cluster: node2's startup pass logged "collected a leaked address" for an address it had allocated ten minutes earlier and "dropped a stale binding" for one living on node3's VNIC. The damage did not stop at the OCI objects. With the binding gone, Lookup.DatapathIP returns the public address unchanged, so the OVN dnat_and_snat row carries the public address instead of the private half. Guest egress then leaves the VNIC sourced from a reserved public IP, which OCI drops because it enforces the registered source per VNIC -- and EnsureEIPSourceRoute no-ops too, since a public address shares no subnet with any host address. One destroyed binding takes out both directions. Binding.VNICID already exists for exactly this ("how a node that comes back from the dead tells mine from moved on without me") and Reconcile never consulted it. A binding is now only dropped when it names this node's VNIC; one naming no VNIC cannot be attributed and is left alone rather than guessed at. * fix(host): find the EIP's source-routing table when the uplink is held as a /32 uplinkSourceRuleFor identified the interface owning an EIP's subnet by finding a host address configured with a prefix that contains it. On OCI both VNICs sit in one subnet, so br-wan has to hold its address as a /32 -- a /23 there puts the subnet's on-link route in the main table at metric 0 and takes the whole subnet off the primary VNIC. A /32 contains nothing but itself, so no rule was ever found, no source rule was installed, and the reply left by the primary VNIC where OCI dropped it for carrying an unregistered source. The address is not the only record of who owns the subnet. The rule names a table, and that table holds the on-link route. Consulted as a fallback, only when the address prefixes matched nobody, so a host that already resolved this the direct way keeps its existing answer -- prod and dev-prod are unaffected. Only scope-link routes count. A default route covers every address and would make every source-routing table an equally good answer. * docs(oci): correct the multi-node story and name the oci_vnic_id constraint The nat-mode warning said EIPs and default egress were both broken on more than one node. Default egress is fine: delete a guest's dnat_and_snat row and it reaches 1.1.1.1 in 2.3 ms from a guest on any of three nodes. Only the public-IP path was broken, and the three defects behind it are now fixed. What remains is not a bug but the shape of the datapath: routed NAT centralises the EIP path on the VPC's gateway chassis, ext-shared is a localnet so the other nodes cannot reach it, and OCI drops any frame whose source is not registered on the VNIC it leaves by. So every node's pool must pin oci_vnic_id to the gateway chassis's VNIC rather than resolving its own. Say so in §5.3, where an operator writing the pool will read it, and in §9, where the cost is that a gateway change kills every public IP until the pools are repointed. The single-node cap itself is gone from the doc: a three-node cluster has been formed and measured, and the warning proceeds rather than refusing. * feat(nat): distribute the routed-NAT EIP path so it works on more than one node Routed NAT processed every EIP on the VPC's gateway chassis, and ext-shared is a localnet, so the transit segment a /32 route pointed into existed only on that one node. Measured on three nodes: all three claimed all three EIPs, the nodes disagreed on the gateway LRP, and only guests on the gateway chassis were reachable. The mechanism to fix it is not new. Pool mode already distributes EIPs by putting external_mac and logical_port on the dnat_and_snat row and letting OVN's ARP responder answer for the external IP on the external segment. Routed mode has an external segment too, the spx-nat veth pair, and it being per node is the distribution: only the node running the instance can answer ARP on it. So a routed rule is now distributed on the same terms as a pool one, and its host route becomes on-link on the transit veth instead of via a gateway LRP that may be on another chassis. bindHostEIP stops demanding that address for a distributed rule, which is what previously failed the bind on every node but one. vpcd plumbs only the ENIs local OVS reports as its own, asked of OVS rather than the Southbound DB because the question is about this host. The gateway LRP keeps its transit-pool address. Default egress routes through it and works across three nodes today, so making it link-local as pool mode does would break a working path to fix one distribution has already fixed. Pool and static are unmoved, and a test asserts that against the same input rather than leaving it to inspection. A rule with no MAC stays centralised, so a cluster part-way through an upgrade is never in a state neither shape handles. Beads: mulga-opyl7, mulga-la0pe * fix(nat): make routed-EIP host plumbing work on every node Three guests on three OCI nodes, one guest per node: only one was reachable. Two defects, both invisible on a single node. The transit veth took a random MAC per node, but OVN holds one MAC binding for the transit nexthop across the whole cluster — so every chassis addressed its egress to whichever node seeded the binding, and the other kernels dropped it at L2. The pair is node-local, so one locally-administered address everywhere is unambiguous; setup-ovn.sh pins it at creation and in the netdev unit, and EnsureUplinkPort repairs a veth made before that. Host routes and proxy-ARP also rode the leader-gated reconcile pass, so a node that never won the lease never plumbed its own guests' EIPs. BindHostEIPs does that half on its own loop on every node; Bind already leaves alone an EIP whose ENI is elsewhere, so two nodes never contend. An unresolvable address skips the sweep rather than shrinking it. Pool and static are untouched: the new pass is a no-op outside routed mode, and a test asserts it. Also adds --teardown and --dry-run to setup-ovn.sh, for re-running it on a node that is already tangled. It deliberately spares the NB/SB databases, /etc/spinifex, the system-id and the packages, and refuses to del-br a Linux bridge. * fix(nat): assert the transit MAC where vpcd actually runs it EnsureUplinkPort is never called in production — host.Wiring is constructed nowhere — so the repair landed in a path the deployed daemon does not take, and three nodes came back with three different MACs. vpcd re-asserts it beside the NAT egress rules instead, which is the routed-NAT branch that does run on every start. * fix(vpcd): make uplink-mode detection say what it saw, and check it detectBridgeMode logged only the branch it took, so a node detected as veth when it was wired for nat looked identical to one that was right. It now reports both probes every time, errors when a node carries both uplink veths (set up twice, and --teardown is the way out), and errors when external_mode and the wiring disagree — the datapath follows the wiring, so the config silently loses that argument. An explicitly configured bridge_mode gets the same cross-check, since nothing else ever compared it against the wiring. The vpc-networking doc now opens nat with "start with pool unless one of these applies" and gains a section on nat across more than one node; the admin warning for --external-mode=nat on two or more nodes says what was verified by hand and what is still uncovered. * test(nat): prove routed NAT delivers each node's own guests, on a cluster natuplink ran single-node only, which is exactly where the two defects that broke three nodes are invisible. It now admits ModeMultinode, and phase 9 takes a second shape there: a guest per node in a spread placement group, and for each one — its public IP plumbed by the node running it and by no other node, the route on-link with proxy-ARP answered there, a dnat_and_snat row carrying external_mac and logical_port, a TCP handshake in from another node, and the guest's own console verdict for egress. Every node's transit veth is checked against the one cluster-wide MAC before any guest boots. Single-node phase 9 gains the same row and on-link assertions, so the shape is held on both topologies. Two gates come from what three OCI nodes actually do: every node holds a gateway_chassis row for every VPC (the active one is the highest priority) and every node installs the VPC ingress route, so only phase 7 needs gating — it runs where the gateway port and the guest are both local, and logs which one is missing otherwise. Cell 20 is real-multi x nat with its own nightly step, leaving the permutation cells' suite set alone. * test(nat): run the VPC egress paths in nat mode, not pool mode only The single suite gated TestVPCEgressPaths and TestWANEgressSMTPBlock on external_mode = "pool", with a comment saying nat clusters have no EIP/NAT-GW support. They have had EIPs since the routed path landed, so the gate now asks whether the cluster can allocate public addresses at all — pool, or nat with a non-transit pool configured. The nat-single cell runs TestVPCEgressPaths beside natuplink, which is the only nightly coverage of a private subnet's isolation and its NAT gateway in routed mode. natuplink never creates a private subnet. * docs(config): list nat among external_mode values * fix(ci): renumber the multi-node nat cell off the bare-metal job Cell 20 is already the bare-metal single-node job, gated on the same cells input but declared in the workflow rather than e2e-cells.json. A matrix entry with the same number runs two unrelated jobs under one name, and naming 20 in a dispatch selects both. * feat(images): add Oracle Linux 10.1 and 9.8, and pin their catalog digests Oracle publishes its qcow2 checksums inline on an HTML page and ships no sums file, so the four new entries carry the expected digest in the catalog rather than a URL to fetch one from. Images gains ChecksumDigest for that, and the catalog import path verifies against it and says which of the two sources it used, so a recorded digest never reads as a fetched sums file. Pinning is safe here and only here: each URL names an immutable build (b291, b293, b178, b182) and never moves. aarch64 has a -kvm-cloud- build alongside the plain -kvm- one; the cloud build is the cloud-init image and is the one that matches x86_64's -kvm-. TestImageCatalogReachable HEADs every URL in the catalog from the general single-node suite. These artifacts live on hosts nobody here controls, so an upstream that retires a build breaks `spx admin images import` for a customer and nothing else in the suite would notice. It also asserts that every entry sets exactly one checksum source, and that no pinned entry points at a "latest" URL whose bytes would change under it. * fix(formation): carry bridge_mode onto the leader and every joiner applyNetworkConfig copied external_mode, external_interface and the pools out of the formation NetworkConfig but never bridge_mode, so a multi-node cluster wrote a spinifex.toml without the key while single-node init wrote it. Every node then fell back to autodetecting a mode that init had pinned. Nightly cell 30 caught it: TestNATUplink failed at the config phase on a spinifex.toml that did not contain bridge_mode = "nat". Deriving it from the mode rather than adding a NetworkConfig field keeps the leader and the joiners on the same rule single-node init already uses. * feat(ui): show how to ssh into an instance, and which user to use An instance detail page gave the addresses and the key pair name but never the login user, which is the one part of the command nobody can derive. It belongs to the AMI, not to Spinifex: nothing in the launch path creates or renames an account, so stock cloud-init applies the distro's own default_user. Oracle Linux is cloud-user, Debian is debian, Ubuntu is ubuntu, and guessing from the distro name gets two of those three wrong. The card resolves the user from the AMI's name and description, and an unrecognised image yields a placeholder rather than a guess, because a wrong username in a copy-paste command is worse than an honest gap. A private-only instance is called out as reachable from inside the VPC only, rather than presented as a command that will simply time out. * fix(nat): plumb host state for a NAT gateway's public address AddNATGateway installed the OVN snat row and stopped there. On a routed-NAT node the host is what delivers a public address, so an egressing private guest left with a source the host held no route back to, and the masquerade rule — which matches the transit 100.127.0.0/24 alone — never saw the packet. The gateway's address needs the same /32 route into OVN, proxy-ARP and FORWARD accepts an EIP gets. hostBindSpecs unions floatingIPSpecs with one deduped spec per NAT gateway address and feeds BindHostEIPs. It is kept apart from floatingIPSpecs deliberately: pruneOrphanEIPs and addLive read that set as the complete list of addresses allowed to hold a dnat_and_snat, and a gateway holds an snat, so widening it would stop the sweep reclaiming a stale row on an address later reused as a gateway's. AddNATGateway also binds directly, on both the create and idempotent-skip paths, so a fresh gateway is not dark for up to 30s waiting on the host EIP pass — and so a reboot that keeps the OVN row still replays the volatile half. Documents NAT gateways in nat mode in vpc-networking, and says at the head of that section that nat is a supported mode covered by nightly cells 19 and 30. * docs(oci): document Oracle Linux as a guest image, and that its user is cloud-user Four catalog entries, all UEFI. The login user is the one that catches people: opc is the default on the images Oracle publishes into OCI itself, while these are the generic KVM cloud images from yum.oracle.com, whose cloud-init default_user is cloud-user. Verified on 10.1 and 9.8 — opc, oracle and ec2-user are all refused. Also records why these four pin a checksum digest inline (Oracle ships no sums file, and each URL names an immutable build), why arm64 takes the -kvm-cloud- build, and that SELinux ships enforcing. * fix(reconcile): stop reading an unreadable intent as an empty one A wedged JetStream KV returned no keys for the ENI bucket while writes were still landing. loadPorts mapped that to an empty port map, pruneOrphanPorts read every guest LSP as an orphan against it, and every guest on the node lost DHCP and IMDS while the API kept answering and the instances stayed "running". The existing re-read guard did not help: it re-reads through the same path. Three defences, because no single one covers every shape of the fault. A failed kv.Get now fails the pass instead of dropping the record. Dropping one VPC narrows localVPCs, which filters out every ENI in it, which the sweep reads as permission to delete every guest port in that VPC — one failed read, same outcome, and it needs no exotic KV state. Unmarshal failures stay skips: that is a poison record rather than an unhealthy store, and failing on it would wedge reconcile forever. The port sweep now asks local OVS which taps are plugged in here and never deletes a port that has one. A running guest's port cannot be an orphan whatever intent says, and OVS is the one liveness signal that does not come through the KV read path the sweep is otherwise entirely driven by. Wired from vpcd; unwired callers fall back to the guard below. An unreadable OVS skips the sweep rather than deciding on intent alone. Each sweep also refuses outright when the keep set it would delete against holds nothing at all — no ENIs, no SGs, or no ports and no external IPs. That is the case local taps cannot cover, because the drift sweep is leader-gated and lists NB cluster-wide, so the leader holds no tap for another node's guest. The cost is real: a genuinely orphaned row now survives until one live record of its class exists. Leaking a stale row is the recoverable half. Three existing prune tests drove their fixtures to a wholly empty intent, which is now the refusal case; they keep one live record of the class so they still prove the prune. The Get-failure path has no unit test — there is no jetstream.JetStream fake in the tree and an embedded server cannot be made to list a key it will then refuse. * docs(oci): separate the two address quotas, and the principal that can read them The regional reserved-public-IP ceiling is tenancy-wide and does not grow with node count, so a cluster hits it before the per-VNIC 64, which does. Records the commands that read both, and that a detached reserved public IP still bills. Reading limits or usage needs a tenancy-level grant, which cannot be compartment-scoped the way the node's policy is. That is a second, read-only principal kept off every node, not an addition to SpinifexOperators. * docs(oci): give quotas their own troubleshooting section, and say we enforce none Spinifex hardcodes no address ceiling: it attempts the allocation and maps OCI's own refusal to InsufficientAddressCapacity. So a quota is raised with Oracle and nothing here is reconfigured or restarted — which is what an operator needs to know before it bites, not after. Separates the two quotas by how they scale: the regional reserved-public-IP default is tenancy-wide and raisable by ticket, the per-VNIC 64 is per node and is not. * fix(oci): follow a guest to its new node when its EIP moves OCI delivers a public address to whichever VNIC holds the private IP behind it, and nothing ever moved that private IP. Binding.VNICID was written once at allocate time, as the VNIC of whichever queue-group daemon happened to serve the AWS request, and never updated — so an address landed on the right node only by chance, and a guest that stopped and started elsewhere went dark while every OVN row, host route and firewall rule on the new node was correct. ClaimLocalAddresses closes the gap from the node that should hold the address. Local OVS decides ownership, the same authority the host EIP binder already trusts for the same question, and UpdatePrivateIp with a new VnicId does the move in one server-side call. The private IP OCID survives it, so the reserved public IP attached to it follows without being touched and the customer's address never changes. A loop per node, not leader-gated: the question is about this host's guests, and a node that never wins the reconcile lease would otherwise never claim the addresses of the instances it is running. * fix(oci): record which ENI an EIP is associated with An EIP is allocated bare and attached by a later AssociateAddress, so the OCI binding written at allocate time named no ENI and nothing ever filled it in. The address then stayed on whichever node happened to serve the allocation: the affinity pass could follow an auto-assigned address, which is allocated with its ENI, but never an EIP — the case customers actually use. BindOwner is on an optional OwnerTracker interface rather than Allocator, because only OCI has a notion of an address being delivered to one node. A static or DHCP address is on the wire everywhere its node can reach, those allocators do not implement it, and ExternalIPAM treats a pool that does not as a success with nothing to do. The same gap made the release owner guard inert for EIPs: it compares against a binding ENI that was always empty, so a stale teardown could free an address already handed to another instance. * fix(ec2): report the address an instance is actually reachable on Three faults kept DescribeInstances answering with the address launch assigned, whatever happened afterwards. The launch-time stamp lives on the stored ec2.Instance, and the stopped path only declined to add an address rather than removing one, so a stopped instance went on advertising an address it had released and that may already belong to someone else. The projection now clears it, and the NIC association with it, working on its copy so a restart can still report the address. The running path filled the public address only when it was nil, which the stamp never is, so an EIP could not be reported however correctly it was recorded. The runtime address now wins. And nothing told the rest of the system about an association: AssociateAddress wrote neither the ENI record, which is what IMDS and the network reconcile read, nor the instance record, which is what a describe answers from. The ENI write is direct; the instance write is a fan-out, because the queue-group worker that serves AssociateAddress is rarely the node running the instance. * fix(net): take the host EIP plumbing off a node its guest has left Bind refuses an EIP whose ENI is on another chassis; the prune beside it kept anything still wanted cluster-wide. A guest that moved therefore left the old node holding the /32 route into OVN and the proxy-ARP entry on the uplink, with nothing that would ever remove them. On a shared segment that node goes on answering ARP for an address another node is now serving. The two halves now ask the same question by the same authority, in the loop that already runs per node. An unreadable local OVS is not a yes: tearing down a live address on a guess would fire everywhere at once, so it keeps the binding and the next pass tries again. A binder with no chassis test behaves exactly as before, and a centralised EIP is exempt because it hairpins through the gateway chassis and every node plumbs it legitimately. * fix(net): refuse to free an Elastic IP on behalf of a departing interface Releasing an instance's public address on teardown releases it by address, scoped to the ENI holding it. Since an association now writes the EIP onto the instance record, that address is the EIP — so terminating an instance gave a customer's Elastic IP back to the pool while describe-addresses still reported it as theirs, free to be handed to another guest. The existing ownership guard cannot catch this. It only fires when the stored allocation already names an ENI, and an EIP is allocated bare and associated later, so the stored ENI is empty and the guard passes straight through. The distinction that holds is who is asking. ReleaseAddress is the address's own owner and names no interface; instance teardown names the interface it is tearing down. So a release that names an owner may free an auto-assigned address and never an EIP, which is enforced in each of the three allocators. DHCP needed the scope carried on the wire, since it releases by address and resolves the lease afterwards. Absent on an older caller, which reads as false and keeps the previous behaviour. Worst on the two providers where it is not recoverable: on DHCP the lease is the only thing holding the address and the server promises nothing about handing it back, and on OCI the address is a reserved public IP object that the release destroys outright. * test(e2e): prove an Elastic IP survives a stop and start Instance teardown releases the public address by value, and since the association writes the EIP onto the instance record, that release aims straight at the customer's Elastic IP. The allocator guards now refuse it, but nothing end to end proved the address comes back on a restart. The step goes inside EIPFlip, which already runs last on the shared public guest for the same reason a stop is safe there: the address the guest returns on is the allocation's, not the pool's. It runs on the static cells and the DHCP ones alike, and on DHCP it doubles as the only outside view of the lease still renewing while the guest is down. * feat: release an auto-assigned public address on stop, take a new one on start AWS lends an instance a public address for as long as it runs and gives it a different one on the next start. Spinifex kept the address across a stop, which holds a pool slot for a stopped instance and, on a DHCP pool, keeps renewing a lease nothing is using. A stop now offers the address back through the instance cleaner, which refuses anything an Elastic IP is behind — the EIP association record answers that, the same question an ENI delete already asks — and records on the VM that the start owes it a new one. A host drain keeps the address: Restore relaunches the guest on the next boot, and changing its address under it is an outage nobody asked for. The start allocates before the launch, so an exhausted pool refuses the start rather than booting a guest the customer cannot reach, adopts an Elastic IP associated while the instance was stopped instead of assigning a second address beside it, and hands the address back if the launch then fails. e2e: a StopStartAddress stage on the public guest asserts a stopped instance reports no address and comes back on a different one from a static pool, and a StopStartKeepsTheEIP step asserts the opposite for an Elastic IP. * docs: restructure the OCI guide around two deployment paths, with diagrams The guide had grown into a numbered wall with no shape: no frontmatter, no contents, and a reader with an OCI account had no way to tell where to start. It now follows the multi-node install template — overview, prerequisites, then the deployment path — and splits that path in two: Terraform for one node or three, which is the preferred route, and a manual single-node walkthrough for an instance that already exists. Both meet at one shared configuration section rather than repeating it. Two ASCII diagrams carry what the prose could not: one OCI instance with the whole AWS surface inside it, and three nodes with the same surface distributed, showing which plane each VNIC carries and how an address follows a guest that moves. The limits section was materially out of date and said so in the places that mattered most. Multi-node is no longer experimental, public addresses are no longer pinned to one node — each allocates on its own VNIC and the affinity pass moves the private half to wherever the guest is — and an auto-assigned address is now returned to OCI on stop, so a stopped fleet stops paying for addresses. What changed is recorded rather than quietly edited. The Terraform configuration moves in from the monorepo as scripts/terraform/oci-spx, so the guide points at something in this repository. Also marks the two new lifecycle test files in-package, with the reason each one needs to be: both substitute unexported dependencies. * docs: rework the OCI guide — SVG diagrams, Terraform variables, iSCSI and br-wan Replaces the two ASCII diagrams with SVGs in the house style, one for a single node and one for three. Renames the two deployment sections to Terraform deployment and Manual deployment, documents what the Terraform builds per node and every variable worth setting, and explains that the data volume is an iSCSI attachment rather than a disk. The manual path now covers the iSCSI login, the fstab entry and the netplan source routing that Terraform writes, and states what br-wan is for on a routed-NAT node: it names the VNIC for the OCI allocator and holds the registered addresses, but is never bridged into OVS. Opens with what Spinifex is and why OCI is worth running it on. * fix: make gateway convergence work on a multi-node routed-NAT cluster Three defects each stopped a multi-node cluster from ever attaching an internet gateway, and all three produced the same uninformative symptom: `terraform apply` sitting on `aws_internet_gateway.igw: Still creating...` forever while every service reported healthy. ovn-northd was given only its local OVSDB socket as a client connection. A RAFT follower does not forward writes -- it answers "not cluster leader, trying another server" -- so with one entry in the list, northd stops translating Northbound intent into Southbound flows the moment database leadership moves to another node. Measured on the OCI cluster: SB_Global.nb_cfg seventeen transactions behind NB_Global.nb_cfg, with no chassisredirect port for any VPC created after leadership moved. setup-ovn.sh wrote the member list only when the caller passed --ovn-remote, which only install-node.sh did; it now derives the list from the live RAFT membership otherwise, and says so when it finds a single member, because that node needs a second pass once the others join. The gateway datapath probe ran wherever the reconcile lease landed rather than on the chassis owning the port. Both probes leave from the local uplink, and under routed NAT the transit subnet exists identically on every node, so a remote probe reported a healthy gateway as dead and aimed a repair at a datapath the node does not own. GatewayPortLocal gates it. On OCI the probe then ARPed the tenant-visible public IP, which the cloud NATs and which appears on no interface anywhere. The reconciler now maps the address through the same DatapathResolver the NAT rules are built from. Also adds DescribeVpcEndpoints, which the AWS provider calls on every VPC delete to sweep GuardDuty-managed endpoints; an InvalidAction there failed the destroy and stranded the VPC. The OCI guide gains a Terraform section recording which shipped workbooks were validated against a three-node OCI cluster and which are still open. * docs: record the validated Terraform workbook results for the three-node OCI cluster Six of the seven workbooks run against the OCI cluster pass, including the three the section exists to prove: RDS, ECS and EKS. Records nginx-alb and the clean eks-quickstart re-run, and names the two lifecycle gaps that bit during the run so an operator recognises them rather than reading them as a broken workbook. * fix: route the NAT gateway's SNAT source through the datapath resolver AddNATGateway was the one NAT path that did not map its external address to the one that rides the wire. Where the public half of an allocation is NAT'd by the cloud it is on no interface anywhere, so every private-subnet guest egressed with a source that was dropped, while the gateway reported available and DNS kept working because the VPC resolver is host-local. The unmapped address also failed the host EIP sweep, which works in resolved addresses, so the bind was pruned as foreign on the next pass and reinstalled on the one after. The public half now survives as an external ID instead. Also restructures the OCI guide around numbered steps through to running Terraform, and moves the cloud-init and metadata sections into troubleshooting where they are read when something is wrong rather than followed when it is not. * fix: place a NAT gateway's OCI address on the node that forwards for it OCI drops a packet whose source is not a registered private IP on the VNIC it leaves from, so a NAT gateway's SNAT source has to be on the gateway chassis. Two independent placements decided those and nothing reconciled them: the address was created by whichever worker served AllocateAddress, while the chassis is a fixed ordering the reconcile pass asserts. On a three-node cluster that made private-subnet egress a coin flip, and the same Terraform passed or died depending on which node answered. The affinity pass already solves this for guests, by asking local OVS where the tap is. A NAT gateway has no tap, and it skipped anything with no ENI — which is exactly and only this address. The equivalent local fact is the VPC's chassisredirect binding, so ownership now has two answers rather than one, and an address that is neither a guest's nor a gateway's is still left alone. The VPC is recorded from the reconcile pass rather than from CreateNatGateway, because the answer is the gateway chassis and reconcile is what decides it. * fix: plumb a NAT gateway's host ingress only on its gateway chassis A NAT gateway's address has no ENI, so neither the bind's locality test nor the prune's could ever fire for it and every node installed its /32 route into OVN. Latent on bare metal, where pool addresses are disjoint from the node subnet; on a cloud that hands out the address from the nodes' own subnet it outranks the connected route and swallows node-to-node traffic. Give that address the same second authority the ocinet affinity pass has: the VPC's chassisredirect binding, read from the local ovn-controller. An unanswerable question still leaves the binding alone, so a single node whose gateway port is not yet claimed behaves exactly as before. The orphan EIP prune built its live set from ENI and user EIPs alone while also sweeping host ingress, so every drift pass tore down the ingress of every live NAT gateway and left it dark until the next host EIP tick. * fix: ask the gateway chassis only about a NAT gateway's address gatewayElsewhere skipped the NAT gateway marker it was named for and answered for every spec, so on a node that is not the VPC's gateway chassis every EIP — including a guest's, whose locality is its own port — was left unplumbed. Two of three nodes bound nothing. * test: assert the two private-subnet EKS workbooks in the tofu harness eks-https-ingress and eks-gitops-argocd put their workers in private subnets behind a NAT gateway, so a worker only registers Ready if its egress to the cluster endpoint and to ECR survives that path. That makes them the workbooks that catch a NAT gateway fault, and neither had an assertion, so neither could be run at all — run_workbook calls assert_<name> by construction. Both share eks-quickstart's readiness check, extracted unchanged. They also name the worker size node_instance_type rather than instance_type, and tofu rejects a -var for an undeclared variable outright, so the name is read from the config instead of kept in a list here. * test: count agents, not every node, when asserting EKS workers Ready The control plane is three K3s servers, so an unfiltered node count reaches a one-worker cluster's desired size before any worker joins — the assertion passed on exactly the failure it exists to catch, reporting "4/1 workers Ready". Filtering on the control-plane role label makes it 1/1, which is the number that says a node-group worker registered. Matters most for the two private-subnet workbooks, where the worker joining is the whole proof that egress through the NAT gateway works. * docs: record the two private-subnet EKS workbooks passing on OCI	2026-09-26 14:31:07 -05:00
Ben Duncan	d65c4f3673	fix: family-qualify unscoped firewall accepts to IPv4 (#999) ... The public, formation and DNS accept rules carried no address family, so on a node that autoconfigures a global IPv6 from an upstream Router Advertisement they served the console, predastore S3 gate, AWS gateway and DNS on that public address with none of the source scoping applied on IPv4. Qualify them with meta nfproto ipv4 so v6 falls to the default drop; the scoped rules are already v4-only via ip saddr.	2026-09-04 13:21:36 -05:00
Ben Duncan	828b5df4aa	fix: pause a guest on a backend I/O error instead of handing it EIO (#977) ... * fix: fail DRAIN when a volume seal failed shutdownAndUnmount logged the volume unmount error and returned nothing, so StopAll returned nil and the DRAIN phase ACKed success with vms_stopped set. The coordinator then advanced to STORAGE and stopped viperblock while the seal was still in flight, tearing the VBState envelope and leaving the guest unable to mount its root volume on the next boot. The deploy reported success. shutdownAndUnmount now returns ErrVolumeSealFailed when the unmount did not seal; QMP, PID-file, fw_cfg and telemetry cleanup stay best-effort. stopCleanup and stopOne carry it without aborting the rest of the teardown, since QEMU is already down by then, and StopAll aggregates the per-VM failures so handleShutdownDrain ACKs with Error set and never reports vms_stopped. volumeMounterAdapter.Unmount was the second swallow site: it logged each failed seal and returned nil unconditionally. It now attempts every volume and returns the aggregate, which is what the VolumeMounter contract already described. Terminate still tolerates a failed seal — the volumes are deleted next, so an unsealed block map loses nothing a caller can ask for. This does not fix the seal itself. It converts silent data loss into a refused upgrade that an operator can act on. * test: add the storagefault e2e suite for backend-outage guest survival Freezes predastore with SIGSTOP to hold connections open while answering nothing, which is the fault a timeout-free path cannot distinguish from a slow peer. On one node that is a total outage; on three it freezes every node but the guest's own, breaking the RS(2,1) shard floor. The suite writes and flushes a verifiable region before the fault and churns a separate unverified region during it, so an interrupted write is never mistaken for lost data. The root-filesystem load runs from the user's home rather than /tmp, which is tmpfs on this image and would move no bytes on the root device. * test: verify the root device is under write pressure, and size the outage The previous guard matched /^ *vda / against /proc/diskstats, where the device name is the third field, and read $7 which is ms_reading rather than sectors_written. It printed empty and asserted nothing, so a green run could not distinguish a surviving guest from an idle disk. It now samples sectors_written twice and fails unless it advances. The outage is sized from viperblock's absorption rather than guessed: a workload must fill DefaultMaxPendingBytes before backpressure engages at all, so a short freeze measures the retry budget rather than the fault. * test: add nbdkit fault injection to the storage-fault e2e suite The predastore freeze takes the whole storage cluster away from every guest on it, which is too broad to run beside anything else. Removing a single volume's NBD server is the same fault at one-volume scope: one host, one guest, nothing cluster-wide stopped. Both signals are here because they sit on opposite sides of the defect. SIGSTOP holds the socket open and answers nothing, so the guest blocks and no error reaches ext4. SIGKILL closes it, and QEMU's nbd driver at the default reconnect-delay of 0 fails requests at once, which under werror=report aborts the journal. Signalling is guarded: a pid is matched on comm, our plugin and the volume argument together, and re-confirmed immediately before the signal, because these hosts run nbdkit for other volumes and pids are reused. * fix: pause a guest on a backend I/O error instead of handing it EIO Every virtio-blk device was created with werror=report,rerror=report, so a backend outage was delivered to the guest as EIO. ext4 aborts its journal on that and remounts read-only, and the write that failed is simply gone. Under werror=stop QEMU pauses the VM and holds the failed request, so a backend that comes back resumes the guest with nothing lost. The boot drive, the cold-boot data volume and the hot-plugged data volume all move to stop. Alongside it the NBD transport gets reconnect-delay=30, which pauses requests rather than failing them while a storage process restarts on the same socket; on the legacy -drive form it has to be file.reconnect-delay, because the bare name reaches the raw format driver, which has no such option. The QMP heartbeat now recognises run state io-error and issues cont. After ten attempts the instance is reported impaired and the cadence drops to one attempt per ten heartbeats, but it never stops: QEMU holds the request for as long as the guest exists, so a backend that returns an hour later can still be resumed. The storagefault suite gains the fault this was written for — kill the volume's own nbdkit, hold, then put it back on the same socket — and it now verifies a sha256 region across the outage rather than trusting the guest to be reachable. Three harness defects went with it: the socket-liveness poll ran unprivileged against a 0770 directory and could never succeed, the relaunched socket was left 0755 where QEMU needs group write, and the two variants shared one guest so a fault in the first was diagnosed in the second. * test: cover the io-error resume policy * test: stop two parallel tests racing on the terminate backoff variable * fix: report a guest paused on a backend I/O error as impaired A guest held by werror=stop kept reporting ok on both status checks. QMP stays responsive while the guest is paused -- cont is being sent and answered -- so the QMP failure gate that DescribeInstanceStatus keys on never trips, and the one state a customer most needs to see looked healthy. Instance status now reports impaired/failed while the resume budget is non-zero, timestamped from IOErrorSince so impairedSince is when the guest paused rather than when it was escalated. System status reports impaired too: a backend I/O error is an infrastructure fault, not a guest one, and that distinction is what the two checks exist to draw. The counter is cleared by the first poll that sees the guest running, so both clear on recovery. * test: reap the relaunched nbdkit when the fault test ends relaunchNbdkit detaches the server with setsid so it outlives the SSH session that started it, and viperblockd never learned it exists. Nothing reaped it, so every run left an nbdkit holding a socket and a cache for a volume that was later detached or deleted -- four were found running on dev-prod against two detached and two deleted volumes. The PID is resolved from the socket path rather than the volume id, so a legitimate viperblockd server for the same volume is never the one killed. * test: prove the backend is serving before a fault is called cleared Thawing the process a fault stopped is not the same as restoring the service. The outage test treated SIGCONT as the end of the fault and went straight to judging the guest, so it passed while the cluster it ran on stayed unable to serve a single object for the next 35 minutes, and the next test would have inherited that. thawPredastore now puts and gets an object through every gate, with a budget, and fails the test if the backend does not come back. * fix: do not hand back a mount entry whose export is gone A volume whose unmount failed kept its entry in MountedVolumes while its nbdkit was gone. Every later ebs.mount then matched that entry and returned its socket path, so vmMgr waited for a socket nothing would ever create and the instance could not be started again — on dev-prod a guest stopped against a frozen backend was left permanently stopped, four start attempts failing identically on a socket the node had already removed. Drop the entry when there is positive evidence the export is dead: the socket is absent, or the recorded pid is not running. Neither can be true of a live export, so the double-writer hazard the guard exists to prevent is unaffected. Also make the durability test's pattern helpers report failures. The read dd sat in a pipeline, so its exit status was discarded and the harness hashed nothing and returned the empty-string sha256 as a checksum. * test: report the guest console when a restart never answers SSH A restart that reaches running but never comes back is indistinguishable from a dead root filesystem without the serial console, and the log is gone once the fixture terminates the instance. * fix: converge guest port datapaths concurrently, not one deadline after another Each guest port that misses its SB binding burns the full convergence deadline, and applyEIPs spent them in series. Three ports belonging to long-stopped instances therefore held a newly launched guest's EIP dark for the sum of them before its own probe started. AddEIP stays serial since it mutates NB. Only the waits fan out, bounded, and sb-cluster-state-reset is now serialised with a minimum interval so concurrent probes cannot each fire a cluster-wide resync. * fix: refuse a stale cross-node start after the owning node is gone The volume lease already refused a cross-node mount, but it is a liveness claim: 45s TTL, renewed every 15s. A node that dies holding un-uploaded writes stops renewing, and 45 seconds later nothing refuses a start elsewhere that loads a stale backend checkpoint. A failed seal now writes a durable marker naming the node holding the only current copy, cleared on a seal that reaches the backend and on delete. The lease covers a live holder, the marker a dead one. The refusal also reaches the caller instead of being flattened to ServerInternal, which told an operator to retry something that cannot succeed while the holder is down. * fix: keep the daemon's message when a start is refused Rebuilding the error from the code alone rendered the generic ErrorLookup text, so a refusal naming the node holding the volume reached the caller as boilerplate about attaching volumes. * fix(ec2): terminate detaches every DeleteOnTermination=false volume TerminateStoppedInstance skipped the detach for a non-Boot DoT=false volume on the assumption that Stop's Unmount had already cleared its attachment. Unmount only clears it when the seal succeeded, so a volume whose seal failed stayed in-use pointing at an instance that terminate then deleted, leaving it unreachable with no owner to release it. The running-instance path already detaches every DoT=false volume; the stopped path now matches it, and tolerates an already-gone volume the same way. DeleteVolume also keeps its provider error on the way out. The terminate reaper retries the delete every two minutes and logged a bare ServerInternal, which made a blob node being down indistinguishable from a transient fault. * fix(ec2): DetachVolume --force releases a volume whose instance is gone The ordinary detach routes to ec2.cmd.{instanceID} for the guest unplug, so a volume left attached by a part-way teardown could never be detached: the instance no longer runs anywhere and nothing answers. That also made it undeletable, since DeleteVolume refuses an attached volume. ForceDetachVolume already existed for this deadlock but was reachable only from account teardown. Force now falls back to it when the instance has no responder, which is the one case where clearing the attachment cannot strand a live guest. * fix(viperblock): unsealed writes steer placement instead of pinning a volume The dirty marker refused any mount from a node other than the one holding unsealed writes. Instance start already forwards to the node that last ran the instance and falls back after a 30s window, so that refusal turned a working fallback into a volume that could run nowhere once its owner was gone. It traded a rare data-loss risk for a permanent availability failure. The marker is now a record and a warning, not a gate. A takeover logs the previous holder and why it could not seal, then adopts the marker so the old node is no longer treated as the better source. Live exclusion is unchanged and still enforced by the lease. It is also written on open rather than on seal failure. A node killed mid-write never reaches its seal, so the old trigger left no trace in exactly the case the marker exists to describe. Adds 'spx admin volumes unsealed' to see which volumes would cost something if started elsewhere. * fix(viperblock): fence a volume whose lease moved instead of logging it The lease was checked once, at mount, and never again. A node that lost it kept its export up and kept writing into a volume another node owned, so both copies advanced and the backend settled it by arrival order. Losing a renewal now decides: another holder means tear the export down, no holder means the entry merely aged out, so reclaim it and leave the guest running. Fencing never seals — this node's copy is the stale one — and leaves the dirty marker for the winner to take over. This is not an epoch fence. It narrows the window from the 45s TTL to one renewal interval at no write-path cost; the backend still cannot refuse a stale writer. * fix(network): advertise EIPs so a recycled address is reachable The gateway switch port only asked ovn-northd to publish the router's NAT addresses in centralised and routed modes, so on a distributed cluster nothing ever announced an elastic IP. OVN still answers ARP for it, which is why a first-time allocation works and hid this — but no gratuitous ARP is sent when the address moves to a new ENI, so every host on the segment keeps resolving it to the previous holder's MAC until its cache ages out. Reusing an address from the pool then lands on a dead MAC and the instance is unreachable from outside. Distributed mode needs the option as much as the others: northd scopes each entry with is_chassis_resident(<vif port>), so the announcement leaves from the chassis running the instance rather than the gateway, which is what distributed NAT wants. Converge on the already-attached path too. AttachIGW returns early once the gateway port exists and nothing else rewrites it, so without that an existing VPC would stay silent for its whole life. Confirmed on dev-prod: setting the option moved three EIPs from unadvertised to carrying their correct external_mac, each scoped to its own ENI's port. * fix(viperblock): make the volume fence and its dirty marker ownership-safe Four corrections to the lease-loss fence, all found in review. Drop the reacquire path. Losing a lease and finding no successor was treated as a lapsed entry worth reclaiming, but leases.acquire returns the cached entry — already flagged lost, with a dead renewal goroutine — so the "reclaimed" lease renewed nothing and the node went on writing a volume it could not prove it owned. The two errors are not symmetric: a false fence costs one guest a restart, a false reclaim leaves two nodes writing one volume. Fence in both cases, and report which happened through the outcome attribute. Surrender a lease this node cannot confirm. Renewal only noticed a loss when the server rejected an update, so a node partitioned from NATS kept its export up while the TTL expired and another node took the volume. Track the last confirmed renewal and give the volume up once that ages past the point where the server may admit another owner. Bound the renewal call too, so a hung JetStream request cannot hold the goroutine past that point. Do not act on a kill that did not take. Both the fence and ebs.unmount removed the mount entry, released the lease and announced the result before knowing nbdkit had exited — the fence then reported a writer stopped that was still running, and the unmount sealed into a directory a live process was writing. Wait for the exit, and on failure leave the volume mounted and say so. An already-exited process now counts as success, which is what the caller asked. Give the dirty marker a generation. It recorded an owner and nothing to order two nodes' claims, so a node returning after a takeover could clear or overwrite the winner's marker and point placement back at the stale copy. Stamp the lease generation, refuse a write from an older one, and only clear a marker this node still owns. A marker that cannot be written now fails the open rather than leaving writes nothing records. * test(viperblock): cover the stall surrender and the unmount kill guard The two ownership cases the fence work left untested. A holder partitioned from NATS while the winner keeps it never gets an update rejected, so revision checking cannot see it — it has to give the volume up on its own clock. Pins that validity sits below the server TTL with room for a check in between, that a lapsed holder reports the loss as a stall rather than as a takeover, and that releasing a surrendered lease leaves the entry for whoever holds it now. The unmount side of the kill guard: sealing under a live nbdkit writes a state torn between two writers, so a kill that did not take has to fail the unmount and leave the entry mounted for a retry. * fix(e2e): keep the guest private key for as long as the key pair EnsureKeyPair computed the PEM path from the caller's directory but wrote the file inside the ensureOnce closure, which runs only on the first call. Callers passed a per-test artifact dir, and ArtifactDir removes that dir when the test passes — so the first test's success deleted the only copy of the private key while the key pair stayed registered, and every later call returned a path nothing had ever been written to. Guests after the first were then launched against a public key whose private half no longer existed, so the harness could never authenticate and gave up after the full five-minute budget. The guests were healthy and reachable on tcp/22 throughout, which is why this read as a storage or network fault for so long. The directory parameter was the defect, so it is gone: the PEM now lives in a directory the fixture owns and removes with the key pair, and no caller can supply one that is shorter-lived. Also reorder the killed-nbdkit teardown. The reaper registered its own cleanup after the detach registered one, so LIFO killed the block server before the platform detached the disk and DetachVolume then failed with "Node ... is in use". The detach cleanup now reaps once it is done. * fix: stop leaking an nbdkit export when recovery cannot adopt it Restart recovery adopts a guest's boot volume, then routinely fails the -efi volume with "claim lease: context deadline exceeded" -- JetStream is still catching up at daemon start and the KV claim runs on the client's 5s default. Recovery gave up on the first error and never retried, so the export stayed live with no registry entry. At terminate the unmount found nothing to seal, returned NotFound and killed nothing, and the process outlived its guest holding the volume and socket open. EFI is always the second volume adopted, which is why it was always the one that leaked. Two changes, because either alone leaves a hole: - rebuildMountedVolume retries the build (4 attempts, 3s apart) so a JetStream that is merely slow no longer orphans an export for the life of the process. - unmountVolume reaps a corroborated export for the volume it was asked to unmount when the registry has no entry for it, so an export orphaned by any means is still cleaned up. The reap goes through reapOrphanedNbdkit, which declines anything a live process references: at unmount the guest's blockdev may not be torn down yet, and killing the server underneath it strands QEMU. * test: hold the backend-serving probe and assert predastore readiness The backend-outage test declared the backend restored on one successful round trip through each gate. A gate can answer the request that arrives just after a thaw and then lose its blob peers under the load that follows, which is a backend that never recovered reported as one that did -- and that is what happened: the probe reported all three gates serving minutes before the guest was still taking BLOCK_IO_ERROR on every resume. The round trip has to hold across three rounds now, and the count restarts rather than decays when one fails, so a gate that serves and then stops is the failure the hold is looking for rather than noise it averages out. It also checks /readyz on each host and reports any failed check by name. The round trip only proves this workstation can write through a gate; it says nothing about which blob peers that gate can reach, and a gate one node short of the write floor serves a small object and fails a guest. A host that serves no admin listener is logged and skipped, since the port is optional configuration. * fix: scope predastore's admin port to the cluster plane in the host policy predastore's admin listener binds the host's cluster bind_addr and serves /healthz and /readyz unauthenticated. Readiness now names which blob peers a gate cannot reach, which is exactly the kind of topology detail 4432 is peer-scoped for, so 8660 joins it in the peer-scoped rule rather than the public one. * fix: keep resuming a paused guest at heartbeat rate after a long outage Past ten attempts the cont dropped to one per ten heartbeats, so a guest whose backend had come back could sit paused for another five minutes before anything retried it. Measured on dev-prod: the backend was proved serving through all three gates while the guest was still waiting on attempt 60, five minutes after attempt 50. The slowdown was there to avoid paying a failed I/O every thirty seconds on a backend still down. That is nearly free, and five minutes of downtime after recovery is not, so only the log line is throttled now. * test: fail the run when a key pair name resolves to more than one pair A create that lands twice leaves two key pairs with one name and different material. RunInstances resolves the name to one of them while the harness holds the private key of the other, and the only symptom is every guest in the run refusing the key at the SSH gate — five minutes of waiting per guest, four guests, and nothing anywhere naming the cause. Observed on dev-prod: two pairs created a millisecond apart, both named e2e-key-30de10ba. One DescribeKeyPairs after the create says so outright instead. * test: tolerate an already-released volume in the detach teardown Teardown runs DetachVolumeWait after a terminate that releases the volume itself, so the detach can arrive at a volume already on its way to available. That is the outcome it wanted, and treating the IncorrectState refusal as fatal failed a backend-outage run whose assertions had all passed — the guest came back, every acknowledged write verified, and the test still went red on the way out. VolumeInUse is still fatal: that one means the volume is attached somewhere it should not be. * test: carry the merged suites onto the reworked key-pair and account-scoped seams The diskperf suite arrived on dev calling EnsureKeyPair with the artifact dir this branch removed, and DeleteVolume's metadata lookup is account-scoped now, so the provider-failure test has to seed and delete under the same account. * refactor: make the fence outcome a type in one const block Four separate exported string constants with a paragraph of comment each, passed to RecordVolumeFence as a bare string. Group them, give them a FenceOutcome type, and take that type at both call sites so an arbitrary value can no longer reach the metric attribute. * refactor: move viperblockd's wire contract into a leaf vbwire package The daemon imported viperblockd for exactly two symbols -- the fenced volume subject and its event -- which put the storage service back on the daemon's dependency graph to name something it only listens for. vbwire holds the contract instead: the subject and event, the dirty bucket name and record shape, and the read-only listing an operator tool needs. It imports nothing from spinifex, so both the daemon and the admin CLI drop their edge to viperblockd entirely. cmd/service.go keeps its one, which is genuine: it constructs the service. * docs: drop a specific host name from the start-refusal comment Named a real dev-prod host in a comment and in test fixtures. Say what the name is instead. * ci: give storagefault a dedicated nightly cell The suite is eligible for both topologies and runs on the PR gate, but no nightly cell ran it: the permutation cells run a deliberately narrow subset and none of the dedicated suite cells named it. Cell 29, single-node, following eks/ecs/rds. It cannot join a permutation cell -- it SIGSTOPs a cluster-wide service, so every suite sharing that environment would report the outage as its own failure. * ci: run storagefault nightly on three nodes, not one The previous commit put its dedicated cell on a single-node runner. That was the wrong topology and the wrong justification: single-node is RS(1,0), where the shard floor cannot be broken and a cross-node takeover after a failed seal does not exist, so the two assertions worth having a dedicated cell for were the two it could not make. Cell 29 is now real-multi on ci-multi. It also corrects the claim that the PR gate already covers the multi-node case. The suite is eligible for both topologies, but e2e.yml's default suite list never names it, so it has never run there. A full pass is ~63 minutes on dev-prod bare metal, almost all of it deliberate waiting -- one test holds predastore SIGSTOPped for seven minutes -- which is too long to add to a gate that currently finishes well inside its budget. Nightly is where it belongs. Consequences: a per-cell timeout_minutes, defaulting to the previous 50, since 90 minutes of suite cannot run in a 50-minute job; the permutation steps gated on matrix.suite == '' so the dedicated cells do not also run them; and a Run Multi Suite E2E step that builds NODES_CSV from the tofu inventory and asserts the suite against E2E_SUITES_MULTI before running it. * fix: stop a startup KV race leaving vpcd serving keyless guests vpcd opens five KV buckets while NATS is still coming up, each attempt carrying a ~5s JetStream API deadline, and applied three different policies to the same failure: spinifex-vpc-enis was fatal, while instance-state, security-groups, subnets and vpcs each degraded silently for the lifetime of the process. A five-second blip during a cold start therefore decided how a node served metadata until someone restarted it. On dev-prod all three nodes hit this after a deploy. Two were saved by accident: their ENI bucket also timed out, so they exited 1 and systemd restarted them cleanly. The third's ENI bucket opened in time, so it never crashed and ran for hours with no instance record loader. With the local state file unreadable as well, every instance-backed IMDS leaf 404'd, cloud-init's crawl died on ami-id and fell back to DataSourceNone, and every guest launched on that node booted with no authorized_keys. kvstore.OpenWithRetry retries for a bounded window before any policy applies, logs the cause per attempt and the wait on a late success, wraps the last error with the bucket name, and returns at once on a cancelled context. All five opens use it. Failing to open the instance record space is now fatal, matching the ENI bucket: the dependency whose loss stops guests logging in should not be treated more leniently than the one whose loss stops ENI resolution. Also fixes the reason the fallback was load bearing at all. The local state file was written 0600 and vpcd reads it as a different user in the shared spinifex group, so the local-first path had never once worked and its mtime+size cache had never fired. It is 0640 now, set with an explicit Chmod so an inherited umask cannot undo it. Closes mulga-gtpzz, mulga-dhqxr. Still open: mulga-mm2oc, IMDS serving an unresolvable instance as "no key pair" with nothing logged. * fix: scope the ime-* firewall accept to IMDS and VPC DNS only The per-tap OVS demux delivers any guest packet addressed to 169.254.169.254 or .253 onto an ime-* host port regardless of L4 port, and iifname "ime-*" accept then admitted all of it. Every wildcard-bound host service — sshd, the AWS gateway, northstar, the console — answered guest packets on those addresses; only asymmetric reply routing kept the handshake from completing, and a guest could still drive northstar to recurse unthrottled. Scope the accept to IMDS (.254:80) and VPC DNS (.253:53), letting everything else fall through to policy drop. * docs: document the instance-to-host plane and how to protect it without the ISO The metadata/VPC-DNS link-local plane (169.254.169.254/.253) is served by host services and is invisible to a NIC-level or cloud firewall, since the traffic is intra-host on the OVS bridge. Explain that the ISO ships the scoping rule armed, that the binary installer ships the firewall off, and that a non-ISO install must enable it with setup.sh --firewall=on or firewall_enabled=true. Include verification from an instance and the host. * fix: withhold the IMDS meta-data listing when the instance does not resolve Every omission from the meta-data root is a positive claim about the instance — public-keys/ absent means "launched without a key pair", and cloud-init believes it, writes no authorized_keys and boots the guest keyless. So resolving the instance was best-effort for a listing whose whole meaning depends on it: a backend error or a miss shortened the listing instead of failing it, and nothing anywhere recorded that the answer was a guess. Withhold the listing with a 500 instead. cloud-init retries a 5xx and cannot retry a 200 that is merely wrong, so this converts a permanent silent failure into a transient loud one. servePublicKeys already had this policy for the leaf; the root is what contradicted it. A profile lookup that fails still serves the listing minus iam/, and now logs. The key pair is the field a guest cannot recover from losing, while a missing role surfaces through the credentials leaf, which reports the miss itself. Also stops the storagefault cross-node test failing on its own access path. It tolerates the pattern changing across a takeover, but readPattern needs sudo, and a volume opened from the backend checkpoint may predate cloud-init writing its sudoers drop-in — indistinguishable from an unreadable device at the dd call site. It now checks sudo first, and reads cloud-init's recorded datasource to tell a rolled-back root filesystem from IMDS never having answered, failing loudly on the latter rather than accepting it as storage loss. --------- Co-authored-by: brynmailer-mdc <bdm@mulgadc.com>	2026-09-03 19:45:33 -05:00
Julian Sommer	62c1378b79	fix: raise the Ubuntu minimum to 24.04 (mulga-ucvsq) ... Ubuntu 22.04 was accepted by detect_os but the runtime apt list has never resolved on jammy, which ships dhcpcd5 rather than dhcpcd-base. No CI cell or E2E matrix installs on it, and the install docs already claim Ubuntu 26.04 and Debian 13 only, so drop the claim instead of carrying a jammy-conditional package branch for an untested suite.	2026-08-27 10:49:46 +10:00
Ben Duncan	92c2de0017	feat(firewall): scope ssh to an operator allowlist in custom.nft (#881) ... The generated policy accepted ssh from any source and setup.sh rewrote spinifex.nft on every run, so there was nowhere to record an allowlist that survived an upgrade. custom.nft is created once, never rewritten, and kept across a node reset. It holds a define rather than rules, so an operator supplies a value and cannot write a rule that fails the ruleset. The default accepts every source, which is what the policy already did.	2026-08-25 14:37:43 -05:00
Ben Duncan	42d87d7a5a	feat: EBS export tunables, plus blob-read fault isolation for ebsmetadata (#799) ... * feat(ebs): make nbdkit thread count and per-volume read cache configurable Both numbers that set a volume's export concurrency and memory were hard-coded: nbdkit ran on its own default of 16 threads per connection because no -t was ever passed, and the read cache was a 128 MiB literal in the mount path. Neither could be changed without a recompile, and neither was visible in the logs or the process argv. Add default_threads and cache_size_mb to [nodes.<node>.ebs]. Defaults are unchanged, so a config that does not set them behaves exactly as before. CacheSizeMB is a pointer so an explicit 0 (cache disabled) stays distinguishable from an unset key, matching how ViperblockConfig treats its own toggles. -efi volumes stay uncached regardless. Validation is a range check only. Both are operator tunables applied per nbdkit process, and there is one of those per volume, so the host cost is the value times the volume count -- left for the operator to weigh rather than second-guessed here. * feat(security): arm the firewall by install path and sync the clock at firstboot The host firewall armed itself from a code default, so a curl-to-bash install onto an existing Debian or Ubuntu machine applied a default-deny INPUT policy seconds after bootstrap. Anything the operator was already running that is not on the allow list stopped accepting connections, with nothing logged and no config key to set beforehand — no template writes firewall_enabled, so every install fell through to the default. Arming is now the install path's decision. setup.sh takes --firewall=on|off and records it in /etc/spinifex/firewall/mode, defaulting on under ISO_BUILD and off everywhere else: the ISO is an appliance on hardware we define, a package install is a guest on a machine that was already doing something. firewall_enabled becomes a three-state *bool so "unset" and "explicitly false" stay distinguishable. Flipping the default to false instead would have torn down a working policy on every existing node, because false is an instruction to disable rather than an absence. Resolution is config, then mode file, then on, so a node installed before any of this keeps what it has. The ISO had the opposite problem. setup.sh is the only place that enables spinifex-firewall.service and it returns before that line under ISO_BUILD, with no systemd in the chroot to talk to. Nothing else enabled it, so after a reboot the ruleset — runtime state — was gone, and ReconcileFirewall short-circuited on an unchanged peer file and never put it back. An ISO node was firewalled until its first reboot and open permanently after. firstboot now enables the unit, and the reconcile checks the table is loaded rather than treating an unchanged peer set as proof the policy is applied. Two policy changes: 443 is held for the nginx TLS edge so that rollout needs no per-node firewall change, and the SSH port is read from sshd_config and its Include drop-ins instead of being hardcoded to 22 — a host on Port 2222 was allowed exactly one port that was not the one it uses, and ct state established kept the operator's session alive until they disconnected. 53 stays open on every node, now with a comment saying so: northstar serves public authoritative DNS. Clock: firstboot steps and waits for sync before spx admin init mints credentials, since a certificate issued against a wrong RTC carries a NotBefore that outlives the correction. Bounded to 60s and non-fatal so an air-gapped install still completes, and once is enough because rtcsync writes the corrected time back. A chrony drop-in replaces Debian's makestep 1 3, which stops stepping after three updates and then slews at roughly 7 seconds a day — a node that burned those before the network came up never visibly converged. * fix(security): let a node join a cluster whose firewall is already armed Expanding a single-node ISO install into a cluster could not succeed. Two independent blockers, both from the same root: the peer set is derived from converged cluster state, but the ports needed to reach convergence are gated on that same set. The formation port is peer-scoped, and a node dialling in to join is not a peer yet, so the handshake was dropped rather than refused and the joiner retried in silence until it timed out. It cannot simply move to the public rule: outside formation the same port carries the daemon cluster manager, whose /health and /local/* routes report node topology, service inventory and running instances unauthenticated. So it is opened for the formation window only, by spx admin init, and closed again on every exit path. The reconcile then deadlocked. Holding out for the full chassis list is right on a node with no policy loaded, but on one armed for a smaller cluster the ports a joining node needs in order to register its chassis are exactly the ones the stale set blocks, so the list can never complete. Widen the peer set on the tunnel set already loaded, and report the partial state so the caller retries sooner than the steady-state interval. A converged cluster whose Southbound DB is briefly unreadable is still left untouched. * docs(install): explain the firewall around cluster formation An ISO-installed server is a cluster of one, and its firewall admits internal cluster traffic only from members it recognises. Forming a cluster out of such servers therefore needs the firewall off first: step 3 of the multi-node guide has each server dial node 1's OVN database on the raft ports, and those are peer-scoped like everything else on the cluster plane. Neither guide said so, and the failure is a silent drop that looks like a network fault. Both guides now walk the off / form / on sequence, and the multi-node guide gains a reference section covering which ports are public, how arming differs between the ISO and the binary installer, and how to verify membership landed. Re-enable the boot-time unit whenever the daemon arms the policy. Disabling the firewall to form a cluster also disables that unit, so re-arming used to look complete while leaving the node bare after its next reboot. * fix: make the ISO-to-cluster conversion path work end to end Forming a cluster from four ISO-imaged servers surfaced a run of bugs that share a starting state CI never exercises: the ISO delivers an initialized, armed, running node, while the conversion path was written against the binary installer's unconfigured one. - install-node.sh never disarmed the host firewall, so nodes 2 and 3 could not dial the first node on the OVN raft ports and formation could not proceed. It now owns the whole lifecycle: disarm, form, re-arm, then verify the loaded policy against the expected peer set. - spx admin init left the admin access key and secret empty on the preserve path. The pair cannot be read back, and the IAM store is rebuilt by formation, so the empty strings reached every joiner's bootstrap.json and crash-looped awsgw. It is reissued instead, and IAM seeding now skips an empty admin block rather than taking the gateway down on every restart. - node-reset.sh left /etc/default/ovn-central behind. Raft membership lives in that file, not in the databases the reset deletes, so ovn-central recreated the database as a cluster member dialling a peer that had itself been wiped. - node-reset.sh also destroyed the installed firewall policy, which nothing regenerates. set-peers then failed on every attempt and the node stayed open permanently and silently. The arming mode is read before the wipe and the stage regenerated after it. - spinifex-firewall-apply rejected every verb when the policy file was missing, including disable. Turning the firewall off is the recovery path, so that left a node that could be neither armed nor disarmed. - viperblock exited 0 on a fatal startup error, so Restart=on-failure never fired. During formation its JetStream bucket has no leader yet, so the EBS provider died for good and every volume request failed with no responders. predastore and qemunbd had the same bug. - spx admin images import created image directories at 0700, so the first sudo import locked group members out of a tree deliberately made group-writable. Three of these hid behind checks that cannot succeed: ovn-sbctl show against a clustered database, and the OVN northbound and southbound waits in setup-ovn.sh. All three failed to stderr and let the run continue, so a check that could never pass was indistinguishable from one that was merely slow. The USB install guide claimed the ISO leaves Spinifex unconfigured and told the operator to run spx admin init. firstboot has always done that itself, so the guide now describes what the ISO actually produces and how to start using it. * fix: let spread placement launch across every node A spread placement group reserved four distinct nodes correctly and then failed the launch with ServerInternal, having started only two or three of them. The scheduler was never at fault; two faults downstream stopped the instances following the reservation. vpcd subscribes to vpc.add-nat with QueueSubscribe, which delivers one message at a time, and the handler holds its reply across the OVN flows barrier. The 45s caller deadline was sized against a single 30s barrier, which is right for one instance and wrong for the N a spread launch issues at once — the third request waits behind two full barriers and times out. That one subscription now overlaps its handlers under a bounded semaphore. Each barrier still syncs after its own write, so the guarantee is unchanged while N barriers collapse into roughly one. With the requests overlapping, every barrier still ran to its alarm. install-node.sh computed OVN_REMOTE after building the database nodes and gave it only to compute nodes, so the database nodes were configured without --ovn-remote. setup-ovn.sh writes ovn-northd's NB/SB connections from that list and skips it on the localhost default, and it already documents what follows: a northd dialling only its local member stops advancing nb_cfg once raft leadership moves, wedging ovn-nbctl --wait=hv. The list is knowable before any node is touched, so it is computed first and passed to the database nodes too. Nothing reported an OVN fault while this was happening. Every chassis was registered, ovn-controller and IPsec were up, and the raft cluster was healthy. The only signal was the barrier logging that it had overrun and was continuing without confirmation — which reads like slowness and was a permanent wedge. * feat: sweep block sizes in the disk benchmark and add an iperf3 network benchmark * fix: use iperf 2 with CSV output and report why a remote start failed * fix: stop pkill matching its own shell and verify the iperf server is listening * ci: run the predastore throughput benchmark at the end of the nightly * ci: boot the ISO in the nightly, single-node and three-node * ci: use the existing CI node password secret in the ISO cells * ci: report free space before the benchmark instead of failing inside warp * ci: keep the benchmarks and node disks off tmpfs * ci: export the work directories from RUNNER_TEMP rather than job env * ci: run the ISO and benchmark workloads from the nvme scratch area * perf: run guest NICs on vhost-net with multiqueue Guest-to-guest VPC throughput measured 1.68 Gbit/s single-stream against 21.8 Gbit/s host-to-host on the same underlay, and did not improve with more streams. QEMU's -netdev tap defaults vhost to off, so the whole datapath ran in the main loop thread in userspace: that thread sat at 59-65% CPU while all four vCPU threads idled at ~4%. Every tap netdev now carries vhost=on, and VPC ENIs additionally run multiqueue at min(vCPUs, 8) queue pairs. Multiqueue needs all three of the tap, the netdev and the device to agree, so TapNetDev and NetDevice derive the netdev options and the 2N+2 MSI-X vector count from one queue figure rather than leaving callers to keep them in step. The QMP hot-plug path takes the same options as cold boot, with a test comparing it against the rendered cold-boot strings so the two cannot drift. MMIO machines stay single-queue: they have no MSI-X. * ci: check out the sibling repos the go workspace needs in the ISO cells * ci: skip cell-21 workload cleanup when the workload never ran * ci: size the cell-21 node at 8 GiB and keep its serial log * ci: keep the cell-21 node diagnostics alongside the serial log * fix: let the daemon open /dev/vhost-net and delete multi_queue taps Enabling vhost=on broke instance launch two ways. DevicePolicy=closed gates the cgroup device list before capabilities are consulted, so QEMU's open of /dev/vhost-net returned EPERM despite CAP_DAC_OVERRIDE; add the device to the allow list and bump the unit version. Separately, `ip tuntap del` re-opens the device with TUNSETIFF and fails with EINVAL on a multi_queue tap, leaking it on every teardown; route deletion through `ip link del`, which is flag-agnostic. * perf: default guest NICs to a single queue Multiqueue measured a net loss. Spreading one flow across queue pairs reorders packets and TCP reads that as loss: a 4-vCPU guest fell from 1.76 to 1.50 Gbit/s, with cwnd collapsing on spurious retransmits. The host overlay funnels a node pair onto one softirq core, so the extra queues have no headroom to win back what the reordering costs. Keep the plumbing and its tests behind MultiqueueNICs so re-enabling is one line once that funnel is fixed. vhost-net stays on unconditionally; it moves the datapath off the QEMU main loop, measured at 60% of a core under load against 3% with it. * fix: raise the UDP receive buffer for predastore's QUIC transport Predastore carries blob, meta and raft over QUIC and quic-go asks for a 7MiB receive buffer. Left at the kernel default of 208KiB it silently dropped datagrams under load: the four-node cluster had shed 80k-125k each, which failed shard writes, put raft into repeated elections, and ultimately handed guests an I/O error that cost two of them their root filesystems. Adds a sysctl stage writing /etc/sysctl.d/99-spinifex-net.conf, so the setting lands in the ISO image and on every node push rather than living in a runbook. * ci: give the cell-21 workload an identity the node accepts * feat: tie the guest MTU and NIC multiqueue to ipsec_enabled ESP costs 34 bytes on top of geneve's 58, so a cluster running the overlay in plaintext can advertise 1442 instead of 1408 — worth roughly 10% of guest throughput. Derive it from network.ipsec_enabled rather than hardcoding the conservative figure. Multiqueue moves with the same flag, because its sign depends on it. Behind IPsec it costs ~12%: ESP has no L4 ports, so RSS funnels a node pair onto one core and extra queues only reorder packets, which TCP reads as loss. Without IPsec the sending host's single vhost thread saturates first and queues are worth +54%. IPsec on with multiqueue on is the worst of the four combinations, so it stops being a standalone knob and becomes vm.Deps.MultiqueueNICs. The reconciler only ever created a subnet's DHCPOptions row and never revisited it, so the create-time MTU stuck for the life of the subnet and this change would have applied to new subnets only. Worse in the other direction: turning IPsec back on would have left subnets advertising 1442 over ESP, which drops large segments with no ICMP to recover from. applySubnets now diffs and updates in place. reconcile.Config takes IPSecDisabled rather than IPSecEnabled so the zero value is the conservative MTU — a caller that forgets it is merely slower. * docs: add an install-time performance tuning guide Measured on the four-node cluster rather than inferred: the UDP receive buffer sysctl, what IPsec actually costs and why, the multiqueue sign flip, the overlay MTU budget, and the jumbo-frame check that must precede enabling it. A starter doc — several sections end in an open question rather than a recommendation. * ci: keep the cell-21 node alive for the workload step * ci: collect the cell-21 qemu log and widen its timeout * feat: derive the guest MTU from the fabric, and deepen the virtio NIC config The overlay overhead is passed on to the guest rather than hidden from it. Hiding it means running the fabric far enough above 1500 that a standard guest MTU fits in the headroom, which is what the large public clouds do — but they own the switches. On a fabric we do not control, the honest figure is the one the path can carry, and a jumbo underlay should reach the guest: 9000 gives 8942, or 8908 with IPsec. network.underlay_mtu is the lever; SubnetMTU spends it. Configured rather than probed, because a node's own NIC MTU says nothing about the switch between it and its peers. host_mtu sets VIRTIO_NET_F_MTU so a guest that never takes a DHCP lease — statically addressed, or IPv6-only — still learns the ceiling. rx/tx_queue_size raise the virtio ring from QEMU's default 256. That default is a packet-rate ceiling rather than a memory saving: a vCPU preempted for a fraction of a millisecond fills all 256 slots and the overflow is dropped. ethtool is the only way to read NIC offload, ring and channel state, so a node without it cannot be diagnosed for throughput at all. * ci: give the cell-21 workload its own guest key * fix: set the guest MTU on the tap, not just in the guest The tap was created and brought up without an MTU, so it stayed at the kernel default of 1500 while DHCP and host_mtu told the guest something else. Harmless at 1442; it silently blackholed every large frame on a jumbo underlay, with no ICMP back because the drop is at the first hop. TapSpec.MTU comes from the same guest overlay MTU the DHCP option and host_mtu already use, and is applied in the bring-up call so a size the fabric cannot carry fails the launch instead of degrading quietly. * fix: stop setting tx_queue_size, which QEMU caps at 256 for tap QEMU only honours a tx_queue_size above 256 for vhost-user and vhost-vdpa backends. On a tap it rejects the value outright rather than clamping it, so every launch died with "must be a power of 2 between 256 and 256" and the instance went straight to terminated. rx_queue_size is unaffected and stays at 1024. A test pins the absence of tx_queue_size across machine types and queue counts. * docs: lead the tuning guide with the overlay segmentation finding The guide's headline was a 5.4x that came almost entirely from turning IPsec off, and it treated MTU and the fabric as the main levers. The host reaches 92.6 Gbit/s at MTU 1500 on the same NIC, so the fabric is not the constraint and the encrypted default is not the thing to tune away from. Adds the host-versus-guest comparison up front, documents network.underlay_mtu, and makes the missing segmentation offload the first open question. * ci: stop deselected matrix cells failing in teardown * docs: quantify what disabling overlay IPsec is worth The guide described the trade without the numbers an operator needs to make it. Adds the measured 4.9x, notes that the MTU and multiqueue changes follow ipsec_enabled rather than being separate knobs, and records two transition hazards: a stale wide MTU on running guests, and RunInstances failing while the OVN flows barrier catches up with the new SAs. * ci: collect and retain the e2e benchmark and cluster logs Cells 21 and 22 both copied two paths that only ever exist on the nodes, so the benchmark JSON was never uploaded and cell-22 collected nothing at all. Both now gather what they actually produce, and keep it for 90 days. * fix: quieten spx by installing an explicit CLI logger default spx never set a process logger, so it inherited whatever a linked package left behind — until now a JSON handler at Info on stdout, interleaving library log records with command output and breaking piped use. Execute() claims the default before any command runs: text on stderr, LevelError, so commands report failures through their own stderr writes and libraries stay quiet. --verbose/-v raises to info and SPX_LOG_LEVEL sets an arbitrary level, the flag winning over the environment. Service subcommands still call initTelemetry, which replaces the handler outright. No root --debug flag: viperblock, qemunbd, nats and awsgw each define their own --debug with a service-specific meaning, and a root persistent flag would shadow inconsistently depending on the command. The guard test samples the handler at package-init time rather than inside the test, because tests in this package start services that install their own logger legitimately. * feat: private /admin/CreateAccount for self-service tenant signup Add POST /admin/<Method> to the AWS gateway: SigV4-signed, JSON in and JSON out, callable only by an IAM user in the super-admin account holding spinifex:<Method>. CreateAccount is the one method, so a signup form can provision a tenant without an operator shell. No new service, port, unit or certificate — it mounts inside the gateway's existing authenticated group and inherits its auth, throttling and tracing. Provisioning is resumable rather than transactional, which fits stores that have no transaction. Two CAS creates carry the correctness: the client token admits one in-flight attempt and makes a lost response replayable for 24 hours, and the name reservation makes duplicate account names impossible. Every step after the account exists is create-if-absent, so a retry with the same token finishes the job instead of failing. The CLI and the endpoint share one provisioning helper, which closes a gap on both at once: CreateAccount never checked names, so running `spx admin account create` twice produced two accounts that differed only by ID. The response waits for the account's default VPC over a new ec2.EnsureDefaultVpc request/reply, rather than trusting the existing fire-and-forget iam.account.created publish — credentials for an account that cannot launch anything are worse than a slower call. [signup] max_accounts bounds how many accounts may exist; absent means uncapped, which is what every cluster does today. * fix: stop the awsgw template hardcoding one deployment's console URL console_url shipped as https://console.spx3.com, so every fresh install — including an on-premise cluster that will never run signup — was configured to point new accounts at our console. It is the only deployment-specific domain in any template. It now ships empty, and the gateway warns when it provisions an account without one, since a blank sign-in link is otherwise visible only to the customer who receives it. Extract the remote client from the command body while here. The signing, request shape and error decoding are the part a Cloudflare Worker mirrors, and as a function they are testable against a TLS test server rather than only by hand against a live cluster. * refactor: drop console_url from signup and default max_accounts to 128 The gateway never read console_url — it passed the value straight through to the response, and the idempotency record froze it for 24 hours so a corrected typo still replayed stale. The client that made the request already knows where it wants to send the new tenant. max_accounts is now a pointer in the parsed config so an absent [signup] section takes the 128 default while an explicit 0 still means uncapped. * feat: run the predastore fault-injection harness as cell-24 Cell-23 asks how fast predastore is when every node is healthy; cell-24 asks whether it survives one that is not. It runs after cell-23 rather than beside it: both want a quiet single runner, and a benchmark sharing a machine with a fault injection run measures the wrong thing. The port offset keeps the two clear of each other even so, and the job clears stale nodes on entry and exit, since a run killed before its cleanup trap leaves one holding ports that fail the next run somewhere unrelated. * ci: keep the throughput benchmark's output with its results The run log only existed in the job log, so the artifact it uploaded could not be read without going back to the run that produced it. * ci: keep the fault-injection run log with its results events.txt carries the timeline and the FAIL line, but not the build output or anything Warp wrote before the harness took over, so a failed cell could only be read from the job log it was printed into. * fix: serve the cluster region to the console instead of hardcoding it The console hardcoded ap-southeast-2 as its SigV4 region in seven files, so on any other cluster the credential scope was wrong and the gateway rejected every signed request. The SPA is embedded in the binary, so a build-time value would pin one binary to one region. The UI server now serves GET /api/config with the region, no-store so a proxy fronting several clusters cannot serve one the other's region, and the SPA loads it once before render. The region comes from the serving node's config because awsgw pins signature verification to that same field. Reading the cluster-wide [aws] region instead would let a cluster whose two keys disagree pass review and still fail to authenticate. Also relabels the login card as Spinifex credentials, since the fields are not AWS ones, and hides the CA install banner unless the console is reached directly: behind a proxy with a publicly trusted certificate there is nothing for the cluster CA to fix. * fix: say once when the host firewall will not be maintained Both quiet outcomes of the reconcile — no cluster config, and the policy switched off — return nil without logging, so a node that never arms is indistinguishable in the journal from one that armed on its first pass. * test: arm the host firewall before asserting on its peer set Only the ISO install arms the policy; every other path writes off to the mode file, so the daemon disables the firewall and never writes a peer set. The test read that file as its baseline, which it could never have. It now switches the mode on for its own duration and puts it back, waits for the peer set the way install-node.sh does, and on failure reports the mode, the node count and what the daemon said rather than just the missing file. * test: move the console region test to the external cmd package It was in package cmd, which test-package-check rejects, so preflight failed on the branch. The hook follows the pattern already in export_test.go. * test: make the reboot EIP diagnostic capable of naming the drop The two ofproto/trace probes injected packets with no TTL, so OVN's router killed them at lr_in_ip_input and both traces ended early for a reason that had nothing to do with the failure under investigation. A keyword filter then stripped the table-match lines, hiding where each trace stopped. Traces now carry a TTL and are kept whole, the real datapath megaflows for the guest reply are dumped alongside them, and any drop matching the guest source is resolved from its OpenFlow cookie back to the Southbound logical flow that installed it. * fix: exempt an EIP instance from the subnet drop gate with allow, not reroute A subnet with no route of its own carries a priority-1100 drop policy, and an instance with an external IP is lifted above it by a priority-1200 policy. That policy rerouted to the pool gateway, which in distributed NAT is outside every logical router port subnet, because the gateway LRP is the link-local /30. The static default route survives that by naming an output port; a routing policy has no such field, so ovn-northd discarded all three of them and the drop gate below swallowed everything the instances sent. The effect was that every external IP answered inbound traffic and none of it came back: the SYN reached the guest, the guest replied, and the reply died in the router's policy stage. Guest-initiated egress was lost the same way. The policy only ever needed to lift the gate. The router's default route already points at the gateway, so allow leaves the routing decision intact and gives northd nothing to reject. * test: match the log the firewall quorum gate actually emits The assertion waited for "chassis encap addresses; not writing a partial peer set", which no code has ever logged. The reconcile reports a short chassis list as "OVN reports N of M chassis encap addresses" and the maintain loop carries that into its retry warning, so the wait could only ever time out. Arming the node in the previous commit got the test as far as this assertion for the first time, which is why it surfaced now. The failure also prints the journal tail and the same inputs the earlier wait reports. * fix: stop one unreadable volume document failing every volume listing A single volume document whose shards no longer reconstruct made DescribeVolumes, DescribeVolumeStatus and DescribeVolumesModifications fail for every account, because the listing returned the first fetch error rather than skipping the document. ListVolumes now skips a document it cannot read or decode and logs the skip, mirroring ListAMIs. Callers whose answer would be wrong rather than merely partial take ListVolumesStrict: the orphan scan and the snapshot in-use check both gate destructive work on a complete list. The tolerance covers fetch failures as well as decode failures. An object whose shards no longer join is as unusable as one whose bytes will not parse, and it was the fetch path that took the cluster down.	2026-08-15 14:22:49 -05:00
Josh Siviour	91d22d9f12	feat(security): add an nftables host firewall scoping cluster ports to peers (#769) ... * feat(security): add an nftables host firewall scoping cluster ports to peers Nodes shipped no host firewall, so every cluster listener — OVN NB/SB, the formation port, NATS, predastore — was reachable from anything that could route to the node. On a single-NIC node the planes collapse onto the public address, which made that the default rather than the exception. setup.sh installs a policy and an apply helper but does not enable them: the peer set is unknown until the cluster's planes resolve, and a drop policy with an empty set would cut the first node that applied it out of its own formation. install-node.sh writes the peers and enables the unit after formation. The policy owns only table inet spinifex_filter and replaces it in one transaction. vpcd writes MASQUERADE and per-EIP FORWARD rules into the ip nat/filter tables and reinstalls them only on start, so a whole-ruleset flush — what Debian's nftables.service does on both start and stop — would break elastic IPs silently. INPUT only, for the same reason: a forward-hook drop here could not be rescued by an accept in the other table. Guest traffic keeps working: IMDS and VPC DNS terminate on ime-* ports in the host netns, and every guest agent reaches its control plane over the AWS gateway rather than NATS, so no cluster port needs to face the guests. * feat: reconcile the host firewall peer set from the daemon The peer set was written by install-node.sh and by cluster-bootstrap.yml. That missed two of the four formation paths (bm-bootstrap.sh and the ISO firstboot), duplicated the same logic in each, and went stale as soon as a node was added or replaced. host.ReconcileFirewall now owns it: every formation path starts the daemon, and it re-runs on each startup. A node's own config names its peers by their wan address only, so a set built from config alone drops every peer's NATS, OVSDB and formation traffic. The lan plane comes from the generated NATS cluster routes and the OVN remote lists, and the vpc plane from the Southbound Encap table; an empty encap answer is an error rather than a narrower set. ProtectSystem=full applies to the sudo child too, so the root helper could not write /etc/spinifex/firewall either — it is now in the daemon's ReadWritePaths, optional so a node installed before the firewall stage still starts. The apply helper gains set-peers and disable verbs, re-validates every address as a dotted quad with in-range octets, and rolls the peer file back if the ruleset will not load. * fix: retry the host firewall reconcile until every chassis registers The peer set is only safe to write once OVN reports an encap address for every node. On a fresh bootstrap the daemon routinely wins the race against ovn-controller, so a single startup attempt saw a partial chassis list and either wrote a set that dropped Geneve from the nodes missing from it, or failed outright and left the node with no policy until a restart. Gate on the full chassis count and schedule the reconcile instead of running it once: back off from 15s to 5m while it fails, then re-run every 5m so a node added or replaced after formation is picked up without a daemon restart. An unchanged peer set short-circuits before the helper runs. * test: cover the firewall chassis quorum gate in the multinode e2e suite A partial chassis list is the normal state early in a bootstrap, and writing a peer set from one produces a firewall that drops Geneve from every node missing from it. The gate that refuses that write had no live coverage: a 5-node bootstrap converges too fast for any daemon to observe a partial list, so the verification run only ever exercised the OVN-unreachable path. Reach the branch by inflating the expected count instead of breaking OVN: a phantom node in one node's config makes the cluster one chassis short, with nothing in the Southbound DB disturbed. Asserts the gate logs, that no peer file is written while it holds, that the loaded ruleset survives, and that the node recovers to a byte-identical peer set once the phantom is removed.	2026-08-12 22:37:27 +10:00
Josh Siviour	5c6429bde1	fix(ipsec): stop charon on nodes that do not use OVN IPsec (#768) ... The daemon only called EnableOVNIPSec when network.ipsec_enabled was true, so the disabled case was never reached and nothing ever turned the host's IPsec services off. Since setup.sh installs openvswitch-ipsec and strongswan-charon unconditionally and the packages enable their units, charon listened on 0.0.0.0:500 and 0.0.0.0:4500 on every node -- including single-node clusters, where EnableOVNIPSec short-circuits and IPsec is never switched on at all. ReconcileOVNIPSec runs on every startup and resolves the state both ways. It self-heals: a single-node box that later joins an IPsec-on cluster gets the services back through the same call, which a static install-time mask could not do because setup.sh runs before the topology is known. The unit changes go through a new fixed-verb root helper rather than systemctl. spinifex-daemon holds three narrow sudo grants and none of them is systemctl, which is deliberate: a NOPASSWD systemctl rule takes an arbitrary unit name and is root-equivalent. The helper takes only on|off and names the units itself, matching the existing spinifex-set-endpoint-sysctl pattern. Only the read-only is-enabled and is-active probes are run directly, and a test pins that. strongswan-starter.service is disabled in both states. ovs-monitor-ipsec execs the strongSwan starter itself rather than going through that unit, so it only ever competes for UDP 500/4500 -- confirmed on a live IPsec-enabled cluster, where charon's parent is /usr/lib/ipsec/starter and the unit is inactive. A nil cluster config leaves the services alone rather than guessing. This is the bind-address half only. On a multi-node cluster charon must run, and it still binds the wildcard; restricting that reach to the encap plane is the host firewall's job.	2026-08-12 12:48:08 +10:00
Julian Sommer	2e09461080	chore: cleanup code comments	2026-08-04 14:20:19 +10:00
Ben Duncan	b76aab8b50	feat(installer): three-plane networking, ZFS root, distributed NAT and multi-node install (#663)	2026-08-04 10:21:12 +10:00
Josh Siviour	a2c451128d	fix: unblock the Ubuntu 26.04 installer (qemu-kvm, sudo-rs) (#665) ... setup.sh aborts twice on resolute. qemu-kvm is a virtual package with two providers there, so apt refuses to resolve it; the token was redundant anyway since $QEMU_PACKAGES already installs qemu-system-x86/arm and spinifex execs the qemu-system-* binaries directly. sudo-rs, Ubuntu's default sudo since 25.10, then rejects the sudoers file: it implements no pam_session Defaults setting, and forbids a wildcard inside a command argument. pam_session is now emitted only for classic sudo, and the sysctl grant names a fixed-verb helper instead of a wildcard. The helper is narrower than the rule it replaces: it accepts one interface, a key restricted to rp_filter or accept_local, and a value restricted to 0 or 1, where the old grant permitted any net.ipv4.conf key at any value. Granting sysctl with an unconstrained argument was rejected as root-equivalent. Adds scripts/lint-apt-packages.sh and an Apt Packages workflow so the runtime package list is resolved against each supported suite's real indices in CI.	2026-08-03 08:57:50 +10:00
Josh Siviour	dad5d702e1	fix: stop lbagent heartbeats on positive LB deletion, and actually enable the JetStream watchdog (#643) ... * fix(lbagent): stop heartbeating a load balancer once it is confirmed deleted A LoadBalancerNotFound response is the gateway's positive confirmation that the store has no record for the LB, not a connectivity failure — unlike a dial error, timeout, or 5xx, which leave its existence unknown and must keep retrying. Past the startup registration grace window (or after any prior successful heartbeat), it can no longer be the record's own replication race, so the agent now stops rather than retrying forever and filling the log with heartbeat noise after teardown. * fix(nats): enable the JetStream ENOSPC-latch watchdog timer nats-server disables JetStream process-wide on ENOSPC (server/jetstream.go handleOutOfSpace -> ShutdownJetStream) and never re-enables it itself — that decision is nats-server's own internal state, not spinifex's, so a restart is the only way back. The spinifex-nats-watchdog.timer + service already probe /healthz?js-enabled-only=true and try-restart on a sustained failure, but the timer's WantedBy=timers.target never took effect: neither setup.sh nor firstboot enabled it, so the watchdog shipped inert and a full disk needed a manual restart forever. Enable --now it alongside the rest of the unit set.	2026-08-02 08:21:46 +10:00
Josh Siviour	9f5f877e50	fix: batched hygiene — slog key, ui pid file, race-scaled timeouts, GPU console capture (#637) ... * fix(ec2): rename ambiguous 'code' slog key to firmwarePath The EFI variable store prep log used 'code' for the OVMF firmware path, which reads as an error/status code in log search. Rename to firmwarePath to avoid confusion. * fix(spinifexui): write pid file to a service-owned BaseDir spinifexui.Start() called the legacy utils.WritePidFile, which has no MkdirAll and falls back to $HOME/spinifex. Under the spinifex-ui unit's ProtectHome=yes this directory is neither guaranteed to exist nor writable. Add Config.BaseDir, switch to utils.WritePidFileTo (which does MkdirAll) for Start/Stop/Status, wire it through the CLI as a spinifex-ui-base-dir flag/SPINIFEX_UI_BASE_DIR env var, provision /var/lib/spinifex/spinifex-ui in setup.sh, and add it to the unit's ReadWritePaths. * test(viperblockd): replace fixed sleep with a readiness poll TestEBSConfigQueueGroup_DetachedOpenFails slept 500ms before assuming launchService's ebs.config subscriber was bound, then raced a hardcoded 5s request deadline. Poll with a cheap malformed request (rejected at json.Unmarshal, before the slow S3 path) until the subscriber answers, then issue the real request with room for openVolumeVB's S3 backoff. * fix(e2e): capture GPU worker console log on test failure Wire harness.OnFailure into TestEKSGPUPodExposure to dump the GPU worker's serial console via the established EC2 GetConsoleOutput path, matching lb_test.go's captureLBConsoleOnFailure. A driver or device-plugin hang after boot only ever surfaces on the guest console.	2026-08-02 08:17:18 +10:00
Josh Siviour	628fbbc0c3	fix: group-scope the OVS/OVN sockets and retire the vpcd sudoers grants (#609) ... * fix: group-scope the OVS/OVN sockets and stop sudoing read-only probes The OVS DB socket and both sets of ctl sockets were chmod 0666 by setup-ovn.sh, with a systemd override re-applying it on every openvswitch-switch restart. Write access to db.sock is full control of the local datapath, so this handed the datapath to any account on the box. Both service users run with Group=spinifex, so the access is now 0660 root:spinifex, /var/run/ovn is 0750, and a shared helper re-applies it from ExecStartPost on openvswitch-switch and ovn-controller alike — the latter is new, and matters because the ovn-controller ctl socket name embeds the pid and is recreated on every start. That also fixes the health probe reporting ovn-controller as not_running on a healthy node. It shelled out to `sudo ovn-appctl`, which only spinifex-vpcd is granted; as spinifex-daemon sudo demanded a password and the swallowed error read as OVN being down. With the ctl socket group-owned the probe needs no privilege, so health.go moves to a non-escalating Runner rather than the alternative of granting the daemon a root-equivalent ovn-appctl rule to read a status. ListIMDSTaps ran `list-ports` plus one `get` per patch port on vpcd's 15s tick, so 1+N sudo invocations every tick, each one logging a pam_limits warning about the core rlimit the unit deliberately pins to 0. It now issues a single JSON query per pass, and the sudoers drop-in disables the PAM session for both service users, which also leaves sudo children inheriting the unit's rlimits. An absent iface-id on a patch port is now skipped with a warning instead of aborting the pass: it is the only record of the full ENI, so such a port cannot join the live set either way, and failing the pass stalled reconcile for every healthy port behind it. A transient query failure still aborts. * fix: group-own the OVS/OVN pidfiles too, and cover the ipsec ctl socket Group-owning the ctl sockets was not enough for `ovn-appctl -t ovn-controller`: it resolves the pid-named socket through the pidfile, and OVS opens that O_RDWR to test the fcntl liveness lock, so a group-readable pidfile still fails the probe with EACCES. The pidfiles now get the same 0660 root:spinifex, which also tightens them from the 0644 they ship with. openvswitch-ipsec joins the override list because setup-ovn.sh restarts it after the sweep runs, leaving its ctl socket as the one file still at the shipped mode. Verified on the dev node: every socket and pidfile is 0660 root:spinifex and survives a restart of all three units, an unprivileged user is refused, and /health reports ovn-controller ok where the sudo probe read not_running. * refactor: stop escalating the OVS/OVN socket clients Now that setup-ovn.sh group-owns the control sockets, ovs-vsctl, ovs-appctl, ovn-nbctl, ovn-sbctl, ovn-appctl and systemctl reach their daemons as the service user. utils.NeedsPrivilege carries that policy and both command builders consult it, so ip, iptables, dhcpcd, sysctl and arping still escalate while the socket clients no longer do. Their sudoers grants are removed: each took unrestricted args and all of them accept --log-file=PATH, so every one was a root-equivalent grant handed out to run a query. ovs-ofctl keeps its grant. It talks to a per-bridge <bridge>.mgmt socket that ovs-vswitchd creates when the bridge appears, including bridges spinifex creates at runtime, so the provisioning sweep cannot group-own it up front. The NB and SB database sockets join the sweep for ovn-nbctl and ovn-sbctl. vpcd carried its own sudoCommand copy that escalated unconditionally, bypassing the policy entirely. It survived only because the grants existed and broke the OVN flows-ready barrier the moment they were removed, so it now delegates to utils.SudoCommand. TestRG10_SudoOnlyThroughThePolicy fails the build if another package reintroduces a local copy. Verified on the dev node: an instance launches, IMDS is served, the flows-ready barrier completes, and no sudo denial appears anywhere in the journal. * refactor: drop every spinifex-vpcd sudoers grant for ambient capabilities vpcd held NOPASSWD grants for ip, iptables, sysctl and arping. The ip grant alone was root-equivalent: sudoers cannot constrain its arguments, so `sudo ip netns exec <ns> /bin/sh` was a root shell for the service account. The unit already grants CAP_NET_ADMIN and CAP_NET_RAW ambiently, and the kernel passes an ambient set to each child on exec, so those tools work as the service user with no escalation at all. Verified on a live node: all sixteen ip subcommands vpcd issues, plus iptables reads and writes, sysctl -w on the net.ipv4.neigh tree, and arping, all succeed under exactly the unit's ambient set. spinifex-vpcd now has no sudoers rules. NeedsPrivilege had to become capability-aware to do this. spx is one binary shared by both service units, and the daemon holds CAP_SYS_ADMIN and CAP_DAC_OVERRIDE but no CAP_NET_ADMIN, so a static tool list cannot express "vpcd runs ip directly, the daemon escalates it". The decision now reads CapAmb from /proc/self/status once and matches it against the capability each invocation needs. sysctl is resolved per key, since CAP_NET_ADMIN covers the net.* trees and nothing else. The dhcpcd grants go too: neither daemon invokes it. Only setup-ovn.sh, which runs as the provisioning user, and the in-guest ecs-agent. TestRG10_VPCDHoldsNoSudoersGrant pins both halves together, so a future grant cannot be re-added without also explaining why the capability stopped working. * chore: restore routeTree.gen.ts swept in by mistake * fix: probe ovn-controller with ovn-appctl, not ovs-appctl checkOVNController's first probe passed a bare -t ovn-controller to ovs-appctl, which resolves it against OVS_RUNDIR — /var/run/openvswitch. ovn-controller's socket and pidfile are in /var/run/ovn, so the probe could never succeed and logged "ovn-controller.pid: open: No such file or directory" on every check. Not a functional break: the next probe globs the full /var/run/ovn path and succeeds, so the function still returned nil. It cost an exec per check and made a healthy node look like it was failing an OVN liveness probe. ovn-appctl defaults OVS_RUNDIR to /var/run/ovn, so the common case now matches first. The glob and systemctl fallbacks stay for older layouts.	2026-07-30 17:33:24 +10:00
Julian Sommer	e79ca59d4b	fix: configure host DNS from spx admin init/join (#582)	2026-07-24 10:02:11 +10:00
Julian Sommer	1d55404a73	revert: host DNS setup on resolved/resolvconf hosts (#580) ... This reverts commit f9c9f0b38d. #580 switched setup.sh to read the northstar :53 listener and zones from northstar.toml, so the resolved stage now defers at install time (before formation) and must be re-run afterward. Only firstboot re-runs it; the manual setup.sh + spx admin init install and CI never do, so those hosts get no host DNS and cluster-zone names (ELB/EKS) fail to resolve from the node. Reverting restores the pre-#580 behaviour: systemd-resolved is configured at install time from the WAN IP, with no post-formation re-run needed. This reintroduces the original gap where resolvconf hosts get no host DNS. The proper fix configures host DNS from spx admin init/join for both resolver managers (mulga-x30yv).	2026-07-23 11:23:21 +10:00
Julian Sommer	f9c9f0b38d	fix: configure host DNS for Spinifex zones on resolved and resolvconf hosts (#580)	2026-07-23 09:43:38 +10:00
Ben Duncan	28a876d17d	feat(dns): resolvable ALB/NLB + EKS DNS, forwarder hardening, cross-tenant reconcile (#532)	2026-07-17 12:02:59 +10:00
Josh Siviour	e1c4f85a36	feat: routed-NAT external mode with two-tier ingress (#475) ... * feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper * feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode * feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules) * feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode * feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks * feat(scripts): add --nat-uplink routed mode to setup-ovn.sh * test(e2e): add routed-NAT single-node natuplink suite Covers the external_mode=nat lane end to end on a nat-initialized node: host transit wiring (veth, br-ext port, ip_forward, iptables egress rules), spinifex.toml contents, disabled EIP surface (empty describe-addresses, UnsupportedOperation on allocate), default subnet MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit /24, guest outbound egress proven via a serial-console probe (nat mode has no inbound path), and unique transit gateway IPs across VPCs. Suite self-gates on external_mode=nat; registered as e2e-natuplink and in SUITES_BUILD so it compiles in CI. Manifest entry stays commented until CI can provision a nat-mode environment. * feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column Adds the exempted_ext_ips column (optional strong Address_Set ref) to the nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an in-place SetNATExemptedExtIPs update on the client interface, live client, and mock. Test NB schema gains the column and covers insert-resolves-UUID, converge, clear, and missing-NAT paths. * feat(policy): exempt address set on routed SNAT/EIP rows Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR) with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip path patch the ref in place on pre-existing rows. The set is a singleton strong ref and is never deleted. Non-routed modes are unchanged. * feat(host): VPC ingress route primitives for routed-NAT mode EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the spx-nat veth. Replace/del semantics keep both idempotent; a missing route on removal is not an error. * feat(external): routed ingress leg on IGW attach/detach, reconcile-safe AttachIGW's routed leg now also installs the host VPC ingress route via RoutedIngressHooks, and re-runs on the already-attached path so the reconcile loop restores host routes (and patches legacy SNAT exempt refs) after a reboot — OVN state survives restarts but host routes do not. DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs, validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct pass nil hooks and are unchanged. * test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row pointing at spinifex_nat_exempt (containing the transit /24), and — after opening the SG from the transit net (default-closed, AWS parity) — ping and an sshd TCP handshake against the instance private IP from the host, which proves the un-SNATted return path. Phase 8 additionally waits for the second VPC's ingress route to appear on IGW attach. vpc-networking README documents the jumpbox pattern, SG note, and nat_exempt_cidrs. * feat(config,admin): public wan pool alongside nat-transit Restructure admin config generation from single-pool scalars to a Pools slice so nat mode can emit both the transit pool and an optional public wan pool. nat mode now accepts --external-pool/--external-source (plus --external-bind-bridge/--gateway-ip when a public pool is requested), validated by the shared resolvePublicPoolFlags helper also used by pool mode. Transit pool always renders first so the IGW allocator picks it. Formation wire format unchanged (nat mode is single-node only). * feat(daemon): enable EIP service and auto-assign in nat mode with public pool External IPAM (and with it the real EIP service, RunInstances public IP auto-assign, and MapPublicIpOnLaunch on default subnets) now activates whenever public pools exist: pool mode as before, plus nat mode when a wan pool rides alongside the transit segment. The transit pool is filtered out of IPAM — its addresses are gateway-LRP plumbing, never allocatable EIPs. Transit-only nat configs keep the disabled EIP stub. * feat(host): per-EIP ingress primitives for routed-NAT mode DetectUplinkFor parses ip route get to find the interface and source IP facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move). EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC gateway LRP transit IP (src-hinted so host-local traffic still DNATs), answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC, WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated. * feat(policy,vpcd): host EIP binder option on NATManager WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds after priming on both the fresh-add and idempotent-skip paths (host routes and proxy-ARP are volatile even when the OVN row survives, so reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort. The next hop comes from the new gatewayPortIP helper reading the gateway LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID key now an exported policy const shared with the external package. vpcd selects the IGW pool by name instead of index — with two pools in nat mode the gateway allocator must stay on the transit pool — and wires the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced pools (gateway unknown until lease) proxy-ARP on their bind bridge. * feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection dhcp_mac = "derived" (default) | "interface" on external pools. The interface variant leases with the bind interface's own MAC — WiFi/WWAN uplinks drop frames with foreign source MACs, so derived chaddr MACs never complete DORA there. Option 61 stays the lease discriminator. UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and persists through KV so restart re-acquires repeat it. Routers that key leases on MAC and ignore client-id would hand every lease the same IP; the manager detects an ACK matching the interface's own address or another client-id's live lease, releases the colliding lease, and hard-errors advising a static range instead. admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp pools bound to non-bridgeable uplinks (wlan/wwan/ppp). * test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs The natuplink suite now detects a public (non-transit) pool in spinifex.toml and switches lanes: Tier-1-only configs keep the EIP-disabled and no-public-IP assertions; public-pool configs assert the EIP surface is enabled, the default subnet has MapPublicIpOnLaunch=true, instances auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink, per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the EIP, reconcile replay across a vpcd restart, and teardown on disassociate. Docs: vpc-networking README gains a public-pool section for routed-NAT nodes (init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool. * fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd Routed-NAT mode shells out to iptables for the transit masquerade and per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for host-delivered EIPs. Without these sudoers entries vpcd crash-loops with "command not allowed" on nat-mode nodes. * fix(ovn): scope DeleteStaticRoute prefix match to the target router Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache match could grab another router's row and fail with a referential integrity violation, wedging IGW attach on multi-VPC nodes. Match only rows referenced by the target router, mirroring FindStaticRoute. Mock updated to the same semantics, with a regression test covering drift replacement across two routers. * fix(reconcile): always replay AttachIGW so host ingress state is re-ensured applyIGWs skipped AttachIGW whenever the gateway switch port existed in actual state, so the already-attached re-ensure path never ran and routed-NAT host ingress routes lost on reboot were never reinstalled. AttachIGW is idempotent; run it unconditionally. * fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes collide on the single host route table and the last IGW attach silently stole the route (route replace). Pass vpcID and the gateway LRP IP through RoutedIngressHooks; the bootstrap VPC always installs its route, other VPCs only install when the CIDR is free and only remove a route their own gateway IP holds. Adds host.VPCIngressRouteVia to read the current route holder. * ci(e2e): add nightly cell 19 for routed-NAT single-node suite Cell 19 bootstraps with --external-mode=nat plus the CI static pool as the public wan pool and runs the natuplink suite, covering the Tier-1 transit datapath and Tier-2 EIP ingress. The run step exits 0 when natuplink.test is absent from the tests tar so scheduled runs against branches predating the suite stay green. * fix(admin): nat-mode init summary reflects configured public pool The summary printed 'outbound-only — no public IPs/EIPs' even when --external-pool/--external-source configured a public wan pool. Print the pool range (or dhcp bind) and note EIPs are enabled. * fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites Reconcile the routed-NAT branch with dev's ctx-first service APIs after rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService interface, the vm_adapters public-IP resolver asserts the ctx-signed AssociatedPublicIPForInstance, and the default-subnet mapping test threads ctx into DescribeSubnets.	2026-07-09 09:43:32 +10:00
Julian Sommer	e11437cec9	feat(imds): per-tap responder datapath (#408)	2026-06-23 16:44:53 +10:00
Josh Siviour	c22a6f4ffc	fix: install chrony on host via curl|bash setup path (#366) ... The host clock needs an NTP daemon; without one it drifts and SigV4 request signatures fall outside the skew window, kicking the web console back to /login. Mirrors the ISO rootfs (iso-builder packages.list) so both host-install paths ship the same software. Guest VMs track the host via kvm-clock. Bug: mulga-siv-335	2026-06-16 16:03:34 +10:00
Bryn Mailer	f6f3956952	feat: enable viperblock at-rest encryption by default on fresh installs (#323) ... * feat: enable viperblock at-rest encryption by default on fresh installs Fresh installs left nodes.<node>.viperblock.encryption_key_file empty, which viperblockd treats as legacy cleartext mode. The encryption machinery existed but the install path never generated a key or set the field. Provision a cluster-wide viperblock master key in all three fresh-install flows and render its path into spinifex.toml: - admin init (single node): generate the key locally. - admin init --multi-node (leader): generate once, save locally, and distribute via the formation server (mirrors IAM master key distribution). - admin join (joiner): receive the shared key from the leader rather than minting its own, so volumes sealed on any node open everywhere. Lenient on absence — an older leader returns no key, so the joiner warns and stays cleartext instead of failing the join. The key file is omitted from the template when unset, so upgrades with no existing key keep running cleartext. * fix: make viperblock encryption key group-readable for both readers The key is loaded by two services per node — viperblockd (spinifex-viperblock) and the awsgw EC2 handlers (spinifex-gw), both in the spinifex group — so it must be group-readable, not owner-only like the per-node predastore key. The previous commit generated the key but never registered its path with the deployment ownership machinery, leaving /etc/spinifex/viperblock as root:root 0700 (non-traversable) and the key as root:root 0600. The daemon failed at startup with "stat ... permission denied". Mirror master.key: setup.sh creates /etc/spinifex/viperblock (0750), and SetServiceOwnership chowns the dir to spinifex-viperblock and promotes the key to root:spinifex 0640. * fix: read viperblock config.json through StateBody for encrypted volumes The metadata layer parsed config.json with json.Unmarshal/NewDecoder directly, which fails on encrypted volumes/AMIs now that viperblock wraps VBState in a valid-JSON envelope ({"v":1,"payload":...,"authtag":...}). DescribeImages silently skipped encrypted AMIs (count=0 → "no AMIs available"), bricking the whole launch path. Route every config.json read through viperblock.StateBody, which unwraps the envelope (no key needed — the metadata is authenticated, not encrypted) and is a no-op for unencrypted volumes: - image: DescribeImages, getVolumeConfig, amiNameExists, GetAMIConfig - volume: getVolumeConfigAndEncryption, and mergeVolumeConfig (so the refuse-to-merge-encrypted guard sees the real VBState instead of clobbering the envelope with a tag-less blob) - admin: readAMIConfig, readVolumeConfig	2026-06-15 09:31:23 +10:00
Josh Siviour	2ea3e22db6	feat(systemd): cgroup-confine spinifex to a resource-limited slice + host swap (#324) ... * feat(systemd): bound spinifex to a resource-limited slice Add a spinifex.slice parent ceiling (MemoryHigh=80%, MemoryMax=90%, MemorySwapMax=0) with a spinifex-system.slice child holding the seven control/infra services, so a runaway hits the slice instead of the host. A system.slice drop-in (MemoryMin=1G, CPUWeight=300) reserves RAM and CPU for sshd/journald/the operator, keeping the box reachable when spinifex is maxed. The drop-in is generated in setup.sh install_systemd() rather than shipped as a staged file, since the packaging globs flatten the systemd/ dir; this mirrors the existing sshd-keygen drop-in pattern. The two .slice files flow through every staging path unchanged. Within the single effective cgroup, a slice MemoryMax OOM picks the highest oom_score victim — guest QEMU (+500) before any service (-500) — so a slice OOM degrades by killing one guest while the daemon, storage tier, NATS, and the host all survive. * feat(setup): provision 8G host swapfile + vm.swappiness=10 Add a setup_swap stage that creates an 8G /swapfile, enables it, persists it to fstab, and writes vm.swappiness=10 so the kernel only swaps under real pressure. Backs spinifex.slice MemorySwapMax so the slice can spill to swap for graceful degradation instead of OOMing a guest. Idempotent; skipped under ISO_BUILD (firstboot provisions swap on ISO hosts). Reverses the prior swap=0 assumption.	2026-06-09 15:21:13 +10:00
joshsiv-mulga	41f840ee85	fix(setup): grant spinifex-vpcd sudo for ovn-appctl (siv-167) ... The W2 inject-garp fix landed in 72a6a59d adds a shellout to `ovn-appctl -t ovn-controller inject-garp <port>` via the L0 execRunner, which prefixes `sudo` when not running as root. spinifex-vpcd runs as user spinifex-vpcd, so every emission was hitting `sudo: a password is required` and silently returning a WARN that nobody read until W5 diag capture surfaced the journal. Net: W2 has been a complete no-op since it shipped. Every recycled-EIP scenario has continued to leave stale upstream ARP. Evidence: diag bundle from run 26813235295 — 12/12 GARP attempts denied by sudo, 0 successful emissions; pcap shows runner ARP for recycled EIP .232 stuck on the prior VPC's gw chassisredirect MAC (02:75:d9:ef:d8:4b) while OVN NB has handed .232 to a new VPC with a different gw MAC; pcap also shows runner sending 174 SYNs to recycled EIP .233 with 0 SYN+ACKs received. Fix: add /usr/bin/ovn-appctl to the spinifex-vpcd NOPASSWD grant. Trivial; ovn-appctl is already trusted via the existing NB/SB read-write grants. Follow-ups (separate change): tighten host.InjectGARP error classification so sudo-deny is loud (Error or fatal) rather than silently warn-and-continue; add a boot-time self-check that verifies sudo permissions for every host-tool the daemon will shell out to.	2026-06-02 21:00:20 +10:00
tomnewton-mulga	9d8597317a	Feat/mig capable (#271) ... * feat(gpu): add NVIDIA MIG multi-instance GPU support Partitions MIG-capable GPUs (A100, A30, H100) into isolated slices exposed as mdev devices instead of whole-GPU VFIO passthrough. VMs attach via -device vfio-pci,sysfsdev=<mdev-path> using the GPUAttachment discriminated union (PCIAddress for whole-GPU, MdevPath for MIG). Key changes: - gpu: new MIGProfile/MIGInstance types, IsMIGCapable model map, mig.go ops (EnableMIGMode, DisableMIGMode, CreateInstances, ListInstances, DestroyAllInstances, enrichMdevPaths) - gpu.Manager: Claim() returns (*GPUDevice, *MIGInstance, error); AddMIGInstances, ReclaimByMdev for restart recovery; MarkMIGFailed per-slice - daemon: buildGPUPool helper partitions devices into whole-GPU vs MIG pools; MIGProfile config field selects profile per node or per GPU override - vm.VM: GPUAttachments []gpu.GPUAttachment replaces GPUPCIAddresses []string - admin CLI: spx admin gpu mig status/enable/disable subcommands with admin.SetMIGProfile for spinifex.toml writes - docs: COMMANDS.md updated with three new MIG management command rows * test(gpu): unit tests for MIG parsers and pool manager Cover all pure parsing functions (parseMIGMemory, isCapacityExhaustedError, extractProfileFields/GIFields, parseMIGProfiles, parseMIGInstances, parseCreatedGIID/CIID) with realistic nvidia-smi fixture output. Manager MIG tests cover AddMIGInstances, Claim/Release on mdev-backed entries, MarkMIGFailed, ReclaimByMdev, and interaction with whole-GPU MarkFailed — all using fake mdev directories in t.TempDir() so no GPU hardware is required. * feat(gpu): dynamic MIG profile-per-request with on-demand GI/CI lifecycle Replaces static config-driven MIG pre-carving with a dynamic model where the MIG profile is specified per RunInstances call via mig.<profile> instance type names (e.g. mig.1g.10gb) that mirror nvidia-smi nomenclature directly. - Manager.Claim(instanceID, profileName): fast path claims a pre-carved slice; slow path dequeues a free MIG GPU, releases the lock, calls nvidia-smi mig -cgi/-cci to carve all slices, then reclaims the lock to commit and return the first slice - Manager.Release: last release on a dynamically-carved GPU calls DestroyAllInstances and returns the GPU to freeMIGGPUs, allowing a different profile on the next claim - AddMIGGPU(dev): registers a fresh MIG GPU for dynamic allocation; AddMIGInstances (restart recovery) keeps its no-destroy semantics - GenerateMIGTypes: produces mig.<profile> InstanceTypeInfo entries from gpu.ListProfiles at daemon startup; IsMIGType/MIGProfileFromType let callers inspect these names - buildGPUPool: calls ListInstances for recovery, AddMIGGPU for fresh GPUs, ListProfiles to collect MIG profile specs for type generation; removes setupMIGInstances and migProfileForDevice - GPUClaimer.Claim gains profileName param; daemonGPUClaimer, service_impl call sites, and fakeGPUClaimer all updated * fix(gpu): correct mdev parent detection and make --profile optional for mig enable enrichMdevPaths was using os.Readlink on "parent_bus_0" (a non-existent sysfs attribute) and then on ".." (a directory, not a symlink) — both would fail on real NVIDIA hardware, breaking CreateInstances and ListInstances on daemon restart recovery. Fix: read the mdev UUID's own symlink target, which always encodes the parent GPU's PCI address in its path (e.g. .../0000:01:00.0/<uuid>). Make basePath injectable in enrichMdevPaths so the fix is covered by a new set of unit tests using temp-dir symlinks (TestEnrichMdevPaths_*). Remove the MarkFlagRequired("profile") from "spx admin gpu mig enable": Phase 2 is fully dynamic — the daemon carves slices on demand at RunInstances time. Without --profile, the command just enables MIG mode and destroys any pre-carved slices so the daemon starts with a free GPU. With --profile, behaviour is unchanged (pre-carve a static pool). * chore: writeups for smci and arcee * test: add coverage for MIG exec, manager slow-path, instancetypes and admin - Add mig_exec_test.go with fake nvidia-smi harness covering Enable/Disable MIGMode, IsMIGModeEnabled, DestroyAllInstances, ListProfiles, CreateInstances, ListInstances - Extend manager_mig_test.go with slow-path Claim/Release tests using fake nvidia-smi - Add generate_test.go coverage for IsMIGType, MIGProfileFromType, GenerateMIGTypes - Add admin_test.go coverage for SetMIGProfile (create, update, idempotent, error paths) - Make mdevBasePath a var to allow sysfs injection in tests - Add json tags to GPUAttachment fields to satisfy musttag linter Diff coverage: 75.6% (threshold: 60%). * fixed gpu passthrough doc * Revert "fixed gpu passthrough doc" This reverts commit 1e20f5ab13. * feat: Added RTX 6000 GPU MIG support and fixed docs * feat: expose GPU/MIG inventory in GetNodes and GetVMs admin API Add Manager.Snapshot() for lock-safe pool inspection. Extend NodeStatusResponse with GPUs []GPUInfo (per-physical-GPU inventory with per-slice detail for MIG) and VMInfo with GPU *VMGPUInfo (model, VRAM, profile, mdev path for the attached GPU). Populate both in handleNodeStatus and handleNodeVMs via buildGPUInventory, buildPoolLookup, and resolveVMGPU helpers. * feat: surface GPU info on EC2 instance detail page (mulga-new-03) Admin users see a GPU card (model, VRAM, attachment type, MIG profile/mdev path or PCI address) populated from the GetVMs admin API. Adds formatVRAMMiB helper and VMGPUInfo/GPUInfo/GPUSliceInfo types to the frontend. * feat: GPU columns + inventory card on Nodes page (mulga-new-04) Adds GPU column to VMsTable (model/VRAM or MIG profile) and ResourceTable (alloc/total GPUs). Section 4 GPU Inventory card shows per-node GPU state with per-slice MIG drill-down. Also adds total_gpus/alloc_gpus to NodeInfo type. * feat: grouped GPU-aware instance type picker in Run Instances (mulga-new-05) Splits the flat instance type dropdown into Standard and GPU-Accelerated groups. GPU items show total VRAM alongside the type name. Selecting a GPU type surfaces an info line with GPU name and VRAM. * testing image builder scripts * fixed nvidia build script * more descriptive errors for image build script * more fixes * more fixes * fix: surface PCI address in whole-GPU VMGPUInfo and rebuild frontend dist Add PCIAddress to VMGPUInfo, populate it in resolveVMGPU for whole-GPU passthrough, and fix the instance detail page which was incorrectly showing instance type in the PCI address field. Rebuilt frontend dist to include GPU UI changes (inventory card, resource columns, GPU-aware instance type picker). * fix: improve kernel version detection in nvidia image build script Fall back to /lib/modules when /boot/vmlinuz-* is absent, and add diagnostic output to help debug kernel detection failures. * fixing nvidia image build script * fix ami name issue * correct kernel versions * another fix * image build works, added docker and nvidia toolkit for QoL * fix: add gpg * fixing amd image builder * add universe * another fix * nvm * fix docker user group issues * make rocm land where it is supposed to * fix docker groups * feat: add Ubuntu 26.04 GPU AMI catalog entries and fix build script parted detection Add ubuntu-26.04-nvidia-gpu-x86_64 and ubuntu-26.04-amd-gpu-x86_64 to AvailableImages, hosted on iso.mulgadc.com/system-ami/ with SHA256 checksums. Fix build-system-image.sh to detect parted in /usr/sbin when not on PATH. Suppress gosec false positives in DHCP manager (cancel stored in leaseLoop) and promisc ioctl (standard syscall uintptr pattern). * Updated passthrough doc regarding GPU images * fix: remove orphaned nolint directives (golangci-lint v2.12.2 no longer flags G115/G118) * chore: remove stale spinifex@192.168.3.107 directory Scripts were accidentally committed with a node hostname as the path prefix during initial testing. Correct copies live in scripts/images/.	2026-06-01 13:48:13 +10:00
Josh Siviour	fd3dc6b680	feat(network): ADR-0006 layer redesign + IPsec/OVN hardening (#255) ... * refactor(vpcd): remove macvlan bridge mode Macvlan was a single-NIC SSH-safe hack that predated veth mode and was never selected by any active config path. Deletion is Phase 0 item 1 of the network layer redesign (mulga-siv-125.1). Removed: - BridgeModeMacvlan const + setMacvlanMAC + ifaceIsMacvlan probe - Macvlan MAC alignment block in launchService (only fired on dead path) - Macvlan branch in detectBridgeMode + matching tests - Macvlan duplicates of IGW/AddNAT centralized-NAT tests (veth versions already cover the same path) - --macvlan flag + macvlan case branch in setup-ovn.sh Refactored: - useCentralizedNAT() now == BridgeModeVeth (was != BridgeModeDirect); verifyBridgeMode already rejects empty/unknown at startup - Three IGW-pool tests now explicitly WithBridgeMode(BridgeModeVeth) (previously relied on empty-mode defaulting to centralized) Kept intentionally: - Deprecation notes in docs/compute/vpc-networking/README.md - Legacy macvlan-interface cleanup in reset-dev-env.sh + ansible teardown role (migration for operator systems with stale spx-ext-* interfaces from pre-veth installs) * feat(vpcd): resolve external CIDR at startup with bounded retry Block service start until the WAN bridge has an IPv4 address, with a 30s retry window. Phase 0 item 2 of the network layer redesign (mulga-siv-125.1). Guards the boot race where vpcd starts before systemd-networkd finishes DHCP on br-wan, or before netplan applies a static config. Skipped when ExternalMode is empty (overlay-only deployments). Forward-compatible with ADR-0006: the new resolveExternalCIDR / externalCIDRFromBridge helpers become the backing implementation of HostWiring.ExternalCIDR() once the network/host package lands in Phase 1. Until then the resolved value is only used to fail-start on missing uplink. * fix(vpcd): cover external CIDR startup paths Extract launchService's CIDR-resolve block to ensureExternalCIDRReady helper so it's unit-testable, and add direct tests for externalCIDRFromBridge (loopback + unknown-interface) to exercise the original literal body. Closes the diff-coverage gap on Phase 0 commit 15b564d which left these paths unreachable from unit tests. * refactor(network/ovn): extract L1 OVN client to network/ovn package Move OVNClient interface, LiveOVNClient, ACLSpec, ErrNATNotFound and namedUUID helper from services/vpcd to a new spinifex/network/ovn package. The mock relocates to spinifex/network/ovn/mock as a sub-package. Storage fields on the mock become exported so vpcd tests can continue to seed/inspect state across the new package boundary. services/vpcd retains thin type-alias and var-alias shims so existing callers compile unchanged; Phase 2 removes those shims as vpcd is torn down. Adds -coverpkg=./spinifex/... to test-cover so cross-package test exercise of mock methods continues to count toward coverage now that the mock lives outside vpcd. First sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md (bead mulga-siv-125.2). Pure refactor; no functional change. All existing tests pass; preflight green. * feat(network/topology): TopologyManager interface + vpcd adapter Add spinifex/network/topology with Manager interface, VPC/Subnet/Port specs, and OVN object naming helpers covering the L2 stable contract (§7 of the redesign plan). Phase 1 keeps existing in-tree OVN names (rtr-, ext-port-, sg_) and notes the §7.3 targets for a later reconciliation migration. TopologyHandler in services/vpcd grows EnsureVPC / DeleteVPC / EnsureSubnet / DeleteSubnet / EnsurePort / DeletePort / SetPortSecurityGroups methods that mirror the corresponding NATS handlers' OVN ops with Spec inputs instead of nats.Msg envelopes, satisfying topology.Manager via a compile-time check. Existing handlers are unchanged; the duplication is short-lived (Phase 2 collapses them). Smoke tests in topology_manager_test.go cover the idempotent create path, delete cascade, and declarative SG membership move. Deliverable: TopologyManager.EnsureVPC/Subnet/Port callable; all existing tests green. Second sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md (bead mulga-siv-125.2). * feat(network/host): L0 HostWiring interface + physical/veth impls Phase 1 deliverable for mulga-siv-125.2. Introduces network/host (L0 of ADR-0006): UplinkMode enum, Wiring interface, and Physical/Veth implementations with injectable Runner + InterfaceReader for testability. Not yet wired into daemon/vpcd — Phase 2 replaces daemon.SetupComputeNode and vpcd's resolveExternalCIDR with calls through this package. * feat(network/policy): L3 SecurityGroup/NAT/Route managers Introduce network/policy as the data-plane policy layer (ADR-0006 L3). Three managers, all backed by network/ovn: - SecurityGroupManager: clear+add ACL replacement on an existing port group; port group lifecycle stays with L2 (topology.Manager). - NATManager: bound to one NATMode at construction (derived from host.UplinkMode). Distributed sets logical_port/external_mac on EIP dnat_and_snat rules; centralised leaves them unset. Per-EIP, NAT GW, and IGW-default SNAT entrypoints; all deletes idempotent. - RouteManager: AddStaticRoute is read-then-decide so retries no longer duplicate rows (closing the long-standing non-idempotency bug in vpcd/topology.go's IGW default-route path). ACL priority constants and infrastructure ACL builders move from vpcd/{security,acl}.go into policy/acl.go with their semantics preserved (default-deny 900/800 logged, DHCP-allow 1050, tenant 1000). No vpcd wiring changes here — additive only; the reconciler (bead .3.2) and network/external (bead .3.4) will consume these managers. bead: mulga-siv-125.3.1 * feat(network/external): L5 IGW/EIP/NATGW managers Pure additive — new package, no topology.Manager changes, no rewiring of existing vpcd handlers. IGWManager builds the external switch + localnet + gateway LRP + default route + chassis HA, with a flows barrier callers can inject (production: ovn-nbctl --wait=hv sync). EIP/NATGW are thin L5 wrappers over policy.NATManager. GatewayIPAllocator interface lets the DHCP-backed allocator stay in vpcd until the DHCP-mgr removal bead lands; LinkLocalAllocator + StaticRangeAllocator ship here. * feat(network/reconcile): L6 intent-actual reconciler replaces 5-pass startup Build spinifex/spinifex/network/reconcile/ package per Phase 2.2 of the network layer redesign. Reads desired state from NATS KV filtered by vpc.AZ (with legacy empty-AZ matches-all rule), scans actual state from OVN NB DB, diffs and applies with VPC → Subnet → Port → SG → IGW → EIP/NATGW topological ordering. Atomic port-group join via CreateLogicalSwitchPortInGroups closes the racy gap the legacy scanENIPortMembership pass used to revive 30s later. A 5-minute drift goroutine (leader-gated via JetStream KV CAS) replaces the old 30s ReconcileSGsLoop. Adds AZ field to VPCRecord stamped at create time. Empty AZ on pre-Phase 2.2 records matches every local AZ, so no migration pass is required. Replaces and removes the 5-pass startup sequence in services/vpcd: - Reconcile + ReconcileFromKV - ReconcileSGsOnce + ReconcileSGsLoop - RetrofitAllExternalLocalnetOptions - RetrofitAllGatewayPortNetworks IGW fresh-create continues to flow through the legacy NATS handler path (handleIGWAttach) so the dhcpManager wiring stays intact until Phase 2.3; the reconciler only rebinds gateway-chassis on existing IGW topology. Closes mulga-siv-125.3.2. * refactor(network/topology): promote live Manager impl to network/topology Move the topology.Manager implementation out of services/vpcd/topology_manager.go into network/topology/manager_live.go as a standalone liveManager constructed via NewLiveManager(ovn.Client, opts...). Inputs that varied per-handler (DNS server resolution from the external pool) are now passed as functional options (WithDNSServer). TopologyHandler still satisfies topology.Manager but its methods delegate to a cached liveManager. The shim stays until the rest of mulga-siv-129 routes NATS subscribers through topology.Manager directly. Also exports SubnetGatewayCIDR + FormatDNSServerList from network/topology so the reconciler in network/reconcile can share helpers with the live manager (audit pass to follow per plan §6). Test cases ported one-for-one from services/vpcd/topology_manager_test.go to network/topology/manager_live_test.go; the vpcd-side test file is deleted. New WithDNSServer test asserts the option propagates into DHCPOptions rows. Refs mulga-siv-129 (Phase 2.6 vpcd subscriber migration, Slice 1 of N). * refactor(vpcd): collapse SG subscribers onto network/{topology,policy} managers Phase 2.6 / mulga-siv-129 slice 2. Rewrites the vpc.create-sg / vpc.update-sg / vpc.delete-sg NATS subscribers in services/vpcd as thin wrappers over topology.Manager (port group lifecycle) and policy.SecurityGroupManager (ACL programming). Deletes the ~1000 lines of duplicated ACL builder and provisioning logic — its authoritative copy lives in network/policy. Surface additions: - ovn.ErrPortGroupNotFound sentinel; mock + live GetPortGroup wrap not-found via fmt.Errorf("%w: ...") so callers can use errors.Is. - topology.Manager grows EnsureSGPortGroup / DeleteSGPortGroup (resolves the apply.go TODO from §2.1 of the redesign plan). liveManager implements them with GetPortGroup-then-Create / GetPortGroup-then-Clear+Delete idempotency. Behaviour delta: the old provisionSG rolled back the OVN port group on AddACLs failure ("all-or-nothing"). The new path lets the port group survive and relies on network/reconcile to retry EnsureSG on its next sweep. The all-or-nothing fail-fast test cases are dropped; ACL emission semantics are covered by network/policy/sg_test.go. * refactor(vpcd): migrate VPC/Subnet/ENI subscribers to topology.Manager Phase 2.6 / mulga-siv-129 slice 3. Rewrites vpc.create / vpc.delete / vpc.create-subnet / vpc.delete-subnet / vpc.create-port / vpc.delete-port / vpc.update-port-sgs NATS subscribers as thin wrappers (≤30 lines each) over topology.Manager. Subscriber bodies now: parse event → translate to spec → delegate to manager. Deletes: - vpcd.subnetGateway (and helpers_test.go + topology_test.go's TestSubnetGateway block); parity coverage lives in network/topology.TestSubnetGatewayCIDR. - vpcd.reconcilePortSGs (the live manager already exposes the same diff-then-transact via reconcilePortSGs on the L2 side). - vpcd.TopologyHandler.dnsServer; lmDNSServer (passed via topology.WithDNSServer to the live manager) is the live path. Behaviour delta: handlers now strict-parse PrivateIpAddress / MacAddress / CidrBlock at the NATS boundary and reject malformed payloads instead of degrading to "no DHCP options". The reconciler guarantees that field validity is upheld upstream of the publish. * refactor(network/reconcile): drive SG port groups via topology.Manager applySGs now calls topology.Manager.EnsureSGPortGroup / DeleteSGPortGroupByName instead of reaching directly into the OVN client. Reconcile.Config gains a Topology field (required); the production caller in services/vpcd passes the TopologyHandler shim, the reconcile package tests construct a fresh topology.NewLiveManager bound to the mock OVN client. DeleteSGPortGroupByName is added to the topology.Manager interface for the reconciler's orphan-removal path, which iterates raw OVN port-group names without a groupID round-trip. The existing DeleteSGPortGroup(groupID) delegates to it. Resolves the TODO at apply.go:137 carried over from slice 2 of mulga-siv-129. * feat(network/policy): add FlowsBarrier option to NATManager NATManager.AddEIP needs to block on `ovn-nbctl --wait=hv sync` after writing the dnat_and_snat row so the NATS reply only returns once every chassis has the rewrite flow. The legacy vpc.add-nat handler does this at the tail of handleAddNAT; the manager has lacked it since slice 2.1. NewNATManager grows a functional-options tail (Option / WithFlowsBarrier). Defaults to a no-op for tests; production wiring in services/vpcd passes waitForFlowsHV. AddEIP now invokes the barrier after AddNAT, logging at warn level on failure to match the legacy handler's _ = waitForFlowsHV() swallowing semantics. Pre-work for slice 4 of mulga-siv-129 — without the barrier the NAT EIP subscriber migration would lose the post-write flow-install gate. * refactor(vpcd): migrate NAT GW subscribers to policy.NATManager handleAddNATGateway / handleDeleteNATGateway now decode the NATS event and delegate to policy.NATManager.{AddNATGateway,DeleteNATGateway}. The manager carries identical OVN logic — same snat type, same ExternalIDs (spinifex:vpc_id + spinifex:nat_gateway_id) — so existing TestTopologyHandler_AddNATGateway / DeleteNATGateway suites stay green. Lazy-constructed NATManager added on TopologyHandler (natmOnce). Mode is derived from the bridge mode (veth → centralized, direct → distributed) and the FlowsBarrier is bound to waitForFlowsHV for parity with the legacy EIP path's tail barrier (used by the next slice, vpc.add-nat). Slice 4 of mulga-siv-129, sub-step b. * refactor(vpcd): migrate EIP NAT subscribers to policy.NATManager handleAddNAT / handleDeleteNAT now decode the NATS event and delegate to policy.NATManager.{AddEIP,DeleteEIP}. The behavioural contract is preserved end-to-end: - Idempotency-skip on a matching dnat_and_snat row (same VPC + LogicalIP, plus MAC + LogicalPort under distributed NAT) moves into AddEIP so duplicate vpc.add-nat publishes never tear down the rule the previous invocation just installed (mulga-siv-124). - Stale-rule cleanup across every router stays in AddEIP. - Post-write waitForFlowsHV barrier is now driven by the manager via the FlowsBarrier option wired in services/vpcd (slice 4a). - DeleteEIP returns nil on ErrNATNotFound, matching the handler's fire-and-forget "already absent = success" semantic. The vpcd "errors" import drops out with the inline error-comparison. Slice 4 of mulga-siv-129, sub-step c. * refactor(vpcd): migrate IGW subscribers to external.IGWManager (partial) handleIGWAttach now branches at decode time: - Centralised NAT + DHCP-sourced pool keeps using attachIGWLegacy because the gateway-LRP IP comes from vpc.dhcp.acquire, and that DHCP client is the vpcd-local manager that bead mulga-siv-125.3.3 will retire. - All other shapes (distributed NAT, static-range pool, no pool) delegate to external.IGWManager.AttachIGW. The manager mirrors every step the legacy handler ran — external switch + localnet port + gateway LRP + switch-side router peer + default route + chassis HA + waitForFlowsHV tail barrier — so observable behaviour is unchanged. handleIGWDetach drops its inline OVN walk entirely; both paths exit through external.IGWManager.DetachIGW followed by releaseGatewayLRPLease. The lease release stays in vpcd because the upstream client lives in services/vpcd/dhcp; calling it after every detach is safe because the upstream server expires unknown leases on its own. Lazy-constructed IGWManager added on TopologyHandler (igwmOnce). Pool, allocator (StaticRangeAllocator), chassis, NAT mode, NAT manager, route manager, and FlowsBarrier are all derived from the existing TopologyHandler state. The DHCP-coupled legacy branch retains its full inline OVN logic; it will disappear in slice-after-125.3.3 follow-on. Slice 4 of mulga-siv-129, sub-step d. * feat(ec2/instance): honor AssociatePublicIpAddress per-NIC override RunInstances --network-interfaces "AssociatePublicIpAddress=true|false" was silently ignored, only subnet.MapPublicIpOnLaunch gated public IP assignment. AWS contract is per-NIC override wins. Extend TestPrepareRunInstances_PublicIPAutoAssigned to table-driven 3 cases: subnet true no-override, subnet false override true, subnet true override false. * refactor(config): drop upstream-DHCP config surface (static WAN pool only) Hard-error in LoadConfig on legacy keys: [network] external_dhcp, [nodes.<n>.vpcd] dhcp_bind_bridge, and [[network.external_pools]] source = "dhcp". Add pool range validation (must fit inside the gateway/prefix_len CIDR; per-cluster non-overlap). Strip ExternalDHCP from config.NetworkConfig + formation.NetworkConfig + admin.ConfigSettings, and DhcpBindBridge from config.VPCDConfig + admin.ConfigSettings + vpcd.Config. vpcd.ExternalPoolConfig.IsDHCP() short-circuits to false; the field and downstream DHCP branches in external_ipam / topology.go remain for Slice C to delete together with dhcp_manager. cmd/spinifex/cmd/admin.go: drop useExternalDHCP path and the detectedDhcpBindBridge helper; require --external-pool when --external-mode=pool; auto-detect default no longer infers source=dhcp. service.go's wan_bridge legacy-key rejection now advises removing the key entirely (vpcd auto-detects br-wan), not renaming to dhcp_bind_bridge. Two topology_test cases that exercise the DHCP gateway-LRP path are skipped — Slice C deletes attachIGWLegacy and these tests together. * refactor(vpcd): delete dhcp_manager + DHCP gateway-LRP path (Slice C) Removes upstream-DHCP-client gateway-LRP allocation in favor of static WAN-pool allocation (StaticRangeAllocator). Part of bead mulga-siv-125.3.3. - delete services/vpcd/dhcp_manager.go + dhcp/ subpackage (~865 lines) - delete allocateGatewayLRPIPViaDHCP + releaseGatewayLRPLease paths - drop ExternalPoolConfig.Source + DhcpBindBridge + IsDHCP() fields - drop ExternalIPAllocation lease fields (Expiry/ServerID/HWAddr) and ExternalIPAMRecord.Source - drop NewExternalIPAM nc *nats.Conn argument; update all 4 call sites - rename verifyBridgeMode local dhcpBindBridge -> wanBridge; update error strings and tests Slice D (delete attachIGWLegacy + dispatcher) intentionally deferred. * refactor(vpcd): delete attachIGWLegacy + IsDHCP stub (Slice D) Closes mulga-siv-129 slice 4d. Part of bead mulga-siv-125.3.3. handleIGWAttach now unconditionally delegates to external.IGWManager. attachIGWLegacy (~207 lines, ~5 OVN write steps + chassis scheduling) deleted along with its dispatcher branch in handleIGWAttach. The ExternalPoolConfig.IsDHCP() dead stub is gone. No code paths or NATS subjects regress: the static-pool / distributed branches were already going through external.IGWManager since Slice 4d (6719dc3). * feat(ipam): Purpose field on every IPAM record (Slice B) Part of bead mulga-siv-125.3.3. Adds a Purpose enum (handlers/ec2/vpc/purpose.go) tagging every IP allocation with its semantic role (eni-primary, eni-public, eip, igw-lrp, natgw-external, …) so multi-VPC clusters can reclaim/audit by (owner, purpose). External IPAM (spinifex-external-ipam, v1→v2): - ExternalIPAllocation.Type → .Purpose; legacy values mapped via LegacyExternalTypeToPurpose (gateway→igw-lrp, auto_assign→eni-public, elastic_ip→eip). Unknown legacy values warn and become "unknown". - AllocateIP / AllocateFromPool / allocateFromPool param rename allocType→purpose. All call sites pass Purpose* constants. - migrate/002_external_ipam_purpose.go: idempotent CAS-retry migration with v1/v2-union record shape so partial migrations resume cleanly. Internal IPAM (spinifex-vpc-ipam, v1→v2): - IPAMRecord.Allocated []string → []IPEntry{IP, Purpose, OwnerID}. - AllocateIP gains purpose+ownerID args; ReleaseIP unchanged (still keyed on IP). - AllocatedIPs returns []IPEntry. - eni.go reorders eniId generation before AllocateIP so the owning resource ID lands on the new entry. - migrate/003_ipam_purpose.go: cross-bucket backfill that reads the ENI bucket via the new RunKVWithJetStream path. Orphaned IPs (no owning ENI) warn and become Purpose=unknown. Migrate framework: - KVContext gains an optional JetStream field; new Registry.RunKVWithJetStream method threads it through. RunKV signature unchanged — all other callers are unaffected. Tests: - ipam_test.go: AllocateIP signature updates; AllocatedIPs now asserts IP / Purpose / OwnerID on the returned entries. - external_ipam_test.go: replaces "auto_assign"/"elastic_ip" literals with PurposeENIPublic/PurposeEIP and asserts alloc.Purpose. - migrate/002 + 003 tests: schema-detect, idempotent rerun, no-JetStream fallback, empty bucket. * refactor(daemon): tombstone obsolete spinifex-dhcp-leases KV bucket The dhcp_manager removed in Phase 2.3 (bead mulga-siv-125.3.3) owned the spinifex-dhcp-leases KV bucket. The migrate framework only handles per-bucket schema version bumps, not bucket deletion, so the obsolete bucket would linger on upgraded clusters. Add a one-shot tombstone after initJetStream that deletes the bucket if present. DeleteKVBucketIfExists treats ErrBucketNotFound and ErrStreamNotFound as success, so the call is idempotent across restarts. * feat(config): add network.ipsec_enabled cluster flag (default true) Add IPSecEnabled to NetworkConfig in both the cluster TOML schema and the formation handshake payload. Default true so AWS-parity intra-AZ encryption is on by default. Operators can flip false in [network] for trusted single-rack lab topologies. Phase 2.5 (bead mulga-siv-125.3.5) consumes the flag from admin init (strongSwan provisioning), formation server (propagates to joiners), and daemon startup (ovs-vsctl ipsec_encapsulation toggle). * feat(admin): IPsec peer cert generator signed by cluster CA Add GenerateIPSecPeerCert + IPSecCertPaths in spinifex/admin so admin init (and the future admin add-node path) can issue per-node IKEv2 peer credentials signed by the existing cluster CA. The cert carries id-kp-ipsecIKE (RFC 4945) so strongSwan accepts it as an IKEv2 peer credential when ovs-monitor-ipsec materialises tunnel configs. Cluster CA is reused directly — no intermediate strongSwan PKI today; when intermediate CAs land that becomes a separate workstream. The cert/key pair lives under {configDir}/ipsec/peer.{pem,key}; the daemon points OVS Open_vSwitch.other_config (certificate, private_key, ca_cert) at these paths when network.ipsec_enabled is true. * feat(admin): wire IPsec peer cert into admin init + join paths Add --ipsec bool flag (default true) on adminInitCmd. Single-node init and the multi-node leader path both emit an IPsec peer cert after the existing CA+server-cert generation. The multi-node formation server propagates the IPSecEnabled flag to joining nodes via NetworkConfig; admin join honours it and emits its own peer cert during the same cert-issuance step that already produces the server cert. spinifex.toml template gains an unconditional [network] block carrying ipsec_enabled so the value is visible to operators on every install regardless of whether external networking is configured. The daemon reads it via cluster config (NetworkConfig.IPSecEnabled). Bead mulga-siv-125.3.5. * build: install openvswitch-ipsec for OVN native IPsec openvswitch-ipsec provides ovs-monitor-ipsec, which reads the Open_vSwitch.other_config cert/key/CA pointers and materialises strongSwan configs for every Geneve tunnel managed by ovn-controller. Pulls in strongswan + strongswan-pki as transitive deps. Required by bead mulga-siv-125.3.5. * feat(daemon): enable OVN native IPsec on intra-AZ Geneve at startup When network.ipsec_enabled is true (default), the daemon points the local OVS DB other_config at the per-node IPsec peer cert that admin init / admin join provisioned (configDir/ipsec/peer.{pem,key} + the cluster CA at configDir/ca.pem), then sets other_config:ipsec_encapsulation=true. ovs-monitor-ipsec (shipped by the openvswitch-ipsec package) consumes those pointers and materialises strongSwan configs for every Geneve tunnel ovn-controller programs — including OVN-IC inter-AZ tunnels when federation lands in Phase 3a. Missing cert files log a warning and leave Geneve plaintext; operators upgrading an older cluster reissue certs via spx admin to close the gap. Bead mulga-siv-125.3.5. * test(e2e): multinode IPsec wiring + xfrm SAs + ESP capture TestMultinodeIPSec asserts: - OVS DB on every node carries certificate/private_key/ca_cert pointers and other_config:ipsec_encapsulation=true (what daemon.enableOVNIPSec writes). - ip xfrm state on every node has at least one AEAD SA in AES-GCM mode (rfc4106(gcm(aes)) or gcm(aes)) within 90s. - tcpdump catches ESP (ip proto 50) on the underlay (best-effort — logged only; geneve may be idle during the 5s capture window). Skips cleanly when the cluster was bootstrapped with --ipsec=false (no ipsec_encapsulation key on first node). Read-only over SSH; runs in the parallel bucket alongside the other non-mutating multinode tests. * fix(igw): write attachment State="attached" for AWS parity Reconciler intent loader (network/reconcile/intent.go::loadIGWs) filters IGW records with `State == "attached"`, but AttachInternetGateway was writing `State = "available"`. The reconciler therefore never picked up any attached IGW — `intent_igws=0` on every pass — so the OVN external switch + gateway LRP + WAN bridge wiring built by IGWManager.AttachIGW never ran for default-VPC IGWs (whose live `vpc.igw-attach` publish at daemon bootstrap is lost — vpcd subscriber not yet wired). Net effect on single-node e2e: every SSH probe to 192.168.x public IPs timed out (TestSSHProbe, TestModifyInstanceAttribute, TestRebootInstance, TestNATGateway, TestSGToSGDatapath) because SNAT/DNAT rules on the VPC router had no path to the external network. Match AWS InternetGatewayAttachment.State semantics: - Attach: State="attached" - Detach: State="detached" Detach value is internal — recordToEC2 gates Attachments on VpcId != "", so the detached State is never surfaced to API consumers. Bead: mulga-siv-131 * fix(reconcile): call AttachIGW when IGW topology missing applyIGWs previously deferred to the NATS handler when an intent IGW had no external switch in OVN. When the vpc.igw-attach event is lost (bootstrap-time default-VPC IGW published before vpcd subscriber is ready), deferral never resolves and VM public-IP traffic dies. Call IGWManager.AttachIGW directly — it is idempotent (short-circuits when the external switch already exists), so the reconciler converges default-VPC IGWs on first pass without duplicating work on subsequent passes. Closes mulga-siv-132. * fix(test): align NewExternalIPAM call with 2-arg signature Rebase onto dev pulled in daemon_system_instance_test.go which used the legacy 3-arg NewExternalIPAM(jsNC, js, pools). The feature branch already narrowed the signature to (js, pools) via the upstream-DHCP config drop; align the test with the current signature. * fix(vpcd): EnsureSubnet must EnsureVPC first to avoid NATS race vpc.create-vpc and vpc.create-subnet are independent NATS events with no inter-event ordering guarantee. Tenant bootstrap on iam.account.created publishes both within <1ms, so the subnet handler's CreateLogicalRouterPort can race the parent VPC's CreateLogicalRouter and fail with "logical router not found". The cascade fails CreateNetworkInterface which fails RunInstances which surfaces as InsufficientInstanceCapacity to the SDK — masking the underlying topology race. handleSubnetCreate now calls EnsureVPC(VPCID-only) before EnsureSubnet. EnsureVPC short-circuits on existing LR (idempotent), so the race-winning vpc.create-vpc handler's full CIDR/VNI ExternalIDs are preserved; if subnet wins the race the LR is created with bare ExternalIDs (cosmetic) and the follow-on vpc.create-vpc short-circuits. Adds TestIntegration_SubnetBeforeVPC regression covering both orderings. Closes mulga-siv-133. * feat(ansible): require --external-pool, drop NAT, forward EXTRA_VARS spx admin init now rejects --external-mode=nat and demands --external-pool when mode=pool. The ansible init role silently skipped pool flags when the range was unset, so installs fell over inside the spx validator. Make targets also swallowed `-e` (make's env-override flag), so operators could not override pool start/end without editing defaults.yml. - vars/defaults.yml: rename nat_gateway_ip → external_gateway_ip; add spinifex_external_no_external opt-out; document the spx flag contract. - roles/init/tasks/main.yml: drop NAT branch; assert pool range upfront with a fail_msg pointing at EXTRA_VARS / defaults.yml; emit gateway, prefix-len, gateway-ip only when explicitly set (let spx auto-detect); honour --no-external. - playbooks/dev-reset.yml: rename captured var; coerce legacy external_mode=nat to pool with a warn; tighten gateway regex with \b. - Makefile: ansible-dev-* targets now forward EXTRA_VARS="key=val ..." through --extra-vars so `make` callers can override without editing defaults.yml. - scripts/ansible/README.md: document pool requirement, both invocation styles (direct -e vs make EXTRA_VARS), NAT removal. * fix(ipsec): guard ipsec_encapsulation on ovs-monitor-ipsec liveness enableOVNIPSec set OVS cert pointers + ipsec_encapsulation=true without ensuring ovs-monitor-ipsec.service was running. Without that daemon, OVS holds the cert pointers unused, strongSwan never gets IKE config, and tunnel traffic ovn-controller marks for encryption gets silently dropped. TestMultinodeIPSec fails on `node1 xfrm has no AEAD SAs`. - systemd: spinifex-daemon Wants=/After= ovs-monitor-ipsec.service - daemon: ensureOVSMonitorIPSecActive — systemctl enable --now + is-active poll; refuses to flip ipsec_encapsulation if daemon is dead - daemon: single-node short-circuit — no peers means no tunnels means the flag is at best inert and at worst load-bearing-silent-drop - ci: capture-net-diag.sh + workflow steps to snapshot OVS/OVN/xfrm state on failure, fan out across all 3 multinode nodes Refs: mulga-siv-136 * test(e2e): sticky-skip downstream SSH tests when probe fails When TestSSHProbe times out (3 min, typically because of mulga-siv-134 datapath bug), four downstream tests each re-probe SSH on the same broken datapath and time out identically — single-node job runs ~24 min instead of ~12 min, with the same error logged five times. First waitForSSHReady timeout now sets a package-level atomic flag; subsequent waitForSSHReady / trySSHReady / requireSSHHealthy calls skip immediately. Preserves the original red signal on TestSSHProbe and on any test that genuinely needed SSH (Skip ≠ Pass). Bool variant (trySSHReady) consults but does not set the flag — its caller already treats False as advisory warning, not fatal. TestSGToSGDatapath has its own EventuallyErr SSH wait (not via waitForSSHReady) and needs an explicit requireSSHHealthy(t) gate at the top of runSGToSGDatapath. TestNATGateway already calls waitForSSHReady first so the gate fires naturally. Refs: mulga-siv-137 * fix(ipsec): use Debian unit name openvswitch-ipsec.service CI artifact shows `Unit ovs-monitor-ipsec.service could not be found` on every node. Debian's openvswitch-ipsec package ships the unit as openvswitch-ipsec.service (the binary inside is ovs-monitor-ipsec, but systemd's name follows the package). The previous commit guarded on the wrong unit name, so: - spinifex-daemon.service Wants=/After= silently no-op'd (missing unit) - enableOVNIPSec's `systemctl enable --now ovs-monitor-ipsec.service` failed, daemon logged a warning and continued without flipping ipsec_encapsulation=true - TestMultinodeIPSec skipped with `node1 reports ""` because OVS DB never received the cert pointers Rename across all 5 spots (daemon, systemd unit, test, CI diag script, workflow journal capture) to match the Debian package layout. Note: the package itself still must be present on the CI VM image — that's tracked in the mulga repo (image-builder provision.sh + bootstrap-install.sh shim). Refs: mulga-siv-136 * fix(e2e): authorize default SG tcp/22 + icmp in single-node fixture admin init leaves the default SG closed (AWS parity); the Go single suite never ported the Phase-5 SG-ingress block that run-e2e.sh has had since day one, so every test runner SSH/ping probe hits the OVN port-group ACL drop. - ec2helpers: add AuthorizeICMPIngress + EnsureDefaultSGOpen helpers. - single/main_test: open default SG once per process in pkgFixOnce.Do. - witness: extend ensureDefaultSGSSHIngress to authorize ICMP too. - run-e2e.sh: append matching ICMP authorize next to tcp/22 block. mulga-siv-134 * fix(network/reconcile): key DHCP_Options lookup by subnet_id not CIDR Multi-tenant default VPCs share the same subnet CIDR (172.31.0.0/20) per account. The existing FindDHCPOptionsByCIDR gate at apply.go:107 found the first account's DHCP_Options row and skipped creation for every subsequent subnet with the same CIDR. Those subnets' ports then attached nothing (FindDHCPOptionsByExternalID misses on subnet_id), VMs got no DHCPv4 lease, and SSH/ICMP into them silently failed even though SG ACLs and NAT were correct. Switch the gate to FindDHCPOptionsByExternalID keyed on spinifex:subnet_id so each subnet owns its own row, matching the attach lookup at line 203. Verified: account 000000000001 default VPC RunInstances now produces an LSP with dhcpv4_options populated, VM cloud-init reports a 172.31.0.5/20 lease + default route via 172.31.0.1, and ping + ssh TCP handshake to the EIP succeed end-to-end. Stale DHCP_Options row for the prior dead subnet was wiped manually; reconciler recreated the correct row for the live subnet on its next pass. * fix(ipsec): enable openvswitch-ipsec at provision time, drop daemon write Provisioning now runs `systemctl enable --now openvswitch-ipsec.service` alongside openvswitch-switch and ovn-controller. The spinifex-daemon process drops its `enable --now` shell-out and only polls is-active. Sudoers gains one read-only line: spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/systemctl is-active openvswitch-ipsec.service This mirrors the existing vpcd ovn-controller is-active precedent and removes the daemon's need for systemd-write capability. Aligns with least-privilege service account pattern: provision enables services, daemon only reads status. If the unit is inactive at runtime, the daemon refuses to flip ipsec_encapsulation=true (silent-drop guard from mulga-siv-136) and surfaces a clear error pointing at scripts/setup-ovn.sh. * chore: unblock preflight on feat/network-layer-redesign Two pre-existing failures blocked preflight on the branch: - govulncheck flagged golang.org/x/net@v0.54.0 (GO-2026-5026, ASCII Punycode rejection bug in idna.ToASCII); reached via utils.DownloadFileWithProgress. Bump to v0.55.0; go mod tidy carries x/sys to v0.45.0. - TestTopologyHandler_SubnetCreate_RouterPortFails asserted subnet create fails when the parent VPC router is missing. That contract changed in aed7b2e (mulga-siv-133), which made handleSubnetCreate call EnsureVPC idempotently before EnsureSubnet to avoid a NATS publish-order race. Delete the now-impossible failure case. * refactor(network/ovn): relocate nbdb model package from services/vpcd Phase 4 / mulga-siv-125.6 Slice A. Pure import rename, no logic change. The OVN Northbound DB model (LogicalSwitch, LogicalRouter, DHCPOptions, etc.) lived under services/vpcd/nbdb/ because that was the original home of every OVN caller. After Phase 1+2 the L1 client (network/ovn) and all L3/L5 managers (network/policy, network/external, network/topology, network/reconcile) became the primary callers — 8 importers in network/* vs 2 in services/vpcd/. The directory move puts the schema model where its consumers are, and unblocks deleting services/vpcd/ entirely once the subscriber/launcher relocations land (Slices B–D). * refactor(vpcd): delete dead retrofit/reconcileIGW helpers + bridge-mode bleed Phase 4 / mulga-siv-125.6 Slice B. Production path no longer touches any of: - RetrofitAllGatewayPortNetworks / RetrofitAllExternalLocalnetOptions — the 5-pass startup leftovers that network/reconcile (Phase 2.2) made redundant. Nothing called them outside tests. - reconcileIGW + its allocateGatewayLRPIP / gwLrpRange helpers + the ipv4ToUint32 / uint32ToIPv4 utilities — superseded by external.IGWManager (Phase 2.4) which carries its own pool logic. - ensureLocalnetOptions / ensureGatewayPortNetworks / expectedGatewayPortNetwork — read-before-write retrofit shims for the same code paths. - generateMAC — only callers were reconcileIGW + its tests. useCentralizedNAT (the bridge-mode bleed flagged in plan §2.2) is replaced by a policy.NATMode value injected via WithNATMode(...). The mode is resolved once in vpcd.go from host.UplinkMode and threaded into both the topology-handler lazy managers and the reconciler stack, so "is this veth?" is no longer a question topology.go can ask. The TopologyHandler defaults to NATModeDistributed when WithNATMode is omitted, matching the previous default. Drops services/vpcd/topology.go from 1220 LOC to 693, deletes ~700 LOC of obsolete tests (TestGenerateMAC, TestEnsureLocalnetOptions_*, TestRetrofitAllExternalLocalnetOptions_FixesClearedOptions, TestReconcileIGW_*, TestGwLrpRange), and frees the nbdb + utils + net imports in topology.go. Stale "phase 4 migrates those onto the new managers" comment in vpcd.go fixed in passing — the subscribers were migrated in Phase 2.6 (mulga-siv-129); only the file-relocation parts of Phase 4 remain. Stale ensureLocalnetOptions / ensureGatewayPortNetworks references in network/ovn/{live.go,mock/mock.go} doc comments also cleaned up. * refactor(network): move NATS VPC subscribers to network/subscribers Phase 4 Slice C (mulga-siv-125.6). Promotes the NATS event handlers out of services/vpcd into a new spinifex/network/subscribers package. Subscriber holds the topology/policy/external managers directly — the lazy *Once adapter shim (topology_manager.go) is deleted along with vpcd's local event-struct copies. launchService now constructs topology.LiveManager eagerly with a DNS callback derived from external pools (preserving the unscoped-pool lookup old TopologyHandler.lmDNSServer did), builds every manager up front, then hands them to subscribers.New + Subscribe. reconcile.New receives topology.Manager directly instead of through the adapter. Bridge-mode constants and waitForFlowsHV move into vpcd.go alongside the launcher; they were left behind as topology.go's last responsibilities. Handler-layer tests (topology_test.go, security_test.go, integration_test.go) are dropped: substance is covered by network/topology, network/policy, network/external manager tests. subscribers/subscribers_test.go adds focused coverage: missing-dep rejection, topic registration smoke, and bad-JSON envelope per topic. services/vpcd is now down to the launcher + the ovn/mock compat aliases (removed in Slice D when the launcher promotes to cmd/vpcd/). * refactor(vpcd): relocate services/vpcd → spinifex/vpcd; drop dead aliases Phase 4 Slice D (mulga-siv-125.6). Final cleanup of the network-redesign package layout. services/vpcd/ disappears entirely; the launcher remains an in-process Service impl, just rehomed under spinifex/vpcd/ (no separate binary — spx vpcd start stays the cobra entrypoint). - Delete services/vpcd/mock_ovn.go + ovn.go (compat aliases now dead post Slice C; only callsite was launchService's NewLiveOVNClient, inlined to ovn.NewLiveClient). - git-rename services/vpcd/vpcd.go + vpcd_test.go to spinifex/vpcd/. - Retarget three importers (spinifex/service/service.go + service_test.go, cmd/spinifex/cmd/service.go) to the new path. - Fix stale services/vpcd / TopologyHandler comments in network/reconcile/ + network/external/types.go + spinifex/vpcd/vpcd.go. - service-interfaces.yaml vpcd path updated. - Plan doc §13/§14/§16 + §9.2/§19 updated: drop cmd/vpcd/main.go (was speculative; never built) in favour of in-process Service impl at spinifex/vpcd/vpcd.go. Plan §16 Created list now lists subscribers/. Phase 4 plan items 1+2 (remove services/vpcd, entrypoint location) satisfied. Items 3+4 remain blocked on mulga-siv-125.4 + 125.5 (bundled join token, plan archive). * fix(ovn): install openvswitch-ipsec + strongswan explicitly ovs-monitor-ipsec.service fails to start without an IKE daemon. The openvswitch-ipsec package only Recommends strongswan, so any apt config that drops Recommends (or a base image with the recommends already skipped) leaves the service in a Restart loop with 'IKE daemon is not installed'. Hoist both packages into setup-ovn.sh so provisioning is the single source of truth — the bootstrap shim in mulga is no longer needed. * test(network): pin ADR-0006 invariants to executable suite Add static-analysis + behavioural tests that bind every safety and liveness clause in ADR-0006 (S1-S9, L1-L3) to a named Go test. Failure output quotes the ADR clause verbatim and points at the offending file:line so future debugging sessions can't quietly bypass a layer contract to make a test green. Static-analysis suite (spinifex/network/invariants/): - S1 layers_test.go: parses `go list -json`; rejects upward layer imports and layers pulling from cross-cutter packages - S2 host_containment_test.go: AST-detects ovs-vsctl / ovn-nbctl / ovs-ofctl shell-out outside network/host/ - S3 nat_mode_test.go: rejects .UplinkMode() / NATModeFromUplinkMode() calls outside the init-time resolver locations - S4 names_test.go: AST-detects OVN object name construction (concat, fmt.Sprintf, strings.Join) outside topology/names.go - S5 policy_no_create_test.go: rejects CreateLogical{Switch,Router,...} and DeleteLogical{...} calls inside network/policy/ - S6 federation_isolation_test.go: rejects TransitSwitch / TransitRouterPort references and ovn-ic literals outside network/federation/ (pre-protects the layer before it lands) - S8 ipsec_test.go: rejects strongswan / xfrm / IKEv2 / libreswan / openswan imports anywhere in network/ - S9 reconciler_singleton_test.go: rejects Reconcile…Loop and Reconcile…Pass<n> function names anywhere in network/ - S7 nats_subjects_test.go: scans for `vpc.<verb>` subjects missing the {azID} segment; calls t.Skip with the gap report so every test run prints the work-to-do without breaking CI. Flip to t.Fatalf when the cluster-wide subject rename ships. Behavioural suite (spinifex/network/reconcile/liveness_test.go): - L1 reconciler converges: scaled-intent (N=1,3,8) double-Reconcile property test; asserts second pass is a no-op - L2 drift bound: structural check that DriftInterval is positive and ≤30 min support window; full fake-clock behavioural test deferred - L3 federation re-enables: t.Skip stub pointing at the missing network/federation/ package Fixes the one production S4 violation the new walker found: network/external/igw.go was constructing "gw-port-"+vpcID inline. Adds topology.GatewaySwitchPort(vpcID) helper and retargets both AttachIGW and DetachIGW callsites through L2. Plan + per-clause mechanic + phasing in docs/development/improvements/network-invariants.md. Refs: mulga-siv-140 * fix(ovn): install strongswan-charon (not strongswan metapackage) The strongswan metapackage's '\''charon-systemd | strongswan-charon'\'' alternative resolves to charon-systemd by default, which ships /usr/sbin/charon-systemd but not the legacy /usr/sbin/ipsec helper or /usr/lib/ipsec/charon binary that ovs-monitor-ipsec --ike-daemon=strongswan shells out to. Install strongswan-charon explicitly to pull strongswan-starter (legacy ipsec controller) + the matching charon binary, matching what openvswitch-ipsec.service expects. Also reset-failed + restart openvswitch-ipsec.service after enable to recover from the deb post-install starting the service before its dependencies finished unpacking. * chore: add CODEOWNERS — joshsiv-mulga gates network/ + vpcd/ Fall-through reviewer for any change is the maintainer. network/ and vpcd/ are explicitly listed so the matching branch protection rule ("Require review from Code Owners") routes those PRs to joshsiv-mulga. Those trees are governed by ADR-0006 invariant tests (spinifex/network/invariants/); CODEOWNERS adds a human review gate on top of the mechanical fence. Refs: mulga-siv-141 * chore: CODEOWNERS fallthrough → @mulgadc/mulgadc-engineers team Fallthrough reviewer should be the team, not a single maintainer, so any engineer can approve PRs outside the explicitly-gated paths. network/ and vpcd/ keep joshsiv-mulga as the required reviewer per ADR-0006 invariant ownership. Refs: mulga-siv-141 * feat(ci): add test_run workflow input for -test.run filter Pass an optional regex through to each suite's '\''go test -test.run'\'' so dispatches can target a single test (e.g. '\''^TestMultinodeIPSec$'\'') without running the full multinode suite. Threaded through single, multinode, and reboot suite invocations; empty input preserves current behaviour. * fix(daemon): flip NB_Global.ipsec=true on management node Setting Open_vSwitch.other_config:ipsec_encapsulation=true on each chassis is necessary but not sufficient for OVN native IPsec — without NB_Global.ipsec=true, ovn-controller never adds options:remote_name to the Geneve tunnel ports, so ovs-monitor-ipsec sees no IPsec-enabled tunnels and the kernel xfrm tables stay empty. After enabling ipsec_encapsulation locally, also write NB_Global.ipsec=true when the local ovnnb_db socket is present (management node only); workers skip silently — one writer is enough for a cluster-wide flag. Adds a scoped sudoers entry pinning the daemon to that exact ovn-nbctl invocation so the grant can't be reused for other NB mutations. * fix(ovn): grant charon AppArmor read on /etc/spinifex/ipsec The strongswan-charon profile only allows reads from /etc/ipsec.*, /etc/strongswan.*, and a small fixed allowlist. ovs-monitor-ipsec writes the per-tunnel strongSwan config with absolute paths to our peer cert + key under /etc/spinifex/ipsec/, so charon hits 'Permission denied' loading them (it runs as uid 0 but AppArmor still blocks) and no xfrm SAs come up. Drop a read grant for /etc/spinifex/ipsec/** into the profile's local override hook + reload. * fix(ovn): pin OVS system-id to hostname for IPsec cert identity match ovs-monitor-ipsec uses chassis-id as the IKEv2 `@<name>` peer identity in the strongSwan config it generates. Our per-node IPsec peer cert (admin.GenerateIPSecPeerCert) carries the node name as CN + dnsName SAN. With chassis-id left at the package-generated UUID, charon validates `@<UUID>` against a cert with dnsName=<hostname> — mismatch returns AUTHENTICATION_FAILED and no xfrm SAs are negotiated. Pin system-id to hostname (which equals the node name on Mulga clusters) by writing both the on-disk source of truth (/etc/openvswitch/system-id.conf, read by ovs-ctl on boot) and the running OVS DB value. Writing only one would drift on the next reboot — the original "do not override" comment was right about that hazard, just wrong about the resolution. Cross-link the invariant in admin/ipsec.go: changing the cert identity without updating setup-ovn.sh (or vice versa) breaks IKE silently. Refs: mulga-siv-136 * fix(ovn): take --node-name flag so chassis-id matches cluster identity CI hostnames are bead-themed (env20-casuarina) and don't match the cluster node name (node1/node2) baked into the IPsec peer cert dnsName SAN at `spx admin init/join --node NAME`. Pinning system-id to `hostname -s` produced env20-casuarina; charon then validated `@node1` against dnsName=env20-casuarina and IKE failed with AUTHENTICATION_FAILED. Add a --node-name=NAME flag to setup-ovn.sh, default to hostname for manual/dev invocations, and update the ansible install role to pass {{ spinifex_node_name }}. The matching bootstrap-install.sh change lives in mulga. Refs: mulga-siv-136 * refactor(daemon): relocate L0 helpers to network/host + widen ADR-0006 S2 Phase 4 Slice E of mulga-siv-125.6. Cleans up spinifex/daemon/network.go per the audit and widens the S2 invariant to forbid OVS/OVN shell-outs in spinifex/daemon/ (daemon is an L0 consumer; S2 applies). Moved to network/host/: - OVSNetworkPlumber + SetupTap/CleanupTap → host.OVSPlumber (host.NewOVSPlumber) - GetBridgeIPv4 → host.GetBridgeIPv4 - OVNHealthStatus + CheckOVNHealth → host.OVNHealth + host.HealthStatus - daemon/ipsec.go OVS/OVN ops → host.SetIPSecCertPaths, host.EnableIPSecEncapsulation, host.SetNBGlobalIPSec Moved to utils: - sudoCommand → utils.SudoCommand + utils.SetSudoCommandForTest test seam (cross-package reassign would trip the reassign linter; helper keeps the indirection inside utils). Moved to vm: - generateMgmtMAC → vm.GenerateMgmtMAC (pairs with vm.GenerateDevMAC) Deleted (dead): - SetupComputeNode (replaced by network/host/bridges.go:ensureBridges + scripts/setup-ovn.sh:572-579) - EnsureDataRoute - findInterfaceByIP - daemon/network.go + dead tests in daemon/network_test.go (duplicates of vm package tests retained; mgmt MAC test moved to vm/lifecycle_test.go) S2 invariant (network/invariants/host_containment_test.go) now walks both spinifex/network/ and spinifex/daemon/, skipping network/host/. * fix(ovn): broaden AppArmor read grant to /etc/spinifex (covers CA cert) The earlier override granted reads on /etc/spinifex/ipsec/** only, which covers the peer cert + key but NOT the cluster CA at /etc/spinifex/ca.pem. charon then can't load the CA, can't validate the peer cert chain, and surfaces the failure as `no trusted RSA public key found for '<peer>'` even though identity matching now succeeds — IKE_AUTH fails with AUTHENTICATION_FAILED and no xfrm SAs negotiate. Grant /etc/spinifex/** instead (covers ca.pem + ipsec/peer.{pem,key}) and always rewrite the override so re-runs replace the older narrower rule. Refs: mulga-siv-136 * fix(ipsec): symlink cluster CA into charon's default cacerts dir ovs-monitor-ipsec writes per-conn `cacert=/etc/spinifex/ca.pem` references in the strongSwan config, but strongSwan 6.0 only loads CAs declared in standalone `ca` sections or files placed in /etc/ipsec.d/cacerts at startup. Without this, charon authenticates the local side fine but rejects every inbound peer cert as `no trusted RSA public key found for '<peer>'`, AUTH_FAILED, and no xfrm SAs negotiate. Do the symlink in admin.GenerateIPSecPeerCert (right after writing the peer cert) — setup-ovn.sh runs before `spx admin init/join` creates /etc/spinifex/ca.pem, so the wiring can't live there. Use a stage-and- rename pattern so concurrent test invocations don't race on the symlink. /etc/ipsec.d/cacerts is on charon's default AppArmor allowlist so no profile change needed. Refs: mulga-siv-136 * chore(setup-ovn): drop apt-get install_packages block OVS/OVN/openvswitch-ipsec/strongswan-charon are baked into the CI gold image via mulga's image-builder/scripts/provision.sh (commits 116bb05, a9b3a9c). Keeping a runtime apt-get install path here pulled CI through apt-cacher-ng on every bootstrap and tripped on intermittent "Message has been manipulated" InRelease cache corruption (env21/env20 runs). Drop the install block. Gold-image rebuild is the canonical place to add new system packages. If a package is missing, downstream ovs/ovn commands fail loudly during setup-ovn.sh's later steps — clearer signal than silently re-installing through a flaky proxy. setup.sh already supports INSTALL_SPINIFEX_SKIP_APT=1 (passed by mulga's bootstrap-install.sh:237) so this removes the last runtime apt-get call on the CI provisioning path. * fix(ovn): gate system-id pin on explicit --node-name setup-ovn.sh used to fall back to `hostname -s` when --node-name wasn't passed, and then unconditionally wrote that value to /etc/openvswitch/system-id.conf + ovs-vsctl external_ids:system-id. On CI single-node, the gold image already registered a chassis with name=<UUID>; setup-ovn.sh then rewrote system-id to the bead-themed hostname (env10-banksia). vpcd's discoverChassis treats any chassis row whose name != local system-id but hostname == local hostname as a stale local chassis and skips it, so vpcd reported "no OVN chassis registered in SBDB" and refused to start, cascading into the e2e + e2e-go failures on run 26430839508. Drop the hostname fallback. If --node-name is given, pin both the on-disk file and the running OVS DB to it (callers that care about IPsec cert identity matching still get it). If no --node-name is given, leave the existing system-id alone — that's always safe whether the value came from a gold-image UUID, bootstrap-install.sh writing nodeN before this script runs, or a manual ansible value. * fix(network/reconcile): skip orphan prune on startup; drift owns GC The vpcd startup reconcile races against the daemon's EnsureDefaultVPC: the daemon writes SG KV rows + publishes vpc.create-sg on each node in parallel with vpcd starting up. The leader-gated reconcile can load intent before any SG row is visible on its local JetStream replica while subscribers on peer nodes are concurrently creating port groups from the same vpc.create-sg events. A startup orphan-port-group sweep then deletes managed sg_* port groups that intent hasn't caught up to. CI run 26434124790 trace: node2 reconcile/lock: elected holder=env20-bottlebrush node2 reconcile: starting intent_sgs=0 node2 reconcile/apply: removed orphan port group pg=sg_161c70ce23e704a2a node2 reconcile/apply: removed orphan port group pg=sg_a0d078650a3150391 node2 reconcile/apply: removed orphan port group pg=sg_54b564b374e9162bb Every subsequent RunInstances → vpc.create-port failed with "port group not found" for ~3 minutes until the 5-minute drift tick recreated the port groups. Tests give up after 60s; six multinode suites cascade-fail on InsufficientInstanceCapacity. Split the reconciler into Reconcile (full prune, drift loop) and ReconcileApplyOnly (no prune, startup). applySGs takes a pruneOrphans bool. vpcd startup calls ReconcileApplyOnly; drift loop unchanged. Legitimate orphans get garbage-collected on the next drift tick — at worst 5 minutes of lingering rows, no production impact. Add TestReconcile_ApplyOnlyKeepsOrphanPortGroup to lock the contract. * fix(ipsec): trigger charon rereadcacerts after CA symlink install strongSwan scans /etc/ipsec.d/cacerts only at charon startup. The unit openvswitch-ipsec.service is enabled at provision time and starts before admin init writes the cluster CA symlink, so charon ends up running with zero CAs loaded. Every IKE_AUTH then fails with `no trusted RSA public key found for '<peer>'` and no SAs negotiate — the failure mode exposed by TestMultinodeIPSec. Append a stroke rereadcacerts call at the end of installCAIntoCharonTrustStore so an already-running charon picks up the freshly-placed CA without a service restart. Failure is swallowed at debug — a missing/stopped charon at admin-init time is expected and a later startup will scan cacerts naturally. * refactor(network/host): move IPSec orchestrator from daemon (ADR-0006 S8) ADR-0006 S8: "IPSec is OVN-native only. No layer implements custom IKEv2, XFRM rules, or direct strongSwan management. IPSec SA lifecycle is delegated entirely to OVN native IPSec and is invisible above L0." The enableOVNIPSec orchestrator and its ensureOVSMonitorIPSecActive helper lived in daemon/ipsec.go but composed L0 primitives (SetIPSecCertPaths, EnableIPSecEncapsulation, SetNBGlobalIPSec) that already lived in network/host. The orchestrator now lives alongside its callees as host.EnableOVNIPSec, taking configPath and clusterConfig from the daemon at the call site. Helper and test-override package vars (systemctlActiveTimeout, ovnNBSocketPath) move with it; tests are translated to call EnableOVNIPSec directly rather than via the Daemon struct. Closes mulga-siv-142 * refactor(daemon): use topology.Port for OVN port names (ADR-0006 S4) ADR-0006 S4 forbids any layer above L2 from constructing OVN object names. Three daemon sites formatted "port-" + ENIId inline; route them through topology.Port instead. Closes mulga-siv-143 * refactor: use topology.Port for OVN port names in handlers + vm (ADR-0006 S4) Replace inline "port-"+eniID formatting across five sites (eip, instance x2, vpc/eni, vm/network_helpers) with topology.Port. OVSIfaceID stays as a thin wrapper to preserve the OVN binding contract comment and existing call sites. Closes mulga-siv-144 * test(network/invariants): widen scan roots to daemon/handlers/vm/testutil/utils (ADR-0006) Widen ADR-0006 invariant scan roots so drift in upstack consumers surfaces in CI. Backlog of violations in widened roots was drained by mulga-siv-142/-143/-144; this commit makes the wider scan permanent. - S2 (host_containment): add vm/ alongside network/ + daemon/ - S3 (nat_mode): add daemon/ alongside network/ + vpcd/ - S4 (names): add daemon/, handlers/, vm/ alongside network/ + vpcd/; add narrow AWS-ID exemption (vpc-cidr-assoc-) so AWS resource IDs are not misclassified as OVN object names - S6 (federation): add daemon/, handlers/ alongside network/; convert single-root walk to slice - S7 (nats_subjects): add daemon/, handlers/, testutil/, utils/ — stays t.Skip pending cluster-wide subject rename; gap report now lists 58 subjects CODEOWNERS: add /spinifex/network/invariants/ rule so widening or narrowing the scan needs explicit owner review. Closes mulga-siv-145 Refs: audit-adr-0006.md * chore(e2e): disable IPsec in pseudo multi-node test Pseudo multi-node runs three simulated nodes as unprivileged tf-user. The charon trust-store install in 904b45e writes /etc/ipsec.d/cacerts, which fails with EACCES under a non-root caller and aborts admin init. Disable --ipsec for this suite — loopback-only nodes don't exercise real Geneve encryption anyway. Real-multi-node + single-node e2e (both run admin init via sudo) continue to cover the IPsec path. * fix(ovn): atomic Ensure* primitives prevent duplicate logical entities OVN NB has no unique constraint on Name. The previous GetLogicalRouter→CreateLogicalRouter sequence in EnsureVPC was a TOCTOU race: when vpc.create on one node and the defensive vpc.create-subnet EnsureVPC call on another both observed absence, both inserted, yielding two routers per VPC. The orphan held DNAT references blocking EIP detach, and DNAT/flow installation landing on the orphan caused TestMultinodeGuestSSH SSH timeouts. Add EnsureLogicalRouter / EnsureLogicalSwitch / EnsurePortGroup to the libovsdb client. Each builds a single OVSDB transaction combining a wait-op (Until=equal, Rows=[]) gated on absence of a Name match with the insert. The server serialises the wait+insert atomically; concurrent callers see exactly one survivor and the rest fall through to a re-fetch that returns the canonical row. Switch all production callers (topology.EnsureVPC / EnsureSubnet / EnsureSGPortGroup, external.AttachIGW, reconcile.applyVPCs / applySubnets) to the Ensure* variants. Bare Create* remains available to the L1 wrapper. Add invariant test TestS4_OVNNamedCreatesUseEnsure (AST walk) to block future reintroduction of the bare Create* call sites outside network/ovn. Mock implements Ensure* under its existing mutex so the concurrency tests (50/30/30 goroutines for Router/Switch/PortGroup) prove a single UUID survives. Refs: mulga-siv-146 * fix(ovn): wait-op rows must be non-empty; flip to NotEqual predicate The Ensure* primitives sent a wait-op with `Rows: []ovsdb.Row{}` plus `Until: ==` (RFC 7047 §5.2.6 absence test). The libovsdb Operation struct tags Rows as `json:"rows,omitempty"`, so the empty slice was dropped on the wire and ovsdb-server rejected every transaction with "Required 'rows' member is missing", breaking reconciler bootstrap and every downstream create-port lookup ("logical switch X not found"). Flip the predicate to the equivalent `Until: !=` formulation with a single-row projection containing the Name we want absent: no row → projection [] != [{name}] → predicate holds row exists → projection [{name}] == [{name}] → predicate fails Same atomic guarantee, JSON now well-formed. Mock tests can't catch this — only a real ovsdb-server validates the envelope. * chore: remove generateMAC wrapper * chore: remove all bead / plan references * fix(network/policy): wait for chassis flows after AddNATGateway AddEIP fires m.barrier() to block until ovn-northd compiles NB→SB and every chassis installs the new dnat_and_snat flows; AddNATGateway returned right after the NB commit. So `aws ec2 create-route … --nat-gateway-id` would publish vpc.add-nat-gateway, the SNAT row would land in NB, and the API call would succeed before the gateway chassis had programmed the SNAT flow for the private subnet. Phase 8d's 10×5s=50s ping budget races that flow-install latency on slow CI hosts (last seen run 26455602615 single-node, identical flake on unmodified dev run 26434711010 — pre-existing, not introduced by Ensure*). Mirror AddEIP: call m.barrier() after AddNAT returns. Add unit test TestNATManager_AddNATGateway_FlowsBarrier_Fires covering the new path. * chore: remove more plan refs * chore: remove GenerateIPSecPeerCertIfEnabled * chore: remove dupliate subnetGatewayCIDR function * chore(network): trim superfluous comments across network/ tree Collapse multi-paragraph godocs to 1-2 sentences, drop WHAT-restatement comments, remove section dividers grouping <10 items, and cut prose narration in tests. ADR clause references, one-line WHY comments for non-obvious OVN/OVSDB/AWS/NATS semantics, and exported-symbol godoc are preserved. No behavioural changes. Packages: invariants, topology, host, external, ovn, reconcile, policy, subscribers. * fix(igw): restore attachment State="available" for AWS parity AWS InternetGatewayAttachment.State is "available" when an IGW is attached to a VPC, per the AWS SDK doc ("the state is available when attached to a VPC; otherwise, this value is not returned"). Commit b7505f8 wrote "attached"/"detached" to align with the reconciler filter, which inverted AWS parity. Fix the producer to write "available" again and update the consumer: - handlers/ec2/igw: AttachInternetGateway and DetachInternetGateway both persist State="available"; IGWRecord comment narrowed. - network/reconcile/intent.loadIGWs: filter accepts "available" (was "attached"); attachment is identified by VpcId != "". - tests updated: igw service_impl_test, routetable service_impl_test fixture, reconcile intent_test fixture. * Update spinifex.toml * refactor(subscribers): route EIP/NATGW through L5 wrappers (ADR-0006 §S1) Subscriber previously called policy.NATManager directly for EIP and NAT Gateway operations, bypassing the L5 wrappers (EIPManager, NATGWManager) that existed on paper. ADR-0006 §S1 requires every cross-layer call to pass through the immediate lower neighbor's typed interface; IGW already followed this pattern. Wire EIPManager and NATGWManager into the Subscriber struct, construct them in vpcd alongside IGWManager, and swap the four direct-L3 call sites (AddEIP, DeleteEIP, AddNATGateway, DeleteNATGateway) onto the L5 surface (AttachEIP, DetachEIP, AttachNATGateway, DetachNATGateway). AttachEIP now fires the flows-barrier, giving callers the same "reachable on return" guarantee IGW already had. The reconciler still calls policy.NATManager directly; that's a cross-cutter and a separate concern. * test(reboot): port 8MB WAL pad workaround from bash run-reboot-e2e.sh App VMs' user-data writes (unit file + index.html) sit in the on-host WAL until WriteWALToChunk crosses the 4MB threshold. Host reboot before that loses them — systemd then refuses to load the zero-byte unit and the responder never starts. Mirror the bash driver's dd+sync pad so the WAL flushes pre-reboot. Drop when the upstream WAL fix lands. * test(e2e): dump OVN/OVS/SG state on Phase 8d NATGW failure Phase 8d's existing failure path only prints lr-nat-list and lr-route-list. CI artifacts are not uploaded when run-e2e.sh exits nonzero, so debugging the NATGW regression requires inline diagnostics in the journal. Expand the dump to include nb/sb show, chassis bindings, port_group ACLs, br-int flows filtered to the NATGW IP/private subnet, host routes, and the private VM's own ip/route/neigh state. * fix(ovn,reconcile): scope NAT deletes to router; NATGW snat per associated subnet Two compounding bugs caused Phase 8d NATGW E2E flake (~60% CI failure): 1. ovn.DeleteNAT/DeleteNATByExternalIP searched all NAT rows globally and only mutated the named router afterwards. With AWS subnet CIDRs repeating across VPCs, (type, logicalIP/externalIP) is not globally unique — the delete picked a NAT row owned by a different router, the mutate was a no-op, and the row delete failed with "remaining reference(s)", leaving the original router's NAT orphaned. Both calls now resolve the LR first and filter NAT lookups to lr.NAT UUIDs. Mock client mirrors the scoping. 2. reconcile.loadNATGWs used the NATGW's home (public) subnet CIDR as SubnetCIDR, producing an snat for the wrong CIDR that collided with the correct event-published snat. loadNATGWs now walks route tables and emits one NATGWSpec per (NATGW, associated private subnet) pair, mirroring publishNatGatewayEventsForAssociation. Intent map keyed by natgwID|subnetCIDR so multi-subnet NATGWs produce distinct specs. Tests: - nat_test.go: cross-router same-logicalIP isolation for DeleteEIP and DeleteNATGateway - intent_test.go: NATGW snat uses associated private subnet CIDR, not the NATGW home subnet; NATGW with no associations produces no spec --------- Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>	2026-05-27 21:05:01 +10:00
Julian Sommer	20a8a39608	feat: flip images to UEFI default (#237)	2026-05-20 12:34:51 +10:00
Josh Siviour	4ae2a4ac53	feat(elbv2): QEMU microvm direct-boot for LB system instances (#228) ... * feat(vm): direct-boot foundations + config/input extensions + image build pipeline vm package (mulga-siv-62): - Add KernelImage, Initrd, KernelCmdline, FwCfg/FwCfgEntry to vm.Config - Emit -kernel/-initrd/-append/-fw_cfg args when fields set - Relax drive invariant: accept kernel image in place of drives - Add devices.go: IsMMIO predicate, NetDevice/BlkDevice/RngDevice helpers - Refactor lifecycle.go + network_helpers.go call sites to use helpers - shutdown.go: os.Remove fw_cfg tmpfiles after QEMU exits - Add devices_test.go, microvm_test.go; extend vm_test.go config/input extensions (mulga-siv-63): - Add MicrovmConfig{ELBv2Enabled, ImagePath} to DaemonConfig - ImagePath falls back to /usr/share/spinifex/microvm when empty - Add commented [microvm] section to spinifex.toml template - Add DirectBoot bool + NICs []NICConfig to SystemInstanceInput image build pipeline (mulga-siv-64): - scripts/build-microvm-image.sh: Alpine chroot build producing vmlinuz + initramfs.cpio.gz - build/microvm/init.sh: fw_cfg-driven PID 1 init (mounts, modprobe, NIC config, lb-agent-env, ca-cert) - build/microvm/inittab: BusyBox init with lb-agent respawn - make build-microvm-image target - setup.sh: install microvm/* to /usr/share/spinifex/microvm/ - .gitignore: exclude built vmlinuz + initramfs.cpio.gz artifacts * feat(daemon): direct-boot launch path for microvm system instances When SystemInstanceInput.DirectBoot=true: - Skip GetAMIConfig and GenerateVolumes - Build vm.Config directly (no buildBaseVMConfig, no PCIe root ports) - MachineType=microvm,x-option-roms=off,pic=off,pit=off,rtc=on,acpi=on,isa-serial=on - Resolve kernel/initrd from Microvm.ImagePath config field - Serialise NICs to three fw_cfg tmpfiles (netcfg, lb-agent-env, ca-cert) - Validate exactly one NICConfig.IsDefault=true before writing netcfg blob - Shutdown path cleans up tmpfiles via existing Config.FwCfg iteration PC-machine path (DirectBoot=false) unchanged. Closes mulga-siv-65 * feat(elbv2): wire microvm.elbv2_enabled flag into ELBv2 service When ELBv2Enabled=true: - Drop lbVMUserData cloud-config; emit KEY=value LBAgentEnv blob instead - Set SystemInstanceInput.DirectBoot=true; omit ImageID - Populate NICs []NICConfig (index 0=primary ENI IsDefault, 1=mgmt, 2+=extra ENIs) - Validate exactly one IsDefault NIC at service layer When ELBv2Enabled=false: existing PC-machine path unchanged. Adds buildLBAgentEnv, buildMicrovmNICs, subnetCIDRForIP, subnetGatewayIP helpers. Tests in arn_test.go cover both flag-on and flag-off paths. Closes mulga-siv-66 * test(daemon,vm): add unit tests for direct-boot helpers to meet diff coverage threshold * feat(daemon): Phase 1 perf instrumentation + e2e runner microvm flag - Log t1→t2 boot timing via slog on the direct-boot path - measureQEMURSSMiB helper reads /proc/<pid>/status (tested) - Gate constants: phase1BootGateMs=500, phase1RSSGateMiB=25 - run-lb-e2e.sh: --microvm flag, dual-path CI annotation, density test note Closes mulga-siv-67 * fix(build): remove busybox-initscripts — not in Alpine 3.20, not needed * fix(microvm): end-to-end fixes for direct-boot LB path Serial output was silent (x-option-roms=off suppresses BIOS serial init); fixed by removing it from the machine string and using -serial chardev:console0 to attach to the machine-created ISA serial device instead of adding a duplicate isa-serial device (which would create ttyS1 while kernel uses ttyS0). virtio-net interfaces were invisible to the kernel because QEMU's auto-kernel-cmdline does not append virtio_mmio.device= params when -append is used. Fixed by generating explicit virtio_mmio.device=0x200@0xfebNNNNN:N params for each NIC in KernelCmdline, and modprobing virtio_mmio + virtio_net in init.sh. sys.micro bumped to 512 MiB — 128 MiB caused kernel OOM during initramfs decompression (~68 MiB uncompressed). BusyBox init ::respawn: entries do not source or export env files; fixed inittab to use set -a; . /etc/conf.d/lb-agent; set +a before exec. wireLBAgentConfig was documented as reading the CA cert but never set elbv2Service.CACert; lb-agent TLS handshakes to AWSGW failed with x509: certificate signed by unknown authority. Fixed by reading d.config.NATS.CACert (same local CA that signs the AWSGW server cert) and assigning to elbv2Service.CACert so writeFwCfgBlobs writes a non-empty cacert blob into fw_cfg. Adds vm_test TestExecute_MicrovmFileChardev and updates lifecycle_test to assert SerialSocket is intentionally empty on direct-boot VMs. * fix(build): auto-detect docker/podman when apk unavailable for microvm image build * fix(build): create /dev/console and /dev/null as proper device nodes in initramfs tar extraction without root silently demotes device files to 0-byte regular files (mknod requires CAP_MKNOD). The kernel opens /dev/console when wiring init's stdio to the serial console; a regular file there causes all init output to be silently discarded to RAM — the VM appears stuck at 'Run /init as init process' with no subsequent console output and lb-agent never heartbeats. Fix: after rootfs population, wipe /dev and recreate console (c 5 1) and null (c 1 3) via sudo mknod. devtmpfs provides all other devices at runtime once init.sh mounts it. * fix(microvm): fix mgmt NIC routing for all LB schemes Two bugs prevented lb-agent from reaching the AWSGW via the mgmt interface: 1. RouteDst/RouteVia swapped — resolveMgmtRoute returns (gateway, target) but buildMicrovmNICs assigned them as (Dst=gateway, Via=target), producing 'ip route add 10.15.8.1 via 192.168.1.33' instead of the correct 'ip route add 192.168.1.33 via 10.15.8.1'. 2. NIC[1] MAC and CIDR never filled — the daemon allocates mgmt IP/MAC after buildMicrovmNICs returns, but didn't inject them into input.NICs[1] before calling writeFwCfgBlobs, leaving NIC1_MAC= empty in the netcfg blob and skipping interface setup in init.sh entirely. Also extends mgmt routing to internet-facing scheme in the microvm path: VPC SNAT is not reliable across all VPC topologies for control-plane heartbeats; mgmt is always a direct L2 path to the host. * feat(microvm): Phase 2 kernel stripping + boot optimisation - cpu: switch to host,-pmu (disable PMU — not needed in microvm guest) - machine type: make isa-serial configurable via microvm.isa_serial config field; default true (on) preserves console log path - build: strip unused kernel modules (USB, IDE, ATA, GPU, media, input, HID, bluetooth, staging, sound, wireless) from initramfs before cpio - build: add Phase 2 artifact size gate — fail if vmlinuz + initramfs.cpio.gz exceeds MICROVM_ARTIFACT_MAX_MiB (default 50 MiB); override via env var during development - perf: add phase2BootGateMs=300 and phase2ArtifactGateMiB=50 constants with canary tests to catch unintentional gate changes - tests: cover buildMicrovmNICs nil-VPC and extra-ENI paths - tests: cover microvmMachineType nil/true/false branches * fix(build): use find-based virtio+fw_cfg module keep-list in initramfs strip The directory-only keep-list (drivers/virtio, drivers/firmware) missed virtio_net.ko which lives in drivers/net/, virtio_blk in drivers/block/, and virtio-rng in drivers/char/hw_random/. Switch to find-based approach: collect all virtio*.ko* and *fw_cfg*.ko* regardless of subdirectory, then restore them into their original relative paths under kernel/. Artifacts now: 42 MiB (vmlinuz 11 MiB + initramfs 31 MiB), gate passes. * fix(build): keep failover modules in initramfs strip virtio_net depends on net_failover/failover for live-migration support. Stripping them left virtio_net with unresolved symbols (net_failover_create/destroy), so the module failed to load and the guest came up with only lo. init.sh then skipped every NIC ("no interface found for MAC ..."), leaving lb-agent with no network path to AWSGW and the LB stuck in provisioning. Add *failover*.ko* to the keep-list. Image size unchanged at 42 MiB (under the 50 MiB Phase 2 gate). ELBv2 e2e now reaches active and 2/2 targets healthy for both ALB and NLB internal LBs. * fix(daemon): restore slog default after viperblock.New mutates it viperblock.New unconditionally calls vb.SetDebug, which calls slog.SetDefault with a TextHandler at LevelError — globally silencing every slog.Info and slog.Warn in the caller process for the rest of its life. Discovered while validating microvm Phase 2 (mulga-siv-68): the daemon stopped emitting Info logs the moment the first customer VM with volumes launched, hiding direct-boot timing, LaunchSystemInstance completed, and CreateLoadBalancer completed lines that we need for boot-time tuning. Workaround in newViperblock: re-install an Info-level TextHandler on os.Stderr after each viperblock.New call. Also drop the three explicit vb.SetDebug(false) calls in prepareRootVolume / prepareEFIVolume / prepareCloudInitVolume — they were no-ops (viperblock.New already runs SetDebug internally) and each one re-triggered the global mutation we are trying to undo. Proper fix is in viperblock itself — tracked as mulga-siv-70: SetDebug should not call slog.SetDefault. Until then, this restore call is the shortest path to getting the daemon's observability back so we can measure t1_to_t2_ms and tune Phase 2. Preflight note: golangci-lint not available on this dev host, so the project preflight target cannot run end-to-end. Verified with go vet ./spinifex/handlers/ec2/instance/... (clean) and go test ./spinifex/handlers/ec2/instance/... (pass). * perf(microvm): hit Phase 2 boot/memory gates t1→t2 hypervisor: 3224 ms → 264 ms (12×). 128 MiB guest now boots cleanly without OOM, reverting the temporary 512 MiB bump. Root causes: 1. build-microvm-image.sh staged the entire chroot into the cpio without dropping /boot, embedding a duplicate vmlinuz, Alpine's stock initramfs-virt, System.map, and a "boot → ." symlink loop (~25 MiB). Decompression peak exceeded 128 MiB. Strip /boot after the kernel is copied out, before the cpio. Initramfs drops from 31 MiB → 11 MiB (53 MiB → 28 MiB uncompressed); total artifact 42 MiB → 22 MiB. 2. vm/lifecycle.go burned 3 s of fixed sleep on every direct-boot launch: 2 s nbdkit pre-exec wait (no nbdkit on direct-boot) and 1 s post-fork settle (QEMU binds QMP within ~10 ms when there is no firmware, ROM, or block layer). Both extracted to directBoot-aware helpers — nbdkitPreExecWait collapses to 0, and qemuStartSettleWait to 50 ms. 3. KernelCmdline only carried `console=ttyS0 virtio_mmio.device=…`. Added boot-time perf flags: quiet loglevel=3 (silence printk over 115200-baud isa-serial), mitigations=off (KVM guest on trusted host, single-tenant LB workload), tsc=reliable, no_timer_check, reboot=t, i8042.{nopnp,noaux,nokbd} (microvm has no PS/2). 4. init.sh dropped modprobe virtio_mmio (built-in: CONFIG_VIRTIO_MMIO=y) and tightened polling: fw_cfg/iface waits go from 10 × 200 ms (up to 2 s) to 5 × 20 ms (up to 100 ms). Phase 2 gates: - t1→t2 ≤ 300 ms ✅ 264 ms / 281 ms - artifact ≤ 50 MiB ✅ 22 MiB (gate already in build script) - ELBv2 e2e green ✅ 32/33 (pre-existing client-VM failure, unrelated to microvm path) * build(microvm): wire build into deploy + distro + release pipeline Adds install-microvm Makefile target (with grouped-target staleness detection on vmlinuz/initramfs.cpio.gz), wires it into dev-install.sh and reset-dev-env.sh, makes 'make deploy' reinstall artifacts when present, and makes 'make distro-amd64' stage build/microvm/* into the release tarball staging dir. arm64 distro stays unchanged (no aarch64 microvm artifact yet — Phase 4). Release workflow gains a setup-go step and a 'make build-microvm-image' step before 'make distro-amd64' on the amd64 runner so the resulting tarball includes the kernel + initramfs without a manual copy. Precursor for the Phase 3 PC-machine LB removal (mulga-siv-69). Closes mulga-siv-71. * build(ansible): include microvm artifacts in build role tarball Adds a build-microvm-image task to the ansible build role and stages build/microvm/* into the role's release tarball so the install role's setup.sh run picks them up via INSTALL_SPINIFEX_TARBALL. Skipped gracefully when the build script is absent (older branches without direct-boot LB wiring). * feat(config): enable microvm direct-boot LB path by default Uncomments the [nodes.<node>.daemon.microvm] section in the generated spinifex.toml template and sets elbv2_enabled = true so all new clusters bring up ELBv2 LBs via the direct-boot microvm path. Operators can opt back to the legacy PC-machine path by flipping the flag to false in the rendered config. Required for end-to-end CI on this branch to exercise the direct-boot path. Existing clusters' configs are unchanged (the template only affects fresh 'spx admin init' renders). * fix(microvm): bootstrap apk.static when no host rootfs tool exists Self-hosted CI runners and the GitHub-hosted ubuntu-24.04 image often have none of apk, docker, or podman installed. apk-tools is not in the Debian 13 (trixie) repositories. Without one of these the rootfs build fails up front with 'no rootfs build tool available'. When all three are absent, fetch apk-tools-static from the configured Alpine mirror, extract it to a tempdir, and prepend it to PATH so the existing native-apk strategy below picks it up transparently. Requires only curl + tar, both of which are present on every CI runner that can already check out source. No-op when apk, docker, or podman is already present. Unblocks https://github.com/mulgadc/spinifex/actions/runs/25988830244 * revert: drop apk.static bootstrap in build-microvm-image Reverts 0bd76b7. Self-hosted runners will get docker.io installed out of band; existing Strategy 2 (docker run alpine:3.20) handles the rootfs build without host sudo or unsigned package install. * docs(workbook): drop lb-alpine import for microvm direct-boot LB LBs now launch via direct-boot QEMU microvm using kernel + initramfs bundled at /usr/share/spinifex/microvm/, so the legacy lb-alpine system AMI import is no longer required. * fix(daemon): own tap devices by daemon user so QEMU can attach without CAP_NET_ADMIN When the daemon runs non-root (production: User=spinifex-daemon), child QEMU processes inherit that uid but lack CAP_NET_ADMIN. The kernel's tun attach check then requires the tap owner uid to match the caller's euid. Without the `user` flag, `ip tuntap add` defaults to root:root and qemu's TUNSETIFF fails with EPERM ("could not configure /dev/net/tun: Operation not permitted"). Surfaced by the e2e-nightly cell-17 microvm LB launch on the ci-pseudo runner where the kernel enforces strictly; legacy EC2 launches on ci-single runners happened to attach successfully despite the same code path. Adding `user $current group $gid` to `ip tuntap add` ensures the tap is owned by whatever uid the daemon (and its qemu children) run as. Root daemons skip the flag — they have CAP_NET_ADMIN ambient. * fix(e2e): redirect lb probe chatter to stderr so stalls are visible test_http_from / test_tcp_from echoed progress to stdout, which the callers captured into *_OK vars via $(...). Result: 6+ minute retry loops ran silently in CI, and the "$ok" comparison never matched "true" because the captured value contained the full multi-line chatter. Reachability pass/fail counters were also lost in the subshell. Redirect helper output to stderr, leave only the true/false flag on stdout, and move pass/fail to the caller so they actually count. * fix(microvm): don't /32 AdvertiseIP via mgmt for internet-facing single-node buildMicrovmNICs overrode resolveMgmtRoute's deliberate empty return for the single-node internet-facing case by falling back to (MgmtBridgeIP, AdvertiseIP). That installed a /32 route to AdvertiseIP via the mgmt NIC inside the LB microvm. On single-node where the AWSGW host IP equals the WAN client IP, the /32 also catches reply traffic for inbound HTTP/TCP to the LB's floating IP — replies leave via the mgmt NIC with src=data-NIC IP, the host kernel drops the asymmetric flow, and same-chassis ingress hangs. resolveMgmtRoute already encodes the correct policy: - multi-node any scheme → explicit MgmtRoute{Gateway,Target} - single-node internal → fallback to (MgmtBridgeIP, AdvertiseIP) - single-node internet-facing → empty (reach AWSGW via VPC SNAT) Honor it. Matches the legacy lbVMUserData path, whose own comment flagged the stolen-WAN-return-path symptom this introduced. Test covers all three branches. * feat(ansible): dev-deploy playbook for in-place artifact swap Fills the gap between ansible-dev-install (skips when already installed) and ansible-dev-reset (full teardown + reinstall + smoketest). Mirrors Makefile `deploy` target: rebuild + replace binaries and microvm artifacts on a running node, restart spinifex.target, wait for awsgw. Skips setup.sh (no user/dir/systemd-unit changes) and smoketest so iteration is fast. Use ansible-dev-reset when changing systemd units, helper scripts, logrotate, or setup.sh itself. Adds: - roles/deploy/tasks/main.yml — install spx + nbdkit plugin + microvm artifacts, with plugindir auto-detection via `nbdkit --dump-config`. - playbooks/dev-deploy.yml — chains build + deploy roles. - Makefile ansible-dev-deploy target. - README playbook table + "when to use which" guidance. * chore(microvm): remove legacy LB Alpine image build pipeline Phase 3 microvm removal — strip the PC-machine LB image build artifacts that direct-boot no longer needs. The microvm initramfs already carries lb-agent + HAProxy via build-microvm-image, so the lb.conf manifest, lb-setup.sh chroot script, build-lb-image target, dev-install / reset-dev-env import steps, and ansible LB-image tasks all become dead. build-system-image stays — other manifests (e.g. GPU images) still use it. * refactor(microvm): remove legacy PC-machine LB boot path Phase 3 — direct-boot becomes the only ELBv2 LB launch path. - Drop SystemInstanceInput.DirectBoot / ImageID / UserData (every LB caller set DirectBoot=true; legacy cloud-init userdata is gone). - Drop ELBv2ServiceImpl.lbVMUserData, systemAMI* fields, SetSystemAMIFunc, getSystemAMI, and the lazy-AMI wiring in daemon.startCluster. - Collapse CreateLoadBalancer to a single direct-boot launch input. - Collapse daemon.LaunchSystemInstance: drop AMI validation, ImageID on RunInstancesInput, UserData base64 encode, GenerateVolumes branch. buildDirectBootConfig hardcodes /usr/share/spinifex/microvm and isa-serial=on (microvmMachineType no longer takes a flag). - Drop MicrovmConfig from config.DaemonConfig and the [daemon.microvm] section from the spinifex.toml template. - Prune dead tests (lbVMUserData_*, PCMachine_Unchanged, AMI resolver paths) and refresh stale comments that still mentioned lbVMUserData. ARM hosts have no LB launch path until Phase 4 adds direct-boot on virt — CreateLoadBalancer on aarch64 fails naturally at QEMU exec. * docs: drop lb-alpine catalog entry from cluster setup guide LB system instances no longer use an AMI — direct-boot microvm artifacts ship in the spinifex package. * feat(ui): remove LB image import banner + LB-AMI gating Direct-boot microvm artifacts ship in the spinifex package, so the LB system AMI no longer exists — the "Load balancer image not imported" banner and the Create-button disable on the LB list / wizard pages can never fire. Strip: - lb-image-missing-banner.tsx component - hasLbImage / isLbImage / LB_IMAGE_NAME / LB_MANAGED_BY_VALUE from system-managed.ts (isSystemManagedImage stays — still used by the EC2 Images pages to hide system AMIs) - Banner gating + ec2ImagesQueryOptions imports from the LB list and wizard pages - The corresponding tests that asserted the banner appears * revert: restore lb.conf + lb-setup.sh for mulga install pipeline Phase 3 microvm removal (commit 6313d74) deleted these too early — the mulga install/iso artifact builders (scripts/tofu-cluster/build-install -artifacts.sh, scripts/tofu-iso/build-iso-artifacts.sh) still invoke scripts/build-system-image.sh scripts/images/lb.conf, breaking e2e with "ERROR: Manifest not found: scripts/images/lb.conf". Restore both files verbatim so the legacy pipeline keeps building until the mulga-side migration to microvm-only artifacts lands post-merge.	2026-05-18 15:48:04 +10:00
joshsiv-mulga	26c2c9fa49	fix(setup): replace /dev/stdin in tmpfiles.d write to work without /dev mount in chroot	2026-05-14 15:01:14 +10:00
tomnewton-mulga	28ca689130	feat: VFIO GPU passthrough (#188) ... * feat: add GPU discovery package (mulga-992) Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group member lookup, and a PCI device ID model table. Vendor-specific enrichment via nvidia-smi is attempted opportunistically and falls back to the built-in table when unavailable. All discovery and IOMMU logic is covered by unit tests using mock sysfs trees. * feat: add VFIO bind/unbind and GPU pool manager (mulga-993) Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device driver switching, and gpu.Manager which owns the claim/release lifecycle for the GPU pool including full IOMMU group binding, rollback on partial failure, and marking entries unavailable when rebind fails after VM termination. * feat: add G5 instance type generator with GPU model support (mulga-994) - Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup - Add generateGPUTypes to emit g5.* types with GpuInfo populated - Add IsGPUType helper for consumers filtering GPU-capable types - Update DetectAndGenerate to accept gpuModels []GPUModel parameter * feat: integrate GPU passthrough into daemon lifecycle (mulga-995) - Add gpu_passthrough config flag; discover GPUs and build manager at startup - Gate g5 instance types on GPU pool availability via canAllocateCount - Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device - Release GPU (VFIO unbind, rebind to host driver) in stopInstance - Re-claim GPU on daemon restart when QEMU process is still alive - Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS) - Fix Claim to use stored OriginalDriver when device is already vfio-pci bound * feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997) Extends build-system-image.sh to support Ubuntu alongside Alpine. Key differences handled: partition table (parted resizepart p1), bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs rc-update. DISTRO defaults to "alpine" for backward compatibility. Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script that installs nvidia-headless-550-server with DKMS pre-built against a pinned linux-image-generic kernel, blacklists nouveau, and prevents unattended-upgrades from invalidating the pre-built module. * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * feat: spx admin gpu status/enable/disable commands (mulga-1000) - admin.SetGPUPassthrough: Go implementation of the idempotent TOML write (replaces the inline Python heredoc in spx-test-gpu.sh) - spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough activation, and advertised g5 instance types via NATS NodeStatusResponse - spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP, polls until daemon confirms GPUPassthrough=true - spx admin gpu disable: refuses while GPU instances are running, otherwise writes TOML, sends SIGHUP, polls until confirmed - spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable * fixed admin * fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter the passthrough pool. enrichFromLspci() uses the system PCI ID database via lspci -vmm to populate the model name for any card not in the hardcoded table, without needing nvidia-smi or driver tools. * fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of file permissions. QEMU subprocesses also need char-vfio rw access to open /dev/vfio/N for the passed-through device. * fixing gpu bining permissions * working state of gpu passthrough on phyiscal box * installer grub image fixes * made gpu-passthrough more user facing * fixed banner * fixed banner * banner fix * gpu passthrough docs * preflight tweak * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * fixes after running /review-branch * feat: re-wire GPU claim and device injection after vm.Manager refactor GPU passthrough claim (VFIO bind) and QEMU device injection were lost when dev's vm.Manager refactor (#185/#186) moved StartInstance/ CreateQMPClient into vm/lifecycle.go. Re-add both in their correct new homes: - daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both RunInstances and StartStoppedInstance handlers; release on failure - vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag derived from instance.GPUXVGAEnabled when GPUPCIAddress is set * fix: Create spinifex user ~/.ssh and keys in firstboot --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>	2026-05-06 09:52:23 +10:00
joshsiv-mulga	9b9bcc5b8f	fix(setup): mask dhcpcd.service even when apt install skipped ... bc33c72 added the mask but placed it after the SKIP_APT early-return. CI bootstrap runs setup.sh with INSTALL_SPINIFEX_SKIP_APT=1 against runners that already have dhcpcd-base, so the mask never ran. On Debian-13 the auto-enabled dhcpcd.service kept binding br-wan and racing vpcd's nclient4, reproducing the mulga-siv-32 DORA timeout on the public-IP allocation path. Lift the disable+mask out of the apt branch so it runs unconditionally.	2026-04-29 16:10:59 +10:00
Josh Siviour	04f63ad8c6	feat(iso-installer): DHCP race fix + vpcd reconcile gate + ELB mgmt route + HashMAC + diagnostics (#172) ... * preflight tweak * fix: pin gateway LRP to link-local, end intra-cluster ARP collision Every VPC's IGW gateway-LRP claimed pool.GatewayIP || pool.RangeStart on attach. Multi-VPC clusters all GARP'd the same .160 with three different deterministic MACs on the shared br-ext localnet, flapping the upstream ARP cache and dropping 100% of inbound traffic to dnat_and_snat external IPs (mulga-siv-26). handleIGWAttach and reconcileIGW now unconditionally set the gateway router port to 169.254.0.1/30. The default route's OutputPort already pins the LRP, so the WAN nexthop need not be on-subnet. Pool IPs stay allocated only for per-VM dnat_and_snat external addresses, which are unique by construction. Self-heal: RetrofitAllGatewayPortNetworks runs at startup and rewrites stale Networks in place via the new UpdateLogicalRouterPort path. Needed because Reconcile gates IGW work on "ext switch missing", so a cluster shipped with the buggy CIDR never re-enters reconcileIGW. Regression guards: TestTopologyHandler_IGWAttach_MultiVPC_NoIPCollision and TestReconcileIGW_RewritesStaleGatewayPortNetworks. Existing tests updated to assert link-local and OutputPort=&gwPortName. Log field gateway_ip renamed to lrp_network to signal the semantic shift. * fix: rewrite utils.HashMAC to SHA-256 / 46-bit output The old 24-bit rolling hash with literal class prefix had birthday-paradox collision probability of 1% at ~580 ids and 50% at 4096 — well within the range of a single deployment's resource population (LRPs, ENIs, dev/mgmt NICs). On a shared L2 segment, two clusters could trivially produce duplicate MACs, breaking neighbour discovery and DHCP. Switch HashMAC to SHA-256 across the entire input, taking the low 6 bytes and forcing the IEEE 802 reserved bits (bit0=0 unicast, bit1=1 LAA) on the first octet. Output entropy goes from 24 bits to 46 bits — 1% collision boundary moves from ~580 to ~1.2M ids. Drop the redundant prefix arg: it was no longer the literal output prefix under the new scheme, just folded into the hash input. Class disambiguation moves to the caller, where it's explicit. Three of four callers had nothing to disambiguate (eni-, subnet-, gw-vpc- ids are already unique); dev and mgmt NICs share instance.ID and now compose "dev:" / "mgmt:" tags. Also consolidate vpcd.generateMAC, which was a local duplicate of the broken algorithm rather than a HashMAC wrapper, into a HashMAC delegation. Plan: docs/development/bugs/vpcd-mac-collisions-shared-l2.md Bead: mulga-siv-27 * fix: pin HashMAC first octet to 0x02 Previous impl forced only the IEEE 802 reserved bits (bit0=0 unicast, bit1=1 LAA) on the first octet, leaving the upper 6 bits hash-derived. Technically valid LAA unicast, but produced MACs like 4a:.. and fe:.. that visually encroach on vendor OUI space and tripped CI/ops checks expecting all generated MACs to start with 02:. Pin byte 0 to literal 0x02; hash bytes 1-5 from sha256[0:5]. Output entropy drops from 46 to 40 bits — birthday-paradox 1% collision at ~150k ids, still 256x the old 24-bit bound (~580). * chore(e2e): dump cross-chassis OVN state on guest SSH failure Adds chassis registrations, port_binding claims, gateway_chassis rows, ovs-vsctl topology, Geneve tunnel state, and br-int tunnel flows to dump_guest_ssh_diagnostics so we can pinpoint where the cross-chassis dataplane stalls (cr-gw chassis vs remote chassis with the VM bound). * fix: gate vpcd startup reconcile behind NATS KV leader election N vpcds in a multi-node cluster all hit Get-then-Create on Logical_Router with no atomicity, producing duplicate rows that ovn-nbctl rejects with "Multiple logical routers named '...'. Use a UUID." Add AcquireReconcileLeader using NATS KV CAS (kv.Create on a single key in spinifex-vpcd-reconcile bucket). Only the elected vpcd runs Reconcile, ReconcileFromKV, and the two retrofit passes — others Subscribe and rely on the vpcd-workers queue group for runtime events. MaxAge=60s on the bucket caps recovery time when a leader crashes mid-reconcile. Falls through (elected=true) when KV is unavailable so first-boot does not deadlock on a missing JetStream. mulga-siv-29 * fix(elbv2): force mgmt host route for internal-scheme LBs (mulga-siv-30) Single-node CI e2e Phase 3b ALB internal hangs in 'provisioning' for 270s because the lb-agent inside the ALB VM has no return path to AWSGW. The gateway-URL decision tree in daemon.wireLBAgentConfig picks advertiseIP for single-node and deliberately skips adding the /32 mgmt host route — the WAN-return-path concern in the comment is real but applies only to internet-facing LBs, which have an EIP and an OVN dnat_and_snat pair that gives reply traffic a way home. Internal LBs have no EIP / no SNAT, so when the agent dials advertiseIP the reply targets the VM's private subnet IP from the host's WAN with no route back. Make lbVMUserData scheme-aware. New helper resolveMgmtRoute returns the existing (MgmtRouteGateway, MgmtRouteTarget) when set (multi-node case) and falls back to (MgmtBridgeIP, AdvertiseIP) when scheme is internal — only internal LBs need the /32 route, so internet-facing single-node still gets no bootcmd and the host-initiated WAN path stays clean. Daemon now exposes MgmtBridgeIP and AdvertiseIP unconditionally so the fallback has the inputs it needs. * fix(daemon): stub sudoCommand in TestSetupComputeNode_ValidatesArgs (mulga-siv-31) The test was calling real SetupComputeNode("chassis-test", ...) which shelled out via sudoCommand to set external_ids:system-id=chassis-test on the host's live OVS. On a dev box with OVS installed this overwrote the persistent UUID (owned by openvswitch-switch and re-applied only on reboot), so vpcd's discoverChassis filtered out the legitimate SBDB chassis row as 'stale local' and crashlooped — taking out all VPC ops including ALB ENI provisioning. Bind sudoCommand to a var so tests can swap it. Stub it in the test to return exec.Command("/bin/false") and assert SetupComputeNode returns an error. The test's intent (signature + error-path coverage) is preserved; nothing on the host is touched. * chore(vpcd): log DHCP Acquire concurrency + duration (mulga-siv-32) Diagnostics-only change to surface whether nightly e2e cell-3 / cell-8 DORA timeouts correlate with concurrent Acquire calls on the same bridge. Tracks per-bridge in-flight count and wallclock duration of each Acquire; logs WARN on failure with both fields, DEBUG on success. No behavioural change. * ci(e2e-nightly): dump cluster journals on failure (mulga-siv-32) run-multinode-e2e.sh dumps node journals via its own EXIT trap, but run-cert-e2e.sh / run-lb-e2e.sh / reboot-e2e do not. When those scripts fail the cluster is destroyed in Teardown without surfacing any daemon logs, blocking root-cause analysis. Add a workflow step that runs after the test scripts but before Teardown when a previous step failed, ssh'ing to each node and pulling the last 200 lines of every spinifex-* unit. Best-effort: skips gracefully when inventory or nodes are unreachable. Diagnostic-only; no behavioural change to the test path. * fix(setup): mask dhcpcd.service to stop it racing vpcd's DHCP client (mulga-siv-32) Debian 13 (Trixie) auto-enables a standalone dhcpcd.service when dhcpcd-base is installed. It binds br-wan and races vpcd's in-process nclient4 client for upstream OFFERs, draining the lab DHCP pool with duplicate leases and causing intermittent DORA timeouts in the e2e public-IP path (worst on cell-8 real-multi/dhcp/debian-13). The ISO installer already masks the unit at install time (cmd/installer/install/install.go:316). The binary install path used by e2e nightly missed it, so debian hosts installed via setup.sh ran with both DHCP clients live. Mirror the ISO mask here. Ubuntu hosts are unaffected (dhcpcd-base does not auto-enable dhcpcd.service on Noble). * ci(e2e-nightly): also dump dhcpcd.service state on failure (mulga-siv-32) Confirms or rules out the dhcpcd-vs-vpcd race hypothesis. Pulls the dhcpcd.service journal alongside the spinifex-* units, plus systemctl is-enabled / is-active and the DHCP client port (UDP 68) listener so we can see directly whether dhcpcd is running and bound to br-wan when an Acquire times out. --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>	2026-04-29 14:38:29 +10:00
Ben Duncan	27d7fdd12b	Latest changes re ISO and binary install path	2026-04-20 15:14:32 +10:00
Ben Duncan	b591c04880	firstboot tweaks	2026-04-20 11:27:49 +10:00
Ben Duncan	a64ee2b2da	ISO tweaks	2026-04-17 14:22:34 +10:00
Ben Duncan	402a04e1b9	Merge branch 'feat/iso-installer' of github.com:mulgadc/spinifex into feat/iso-installer	2026-04-16 09:29:15 +10:00
Ben Duncan	25d02707d0	ISO tweaks	2026-04-16 09:29:00 +10:00
Julian Sommer	7d6e0780b5	feat: action rate limiting (#140)	2026-04-15 12:19:30 +10:00
Julian Sommer	1ad03bb518	feat(gateway): per-IP auth failure rate limiting (#138)	2026-04-15 09:56:04 +10:00
Julian Sommer	a4ae813e38	fix: recursive chmod on shared data dirs so existing image files are group-readable ... chmod 0770 without -R only affected the top directory; pre-existing files (e.g. images imported while owned by another user at mode 0600) stayed unreadable to the spinifex group, breaking image import on upgrade paths.	2026-04-14 11:05:01 +10:00
Julian Sommer	e0118975d1	fix: setup.sh	2026-04-14 10:17:01 +10:00
Julian Sommer	36e3481ca3	fix: enable spinifex.target so services autostart on boot	2026-04-14 10:14:09 +10:00
Julian Sommer	515c313e07	fix: setup.sh newgrp	2026-04-14 10:07:59 +10:00
Julian Sommer	48c90b795a	feat: migration framework (#127)	2026-04-10 14:34:15 +10:00
Julian Sommer	e27d909e8d	feat: move NBD sockets to /run/spinifex/nbd/ and enable PrivateTmp (#119)	2026-04-09 11:11:54 +10:00
Julian Sommer	9a7e914d28	feat: awsgw directory (#117)	2026-04-08 15:19:29 +10:00
Julian Sommer	a56a2baf1f	feat: privilege separation (#114)	2026-04-07 16:22:19 +10:00
Ben Duncan	e5b6204875	Fix service helper sequence	2026-04-02 13:53:29 +10:00