118 Commits

Author	SHA1	Message	Date
Julian Sommer	47fa76e09d	chore: coverage badge	2026-10-07 13:06:41 +10:00
Julian Sommer	4e8e854fc0	fix(ci): stop dev release tags nesting the previous dev tag ... The dev release tagged itself with git describe plus -dev, and describe resolved to the previous dev tag, so every tag appended to the last until the tarball name passed the 255-byte filename limit. Describe against release tags only, here and in the Makefile and bare-metal VERSION.	2026-10-06 11:16:25 +10:00
Ben Duncan	c6841ce1f0	feat(oci): separate using an instance principal from creating one (#1157) ... * feat(oci): separate using an instance principal from creating one The allocator needs OCI credentials at runtime, and a node with none forms, passes every health check, then refuses every launch wanting a public address. Instance principal is the credential route that puts no key material on a node, but it needs a dynamic group and a policy, and those are tenancy-root resources the compartment-scoped user the rest of the configuration runs as cannot create. enable_instance_principal conflated the two, so the better credential route was only reachable by a tenancy admin. instance_principal now takes off, adopt or create: adopt selects the credential and creates nothing, because the matching rule covers the whole compartment and so names no instance to reference. setup-identity.sh owns the create case, with its own state. validate-topology.sh destroys its state at the end of every run, so holding tenancy resources there would let a teardown delete the tenancy's policy. Also package install-node.sh and smoke-test.sh, so a cluster can be formed from a curl | bash install with no checkout. smoke-test.sh is not optional: formation scp's it from its own directory for the post-formation verification. * feat(oci): take the credential handoff from an operator hook, and widen the workflow name An API-key deployment needs a credential on each node, and nothing in this repository can supply one: a key in user-data is readable from instance metadata for the life of the instance, and this host runs other people's guests. A manual install does not survive either, because a topology destroys itself on success. So the harness calls an operator-owned executable after formation and before the pool, the only window where a node has /etc/spinifex and has not started the allocator. Instance principal needs no handoff, so the hook is skipped there. The workflow is renamed for the vendors it will cover rather than the first one. Nothing is orphaned in the Actions UI: it has never been on the default branch and has never run, so there is no history attached to the old filename. * docs(oci): the API-key path needs the config file, not just the PEM oci_config_file names an ordinary OCI SDK config and nothing creates it, so a node holding only the key forms and then fails every allocation with no stated cause. * fix(ci): the runner context is not available in a job-level env block A workflow referencing it there is rejected before any job starts, which is why the rename produced a failed run attributed to the push. The reference had never been validated: a dispatch-only workflow that is never dispatched is never parsed. * fix(deps): bump containerd to v2.4.1 for GO-2026-6597 The image-pull DoS via crafted OCI index graph amplification is reachable from the ECS agent's containerd puller, so the vulnerability scan fails preflight on v2.4.0. A patch release with no API change. * fix(e2e): probe guest public addresses from outside the cloud's own network A VCN does not reflect its own public addresses back into itself, so a node asking for one gets nothing however well the datapath works. The workbook driver runs on the first node, so every assertion that curls or SSHes a guest's public address failed on OCI against a cluster that was serving those addresses correctly to the internet. The driver now takes E2E_PUBLIC_PROXY, a SOCKS5 proxy used for public addresses only, and validate-topology.sh supplies one with a remote dynamic forward on the connection that launches the driver. Egress is then the host running the validation, which is outside the VCN and is where a customer sits. Unset leaves every other platform on a direct probe, which is already correct there. Also adds --version TAG for a repeatable install, separates the workflow's auth choice from its external-pool choice so an API key can run with the allocator live, and captures allocator diagnostics before the teardown destroys them. * ci: read the Go version from go.mod in the release workflows Both pinned 1.27.0 by hand while every other workflow reads the file, so the toolchain they build with drifts from the one the module declares the day go.mod moves. release.yml was already inconsistent with itself: its second setup-go step reads spinifex/go.mod. * feat: publish the OCI topology results as tables on the run page A green tick on one job with 4,000 lines of log behind it is confidence for whoever wrote the harness and nothing to anybody else. Someone assessing whether Spinifex works on their cloud should be able to see that it formed and served guests on a single VM, on three VMs and on bare metal without opening a log. record now takes <gate> <status> [detail] and writes results.tsv beside the human block, so a gate's status is a field the script sets rather than a regex over prose that reads fine and matches wrong. record_workbooks turns the driver's own go-test-shaped lines into workbooks.tsv, and runs on the failing branch too: which four passed is most of what the fifth failing means. e2e-oci.sh --summary-only renders both tables to stdout and to GITHUB_STEP_SUMMARY, and the workflow calls it under always() before the sweep, because the sweep destroys the state directories the tables are read from. A skipped gate gets its own mark, and a topology that never ran says so: the one way a results table can do real harm is a reader taking an absent row for a passing one. * docs(oci): cut the deploy-by-hand appendix, and validate the config for real The appendix was 444 of 1014 lines restating an install the Terraform already does, which made the document read as though deploying by hand were a normal route. It is not: the configuration builds the VCN, the volumes and both VNICs per node and renders the cloud-init, in the same order every time. What is genuinely OCI-specific is four things, each of which already has a section, so the replacement names those and sends everyone else to the single-node and multi-node install guides. Verify end to end told the reader to ping a guest's public address from the host, which cannot work: a VCN does not route its own public addresses back into itself. That is the finding this branch is built on and the document was asserting the opposite. It now says where to probe from and why, and carries the allocator's `resolved the external VNIC` check, which existed only in the cut appendix and is the one line proving the credentials and compartment are right. `spx config validate` is not a command -- spx has admin, get, service, top and version -- so that line could only ever fail and its output and status were both discarded. It is replaced by parsing the TOML, which is what was wanted: the remap and the pool append are textual edits, and a broken result would otherwise surface as every service failing to start with nothing saying why. The remap now checks its own substitution count for the same reason, since re.sub with no match rewrites the file unchanged and exits 0. Publication of the run-page tables moves to one owner. GITHUB_STEP_SUMMARY is per step, so appending from the run as well as the publish step would have put two copies of every table on the run page. * fix(oci): check the credential hook before building anything The hook is a path from a CI variable, and whether it exists on the runner is knowable before Terraform runs. Checked where it was used, a wrong path cost the apply, the install and the formation first -- forty minutes and a bare-metal bill to be told a file is missing. Nothing about the check depends on the apply, so it moves to the top, where --dry-run reaches it too. * fix(oci): the summary must say what was built and what was not tested The first CI run headed its tables "Published release `latest channel`" over a build of a feature branch, which is the one error in a results table that would mislead a reader who trusted it. The header now states a tree build as a tree build, and names how the allocator authenticated. A run with no external pool says so in the header rather than leaving it to a SKIPPED row further down. Public-address allocation is the claim a green table is most likely to be read as making, and it is exactly the claim such a run cannot support. Both flags are passed to the publish step too, so the summary rendered after a failure describes the same run as the one that failed. Raise the dev prerelease prune from five to twenty. A pinned dev tag is how an external evaluator installs a specific build, and five is about a week of them -- short enough that the tag someone was handed stops resolving while they are still using it. * docs(oci): the workbook table is a snapshot, and the nightly is the current answer It is dated against a build five releases of work ago, and the nightly now runs the same workbooks on all three topologies and publishes its own table. Say which one to read rather than letting a stale table be taken as current. * ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file A workflow's displayed name is captured when GitHub first registers it and is not refreshed afterwards. Both OCI entries were registered from a run that could not be parsed -- a job-level env block referencing runner.temp -- so both list as their path, which reads as a missing name: key and is not one. The 2026-10-02 run was green on all three topologies and the entry still read as a path, with updated_at unchanged from the broken first run, so fixing the file is necessary and not sufficient. A new path is a new registration, and its first run reads a valid file. Dropping -nightly is also more honest: there is no schedule: here, because GitHub registers one only from the default branch, so a cron could never run dev and a feature branch as separate nightlies. * Revert "ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file" The rename makes the workflow undispatchable. `gh workflow run` resolves a filename against the default branch, and the raw dispatch endpoint returns 404 for a path GitHub has not registered -- checked twice, several minutes apart, so it is not an indexing delay. The old name works only because it is already registered, and an unrunnable workflow is a worse problem than a mislabelled one. The sidebar label needs a fresh registration, which a rename cannot provide from a feature branch. It corrects itself when the file reaches main, where GitHub re-reads it; the alternative is deleting the dead registrations' runs, which is an operator's call. * fix(ci): never publish the credential hook's output as an artifact The hook is the operator's own executable, and the one thing known about it is that it handles a private key. Its stdout and stderr are captured to credential-hook.log, which the artifact's *.log glob was collecting into a downloadable archive with a fourteen-day retention. Excluded rather than trusted to have been quiet. A hook that prints its key once while someone is debugging it should not thereby publish it. * fix(oci): the hairpin probe measured nothing on one node, and claimed too much With a single host, HOSTS[0] and HOSTS[-1] are the same address, so the node connected to itself -- which always succeeds. It reported "present" on vm-single and bm for that reason and not as a measurement. It now needs two nodes and says so otherwise. The claim it carried was also wrong. "A VCN does not reflect its own public addresses back into itself" was read off a workstation run whose node_client_cidr_allow_list was narrowed to one office address, which makes a node's own public address a non-permitted source: the refusal that follows is identical to an absent hairpin. With the committed 0.0.0.0/0 default the probe reports the hairpin present between two nodes, so the original reading was confounded and is retracted. None of this changes the remedy. A customer reaches a guest from outside the VCN, which is where the proxy's egress is, and probing from there is correct whichever of the two causes is in play -- 15 of 15 workbooks pass through it on all three topologies. What changes is what the documentation may assert: the guide now names both causes as indistinguishable rather than stating one as fact. * docs(oci): a reader could not run the first command, because nothing told them to clone The guide's first code block was `cd scripts/terraform/oci-spx`, into a directory no step had obtained. It also never mentioned the SSH keypair the driver refuses to start without, nor that the configuration is not on main or in v1.21.0 yet, so all three had to be inferred from the deploy failing. Restructured so the deploy comes first and reads as four steps and one command: get the configuration, choose one node or three, write the inputs, run it. The credential decision stays before step 4, because it changes an argument there and a node without one forms, looks healthy, and then refuses every launch that wants a public address. Everything else -- why OCI, sizing, the architecture, the IAM policy, quotas, the oci CLI, what Terraform builds, every variable -- moves under Reference, unchanged in substance. Adds what a successful run prints, naming `allocator ready` as the line to look for: every line above it appears on a node that cannot allocate, and that is the one failure the run does not announce. Also drops an empty verdict.txt committed from a local run. The directory the guide now tells customers to work in should not ship one run's artifacts.	2026-10-04 09:04:55 -05:00
Julian Sommer	a06eccfdab	test: fuzz awsec2query parser and add make fuzz ... Fuzz QueryParamsToStruct over query bodies decoded into EC2, ELBv2 and RDS inputs: no panic, list entries bounded by MaxSliceLen and the input size, and an entry past the first gap never changes the parse. make fuzz runs every Fuzz* target for FUZZTIME each.	2026-09-30 14:35:48 +10:00
Julian Sommer	7bbf78df78	chore: cleanup preflight	2026-09-30 13:52:47 +10:00
Julian Sommer	2d2d2f27ed	feat: model-driven request conformance in the integration suite (#1122)	2026-09-28 14:14:15 +10:00
Ben Duncan	7163f7d62d	feat(oci): run Spinifex on Oracle Cloud, and fix the public-address and NAT gateway lifecycle (#1108) ... * feat: ship uninstall-spx.sh and fix the OCI bring-up path Adds scripts/uninstall-spx.sh, the inverse of setup.sh: it removes the software where node-reset.sh clears state and leaves the node installed. An operator who installed with `curl ... | bash` had no teardown path at all. State teardown is delegated to node-reset.sh rather than reimplemented, and setup.sh now ships both to /usr/local/share/spinifex so they are reachable without a checkout. It refuses on a multi-node cluster and --yes does not override that: a flag meaning "don't ask me" must not also mean "ignore a safety property". /var/lib/spinifex and every apt package are kept unless asked for, and both opt-ins confirm separately even under --yes. Proved by a full uninstall/reinstall round trip on dev-oci, which found two bugs neither prod nor dev-prod can reach, because both are ISO-installed: node-reset.sh exited 2, silently, at its last step and after the wipe. setup.sh creates the spinifex *group* and the spinifex-* service users but never a *user* named spinifex, so `getent passwd spinifex` exits 2 and pipefail carried it into set -e. The uninstaller broke openvswitch-switch on its first run by removing the socket-perms helper while leaving the drop-in referencing it, failing the unit at step EXEC with status 203. Fixed on both sides: the uninstaller removes drop-ins it wrote into other units' namespaces and restarts anything it failed, and setup-ovn.sh writes ExecStartPost=- so a missing helper can never stop another package's daemon. setup-ovn.sh also learns the shape OCI needs: when --wan-bridge names a Linux bridge it links it to OVS with a veth pair instead of enslaving the NIC, so the VNIC keeps its MAC identity and netplan keeps owning the addresses. It now fails on OVS datapath errors instead of reporting success, and the --management demotion guard keys on /etc/default/ovn-central rather than on database presence, which the ovn-central package creates on every node. Tests: scripts/uninstall-spx_test.sh, wired into make test-build-scripts along with shellcheck over the four scripts that ship to every node. * feat: allocate external IPs from the Oracle Cloud API An OCI VNIC drops any packet whose source is not a registered private IP object on it, so an external pool cannot be range math there — the address has to be created through the provider before it exists at all. Add source = "oci" and the allocator behind it. One allocation is two OCI objects: a secondary private IP that rides the wire, and a RESERVED public IP attached to it that the internet reaches. Allocate returns the public address because that is what AWS semantics are about, and records the pair so the host plumbing and release path can find the private half. The OCI objects are created before the binding is written, and dropped after it on release — both orders chosen so a crash leaves a discoverable leak rather than a record that lies. Reconcile squares the two on startup and is required, not optional: a leaked reserved public IP bills and holds one of the 50 per-region slots with nothing referencing it. It never touches an object without our display-name prefix, because the reference host carries operator-created secondaries on the same VNIC. Resolve the VNIC from an interface name by MAC through IMDS rather than by Oracle's VNIC ordering, which is not a documented contract. Generalise ExternalIPAM.EnableDHCP into InstallAllocator so the OCI wiring is not a second single-purpose installer. * fix: refuse source = "oci" outside routed mode, and cover the SDK path An OCI VNIC accepts exactly one MAC, its own, and delivers every inbound frame to it regardless of which of its addresses the packet is for. Measured on the reference host: an internet source reaches 10.200.0.144 as router-MAC to VNIC-MAC, the public address never appears on the wire at all, and every off-subnet destination resolves to the same next hop. Pool mode puts a per-VPC gateway LRP MAC and a per-rule external NAT MAC on the uplink and ARPs for the address, which policy.go:50 justifies with "the Linux bridge learns the per-rule external MAC on the veth port like any other MAC". An OCI VNIC does not learn MACs; it accepts one. So a pool-mode guest's egress is dropped and its ingress never reaches OVN. Routed mode keeps the address on the host, which already wears the VNIC's MAC. Pinning every LRP to that MAC does not rescue pool mode: they share one external logical switch and OVN resolves inbound frames by an L2 lookup on it, so duplicate MACs work for one VPC and silently misroute past that. The refusal names the MAC reason, because a refusal that only says "use nat mode" reads as arbitrary and gets worked around. Also test the SDK request shapes against a local endpoint. Two of them are wrong in ways reading does not catch: pagination stopping early reports a partial VNIC to reconcile, which then drops live bindings, and a nil privateIpId on detach is omitted from the body entirely, so OCI reads the update as a no-op, returns 200, and leaves the address attached and billing. * fix: install host firewall rules where a distro's catch-all reject cannot precede them Oracle's Ubuntu cloud image ships a catch-all REJECT in both INPUT and FORWARD. Two Spinifex datapaths were silently dead behind it on any such host. EnsureNATEgressRules probed with -C and appended with -A, so its ACCEPT rules landed after the REJECT and never matched: 24113 packets on the reject, 0 on both accepts, and guests had no egress. The comment named the case it did not handle, anticipating a FORWARD policy of DROP — which an append does survive — rather than an explicit REJECT rule, which it cannot. Insert at the head, and drain existing copies first, because a rule an earlier build appended still satisfies -C and would otherwise stay stranded across an upgrade. Nothing opened INPUT for the IMDS endpoints at all. The responder binds .254 and .253 per instance on the ime- devices, so guest requests are locally delivered and meet INPUT. Requests arrived and no reply ever left; cloud-init spent 490s timing out and reported the endpoint as disabled. With one wildcarded accept the same image boots in 18.6s and finds its datasource. The natuplink e2e check had the same blind spot the code did, asserting -C and therefore presence but never position. It now asserts the accepts precede the first reject. * feat: OCI address pairs, and an IMDS remap so the host keeps its own metadata An OCI external address is two objects, not one: a RESERVED public IP that OCI 1:1-NATs upstream and never puts on the wire, and a secondary private IP on the VNIC that the wire actually carries. Every AWS-facing record holds the public half; every datapath object must hold the private one. The mapping goes in policy.natManager and nowhere else. AddEIP, DeleteEIP, PruneOrphanEIPs and the HostEIPBinder all converge there, and reconcile rebuilds the rule set from AWS-facing intent on every pass — so a correctly mapped rule installed further upstream is swept on the next reconcile and replaced with one matching an address that never arrives. The seam is a DatapathResolver defaulting to identity, so every non-OCI deployment is unchanged. A resolve failure is fatal: falling back to the public address would install a rule that reads correctly in ovn-nbctl and blackholes. ocinet.Lookup is the OCI side — read-only, KV-backed, no client and no credentials, because vpcd is a different process from the daemon that owns the allocator and the bindings record is all either would consult. Two live bugs the in-memory test store was hiding, both found by making the fake report ErrNotFound the way the real store does: - KVStore.Mutate could not create, so every pool failed its own first allocation after OCI had already created and billed the pair. - Reconcile treated a missing record as a failure, aborting the one pass that exists to find an object created before its binding was written. Guest IMDS must stay on 169.254.169.254, which is also where a cloud host's own metadata service lives — the first guest launch made it a local address and took the host's metadata and DNS with it. The datapath is OpenFlow steered, so the guest-facing addresses stay put and only the host-side endpoint address moves, with a PREROUTING DNAT on ime-+ translating and conntrack un-NATing the reply. Off by default; identity when unset. The IMDS firewall rules move to vpcd. They are wildcarded over every endpoint rather than per-launch work, and spinifex-daemon holds no iptables grant and no CAP_NET_ADMIN by design, so attempting them aborted every launch. * fix: name the real constructor in the IMDS bind-address error, and mark the NAT datapath test in-package The error string pointed at DefaultHostBindAddrs, which does not exist — a caller following it would have looked for a symbol that was never written. nat_datapath_test.go reuses nat_test.go's in-package fixtures, so it carries the //test:in-package marker the package check asks for. * docs: index what each e2e suite proves Written because a datapath comment was read as an end-to-end behavioural claim and the suite that already answered it was found only after the wrong answer had been given. TestVPCEgressPaths/NATBaseline is the authority on private subnet egress, and nothing pointed at it. * fix: source-route an EIP's reply out the VNIC that holds it An EIP's reply carries the EIP as its source. On a host that pins its uplink with a policy rule, the reply took the main table instead and left by the wrong interface — which a cloud VNIC drops, because the address is registered on a different one. Ingress reached the guest and nothing came back. Every bind now mirrors the uplink's rule onto the EIP. The selector is the rule rather than the interface: both OCI VNICs sit in one subnet, so "the interface owning the EIP's subnet" names two of them, and two candidate rules is a refusal rather than a guess. A host with no source routing gets no rule and no change. * docs: name the setup-ovn recipe routed mode actually needs on OCI The OCI guide told operators to run `--wan-bridge=br-wan`, which is mutually exclusive with `--nat-uplink` and deletes the spx-nat transit veth routed mode runs on. The node then refuses to bring the uplink up, with an error naming the flag the guide should have used. Several messages also claimed routed mode is outbound-only with no public IPs. That stopped being true when the routed EIP path landed, and it reads as "what you are about to do cannot work" during the one init the OCI path documents. * feat: let routed NAT form a multi-node cluster, and carry every pool through formation Lifts the --nodes >= 2 refusal on --external-mode=nat. The refusal carried no technical justification -- it was a v1 scope decision -- but routed NAT is the only datapath a cloud VNIC can carry, so "routed NAT is single-node" had become "Spinifex on a cloud host is single-node". It is not fixed by lifting it. Measured on three OCI nodes: the cluster forms, all chassis register and guest-to-guest traffic runs at 7.1-7.7 Gbit/s, but every node claims every EIP, nodes disagree on the VPC's gateway LRP, and only the gateway chassis reaches its own guests. init now says so on stderr rather than refusing, because the refusal made the defects unreachable and therefore unfixable. Distributing the EIP datapath is the actual fix. Lifting it exposed a second defect that stops formation outright. NetworkConfig held nine flattened Pool* scalars and the init node set PoolName = "wan" unconditionally; routed NAT renders two pools and the one that matters -- nat-transit, carrying the gateway-LRP range -- was dropped. Each joiner got a single pool named "wan" holding the transit gateway and the detected WAN prefix, and the leader re-rendered its own config from the same struct, so all three nodes were wrong. Pools is now []admin.PoolData in render order, and PoolData gains json tags because it is on the wire now rather than a template input. * feat(install-node): add --external-mode so a cluster can be formed in routed NAT install-node.sh could not express routed NAT at all. It required --external-pool/--external-gateway/--external-prefix-len and passed --external-mode=pool to every spx admin init unconditionally, so the only supported way to form a routed-NAT cluster was by hand. --external-mode nat refuses the three pool flags rather than ignoring them, and threads --nat-uplink into setup-ovn.sh -- without it the WAN interface check rejects a physical NIC during formation, which is correct in pool mode and wrong in routed mode, where nothing is bridged. * fix(ocinet): stop each node deleting its peers' OCI addresses on startup Reconcile compares the binding record against the private IPs on one VNIC. The record is one key per pool and therefore cluster-wide, so on more than one node every peer's binding looks exactly like a binding whose objects OCI has lost. Each node dropped them, and direction 1 on the owning node then collected the now-unclaimed objects as leaks -- deleting the reserved public IP and private IP of a running instance. Observed on the three-node OCI cluster: node2's startup pass logged "collected a leaked address" for an address it had allocated ten minutes earlier and "dropped a stale binding" for one living on node3's VNIC. The damage did not stop at the OCI objects. With the binding gone, Lookup.DatapathIP returns the public address unchanged, so the OVN dnat_and_snat row carries the public address instead of the private half. Guest egress then leaves the VNIC sourced from a reserved public IP, which OCI drops because it enforces the registered source per VNIC -- and EnsureEIPSourceRoute no-ops too, since a public address shares no subnet with any host address. One destroyed binding takes out both directions. Binding.VNICID already exists for exactly this ("how a node that comes back from the dead tells mine from moved on without me") and Reconcile never consulted it. A binding is now only dropped when it names this node's VNIC; one naming no VNIC cannot be attributed and is left alone rather than guessed at. * fix(host): find the EIP's source-routing table when the uplink is held as a /32 uplinkSourceRuleFor identified the interface owning an EIP's subnet by finding a host address configured with a prefix that contains it. On OCI both VNICs sit in one subnet, so br-wan has to hold its address as a /32 -- a /23 there puts the subnet's on-link route in the main table at metric 0 and takes the whole subnet off the primary VNIC. A /32 contains nothing but itself, so no rule was ever found, no source rule was installed, and the reply left by the primary VNIC where OCI dropped it for carrying an unregistered source. The address is not the only record of who owns the subnet. The rule names a table, and that table holds the on-link route. Consulted as a fallback, only when the address prefixes matched nobody, so a host that already resolved this the direct way keeps its existing answer -- prod and dev-prod are unaffected. Only scope-link routes count. A default route covers every address and would make every source-routing table an equally good answer. * docs(oci): correct the multi-node story and name the oci_vnic_id constraint The nat-mode warning said EIPs and default egress were both broken on more than one node. Default egress is fine: delete a guest's dnat_and_snat row and it reaches 1.1.1.1 in 2.3 ms from a guest on any of three nodes. Only the public-IP path was broken, and the three defects behind it are now fixed. What remains is not a bug but the shape of the datapath: routed NAT centralises the EIP path on the VPC's gateway chassis, ext-shared is a localnet so the other nodes cannot reach it, and OCI drops any frame whose source is not registered on the VNIC it leaves by. So every node's pool must pin oci_vnic_id to the gateway chassis's VNIC rather than resolving its own. Say so in §5.3, where an operator writing the pool will read it, and in §9, where the cost is that a gateway change kills every public IP until the pools are repointed. The single-node cap itself is gone from the doc: a three-node cluster has been formed and measured, and the warning proceeds rather than refusing. * feat(nat): distribute the routed-NAT EIP path so it works on more than one node Routed NAT processed every EIP on the VPC's gateway chassis, and ext-shared is a localnet, so the transit segment a /32 route pointed into existed only on that one node. Measured on three nodes: all three claimed all three EIPs, the nodes disagreed on the gateway LRP, and only guests on the gateway chassis were reachable. The mechanism to fix it is not new. Pool mode already distributes EIPs by putting external_mac and logical_port on the dnat_and_snat row and letting OVN's ARP responder answer for the external IP on the external segment. Routed mode has an external segment too, the spx-nat veth pair, and it being per node is the distribution: only the node running the instance can answer ARP on it. So a routed rule is now distributed on the same terms as a pool one, and its host route becomes on-link on the transit veth instead of via a gateway LRP that may be on another chassis. bindHostEIP stops demanding that address for a distributed rule, which is what previously failed the bind on every node but one. vpcd plumbs only the ENIs local OVS reports as its own, asked of OVS rather than the Southbound DB because the question is about this host. The gateway LRP keeps its transit-pool address. Default egress routes through it and works across three nodes today, so making it link-local as pool mode does would break a working path to fix one distribution has already fixed. Pool and static are unmoved, and a test asserts that against the same input rather than leaving it to inspection. A rule with no MAC stays centralised, so a cluster part-way through an upgrade is never in a state neither shape handles. Beads: mulga-opyl7, mulga-la0pe * fix(nat): make routed-EIP host plumbing work on every node Three guests on three OCI nodes, one guest per node: only one was reachable. Two defects, both invisible on a single node. The transit veth took a random MAC per node, but OVN holds one MAC binding for the transit nexthop across the whole cluster — so every chassis addressed its egress to whichever node seeded the binding, and the other kernels dropped it at L2. The pair is node-local, so one locally-administered address everywhere is unambiguous; setup-ovn.sh pins it at creation and in the netdev unit, and EnsureUplinkPort repairs a veth made before that. Host routes and proxy-ARP also rode the leader-gated reconcile pass, so a node that never won the lease never plumbed its own guests' EIPs. BindHostEIPs does that half on its own loop on every node; Bind already leaves alone an EIP whose ENI is elsewhere, so two nodes never contend. An unresolvable address skips the sweep rather than shrinking it. Pool and static are untouched: the new pass is a no-op outside routed mode, and a test asserts it. Also adds --teardown and --dry-run to setup-ovn.sh, for re-running it on a node that is already tangled. It deliberately spares the NB/SB databases, /etc/spinifex, the system-id and the packages, and refuses to del-br a Linux bridge. * fix(nat): assert the transit MAC where vpcd actually runs it EnsureUplinkPort is never called in production — host.Wiring is constructed nowhere — so the repair landed in a path the deployed daemon does not take, and three nodes came back with three different MACs. vpcd re-asserts it beside the NAT egress rules instead, which is the routed-NAT branch that does run on every start. * fix(vpcd): make uplink-mode detection say what it saw, and check it detectBridgeMode logged only the branch it took, so a node detected as veth when it was wired for nat looked identical to one that was right. It now reports both probes every time, errors when a node carries both uplink veths (set up twice, and --teardown is the way out), and errors when external_mode and the wiring disagree — the datapath follows the wiring, so the config silently loses that argument. An explicitly configured bridge_mode gets the same cross-check, since nothing else ever compared it against the wiring. The vpc-networking doc now opens nat with "start with pool unless one of these applies" and gains a section on nat across more than one node; the admin warning for --external-mode=nat on two or more nodes says what was verified by hand and what is still uncovered. * test(nat): prove routed NAT delivers each node's own guests, on a cluster natuplink ran single-node only, which is exactly where the two defects that broke three nodes are invisible. It now admits ModeMultinode, and phase 9 takes a second shape there: a guest per node in a spread placement group, and for each one — its public IP plumbed by the node running it and by no other node, the route on-link with proxy-ARP answered there, a dnat_and_snat row carrying external_mac and logical_port, a TCP handshake in from another node, and the guest's own console verdict for egress. Every node's transit veth is checked against the one cluster-wide MAC before any guest boots. Single-node phase 9 gains the same row and on-link assertions, so the shape is held on both topologies. Two gates come from what three OCI nodes actually do: every node holds a gateway_chassis row for every VPC (the active one is the highest priority) and every node installs the VPC ingress route, so only phase 7 needs gating — it runs where the gateway port and the guest are both local, and logs which one is missing otherwise. Cell 20 is real-multi x nat with its own nightly step, leaving the permutation cells' suite set alone. * test(nat): run the VPC egress paths in nat mode, not pool mode only The single suite gated TestVPCEgressPaths and TestWANEgressSMTPBlock on external_mode = "pool", with a comment saying nat clusters have no EIP/NAT-GW support. They have had EIPs since the routed path landed, so the gate now asks whether the cluster can allocate public addresses at all — pool, or nat with a non-transit pool configured. The nat-single cell runs TestVPCEgressPaths beside natuplink, which is the only nightly coverage of a private subnet's isolation and its NAT gateway in routed mode. natuplink never creates a private subnet. * docs(config): list nat among external_mode values * fix(ci): renumber the multi-node nat cell off the bare-metal job Cell 20 is already the bare-metal single-node job, gated on the same cells input but declared in the workflow rather than e2e-cells.json. A matrix entry with the same number runs two unrelated jobs under one name, and naming 20 in a dispatch selects both. * feat(images): add Oracle Linux 10.1 and 9.8, and pin their catalog digests Oracle publishes its qcow2 checksums inline on an HTML page and ships no sums file, so the four new entries carry the expected digest in the catalog rather than a URL to fetch one from. Images gains ChecksumDigest for that, and the catalog import path verifies against it and says which of the two sources it used, so a recorded digest never reads as a fetched sums file. Pinning is safe here and only here: each URL names an immutable build (b291, b293, b178, b182) and never moves. aarch64 has a -kvm-cloud- build alongside the plain -kvm- one; the cloud build is the cloud-init image and is the one that matches x86_64's -kvm-. TestImageCatalogReachable HEADs every URL in the catalog from the general single-node suite. These artifacts live on hosts nobody here controls, so an upstream that retires a build breaks `spx admin images import` for a customer and nothing else in the suite would notice. It also asserts that every entry sets exactly one checksum source, and that no pinned entry points at a "latest" URL whose bytes would change under it. * fix(formation): carry bridge_mode onto the leader and every joiner applyNetworkConfig copied external_mode, external_interface and the pools out of the formation NetworkConfig but never bridge_mode, so a multi-node cluster wrote a spinifex.toml without the key while single-node init wrote it. Every node then fell back to autodetecting a mode that init had pinned. Nightly cell 30 caught it: TestNATUplink failed at the config phase on a spinifex.toml that did not contain bridge_mode = "nat". Deriving it from the mode rather than adding a NetworkConfig field keeps the leader and the joiners on the same rule single-node init already uses. * feat(ui): show how to ssh into an instance, and which user to use An instance detail page gave the addresses and the key pair name but never the login user, which is the one part of the command nobody can derive. It belongs to the AMI, not to Spinifex: nothing in the launch path creates or renames an account, so stock cloud-init applies the distro's own default_user. Oracle Linux is cloud-user, Debian is debian, Ubuntu is ubuntu, and guessing from the distro name gets two of those three wrong. The card resolves the user from the AMI's name and description, and an unrecognised image yields a placeholder rather than a guess, because a wrong username in a copy-paste command is worse than an honest gap. A private-only instance is called out as reachable from inside the VPC only, rather than presented as a command that will simply time out. * fix(nat): plumb host state for a NAT gateway's public address AddNATGateway installed the OVN snat row and stopped there. On a routed-NAT node the host is what delivers a public address, so an egressing private guest left with a source the host held no route back to, and the masquerade rule — which matches the transit 100.127.0.0/24 alone — never saw the packet. The gateway's address needs the same /32 route into OVN, proxy-ARP and FORWARD accepts an EIP gets. hostBindSpecs unions floatingIPSpecs with one deduped spec per NAT gateway address and feeds BindHostEIPs. It is kept apart from floatingIPSpecs deliberately: pruneOrphanEIPs and addLive read that set as the complete list of addresses allowed to hold a dnat_and_snat, and a gateway holds an snat, so widening it would stop the sweep reclaiming a stale row on an address later reused as a gateway's. AddNATGateway also binds directly, on both the create and idempotent-skip paths, so a fresh gateway is not dark for up to 30s waiting on the host EIP pass — and so a reboot that keeps the OVN row still replays the volatile half. Documents NAT gateways in nat mode in vpc-networking, and says at the head of that section that nat is a supported mode covered by nightly cells 19 and 30. * docs(oci): document Oracle Linux as a guest image, and that its user is cloud-user Four catalog entries, all UEFI. The login user is the one that catches people: opc is the default on the images Oracle publishes into OCI itself, while these are the generic KVM cloud images from yum.oracle.com, whose cloud-init default_user is cloud-user. Verified on 10.1 and 9.8 — opc, oracle and ec2-user are all refused. Also records why these four pin a checksum digest inline (Oracle ships no sums file, and each URL names an immutable build), why arm64 takes the -kvm-cloud- build, and that SELinux ships enforcing. * fix(reconcile): stop reading an unreadable intent as an empty one A wedged JetStream KV returned no keys for the ENI bucket while writes were still landing. loadPorts mapped that to an empty port map, pruneOrphanPorts read every guest LSP as an orphan against it, and every guest on the node lost DHCP and IMDS while the API kept answering and the instances stayed "running". The existing re-read guard did not help: it re-reads through the same path. Three defences, because no single one covers every shape of the fault. A failed kv.Get now fails the pass instead of dropping the record. Dropping one VPC narrows localVPCs, which filters out every ENI in it, which the sweep reads as permission to delete every guest port in that VPC — one failed read, same outcome, and it needs no exotic KV state. Unmarshal failures stay skips: that is a poison record rather than an unhealthy store, and failing on it would wedge reconcile forever. The port sweep now asks local OVS which taps are plugged in here and never deletes a port that has one. A running guest's port cannot be an orphan whatever intent says, and OVS is the one liveness signal that does not come through the KV read path the sweep is otherwise entirely driven by. Wired from vpcd; unwired callers fall back to the guard below. An unreadable OVS skips the sweep rather than deciding on intent alone. Each sweep also refuses outright when the keep set it would delete against holds nothing at all — no ENIs, no SGs, or no ports and no external IPs. That is the case local taps cannot cover, because the drift sweep is leader-gated and lists NB cluster-wide, so the leader holds no tap for another node's guest. The cost is real: a genuinely orphaned row now survives until one live record of its class exists. Leaking a stale row is the recoverable half. Three existing prune tests drove their fixtures to a wholly empty intent, which is now the refusal case; they keep one live record of the class so they still prove the prune. The Get-failure path has no unit test — there is no jetstream.JetStream fake in the tree and an embedded server cannot be made to list a key it will then refuse. * docs(oci): separate the two address quotas, and the principal that can read them The regional reserved-public-IP ceiling is tenancy-wide and does not grow with node count, so a cluster hits it before the per-VNIC 64, which does. Records the commands that read both, and that a detached reserved public IP still bills. Reading limits or usage needs a tenancy-level grant, which cannot be compartment-scoped the way the node's policy is. That is a second, read-only principal kept off every node, not an addition to SpinifexOperators. * docs(oci): give quotas their own troubleshooting section, and say we enforce none Spinifex hardcodes no address ceiling: it attempts the allocation and maps OCI's own refusal to InsufficientAddressCapacity. So a quota is raised with Oracle and nothing here is reconfigured or restarted — which is what an operator needs to know before it bites, not after. Separates the two quotas by how they scale: the regional reserved-public-IP default is tenancy-wide and raisable by ticket, the per-VNIC 64 is per node and is not. * fix(oci): follow a guest to its new node when its EIP moves OCI delivers a public address to whichever VNIC holds the private IP behind it, and nothing ever moved that private IP. Binding.VNICID was written once at allocate time, as the VNIC of whichever queue-group daemon happened to serve the AWS request, and never updated — so an address landed on the right node only by chance, and a guest that stopped and started elsewhere went dark while every OVN row, host route and firewall rule on the new node was correct. ClaimLocalAddresses closes the gap from the node that should hold the address. Local OVS decides ownership, the same authority the host EIP binder already trusts for the same question, and UpdatePrivateIp with a new VnicId does the move in one server-side call. The private IP OCID survives it, so the reserved public IP attached to it follows without being touched and the customer's address never changes. A loop per node, not leader-gated: the question is about this host's guests, and a node that never wins the reconcile lease would otherwise never claim the addresses of the instances it is running. * fix(oci): record which ENI an EIP is associated with An EIP is allocated bare and attached by a later AssociateAddress, so the OCI binding written at allocate time named no ENI and nothing ever filled it in. The address then stayed on whichever node happened to serve the allocation: the affinity pass could follow an auto-assigned address, which is allocated with its ENI, but never an EIP — the case customers actually use. BindOwner is on an optional OwnerTracker interface rather than Allocator, because only OCI has a notion of an address being delivered to one node. A static or DHCP address is on the wire everywhere its node can reach, those allocators do not implement it, and ExternalIPAM treats a pool that does not as a success with nothing to do. The same gap made the release owner guard inert for EIPs: it compares against a binding ENI that was always empty, so a stale teardown could free an address already handed to another instance. * fix(ec2): report the address an instance is actually reachable on Three faults kept DescribeInstances answering with the address launch assigned, whatever happened afterwards. The launch-time stamp lives on the stored ec2.Instance, and the stopped path only declined to add an address rather than removing one, so a stopped instance went on advertising an address it had released and that may already belong to someone else. The projection now clears it, and the NIC association with it, working on its copy so a restart can still report the address. The running path filled the public address only when it was nil, which the stamp never is, so an EIP could not be reported however correctly it was recorded. The runtime address now wins. And nothing told the rest of the system about an association: AssociateAddress wrote neither the ENI record, which is what IMDS and the network reconcile read, nor the instance record, which is what a describe answers from. The ENI write is direct; the instance write is a fan-out, because the queue-group worker that serves AssociateAddress is rarely the node running the instance. * fix(net): take the host EIP plumbing off a node its guest has left Bind refuses an EIP whose ENI is on another chassis; the prune beside it kept anything still wanted cluster-wide. A guest that moved therefore left the old node holding the /32 route into OVN and the proxy-ARP entry on the uplink, with nothing that would ever remove them. On a shared segment that node goes on answering ARP for an address another node is now serving. The two halves now ask the same question by the same authority, in the loop that already runs per node. An unreadable local OVS is not a yes: tearing down a live address on a guess would fire everywhere at once, so it keeps the binding and the next pass tries again. A binder with no chassis test behaves exactly as before, and a centralised EIP is exempt because it hairpins through the gateway chassis and every node plumbs it legitimately. * fix(net): refuse to free an Elastic IP on behalf of a departing interface Releasing an instance's public address on teardown releases it by address, scoped to the ENI holding it. Since an association now writes the EIP onto the instance record, that address is the EIP — so terminating an instance gave a customer's Elastic IP back to the pool while describe-addresses still reported it as theirs, free to be handed to another guest. The existing ownership guard cannot catch this. It only fires when the stored allocation already names an ENI, and an EIP is allocated bare and associated later, so the stored ENI is empty and the guard passes straight through. The distinction that holds is who is asking. ReleaseAddress is the address's own owner and names no interface; instance teardown names the interface it is tearing down. So a release that names an owner may free an auto-assigned address and never an EIP, which is enforced in each of the three allocators. DHCP needed the scope carried on the wire, since it releases by address and resolves the lease afterwards. Absent on an older caller, which reads as false and keeps the previous behaviour. Worst on the two providers where it is not recoverable: on DHCP the lease is the only thing holding the address and the server promises nothing about handing it back, and on OCI the address is a reserved public IP object that the release destroys outright. * test(e2e): prove an Elastic IP survives a stop and start Instance teardown releases the public address by value, and since the association writes the EIP onto the instance record, that release aims straight at the customer's Elastic IP. The allocator guards now refuse it, but nothing end to end proved the address comes back on a restart. The step goes inside EIPFlip, which already runs last on the shared public guest for the same reason a stop is safe there: the address the guest returns on is the allocation's, not the pool's. It runs on the static cells and the DHCP ones alike, and on DHCP it doubles as the only outside view of the lease still renewing while the guest is down. * feat: release an auto-assigned public address on stop, take a new one on start AWS lends an instance a public address for as long as it runs and gives it a different one on the next start. Spinifex kept the address across a stop, which holds a pool slot for a stopped instance and, on a DHCP pool, keeps renewing a lease nothing is using. A stop now offers the address back through the instance cleaner, which refuses anything an Elastic IP is behind — the EIP association record answers that, the same question an ENI delete already asks — and records on the VM that the start owes it a new one. A host drain keeps the address: Restore relaunches the guest on the next boot, and changing its address under it is an outage nobody asked for. The start allocates before the launch, so an exhausted pool refuses the start rather than booting a guest the customer cannot reach, adopts an Elastic IP associated while the instance was stopped instead of assigning a second address beside it, and hands the address back if the launch then fails. e2e: a StopStartAddress stage on the public guest asserts a stopped instance reports no address and comes back on a different one from a static pool, and a StopStartKeepsTheEIP step asserts the opposite for an Elastic IP. * docs: restructure the OCI guide around two deployment paths, with diagrams The guide had grown into a numbered wall with no shape: no frontmatter, no contents, and a reader with an OCI account had no way to tell where to start. It now follows the multi-node install template — overview, prerequisites, then the deployment path — and splits that path in two: Terraform for one node or three, which is the preferred route, and a manual single-node walkthrough for an instance that already exists. Both meet at one shared configuration section rather than repeating it. Two ASCII diagrams carry what the prose could not: one OCI instance with the whole AWS surface inside it, and three nodes with the same surface distributed, showing which plane each VNIC carries and how an address follows a guest that moves. The limits section was materially out of date and said so in the places that mattered most. Multi-node is no longer experimental, public addresses are no longer pinned to one node — each allocates on its own VNIC and the affinity pass moves the private half to wherever the guest is — and an auto-assigned address is now returned to OCI on stop, so a stopped fleet stops paying for addresses. What changed is recorded rather than quietly edited. The Terraform configuration moves in from the monorepo as scripts/terraform/oci-spx, so the guide points at something in this repository. Also marks the two new lifecycle test files in-package, with the reason each one needs to be: both substitute unexported dependencies. * docs: rework the OCI guide — SVG diagrams, Terraform variables, iSCSI and br-wan Replaces the two ASCII diagrams with SVGs in the house style, one for a single node and one for three. Renames the two deployment sections to Terraform deployment and Manual deployment, documents what the Terraform builds per node and every variable worth setting, and explains that the data volume is an iSCSI attachment rather than a disk. The manual path now covers the iSCSI login, the fstab entry and the netplan source routing that Terraform writes, and states what br-wan is for on a routed-NAT node: it names the VNIC for the OCI allocator and holds the registered addresses, but is never bridged into OVS. Opens with what Spinifex is and why OCI is worth running it on. * fix: make gateway convergence work on a multi-node routed-NAT cluster Three defects each stopped a multi-node cluster from ever attaching an internet gateway, and all three produced the same uninformative symptom: `terraform apply` sitting on `aws_internet_gateway.igw: Still creating...` forever while every service reported healthy. ovn-northd was given only its local OVSDB socket as a client connection. A RAFT follower does not forward writes -- it answers "not cluster leader, trying another server" -- so with one entry in the list, northd stops translating Northbound intent into Southbound flows the moment database leadership moves to another node. Measured on the OCI cluster: SB_Global.nb_cfg seventeen transactions behind NB_Global.nb_cfg, with no chassisredirect port for any VPC created after leadership moved. setup-ovn.sh wrote the member list only when the caller passed --ovn-remote, which only install-node.sh did; it now derives the list from the live RAFT membership otherwise, and says so when it finds a single member, because that node needs a second pass once the others join. The gateway datapath probe ran wherever the reconcile lease landed rather than on the chassis owning the port. Both probes leave from the local uplink, and under routed NAT the transit subnet exists identically on every node, so a remote probe reported a healthy gateway as dead and aimed a repair at a datapath the node does not own. GatewayPortLocal gates it. On OCI the probe then ARPed the tenant-visible public IP, which the cloud NATs and which appears on no interface anywhere. The reconciler now maps the address through the same DatapathResolver the NAT rules are built from. Also adds DescribeVpcEndpoints, which the AWS provider calls on every VPC delete to sweep GuardDuty-managed endpoints; an InvalidAction there failed the destroy and stranded the VPC. The OCI guide gains a Terraform section recording which shipped workbooks were validated against a three-node OCI cluster and which are still open. * docs: record the validated Terraform workbook results for the three-node OCI cluster Six of the seven workbooks run against the OCI cluster pass, including the three the section exists to prove: RDS, ECS and EKS. Records nginx-alb and the clean eks-quickstart re-run, and names the two lifecycle gaps that bit during the run so an operator recognises them rather than reading them as a broken workbook. * fix: route the NAT gateway's SNAT source through the datapath resolver AddNATGateway was the one NAT path that did not map its external address to the one that rides the wire. Where the public half of an allocation is NAT'd by the cloud it is on no interface anywhere, so every private-subnet guest egressed with a source that was dropped, while the gateway reported available and DNS kept working because the VPC resolver is host-local. The unmapped address also failed the host EIP sweep, which works in resolved addresses, so the bind was pruned as foreign on the next pass and reinstalled on the one after. The public half now survives as an external ID instead. Also restructures the OCI guide around numbered steps through to running Terraform, and moves the cloud-init and metadata sections into troubleshooting where they are read when something is wrong rather than followed when it is not. * fix: place a NAT gateway's OCI address on the node that forwards for it OCI drops a packet whose source is not a registered private IP on the VNIC it leaves from, so a NAT gateway's SNAT source has to be on the gateway chassis. Two independent placements decided those and nothing reconciled them: the address was created by whichever worker served AllocateAddress, while the chassis is a fixed ordering the reconcile pass asserts. On a three-node cluster that made private-subnet egress a coin flip, and the same Terraform passed or died depending on which node answered. The affinity pass already solves this for guests, by asking local OVS where the tap is. A NAT gateway has no tap, and it skipped anything with no ENI — which is exactly and only this address. The equivalent local fact is the VPC's chassisredirect binding, so ownership now has two answers rather than one, and an address that is neither a guest's nor a gateway's is still left alone. The VPC is recorded from the reconcile pass rather than from CreateNatGateway, because the answer is the gateway chassis and reconcile is what decides it. * fix: plumb a NAT gateway's host ingress only on its gateway chassis A NAT gateway's address has no ENI, so neither the bind's locality test nor the prune's could ever fire for it and every node installed its /32 route into OVN. Latent on bare metal, where pool addresses are disjoint from the node subnet; on a cloud that hands out the address from the nodes' own subnet it outranks the connected route and swallows node-to-node traffic. Give that address the same second authority the ocinet affinity pass has: the VPC's chassisredirect binding, read from the local ovn-controller. An unanswerable question still leaves the binding alone, so a single node whose gateway port is not yet claimed behaves exactly as before. The orphan EIP prune built its live set from ENI and user EIPs alone while also sweeping host ingress, so every drift pass tore down the ingress of every live NAT gateway and left it dark until the next host EIP tick. * fix: ask the gateway chassis only about a NAT gateway's address gatewayElsewhere skipped the NAT gateway marker it was named for and answered for every spec, so on a node that is not the VPC's gateway chassis every EIP — including a guest's, whose locality is its own port — was left unplumbed. Two of three nodes bound nothing. * test: assert the two private-subnet EKS workbooks in the tofu harness eks-https-ingress and eks-gitops-argocd put their workers in private subnets behind a NAT gateway, so a worker only registers Ready if its egress to the cluster endpoint and to ECR survives that path. That makes them the workbooks that catch a NAT gateway fault, and neither had an assertion, so neither could be run at all — run_workbook calls assert_<name> by construction. Both share eks-quickstart's readiness check, extracted unchanged. They also name the worker size node_instance_type rather than instance_type, and tofu rejects a -var for an undeclared variable outright, so the name is read from the config instead of kept in a list here. * test: count agents, not every node, when asserting EKS workers Ready The control plane is three K3s servers, so an unfiltered node count reaches a one-worker cluster's desired size before any worker joins — the assertion passed on exactly the failure it exists to catch, reporting "4/1 workers Ready". Filtering on the control-plane role label makes it 1/1, which is the number that says a node-group worker registered. Matters most for the two private-subnet workbooks, where the worker joining is the whole proof that egress through the NAT gateway works. * docs: record the two private-subnet EKS workbooks passing on OCI	2026-09-26 14:31:07 -05:00
Josh Siviour	451df1a8cd	docs: publish AWS operation coverage inventory (#1075)	2026-09-17 21:33:53 +10:00
Julian Sommer	5991b92b06	chore: concurrent preflight	2026-09-17 11:31:42 +10:00
Julian Sommer	ba98509c5b	fix: simplify mulga-docs build step by collapsing coverage paths + hardware	2026-09-14 12:41:02 +10:00
Julian Sommer	fb6494da01	feat: coverage docs	2026-09-14 11:42:01 +10:00
Josh Siviour	8e11908dfb	fix: make the EKS control plane's disk latency observable, and give it room to run (#1009) ... * fix: report CPU and memory pressure in the EKS control-plane health diagnostic An unhealthy control plane reported readyz, etcd reachability and etcd-disk free space. When etcd goes unreachable with disk:ok and no OOM kill, that set cannot distinguish a starved guest from a genuine etcd fault, so add the loadavg and MemAvailable that separate them. An empty /readyz body also rendered as readyz:[none], which reads as "no subcheck failed" when it means the probe never answered. Report that as unreachable instead. * fix: sample the node's load, not just the runner host's Every runner agent runs on one host while their nodes sit on several, so the existing sample measured the runner's host and was labelled as the hypervisor. It happens to coincide when the node is on that same host and misleads otherwise. Relabel it and add a node-side sample, which is where the guests are actually scheduled and so where a starved guest shows up. * feat: build the mkosi EKS image in CI and give its control plane telemetry The Ubuntu/systemd EKS image is what customers run, and nothing built it: the composition existed in mkosi-build.sh's image_profiles table but had no caller, so a change that broke it was found by hand or not at all. Cell-25 builds the Alpine image, which leaves the published one with no gate. Add a Makefile target beside the Alpine one and a job in the existing path-filtered image workflow. The guest also collected nothing. k3s has served embedded-etcd metrics on 127.0.0.1:2381 since the daemon started setting etcd-expose-metrics, and no collector ever read them, so an etcd stall was only ever visible as "etcd:unreachable" in the state report — which cannot separate a slow disk from a dead one. Ship otelcol-contrib pinned to the version the cluster hosts already run, scraping etcd and guest host metrics. The unit is gated on ConditionPathExists for its endpoint file rather than started unconditionally: whether the control-plane guest can reach the sink from the isolated CP VPC is unverified, and a collector with nowhere to export is a restart loop, not telemetry. * fix: report etcd fsync latency in the control-plane health diagnostic disk:ok measures free space, which says nothing about how long a write takes. etcd stalls on fsync latency long before it runs out of room, and this guest's disk is an EBS volume served by viperblock, so a fsync is a network round trip. An unhealthy control plane reporting healthy memory and disk was uninterpretable without it. Derived from the histogram's sum and count rather than a bucket percentile: two greps against an interpolation this script has no business doing, and the difference between a healthy single-digit reading and a stalled one is an order of magnitude, not a tail. Rides the state report's existing gateway path, so it needs no route the CP VPC does not already grant, and lands on both images because the script is shared. The scrape's failure is guarded explicitly. Under set -e a bare assignment from a curl against a port nothing answers aborts the whole diagnosis, dropping the reason field exactly when it is wanted. * feat: publish etcd fsync on every control-plane report, not only failing ones A figure that appears only once the apiserver is already unhealthy has no baseline to be read against, which is the same gap a passing cell that uploads no journal leaves: the number is unanchored when it finally arrives. Move the scrape into its own helper, emit it on healthy reports too, and carry it as a typed field rather than only inside the reason string. Slowness is logged, never gated. observe() decides cluster health, and a working cluster on a slow disk is working — returning an issue there would fail customers' clusters on a latency threshold. The warning is latched on the transition instead, because the control plane republishes on its own timer and a line every interval is how a real signal gets tuned out. An absent sample is not a fast one. Too few fsyncs to average, an etcd serving no metrics and an older AMI all arrive as nil, and the field is omitted rather than defaulted so none of them can be read as healthy. The minimum sample count is the same lesson as the slow-shard floor: a mean over three fsyncs on a just-started etcd is a number with no meaning. * fix: log an etcd fsync baseline and name the two no-sample cases apart A healthy cluster logs no health lines, so the fsync figure reached the daemon on every report and was never surfaced — a passing cell left nothing for a later slow reading to be compared against. Log the first known sample. Split the "unknown" sentinel into nometrics (etcd served none) and warmup (too few fsyncs to average): different faults, different fixes. The payload guard now tests for a number rather than the sentinel word, so a future sentinel cannot leak into the JSON as a bare value. * feat: measure fdatasync latency in the diskperf gate The two pinned profiles are buffered-bypassing throughput jobs at queue depth 32. Neither measures the cost of making a write durable, which is what a consensus datastore pays per raft entry and stalls on long before it runs short of bandwidth. A 5ms barrier on bare metal went unseen as a result. syncwrite-4k is the etcd-shaped profile: sequential 4k, queue depth 1, fdatasync every write, buffered psync so the sync has a page cache to flush. fio reports the sync distribution apart from the write stream, so the parser reads it separately — gating on write clat would score the time to reach the cache and call it durability. No figures captured on this hardware yet, so the baseline entry is zero and the job records without gating. * fix: use the real otelcol checksums asset name in the EKS server image The checksums file is named for the release repo and the distribution and carries no version, unlike the tarball beside it. The guessed name 404'd and took the image build with it. * fix: extract otelcol as root in the EKS server image The upstream tarball records its build uid (runner/docker, 1001:118), which does not exist in the image. tar running as root tries to restore it and fails outright rather than degrading, taking the build with it. --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>	2026-09-07 21:53:41 +10:00
Bryn Mailer	18aa67b0e9	feat: add the e2e guest disk-performance gate (#964) ... * feat: add the e2e guest disk-performance gate Nothing measured guest disk performance in CI, which is why a quadratic index scan shipped in viperblock 35dd9f2 and survived three subsequent fixes to adjacent code. It is also why the two availability clauses at the end of the ebs-performance plan's Stage 0 have never been evaluated: a sustained random-write workload took the guest, and then the host, down, and nothing since has demonstrated that it no longer does. tests/e2e/diskperf/ launches an instance, attaches a dedicated blank volume per repetition, and drives a pinned fio profile over guest SSH. Two profiles: randwrite-4k at 4 jobs x 4 GiB, the shape the exit criterion names, and randrw-70-30-4k, the mixed workload the headline IOPS came from. The target is a raw device with direct=1 rather than a file on the boot disk, so guest page cache, filesystem allocation and journal traffic are out of the path. Two classes of assertion, kept apart because folding them together lets a run that left the host unreachable pass on a respectable IOPS figure. Availability is hard pass/fail and absolute, so it gates from the first run: the job completes inside its budget, a guest responsiveness probe answers throughout on a second SSH channel, and the host's own sshd answers throughout. The guest probe reads the root volume while the workload runs on a different one, so a pass is evidence of isolation between volumes rather than of the job merely being slow. The host probe is a TCP connect plus an SSH banner read, which needs no credential and still proves the kernel scheduled sshd. Throughput and latency are banded against a committed baseline: fail above 25% regression, warn above 10%, per-metric median across repetitions. baseline.json is committed and embedded, with every figure zeroed, and a zero reference records and warns rather than failing. Only the availability class gates until numbers are captured on the bare-metal cell; a throughput band nobody measured would be a fabricated gate. A fresh volume per repetition is the reset. Extent count grows monotonically, so reusing a volume makes the gate order-dependent and drifts its own numbers upward over time, slowly normalising the regression it exists to catch. Each job is its own guest invocation and fio's JSON is fetched by a separate command, so a wedged job loses only itself and reports as a budget breach rather than a parse failure. e2e-baremetal.yml gains suites, suite-env and cell inputs so one reusable workflow serves both the correctness cell and nightly cell 28, which queues behind cell 20 on the shared concurrency group. Cells 25-27 are the dedicated eks, ecs and rds cells. The per-suite Go test timeout is resolved from e2e_suite_timeout in suite-sets.sh, where diskperf declares its 180-minute budget beside rds's rather than in a second place. The gate's own arithmetic runs in preflight via test-harness: band edges, the unmeasured cases, per-stream metric selection, fio JSON parsing, and the probe accounting including a slow-but-successful answer counting as a breach. A gate whose comparison logic is untested is the failure mode the plan is otherwise full of. The extent-count ceiling named as a fourth availability assertion is not implemented: it needs viperblock.index.entries from S1.5, which does not exist. * feat: let a manual bare-metal run choose its e2e suites A dispatch run only sees inputs declared under workflow_dispatch, which had none, so suites and suite-env resolved empty and fell back to the correctness set. Capturing a diskperf baseline by hand was therefore unreachable without a workflow_call caller. * fix: build the bare-metal suites the run step will execute The build step packs pre-built binaries into the tarball and did not see the suite selection, so a selected suite was absent from the node unless it happened to be one of the three the build script hardcoded. * feat: commit the measured disk-performance baseline Medians of three repetitions on the bare-metal cell. The throughput and latency bands now gate instead of warning, so a 2x regression in either job fails the suite. randwrite-4k's write tail is two orders above the mixed job's. It is recorded as measured rather than zeroed: an ungated metric would miss the regression class the gate exists to catch.	2026-09-03 14:33:21 +10:00
Julian Sommer	c81abd1836	fix: add fipsboot guard to missing services	2026-08-27 15:05:49 +10:00
Julian Sommer	74117e5b26	fix: remove the broken golangci-lint cache path	2026-08-20 11:37:10 +10:00
Julian Sommer	e36c8f6ba8	chore: update to go 1.27	2026-08-20 10:37:52 +10:00
Ben Duncan	225bf65a8e	feat: EBS export tunables, teardown reapers, fan-out and OVN fixes (#800) ... Merges `feat/ebs-export-tunables` (83 commits) into dev. The branch could not be rebase-merged because it carries five merges from dev, so it lands as a squash. ## What is in it - **EBS export tunables** and the nbdkit/viperblock export path they configure. - **Teardown reapers and backoff** for S3 buckets, load balancers, gateways, spot requests, certificates, endpoints, elastic IPs and network interfaces, so a dependent that keeps failing is backed off rather than retried hot. - **Retry and tolerance paths**: ECS IAM service init, ECR signing keys deleted between the list and the read, an unreadable signing key at awsgw startup, and an ECS instance-profile policy attach converging on a stale read. - **Key pair validation reads the material by name** instead of listing the account, closing the `InvalidKeyPair.NotFound` race that a single-replica LIST could produce. - **Concurrency and fan-out**: concurrent metadata document fetches, bounded metadata reads for describes, and the node-discovery timeout removed from every fan-out call. - **OVN correctness against the configured NB cluster**: the flows barrier, nexthop MAC binding, and stopping a compute node from running the package's standalone database. - **nats-js-watchdog can re-arm itself.** The escalation check ran above the probe loop while the only line that cleared the counter sat below it, so a latched watchdog could never observe a recovery. It now probes first and decides second, and says so once per latch rather than once per tick. - **CI**: 19 commits of nightly and build-pipeline work. ## Verification `make preflight` green locally. Build and Test, Integration Tests, Race Detection, and Lint and Security all green on `c2da0385`. ## Known red, not introduced here Two nightly cells fail on this branch and are pre-existing: - **cell-10** — a partial OVN IPsec enable leaves `NB_Global ipsec=true` asserted cluster-wide while two of three chassis failed their local reconcile, black-holing cross-chassis guest traffic for roughly 5.5 minutes while every health signal stays green. - **cell-19** — a total layer-3 egress blackhole below a correctly and promptly programmed NAT/EIP datapath, aggravated by a natuplink probe whose retry budget outlives the test's own wait, so it can never emit its FAIL verdict. Both are filed separately and should be weighed before a release is cut.	2026-08-17 17:42:27 -05:00
Julian Sommer	333248568e	fix: use actions/cache@v6 + test-actions clean	2026-08-17 11:34:50 +10:00
Julian Sommer	ae0337d31b	test: run the unit and race tiers clean in CI ... Go stores test results in the build cache alongside compiled objects, so now that the cache persists across runs an unchanged package replays its previous result instead of executing. That is what makes a local run fast and exactly what a merge gate should not do: a green check would no longer mean the suite ran against that commit. Add a GOTESTFLAGS hook, empty by default so local runs keep reusing results, and pass -count=1 from the two CI steps that gate a PR. Compilation is still cached either way, so this costs no build time.	2026-08-17 11:28:59 +10:00
Julian Sommer	cba51b59e3	fix: stop GOFIPS140 splitting the Go build cache in two ... GOFIPS140 is a global build-config change: it alters every package's action ID, stdlib included. A target that omits it therefore shares no build-cache entry with one that sets it, and the two flavours cost roughly a full cold build to switch between. Seven targets set it inline while eight go invocations did not, so test-integration, test-harness, test-actions, test-segscan-oracle, bench and the go run tools all built the non-FIPS flavour. In CI this is load-bearing: the integration job wins the shared setup-go cache-save race, stores a non-FIPS cache, and every FIPS build in the other jobs then misses on every lookup. Export it once so the two cannot drift apart again.	2026-08-17 11:02:17 +10:00
Josh Siviour	c6bc2b4d29	fix: only restart NATS when JetStream has actually failed (#795) ... The watchdog's is-active guard cannot fire against spinifex-nats.service (Type=simple reports active the instant the process forks, long before JetStream replays its state), so it restarted a NATS that was still recovering. Separately, any non-zero exit from spx js-probe was read as a JetStream write failure, so a broken spx binary read as a cluster-wide outage instead of an unrunnable tool. Replaces the is-active guard with an age gate on ExecMainStartTimestampMonotonic, adds a post-restart cooldown and a restart-count escalation that gives up rather than looping, and teaches the watchdog to treat js-probe exit 2 (and any exit code that isn't 0 or 1) as inconclusive rather than as proof of failure. Bounds the unit's own start with TimeoutStartSec=120, matching the existing TimeoutStopSec derivation.	2026-08-14 15:01:00 +10:00
Josh Siviour	9102588e5a	chore: bump Go toolchain to 1.26.6 (#793)	2026-08-14 09:29:03 +10:00
Julian Sommer	65d2501e6b	feat: add MariaDB as the second RDS engine (#762)	2026-08-13 10:38:18 +10:00
Josh Siviour	fd8be8a814	refactor(test): route duplicated NATS bootstraps through testutil (#776) ... * refactor(test): route duplicated NATS bootstraps through testutil Thirteen test sites hand-rolled a NATS or JetStream server bootstrap that spinifex/testutil already provides, and one seedKV clone duplicated testutil.SeedKV. Route them through the shared helpers. Sites whose options genuinely diverge are left alone: fixed ports needed for reconnect and restart scenarios, custom TLS and token auth, a custom MaxPayload, and TestMain entry points that have no testing.TB for t.Cleanup. The natgw and routetable seedKV clones use CreateOrUpdateKeyValue rather than CreateKeyValue, so their failure behaviour differs and they stay as they are. Also moves eight test files to external test packages where they declare no helper used by a sibling in the same package. Coverage is unaffected: Go compiles internal and external test packages into one binary per package, so attribution is identical either way. * chore(test): require new test files to use an external test package Adds a preflight check that fails when a newly added test file declares an in-package test. External test packages bind tests to the exported API and let packages share test doubles, which an in-package double cannot be. Only added files are checked, so the existing in-package tests are untouched. e2e and integration suites are exempt: both share a package-level fixture built in main_test.go, so their files cannot move individually. A file that needs unexported access can opt out with a //test:in-package comment.	2026-08-12 22:38:15 +10:00
joshsiv-mulga	1e865b3a4b	fix(ci): stop a cached integration run from failing the build ... make test-integration cats the AWS model conformance report immediately after the test run, but the report is written by the test binary itself. Go's test cache means an unchanged package reports "ok (cached)" without executing, so no report is written and the cat fails the target with exit 1. Any branch whose changes do not invalidate that package's cache hits it, which is why it fires on unrelated branches -- a docs-only branch cannot break a build yet failed the same way. -count=1 forces the run. That also keeps the conformance gate honest: a cached result would mean the check never ran against this commit. Making the cat tolerant instead would let the gate silently no-op.	2026-08-12 12:38:41 +10:00
Josh Siviour	d178ef5992	feat: version-aware host-asset preflight for binary-swap deploys (#750) ... A binary-swap deploy (make deploy or a manual spx copy) ships a new binary whose host-config prerequisites — the privileged helper scripts and their sudoers grant, placed only by setup.sh / setup-ovn.sh — it does not ship. The node looks healthy and the gap surfaces later as a cryptic launch failure or a silent security regression. Add a preflight package that checks the host against what this build expects, comparing asset content, not existence, so a stale-but-present asset is caught too. The managed helpers install from quoted heredocs, so their on-disk bytes equal the heredoc body verbatim; the binary carries the canonical sha256s and compares. - spinifex/preflight: CheckHost / CheckHostAt, content hash of the two managed helpers and a NOPASSWD grant-line check for the sysctl helper. - Generated canonicalHashes plus a drift guard that re-extracts the heredocs and fails if the committed manifest no longer matches. - Three surfaces: spx admin preflight, a make deploy gate (SKIP_ASSET_PREFLIGHT=1 to bypass), and a non-fatal daemon boot warning.	2026-08-10 07:52:39 +10:00
Josh Siviour	272b4b22e8	feat(ochre): add the vLLM serving AMI and give the daemon a route to it (#740) ... * feat(images): add a vllm-serving AMI as a stacked mkosi profile LaunchServingVM resolves its AMI by the tags spinifex:managed-by=bedrock and spinifex:bedrock-role=vllm-serving. No image carried either, so every Ensure failed at resolveServingAMI. The only GPU image ships the NVIDIA driver and Docker but no Python, no vLLM and no serving unit, so tagging it would have moved the failure from admission to first boot. Add a vllm profile that stacks onto gpu-nvidia and owns the whole guest side of the launcher contract: the runtime, vllm-serve.service, the weights mount and the tags. vLLM is baked into a uv-managed venv rather than pulled as a container at boot — a runtime docker pull of vllm/vllm-openai is known to stall this platform's write plane, and mkosi's unprivileged build has no daemon to bake an image from anyway. Also bring images/mkosi.profiles/ under make test-images. Its scripts have no .sh suffix, so the existing name glob skipped exactly the files worth linting; sources are found by shebang instead. * fix(images): make the vllm postinst survive the build sandbox Two failures found by actually running the build. uv installs to /usr/local/bin, which mkosi's postinst PATH need not carry, so `command -v uv` reported a good install as missing. Call it by absolute path, and check it the same way. `vllm --version` initialises device config and dies with "Failed to infer device type" on a builder with no GPU, so it could never have verified anything. Assert the pinned version through package metadata instead, which also catches a version mismatch rather than merely a broken entrypoint. * fix(images): pin the serving AMI datasource and mirror cloud-init to the console A serving VM boots through LaunchSystemInstance, so it comes up dual-NIC with a fw_cfg netcfg blob and takes its user-data solely from IMDS — the same shape as the EKS control plane. That path carries a cloud-init datasource pin; the vllm profile never got one, and the base config deliberately leaves datasource_list unpinned, so the serving image was the only system VM booting without it. Reuse the EKS pin rather than copying it. The 99- prefix is load-bearing: read_conf_d() applies cloud.cfg.d in REVERSE and mergemanydict() is first-source-wins, so a 90- name loses to cloud-init's own 90_dpkg.cfg ('_' sorts above '-') and the pin is discarded in silence. The postinst asserts the MERGED datasource_list via cloud-init's own read_conf_with_confd rather than checking the file exists, because it existed at 90- too and did nothing there. Also mirror every cloud-init stage to the serial console. A serving VM is headless, and a guest that wedges before handover never surrenders /var/log/cloud-init-output.log, so the console get-console-output reads is the only channel an operator has. The images_test.go case covers the two tags resolveServingAMI filters on; it matches on those alone and never on name, so a typo hides the image entirely. * fix(images): make the dhcpcd data-NIC lease genuinely one-shot dhcp_oneshot() promised to configure the link and exit "leaving no daemon to fight cloud-init", but the dhcpcd branch passed only -q, which is quiet rather than one-shot. Every Ubuntu image takes that branch, so a dhcpcd daemon survived on the data NIC holding the control socket. cloud-init's Ec2 datasource then runs its own ephemeral dhcpcd on the same NIC. Instead of spawning, dhcpcd forwards the invocation to the resident daemon as a control command and exits 0, and --dumplease returns the resident daemon's lease, so discovery looks successful. Cleanup then waits for a pid file at /run/dhcpcd/<iface>-4.pid that no process ever wrote, spinning its full 30000 x 10ms budget: init-local stalls for exactly 300s on every system-instance boot. Adding -1 exits once the interface is configured and -p keeps the address and routes across that exit. Both are needed; -1 alone tears the link back down. The test now models the real clients — the stub stays resident unless given the flag that makes its client exit, and the dhcpcd branch is forced by dropping udhcpc from PATH rather than left to whether the host happens to have /usr/share/udhcpc/default.script. * fix(images): serve without a CUDA toolkit and stop vllm-serve crash-looping vLLM picks FlashInfer for top-k/top-p sampling, and FlashInfer JIT-compiles that kernel on first use. The serving image ships the driver and the container toolkit but no nvcc, so the sampler warmup died with "Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist" minutes into a GPU cold start, after the model had already loaded. VLLM_USE_FLASHINFER_SAMPLER=0 selects vLLM's native top-k/top-p sampler, which is the same operation with no CUDA toolchain at runtime. Everything else on the serving path is already precompiled in the wheels: the attention backend resolves to FLASH_ATTN and CUDA graph capture completes, both without nvcc. Baking FlashInfer's AOT modules is the alternative if the fused kernel is ever wanted, and it keeps the toolchain in the builder rather than the appliance. The unit had Restart=on-failure with no bound, so an unfixable start failure restarted forever and buried its own first traceback under identical copies on the serial console — the only view an operator has of a headless guest. StartLimitIntervalSec/StartLimitBurst make it report instead. The postinst asserts the installed unit carries the sampler setting, for the same reason the datasource pin is asserted there: the alternative is finding out at runtime, on a GPU, five minutes in. Also corrects the shared datasource pin's header, which still claimed it installs as 90-spinifex-ec2-imds.cfg. It is installed at 99- by the vllm profile precisely because a 90- name loses the reverse-sorted, first-source- wins cloud.cfg.d merge to cloud-init's own 90_dpkg.cfg. * fix(bedrock): raise the serving-endpoint readiness bound to 15 minutes The 5 minute bound was chosen before any cold start had been measured, and its own comment asked to be corrected once a model actually ran on wattle. It now has: Llama 3.2 1B on an RTX A1000 reached "Application startup complete" 257.6s after launch, leaving ~42s of margin on the smallest model this platform can serve, with a cold torch.compile accounting for 26.3s of it. A bigger card and a bigger model are unmeasured. The two failure modes are not symmetric. Too low tears down an endpoint that was about to come up and reports it as a failed launch; too high only makes a genuinely stuck launch take longer to give up. So the bound is set well clear of the single data point rather than close to it. The CLI's own wait moves to 20 minutes to stay above it. The daemon starts its clock when the VM is launched, up to a minute after the CLI starts its own, so leaving both at 15 would expire the client first and report a bare "timed out" in place of the daemon's abort, which carries the reason. * fix: give the Ochre daemon a routed port in the Bedrock system VPC The readiness probe and every InvokeModel call dial a serving VM's system-VPC address from inside the daemon, but that subnet lives in OVN on br-int and the host has no route to it, so a healthy VM never reached READY. Each daemon now owns an ENI in the system VPC's private subnet, bound to an OVS internal port on br-int by iface-id, exactly as ovn-controller binds a guest tap. The ENI address is installed at the subnet's prefix length, which is what supplies the connected route. BaseURL is unchanged: the address it already carried is now reachable. Describe-or-create keyed on a per-node description, since ENIs carry no tag filter, so a restart or a second endpoint reuses one address instead of leaking one per launch. The port is ensured before anything else the launch creates: a VM the daemon cannot dial holds a GPU for the whole readiness window before unwinding.	2026-08-06 15:26:05 +10:00
Josh Siviour	688ce5ccfd	feat: add AWS model conformance coverage (#736)	2026-08-05 18:46:37 +10:00
Josh Siviour	a2c451128d	fix: unblock the Ubuntu 26.04 installer (qemu-kvm, sudo-rs) (#665) ... setup.sh aborts twice on resolute. qemu-kvm is a virtual package with two providers there, so apt refuses to resolve it; the token was redundant anyway since $QEMU_PACKAGES already installs qemu-system-x86/arm and spinifex execs the qemu-system-* binaries directly. sudo-rs, Ubuntu's default sudo since 25.10, then rejects the sudoers file: it implements no pam_session Defaults setting, and forbids a wildcard inside a command argument. pam_session is now emitted only for classic sudo, and the sysctl grant names a fixed-verb helper instead of a wildcard. The helper is narrower than the rule it replaces: it accepts one interface, a key restricted to rp_filter or accept_local, and a value restricted to 0 or 1, where the old grant permitted any net.ipv4.conf key at any value. Granting sysctl with an unconstrained argument was rejected as root-equivalent. Adds scripts/lint-apt-packages.sh and an Apt Packages workflow so the runtime package list is resolved against each supported suite's real indices in CI.	2026-08-03 08:57:50 +10:00
Josh Siviour	3e1f9b1198	feat: segscan storage oracle for the real-predastore integration tier (#657) ... * feat: add segscan storage oracle for the real-predastore integration tier Adds spinifex/testutil/segscanoracle, which copies a fixture node data dir and execs the mulga umbrella repo's scripts/segscan (a separate Go module that cannot be imported) to assert live/dead-tombstoned/dead-orphan byte accounting against real predastore writes. Kept off the default PR path: a new segscanoracle build tag excludes the test from make test-integration, and a separate segscan-oracle.yml workflow (push to main/dev + manual dispatch only, mirroring e2e.yml's non-gating trigger) checks out scripts/segscan rather than adding that checkout cost to spinifex.yml. * test: cover segscanoracle's pure logic to satisfy diff coverage segscanoracle has no build tag and always compiles into the default build, but nothing in that build exercised it, so every new line counted as uncovered against the 60% diff-coverage gate. Adds unit tests for JSON decoding (including the malformed-payload path), the Locate resolution order, and the directory-copy routine, plus a throwaway stand-in module (not the real scripts/segscan) to exercise build+exec+decode end to end. Pulls decodeReport out of Run and resolveSegscanDir out of Locate so the pure logic is testable without a *testing.T or a real binary.	2026-08-03 00:40:45 +10:00
Josh Siviour	9c9bb9430c	test: gate image shell scripts in CI and pin the spot vCPU quota ordering (#641) ... * test: gate scripts/images/ shell suites + shellcheck in CI The eks-node/rds-postgres *_test.sh suites and shellcheck only ever ran by hand: spinifex.yml filters pushes on Go-only paths and no job invoked them, so a broken helper could land unnoticed. Add a `make test-images` target and a path-filtered image-scripts.yml workflow so both run automatically on scripts/images/** and images/** changes, kept separate from preflight so image-asset churn doesn't run on every Go contributor's commit. Also clears the shellcheck findings that would have started this gate red (SC1090 on a runtime-resolved source path, SC2154 x2 on eval-assigned variables shellcheck can't trace). * test: pin vCPU quota gate ordering in RequestSpotInstances Nothing asserted that RequestSpotInstances runs the vCPU quota gate before dispatching capacity or launching a VM, so a regression that dropped or reordered EnforceLaunch would pass silently. Add an ordering test using a real quota.Service (embedded JetStream KV) and real NATS responders standing in for the capacity and launch daemons, recording a global call sequence and asserting the gate's index precedes both. Verified by removing the gate: the first quota-KV read then comes from the post-launch charge instead, landing after capacity dispatch and launch, and the ordering assertion fails as expected.	2026-08-02 08:21:04 +10:00
Josh Siviour	6ce8b9ff18	fix: fail preflight when a gate tool is killed instead of reporting it clean (#633) ... A tool whose child dies by signal can still exit 0 through go tool, so govulncheck printed 'ok' after being OOM-killed and preflight passed without a vulnerability scan having run. Route lint, govulncheck and nilaway through scripts/run-gate.sh, which fails on a non-zero exit or on signal-death output.	2026-08-02 08:15:16 +10:00
Josh Siviour	135da2371e	fix: scope golangci-lint cache per worktree, correct eks-node-gpu catalog description (#629) ... * fix: scope golangci-lint cache per worktree golangci-lint's default cache under ~/.cache/golangci-lint is keyed by module path rather than worktree path, so parallel worktrees of this module collide and surface each other's lint issues, including from worktrees that have since been deleted. * fix: correct eks-node-gpu catalog description to match agent-only image The GPU node image never installs eks-token-webhook or a server/agent role selector, per scripts/images/eks-node-gpu/manifest.conf; it ships ecr-credential-provider only and is agent-only because GPU nodes never run the EKS control plane.	2026-08-02 08:01:45 +10:00
Julian Sommer	32d80b473a	feat: RDS v1 foundations (#592)	2026-07-29 10:38:22 +10:00
Julian Sommer	b505887b36	chore: add nilaway	2026-07-23 11:28:02 +10:00
Josh Siviour	88b57b1805	chore: give the integration tier its own CI job and drop it from preflight (#577) ... The tier ran inside the Build and Test job, serialised ahead of test-cover, and again in preflight. Neither placement bought signal that a parallel job does not: it provisions nothing and finishes in seconds, so the only effect was wall time on the two paths developers wait for most. It stays PR-blocking as its own job, which is what distinguishes it from e2e.yml's front gate — that one is push-triggered on a self-hosted runner and so never gates a PR before merge.	2026-07-23 07:56:07 +10:00
Josh Siviour	48749576c3	test: rebalance E2E into in-process integration tier + shared-guest live scenarios (#556) ... * test: add in-process integration test tier Roughly half of the behavioural tests under tests/e2e/ assert control-plane API shapes, authz and error codes that need no live environment, yet they can only run against a tofu-provisioned environment on a scarce self-hosted runner. This adds the tier they can move down to. The real gateway router runs in-process: SetupRoutes() is a pure function of GatewayConfig, so it serves from an httptest server over embedded NATS JetStream with real IAM/STS services and genuine SigV4 verification. Only the daemon side of the wire is faked -- the ec2.*/ebs.* NATS subjects a live node would answer are stubbed per test. Nothing is provisioned and no fixed ports are bound, so the tier runs concurrently on a shared build host and is safe as a CI front gate ahead of the live suites. It is build-tagged and kept out of preflight, which stays unit-only. * test(integration): add IAM/STS client accessors and port the zero-stub authz tests Adds IAMClient/STSClient plus ClientsWithCreds/ClientsWithSessionCreds so a test can act as a principal other than the seeded root user, and stubs the node-discovery subject in StartGateway so call paths that trigger discovery resolve immediately instead of riding out the fan-out timeout. Ports the four enforcement tests that guard on ec2:DescribeRegions, a gateway-local action, so they need no daemon-side stub at all. * test(integration): port remaining IAM/STS authz tests from e2e Ports TestIAMUserCRUD, TestIAMAccessKeyLifecycle, TestIAMUserAuthentication, TestIAMPolicyCRUD, TestIAMPolicyAttachmentEnforcement, TestIAMPolicyLifecycle, TestIAMGroupsLifecycle, TestIAMRolesAndProfiles and TestSTSAssumeRoleAndGetCallerIdentity from tests/e2e/iam/ into the in-process integration tier. Each test now self-bootstraps its own users/policies/roles instead of relying on shared suite-level state, since every integration test gets its own gateway and embedded NATS. TestIAMRolesAndProfiles stubs ec2.IamProfileAssociation.describe because DeleteInstanceProfile always counts live associations over NATS regardless of which guard actually fires. TestIAMCleanup is not ported: it only swept a shared long-lived E2E environment, and every ordering/dependency assertion it made is already covered by TestIAMPolicyLifecycle, TestIAMGroupsLifecycle and TestIAMRolesAndProfiles's own delete-conflict guards. * test(e2e): remove IAM/STS authz tests migrated to the integration tier All 13 signed-request authz tests plus TestIAMCleanup now have faithful, mutation-proven equivalents in tests/integration/ — the gateway dispatches IAM/STS actions directly with no daemon hop, so the in-process tier is a faithful and far cheaper substitute. The iam/ e2e suite now runs only TestIAMInstanceProfileAssociation and TestIMDS. Drop the now fully-dead iam_helpers_test.go alongside the six deleted test files, and remove harness.IAMGroupARN/IAMDeleteGroupBestEffort, which had no remaining caller anywhere in the repo once iam_groups_test.go was gone. * feat(integration): daemon-lite harness for key/tags/routetable/vpc Adds DaemonLite, an in-process stand-in for a live spinifex daemon that subscribes the real key, tags, route table, and VPC/subnet/SG/IGW service implementations directly to their NATS subjects, backed by memory and embedded-JetStream stores. Ports TestKeyPairs, TestTagManagement, TestRouteTableValidation, and TestReplaceRouteConvergence from tests/e2e/single so the integration tier drives genuine daemon-side business logic instead of StubSubject canned replies. vpcd (OVN) is out of scope, so its "vpc.create-sg"/"vpc.delete-sg" acknowledgements are stubbed with a fixed success reply — the SG record itself is still written by the real service impl before that event fires, so the stub only unblocks the synchronous round-trip rather than replacing tested behavior. * ci(e2e): gate live E2E jobs behind the in-process integration tier Runs the in-process integration suite first, on the ci-single runner host rather than inside a provisioned VM, so a broken control plane is caught before either live job pays the Spinifex install. Both e2e_single and e2e_multi now require it, and e2e_status treats a skipped integration job as a failure so the gating status cannot go green on a gate that never ran. Drops the shadow-mode suite selector: targeted per-diff suite selection is not the direction being pursued, and leaving a non-functional shadow job in the gating workflow only invited it to be mistaken for a live gate. * test(e2e): remove single/ tests superseded by daemon-lite ports TestKeyPairs, TestTagManagement, TestRouteTableValidation, and TestReplaceRouteConvergence now have proven-equivalent ports in tests/integration/ (mutation testing confirmed parity). Delete the e2e originals and their runX helpers, along with keypair_test.go's and tags_test.go's file-local helpers left with no caller. generateImportPubKey is shared by other single/ tests so it moves to helpers_test.go instead of being deleted. Slims the manifest-lint baseline for the removed create-call entries. * ci: run the lb e2e suite on multinode only Both e2e jobs filter SUITES by exclusion, and neither excluded lb, so the suite ran twice per e2e run: 354s on single and 654s on multi. Balancing across instances on one node is not a deployment mode we support, and ELBv2 proper requires two AZs, so the single-node run was redundant. Drops ~354s from the single-node job, which is the gate critical path. * test(integration): port 3 of 7 ECR tests to the in-process tier Wires the gateway's ECR control plane (repository/policy/lifecycle metadata over an in-process daemon-lite NATS subscriber, blob and manifest bytes via a memory object store, JWT issuer/verifier keyed off the existing IAM master key) into the integration harness, and ports TestECRControlPlane, TestECRGetLoginPassword, and TestECRIAMAuthorization_AssumedRolePullOnly from tests/e2e/ecr — each proven equivalent by mutation (evaluatePrincipalPolicy forced to allow, and the repo-describe path forced to not-found) before its E2E original was removed. The other 4 authorization tests (PullOnly, ReadOnly, DetachPolicyDeniesImmediately, DeactivatedKeyReturns401) stay on the E2E tier: AttachUserPolicy lacks the AWS-managed-ARN existence-check bypass that AttachRolePolicy/AttachGroupPolicy already have, so attaching AmazonEC2ContainerRegistryPullOnly/ReadOnly straight to a user fails NoSuchEntity in both tiers alike. Left untouched pending a fix to that shared IAM code. * fix(iam): accept AWS-managed policy ARNs on AttachUserPolicy AttachUserPolicy called getPolicyByARN unconditionally, so attaching an AWS-managed ARN to a user returned NoSuchEntity. AttachRolePolicy and AttachGroupPolicy already skip that lookup for AWS-managed ARNs and store them opaquely; the user path was missed when that guard was introduced. ListAttachedUserPolicies had the mirror of the same gap: it resolved every ARN through getPolicyByARN and silently dropped the ones it could not find, so a managed ARN would disappear from the listing rather than round-trip. Evaluation already handled these correctly via resolveAttachedPolicy, so only the write and list paths needed the guard. * test(integration): port remaining 4 ECR authorization tests Ports TestECRIAMAuthorization_PullOnly, _ReadOnly, _DetachPolicyDeniesImmediately, and _DeactivatedKeyReturns401 from tests/e2e/ecr now that AttachUserPolicy accepts AWS-managed policy ARNs. Each was proven equivalent by mutation before its E2E original was removed: PullOnly/ReadOnly/DetachPolicyDeniesImmediately fail when evaluatePrincipalPolicy is forced to always allow, and DeactivatedKeyReturns401 fails when the access-key-active check in resolveECRLongLivedPrincipal is bypassed. All mutants were reverted and a clean run confirmed before committing. tests/e2e/ecr/iam_authorization_test.go is now fully removed — every test it held has an in-process equivalent, and its helpers had no other callers in the package. * ci: select e2e suites by allow-list instead of deny-list Both jobs filtered SUITES by exclusion, so any suite named in neither deny-list ran in both jobs. That is how lb came to run twice per e2e run, and it makes adding a suite silently fan it out everywhere. Each job now names the suites it is eligible to run. Selection is still driven by SUITES, so this is behaviour-preserving for the current default: single/iam/cert on the single-node job, multinode/lb/cert on the multi-node job, exactly as before. eks, ecs and storagegrowth are listed as eligible on the single-node job but are not yet in the default SUITES. * test(e2e): delete seven tests with no coverage TestFinalClusterStats and TestInstanceLaunch assert nothing beyond what other callers of their shared helpers already imply. The multinode baseline, volume, guest-SSH, and preflight tests duplicate single-node coverage or exercise per-guest behaviour that isn't cluster-specific. TestMultinodeInstanceDistribution can't fail: three independent RunInstances calls with no placement group make a strict spread assertion flaky and a loose one meaningless. Preflight checks (/dev/kvm writability, peer SSH reachability) move into requireMultiNodeFixture so the package fixture singleton still fails fast with a clear message instead of disappearing along with the dedicated test. The remaining callers of needInstanceTrio (CrossNodeOps, CrossNodeGateway, NodeFailure, VolumeDurability) still trigger the shared trio launch on their own, so removing the distribution and volume-lifecycle tests doesn't leave it unbuilt. Also slims the manifest-lint baseline entry for the deleted multinode volume test. * fix(e2e): require full 3-node spread in TestMultinodeSpread SpreadPlacement passed at unique>=3, warned via t.Logf at unique==2, and only failed below that. A 33% spread failure (two instances colocated) passed with a log line. The 243.6s NAT-egress scenario that follows only proves what it claims to if the private trio actually landed on three distinct chassis, so that premise needs to be a hard assertion, not a warning. * refactor(e2e): drop the unused spread-assertion harness helpers AssertSpreadAcrossNodes and CountInstancesPerNode lost their only caller when the instance-distribution test was removed. Beyond being dead, the assertion helper documented minNodes=1 as an acceptable logging-only mode inherited from the bash driver, which is the pattern that let a placement test pass while every instance landed on one node. InstanceHostingNode and its ps helpers stay; five multinode tests use them. * fix(ec2): stop leaked goroutines on prepareEFIVolume error paths newViperblock starts the chunk uploader and WAL syncer unconditionally, but prepareEFIVolume only released them on the happy path via Close(). Six error returns between acquisition and Close left both goroutines running. Add a defer immediately after acquisition, matching the sibling prepareRootVolume; Close() remains the durability boundary. * test(predastore): add real predastore fixture for integration tests Extract predastore_integration_test.go's inline daemon startup into a reusable testutil/predastore package so tests/integration can point DaemonLite-style wiring at a real S3-compatible backend instead of an unreachable host. Kept as its own leaf package rather than folding into the shared testutil package: it transitively imports predastore's quicclient, whose package-level connection pool starts a permanent goroutine at init that would trip goleak checks in any of the ~80 other packages that import testutil for unrelated NATS helpers. * test(integration): migrate CreateVolume to assert real predastore persistence Add StartVolumeDaemonLite, an opt-in DaemonLite variant wired to the real predastore fixture, and migrate TestCreateVolume_PersistsToRealPredastore off memory-backed DaemonLite so CreateVolume/DescribeVolumes exercise viperblock's real S3 backend end to end instead of stopping at input validation. * test(integration): add real storage-state oracle for CreateVolume DescribeVolumes only proves the handler's own read path agrees with its own write path, both via the same deserialization; it cannot catch a bug in a field it never re-derives (e.g. a unit-conversion error in the bytes handed to viperblock, since VolumeMetadata.SizeGiB is copied from the request directly). vbscan reads a volume's persisted config.json and chunk objects straight over S3, independent of any spinifex handler code, so TestCreateVolume_PersistsToRealPredastore can assert on what viperblock actually wrote rather than only on what the handler claims. vbscan is a local reimplementation of scripts/vbrefscan's read-only scan rather than an import: spinifex cannot depend on mulga, the module that contains it. It only reimplements the config/chunk-listing half of vbrefscan — the checkpoint-based reference-integrity half needs viperblock.ParseBlockCheckpointBytes, which does not exist in the viperblock version spinifex currently depends on (v1.12.1) and is also moot for this volume: CreateVolume never calls SaveLiveCheckpoint, so a freshly created volume has no checkpoint to scan regardless of version. * refactor(test): move the predastore daemon fixture out of the coverage scope test-cover runs go test ./spinifex/..., so anything under spinifex/ is compiled and measured. The daemon fixture cannot be exercised by unit tests, so it reported a coverage gap no unit test could close. The e2e harness already avoids this by living under tests/; the fixture now does the same. vbscan stays put, being ordinary library code with its own unit tests. * test(integration): migrate RunInstances multi-count expansion off live guests RunInstances with MinCount/MaxCount=2 expands into a single per-node targeted launch (distributeInstances in gateway/ec2/instance/placement.go) before the daemon-facing NATS hop, so the real gateway router plus a stubbed daemon response is sufficient to prove it — no throwaway VM is needed to exercise the expansion logic itself. The live single-node E2E test's WaitForInstanceState/TerminateInstances sequence was teardown hygiene for the real guests it booted, not an assertion of the expansion behavior, so it is dropped rather than emulated. The migrated test additionally asserts the per-node launch request itself carries MinCount=MaxCount=2, proving the count actually propagates rather than the launch succeeding by coincidence or via the zero-daemon degenerate path in placement.go. * docs: drop a doc reference to the deleted live test * test(integration): migrate launch-template RunInstances expansion off live guests RunInstances --launch-template resolves ImageId/InstanceType from the stored template entirely gateway-side (expandLaunchTemplate, before validation/placement/the daemon-facing NATS hop), so proving it forwards the resolved fields never needed a real guest boot. Ports the assertion to tests/integration via a new opt-in StartLaunchTemplateDaemonLite (real LaunchTemplateServiceImpl over embedded JetStream) plus a direct subscription on the per-node launch subject to capture what the gateway actually dispatched, rather than only the client-visible response. The live test keeps its CRUD/versioning lifecycle (create/modify/delete version, tag-filtered describe) — none of it touches a guest — and drops only the RunInstances-from-template + wait-for-running + terminate tail, whose only job was waiting out a real boot to read back fields already proven resolved. * test(integration): migrate KV-only account-scoping checks off live tenants Ports the KeyPair/VPC-Subnet/IGW-EIGW/AccountSettings cross-account isolation assertions from tests/e2e/single/account_test.go to the in-process integration tier. Each of these resources resolves ownership from a plain account-scoped KV key (utils.AccountKey(accountID, id), or an account-keyed settings record) with no vm.Manager or real guest involved, so DaemonLite now also wires the real EgressOnlyIGWServiceImpl and AccountSettingsServiceImpl alongside the existing VPC/IGW/Key services. Two independently minted tenant accounts (via IAMService.CreateAccount/CreateUser/CreateAccessKey/CreatePolicy/ AttachUserPolicy, in-process — the same calls runAccountCreate makes without the CLI/stdout round-trip) replace the live tier's carousel so isolation is checked between real, separately owned accounts. Left live: instance scoping (Step2) and volume scoping (Step3) — checkInstanceOwnership, the actual cross-account gate for stop/start/ reboot/terminate/attach-volume, runs daemon-side only after a real vm.Manager lookup, so it cannot be exercised without a live guest. Snapshot scoping (Step5) needs a real backing volume for CreateSnapshot (viperblock, out of DaemonLite's scope), so it stays live entirely rather than split off only its two KV-only assertions. Global-resources (Step9) carries no account branching in the handlers at all, so asserting its equality across two accounts wouldn't exercise anything a fake node responder didn't already trivially satisfy — not migrated. * test(integration): migrate launch-template CRUD lifecycle off live guests TestLaunchTemplates booted zero guests: its RunInstances coverage had already moved to TestRunInstances_LaunchTemplateExpansion, leaving only template CRUD/versioning assertions occupying a full live-cluster slot. Port the remaining create/describe/version/modify/delete lifecycle, including the SourceVersion merge and a nested timestamp round-trip, into TestLaunchTemplates_CRUDLifecycle against the in-process integration gateway. * refactor(tests): move spot instance lifecycle coverage to integration tier RequestSpotInstances is a mock over on-demand RunInstances (no bidding, interruption, or reclamation), so booting real VMs in the e2e tier proved nothing spot-specific. Migrate the SIR CRUD lifecycle (request, describe, cancel, close-on-terminate, insufficient-capacity error) to a NATS-level integration test that exercises the real gateway orchestration and daemon service impl without a live guest. Real-VM-boot assertions are already covered by TestRunInstances_MultiCountExpansion and the e2e baseline tests. * fix(tests): share one embedded NATS server across the integration package ClientTokenStore in spinifex/gateway/ec2/instance is a process-wide sync.Once singleton bound to whichever *nats.Conn first initialises it. Every integration test booted its own embedded NATS server, so the first RunInstances-calling test captured the singleton, and any later test inherited a KV handle bound to that server's now-closed connection. TestMain now boots one embedded NATS+JetStream server for the whole package; every StartGateway call connects into its own dynamically provisioned NATS account (accountAuthenticator in shared_nats.go), giving each test a private JetStream namespace despite the shared server, and none of the per-test connections are closed until every test has finished - whichever one the singleton latches onto stays live for the run. Also wire the suite into the required PR gate: it was previously only invoked from e2e.yml's push-triggered (main/dev), self-hosted "Integration (front gate)" job, so a regression here never blocked a PR. The whole package runs in single-digit seconds, so it now runs inline in `make preflight` and as its own step in spinifex.yml's build_and_test job. * test(integration): migrate serial-console access checks off live guests Enable/Disable/GetSerialConsoleAccessEnabled is pure gateway+account-settings logic with no guest involved, so it moves to the in-process integration tier against the real AccountSettingsServiceImpl wired into DaemonLite. * test(integration): migrate DescribeImages round-trip off live guests runImage's AMI describe round-trip needs no real cluster: it registers an AMI against a hand-seeded snapshot via the real ImageServiceImpl, backed by a new StartImageDaemonLite helper wiring RegisterImage/DescribeImages onto an in-memory object store. * test(integration): migrate predastore object lifecycle off live guests runPredastoreObjectLifecycle only needs a real predastore daemon, not a tofu-provisioned cluster: it now runs against tests/fixtures/predastore's in-process fixture, covering the same create/put/get/delete/list contract. * test(integration): migrate ListFoundationModels catalog check off live cluster ListFoundationModels is pure gateway-side static catalog logic with no NATS/daemon hop, so it moves to the integration tier via a new BedrockClient harness accessor; this tier configures no credential resolver, so the Anthropic entry's absence is asserted explicitly rather than left optional. * test(integration): cover RunInstances client-token idempotency Proves replaying the same ClientToken through the real gateway returns the original reservation without a second daemon dispatch, and a different token launches independently. * test(integration): cover RunInstances launch-template expansion Proves version selection (omitted Version resolves to the template default, explicit Version selects that version) and field precedence (explicit RunInstances parameters override the template, unset fields inherit it) through the real gateway. * test(integration): cover RunInstances malformed-input rejection Proves a malformed ImageId and a malformed targeted-launch CapacityReservationId are rejected before any NATS dispatch, with no daemon stub registered so a regression that let either through would fail fast instead of hanging. * test(integration): cover RunInstances iam:PassRole enforcement Proves a caller granted ec2:RunInstances but not iam:PassRole on an instance profile's role is denied, and the identical session succeeds once PassRole is granted for that exact role ARN. * test(integration): cover RunInstances placement-group strategy routing Adds a placement-group daemon-lite wiring a real PlacementGroupServiceImpl over NATS, then proves spread strategy assigns exactly one instance per node while cluster strategy pins the whole launch to the single highest-capacity node. * test(integration): cover RunInstances capacity distribution across nodes Proves the default (non-placement-group) path packs a multi-instance launch across nodes proportionally to spare capacity rather than exhausting one node first. * test(e2e): merge SameSGComms and SGToSGDatapath around one client/target pair Both booted their own instance pair to check SG-to-SG datapath enforcement and same-default-SG east-west connectivity. Merging them around one shared client/target pair cuts four boots to two, and converting the inline harness.Step sequence into named t.Run stages (PortGroupMembership, AllowedTraffic, DeniedTraffic, RevokeAndVerifyBlocked, ReauthorizeAndVerifyRestored, SameSGComms) gives each its own pass/fail attribution instead of one opaque test result. SameSGComms now runs last and joins the default SG onto each ENI via ModifyNetworkInterfaceAttribute only after every SG-to-SG allow/deny assertion has already recorded its own result, since the default SG's self-reference rule admits all traffic between co-members and would otherwise defeat the denied-traffic check. * test(e2e): merge default-SG reachability, guest DNS, and egress enforcement DefaultSGReachabilityBaseline, GuestDNSResolution, and SecurityGroupEgress each booted their own instance to check one facet of "a guest is reachable only when policy says so": ingress deny/authorize, resolver + WAN egress, and egress ACL enforcement. Merging them around one shared guest cuts three boots to one, with named t.Run stages (Deny, Authorize, Resolver, WANEgress, RevokeEgress/RestoreEgress) giving each its own attribution. The egress revoke/restore round-trip now mutates this scenario's own dedicated SG instead of the shared default SG the rest of the suite depends on, isolating its egress-rule mutation from every other test. * fix(ci): single-quote e2e test_run regex across the ssh boundary * fix(e2e): anchor SG egress-enforcement probe to a WAN IP, not the gateway * test(e2e): merge single-node guest churn tests into one durability scenario Combines TestSSHProbe, TestENIHotplug, TestENIHotplugReconcile, TestVolumeDurability, TestModifyInstanceAttribute, and TestRebootInstance into TestGuestChurnDurability, a single shared-guest scenario that re-verifies a data-integrity sentinel sha256 after every perturbation stage (ENI hotplug, daemon restart, ENI detach, instance-type resize, stop/start, reboot). This proves the sentinel survives the full churn sequence rather than just one isolated stop/start. * fix(vpc): clear subnet drop gate when installing egress reroute A subnet with an attached IGW but no 0.0.0.0/0 route carries a priority-1100 drop gate on the VPC router. When a route later lands (NATGW reroute at 900, IGW reroute at 1000) the reroute is installed but the drop gate was never removed, so the higher-priority gate wins and black-holes the subnet's egress in lr_in_policy before it reaches routing or SNAT. This left NAT-gateway private-subnet egress with 100% packet loss: the SNAT stage saw zero packets and no conntrack was committed. Remove the subnet's drop gate for the same prefix once a reroute is installed. Idempotent and prefix-scoped, so a specific-prefix reroute is a no-op and leaves the default gate intact. * test(e2e): merge VPC subnet, NAT gateway, and EIP single-node tests Merges the former TestVPCSubnetE2E, TestNewVPCEgressBaseline, TestNATGateway, and TestInstanceEIP into one staged scenario, TestVPCEgressPaths, around one scenario-owned VPC and its two guests, cutting five guest boots to three. The public guest serves PublicSubnetEgress, then the NAT Gateway bastion hop, then (last, since association rewrites its public IP) the EIP flip. The private guest serves the NAT Gateway path, launched directly into the same VPC's already-existing private subnet instead of a dedicated one. RouteBeforeSubnet keeps its own second VPC and instance since it exercises the implicit main-route-table code path, a different one than the explicit route table this scenario's own VPC uses for its public subnet — collapsing the two would drop coverage of whichever path got merged away. The NAT Gateway up/down flip now repeats for an env-tunable round count (SPINIFEX_VPCEGRESS_NAT_ROUNDS, default 1), matching every other merged single-node scenario. * test: TEMPORARY mutation-check — revert a795c2b2 drop-gate clear Reverts the RemoveSubnetEgressDrop call in ensureSubnetEgressAtPriority to confirm TestVPCEgressPaths/Round1/NATGatewayUp actually detects the regression it guards against. This commit will be reverted immediately after the live run confirms a failure. * test: TEMPORARY mutation-check v2 — neutralize RemoveSubnetEgressDrop Restores the real a795c2b2 fix in ensureSubnetEgressAtPriority (the first mutation attempt only reverted this call site and was masked by the independent gate-recompute path in routetable service_impl.go's CreateRoute, which also clears the drop gate on every 0.0.0.0/0 route add and predates a795c2b2). Neutralizing RemoveSubnetEgressDrop itself disables gate-clearing regardless of caller, reproducing the literal regression. Will be reverted immediately after live verification. * Revert "test: TEMPORARY mutation-check v2 — neutralize RemoveSubnetEgressDrop" Mutation-check confirmed: neutralizing RemoveSubnetEgressDrop reproduced a795c2b2's regression exactly (stale priority-1100 drop gate outranking the priority-1000 IGW reroute, confirmed via OVN flow dump; PublicSubnetEgress failed with an SSH handshake timeout to the public instance). Restoring the real fix; igw.go is now byte-identical to a795c2b2. * fix(e2e): stop pruning shared multinode SSH key pem mid-run The multinode package fixture cached Fixture.Artifacts once, computed from harness.ArtifactDir(t, env) against whichever top-level Test* happened to win the pkgFixOnce race. That helper both names the directory after that one test and registers a t.Cleanup that removes it once that test passes. needKeyPair wrote the shared EC2 key pair's pem into that same directory, so as soon as the winning test finished, its own artifact-dir cleanup deleted the pem every later test in the package still needed to SSH into fresh guests, even though the underlying EC2 key pair itself was untouched (its DeleteKeyPair cleanup already correctly drains at process Close() in TestMain). Fix: give Fixture an ArtifactDir(t) method — the same per-call pattern already used by the gpu/single/storagegrowth/iam fixtures — so each diagnostic dump site recomputes its own dir against the CURRENT test. The shared key pair pem now lives under Fixture.TmpDir, which is genuinely package-scoped and only removed once in TestMain after m.Run() completes. * refactor(e2e): parallelize LB e2e scenarios on multinode clusters Each of the 8 TestLoadBalancer scenarios already stands up its own dedicated LB + target group (ModifyListener/ListenerRules mutate their LB's listener in place, so it can never be shared). The bottleneck was running them serially: each teardown made createActiveLB back off waiting for the previous LB's sys.micro slot to be reclaimed before the next one could stand up. On multinode clusters (peer != "", i.e. the 3-node cluster this suite targets) there's spare capacity to stand up all 8 dedicated LBs at once, so scenarios now run as t.Parallel() subtests instead of serial create/teardown/backoff. Since each scenario owns an independent LB, one never reaching active only fails that scenario — it no longer cascades into five sequential 5-minute timeouts, so the serial fail-fast gate is skipped in this mode; concurrency itself bounds the worst case to a single timeout window. Single-node dev boxes (no peer) keep the original serial path, including the fail-fast gate, since they lack the spare capacity for concurrent LB standup. No subtest body, assertion, or call signature changed — only the dispatch/gating logic in TestLoadBalancer. * fix(ci): gate e2e jobs on the suites they actually run * fix(e2e): serialize internet-facing LB subtests among themselves Live multinode verification showed InternetFacing_ALB and InternetFacing_NLB running concurrently each clobber the other's public-LB DNS record mid-registration, so neither ever resolves within the 90s wait. The old serial suite never had two internet-facing LBs alive at once, so this never surfaced. Drop t.Parallel() from the three internet-facing subtests so their register+resolve windows never overlap each other, while leaving the internal subtests parallel as before. They may still run alongside the internal group, which resolves internal (not public) DNS and isn't affected. * fix(e2e): give each ALB LB scenario its own security group Concurrent internal subtests all created ALBs with no explicit SecurityGroups, so each ALB's ENI fell back to the one shared VPC default SG. One scenario authorizing/reconfiguring its own listener racing another mutating the same SG made Internal_ALB_ModifyListener intermittently fail to serve on the modified port under parallel load. Each ALB scenario now creates a dedicated SG (createScenarioSG), authorized only for the ports that scenario's listener(s) use, and passes it to CreateLoadBalancer. The shared app-instance/probe-client SG is untouched, since SG ingress is enforced at the destination ENI regardless of the source ENI's SG. NLBs are unaffected: they already get an auto-managed per-LB SG when none is supplied. * test(e2e): annotate intentional create-path calls for manifest-lint The vpcegress and per-scenario LB security-group tests create raw AWS resources on purpose — the egress datapath, the route-before-subnet ordering, and per-scenario SG isolation are the subjects under test, so they cannot route through a shared Ensure* fixture. Mark each with // e2e:allow-create and slim the baseline of entries for the removed single-node network tests. * test(e2e): restore live spot terminate-close and cancel-running coverage The integration-tier spot test calls CloseForInstance directly and infers cancel-keeps-running from the SIR's InstanceId, because DaemonLite has no vm.Manager to drive the real TerminateInstances -> daemon teardown cleaner -> RemoveFromSpotRequest -> CloseForInstance chain or to observe a real VM's running state. Add a thin live e2e test that drives both terminal transitions through a real backing VM: cancel one spot-backed instance and observe it stays running, terminate the other and observe its SIR reach closed through the genuine async delegation. * fix(network): refresh orphan-EIP prune liveness to spare mid-pass launches pruneOrphanEIPs listed OVN NAT rows live but matched them against the intent snapshot captured at the start of the reconcile pass. Because the apply phase can block for tens of seconds on guest-port datapath waits, a guest launched during the pass had a live dnat_and_snat row absent from the stale snapshot, so the prune swept it and blackholed the guest's auto-assigned public IP until the next pass re-added it — an intermittent multinode SSH-dark flake. Add an optional FreshIntent re-read callback (wired in vpcd to LoadIntentFromKV) and union its ports into the live set before pruning, so a mid-pass launch counts as live. Skip the prune entirely when the re-read fails rather than risk a false sweep against a snapshot known to be stale. Self-healing and idempotent: a genuine orphan absent from both snapshot and fresh read is still pruned next drift. Single-node behaviour is unchanged (callback nil in tests).	2026-07-21 16:46:18 +10:00
Julian Sommer	47604f43dc	chore: go 1.26.5	2026-07-09 09:43:32 +10:00
Josh Siviour	24bd50a53c	feat: ecs-node AMI (alpine + containerd + ecs-agent baked) (#415) ... Adds the spinifex-ecs-node AMI variant: alpine 3.21.7 + containerd + runc + cni-plugins + ca-certificates with the ecs-agent binary baked to /usr/local/bin and three OpenRC services wired in dependency order (ecs-ca-install -> containerd -> ecs-agent). - manifest.conf drives build-system-image.sh (no host mounts). - ecs-ca-install.initd: first-boot oneshot installing the cloud-init delivered gateway CA into the system trust; no-op when absent. - ecs-agent.initd: sources /etc/spinifex-ecs/agent.env, backgrounds the agent after net + containerd. - 10-bridge.conflist: CNI bridge for task networking (Sprint 4d). - setup.sh: exec bits, /opt/cni/bin symlink, content-store dirs, ttyS0 console reorder + fast boot menu. - Makefile: build-ecs-node-image / import-ecs-node-image targets.	2026-06-24 21:54:35 +10:00
Josh Siviour	93db7e4d1b	feat(ecs): ecs-agent skeleton + containerd image pull (Sprint 4c) (#414) ... * feat(ecs): ecs-agent skeleton + containerd image pull (Sprint 4c) Add the ecs-agent binary that runs inside an ECS container instance: it resolves host identity from IMDS, registers on the Layer-2 NATS bus, and heartbeats while alive. Image pulls go through a containerd-backed ImagePuller behind an interface, with an ECR resolver that mints GetAuthorizationToken creds from the instance role. Extract the IMDS-credential and ECR-auth flow out of ecr-credential-provider into internal/imdscreds and internal/ecrauth so the agent and the kubelet provider share one implementation. Add spinifex/handlers/ecs/bus as the single source of truth for Layer-2 subject builders and wire payloads, shared by the agent now and the scheduler in Sprint 4e. * build: pin genproto via replace to fix containerd/v2 ambiguous import go.opencensus.io -> grpc@v1.33.2 drags the pre-split monolithic genproto into the module graph; it still ships googleapis/rpc/status, which collides with the split genproto/googleapis/rpc module that containerd/ttrpc needs. The main module never imports the monolith, so a plain require bump is dropped by 'go mod tidy'. Pin it with a replace to a post-split version so the fix survives tidy and contributors can run 'go mod tidy' freely. * fix(ecs): agent boot failsafes from runtime verify - Build the ECR gateway HTTP client lazily on first image pull instead of at boot. A missing/malformed gateway CA no longer kills the agent before it can register + heartbeat, matching the best-effort containerd path. - Actively probe containerd with IsServing in runtime.New. The client dials lazily, so a dead socket previously slipped past the boot-time 'containerd unavailable' warn and only surfaced on the first pull. - Make the heartbeat interval configurable via ECS_HEARTBEAT_INTERVAL (default 30s) for test/verify and slow-link tuning.	2026-06-24 21:18:03 +10:00
joshsiv-mulga	022c6cfd2a	Update Makefile ... Signed-off-by: joshsiv-mulga <mrk@mulgadc.com>	2026-06-12 15:26:11 +10:00
Josh Siviour	d076a434d4	Feat/e2e targeted suites (#352) ... * test(e2e): add targeted suite selector + shadow-mode CI step select-suites maps a diff to the e2e suites it must run: changed files → services (manifest path/additional_paths) → consumer closure over depends_on vs publishes/subscribes → suites whose covers intersect. Special cases force-all (infra/harness/workflow, ≥3 services, submodule bumps) or a lone suite dir; empty diffs fall back to default suites. An AST scan of each suite's harness.Ensure* call graph derives per-test service coverage and emits an optional -test.run subset (RUN_PATTERN). Wired into e2e.yml as a non-gating 'Select Suites (shadow)' job that logs the computed selection to the run summary so predicted vs actual can be compared before the selector drives SUITES. * ci(e2e): add dispatch-only per-suite job blocks grouped by env Restructure the looped e2e flow into env-grouped provision -> matrix suites -> teardown blocks so each suite surfaces as its own node in the Actions graph instead of a single green tick: single_provision -> single_suite [single,cert,lb,eks] -> single_teardown multi_provision -> multi_suite [multinode] -> multi_teardown Provision resolves the VM/cluster and ships test binaries, emitting WAN IP + ssh key path as job outputs; each matrix leg runs one suite over SSH and uploads its own junit/analysis artifacts; teardown (if:always) reuses the persisted tofu workspace on the same runner to destroy. workflow_dispatch only -- runs alongside the gating e2e.yml for validation and posts no commit status. * test(e2e): add manifest drift lint (fixture-create + NATS subject) Bead 5 of the targeted-suite-selection plan. Two ratcheted drift guards in tests/e2e/manifest-lint, wired into make preflight and the spinifex.yml PR job: - Fixture lint: AST-flags direct resource-create AWS calls (c.EC2.RunInstances, c.ELBv2.CreateLoadBalancer, ...) in suite test files, steering them through harness.Ensure* fixtures. ~104 pre-existing call sites are captured in a checked-in baseline; the lint fails only on NEW creates. Intentional create-path tests opt out with an inline // e2e:allow-create. - Subject lint: extracts NATS subject literals from the natsSub{} registry and literal Subscribe/Publish/Request args, cross-referenced wildcard-aware against service-interfaces.yaml. Code subjects missing from the manifest fail (ratcheted); manifest subjects with no static reference warn only (many are produced dynamically). Fixed real drift surfaced on first run: declared the five spinifex.cluster.shutdown.* phase subjects under spinifex_daemon. manifest-check is now CI-enforced alongside manifest-lint (previously preflight/local-only). * feat: run-scoped e2e teardown sweep Tag every Ensure* fixture resource with e2e:run=<run-id> when SPINIFEX_E2E_RUN_ID is set, and add a teardown.test suite that sweeps everything carrying that tag at end of run — reclaiming resources leaked by a crashed suite, an interrupted run, or a persistent cluster. - harness/sweep.go: tagRunResources (EC2 CreateTags) + tagRunELB (ELBv2 AddTags) + SweepRunResources in dependency order (LB, instances, NAT, snapshots, volumes, images, SGs, subnets, key pairs). Blank run id no-ops both tagging and sweeping, so the sweep can never touch an untagged or foreign resource. - harness/fixtures.go: stamp the run tag on each created resource. - tests/e2e/teardown: thin suite that runs the sweep; skips without a cluster or run id, logs the report, never fails the run. - e2e-suites.yml: build + ship teardown.test, pass the run id into suite legs, sweep on the VM in both teardown jobs before tofu destroy.	2026-06-11 21:09:50 +10:00
Julian Sommer	773341248d	feat: use STS tokens on frontend (#322)	2026-06-09 15:40:35 +10:00
Josh Siviour	467e4de715	feat(eks): publish eks-node AMI to R2 + spx catalogue entry (#313) ... * feat(eks): publish eks-node AMI to R2 + spx catalogue entry Add a spinifex-eks-node catalogue entry so operators install the EKS node image with a single command: spx admin images import --name spinifex-eks-node The entry pulls the Alpine+K3s image from iso.mulgadc.com, verifies its sha256, and stamps spinifex:managed-by=eks — the tag the control-plane and node-group launchers resolve the boot AMI by, so no extra config is needed. Add scripts/publish-system-image.sh (+ make publish-eks-node-image) to convert the built raw to a compressed qcow2, emit a checksum sidecar, and upload both to the R2 bucket backing iso.mulgadc.com/system-ami via the S3 API. * ci(eks): add manual workflow to build + publish eks-node AMI to R2 Build the eks-node AMI on the runner (libguestfs) and publish via publish-system-image.sh using the existing CF_R2_* secrets. Manual workflow_dispatch with a dry-run input. Fix the publish script's default bucket to spinifex-iso (the bucket iso.mulgadc.com is served from).	2026-06-08 09:48:59 +10:00
Josh Siviour	f14fb64562	feat(eks): unify eks-server + eks-agent into one eks-node AMI (#288) ... * feat(eks): unify eks-server + eks-agent into one eks-node AMI Collapse the two K3s AMIs into a single spinifex-eks-node image whose role (server vs agent) is chosen per-instance at first boot, halving build time, storage, and CVE-rotation surface. - scripts/images/eks-node/: superset manifest + merged setup.sh; all role services baked but only crond + the new eks-node-role selector enabled. - eks-node-role first-boot selector reads SPINIFEX_K3S_ROLE, rc-update adds + starts the matching services (server: k3s + eks-token-webhook + k3s-first-boot publisher; agent: k3s-agent), records /etc/spinifex-eks/role, self-disables. Paths are env-overridable; eks-node-role_test.sh covers all five cases. - mulga-eks-etcd-snapshot.sh early-exits on agent nodes. - k3s_server_vm.go seeds SPINIFEX_K3S_ROLE=server into first-boot.env. - Makefile/dev-install: build-eks-node-image / import-eks-node-image replace the server+agent targets. - Delete scripts/images/eks-server + eks-agent. The launcher resolves the AMI by the spinifex:managed-by=eks tag, so running clusters (which hold the resolved AMI ID) are unaffected. AMI bake + cluster boot are operational integration steps, not yet run. * fix(eks): order eks-node-role after cloud-config/cloud-final The first-boot role selector reads /etc/spinifex-eks/first-boot.env, which cloud-init delivers via write_files in the cloud-config/cloud-final stages. The selector only depended on net+cgroups, so on a fast boot it could start before write_files landed and read a missing role env. Add an after-ordering on cloud-config + cloud-final so the env file is always present first.	2026-06-04 12:28:56 +10:00
Josh Siviour	b0227e7eeb	chore(ansible): remove ansible tree, uplift to mulga/ansible/ (#287) ... Ansible orchestrates the whole stack (spinifex + predastore + viperblock + OVN + tofu hosts) and its spec/beads live in mulga. Moving it to top-level mulga/ansible/ ends the per-change submodule-pointer-bump friction. - git rm -r scripts/ansible/ - Makefile: drop ansible-dev-* / ansible-cluster-bootstrap targets + .PHONY The tree lands at mulga/ansible/ (flat copy, Option B). History pivot: tag feat/uplift-ansible-snapshot-pre-move marks the pre-move HEAD. Plan: docs/development/improvements/ansible-uplift-to-mulga.md (in mulga).	2026-06-04 11:23:08 +10:00
Josh Siviour	bbff004b45	feat(eks): 6b create-cluster → ACTIVE (single-node SQLite + mgmt-IP NLB stop-gap) (#280) ... * feat(eks): cluster_state KV helpers — ClusterMeta + status CAS + prefix sweep Stage 1 of Sprint 6b CreateCluster lifecycle. Introduces the first read/write helpers against the eks-account-{accountID} bucket so later stages (oidc keypair, NLB wiring, cluster reconciler) can persist state without re-inventing CAS/sweep logic. - ClusterMeta struct holds the DescribeCluster source of truth (status, endpoint, OIDC issuer, CA, VPC config, control-plane VM + ENI, NLB ARN). - ClusterStatus enum mirrors the AWS EKS string set verbatim. - PutClusterMeta is unconditional (CreateCluster path); GetClusterMeta returns ErrClusterNotFound on key absence so the service boundary can map to ResourceNotFoundException. - SetClusterStatus is revision-checked with bounded CAS retries so the reconciler and CreateCluster body cannot wedge each other. - DeleteClusterPrefix sweeps every clusters/{name}/* key; called from DeleteCluster after the OIDC private key is zeroized. Tests cover round-trip, missing/corrupt blob, idempotent status set, DELETING transition from every prior state, concurrent revision bump recovery, sibling-cluster preservation, empty-bucket no-op. Plan: docs/development/feature/eks-6b-create-cluster.md * feat(eks): per-cluster ECDSA-P256 OIDC keypair generation + zeroize Stage 2 of Sprint 6b. Lays the at-rest primitives the CreateCluster body will call before launching the K3s server VM. Pure crypto helpers — no daemon, no launcher, no NATS bus. - GenerateClusterOIDCKeypair mints an ECDSA-P256 keypair per cluster (matches eks-v1.md Q8), PEM-marshals the private as PKCS8, envelope-encrypts it via handlers_iam.EncryptSecret (AES-256-GCM, AwsGW masterKey), writes the ciphertext to OIDCSigningKeyKey, and publishes the public JWKS document to OIDCJWKSKey. Returns the JWKS bytes so CreateCluster can hand them straight to user-data without a follow-up KV read. - JWKS shape mirrors what handlers_sts.JWKS already decodes for IRSA verification (RFC 7517 EC P-256, kid = base64url SHA-256 of SPKI DER, alg ES256, use sig). Types are defined locally to avoid a handlers_eks → handlers_sts → handlers_eks import cycle; JSON wire compat is the contract. - LoadClusterOIDCPrivateKey is the recovery-path read for future stages (reconciler restart, restore-snapshot tooling). Validates PKCS8 parses to an ECDSA P-256 *PrivateKey and rejects anything else. - ZeroizeClusterOIDCKey overwrites the encrypted blob with zeros of the same length, then deletes. DeleteCluster calls this *first* so a crash mid-teardown leaves no recoverable key material. Idempotent on missing key — supports retry semantics. Tests cover JWKS shape (RFC 7517 EC P-256 with 32-byte SEC1 coordinates), per-cluster key uniqueness, round-trip private-key decrypt matches JWKS public, wrong-master-key fails, missing-key ErrClusterNotFound mapping, zeroize idempotency, JWKS survives zeroize (carries no secret material), padCoord left-pad correctness. Plan: docs/development/feature/eks-6b-create-cluster.md * feat(eks): cluster SG + NLB ensure/delete helpers Stage 3 of Sprint 6b CreateCluster lifecycle. Adds the customer-account SG provisioning + per-cluster NLB plumbing the CreateCluster body will call between OIDC keypair (Stage 2) and K3s VM launch (Stage 4). Pure interface-call helpers — no daemon, no KV, no NATS bus. - EnsureClusterSGs creates (or returns) eks-cluster-{name}-control-plane-sg and eks-cluster-{name}-nodegroup-sg in the customer VPC, tagged spinifex:managed-by=eks + spinifex:eks-cluster=<name> + spinifex:eks-role. Idempotent on group-name + vpc-id pair. DeleteClusterSGs is best-effort with continue-on-error sweep. - EnsureClusterNLB provisions the eks-<name> NLB (internal scheme), the eks-<name>-cp target group (TCP, port 6443, target-type ip per eks-v1.md Q13), and the TCP :443 forward listener. Returns ClusterNLB {LoadBalancerArn, TargetGroupArn, ListenerArn, DNSName} so CreateCluster can persist every ARN in ClusterMeta. Idempotent on cluster name; resource-name length validated against the AWS 32-char cap so a too-long cluster name fails with a clear error instead of a generic InvalidParameterValue downstream. - RegisterClusterTarget + DeregisterClusterTarget bind the K3s server VM ENI IP to the TG on :6443. Stage 4 calls Register once the VM has its ENI IP; DeleteCluster calls Deregister before VM termination. - DeleteClusterNLB tears down NLB + TG. DeleteLoadBalancer already cascades listener cleanup inside the ELBv2 impl. LB sweep error takes precedence so DeleteCluster surfaces the more visible failure first while the TG sweep still runs. Narrow sgProvisioner + nlbProvisioner interfaces in-package give tests hand-rolled recorder fakes; the production VPCService + ELBv2Service satisfy them structurally so no adapter is needed at Stage 6 wire-up. Tests cover fresh-creates with right names/tags/ports, idempotent reuse, mixed-existence (control-plane exists, nodegroup created), name-too-long rejection, empty-inputs rejection, missing-resource no-op on delete, and first-error-surfaced sweep continuation on delete. Plan: docs/development/feature/eks-6b-create-cluster.md * feat(ec2): RunInstances honors pre-created NetworkInterfaceId Closes the AWS-parity gap where RunInstances always auto-created a primary ENI even when the caller specified NetworkInterfaces[0].NetworkInterfaceId. The EKS Sprint 6b CreateCluster launcher (Stage 4) needs to pre-create the K3s server VM's ENI so the ENI can be tagged/owned distinctly from the launcher's instance; the existing path silently created a second ENI and left the caller's unattached. Changes: - ENICreator interface gains GetENI(accountID, eniID) → *ENIInfo. ENIInfo is a translated struct (SubnetID, VpcID, PrivateIp, MAC, Status, SecurityGroupIDs) so handlers/ec2/instance stays free of an import on handlers/ec2/vpc.ENIRecord. - daemon/vpc_adapter.go gains daemonENICreator.GetENI translating VPCServiceImpl.GetENIRecord into ENIInfo. - PrepareRunInstances inspects NetworkInterfaces[0].NetworkInterfaceId after the existing subnet/SG lift block. If non-empty, the loop dispatches to a new attachPreCreatedENI helper: - Looks up the ENI under accountID. - Rejects with InvalidNetworkInterface.InUse if Status == "in-use". - Calls AttachENI to mark the ENI in-use and bind it to the new VM. - Populates instance.ENIId / ENIMac and ec2.Instance network fields (PrivateIp, SubnetId, VpcId, SecurityGroups, NetworkInterfaces[0]). - Skips the auto-create + public-IP-auto-assign paths; the caller is expected to manage public IP out-of-band. - Auto-create path is left untouched for every other code path: callers that pass SubnetId without a NetworkInterfaceId still get a fresh ENI with the existing semantics. Tests cover the four shapes: happy path (auto-create not called, attach called once, instance populated), in-use ENI rejected with the canonical AWS error code, lookup error surfaces verbatim, attach error surfaces verbatim. * feat(eks): K3s control-plane VM launcher with cloud-init user-data LaunchK3sServerVM resolves the eks-server AMI by name, pre-creates the customer-account control-plane ENI in the supplied subnet with the control-plane SG, renders cloud-init user-data (first-boot env, OIDC private key, K3s config.yaml with tls-san + service-account args), then calls RunInstances with NetworkInterfaces[0].NetworkInterfaceId pointing at the pre-created ENI. RunInstances failures roll back the ENI so the caller does not leak a customer-account resource. TerminateK3sServerVM tears the instance and ENI down for DeleteCluster; missing identifiers are a no-op so retries stay safe. ENI + VM both live in the customer account (deviation from plan §7 which specified a SystemAccount-owned VM); cross-account ENI attach is deferred to a follow-up bead. * feat(eks): NATS bootstrap subscriber + cluster reconciler + leader lease NATSBootstrap subscribes to the four single-shot subjects the K3s server VM publishes (k3s-bootstrap-token, k3s-admin-kubeconfig, k3s-oidc-jwks, k3s-ca) on eks.bus.{accountID}.{clusterName}.{kind}. Token and admin kubeconfig are envelope-encrypted via handlers_iam.EncryptSecret and persisted to KV. The published JWKS is cross-checked against the controller-generated keypair by kid and rejected on mismatch. The CA lands on meta.CertificateAuthorityB64 via the new SetClusterCertificateAuthority CAS updater. ClusterReconciler holds a leader lease on spinifex-eks-leader/{accountID}/ {clusterName} (60s TTL, 20s refresh) and drives: - CREATING -> ACTIVE once the four bootstrap KV artifacts are present and NLB /healthz responds 200. - ACTIVE -> probe /healthz on the configured interval, warn on failure. - DELETING / FAILED -> exit (DeleteCluster body owns teardown). Loss of the leader lease (CAS conflict or different holderID) returns ErrReconcilerLeaseLost so the caller can re-acquire from another node. Tested with embedded JetStream + a stub HTTPDoer; race-clean. * feat(eks): OIDC issuer URL builder + reconciler registry ClusterOIDCIssuer renders the {gatewayBaseURL}/oidc/eks/{region}/{accountID}/ {clusterName} URL kube-apiserver advertises as service-account-issuer and that STS / external IRSA validators dereference. Validates URL safety on the path segments so a hostile cluster name cannot inject scheme/path fragments. ReconcilerRegistry tracks per-(accountID, clusterName) reconciler goroutines with idempotent Spawn (drop duplicate spawn calls), Stop (cancel one + wait for release), and StopAll (graceful daemon shutdown). RunClusterReconciler is the production SpawnReconcilerFn: constructs a ClusterReconciler, acquires the leader lease, runs Run in a goroutine. Both helpers consumed by the upcoming service_impl CreateCluster body. * feat(eks): CreateCluster/DescribeCluster/ListClusters/DeleteCluster bodies Wires the EKSServiceImpl orchestration deps (VPC SG/K3s/Subnet, NLB, Instance launcher, AMI resolver, master key, gateway base URL, region, holder ID, reconciler registry) and implements the four lifecycle methods: - CreateCluster: validate → CREATING meta → SGs → NLB → OIDC issuer + keypair → K3s server VM (cloud-init carries OIDC private PEM) → register ENI IP with NLB target group → spawn NATS bootstrap subscriber + per-cluster reconciler via the registry. - DescribeCluster: KV meta read → eks.Cluster shape, with ErrClusterNotFound translated to ResourceNotFoundException at the service boundary. - ListClusters: KV key-prefix scan + sorted maxResults/nextToken pagination. - DeleteCluster: SetClusterStatus DELETING → zeroize OIDC key → stop reconciler via registry → deregister NLB target + delete NLB → terminate K3s VM → delete SGs → delete cluster KV prefix. Daemon wiring: - buildEKSServiceDeps loads MasterKey from BaseDir/config/master.key best-effort, resolves the gateway base URL via the same precedence the LB-agent block applies later, and passes VPCService + ELBv2Service + ImageService directly while adapting InstanceServiceImpl via daemonEKSInstanceLauncher (RunInstances passes through; Terminate- Instances loops vmMgr.Get + StopOrTerminateInstance in-process). - SubnetVPCResolver is satisfied by daemonEKSSubnetResolver wrapping VPCService.GetSubnet. - Startup spawns reconcilers for every persisted CREATING/ACTIVE cluster via SpawnRegisteredReconcilers; shutdown calls eksService.Shutdown so every per-cluster goroutine releases its lease before exit. Test surface: - service_impl_test: TestEKSServiceImpl_ClusterLifecycleShimMode covers the four lifecycle methods when constructed via the back-compat shim (CreateCluster/DeleteCluster → ServerInternal, DescribeCluster → ResourceNotFound on empty bucket, ListClusters → empty list). - daemon_handlers_eks_test: drop the four lifecycle subjects from the NotImplemented routing loop; handler-side coverage now lives in handlers/eks. - awserrors: add ResourceNotFound (404) for the DescribeCluster path. * fix(eks): load master.key from config dir not service data dir eks_deps read master.key from {BaseDir}/config/master.key, but the daemon systemd unit sets SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex (a per-service data dir). admin init writes master.key to the shared config dir, so the daemon looked in a path that only resolved when a {BaseDir}/config symlink existed — awsgw has one, the daemon's nested data dir does not. Result: MasterKey=nil, depsReadyForOrchestration() false, CreateCluster/DeleteCluster collapse to ServerInternal. master.key is a config artifact written next to spinifex.toml. Read it from filepath.Dir(d.configPath), matching how the daemon already resolves predastore.toml and TLS certs. No symlink or provisioning change needed. * fix(eks): register + resolve eks-server AMI via tag, add import --ami-name CreateCluster's K3s launch failed 'eks: AMI eks-server not found' on every clean install. Two defects: 1. spx admin images import had no way to name a locally-built appliance AMI: --name is a catalog lookup (rejects non-catalog values) and the AMI name was hardcoded to ami-{distro}-{version}-{arch}. build-system-image.sh ran 'import --name $AMI_NAME' for eks-server/eks-agent/GPU images, which all failed 'Image name not found in available images'. 2. The launcher resolved DescribeImages by name=='eks-server', a name the pipeline could never produce. Add --ami-name to override the registered AMI name on --file imports (fixes eks-server, eks-agent, and GPU appliance imports). Resolve the EKS AMI by the spinifex:managed-by=eks tag the pipeline already stamps, newest-wins on multiple matches. Tag-based resolution survives the planned server+agent → single EKS AMI unify untouched (role chosen per-instance at launch). build-system-image.sh now passes --ami-name; Makefile gains import-eks-server-image (IMPORT=1) and dev-install.sh imports the AMI when DEV_INSTALL_EKS=1. * fix(eks): give K3s server VM NATS token + CA so bootstrap can publish The K3s control-plane VM was launched with NATSURL=ConnectedUrl() (loopback, unreachable from the VM) and no auth material, while spinifex NATS requires a shared token over TLS. The VM could not connect to NATS, so it never published its four one-shot bootstrap messages and the cluster stayed in CREATING. Hand the VM the daemon's shared NATS token + CA PEM and a VM-reachable NATS URL (mirrors the ELBv2 LB-agent shared-credential pattern; spinifex has no nkeys hierarchy to scope per-principal creds against). - K3sServerInput: drop dead NATSCredsContent; add NATSToken + NATSCACert, required by validateK3sServerInput. user-data writes SPINIFEX_NATS_TOKEN, SPINIFEX_NATS_CA + the CA file; first-boot.env tightened to 0600. - EKSServiceDeps + CreateCluster: pass NATSURL/NATSToken/NATSCACert. - buildEKSServiceDeps: read CA from config.NATS.CACert; extract resolveGatewayHost(); add resolveNATSURL() (same reachable host, NATS port). - k3s-first-boot.sh: build nats CLI args with --token + --tlsca. * fix(eks): retry transient bootstrap persists, fail cluster on permanent error The NATS bootstrap subscriber tore down on the first persist error and the cluster sat in CREATING forever. Core-NATS delivers a publication once, so a transient KV blip during persist was unrecoverable and a malformed payload (bad JWKS kid, non-base64 CA, empty envelope) silently stranded the cluster. - persistWithRetry rides out transient persist failures in-handler (3 attempts, 200ms backoff) instead of unsubscribing and waiting for a re-publish that never comes. - errBootstrapPermanent classifies validation failures (malformed envelope, empty fields, JWKS kid mismatch, non-base64 CA) as non-retryable so they return immediately. - spawnBootstrap marks the cluster FAILED (with reason) when Run exits with a non-context error, so DescribeCluster surfaces the cause and DeleteCluster can reclaim resources. - MarkClusterFailed transitions to FAILED+StatusReason only from CREATING, so a late bootstrap error cannot clobber a concurrent delete or healthy cluster. - casUpdateMeta extracts the revision-checked RMW loop shared by the meta setters. * fix(eks): resume NATS bootstrap for CREATING clusters on daemon restart SpawnRegisteredReconcilers re-spawned only the reconciler on boot, never the bootstrap subscriber. A daemon restart while a cluster was CREATING (K3s VM still booting) lost the subscriptions; the VM published its four one-shot core-NATS messages to nobody and the cluster hung in CREATING forever. - BootstrapPendingKinds reports which artifacts (token, kubeconfig, CA) are still missing from KV. JWKS is excluded: it persists no artifact and re-waiting on a once-published subject would deadlock — its cross-check is best-effort across a restart. - RunForKinds subscribes to a subject subset (nil = all four); Run delegates to it. subjectHandlers maps each kind to its handler and rejects unknown kinds. - SpawnRegisteredReconcilers resumes bootstrap for CREATING clusters, waiting only on the missing artifacts, so a mid-CREATING restart still drives the cluster to ACTIVE. * fix(eks): DeleteCluster must not wipe KV or report success on teardown failure DeleteCluster ran zeroize, NLB teardown, and VM termination as warn-and-continue then unconditionally swept the per-cluster KV prefix and returned success. A failed NLB or VM teardown leaked billable resources with no owning record (the meta holding their ARNs/IDs was erased) and told the operator the cluster was gone. Zeroize — a stated security guarantee of no recoverable key material — had its failure ignored too. - Zeroize, NLB delete, and VM terminate are now blocking: their failures accumulate (errors.Join) and abort the call before the KV sweep, leaving the cluster DELETING and returning the error so a retry can still reach the ARNs. - KV prefix sweep runs only after NLB + VM teardown confirm. - SG cleanup stays best-effort (plan step 5) — a leaked SG strands no owning record — and target deregister stays best-effort since DeleteClusterNLB tears down the whole NLB + target group. * fix(eks): persist NLB + control-plane ARNs mid-CreateCluster so failures don't leak CreateCluster set meta.NLBArn / target-group / instance / ENI in memory and persisted them only in the final PutClusterMeta, after LaunchK3sServerVM. Any failure between EnsureClusterNLB and that persist left the NLB + target group + listener + backing LB VM created but unrecorded; markFailed did no rollback and DeleteCluster keys teardown off the persisted ARNs, so the resources leaked. - Persist meta immediately after EnsureClusterNLB (records NLB/TG ARNs) and again after LaunchK3sServerVM (records instance + ENI), before the next fallible step. - A failed create now leaves a FAILED meta carrying the resource references, so the DeleteCluster teardown path (165.9) reclaims every NLB/TG/listener/VM. - Tests: NLB-arn-persisted-before-later-failure, failed-create-then-delete reclaims NLB, and a CreateCluster happy path. Shared eksServiceFixture extracted from the delete-cluster fixture. * fix(sts): parse path-based EKS OIDC issuer so IRSA accepts 6b cluster tokens ParseEKSIssuerURL required an `oidc.eks.` host prefix and an exactly-2-segment path (AWS hosted-EKS shape), but ClusterOIDCIssuer emits — and awsgw serves — https://{gateway-host}/oidc/eks/{region}/{accountID}/{clusterName}. Every 6b cluster's iss claim was rejected with "issuer host must start with oidc.eks", so AssumeRoleWithWebIdentity would fail all IRSA tokens once 6e serves them. Reconcile on the path-based shape, which is what is actually reachable on bare-metal/on-prem (no oidc.eks.* vhost or DNS exists). Host trust comes from the IAM OIDC-provider registration (the full issuer string is the registry key) and the JWKS signature check, not a host pattern — so the parser drops the host prefix rule and instead pins accountID/clusterName at fixed positions in a strict 5-segment /oidc/eks/{region}/{accountID}/{clusterName} path. A crafted iss with extra or missing segments still fails closed. Adds a cross-component round-trip test asserting a ClusterOIDCIssuer-built iss parses back to the same (accountID, clusterName). * fix(eks): reclaim FAILED clusters on retry + cluster-scoped exists/not-found errors A failed CreateCluster left a FAILED ClusterMeta that poisoned retries: the preflight returned ErrorELBv2TargetGroupInUse ("The target group is currently in use...") for a cluster-exists condition, and because that maps to a retryable code the SDK's auto-retry surfaced ResourceInUseException instead of the original failure. FAILED clusters were also a dead state — visible to neither a clean recreate nor a clear lifecycle. - Add EKS-specific ResourceInUseException / ResourceNotFoundException codes with cluster-scoped messages; CreateCluster's exists check and DescribeCluster / DeleteCluster not-found now use them instead of the ELBv2 / EC2 codes. - CreateCluster reclaims a FAILED cluster: it purges the prior attempt's recorded resources and clears KV before recreating, so a retry (including the SDK auto-retry) is idempotent rather than blocked. Teardown failure surfaces rather than recreating on top of orphans. - Extract purgeClusterInfra, shared by DeleteCluster and the reclaim path (blocking zeroize/NLB/VM, best-effort SG, KV sweep only after teardown confirms). - FAILED clusters remain observable via Describe (status=FAILED) and List — no state where create blocks but describe/list show nothing. * fix(eks): fail CREATING clusters that exceed create timeout A per-cluster reconciler that never observes a ready control plane (bootstrap artifacts missing or healthz unreachable) previously stayed CREATING forever. Add a configurable create timeout (default 15m): once meta.CreatedAt is older than the deadline and the cluster is still not ready, the reconciler marks it FAILED with a reason, exiting the loop. Within the deadline it stays CREATING. * fix(eks): surface ACTIVE cluster health in describe-cluster The reconciler only slog.Warn'd a failing /healthz probe for an ACTIVE cluster, so describe-cluster kept reporting a dead control plane as healthy. Add HealthIssue + LastHealthProbe to ClusterMeta, write them from the ACTIVE reconcile branch (recording the failure reason, clearing on recovery), and map a non-empty HealthIssue to a ClusterHealth ClusterUnreachable issue in describe-cluster. * test(eks): add plan-mandated CreateCluster/DeleteCluster service cases service_impl_test only covered shim mode. Add the four cases the plan's Testing section requires: CONFIG_MAP auth mode rejected (and API accepted), missing subnetIds rejected, DescribeCluster 404 with full deps wired (past the shim short-circuit), and the DeleteCluster zeroize-before-teardown ordering proven by forcing a teardown failure and asserting the OIDC key is already wiped. * refactor(eks): return OIDC private-key PEM from generator, drop re-read CreateCluster discarded GenerateClusterOIDCKeypair's return then did a second KV get + AES-GCM decrypt via loadOIDCPrivateKeyPEM to re-derive the same PEM the generator already had in hand. Return the plaintext PEM (alongside the JWKS) from the generator and use it directly; remove loadOIDCPrivateKeyPEM and the now -unused crypto/x509 + encoding/pem imports in service_impl. * fix(eks): make CreateCluster/DeleteCluster failures diagnosable Unmet orchestration deps and a missing eks-server AMI both collapsed to a bare ServerInternal with no request-time log, leaving the client a 'contact AWS re:Post' message and zero cause. Enumerate the specific missing deps and log them at ERROR, returning ServiceUnavailable (an operator/config fault, not an unrecoverable internal one). Add an ErrEKSServerAMINotFound sentinel so a missing control-plane AMI is logged and surfaced as ServiceUnavailable too. * fix(eks): serialize REST-JSON responses via jsonutil so the AWS SDK can parse them WriteJSONResponse marshaled aws-sdk-go *Output structs with encoding/json, which ignores their locationName tags and emits Go PascalCase keys. EKS clients parse restjson on the lowercase locationName, so every field of list/describe/create/delete responses was silently dropped (empty endpoint, null identity.oidc.issuer). Marshal via jsonutil.BuildJSON — the same marshaler the SDK uses for restjson bodies — which honors locationName. Adds wire-shape tests asserting lowercase keys present and PascalCase absent. * feat(eks): launch K3s server VM via boot-agnostic system-instance path The EKS K3s control-plane VM launched through the plain tenant RunInstances path, so it had only its VPC ENI and no management-bridge NIC -- no route to the daemon's NATS endpoint. It never published its bootstrap subjects and the cluster stayed CREATING. Generalize LaunchSystemInstance to a boot-agnostic contract supporting both system-instance boot styles: - BootDirect: direct-boot microVM (fw_cfg blobs) -- ELBv2 LB agent - BootAMI: AMI boot + cloud-init seed (q35) -- EKS K3s server VM The launch contract moves to a neutral handlers/sysinstance package so EKS can launch system VMs without an eks->elbv2 dependency; handlers/elbv2 re-exports the types as aliases. The BootAMI branch mirrors the daemon's Prepare -> Insert -> Launch split and allocates the mgmt NIC on the VM record between Insert and Launch, so the cloud-init network-config renders a static mgmt0 interface. The EKS NATSURL is now sourced from the mgmt-bridge IP (the on-link address the VM's mgmt NIC can reach), falling back to the gateway-host resolution when no mgmt bridge exists. Verified: K3s VM now boots from the eks-server AMI under q35 with a mgmt0 NIC (10.15.8.x), reachable from the host over br-mgmt, with NATS on-link at the bridge IP. No AMI rebuild required. * fix(eks): make DeleteCluster K3s teardown idempotent so it can't wedge TerminateK3sServerVM treated every DeleteNetworkInterface error -- and every TerminateSystemInstance error -- as blocking. Because instance terminate is async, the immediate ENI delete races it: the first attempt fails InvalidNetworkInterface.InUse, and once the instance-terminate cascade releases the ENI every retry fails InvalidNetworkInterfaceID.NotFound. Both joined into purgeClusterInfra's teardownErrs gate, which short-circuits before the SG sweep and KV prefix delete -- so once the ENI was gone the cluster was stuck DELETING permanently, leaking its control-plane + nodegroup SGs and KV meta. Classify the teardown errors instead of blanket-propagating: - ENI InvalidNetworkInterfaceID.NotFound / InvalidNetworkInterface.NotFound -> idempotent success (the cascade or a prior retry already deleted it); must not block the downstream SG + KV sweep. - ENI InvalidNetworkInterface.InUse (VM still terminating) and any other error -> retryable; surfaced so the cluster stays DELETING and the reconciler retries once the instance releases the ENI. - Instance already gone -> idempotent. TerminateSystemInstance now returns a sysinstance.ErrSystemInstanceNotFound sentinel that the EKS path tolerates via errors.Is, so a retried delete after the VM drained completes. Add awserrors.IsErrorCode(err, code) as the shared delete-error classifier (substring match through %w wrapping) and route the lone ad-hoc strings.Contains guard in daemon/vm_adapters.go through it. Verified: delete-cluster on three clusters left wedged DELETING / FAILED (ENI + VM already gone) all complete -- clusters erased, SGs + ENIs reclaimed, no leak. * fix(eks): gate ACTIVE on verified JWKS cross-check, not pre-seeded key The controller pre-seeds OIDCJWKSKey at create time, so its presence proved nothing about the running cluster's actual signing key. A cluster could reach ACTIVE — and start signing IRSA tokens — before the K3s VM's published JWKS was ever cross-checked against the generated keypair. Add an OIDCJWKSVerifiedKey marker written by persistJWKS only after assertJWKSMatch passes. The reconciler's ACTIVE gate and BootstrapPendingKinds now key off the marker, not the pre-seeded JWKS. assertJWKSMatch is map-based and also rejects kty mismatch. * fix(eks): collapse duplicated gateway-host switch into resolveGatewayHost startCluster reimplemented the 5-case gateway-host precedence inline, duplicating resolveGatewayHost (used by the OIDC issuer and EKS NATS URL). If one drifted, the OIDC issuer host embedded in cluster tokens would diverge from the host LB VMs use to reach the daemon -> IRSA iss mismatch and cluster-wide token rejection. startCluster now calls resolveGatewayHost and derives the multi-node mgmt host route (mgmtRouteVia) from the resolved host. resolveGatewayHost carries the canonical case enumeration; a table test pins all five branches plus the case-1 skip guards and the no-host fallthrough. * fix(eks): make eks-server VM boot diagnosable + fix first-boot publisher crash Three issues found while diagnosing why K3s clusters never leave CREATING: 1. setup.sh: the stock Alpine cloud image lists `console=tty0` LAST in default_kernel_opts, so Linux binds /dev/console to the framebuffer and the host-side serial capture sees only kernel dmesg — OpenRC, cloud-init, and k3s-first-boot output were invisible. Reorder so console=ttyS0 is last. 2. k3s-first-boot.initd: command_background was "no" while output_log/error_log were set; start-stop-daemon rejects --stdout/--stderr without --background ("Failed to start k3s-first-boot"), so the bootstrap publisher never ran and the cluster wedged in CREATING. Background it (the publisher waits up to 5min on K3s /healthz and must not block the runlevel anyway) and route its output to the serial console, which the orchestrator captures (the VM has no login). 3. k3s-first-boot.sh: on the /healthz timeout, dump the tail of /var/log/k3s.log to the console so the apiserver's own failure is visible host-side. * fix(eks): seed K3s apiserver a public OIDC key, not the private key buildK3sUserData pointed both kube-apiserver service-account args at the same private-key file. --service-account-key-file requires a PUBLIC key — k8s ParsePublicKeysPEM rejects a private-key PEM with "data does not contain any valid RSA or ECDSA public keys" and the apiserver crash-loops, so K3s never reaches /healthz, first-boot never publishes, and the cluster wedges in CREATING. Add PublicKeyPEMFromPrivate (PKIX public PEM from the PKCS#8 private key) and seed it to a separate /etc/rancher/k3s/oidc-signing-key.pub.pem. Point service-account-key-file at the public file; keep service-account-signing-key-file on the private file. Runtime-verified: the apiserver crash is gone and K3s now starts and schedules system pods. * feat(eks): egress-only SNAT + DNS for K3s control-plane VM The K3s server VM (a hidden system instance) could not pull system-pod images: it had no internet egress and no working resolver, so coredns/traefik/metrics-server never started and /healthz timed out. Egress: give the single control-plane VM egress-only internet, confined to its /32, with no inbound exposure and no customer-visible resource. - policy/route.go: SystemInstanceEgressPriority (1200, above the drop gate) + AddSystemInstanceEgress/Delete — a reroute scoped to ip4.src == <vmIP>/32 so peers in a shared subnet are unaffected. - policy/nat.go: AddSystemInstanceSNAT/Delete — a plain snat (not dnat_and_snat, so no inbound path), idempotent on its pool external IP. - external/igw.go: L5 EnsureSystemInstanceEgress/Remove resolves the gateway nexthop and attaches the reroute + snat (mirrors the NAT-GW egress path; ADR-0006 S1/S4/S5 preserved). - subscribers: vpc.add/delete-system-egress topics + handlers. - handlers/eks: allocate a hidden (ManagedBy=eks) pool address on CreateCluster and publish add; release + publish delete on teardown. EIP dep wired through EKSServiceDeps + the daemon. DNS: the eks-server AMI is Alpine/dhcpcd and /etc/resolv.conf is a dangling symlink, so DHCP-supplied resolvers are never persisted. Write a real /etc/resolv.conf from the K3s VM cloud-config via bootcmd (not write_files — a write_files entry follows the dead symlink, fails, and aborts the whole block, dropping first-boot.env + the k3s config). * feat: EKS 6b single-node SQLite pivot + bootstrap-envelope fixes Switch the K3s server VM from embedded etcd to embedded SQLite (kine) for the single-node v1, so CreateCluster reaches a healthy control plane. etcd's per-commit fsync stalls past the apiserver deadline on viperblock's deferred-durability block path; SQLite tolerates it. - k3s server: omit cluster-init (selects SQLite/kine), add anonymous-auth=true so the reconciler's anonymous /healthz NLB probe returns 200. - first-boot readiness gate: anonymous /healthz (401 under CIS anonymous-auth =false) -> authenticated 'kubectl get --raw /readyz'. - first-boot publish: emit a JSON BootstrapEnvelope for all four subjects (token, admin-kubeconfig, oidc-jwks, ca); jwks and ca were absent before. - nats pub: --stdin -> --force-stdin (nats CLI 0.4.0). - manifest: add jq (builds the bootstrap-message JSON envelopes). * feat: EKS reconciler probes control-plane mgmt IP until DNS lands The host-side cluster reconciler cannot resolve or route to the VPC-internal NLB DNS endpoint (no authoritative DNS / Eclipso yet), so CreateCluster timed out in CREATING and was marked FAILED even though the control plane was healthy. Probe the control-plane VM's br-mgmt NIC address directly — the one apiserver endpoint reachable from the daemon. Capture the mgmt IP from the launch result (SystemInstanceOutput.MgmtIP -> K3sServerOutput.MgmtIP -> ClusterMeta. ControlPlaneMgmtIP) and build the reconciler health URL as https://<mgmtIP>:6443/healthz. The probe client already sets InsecureSkipVerify, so the mgmt IP need not be in the apiserver cert SAN. Workaround until authoritative DNS + a host-reachable NLB front-end land, at which point spawnReconciler reverts to the NLB endpoint.	2026-06-03 22:29:31 +10:00
Julian Sommer	9508aaacce	chore: bump go 1.26.4	2026-06-03 08:59:25 +10:00
Josh Siviour	8883984f3f	feat(eks): eks-server + eks-agent AMI manifests (#273) ... * feat(eks): eks-server + eks-agent AMI manifests + builder INSTALL_FILES Manifest-driven AMI build for the two EKS image variants (eks-v1.md Q12). Lifted forward from cs-eks-6b so the K3s boot path can be smoke-tested in isolation before CreateCluster orchestration lands. Builder: - build-system-image.sh gains optional INSTALL_FILES (src:dst pairs, mode 0644, parent dirs auto-created). Symmetric with INSTALL_BINARIES. - ENABLE_SERVICES moves after INSTALL_FILES + SETUP_SCRIPT so init scripts the manifest references are on disk before rc-update enables them. eks-server image (Alpine 3.21.7, q35, 4G): - K3s v1.32.5+k3s1 (downloaded at bake time, SHA256 verified against the upstream sha256sum-amd64.txt trust anchor). - nats-cli v0.4.0 (not in Alpine main/community — pulled from upstream release zip; unzip installed + purged in same setup step). - OpenRC init scripts: k3s, eks-token-webhook, k3s-first-boot. - Cron periodics under /etc/periodic/{15min,daily}/ for state-report + etcd-snapshot. - k3s-first-boot publishes node-token + admin kubeconfig over NATS on first successful K3s healthz, then self-disables (sentinel + rc-update del). - config.yaml.skel without webhook-token-auth — wiring it before the real webhook lands (cs-eks-6c) would block anonymous probes against /healthz. Orchestrator drops the live config + restarts k3s. eks-agent image (Alpine 3.21.7, q35, 2G): - Same pinned K3s release for server/agent lock-step. - k3s-agent OpenRC service refuses to start without K3S_URL+K3S_TOKEN — fail-closed pre-orchestration. eks-token-webhook stub (cmd/eks-token-webhook/main.go): - 503 + Retry-After:30 on every path. Statically linked. Lets the AMI build succeed and kube-apiserver boot cleanly. Replaced wholesale by cs-eks-6c. Init system divergence from spec wording: Alpine ships OpenRC, not systemd. Unit files use .initd extensions and cron periodics instead of .service / .timer. Bead-level intent (auto-enable on boot, periodic execution) preserved. Smoke-tested: - eks-server: builds 300M qcow2; qemu q35 boot reaches K3s apiserver on 6443 within ~120s; HTTPS probes return valid k8s Status JSON (the 401 is K3s v1.32 default RBAC behaviour against anonymous, not a regression). - eks-agent: builds 280M qcow2; qemu boot mounts root cleanly. * feat(eks): make targets for eks-server + eks-agent AMI builds - build-system-image gains subdir-manifest fallback: tries scripts/images/$(IMAGE).conf first, then scripts/images/$(IMAGE)/manifest.conf so the new EKS layout works alongside the existing flat lb manifest. - build-eks-server-image / build-eks-agent-image — explicit convenience targets that delegate to the generic one.	2026-06-01 16:24:49 +10:00
Julian Sommer	10728391af	chore: revert test-cover Makefile	2026-05-29 11:38:42 +10:00