285 Commits

Author	SHA1	Message	Date
Ben Duncan	225bf65a8e	feat: EBS export tunables, teardown reapers, fan-out and OVN fixes (#800) ... Merges `feat/ebs-export-tunables` (83 commits) into dev. The branch could not be rebase-merged because it carries five merges from dev, so it lands as a squash. ## What is in it - **EBS export tunables** and the nbdkit/viperblock export path they configure. - **Teardown reapers and backoff** for S3 buckets, load balancers, gateways, spot requests, certificates, endpoints, elastic IPs and network interfaces, so a dependent that keeps failing is backed off rather than retried hot. - **Retry and tolerance paths**: ECS IAM service init, ECR signing keys deleted between the list and the read, an unreadable signing key at awsgw startup, and an ECS instance-profile policy attach converging on a stale read. - **Key pair validation reads the material by name** instead of listing the account, closing the `InvalidKeyPair.NotFound` race that a single-replica LIST could produce. - **Concurrency and fan-out**: concurrent metadata document fetches, bounded metadata reads for describes, and the node-discovery timeout removed from every fan-out call. - **OVN correctness against the configured NB cluster**: the flows barrier, nexthop MAC binding, and stopping a compute node from running the package's standalone database. - **nats-js-watchdog can re-arm itself.** The escalation check ran above the probe loop while the only line that cleared the counter sat below it, so a latched watchdog could never observe a recovery. It now probes first and decides second, and says so once per latch rather than once per tick. - **CI**: 19 commits of nightly and build-pipeline work. ## Verification `make preflight` green locally. Build and Test, Integration Tests, Race Detection, and Lint and Security all green on `c2da0385`. ## Known red, not introduced here Two nightly cells fail on this branch and are pre-existing: - **cell-10** — a partial OVN IPsec enable leaves `NB_Global ipsec=true` asserted cluster-wide while two of three chassis failed their local reconcile, black-holing cross-chassis guest traffic for roughly 5.5 minutes while every health signal stays green. - **cell-19** — a total layer-3 egress blackhole below a correctly and promptly programmed NAT/EIP datapath, aggravated by a natuplink probe whose retry budget outlives the test's own wait, so it can never emit its FAIL verdict. Both are filed separately and should be weighed before a release is cut.	2026-08-17 17:42:27 -05:00
Julian Sommer	eac0f175ba	feat: require TLS on RDS connections by default, on both engines (#790)	2026-08-17 09:01:42 +10:00
Ben Duncan	42d87d7a5a	feat: EBS export tunables, plus blob-read fault isolation for ebsmetadata (#799) ... * feat(ebs): make nbdkit thread count and per-volume read cache configurable Both numbers that set a volume's export concurrency and memory were hard-coded: nbdkit ran on its own default of 16 threads per connection because no -t was ever passed, and the read cache was a 128 MiB literal in the mount path. Neither could be changed without a recompile, and neither was visible in the logs or the process argv. Add default_threads and cache_size_mb to [nodes.<node>.ebs]. Defaults are unchanged, so a config that does not set them behaves exactly as before. CacheSizeMB is a pointer so an explicit 0 (cache disabled) stays distinguishable from an unset key, matching how ViperblockConfig treats its own toggles. -efi volumes stay uncached regardless. Validation is a range check only. Both are operator tunables applied per nbdkit process, and there is one of those per volume, so the host cost is the value times the volume count -- left for the operator to weigh rather than second-guessed here. * feat(security): arm the firewall by install path and sync the clock at firstboot The host firewall armed itself from a code default, so a curl-to-bash install onto an existing Debian or Ubuntu machine applied a default-deny INPUT policy seconds after bootstrap. Anything the operator was already running that is not on the allow list stopped accepting connections, with nothing logged and no config key to set beforehand — no template writes firewall_enabled, so every install fell through to the default. Arming is now the install path's decision. setup.sh takes --firewall=on|off and records it in /etc/spinifex/firewall/mode, defaulting on under ISO_BUILD and off everywhere else: the ISO is an appliance on hardware we define, a package install is a guest on a machine that was already doing something. firewall_enabled becomes a three-state *bool so "unset" and "explicitly false" stay distinguishable. Flipping the default to false instead would have torn down a working policy on every existing node, because false is an instruction to disable rather than an absence. Resolution is config, then mode file, then on, so a node installed before any of this keeps what it has. The ISO had the opposite problem. setup.sh is the only place that enables spinifex-firewall.service and it returns before that line under ISO_BUILD, with no systemd in the chroot to talk to. Nothing else enabled it, so after a reboot the ruleset — runtime state — was gone, and ReconcileFirewall short-circuited on an unchanged peer file and never put it back. An ISO node was firewalled until its first reboot and open permanently after. firstboot now enables the unit, and the reconcile checks the table is loaded rather than treating an unchanged peer set as proof the policy is applied. Two policy changes: 443 is held for the nginx TLS edge so that rollout needs no per-node firewall change, and the SSH port is read from sshd_config and its Include drop-ins instead of being hardcoded to 22 — a host on Port 2222 was allowed exactly one port that was not the one it uses, and ct state established kept the operator's session alive until they disconnected. 53 stays open on every node, now with a comment saying so: northstar serves public authoritative DNS. Clock: firstboot steps and waits for sync before spx admin init mints credentials, since a certificate issued against a wrong RTC carries a NotBefore that outlives the correction. Bounded to 60s and non-fatal so an air-gapped install still completes, and once is enough because rtcsync writes the corrected time back. A chrony drop-in replaces Debian's makestep 1 3, which stops stepping after three updates and then slews at roughly 7 seconds a day — a node that burned those before the network came up never visibly converged. * fix(security): let a node join a cluster whose firewall is already armed Expanding a single-node ISO install into a cluster could not succeed. Two independent blockers, both from the same root: the peer set is derived from converged cluster state, but the ports needed to reach convergence are gated on that same set. The formation port is peer-scoped, and a node dialling in to join is not a peer yet, so the handshake was dropped rather than refused and the joiner retried in silence until it timed out. It cannot simply move to the public rule: outside formation the same port carries the daemon cluster manager, whose /health and /local/* routes report node topology, service inventory and running instances unauthenticated. So it is opened for the formation window only, by spx admin init, and closed again on every exit path. The reconcile then deadlocked. Holding out for the full chassis list is right on a node with no policy loaded, but on one armed for a smaller cluster the ports a joining node needs in order to register its chassis are exactly the ones the stale set blocks, so the list can never complete. Widen the peer set on the tunnel set already loaded, and report the partial state so the caller retries sooner than the steady-state interval. A converged cluster whose Southbound DB is briefly unreadable is still left untouched. * docs(install): explain the firewall around cluster formation An ISO-installed server is a cluster of one, and its firewall admits internal cluster traffic only from members it recognises. Forming a cluster out of such servers therefore needs the firewall off first: step 3 of the multi-node guide has each server dial node 1's OVN database on the raft ports, and those are peer-scoped like everything else on the cluster plane. Neither guide said so, and the failure is a silent drop that looks like a network fault. Both guides now walk the off / form / on sequence, and the multi-node guide gains a reference section covering which ports are public, how arming differs between the ISO and the binary installer, and how to verify membership landed. Re-enable the boot-time unit whenever the daemon arms the policy. Disabling the firewall to form a cluster also disables that unit, so re-arming used to look complete while leaving the node bare after its next reboot. * fix: make the ISO-to-cluster conversion path work end to end Forming a cluster from four ISO-imaged servers surfaced a run of bugs that share a starting state CI never exercises: the ISO delivers an initialized, armed, running node, while the conversion path was written against the binary installer's unconfigured one. - install-node.sh never disarmed the host firewall, so nodes 2 and 3 could not dial the first node on the OVN raft ports and formation could not proceed. It now owns the whole lifecycle: disarm, form, re-arm, then verify the loaded policy against the expected peer set. - spx admin init left the admin access key and secret empty on the preserve path. The pair cannot be read back, and the IAM store is rebuilt by formation, so the empty strings reached every joiner's bootstrap.json and crash-looped awsgw. It is reissued instead, and IAM seeding now skips an empty admin block rather than taking the gateway down on every restart. - node-reset.sh left /etc/default/ovn-central behind. Raft membership lives in that file, not in the databases the reset deletes, so ovn-central recreated the database as a cluster member dialling a peer that had itself been wiped. - node-reset.sh also destroyed the installed firewall policy, which nothing regenerates. set-peers then failed on every attempt and the node stayed open permanently and silently. The arming mode is read before the wipe and the stage regenerated after it. - spinifex-firewall-apply rejected every verb when the policy file was missing, including disable. Turning the firewall off is the recovery path, so that left a node that could be neither armed nor disarmed. - viperblock exited 0 on a fatal startup error, so Restart=on-failure never fired. During formation its JetStream bucket has no leader yet, so the EBS provider died for good and every volume request failed with no responders. predastore and qemunbd had the same bug. - spx admin images import created image directories at 0700, so the first sudo import locked group members out of a tree deliberately made group-writable. Three of these hid behind checks that cannot succeed: ovn-sbctl show against a clustered database, and the OVN northbound and southbound waits in setup-ovn.sh. All three failed to stderr and let the run continue, so a check that could never pass was indistinguishable from one that was merely slow. The USB install guide claimed the ISO leaves Spinifex unconfigured and told the operator to run spx admin init. firstboot has always done that itself, so the guide now describes what the ISO actually produces and how to start using it. * fix: let spread placement launch across every node A spread placement group reserved four distinct nodes correctly and then failed the launch with ServerInternal, having started only two or three of them. The scheduler was never at fault; two faults downstream stopped the instances following the reservation. vpcd subscribes to vpc.add-nat with QueueSubscribe, which delivers one message at a time, and the handler holds its reply across the OVN flows barrier. The 45s caller deadline was sized against a single 30s barrier, which is right for one instance and wrong for the N a spread launch issues at once — the third request waits behind two full barriers and times out. That one subscription now overlaps its handlers under a bounded semaphore. Each barrier still syncs after its own write, so the guarantee is unchanged while N barriers collapse into roughly one. With the requests overlapping, every barrier still ran to its alarm. install-node.sh computed OVN_REMOTE after building the database nodes and gave it only to compute nodes, so the database nodes were configured without --ovn-remote. setup-ovn.sh writes ovn-northd's NB/SB connections from that list and skips it on the localhost default, and it already documents what follows: a northd dialling only its local member stops advancing nb_cfg once raft leadership moves, wedging ovn-nbctl --wait=hv. The list is knowable before any node is touched, so it is computed first and passed to the database nodes too. Nothing reported an OVN fault while this was happening. Every chassis was registered, ovn-controller and IPsec were up, and the raft cluster was healthy. The only signal was the barrier logging that it had overrun and was continuing without confirmation — which reads like slowness and was a permanent wedge. * feat: sweep block sizes in the disk benchmark and add an iperf3 network benchmark * fix: use iperf 2 with CSV output and report why a remote start failed * fix: stop pkill matching its own shell and verify the iperf server is listening * ci: run the predastore throughput benchmark at the end of the nightly * ci: boot the ISO in the nightly, single-node and three-node * ci: use the existing CI node password secret in the ISO cells * ci: report free space before the benchmark instead of failing inside warp * ci: keep the benchmarks and node disks off tmpfs * ci: export the work directories from RUNNER_TEMP rather than job env * ci: run the ISO and benchmark workloads from the nvme scratch area * perf: run guest NICs on vhost-net with multiqueue Guest-to-guest VPC throughput measured 1.68 Gbit/s single-stream against 21.8 Gbit/s host-to-host on the same underlay, and did not improve with more streams. QEMU's -netdev tap defaults vhost to off, so the whole datapath ran in the main loop thread in userspace: that thread sat at 59-65% CPU while all four vCPU threads idled at ~4%. Every tap netdev now carries vhost=on, and VPC ENIs additionally run multiqueue at min(vCPUs, 8) queue pairs. Multiqueue needs all three of the tap, the netdev and the device to agree, so TapNetDev and NetDevice derive the netdev options and the 2N+2 MSI-X vector count from one queue figure rather than leaving callers to keep them in step. The QMP hot-plug path takes the same options as cold boot, with a test comparing it against the rendered cold-boot strings so the two cannot drift. MMIO machines stay single-queue: they have no MSI-X. * ci: check out the sibling repos the go workspace needs in the ISO cells * ci: skip cell-21 workload cleanup when the workload never ran * ci: size the cell-21 node at 8 GiB and keep its serial log * ci: keep the cell-21 node diagnostics alongside the serial log * fix: let the daemon open /dev/vhost-net and delete multi_queue taps Enabling vhost=on broke instance launch two ways. DevicePolicy=closed gates the cgroup device list before capabilities are consulted, so QEMU's open of /dev/vhost-net returned EPERM despite CAP_DAC_OVERRIDE; add the device to the allow list and bump the unit version. Separately, `ip tuntap del` re-opens the device with TUNSETIFF and fails with EINVAL on a multi_queue tap, leaking it on every teardown; route deletion through `ip link del`, which is flag-agnostic. * perf: default guest NICs to a single queue Multiqueue measured a net loss. Spreading one flow across queue pairs reorders packets and TCP reads that as loss: a 4-vCPU guest fell from 1.76 to 1.50 Gbit/s, with cwnd collapsing on spurious retransmits. The host overlay funnels a node pair onto one softirq core, so the extra queues have no headroom to win back what the reordering costs. Keep the plumbing and its tests behind MultiqueueNICs so re-enabling is one line once that funnel is fixed. vhost-net stays on unconditionally; it moves the datapath off the QEMU main loop, measured at 60% of a core under load against 3% with it. * fix: raise the UDP receive buffer for predastore's QUIC transport Predastore carries blob, meta and raft over QUIC and quic-go asks for a 7MiB receive buffer. Left at the kernel default of 208KiB it silently dropped datagrams under load: the four-node cluster had shed 80k-125k each, which failed shard writes, put raft into repeated elections, and ultimately handed guests an I/O error that cost two of them their root filesystems. Adds a sysctl stage writing /etc/sysctl.d/99-spinifex-net.conf, so the setting lands in the ISO image and on every node push rather than living in a runbook. * ci: give the cell-21 workload an identity the node accepts * feat: tie the guest MTU and NIC multiqueue to ipsec_enabled ESP costs 34 bytes on top of geneve's 58, so a cluster running the overlay in plaintext can advertise 1442 instead of 1408 — worth roughly 10% of guest throughput. Derive it from network.ipsec_enabled rather than hardcoding the conservative figure. Multiqueue moves with the same flag, because its sign depends on it. Behind IPsec it costs ~12%: ESP has no L4 ports, so RSS funnels a node pair onto one core and extra queues only reorder packets, which TCP reads as loss. Without IPsec the sending host's single vhost thread saturates first and queues are worth +54%. IPsec on with multiqueue on is the worst of the four combinations, so it stops being a standalone knob and becomes vm.Deps.MultiqueueNICs. The reconciler only ever created a subnet's DHCPOptions row and never revisited it, so the create-time MTU stuck for the life of the subnet and this change would have applied to new subnets only. Worse in the other direction: turning IPsec back on would have left subnets advertising 1442 over ESP, which drops large segments with no ICMP to recover from. applySubnets now diffs and updates in place. reconcile.Config takes IPSecDisabled rather than IPSecEnabled so the zero value is the conservative MTU — a caller that forgets it is merely slower. * docs: add an install-time performance tuning guide Measured on the four-node cluster rather than inferred: the UDP receive buffer sysctl, what IPsec actually costs and why, the multiqueue sign flip, the overlay MTU budget, and the jumbo-frame check that must precede enabling it. A starter doc — several sections end in an open question rather than a recommendation. * ci: keep the cell-21 node alive for the workload step * ci: collect the cell-21 qemu log and widen its timeout * feat: derive the guest MTU from the fabric, and deepen the virtio NIC config The overlay overhead is passed on to the guest rather than hidden from it. Hiding it means running the fabric far enough above 1500 that a standard guest MTU fits in the headroom, which is what the large public clouds do — but they own the switches. On a fabric we do not control, the honest figure is the one the path can carry, and a jumbo underlay should reach the guest: 9000 gives 8942, or 8908 with IPsec. network.underlay_mtu is the lever; SubnetMTU spends it. Configured rather than probed, because a node's own NIC MTU says nothing about the switch between it and its peers. host_mtu sets VIRTIO_NET_F_MTU so a guest that never takes a DHCP lease — statically addressed, or IPv6-only — still learns the ceiling. rx/tx_queue_size raise the virtio ring from QEMU's default 256. That default is a packet-rate ceiling rather than a memory saving: a vCPU preempted for a fraction of a millisecond fills all 256 slots and the overflow is dropped. ethtool is the only way to read NIC offload, ring and channel state, so a node without it cannot be diagnosed for throughput at all. * ci: give the cell-21 workload its own guest key * fix: set the guest MTU on the tap, not just in the guest The tap was created and brought up without an MTU, so it stayed at the kernel default of 1500 while DHCP and host_mtu told the guest something else. Harmless at 1442; it silently blackholed every large frame on a jumbo underlay, with no ICMP back because the drop is at the first hop. TapSpec.MTU comes from the same guest overlay MTU the DHCP option and host_mtu already use, and is applied in the bring-up call so a size the fabric cannot carry fails the launch instead of degrading quietly. * fix: stop setting tx_queue_size, which QEMU caps at 256 for tap QEMU only honours a tx_queue_size above 256 for vhost-user and vhost-vdpa backends. On a tap it rejects the value outright rather than clamping it, so every launch died with "must be a power of 2 between 256 and 256" and the instance went straight to terminated. rx_queue_size is unaffected and stays at 1024. A test pins the absence of tx_queue_size across machine types and queue counts. * docs: lead the tuning guide with the overlay segmentation finding The guide's headline was a 5.4x that came almost entirely from turning IPsec off, and it treated MTU and the fabric as the main levers. The host reaches 92.6 Gbit/s at MTU 1500 on the same NIC, so the fabric is not the constraint and the encrypted default is not the thing to tune away from. Adds the host-versus-guest comparison up front, documents network.underlay_mtu, and makes the missing segmentation offload the first open question. * ci: stop deselected matrix cells failing in teardown * docs: quantify what disabling overlay IPsec is worth The guide described the trade without the numbers an operator needs to make it. Adds the measured 4.9x, notes that the MTU and multiqueue changes follow ipsec_enabled rather than being separate knobs, and records two transition hazards: a stale wide MTU on running guests, and RunInstances failing while the OVN flows barrier catches up with the new SAs. * ci: collect and retain the e2e benchmark and cluster logs Cells 21 and 22 both copied two paths that only ever exist on the nodes, so the benchmark JSON was never uploaded and cell-22 collected nothing at all. Both now gather what they actually produce, and keep it for 90 days. * fix: quieten spx by installing an explicit CLI logger default spx never set a process logger, so it inherited whatever a linked package left behind — until now a JSON handler at Info on stdout, interleaving library log records with command output and breaking piped use. Execute() claims the default before any command runs: text on stderr, LevelError, so commands report failures through their own stderr writes and libraries stay quiet. --verbose/-v raises to info and SPX_LOG_LEVEL sets an arbitrary level, the flag winning over the environment. Service subcommands still call initTelemetry, which replaces the handler outright. No root --debug flag: viperblock, qemunbd, nats and awsgw each define their own --debug with a service-specific meaning, and a root persistent flag would shadow inconsistently depending on the command. The guard test samples the handler at package-init time rather than inside the test, because tests in this package start services that install their own logger legitimately. * feat: private /admin/CreateAccount for self-service tenant signup Add POST /admin/<Method> to the AWS gateway: SigV4-signed, JSON in and JSON out, callable only by an IAM user in the super-admin account holding spinifex:<Method>. CreateAccount is the one method, so a signup form can provision a tenant without an operator shell. No new service, port, unit or certificate — it mounts inside the gateway's existing authenticated group and inherits its auth, throttling and tracing. Provisioning is resumable rather than transactional, which fits stores that have no transaction. Two CAS creates carry the correctness: the client token admits one in-flight attempt and makes a lost response replayable for 24 hours, and the name reservation makes duplicate account names impossible. Every step after the account exists is create-if-absent, so a retry with the same token finishes the job instead of failing. The CLI and the endpoint share one provisioning helper, which closes a gap on both at once: CreateAccount never checked names, so running `spx admin account create` twice produced two accounts that differed only by ID. The response waits for the account's default VPC over a new ec2.EnsureDefaultVpc request/reply, rather than trusting the existing fire-and-forget iam.account.created publish — credentials for an account that cannot launch anything are worse than a slower call. [signup] max_accounts bounds how many accounts may exist; absent means uncapped, which is what every cluster does today. * fix: stop the awsgw template hardcoding one deployment's console URL console_url shipped as https://console.spx3.com, so every fresh install — including an on-premise cluster that will never run signup — was configured to point new accounts at our console. It is the only deployment-specific domain in any template. It now ships empty, and the gateway warns when it provisions an account without one, since a blank sign-in link is otherwise visible only to the customer who receives it. Extract the remote client from the command body while here. The signing, request shape and error decoding are the part a Cloudflare Worker mirrors, and as a function they are testable against a TLS test server rather than only by hand against a live cluster. * refactor: drop console_url from signup and default max_accounts to 128 The gateway never read console_url — it passed the value straight through to the response, and the idempotency record froze it for 24 hours so a corrected typo still replayed stale. The client that made the request already knows where it wants to send the new tenant. max_accounts is now a pointer in the parsed config so an absent [signup] section takes the 128 default while an explicit 0 still means uncapped. * feat: run the predastore fault-injection harness as cell-24 Cell-23 asks how fast predastore is when every node is healthy; cell-24 asks whether it survives one that is not. It runs after cell-23 rather than beside it: both want a quiet single runner, and a benchmark sharing a machine with a fault injection run measures the wrong thing. The port offset keeps the two clear of each other even so, and the job clears stale nodes on entry and exit, since a run killed before its cleanup trap leaves one holding ports that fail the next run somewhere unrelated. * ci: keep the throughput benchmark's output with its results The run log only existed in the job log, so the artifact it uploaded could not be read without going back to the run that produced it. * ci: keep the fault-injection run log with its results events.txt carries the timeline and the FAIL line, but not the build output or anything Warp wrote before the harness took over, so a failed cell could only be read from the job log it was printed into. * fix: serve the cluster region to the console instead of hardcoding it The console hardcoded ap-southeast-2 as its SigV4 region in seven files, so on any other cluster the credential scope was wrong and the gateway rejected every signed request. The SPA is embedded in the binary, so a build-time value would pin one binary to one region. The UI server now serves GET /api/config with the region, no-store so a proxy fronting several clusters cannot serve one the other's region, and the SPA loads it once before render. The region comes from the serving node's config because awsgw pins signature verification to that same field. Reading the cluster-wide [aws] region instead would let a cluster whose two keys disagree pass review and still fail to authenticate. Also relabels the login card as Spinifex credentials, since the fields are not AWS ones, and hides the CA install banner unless the console is reached directly: behind a proxy with a publicly trusted certificate there is nothing for the cluster CA to fix. * fix: say once when the host firewall will not be maintained Both quiet outcomes of the reconcile — no cluster config, and the policy switched off — return nil without logging, so a node that never arms is indistinguishable in the journal from one that armed on its first pass. * test: arm the host firewall before asserting on its peer set Only the ISO install arms the policy; every other path writes off to the mode file, so the daemon disables the firewall and never writes a peer set. The test read that file as its baseline, which it could never have. It now switches the mode on for its own duration and puts it back, waits for the peer set the way install-node.sh does, and on failure reports the mode, the node count and what the daemon said rather than just the missing file. * test: move the console region test to the external cmd package It was in package cmd, which test-package-check rejects, so preflight failed on the branch. The hook follows the pattern already in export_test.go. * test: make the reboot EIP diagnostic capable of naming the drop The two ofproto/trace probes injected packets with no TTL, so OVN's router killed them at lr_in_ip_input and both traces ended early for a reason that had nothing to do with the failure under investigation. A keyword filter then stripped the table-match lines, hiding where each trace stopped. Traces now carry a TTL and are kept whole, the real datapath megaflows for the guest reply are dumped alongside them, and any drop matching the guest source is resolved from its OpenFlow cookie back to the Southbound logical flow that installed it. * fix: exempt an EIP instance from the subnet drop gate with allow, not reroute A subnet with no route of its own carries a priority-1100 drop policy, and an instance with an external IP is lifted above it by a priority-1200 policy. That policy rerouted to the pool gateway, which in distributed NAT is outside every logical router port subnet, because the gateway LRP is the link-local /30. The static default route survives that by naming an output port; a routing policy has no such field, so ovn-northd discarded all three of them and the drop gate below swallowed everything the instances sent. The effect was that every external IP answered inbound traffic and none of it came back: the SYN reached the guest, the guest replied, and the reply died in the router's policy stage. Guest-initiated egress was lost the same way. The policy only ever needed to lift the gate. The router's default route already points at the gateway, so allow leaves the routing decision intact and gives northd nothing to reject. * test: match the log the firewall quorum gate actually emits The assertion waited for "chassis encap addresses; not writing a partial peer set", which no code has ever logged. The reconcile reports a short chassis list as "OVN reports N of M chassis encap addresses" and the maintain loop carries that into its retry warning, so the wait could only ever time out. Arming the node in the previous commit got the test as far as this assertion for the first time, which is why it surfaced now. The failure also prints the journal tail and the same inputs the earlier wait reports. * fix: stop one unreadable volume document failing every volume listing A single volume document whose shards no longer reconstruct made DescribeVolumes, DescribeVolumeStatus and DescribeVolumesModifications fail for every account, because the listing returned the first fetch error rather than skipping the document. ListVolumes now skips a document it cannot read or decode and logs the skip, mirroring ListAMIs. Callers whose answer would be wrong rather than merely partial take ListVolumesStrict: the orphan scan and the snapshot in-use check both gate destructive work on a complete list. The tolerance covers fetch failures as well as decode failures. An object whose shards no longer join is as unusable as one whose bytes will not parse, and it was the fetch path that took the cluster down.	2026-08-15 14:22:49 -05:00
Josh Siviour	9102588e5a	chore: bump Go toolchain to 1.26.6 (#793)	2026-08-14 09:29:03 +10:00
Josh Siviour	d52245001e	feat: Ochre vector store (tenant pgvector, embeddings, ingestion, operator teardown) (#788) ... * feat: add /v1/embeddings gateway adapter * feat: Ochre vector store pgvector backend, KV index registry, orchestration Adds handlers_ochrevector: the VectorBackend interface + pgx/v5 impl with per-account schema/role isolation enforced via SET LOCAL ROLE + search_path inside every transaction, the KV index registry (CREATING/READY/DELETING/STALE), and the create/delete/list/reconcile orchestration tying them together with crash-safe rollback and re-entrant recovery of stuck records. * feat: singleton appliance-ensure state machine for ochre vector store Adds the encrypted control-plane credential store and re-entrant Ensure state machine for the platform pgvector appliance (D2): a fixed-key KV claim mirroring Registry.Reserve elects exactly one winner to launch the appliance via an injected ApplianceLauncher, losers wait or resume a stale crashed provision, and the master password is generated once, stored only AES-256-GCM encrypted, and decrypted solely inside Connect to build a pgxBackend. * feat: Ochre vector store async ingestion (chunker, job store, worker) Adds Stage 3: a deterministic recursive chunk splitter, a per-account ingestion job-record store mirroring the index registry, the transactional per-document replace on the backend, and the IngestService worker that lists a predastore source, chunks and embeds each object, and replaces its rows idempotently. * feat: Ochre vector store similarity query and metadata filter grammar Adds VectorBackend.Query (cosine kNN, k default/cap, iterative HNSW scan under a filter) and a Bedrock-KB-shaped metadata Filter AST that compiles to parameterized SQL over jsonb, closing off arbitrary SQL by allowlisting jsonb keys and always binding values as params. Also fixes RunJob reporting READY on a job that ingested nothing regardless of failure cause, and stamps SourceSpec.Metadata onto ingested rows so the new filters have data to match against. * feat: Ochre vector store request/response API, VectorService, and CLI Adds the daemon-facing VectorService interface (index CRUD, ingest, describe job, query), its server implementation over the existing registry/backend/embedder, a NATS-forwarding client, and the `spx admin ochre vector` operator CLI. Query resolves and embeds against each index's own pinned model; ingest stamps the same model/dimension onto every source spec server-side. accountID is always taken from the caller context, never the request payload. Daemon subscription wiring and manifest registration are a later change; this is pure request/response plumbing with no construction into daemon.go or awsgw.go. * feat: wire Ochre vector store VectorService into the daemon Constructs the platform Postgres appliance and VectorService behind a new OchreVector config section that defaults off, so existing daemons are unaffected. Adds a real RDS-backed ApplianceLauncher that provisions the appliance over rds.CreateDBInstance/DescribeDBInstances, registers the six ochre.vector.* NATS subjects mirroring the bedrock endpoint service pattern, and degrades to a disabled (nil) service on any boot-time appliance failure rather than blocking daemon startup. * fix: fix Ochre vector store boot ordering and self-register subjects startOchreVector ran before subscribeAll, so its appliance launch fired rds.CreateDBInstance/DescribeDBInstances requests with no responder yet registered, timing out the launch on a first-enable boot. It also ran synchronously, blocking daemon startup up to the appliance launch timeout. Move it to a background goroutine started after subscribeAll returns, and have it register its own ochre.vector.* subjects once the VectorService is actually constructed, via a new registerNatsSubs helper shared with subscribeAll's boot-time table. Guard d.natsSubscriptions with d.mu, since this background registration can now race the shutdown path's unsubscribe-all sweep. * feat: bake pgvector into the shared rds-postgres image * fix: reach the ochre appliance over a routed daemon port on its ENI IP The daemon dialled the RDS appliance over rec.Endpoint, a vanity spx3.net hostname the control-plane host can neither resolve nor route to, so a healthy appliance was never reachable and the vector store disabled itself. Resolve the appliance's customer-ENI private IP read-only via ec2.DescribeNetworkInterfaces filtered on the spinifex:rds-db-instance tag, dial that IP for the DSN instead of the hostname, and give the daemon its own routed ENI in the appliance's subnet bound to an OVS host port so the IP is reachable. Retry appliance startup with bounded backoff instead of hard-disabling on the first transient. * fix: install pgvector at bootstrap for the Ochre vector appliance The daemon connects to the appliance as its NOSUPERUSER master, and the image's pgvector is not a trusted extension, so CREATE EXTENSION vector failed with permission denied. rds-init now installs it via the bootstrap superuser, gated on the appliance's fixed master name so tenant instances are unaffected. * fix: grant the Ochre appliance master CREATE on its database The daemon's EnsureAccount runs CREATE SCHEMA per account as the NOSUPERUSER master, which does not own the postgres database, so bootstrap now grants it CREATE on the database alongside the pgvector install. Both are gated to the appliance master, so tenant instances are unaffected. * fix: make the Ochre appliance's pgvector reachable under per-account isolation The daemon connects as a NOSUPERUSER master and runs every account operation under SET ROLE with a search_path pinned to the account schema. Two gaps blocked that against a real non-superuser Postgres: the master could not SET ROLE into the roles it creates, and the account role could not see the vector type. Bootstrap now grants the appliance master createrole_self_grant and installs pgvector into a read-only extensions schema granted to PUBLIC; the backend carries that schema on each account search_path so the vector type resolves without weakening per-account write isolation. All appliance-scoped. * fix: prune stale VPC host port before installing the daemon appliance port A replaced Ochre appliance is handed the same subnet IP, so a new daemon ENI mints a fresh vhp- port at an address a superseded port still holds. The old port survives at the OS/OVS layer even after its ENI record is wiped, so two interfaces claim one address and the kernel can route to the dead one (EHOSTUNREACH). installVPCHostPort now prunes any other vhp- port holding the target address before installing. * fix: gate VM restore's stale-socket kill on backing-store readiness On daemon start, a live VM whose NBD sockets fail to dial was force-killed as an orphan. While predastore/viperblock is still coming up after a restart those sockets are transiently unreachable, so a healthy VM (the Ochre appliance was the observed victim) was destroyed and its RDS record flipped to failed. The kill now only fires when the backing store is confirmed up; otherwise the live QEMU is kept and QMP reconnected. Genuine orphan reaping under a healthy store is unchanged. * fix: drive ochre ingest jobs from a daemon-side sweep scheduler * feat: add operator teardown verb for the ochre platform appliance Adds Appliance.Teardown to remove the singleton platform Postgres appliance (RDS instance, host port, KV record), an admin-only NATS subject and daemon handler to drive it, and `spx admin ochre appliance teardown --confirm` to invoke it without exposing the action through the tenant awsgw surface. * fix: register ochre appliance teardown before Connect so it survives a broken appliance * fix: purge ochre index and job metadata on appliance teardown * test: mark ochre in-package test files for the external-test gate	2026-08-13 15:54:48 +10:00
Julian Sommer	65d2501e6b	feat: add MariaDB as the second RDS engine (#762)	2026-08-13 10:38:18 +10:00
Josh Siviour	fd8be8a814	refactor(test): route duplicated NATS bootstraps through testutil (#776) ... * refactor(test): route duplicated NATS bootstraps through testutil Thirteen test sites hand-rolled a NATS or JetStream server bootstrap that spinifex/testutil already provides, and one seedKV clone duplicated testutil.SeedKV. Route them through the shared helpers. Sites whose options genuinely diverge are left alone: fixed ports needed for reconnect and restart scenarios, custom TLS and token auth, a custom MaxPayload, and TestMain entry points that have no testing.TB for t.Cleanup. The natgw and routetable seedKV clones use CreateOrUpdateKeyValue rather than CreateKeyValue, so their failure behaviour differs and they stay as they are. Also moves eight test files to external test packages where they declare no helper used by a sibling in the same package. Coverage is unaffected: Go compiles internal and external test packages into one binary per package, so attribution is identical either way. * chore(test): require new test files to use an external test package Adds a preflight check that fails when a newly added test file declares an in-package test. External test packages bind tests to the exported API and let packages share test doubles, which an in-package double cannot be. Only added files are checked, so the existing in-package tests are untouched. e2e and integration suites are exempt: both share a package-level fixture built in main_test.go, so their files cannot move individually. A file that needs unexported access can opt out with a //test:in-package comment.	2026-08-12 22:38:15 +10:00
Josh Siviour	91d22d9f12	feat(security): add an nftables host firewall scoping cluster ports to peers (#769) ... * feat(security): add an nftables host firewall scoping cluster ports to peers Nodes shipped no host firewall, so every cluster listener — OVN NB/SB, the formation port, NATS, predastore — was reachable from anything that could route to the node. On a single-NIC node the planes collapse onto the public address, which made that the default rather than the exception. setup.sh installs a policy and an apply helper but does not enable them: the peer set is unknown until the cluster's planes resolve, and a drop policy with an empty set would cut the first node that applied it out of its own formation. install-node.sh writes the peers and enables the unit after formation. The policy owns only table inet spinifex_filter and replaces it in one transaction. vpcd writes MASQUERADE and per-EIP FORWARD rules into the ip nat/filter tables and reinstalls them only on start, so a whole-ruleset flush — what Debian's nftables.service does on both start and stop — would break elastic IPs silently. INPUT only, for the same reason: a forward-hook drop here could not be rescued by an accept in the other table. Guest traffic keeps working: IMDS and VPC DNS terminate on ime-* ports in the host netns, and every guest agent reaches its control plane over the AWS gateway rather than NATS, so no cluster port needs to face the guests. * feat: reconcile the host firewall peer set from the daemon The peer set was written by install-node.sh and by cluster-bootstrap.yml. That missed two of the four formation paths (bm-bootstrap.sh and the ISO firstboot), duplicated the same logic in each, and went stale as soon as a node was added or replaced. host.ReconcileFirewall now owns it: every formation path starts the daemon, and it re-runs on each startup. A node's own config names its peers by their wan address only, so a set built from config alone drops every peer's NATS, OVSDB and formation traffic. The lan plane comes from the generated NATS cluster routes and the OVN remote lists, and the vpc plane from the Southbound Encap table; an empty encap answer is an error rather than a narrower set. ProtectSystem=full applies to the sudo child too, so the root helper could not write /etc/spinifex/firewall either — it is now in the daemon's ReadWritePaths, optional so a node installed before the firewall stage still starts. The apply helper gains set-peers and disable verbs, re-validates every address as a dotted quad with in-range octets, and rolls the peer file back if the ruleset will not load. * fix: retry the host firewall reconcile until every chassis registers The peer set is only safe to write once OVN reports an encap address for every node. On a fresh bootstrap the daemon routinely wins the race against ovn-controller, so a single startup attempt saw a partial chassis list and either wrote a set that dropped Geneve from the nodes missing from it, or failed outright and left the node with no policy until a restart. Gate on the full chassis count and schedule the reconcile instead of running it once: back off from 15s to 5m while it fails, then re-run every 5m so a node added or replaced after formation is picked up without a daemon restart. An unchanged peer set short-circuits before the helper runs. * test: cover the firewall chassis quorum gate in the multinode e2e suite A partial chassis list is the normal state early in a bootstrap, and writing a peer set from one produces a firewall that drops Geneve from every node missing from it. The gate that refuses that write had no live coverage: a 5-node bootstrap converges too fast for any daemon to observe a partial list, so the verification run only ever exercised the OVN-unreachable path. Reach the branch by inflating the expected count instead of breaking OVN: a phantom node in one node's config makes the cluster one chassis short, with nothing in the Southbound DB disturbed. Asserts the gate logs, that no peer file is written while it holds, that the loaded ruleset survives, and that the node recovers to a byte-identical peer set once the phantom is removed.	2026-08-12 22:37:27 +10:00
Josh Siviour	5c6429bde1	fix(ipsec): stop charon on nodes that do not use OVN IPsec (#768) ... The daemon only called EnableOVNIPSec when network.ipsec_enabled was true, so the disabled case was never reached and nothing ever turned the host's IPsec services off. Since setup.sh installs openvswitch-ipsec and strongswan-charon unconditionally and the packages enable their units, charon listened on 0.0.0.0:500 and 0.0.0.0:4500 on every node -- including single-node clusters, where EnableOVNIPSec short-circuits and IPsec is never switched on at all. ReconcileOVNIPSec runs on every startup and resolves the state both ways. It self-heals: a single-node box that later joins an IPsec-on cluster gets the services back through the same call, which a static install-time mask could not do because setup.sh runs before the topology is known. The unit changes go through a new fixed-verb root helper rather than systemctl. spinifex-daemon holds three narrow sudo grants and none of them is systemctl, which is deliberate: a NOPASSWD systemctl rule takes an arbitrary unit name and is root-equivalent. The helper takes only on|off and names the units itself, matching the existing spinifex-set-endpoint-sysctl pattern. Only the read-only is-enabled and is-active probes are run directly, and a test pins that. strongswan-starter.service is disabled in both states. ovs-monitor-ipsec execs the strongSwan starter itself rather than going through that unit, so it only ever competes for UDP 500/4500 -- confirmed on a live IPsec-enabled cluster, where charon's parent is /usr/lib/ipsec/starter and the unit is inactive. A nil cluster config leaves the services alone rather than guessing. This is the bind-address half only. On a multi-node cluster charon must run, and it still binds the wildcard; restricting that reach to the encap plane is the host firewall's job.	2026-08-12 12:48:08 +10:00
Julian Sommer	7dfadd4576	fix: add bluebottle to scripts	2026-08-12 10:37:28 +10:00
Josh Siviour	80faf59a43	fix(ovn): bind the NB/SB client listeners to loopback and the lan plane (#765) ... setup-ovn.sh set the OVN Northbound and Southbound client remotes with a bare set-connection ptcp:6641 / ptcp:6642. The address-free ptcp: form binds every interface, so on a multi-plane node the unauthenticated, unencrypted OVSDB control plane was reachable from the public interface. Write access to the Southbound database is control of the whole tenant network fabric. ovsdb-server takes remotes from two additive sources: the Connection table, which RAFT replicates cluster-wide, and local ovn-ctl command-line flags. That split is what the fix uses. Loopback goes in the replicated row, because 127.0.0.1 is correct on every node, so replication is harmless and local clients keep working. The lan address goes on the per-node command line via --db-nb-addr and --db-nb-create-insecure-remote, which never replicates, so each node binds its own address without pushing it to its peers. A new --lan-addr flag carries that address, falling back to --db-cluster-local-addr so clustered nodes stay correct without caller changes. The standalone branch now writes /etc/default/ovn-central too, since a two-node install still has a remote ovn-controller that needs the lan listener. The guard on --db-nb-addr is load-bearing: it defaults to 0.0.0.0, so emitting create-insecure-remote without a resolved address would recreate the bug. install-node.sh passes the lan address on both branches and warns when the lan plane has collapsed onto the public address, since the listener then lands on the public interface and a firewall is the only remaining control.	2026-08-12 10:37:01 +10:00
Julian Sommer	5363d25ef5	fix: add northstar to scripts	2026-08-12 10:25:36 +10:00
brynmailer-mdc	96dc1af568	refactor: delete the predastore cluster topology migration ... The migration rewrote a v4 topology into v5 and recovered raft state through predastore's clusterrun package, which no longer exists. The on-disk layout it relocated data into has moved too: node directories are now node-<id> under the host root, with no host-N level. Nothing migrates a pre-cutover install; it is re-initialised from scratch. node-reset.sh keeps directories by design, so it now removes the predastore cluster tree outright rather than leaving the old skeleton standing.	2026-08-11 17:23:42 +10:00
Julian Sommer	e68c965ef8	chore: bump coverage req to 70	2026-08-11 09:40:24 +10:00
Julian Sommer	53b31cb76c	fix: keep the RDS master password out of the server and console logs (#754)	2026-08-10 10:16:53 +10:00
Josh Siviour	d178ef5992	feat: version-aware host-asset preflight for binary-swap deploys (#750) ... A binary-swap deploy (make deploy or a manual spx copy) ships a new binary whose host-config prerequisites — the privileged helper scripts and their sudoers grant, placed only by setup.sh / setup-ovn.sh — it does not ship. The node looks healthy and the gap surfaces later as a cryptic launch failure or a silent security regression. Add a preflight package that checks the host against what this build expects, comparing asset content, not existence, so a stale-but-present asset is caught too. The managed helpers install from quoted heredocs, so their on-disk bytes equal the heredoc body verbatim; the binary carries the canonical sha256s and compares. - spinifex/preflight: CheckHost / CheckHostAt, content hash of the two managed helpers and a NOPASSWD grant-line check for the sysctl helper. - Generated canonicalHashes plus a drift guard that re-extracts the heredocs and fails if the committed manifest no longer matches. - Three surfaces: spx admin preflight, a make deploy gate (SKIP_ASSET_PREFLIGHT=1 to bypass), and a non-fatal daemon boot warning.	2026-08-10 07:52:39 +10:00
Julian Sommer	3379b16586	fix: encrypt and acknowledge the RDS bootstrap password (#739)	2026-08-06 15:56:06 +10:00
Josh Siviour	272b4b22e8	feat(ochre): add the vLLM serving AMI and give the daemon a route to it (#740) ... * feat(images): add a vllm-serving AMI as a stacked mkosi profile LaunchServingVM resolves its AMI by the tags spinifex:managed-by=bedrock and spinifex:bedrock-role=vllm-serving. No image carried either, so every Ensure failed at resolveServingAMI. The only GPU image ships the NVIDIA driver and Docker but no Python, no vLLM and no serving unit, so tagging it would have moved the failure from admission to first boot. Add a vllm profile that stacks onto gpu-nvidia and owns the whole guest side of the launcher contract: the runtime, vllm-serve.service, the weights mount and the tags. vLLM is baked into a uv-managed venv rather than pulled as a container at boot — a runtime docker pull of vllm/vllm-openai is known to stall this platform's write plane, and mkosi's unprivileged build has no daemon to bake an image from anyway. Also bring images/mkosi.profiles/ under make test-images. Its scripts have no .sh suffix, so the existing name glob skipped exactly the files worth linting; sources are found by shebang instead. * fix(images): make the vllm postinst survive the build sandbox Two failures found by actually running the build. uv installs to /usr/local/bin, which mkosi's postinst PATH need not carry, so `command -v uv` reported a good install as missing. Call it by absolute path, and check it the same way. `vllm --version` initialises device config and dies with "Failed to infer device type" on a builder with no GPU, so it could never have verified anything. Assert the pinned version through package metadata instead, which also catches a version mismatch rather than merely a broken entrypoint. * fix(images): pin the serving AMI datasource and mirror cloud-init to the console A serving VM boots through LaunchSystemInstance, so it comes up dual-NIC with a fw_cfg netcfg blob and takes its user-data solely from IMDS — the same shape as the EKS control plane. That path carries a cloud-init datasource pin; the vllm profile never got one, and the base config deliberately leaves datasource_list unpinned, so the serving image was the only system VM booting without it. Reuse the EKS pin rather than copying it. The 99- prefix is load-bearing: read_conf_d() applies cloud.cfg.d in REVERSE and mergemanydict() is first-source-wins, so a 90- name loses to cloud-init's own 90_dpkg.cfg ('_' sorts above '-') and the pin is discarded in silence. The postinst asserts the MERGED datasource_list via cloud-init's own read_conf_with_confd rather than checking the file exists, because it existed at 90- too and did nothing there. Also mirror every cloud-init stage to the serial console. A serving VM is headless, and a guest that wedges before handover never surrenders /var/log/cloud-init-output.log, so the console get-console-output reads is the only channel an operator has. The images_test.go case covers the two tags resolveServingAMI filters on; it matches on those alone and never on name, so a typo hides the image entirely. * fix(images): make the dhcpcd data-NIC lease genuinely one-shot dhcp_oneshot() promised to configure the link and exit "leaving no daemon to fight cloud-init", but the dhcpcd branch passed only -q, which is quiet rather than one-shot. Every Ubuntu image takes that branch, so a dhcpcd daemon survived on the data NIC holding the control socket. cloud-init's Ec2 datasource then runs its own ephemeral dhcpcd on the same NIC. Instead of spawning, dhcpcd forwards the invocation to the resident daemon as a control command and exits 0, and --dumplease returns the resident daemon's lease, so discovery looks successful. Cleanup then waits for a pid file at /run/dhcpcd/<iface>-4.pid that no process ever wrote, spinning its full 30000 x 10ms budget: init-local stalls for exactly 300s on every system-instance boot. Adding -1 exits once the interface is configured and -p keeps the address and routes across that exit. Both are needed; -1 alone tears the link back down. The test now models the real clients — the stub stays resident unless given the flag that makes its client exit, and the dhcpcd branch is forced by dropping udhcpc from PATH rather than left to whether the host happens to have /usr/share/udhcpc/default.script. * fix(images): serve without a CUDA toolkit and stop vllm-serve crash-looping vLLM picks FlashInfer for top-k/top-p sampling, and FlashInfer JIT-compiles that kernel on first use. The serving image ships the driver and the container toolkit but no nvcc, so the sampler warmup died with "Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist" minutes into a GPU cold start, after the model had already loaded. VLLM_USE_FLASHINFER_SAMPLER=0 selects vLLM's native top-k/top-p sampler, which is the same operation with no CUDA toolchain at runtime. Everything else on the serving path is already precompiled in the wheels: the attention backend resolves to FLASH_ATTN and CUDA graph capture completes, both without nvcc. Baking FlashInfer's AOT modules is the alternative if the fused kernel is ever wanted, and it keeps the toolchain in the builder rather than the appliance. The unit had Restart=on-failure with no bound, so an unfixable start failure restarted forever and buried its own first traceback under identical copies on the serial console — the only view an operator has of a headless guest. StartLimitIntervalSec/StartLimitBurst make it report instead. The postinst asserts the installed unit carries the sampler setting, for the same reason the datasource pin is asserted there: the alternative is finding out at runtime, on a GPU, five minutes in. Also corrects the shared datasource pin's header, which still claimed it installs as 90-spinifex-ec2-imds.cfg. It is installed at 99- by the vllm profile precisely because a 90- name loses the reverse-sorted, first-source- wins cloud.cfg.d merge to cloud-init's own 90_dpkg.cfg. * fix(bedrock): raise the serving-endpoint readiness bound to 15 minutes The 5 minute bound was chosen before any cold start had been measured, and its own comment asked to be corrected once a model actually ran on wattle. It now has: Llama 3.2 1B on an RTX A1000 reached "Application startup complete" 257.6s after launch, leaving ~42s of margin on the smallest model this platform can serve, with a cold torch.compile accounting for 26.3s of it. A bigger card and a bigger model are unmeasured. The two failure modes are not symmetric. Too low tears down an endpoint that was about to come up and reports it as a failed launch; too high only makes a genuinely stuck launch take longer to give up. So the bound is set well clear of the single data point rather than close to it. The CLI's own wait moves to 20 minutes to stay above it. The daemon starts its clock when the VM is launched, up to a minute after the CLI starts its own, so leaving both at 15 would expire the client first and report a bare "timed out" in place of the daemon's abort, which carries the reason. * fix: give the Ochre daemon a routed port in the Bedrock system VPC The readiness probe and every InvokeModel call dial a serving VM's system-VPC address from inside the daemon, but that subnet lives in OVN on br-int and the host has no route to it, so a healthy VM never reached READY. Each daemon now owns an ENI in the system VPC's private subnet, bound to an OVS internal port on br-int by iface-id, exactly as ovn-controller binds a guest tap. The ENI address is installed at the subnet's prefix length, which is what supplies the connected route. BaseURL is unchanged: the address it already carried is now reachable. Describe-or-create keyed on a per-node description, since ENIs carry no tag filter, so a restart or a second endpoint reuses one address instead of leaking one per launch. The port is ensured before anything else the launch creates: a VM the daemon cannot dial holds a GPU for the whole readiness window before unwinding.	2026-08-06 15:26:05 +10:00
Julian Sommer	1c4d41bffc	fix: reach the DB endpoint from any subnet of the VPC (#738)	2026-08-06 13:56:03 +10:00
Julian Sommer	e3b5701f15	fix: authorize RDS data-volume formatting by exact volume identity (#737)	2026-08-06 11:12:01 +10:00
Julian Sommer	8c07f80032	chore: remove the half line breaks	2026-08-06 11:10:15 +10:00
Julian Sommer	5dd3af70eb	fix: rds handoff path	2026-08-05 15:52:55 +10:00
Julian Sommer	615578a2e7	refactor: simplify RDS internals (#692)	2026-08-04 14:42:45 +10:00
Julian Sommer	2e09461080	chore: cleanup code comments	2026-08-04 14:20:19 +10:00
Julian Sommer	050dbdb060	fix: disallow reserved master names during rds password apply	2026-08-04 13:49:40 +10:00
Julian Sommer	970bf18ab5	fix: guard a half bootstapped datadir against crashes	2026-08-04 13:42:16 +10:00
Julian Sommer	faa0519cf1	fix: the RDS master user is no longer a PostgreSQL superuser (#687)	2026-08-04 10:59:45 +10:00
Julian Sommer	9dfef26e48	fix: rds-init	2026-08-04 10:24:48 +10:00
Ben Duncan	b76aab8b50	feat(installer): three-plane networking, ZFS root, distributed NAT and multi-node install (#663)	2026-08-04 10:21:12 +10:00
Julian Sommer	5e17169661	feat(e2e): complete the RDS end-to-end suite (#608)	2026-08-03 15:38:07 +10:00
Josh Siviour	a2c451128d	fix: unblock the Ubuntu 26.04 installer (qemu-kvm, sudo-rs) (#665) ... setup.sh aborts twice on resolute. qemu-kvm is a virtual package with two providers there, so apt refuses to resolve it; the token was redundant anyway since $QEMU_PACKAGES already installs qemu-system-x86/arm and spinifex execs the qemu-system-* binaries directly. sudo-rs, Ubuntu's default sudo since 25.10, then rejects the sudoers file: it implements no pam_session Defaults setting, and forbids a wildcard inside a command argument. pam_session is now emitted only for classic sudo, and the sysctl grant names a fixed-verb helper instead of a wildcard. The helper is narrower than the rule it replaces: it accepts one interface, a key restricted to rp_filter or accept_local, and a value restricted to 0 or 1, where the old grant permitted any net.ipv4.conf key at any value. Granting sysctl with an unconstrained argument was rejected as root-equivalent. Adds scripts/lint-apt-packages.sh and an Apt Packages workflow so the runtime package list is resolved against each supported suite's real indices in CI.	2026-08-03 08:57:50 +10:00
Josh Siviour	9a1fb44065	fix: CreateImage live drain + Alpine .raw sparsify (#647) ... * fix(ec2): drain the attached volume before CreateImage reads a live checkpoint CreateImage against a running instance (--no-reboot) read the volume's checkpoint directly from predastore without draining in-flight writes first, unlike CreateSnapshot. A checkpoint captured before a guest- triggered write (e.g. growpart rewriting the GPT after a root volume is launched larger than the base image) could be stale, producing an AMI that boots to a UEFI shell instead of the OS. Extracts the existing snapshot-package drain logic into a shared, reusable DrainVolume so both live-snapshot call sites route through the same drain-before-read guarantee instead of being able to diverge. * test(e2e): boot-verify CreateImage AMIs instead of trusting State=available describe-images reporting State=available is not evidence an AMI boots — a well-formed but unbootable AMI (stuck at the UEFI shell) still registers cleanly. Extend the CreateImage e2e test to launch a probe instance from each produced AMI (running-path and stopped-path) and require a real SSH login, with console-dump-on-failure wired up for diagnosis. * fix(images): sparsify the Alpine guestfish .raw before conversion The Alpine build path grows the qcow2 in place via qemu-img resize + guestfish resize2fs, which can leave metadata scattered across the newly-grown region. virt-sparsify zeroes the guest's unused blocks before the raw conversion so they collapse into real holes, matching the Ubuntu virt-resize path's sparseness.	2026-08-02 11:26:04 +10:00
Josh Siviour	dad5d702e1	fix: stop lbagent heartbeats on positive LB deletion, and actually enable the JetStream watchdog (#643) ... * fix(lbagent): stop heartbeating a load balancer once it is confirmed deleted A LoadBalancerNotFound response is the gateway's positive confirmation that the store has no record for the LB, not a connectivity failure — unlike a dial error, timeout, or 5xx, which leave its existence unknown and must keep retrying. Past the startup registration grace window (or after any prior successful heartbeat), it can no longer be the record's own replication race, so the agent now stops rather than retrying forever and filling the log with heartbeat noise after teardown. * fix(nats): enable the JetStream ENOSPC-latch watchdog timer nats-server disables JetStream process-wide on ENOSPC (server/jetstream.go handleOutOfSpace -> ShutdownJetStream) and never re-enables it itself — that decision is nats-server's own internal state, not spinifex's, so a restart is the only way back. The spinifex-nats-watchdog.timer + service already probe /healthz?js-enabled-only=true and try-restart on a sustained failure, but the timer's WantedBy=timers.target never took effect: neither setup.sh nor firstboot enabled it, so the watchdog shipped inert and a full disk needed a manual restart forever. Enable --now it alongside the rest of the unit set.	2026-08-02 08:21:46 +10:00
Josh Siviour	a3fb4cccec	fix(eks): backoff and terminal state on delete re-drive, CSI volume reclaim reporting, independent e2e subtests (#642) ... * fix: bound EKS DELETING re-drive with backoff and a terminal give-up reapCluster re-drove purgeClusterInfra on every GC tick with no attempt tracking, so a permanently-failing teardown (e.g. an unretriable DependencyViolation) was retried forever. Track attempt count and last attempt time on ClusterMeta, back off exponentially between re-drives, and give up after maxDeleteReapAttempts with a single error-level log. Status stays DELETING on give-up so the cluster's infra remains tracked and billable pending operator intervention, preserving the ADR-0006 §6 billable-before-sweep invariant. * fix: create the EKS e2e node role independently in each subtest runEBSCSIVolume string-built the worker node role ARN without ever creating it, only passing because runIRSAPod's t.Cleanup discarded DeleteRole's error while the role was still attached to its instance profile, leaving it behind for EBSCSIVolume to reuse by naming convention. Extract ensureNodeRole and call it independently from both subtests, detaching the role from its instance profile before deleting it and asserting that error instead of discarding it. * fix: reclaim CSI-provisioned EBS volumes on EKS cluster delete purgeClusterInfra tore down the control plane, nodegroups, and networking but never looked at EBS volumes, so any volume the in-cluster CSI driver had provisioned for a PVC was left behind untracked and unbilled for. Add a report-only reclaim step that scans for volumes tagged with the cluster's CSI owner tag and logs them for operator follow-up. Volumes are never auto-deleted: a PV's reclaim policy (Delete vs Retain) is a Kubernetes-side property the CSI driver does not tag onto the EC2 volume, so there is no safe way to tell from tags alone whether deleting a given volume is intended. This mirrors the volume-leak reaper's existing mark-and-alarm policy for the same reason. The cluster-owner tag requires the EBS CSI controller's --k8s-tag-cluster-id flag, added here to the controller manifest; without it a CSI volume carries no cluster-scoped tag and reclaim would have to guess across every cluster sharing the account. * fix: persist the last purge error on a wedged DELETING cluster DeleteReapExhausted told an operator a cluster's teardown had given up, but not why: nothing recorded the failing purge's error. Persist it on every failed re-drive, not only once the reaper exhausts its attempts, so it is visible on a still-retrying cluster too. Status stays DELETING throughout, preserving the billing invariant that untorn-down infra remains tracked. * fix: make the EKS e2e suite confirm its CSI volume is actually deleted The EBSCSIVolume subtest deleted its PVC on cleanup but never confirmed the underlying volume was reclaimed, so a CSI driver that silently failed to delete it would leak a volume with nobody noticing. The suite provisions and owns this volume, so unlike production teardown it can safely wait on its StorageClass's Delete reclaim policy: capture the CSI volumeHandle while the PVC is bound, then have cleanup poll EC2 for that exact volume to disappear before the addon/nodegroup that run the CSI controller are torn down.	2026-08-02 08:21:25 +10:00
Josh Siviour	9c9bb9430c	test: gate image shell scripts in CI and pin the spot vCPU quota ordering (#641) ... * test: gate scripts/images/ shell suites + shellcheck in CI The eks-node/rds-postgres *_test.sh suites and shellcheck only ever ran by hand: spinifex.yml filters pushes on Go-only paths and no job invoked them, so a broken helper could land unnoticed. Add a `make test-images` target and a path-filtered image-scripts.yml workflow so both run automatically on scripts/images/** and images/** changes, kept separate from preflight so image-asset churn doesn't run on every Go contributor's commit. Also clears the shellcheck findings that would have started this gate red (SC1090 on a runtime-resolved source path, SC2154 x2 on eval-assigned variables shellcheck can't trace). * test: pin vCPU quota gate ordering in RequestSpotInstances Nothing asserted that RequestSpotInstances runs the vCPU quota gate before dispatching capacity or launching a VM, so a regression that dropped or reordered EnforceLaunch would pass silently. Add an ordering test using a real quota.Service (embedded JetStream KV) and real NATS responders standing in for the capacity and launch daemons, recording a global call sequence and asserting the gate's index precedes both. Verified by removing the gate: the first quota-KV read then comes from the post-launch charge instead, landing after capacity dispatch and launch, and the ordering assertion fails as expected.	2026-08-02 08:21:04 +10:00
Josh Siviour	9f5f877e50	fix: batched hygiene — slog key, ui pid file, race-scaled timeouts, GPU console capture (#637) ... * fix(ec2): rename ambiguous 'code' slog key to firmwarePath The EFI variable store prep log used 'code' for the OVMF firmware path, which reads as an error/status code in log search. Rename to firmwarePath to avoid confusion. * fix(spinifexui): write pid file to a service-owned BaseDir spinifexui.Start() called the legacy utils.WritePidFile, which has no MkdirAll and falls back to $HOME/spinifex. Under the spinifex-ui unit's ProtectHome=yes this directory is neither guaranteed to exist nor writable. Add Config.BaseDir, switch to utils.WritePidFileTo (which does MkdirAll) for Start/Stop/Status, wire it through the CLI as a spinifex-ui-base-dir flag/SPINIFEX_UI_BASE_DIR env var, provision /var/lib/spinifex/spinifex-ui in setup.sh, and add it to the unit's ReadWritePaths. * test(viperblockd): replace fixed sleep with a readiness poll TestEBSConfigQueueGroup_DetachedOpenFails slept 500ms before assuming launchService's ebs.config subscriber was bound, then raced a hardcoded 5s request deadline. Poll with a cheap malformed request (rejected at json.Unmarshal, before the slow S3 path) until the subscriber answers, then issue the real request with room for openVolumeVB's S3 backoff. * fix(e2e): capture GPU worker console log on test failure Wire harness.OnFailure into TestEKSGPUPodExposure to dump the GPU worker's serial console via the established EC2 GetConsoleOutput path, matching lb_test.go's captureLBConsoleOnFailure. A driver or device-plugin hang after boot only ever surfaces on the guest console.	2026-08-02 08:17:18 +10:00
Josh Siviour	6873875248	fix(eks): tolerate the CriticalAddonsOnly taint on konnectivity-agent (#634) ... On a zero-worker or GPU-only cluster no node tolerates the DaemonSet, so it has zero desired pods. The apiserver then has no tunnel to kube-system pods (including metrics-server), v1beta1.metrics.k8s.io never goes Available, and namespace GC hangs on the kubernetes finalizer for every namespace, not just ones using metrics. Add the same CriticalAddonsOnly + nvidia.com/gpu tolerations already used by every other control-plane-critical addon in this repo (aws-load-balancer-controller, ebs-csi-driver, argocd) so the agent also schedules onto the control plane.	2026-08-02 08:16:00 +10:00
Josh Siviour	6ce8b9ff18	fix: fail preflight when a gate tool is killed instead of reporting it clean (#633) ... A tool whose child dies by signal can still exit 0 through go tool, so govulncheck printed 'ok' after being OOM-killed and preflight passed without a vulnerability scan having run. Route lint, govulncheck and nilaway through scripts/run-gate.sh, which fails on a non-zero exit or on signal-death output.	2026-08-02 08:15:16 +10:00
Josh Siviour	628fbbc0c3	fix: group-scope the OVS/OVN sockets and retire the vpcd sudoers grants (#609) ... * fix: group-scope the OVS/OVN sockets and stop sudoing read-only probes The OVS DB socket and both sets of ctl sockets were chmod 0666 by setup-ovn.sh, with a systemd override re-applying it on every openvswitch-switch restart. Write access to db.sock is full control of the local datapath, so this handed the datapath to any account on the box. Both service users run with Group=spinifex, so the access is now 0660 root:spinifex, /var/run/ovn is 0750, and a shared helper re-applies it from ExecStartPost on openvswitch-switch and ovn-controller alike — the latter is new, and matters because the ovn-controller ctl socket name embeds the pid and is recreated on every start. That also fixes the health probe reporting ovn-controller as not_running on a healthy node. It shelled out to `sudo ovn-appctl`, which only spinifex-vpcd is granted; as spinifex-daemon sudo demanded a password and the swallowed error read as OVN being down. With the ctl socket group-owned the probe needs no privilege, so health.go moves to a non-escalating Runner rather than the alternative of granting the daemon a root-equivalent ovn-appctl rule to read a status. ListIMDSTaps ran `list-ports` plus one `get` per patch port on vpcd's 15s tick, so 1+N sudo invocations every tick, each one logging a pam_limits warning about the core rlimit the unit deliberately pins to 0. It now issues a single JSON query per pass, and the sudoers drop-in disables the PAM session for both service users, which also leaves sudo children inheriting the unit's rlimits. An absent iface-id on a patch port is now skipped with a warning instead of aborting the pass: it is the only record of the full ENI, so such a port cannot join the live set either way, and failing the pass stalled reconcile for every healthy port behind it. A transient query failure still aborts. * fix: group-own the OVS/OVN pidfiles too, and cover the ipsec ctl socket Group-owning the ctl sockets was not enough for `ovn-appctl -t ovn-controller`: it resolves the pid-named socket through the pidfile, and OVS opens that O_RDWR to test the fcntl liveness lock, so a group-readable pidfile still fails the probe with EACCES. The pidfiles now get the same 0660 root:spinifex, which also tightens them from the 0644 they ship with. openvswitch-ipsec joins the override list because setup-ovn.sh restarts it after the sweep runs, leaving its ctl socket as the one file still at the shipped mode. Verified on the dev node: every socket and pidfile is 0660 root:spinifex and survives a restart of all three units, an unprivileged user is refused, and /health reports ovn-controller ok where the sudo probe read not_running. * refactor: stop escalating the OVS/OVN socket clients Now that setup-ovn.sh group-owns the control sockets, ovs-vsctl, ovs-appctl, ovn-nbctl, ovn-sbctl, ovn-appctl and systemctl reach their daemons as the service user. utils.NeedsPrivilege carries that policy and both command builders consult it, so ip, iptables, dhcpcd, sysctl and arping still escalate while the socket clients no longer do. Their sudoers grants are removed: each took unrestricted args and all of them accept --log-file=PATH, so every one was a root-equivalent grant handed out to run a query. ovs-ofctl keeps its grant. It talks to a per-bridge <bridge>.mgmt socket that ovs-vswitchd creates when the bridge appears, including bridges spinifex creates at runtime, so the provisioning sweep cannot group-own it up front. The NB and SB database sockets join the sweep for ovn-nbctl and ovn-sbctl. vpcd carried its own sudoCommand copy that escalated unconditionally, bypassing the policy entirely. It survived only because the grants existed and broke the OVN flows-ready barrier the moment they were removed, so it now delegates to utils.SudoCommand. TestRG10_SudoOnlyThroughThePolicy fails the build if another package reintroduces a local copy. Verified on the dev node: an instance launches, IMDS is served, the flows-ready barrier completes, and no sudo denial appears anywhere in the journal. * refactor: drop every spinifex-vpcd sudoers grant for ambient capabilities vpcd held NOPASSWD grants for ip, iptables, sysctl and arping. The ip grant alone was root-equivalent: sudoers cannot constrain its arguments, so `sudo ip netns exec <ns> /bin/sh` was a root shell for the service account. The unit already grants CAP_NET_ADMIN and CAP_NET_RAW ambiently, and the kernel passes an ambient set to each child on exec, so those tools work as the service user with no escalation at all. Verified on a live node: all sixteen ip subcommands vpcd issues, plus iptables reads and writes, sysctl -w on the net.ipv4.neigh tree, and arping, all succeed under exactly the unit's ambient set. spinifex-vpcd now has no sudoers rules. NeedsPrivilege had to become capability-aware to do this. spx is one binary shared by both service units, and the daemon holds CAP_SYS_ADMIN and CAP_DAC_OVERRIDE but no CAP_NET_ADMIN, so a static tool list cannot express "vpcd runs ip directly, the daemon escalates it". The decision now reads CapAmb from /proc/self/status once and matches it against the capability each invocation needs. sysctl is resolved per key, since CAP_NET_ADMIN covers the net.* trees and nothing else. The dhcpcd grants go too: neither daemon invokes it. Only setup-ovn.sh, which runs as the provisioning user, and the in-guest ecs-agent. TestRG10_VPCDHoldsNoSudoersGrant pins both halves together, so a future grant cannot be re-added without also explaining why the capability stopped working. * chore: restore routeTree.gen.ts swept in by mistake * fix: probe ovn-controller with ovn-appctl, not ovs-appctl checkOVNController's first probe passed a bare -t ovn-controller to ovs-appctl, which resolves it against OVS_RUNDIR — /var/run/openvswitch. ovn-controller's socket and pidfile are in /var/run/ovn, so the probe could never succeed and logged "ovn-controller.pid: open: No such file or directory" on every check. Not a functional break: the next probe globs the full /var/run/ovn path and succeeds, so the function still returned nil. It cost an exec per check and made a healthy node look like it was failing an OVN liveness probe. ovn-appctl defaults OVS_RUNDIR to /var/run/ovn, so the common case now matches first. The glob and systemctl fallbacks stay for older layouts.	2026-07-30 17:33:24 +10:00
Josh Siviour	3e195506d3	feat(images): mkosi system-image migration — CPU eks-node/ecs-node + GPU variants (#605) ... * fix(eks): GPU node AMI declares uefi boot mode The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at firmware with zero serial console output. * fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to initialize under the closed/proprietary kernel module: dmesg shows RmInitAdapter failing with "requires use of the NVIDIA open kernel modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no separate open variant (userspace tools only), so it is unchanged. * fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons A cluster with a GPU-only nodegroup has no untainted node: the CP carries CriticalAddonsOnly (EKS parity keeps user workloads off it) and every worker carries nvidia.com/gpu. Without tolerations for both, ebs-csi-controller and aws-load-balancer-controller have nowhere to schedule and stay Pending forever. Real EKS never needs this since its control plane isn't a schedulable node, so the stock upstream manifests carry no such toleration. * fix(images): give the libguestfs appliance a working resolver The guest's /etc/resolv.conf symlinks to systemd-resolved's stub (127.0.0.53), which isn't running inside the appliance chroot, so DNS is dead until package installs need it. Point it at a real resolver before package installs, then restore the stub symlink afterward so the shipped AMI still gets its DNS from systemd-resolved at boot like a stock cloud image. * fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind A GPU's IOMMU group can include an upstream PCIe bridge when the platform lacks ACS isolation between them. vfio-pci refuses to bind non-endpoint bridge devices, so Claim/ReclaimByAddress must filter bridge-class (0x06) members out of the group before touching driver bindings, rather than failing the whole claim or leaving the bridge rebound incorrectly. * fix(ecs): GPU node AMI declares uefi boot mode The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at firmware with zero serial console output. Same defect and same fix as the eks-node-gpu entry: both images are built by the same script from the same stock Ubuntu cloudimg, so both were EFI-only and only one was corrected. * fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary kernel module. NVIDIA ships no proprietary module for Blackwell (GB202) and later, so ECS GPU container-instances could not bind those cards on any driver branch. It is the third verbatim copy of the routine and was missed when the other two were fixed. Also pins the driver branch across all three copies. The branch was resolved at build time via apt-cache search | sort -rn | head -1, which sorts on the number in the package name — but that number is a branch alias, not a driver version (nvidia-dkms-535-server installs 580.126.20), so the ordering key need not match what installs. Resolving also let identical source produce different drivers on different days. The pin lives in setup.sh rather than manifest.conf because manifest variables are not passed into the guest: the setup script is uploaded and executed by virt-customize --run, with no environment carried across. * fix(eks): tolerate CP and GPU taints in argocd addon All six argocd Deployments and its StatefulSet carried no tolerations. A cluster with a GPU-only nodegroup has no untainted node to place them on: the control plane carries CriticalAddonsOnly and every worker carries nvidia.com/gpu, so the pods stay Pending forever. Same defect the ebs-csi-controller and load-balancer-controller addons already had. Real EKS never hits it because its control plane is not a schedulable node, so the stock upstream manifests carry no such toleration and the divergence is ours to maintain. * refactor(images): extract eks-node initd logic into init-agnostic helpers The OpenRC runscripts carried their start_pre logic inline, so porting a service to systemd meant re-deriving that logic under a distro change rather than just rewriting the init wiring. Move it into /bin/sh helpers that either init system can drive, keeping eks-node on Alpine/OpenRC. k3s-prestart takes the restore-block guard, config.yaml derivation, konnectivity-agent and CoreDNS staging, and the token-webhook wait. konnectivity-server-prestart takes the CA wait and cert mint, plus the server-count derivation that used to run at parse time. Both keep every guard, bound and message as-is. Values flow back the way the GPU image already does it: the helper writes shell assignments to a /run file and the init system sources them, so command_args is built in start_pre where it used to be a top-level assignment. k3s-agent-prestart moves to eks-node and becomes the single copy. The GPU manifest claimed its helpers were reused verbatim from eks-node while hand-copying this one; it now points at the shared file, so the two cannot drift. k3s-agent.initd still sources agent.env and exports itself rather than delegating: OpenRC has no EnvironmentFile= equivalent, and the daemon inherits only what the runscript exports, so a helper subprocess sourcing it would not reach k3s. The helpers run under set -eu, which the runscripts did not. The awk over first-boot.env is guarded on the konnectivity manifest, not on the env file, so a missing env file exits awk nonzero; without || true that turned a warn-and-continue into a refusal to start k3s. Covered by a test that fails on rc=2 if the guard is dropped. * feat(images): build system images in a pinned mkosi container Move the image-build toolchain into a container so a runner needs only Docker. The toolchain cannot drift between runners, and the same build runs identically on a developer's machine. mkosi builds in a user namespace, which two of Docker's default confinements block: the seccomp profile masks CLONE_NEWUSER, and the docker-default AppArmor profile separately blocks unsharing the mount namespace. Both are relaxed; neither is a privilege grant, as capabilities stay dropped and no devices are exposed. mkosi also runs as an ordinary user rather than root, since as root it takes a path that needs real CAP_SYS_ADMIN. Verified end to end: an Ubuntu 26.04 Format=disk image builds from the Debian container and the artefact is a valid GPT disk, with the package cache on a named volume cutting a forced rebuild from ~2min to ~34s. * feat(images): publish system images from the build container Convert mkosi's raw output to a compressed qcow2, write the .sha256 sidecar the spx catalog verifier matches, and upload both to R2 using the same pinned toolchain as the build, so publishing needs nothing on the host either. The toolchain image build moves to a shared helper rather than being duplicated across the two entry points. Publishing needs no relaxed confinement, unlike the build: qemu-img and aws create no namespaces, so it keeps Docker's default seccomp and AppArmor profiles. The object path carries no version and the catalog resolves newest-wins, so an upload overwrites the live image with no way back. Copy the current object to a timestamped archive key first, making a bad publish a restore rather than an unrecoverable loss. * fix(images): build the image on the output's mount, not the cache volume mkosi assembles the image in its workspace and moves the result to the output directory. The workspace lived under the cache volume, which is a different mount from the output bind mount even when both sit on one filesystem, so the move failed EXDEV and mkosi fell back to copying — requiring the finished image to exist twice at once. That is invisible on a small image and fatal on a real one: a ~4G GPU image needed ~8G to land and died with ENOSPC after a successful build. Point the workspace at the output's own mount so the move is a rename again and the image only exists once. The package cache stays on the volume, which is the part worth persisting across runs. * feat(images): add mkosi base config and gpu-nvidia profile Establish images/ with a base mkosi.conf plus a gpu-nvidia profile, replacing the throwaway smoke-test config. The base builds a bootable UEFI disk; the profile adds the NVIDIA stack. Most of ubuntu-gpu-nvidia-setup.sh does not carry over. Its kernel-version hunt, the do_bootloader=no kernel-img.conf hack and the manual update-initramfs all exist only because that pipeline chroots into a prebuilt cloud image that already has a kernel. mkosi installs the kernel and owns the bootloader, so the version is known rather than detected, and the postinst asserts exactly one kernel instead of searching for it. The static configuration (nouveau blacklist, unattended-upgrades blacklist, cloud-init docker group) becomes files under mkosi.extra/ rather than heredocs. linux-image-generic drags in every linux-firmware split package, none of it relevant to a headless VM guest; left at the default the modules initrd embeds all of it into the UKI, which grew to ~880MB and overflowed the 512M ESP. The initrd module set is trimmed to mkosi-initrd's curated default, which is a boot fix and not only a size one. Verified in the built artefact, not just the build log: nvidia.ko.zst and the four sibling modules under /usr/lib/modules/<kver>/updates/dkms, the nouveau blacklist, and the nvidia runtime in docker's daemon.json. * fix(images): unblock guest boot on wait-online and firstboot A guest built from these images stalled twice on the way to a login prompt, both times on a unit waiting for input that never arrives in a VM. systemd-networkd-wait-online holds network-online.target until every managed link is up, and docker.service pulls that target in with Wants=, so any image carrying docker inherits the wait whether or not it wants it. A guest whose links are slow or absent ran the unit to its full 120s default with multi-user.target queued behind it. Override it to --any with a short timeout: one link up is what a single-NIC guest actually needs. systemd-firstboot separately prompts on the console for anything the image does not already carry, before multi-user.target. With no timezone baked in, boot stopped dead at "Please enter the new timezone name" and waited for a keystroke. Setting locale, keymap and timezone at build time leaves firstboot with nothing to ask. Both are baked into the base rather than delivered as first-boot drop-ins: an image built from scratch has no cloud-init to write them. Measured on wattle with an A1000 passed through: login at 42.8s, down from ~210s, and the firstboot prompt is gone. nvidia-smi in the guest reports "NVIDIA RTX A1000, 595.71.05, 8188 MiB". Also widen the images gitignore to output-*/ so a scratch build cannot leave multi-GB artefacts stageable. * feat(images): make the base image reachable — cloud-init, sshd, console An image built from scratch launched unreachable. The images this replaces start from a stock Ubuntu cloud image and inherit cloud-init, sshd, sudo, a network renderer, dbus and a serial console from it; mkosi installs only what is listed, so none of it was there and nothing said so. Guests get their SSH key and user-data from IMDS via cloud-init, so without the package a launched instance has no authorized_keys at all — no error, just no way in. It also made any cloud-init-delivered config inert, including the gpu profile's docker-group drop-in. Each addition is load-bearing on its own: cloud-init consumes IMDS: keys, user-data, hostname, network openssh-server the key needs a sshd to be installed into netplan.io cloud-init's renderer on Ubuntu; with no renderer the NIC is never configured, IMDS is never reached, and cloud-init falls back to DataSourceNone dbus systemd only Recommends it and mkosi installs no Recommends, so systemctl could not reach the system bus — and the EKS and ECS bootstraps drive guests with systemctl from user-data lsb-release cloud-init shells out to it; without it every boot reports "degraded" forever and the status stops being a usable signal The kernel also needs console=ttyS0 baked in. The launcher gives every guest a serial port and get-console-output reads it, but a kernel with no console= writes to none of it: the console stopped after the UEFI bootloader and every boot failure was invisible. ttyS0 goes last because Linux makes the final console= the controlling /dev/console. manage_etc_hosts is set because cloud-init renames the guest per launch but at its default never writes a matching hosts entry, so sudo warned on every call. datasource_list is deliberately NOT pinned: the guest identifies the platform from the SMBIOS system UUID the launcher spoofs, ds-identify narrows to Ec2 on its own, and pinning per image is the customisation this migration deletes. Verified on wattle with an A1000 passed through: ssh in as ubuntu with spinifex-key, cloud-id "aws", cloud-init status done (not degraded), no failed units, nvidia-smi reports the A1000 on 595.71.05, docker active, boot 26.7s. * feat(images): let the guest grow its root into the volume it was given mkosi sizes the disk to its content, so the image ships with a root partition that fills the disk to its last sector and ~1.4G free. Nothing in it can expand that: cloud-init's growpart module is enabled, but the growpart binary lives in cloud-guest-utils, which is not installed and is not pulled in as a Recommends by anything in the package set. systemd-repart is not an alternative — Ubuntu's systemd package carries no repart binary, unit or definitions at all. The result was a node handed a larger volume that it could only ever use the first few GiB of, with the rest allocated and unreachable. That is fatal for a worker whose job is to pull container images. gdisk is deliberately left out despite being a cloud-init Recommends: growpart resizes GPT through sfdisk, which arrives via the hard fdisk dependency, and e2fsprogs brings the resize2fs that the resizefs module runs afterwards. Verified on wattle: the gpu-nvidia image launched with a 20G root reports 20G on df -h / (was 4.4G), via "'/' resized: changed (/dev/vda2) from 4770582528 to 20936900096", with nvidia-smi still reporting the A1000 and no failed units. * feat(images): move eks-node-gpu and ecs-agent-gpu onto mkosi profiles Adds the eks-common, eks-agent and ecs profiles, so both GPU node images build from the composed base rather than a hand-authored tree, and the three verbatim copies of install_nvidia_gpu_stack() collapse into gpu-nvidia alone. Splits Docker out of gpu-nvidia into its own profile. Docker belongs to the standalone GPU AMI, whose purpose is `docker run --gpus all`; the EKS and ECS node images run k3s or containerd and reach the GPU through CDI. Fusing the two was fatal for ECS rather than merely wasteful: gpu-nvidia installed containerd.io from Docker's repository, which Conflicts/Replaces the Ubuntu containerd the ecs profile needs, so gpu-nvidia + ecs could not resolve at all. nvidia-container-toolkit stays in gpu-nvidia — every GPU image needs nvidia-ctk to generate its CDI spec. Closes three gaps the pilot left, each of which only shows up on a booted guest: /etc/modules-load.d/nvidia.conf, without which nothing loads the modules on a headless node and CDI generation finds no GPU; mulga-cdi-generate.service, which no profile shipped; and an apt-mark hold on the toolkit packages, which the unattended-upgrades blacklist does not cover for a plain `apt upgrade`. mkosi-build.sh gains --image, naming an output and expanding it to an ordered profile list. The order is load-bearing: mkosi runs each profile's postinst in the order the profiles are passed, so docker's nvidia-ctk wiring must follow gpu-nvidia. Leaving that to callers meant an argument-order mistake produced a healthy-looking image with no GPU wiring, so docker's postinst also fails loudly when gpu-nvidia is selected but nvidia-ctk is absent. Profiles reference build inputs where they already live via ExtraTrees= instead of copying them, keeping one copy of the helpers shared with the Alpine image. Go binaries are built on the host into a staging tree, since the builder container carries no Go on purpose: go.mod is the single pin for the toolchain and a second pin in the Dockerfile could disagree with it silently. * fix(images): install systemd-resolved so the guest has a DNS resolver mkosi installs only the packages named in Packages= plus their hard dependency closure, and systemd-resolved is merely a Recommends of systemd. So it was never installed: the image shipped a 0-byte /etc/resolv.conf that nothing populated, because netplan renders to systemd-networkd and networkd alone does not write resolv.conf. glibc then fell back to localhost and every lookup failed with "[::1]:53 connection refused", which reads as a network fault rather than a missing package. A node that cannot resolve cannot pull a container image. The ECS GPU node booted, registered as a container instance, then failed every task at image pull. The EKS image carries the same defect but is masked by the cloud-init bootcmd in nodegroup_userdata.go that writes resolv.conf at launch; the ECS launch path has no equivalent, so only ECS exposed it. * fix(images): name build output by image and never report a skipped build as success Two defects in the build wrapper that let the wrong bytes reach a boot test. Every composition wrote the same spinifex.raw, taken from the base config's ImageId, so an artefact carried no record of which profiles produced it and an EKS image was distinguishable from an ECS one only by which build ran last. Passing --image-id names the output after the image, and as a side effect the name lands in the guest's /usr/lib/os-release. Publishing correspondingly loses its default input: --name is the published object and --image the local artefact, and with nothing tying the two together a defaulted input meant whatever raw happened to be in the output directory could be uploaded under the name the caller asked for. mkosi also skips the build whenever the output path merely exists. It does not compare inputs, so it never notices an edited mkosi.conf or a restaged binary, prints "Use --force to rebuild" and exits 0 — a stale image indistinguishable from a fresh one. Builds are now always forced, and the run additionally asserts the expected raw was written during this invocation rather than trusting the exit status. The cache worth keeping is the package download volume, which --force does not touch. * feat: add gpu-amd mkosi profile for the ROCm AMI Ports scripts/images/ubuntu-gpu-amd-setup.sh so ubuntu-gpu-amd builds under mkosi as 'gpu-amd docker'. Without it the legacy scripts/images/ tree has to stay alive for one remaining image. Pins the ROCm repo by apt priority rather than only adding it: Ubuntu 26.04 universe ships its own vendored rocminfo, which unpinned apt preferred over AMD's, mixing release trains within one image. Uses a systemd unit for render/video group membership instead of a cloud-init runcmd, because cloud-init's config merge takes a duplicated runcmd key wholesale from one fragment and silently discards the other, which would have dropped the docker profile's own usermod. Runtime GPU verification is deferred — no AMD hardware is available. * feat(images): add eks-server, ec2-node and CPU eks-node/ecs-node mkosi images Ports the K3s server side of EKS off the legacy Alpine/OpenRC builder: nine systemd units, a templated etcd-snapshot service with daily and 15-minute timers, the addon/coredns/konnectivity assets, and the six server binaries. konnectivity-server is not an in-repo package, so it gets a pinned host-side clone-and-build rather than an entry in profile_binaries(). Splits ec2-node out of eks-common. mulga-ebs-byid mints the by-id links for any attached volume and mulga-mgmt-net configures any multi-NIC system VM, so neither belongs behind an EKS profile that a database image would have to compose to reach them. Start-at-boot moves into preset files. mkosi runs preset-all after every postinst, so a postinst enable is discarded before the image is finished, and with no policy file the fallthrough is enable — which built an image that ran a control plane and a worker at once. Installs the cloud-init datasource pin as 99- rather than 90-. cloud-init-base ships its own 90_dpkg.cfg declaring datasource_list, read_conf_d sorts the directory in reverse and the merge keeps the first source, so a 90- prefixed file lost the key silently. The postinst now asserts the merged result. * refactor(eks-node): make the role selector init-agnostic eks-node-role and k3s-first-boot drove OpenRC directly, so on the systemd image the selector enabled nothing and the node never started k3s. Both now dispatch through svc_enable/svc_start/svc_disable on a detected init, with EKS_NODE_INIT overriding detection so either branch is testable on a host that has only one. Detection prefers rc-update over systemctl: the test suite stubs rc-update on PATH, and preferring systemctl would have silently rerouted those cases to a path they were not written for. Enable and start stay two separate passes. The server path starts mulga-eks-k3s-recovery before k3s so a recovery directive applies before the apiserver comes up, and collapsing to systemctl enable --now would interleave them. The dispatch is duplicated in both scripts rather than shared: they are single-file deliverables with no library-path convention between them. Tests cover both branches across all three roles, including that server-join still gets no k3s-first-boot, and that rc-update wins when both inits are present. * fix(eks-node): enqueue role-selected units instead of blocking on them eks-node-role.service declares Before=k3s.service, and mulga-eks-k3s-recovery.service declares After=eks-node-role.service, so a blocking systemctl start deadlocks: systemd will not run the queued job while the selector is still activating, and the selector cannot finish until that job runs. Type=oneshot defaults TimeoutStartSec=infinity, so nothing breaks the tie and the node wedges mid-boot with the role written but no control plane. On a booted guest the selector sat in activating for 261s until the VM was torn down. Start with --no-block on systemd. The ordering the blocking call was implicitly providing is already declared between the units themselves, so systemd sequences them correctly on its own. OpenRC runs the script directly rather than scheduling it, so that branch keeps blocking and keeps surfacing failures through set -e. * fix: escape $ in konnectivity-server ExecStart so server-count survives systemd expands $VAR/${VAR} in Exec*= from the unit's own environment before the command runs, substituting an empty string for a name it does not know. KONN_SERVER_COUNT is not a unit variable — konnectivity-server-prestart writes it to /run/konnectivity-server-args.env, which the wrapper shell sources at runtime — so systemd rewrote the flag to `--server-count=` before /bin/sh saw it. konnectivity-server then died on flag parsing, which is why the unit logged Started and produced nothing despite --logtostderr=true and journal+console, and why the apiserver's cluster egress selector found no UDS for the life of the cluster. This is a port artefact. konnectivity-server.initd is flag-for-flag identical and shares the same prestart script, but under OpenRC the shell was the only thing reading ${KONN_SERVER_COUNT}, so the same text was correct. Escape it as $$ to defer expansion to the wrapper shell, and add lint_unit_expansions() to reject any unescaped $VAR/${VAR} in an Exec*= line of a profile's units. The linter joins line continuations first, since the offending reference sat mid-way through a wrapped multi-line ExecStart, and runs on the host before the container starts so a bad unit fails the build in seconds rather than at first boot. $(cmd) is deliberately not flagged: systemd leaves it literal. Verified on env12 with a rebuilt image: TestEKS/IRSAPod and TestEKS/EBSCSIVolume, which both previously failed on `kubectl exec ... timed out` through the egress tunnel, now pass, with zero UDS connect errors across the run. * fix: refresh the stale qcow2 mkosi-build leaves beside a fresh raw mkosi only emits the raw. A qcow2 next to it came from a previous run's hand conversion, and nothing about its name says which build it belongs to, so it survives a rebuild and silently ships the old image to a node while the raw beside it carries the new one. Observed on a rebuild that left the raw at 15:34 carrying a fix and the qcow2 at 12:45 carrying the pre-fix unit. This is the same failure the existing freshness assertion guards against, one file over, so extend that block rather than add a second check. Refresh rather than delete: the qcow2 is what gets copied to a node and imported, so removing it would push the manual conversion, and this trap, back onto the caller. Convert via a temp name and rename so an interrupted run leaves the old file rather than a half-written one that looks current. * fix: resolve the awsvpc task DHCP client instead of hardcoding udhcpc ecs-agent leases a task ENI's OVN-assigned address inside a per-task netns, which systemd-networkd does not manage, so it shells out to a DHCP client. That call was hardcoded to busybox udhcpc, which Alpine ships and Ubuntu does not. `ip netns exec` exits 1 rather than 127 when it cannot exec the binary, so the failure surfaced as an opaque "network setup failed" with the task stuck in STOPPED, and it took every awsvpc-mode workload with it: TaskAwsvpc and ServiceWithELB both failed on the mkosi image, the latter as runningCount=0 because its target group targets task ENI IPs. Resolve the client at setup time, preferring udhcpc so Alpine behaviour is unchanged and falling back to dhcpcd. Presence of the udhcpc binary is not sufficient to select it: Debian and Ubuntu's busybox ship the applet without /usr/share/udhcpc/default.script, and udhcpc then takes the lease and exits nonzero having configured nothing. Same order and same reasoning as dhcp_oneshot() in mulga-mgmt-net.sh. Both invocations stay one-shot, leaving no daemon behind in the netns. Install dhcpcd-base in base, because neither client was present in any profile. That also repairs mulga-mgmt-net.sh, whose own udhcpc/dhcpcd fallback resolved to nothing on Ubuntu and would have failed the first time a cross-host control-plane data NIC needed it — the S5 path the step-6 plan notes has never been exercised. The DHCP-client choice is injected so the tests assert a named platform's branch rather than whatever happens to be installed on the machine running them; without that the same test exercises different code on an Alpine CI box and an Ubuntu dev box. * refactor: retire the ec2-node profile, fold its helpers into the base image ec2-node carried mulga-ebs-byid and mulga-mgmt-net, and every EKS composition had to remember to list it — eks-common even guarded on its presence and aborted the build without it. ECS did not list it, so the ECS image shipped with no by-id symlinks for hot-attached volumes at all. Neither helper is answerable at build time: any image may later be attached a volume, and any image may later be launched as a control plane. Both are inert when unused (mulga-ebs-byid is udev-driven; mulga-mgmt-net exits 0 when its fw_cfg blob is absent), so the base image is where they belong. That deletes the composition rule, the guard, and the ECS gap in one move. Also drops ec2-node from the two eks image compositions in mkosi-build.sh. Verified on freshly built spinifex-ecs-node and spinifex-eks-node raws: both carry the helpers and the udev rule, mulga-ebs-byid is enabled via multi-user.target.wants and mulga-mgmt-net via cloud-init.target.wants. * fix(eks): force UseMTU=false on netplan units so DHCP renewal can't reinflate the pinned link mulga-vpc-mtu only won the boot race by ordering: netplan renders UseMTU=true (it does not inherit networkd's own false default), so a DHCP lease renewal at T1 could silently re-apply the advertised MTU to the link. usemtu_off() forces UseMTU=false via a runtime drop-in in the systemd-documented [DHCPv4] section for every rendered unit and reloads networkd, removing the dependency on boot ordering entirely. assert_pinned() then verifies the link and its routes actually match before exiting, so any regression is a loud boot-time failure instead of a silent one. Both are no-ops on the Alpine/dhcpcd image, which has no systemd-networkd. * fix(images): state UseMTU=false under both the current section and the legacy alias netplan renders the DHCP settings under [DHCP] while systemd documents [DHCPv4]. A drop-in naming only one parses cleanly and does nothing if the running systemd disagrees, which is the silent no-op the drop-in exists to prevent. * fix(ecs): install iptables on both ecs-node images, cover task-role credential fetch credendpoint.go DNATs the ECS task credential endpoint via iptables; neither the mkosi ecs profile nor the legacy Alpine manifest installed it, so the endpoint silently failed to start and any task with a task role could not fetch credentials while the instance still looked healthy. Also fixes a latent bug in the e2e task-role trust policy (Service-principal-only, which this platform's HTTPS AssumeRole path can never match) and adds a TaskRoleCredentials subtest that fetches real credentials from inside a task container, so a dead endpoint fails the suite instead of passing unnoticed. * fix(images): ship preset files for base, eks-common and ecs start-at-boot mkosi runs systemctl preset-all after every profile's postinst, discarding whatever systemctl enable a postinst performed. eks-server and eks-agent already carry preset files for this reason; base, eks-common and ecs still relied on the default-enable fallthrough, which happens to match today but is one upstream package change from reversing. Adds 10-mulga-base.preset (mulga-ebs-byid, mulga-mgmt-net), 30-mulga-eks-common.preset (mulga-eks-provider-id, mulga-vpc-mtu) and 40-mulga-ecs.preset (containerd, ecs-ca-install, ecs-agent; also disables cni-dhcp.service/socket, which containernetworking-plugins enables by default with nothing to serve under the baked host-local IPAM conflist). Removes the now-redundant systemctl enable calls from each postinst, matching the eks-server/eks-agent precedent. * fix(sts): attribute ecs-tasks.amazonaws.com to container-instance callers The HTTPS AssumeRole path passed an empty principalSource, so matchServicePrincipal denied every Service clause other than the IMDS-only EC2 one. A role carrying the trust policy AWS documents for an ECS task role was unassumable by construction. Service attribution is now derived from the authenticated SigV4 caller: an assumed-role session of the account's ecsInstanceRole — the only credential the in-guest agent holds — is attributed ecs-tasks.amazonaws.com, and only for roles in its own account. Nothing in the request body contributes, so a task holding its own task-role session cannot claim the service principal. * fix(images): remove baked SSH host keys, pin Seed= for reproducibility openssh-server's postinst bakes an identical host key pair into every build, making every guest from one build mutually MITM-able; sshd-keygen already regenerates keys on first boot (ConditionFirstBoot=yes), so the baked pair is pure non-determinism with no purpose. Also pin Seed= so systemd-repart stops assigning fresh partition UUIDs on every build. Verified on a built spinifex-ecs-node image: no ssh_host_* files ship, cloud-init still lists ssh in cloud_init_modules, and sshd-keygen.service is wired via ssh.service.wants/ (redundant regeneration path). A two-build comparison dropped content-differing files from 9 to 1 (only the UKI, whose zstd internals remain sensitive to unrelated input changes). --------- Co-authored-by: tomnewton-mulga <tom.newton@mulgadc.com>	2026-07-30 12:57:43 +10:00
Josh Siviour	a0659ae7d0	fix: gate ovn-controller on SB RAFT readiness and force recompute (#599) ... Single-node OVN RAFT bring-up gated only on the Northbound DB socket, then restarted ovn-controller without waiting for the Southbound RAFT to elect a leader, serve clients, and for northd to write the gateway logical flows. With no join step to absorb the boot window the controller attached mid-election, programmed a partial datapath (SNAT ct-commit without output/delivery) and never reconverged, so NAT-GW egress and EIP ingress intermittently black-holed on the single-node baremetal CI host. Add an SB readiness gate before ovn-controller starts (SB reachable and, when clustered, an elected leader), and after ovn-controller restart wait for its SB connection then force inc-engine/recompute to rebuild all OpenFlow from the converged Southbound. Applies to 1/2/3-node RAFT alike.	2026-07-29 16:22:30 +10:00
Julian Sommer	32d80b473a	feat: RDS v1 foundations (#592)	2026-07-29 10:38:22 +10:00
tomnewton-mulga	6389b37433	fix(gpu): Blackwell GPU support fixes for EKS/ECS image builds and VFIO (#559) ... * fix(eks): GPU node AMI declares uefi boot mode The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at firmware with zero serial console output. * fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to initialize under the closed/proprietary kernel module: dmesg shows RmInitAdapter failing with "requires use of the NVIDIA open kernel modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no separate open variant (userspace tools only), so it is unchanged. * fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons A cluster with a GPU-only nodegroup has no untainted node: the CP carries CriticalAddonsOnly (EKS parity keeps user workloads off it) and every worker carries nvidia.com/gpu. Without tolerations for both, ebs-csi-controller and aws-load-balancer-controller have nowhere to schedule and stay Pending forever. Real EKS never needs this since its control plane isn't a schedulable node, so the stock upstream manifests carry no such toleration. * fix(images): give the libguestfs appliance a working resolver The guest's /etc/resolv.conf symlinks to systemd-resolved's stub (127.0.0.53), which isn't running inside the appliance chroot, so DNS is dead until package installs need it. Point it at a real resolver before package installs, then restore the stub symlink afterward so the shipped AMI still gets its DNS from systemd-resolved at boot like a stock cloud image. * fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind A GPU's IOMMU group can include an upstream PCIe bridge when the platform lacks ACS isolation between them. vfio-pci refuses to bind non-endpoint bridge devices, so Claim/ReclaimByAddress must filter bridge-class (0x06) members out of the group before touching driver bindings, rather than failing the whole claim or leaving the bridge rebound incorrectly. * fix(ecs): GPU node AMI declares uefi boot mode The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at firmware with zero serial console output. Same defect and same fix as the eks-node-gpu entry: both images are built by the same script from the same stock Ubuntu cloudimg, so both were EFI-only and only one was corrected. * fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary kernel module. NVIDIA ships no proprietary module for Blackwell (GB202) and later, so ECS GPU container-instances could not bind those cards on any driver branch. It is the third verbatim copy of the routine and was missed when the other two were fixed. Also pins the driver branch across all three copies. The branch was resolved at build time via apt-cache search | sort -rn | head -1, which sorts on the number in the package name — but that number is a branch alias, not a driver version (nvidia-dkms-535-server installs 580.126.20), so the ordering key need not match what installs. Resolving also let identical source produce different drivers on different days. The pin lives in setup.sh rather than manifest.conf because manifest variables are not passed into the guest: the setup script is uploaded and executed by virt-customize --run, with no environment carried across. * fix(eks): tolerate CP and GPU taints in argocd addon All six argocd Deployments and its StatefulSet carried no tolerations. A cluster with a GPU-only nodegroup has no untainted node to place them on: the control plane carries CriticalAddonsOnly and every worker carries nvidia.com/gpu, so the pods stay Pending forever. Same defect the ebs-csi-controller and load-balancer-controller addons already had. Real EKS never hits it because its control plane is not a schedulable node, so the stock upstream manifests carry no such toleration and the divergence is ours to maintain. --------- Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>	2026-07-27 12:11:18 +10:00
Julian Sommer	e79ca59d4b	fix: configure host DNS from spx admin init/join (#582)	2026-07-24 10:02:11 +10:00
Julian Sommer	1d55404a73	revert: host DNS setup on resolved/resolvconf hosts (#580) ... This reverts commit f9c9f0b38d. #580 switched setup.sh to read the northstar :53 listener and zones from northstar.toml, so the resolved stage now defers at install time (before formation) and must be re-run afterward. Only firstboot re-runs it; the manual setup.sh + spx admin init install and CI never do, so those hosts get no host DNS and cluster-zone names (ELB/EKS) fail to resolve from the node. Reverting restores the pre-#580 behaviour: systemd-resolved is configured at install time from the WAN IP, with no post-formation re-run needed. This reintroduces the original gap where resolvconf hosts get no host DNS. The proper fix configures host DNS from spx admin init/join for both resolver managers (mulga-x30yv).	2026-07-23 11:23:21 +10:00
Julian Sommer	f9c9f0b38d	fix: configure host DNS for Spinifex zones on resolved and resolvconf hosts (#580)	2026-07-23 09:43:38 +10:00
Josh Siviour	4793b5ac09	fix(eks): node route MTU blackholing image pulls, and teardown wedged in DELETING (#562) ... * fix(eks-node): stop the DHCP MTU overriding the pinned link MTU Pinning eth0 to 1320 was never enough. This image runs dhcpcd, which honours DHCP option 26 and stamps the advertised MTU onto every route it installs: default via 10.32.101.1 dev eth0 proto dhcp src 10.32.101.7 \ metric 1002 mtu 1442 Linux prefers a route's MTU metric over the device MTU, so the node kept advertising an MSS derived from 1442 while the real path MTU through the NAT gateway is 1408. Small packets pass and full-MSS inbound segments — a TLS ServerHello with its certificate chain, image layers — are silently dropped. It presents as `net/http: TLS handshake timeout` on image pulls, not as anything MTU-shaped, and it only affects the host netns, so containerd pulls fail while pod traffic sized off cni0 is fine. Per-destination PMTU exceptions expiring in and out of the route cache made it look intermittent and host-specific. Three changes, innermost first: - dhcpcd.conf gets `nooption interface_mtu`, so no lease can stamp a route MTU. mulga-vpc-mtu is the single source of truth for the MTU of these nodes. - mulga-vpc-mtu re-stamps any route that already carries a different MTU, covering an image built before this fix or a dhcpcd already running. Routes with no MTU metric inherit the device MTU and are left alone. - the subnet DHCP option drops 1442 to 1408. 1442 is geneve-only and ignores the IPsec ESP overhead on the NAT egress path, so it overstates the usable MTU for every guest, not just these nodes. Also corrects the header comment, which claimed Alpine's udhcpc ignores option 26 — this image runs dhcpcd, which honours it. That wrong premise is why the route metric was never considered. Adds a stubbed-`ip` test covering the rewrite, the dev token that `ip route show dev X` omits, idempotency, and routes without an MTU metric. * fix(eks): sweep the CP VPC by identity so teardown can finish DeleteClusterCPVPC finds the managed control-plane VPC's subnets and SGs by role tag. A create that fails partway — the first control-plane server failing to launch, say — leaves resources behind that never got tagged, so the sweep cannot see them. DeleteVpc rejects while any subnet or non-default SG remains, and teardown is re-driven on a timer, so the cluster sits in DELETING forever retrying the same doomed delete every couple of minutes: eks-deleting: re-driving wedged DELETING teardown cluster=toc eks-deleting: re-drive teardown failed err="delete managed CP VPC: eks: delete cp vpc vpc-...: DependencyViolation" Nothing converges: the cluster name cannot be reused and terraform's delete wait can never pass. Observed on env19, wedged for ~11 hours until this change went in, at which point the next re-drive cleared it. After the tagged sweep, clear anything still in the VPC scoped by vpc-id: subnets, and non-default SGs (the default SG goes with the VPC cascade). Teardown had no SG surface, so CPVPCDeps gains an optional SG provisioner — nil is tolerated, since only teardown uses it. * chore: pin viperblock + predastore to the merged storage fixes viperblock f8cfd77 (#42) makes read-modify-write atomic per block, removing the lost update that silently reverted a 4096-byte block to a previous generation of itself. Confirmed as the env19 corruption cause by an A/B: the pre-fix build corrupted image layers within ~15 minutes (failed to extract layer / invalid tar header / unpigz: corrupted) while the fixed build ran 5+ hours clean, absorbing 6407 same-block RMW conflicts on the worker root disks. predastore c0a90d2 (#63) batches FSM.Restore. Without it, any predastore restart on a store whose metadata set outgrew one badger transaction fails with "Txn is too big to fit into one request" on every node at once, taking the metadata plane — and with it the AMI catalogue and cluster creation — down permanently. Both were verified in the built artifact rather than from go.mod alone: go version -m reports the new pseudo-versions, and the rmw_conflicts counter plus the batched-restore error strings are present in the binary. Pinning by go.mod while the submodule pointer says otherwise is how an earlier analysis ended up reading dead code. * test: record the corrected DHCP MTU in the kitchen-sink golden The NB golden pins the DHCP options published per subnet, including the advertised MTU. Dropping the subnet option from 1442 to 1408 changes what the reconciler writes, so the golden diverged and the live scenario test failed — the test doing exactly its job. Regenerated with UPDATE_GOLDEN=1. The diff is only mtu:1442 -> mtu:1408 on the two subnet DHCP records; nothing else in the snapshot moves.	2026-07-22 12:57:36 +10:00
Ben Duncan	28a876d17d	feat(dns): resolvable ALB/NLB + EKS DNS, forwarder hardening, cross-tenant reconcile (#532)	2026-07-17 12:02:59 +10:00
joshsiv-mulga	1281381096	fix(ci): publish-system-image.sh honors SYSTEM_IMAGE_BUILD_DIR ... The build script derives BUILD_DIR from SYSTEM_IMAGE_BUILD_DIR (the AMI publish workflow points it off tmpfs so the 16G Ubuntu GPU raw fits), but the publish step hardcoded /tmp and could not find the freshly built raw. GPU legs failed 'Raw image not found'; Alpine legs silently reused stale pre-existing /tmp raws on the persistent runner. Mirror the build script's derivation so publish looks where the build actually wrote.	2026-07-15 07:54:21 +10:00
Josh Siviour	b41a193cbc	fix(eks): make provider-id.sh the single k3s node-label authority for nodegroup + GPU labels (#539)	2026-07-15 00:39:13 +10:00