mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-11-02 17:23:32 +00:00
Ben Duncan
b4dd606104
* feat: Vultr reserved-IP provider and floating external-address pool Adds a source="vultr" external pool backed by Vultr reserved IPs: an API client pinned by captured responses, an in-memory fake with the measured semantics, and a floating allocator that creates, claims, reconciles and releases labelled addresses. OCI and Vultr claims now share one AddressClaimer pass. Also fixes the host firewall dropping guest IMDS when imds_host_meta_ip is set, by matching the pre-DNAT destination, and arms the Spinifex firewall in place of an active ufw. * fix: detach a Vultr reserved IP before deleting it Vultr was measured refusing delete of an attached address with 400 "Unable to detach subnet", so release, rollback and reconcile now detach first. A first claim after allocation logs at info rather than warn. * fix: release the auto-assigned public IP an Elastic IP displaces AssociateAddress restamped the ENI with the EIP and never handed back the address launch had assigned. Nothing referred to it afterwards, so it stayed allocated for good, and on a cloud pool every associate left a billed address. AWS releases it on associate; withdraw its NAT rule and release it, scoped to the ENI, leaving any address another associated EIP holds alone. * fix: retry Vultr's refused detach on reserved-IP detach and delete Vultr has answered a detach with 400 "Unable to detach subnet" and accepted the same detach moments later. Surfacing it left the address for reconcile to collect, up to twenty billed minutes later; retry it like a 5xx instead. * fix: stop reporting the released launch address after an EIP is disassociated Associating an EIP now releases the auto-assigned public IP, but the stored instance kept its launch stamp, and DescribeInstances fell back to it once the EIP was disassociated. The public-IP change handler now moves the stamp with the runtime address. * feat: add a Vultr deploy script for a single-node Spinifex on a VX VM deploy.sh up/destroy/status drives Terraform for the VPC and VM, installs and forms Spinifex over SSH, copies the API key to the node rather than through user-data, and sweeps boot volumes and reserved IPs Terraform does not own. Cloud-init keeps needrestart off the Spinifex, OVS and OVN units and disables unattended upgrades on these disposable hosts. * feat: form a three-node Vultr deployment with install-node.sh over the VPC Each node comes up as an armed single node, the state install-node.sh forms from, with the cluster and Geneve planes on the VPC NIC. The pool and every node's provider_host_id go on each node after formation. * fix: read the pinned host key with ssh-keygen -F when forming a Vultr cluster ssh_config hashes known_hosts entries by default on Debian, so a grep for the address found nothing and set -e ended the run without a message. * docs: add the Vultr guide, one command to deploy one or three nodes and one to remove them * docs: drop the failover write note from the Vultr guide * fix: keep needrestart from restarting the Spinifex, OVS and OVN units Guest QEMU runs in spinifex-daemon's cgroup, so needrestart charged QEMU's replaced libraries to the daemon and restarted it after every dpkg run. The restart never cleared them, so an unattended upgrade ran the unit into its start limit and left the control plane down. setup.sh now writes /etc/needrestart/conf.d/90-spinifex.conf, deselecting spinifex-, ovs-, ovsdb-, ovn- and openvswitch- units. needrestart still lists them, so restarting after an upgrade stays the operator's choice. * chore: drop the Vultr cloud-init needrestart override now setup.sh writes it * docs: replace the Vultr guide's apt upgrade warning now the installer keeps needrestart off the stack * docs: install the Vultr guide's build from the dev channel until Vultr support is released * fix: answer ARP for Vultr reserved IPs on bare metal Vultr routes a reserved IP to a VM by MAC but ARPs for it on a bare-metal server's public segment, and a Vultr pool gave vpcd neither a gateway nor a bind bridge to proxy-ARP on, so nothing answered and no traffic arrived. A Vultr pool now proxy-ARPs on the interface the node routes the internet through. setup.sh installs arping, which the gratuitous ARP already called. * feat: deploy Spinifex on Vultr bare metal, renamed deploy-vultr.sh A vbm- plan creates bare-metal servers, one or three. Without --name the deployment writes the AWS profile spinifex, up prints every node's address and the API key allowlist, credentials and ssh subcommands are added, no AMI is imported unless --image is given, and OpenTofu or Terraform is used. The host key is pinned only once cloud-init is done, since bare metal first answers SSH from Vultr's installer with a different key. * docs: rework the Vultr guide into the OCI guide's flow, with bare metal * fix: restamp LaunchTime when a stopped or recovered instance relaunches The pending watchdog times a launch from LaunchTime, which still held the original launch. An instance recovered from a failed node, or started more than five minutes after it was first launched, was terminated as stuck in pending seconds after QEMU started. AWS restamps LaunchTime on every start. * feat: spread the nginx-alb workbook across nodes, and power Vultr nodes off and on from deploy-vultr.sh The workbook takes server_count and placement_strategy, so three web servers can run one per node. Its README now matches the template (public subnets, no NAT gateway) and the diagram shows the spread. deploy-vultr.sh gains stop and start, which halt and start a node through the Vultr API. * fix: let a terminate finish tearing down before the reaper calls it wedged OrphanQEMUReaper finalized any shutting-down instance whose QEMU had exited, on the premise that a live terminate still has a live QEMU. A terminate is still deleting volumes, releasing its IP and leaving its placement group for tens of seconds after QEMU exits, so the instance was reported terminated first. Terraform then deleted the placement group and got InvalidPlacementGroup.InUse. The reconcile now waits two minutes from shutting-down, well inside the ten-minute stuck-terminate backstop. * fix: release an ovs-vswitchd left in flow-restore-wait after a node boots ovn-controller clears flow-restore-wait once its flows are installed and sets it again straight after. When both writes reach ovs-vswitchd as one update, it only sees true, starts no upcall threads and forwards nothing. Seen on Vultr bare metal after a hard power-off: instances recovered onto the rebooted node ran but were unreachable. vpcd now clears the flag on any node held that way for a minute, longer than ovs-ctl's own restore takes. * docs: add a resiliency test to the Vultr guide and the nginx-alb workbook The Vultr guide gains step 8, which spreads three web servers across a three-node deployment, powers a node off and back on, and shows what was measured on VMs and bare metal. The workbook gains the same test as step 6. The guide's table of contents is repaired, and its limits now cover losing two nodes and recovery inside a spread placement group. * fix: never terminate an instance that has run because a relaunch failed The pending watchdog terminated any instance pending past five minutes, and terminating deletes DeleteOnTermination volumes. A start of a stopped instance, a recovery from a failed node and a daemon relaunch all pass through pending, so a slow one destroyed an existing instance and its root disk. On Vultr a recovered instance was terminated seconds after QEMU started and its root volume deleted. An instance is now marked FirstLaunch from RunInstances until it first reaches running, cleared before the running state is written. The watchdog acts only on first launches, as AWS terminates only a launch that never ran. MarkFailed refuses to terminate any other instance and marks it recovery-failed instead, which keeps its volumes, ENI and addresses and shows it as stopped. A claimed relaunch is never a first launch. Invariant tests pin ADR-0003:S7. The instancerecovery e2e suite gains a test that ages its guest past the watchdog before recovering it, then stops and starts it, and requires the instance, its root volume and the data on it to survive both. * feat: run the Terraform workbooks against a Vultr deployment on demand, and show the Vultr portal in the guide e2e-cloudvendor-vultr.yml is dispatch-only. It builds the ref, deploys it with deploy-vultr.sh on one or three VMs or bare-metal servers, runs the published workbooks from node1 with the same driver as the hypervisor nightly and OCI, and destroys the deployment, failing if anything is left billing. It needs one secret, VULTR_API_KEY, and generates its SSH key per run. The guide gains screenshots of the API key's access control list and of the nodes in the Vultr portal, and its portal navigation now matches. * fix: install setup.sh on Vultr nodes when the tarball does not carry it build-install-artifacts.sh writes setup.sh beside the tarball, so a node installed from its tarball had no /usr/local/share/spinifex/setup.sh and arming the firewall failed. * fix: pass the remote command after the destination in the Vultr e2e driver * fix: run each workbook in the cluster's own region The workbook driver passed no region, so every workbook used its default, ap-southeast-2, and on a cluster in any other region every request failed SignatureDoesNotMatch. It now passes the region of the node's spinifex profile. * feat: count a node's vCPUs as online CPUs and reserve by host size (ADR-0015) A node counted physical cores, collapsing SMT threads, and held back a fixed 2 vCPU, so a 16-thread host offered 6 vCPU to instances. A vCPU is now one online CPU, as AWS counts it: a thread with SMT on, a core with SMT off or on ARM. The OS and Spinifex keep 1 CPU up to 8, 2 up to 16, 4 up to 63 and 8 from 64, and the rest is allocated one to one. The existing overrides stay operator-only. Invariant tests pin every tier boundary. * docs: state the Vultr guide's sizing under the online-CPU count * docs: drop an unmeasured EKS sizing claim from the Vultr README * fix: let a guest reach an Elastic IP hosted on its own node A guest's packet to an EIP on its own node left its VPC from the transit range, was masqueraded to the transit host address, and went back into OVN, which exempts that range from EIP DNAT, so it was never answered. On Vultr an EKS worker could not reach its control plane's endpoint, so it never joined. AWS lets an instance reach a public address in its own VPC. Hairpinned transit traffic now jumps to SPX-EIP-HAIRPIN, above the egress MASQUERADE, where each EIP's rule sends it from the uplink address, as the host's own traffic to the EIP already is. The rule follows the EIP's ingress plumbing in and out, and anything else returns to the MASQUERADE. * fix: resolve Spinifex's LB and EKS names on Vultr nodes Vultr's Ubuntu image writes a static resolv.conf naming Vultr's resolvers, so lookups bypassed the split DNS Spinifex configures in systemd-resolved and an EKS endpoint name did not resolve on the node. deploy-vultr.sh points resolv.conf at the resolved stub, as stock Ubuntu does. * test: prove a guest reaches an EIP on its own node in the natuplink e2e The nat-single cells run it with a public pool, so the hairpin path that left EKS workers unable to reach their control plane on Vultr is now covered nightly. * docs: record the natuplink hairpin check in the e2e coverage index * fix: keep each Vultr node's logs when an e2e run fails The driver destroyed a node whose install failed together with the log that said why. It now copies /root/*.log from every node into the run's artifacts before the teardown, and the workflow uploads them. ---------
Spinifex Scripts
Node installation, dev-environment lifecycle, guest image builds, and verification tooling for Spinifex.
Install and node setup
| Script | What it does |
|---|---|
setup.sh |
The production binary installer, served from install.mulgadc.com. Installs deps, users, sudoers, the host firewall policy, files, systemd units, logrotate and udev rules, and loads openvswitch, geneve, vhost_net and tun, listing them in /etc/modules-load.d/spinifex.conf for later boots. It masks firewalld wherever it is installed, since the Spinifex nftables policy is the only supported node firewall. It writes a needrestart drop-in so an apt upgrade never restarts the Spinifex, OVS or OVN units; restarting them after an upgrade is left to the operator. An unset --firewall keeps the mode recorded in /etc/spinifex/firewall/mode, and otherwise arms the policy when firewalld was active. Where systemd-resolved is installed but not running, resolvconf is absent and NetworkManager runs, it hands host DNS to resolved: it saves /etc/resolv.conf once to /var/lib/spinifex-reset/resolv.conf.orig, links /etc/resolv.conf to the resolved stub, and stops unless resolved has upstream servers and the package repository host still resolves. It links /usr/bin/spx to /usr/local/bin/spx where sudo's PATH leaves out /usr/local/bin, and takes the nbdkit plugin directory from nbdkit --dump-config, stopping when nbdkit cannot say. SETUP_STAGES runs a subset; ISO_BUILD=1 runs it inside the ISO builder's chroot. |
setup-ovn.sh |
Bootstraps a compute node for OVN VPC networking: installs OVN/OVS, creates br-int, configures the WAN bridge (bridged, DHCP, or routed-NAT uplink), sets the chassis identity, and applies overlay sysctls. It runs OVS and OVN as root, setting OVS_USER_ID and OVN_USER_ID where /etc/sysconfig holds them, and hooks the socket-permission helper onto ovs-vswitchd, ovn-controller, openvswitch-ipsec and, on EL, ovn-northd. --management also starts the OVN central services, writes their ovn-ctl flags to /etc/default/ovn-central (Debian) or /etc/sysconfig/ovn-northd (EL), and records the role in /etc/spinifex/ovn-role. With clustered databases it returns only once this node's NB and SB members are back in their RAFT clusters under a known leader, and fails after OVN_RAFT_REJOIN_WAIT seconds (default 60) otherwise, so reruns one node at a time never take two members of a cluster down. A later run without --management refuses while that file, or the OVN_CTL_OPTS line older nodes carry in /etc/default/ovn-central, records the role. It adds maxsize 100M to the OVS and OVN logrotate files, plus rotate 5 where the file sets no count, and rotates as root, matching the daemons. On a host whose IPsec stack is libreswan it enables, stops and masks openvswitch-ipsec.service and ipsec.service for the daemon to unmask when IPsec is on, unless openvswitch-ipsec.service is already running; elsewhere it enables and starts openvswitch-ipsec.service, unless the daemon has masked it to keep IPsec off. The links it creates persist through systemd-networkd where that is enabled, and otherwise through /etc/spinifex/hostnet.json, which spinifex-hostnet.service (spx admin hostnet apply) replays at boot and on every OVS start, starting ovn-controller again when an OVS stop took it down; the first run records the choice in /etc/spinifex/hostnet-backend. Where NetworkManager runs, it leaves the links Spinifex owns unmanaged. |
install-node.sh |
Forms a multi-node cluster across hosts that are already running Spinifex: rebuilds the OVN database in clustered form, runs spx admin init on the first host and spx admin join on the rest, then verifies. Before touching any host it refuses hosts of different OS families (distro_os_family) or spx versions, and --ipsec on when any host uses libreswan. The scripted form of install-multi-node. Destroys the joining nodes' logical network state and crypto — see its --help. |
dev-install.sh |
Full local dev setup through the production installer: builds from source, packs a tarball, runs setup.sh, initialises the cluster, and starts services under systemd. |
build-oracle-appliance.sh |
Turns an Oracle Linux 10.2 system reachable over SSH with sudo into a sealed gold image, then powers it off. It updates OL and reboots into the newest UEK R8, sets SELinux permissive, installs Spinifex without initialising it, and leaves spinifex.target and the NATS watchdog disabled. It creates the spinifex login with a default password that must be changed at first login, installs appliance/firstboot.sh as spinifex-appliance-firstboot.service, and wipes the system's identity (machine-id, SSH host keys, OVS chassis, NetworkManager profiles, logs). On each unit's first boot that service puts the first physical NIC with carrier on a DHCP br-wan, names the host spinifex-<mac>, runs setup-ovn.sh --management and spx admin init --nodes 1, and shows the IP and login on the console and in /etc/motd. --channel, --version and --tarball pick what to install. |
clone-deps.sh |
Clones or updates the viperblock and predastore checkouts alongside spinifex for cross-repo development. |
lib/distro.sh: host-OS capability probes
The node scripts select host-OS behaviour by probing for the capability, not by reading /etc/os-release; only setup.sh's supported-OS gate keys on the OS ID.
Each probe prints its answer, or names every candidate it tried on stderr and returns 1, which it also does when it finds both candidates of a pair.
| Probe | Answers |
|---|---|
distro_pkg_manager |
apt or dnf |
distro_ovs_unit |
openvswitch-switch.service (Debian, Ubuntu) or openvswitch.service (EL) |
distro_ovn_central_layout |
split (Debian's ovn-central plus separate NB and SB units) or lone-northd (EL's single ovn-northd.service) |
distro_ca_trust |
the trust-store update command and its anchor directory |
distro_ovs_logrotate |
/etc/logrotate.d/openvswitch-switch or /etc/logrotate.d/openvswitch |
distro_ovn_logrotate |
/etc/logrotate.d/ovn-common or /etc/logrotate.d/ovn |
distro_os_family |
the OS family of the /etc/os-release text on stdin: debian for Debian and Ubuntu, otherwise the ID |
distro_has_libreswan is a predicate rather than a probe: it succeeds when libreswan's IKE daemon, /usr/libexec/ipsec/pluto, is installed.
EL's openvswitch-ipsec drives libreswan even beside strongSwan, so finding both is not an error.
distro_missing_ovs_deps <unit-file>... is a check rather than a probe: it prints each OVS or OVN unit the given unit files depend on or order against that is not installed, and fails only when a file cannot be read.
setup-ovn.sh runs it against the installed Spinifex units and stops when one is missing, because systemd silently drops ordering against a unit that does not exist; a missing openvswitch-ipsec.service is only a warning.
setup-ovn.sh, uninstall-spx.sh and install-node.sh source it from lib/ beside themselves, then from /usr/local/share/spinifex/lib/distro.sh, where setup.sh installs it, and stop naming both paths when neither exists.
setup.sh is piped from the install endpoint before the tarball is downloaded, and install-node.sh --wipe copies node-reset.sh alone to each host, so those two carry inline copies of the probes they need between # --- BEGIN distro.sh probes --- markers.
lib/distro_test.sh fails when an inline copy differs from the library.
Go code uses spinifex/runtime/host/hostos for the same purpose.
System packages
setup.sh installs APT_RUNTIME_PACKAGES with apt, or on Oracle Linux OL_RUNTIME_PACKAGES with dnf, choosing by distro_pkg_manager.
Before the dnf install it enables ol10_UEKR8 and ol10_kvm_utils, excludes qemu-kvm* and qemu-img* from ol10_appstream so QEMU always comes from KVM Utilities, and writes /etc/yum.repos.d/spinifex-nfv-sig.repo for OVS 3.5 and OVN 25.03 from the CentOS NFV SIG.
That repo carries an includepkgs= allowlist, and its key is refused unless its fingerprint matches NFV_SIG_KEY_FPR.
It then installs kernel-uek-modules for the running kernel, stopping with a reboot instruction if that version has left the repositories, and loads openvswitch, geneve, vhost_net and tun.
make install-system takes the same dnf path, adding the build packages from CodeReady Builder.
| Script | What it does |
|---|---|
lint-apt-packages.sh <image> |
Resolves the apt list against a Debian or Ubuntu image's real indices with apt-get install -s. |
lint-dnf-packages.sh [image] |
Runs setup.sh's Oracle Linux repository setup in an oraclelinux:10 container, resolves the dnf list, and fails unless QEMU comes from KVM Utilities and OVS and OVN from the NFV SIG. |
check-glibc-ceiling.sh <ceiling> <elf>... |
Fails when an ELF file needs a glibc symbol version newer than the ceiling, naming the symbols. build/Dockerfile.distro runs it on spx and the nbdkit plugin with 2.34, so the one release tarball keeps loading on every supported OS. |
Teardown
Both ship to /usr/local/share/spinifex/ on every node, so they are available on a host installed with curl … | bash and not only from a checkout. They differ in what they take away:
| Script | What it does |
|---|---|
node-reset.sh |
Clears state and leaves the node installed: stops services, destroys guests, wipes /etc/spinifex, /var/lib/spinifex, /var/log/spinifex and the OVN databases, and removes the OVS bridges. On a libreswan host it deletes the ovs_cert_* and ovs_certkey_* entries ovs-monitor-ipsec imported into libreswan's NSS database, leaving other identities and /etc/ipsec.d/*.conf alone. It stops OVS and OVN central by the unit names the host's packages use. It leaves the NetworkManager unmanaged-device config, which the next setup-ovn.sh run rewrites, so NetworkManager cannot take a NIC back in between. The node is then ready for a fresh spx admin init. --keep-data preserves /var/lib/spinifex; SPX_RESET_CONFIRM=reset runs it unattended. |
uninstall-spx.sh |
Removes the software — the inverse of setup.sh. Delegates the state teardown above to node-reset.sh rather than reimplementing it, then removes the binary, the nbdkit plugin, the systemd units and slices, the service users, the sudoers grant, the sysctl/logrotate/chrony/udev drop-ins, the systemd-networkd units, the host firewall policy and its loaded nft table, the Spinifex systemd-resolved drop-in, and the Spinifex CA from the host trust store. It unmasks firewalld and leaves it disabled. Where setup.sh handed DNS to systemd-resolved, it restores the saved /etc/resolv.conf and turns resolved off. Verifies afterwards and exits non-zero naming anything that survived, then lists the host settings setup.sh changed that it leaves in place, such as the dnf repositories, OVS_USER_ID, masked IPsec units and the SELinux mode. |
node-reset.sh finds guests by process name, qemu-system-* or qemu-kvm, owned by the daemon's service user, and waits for them to exit before it deletes any state.
node-reset_test.sh checks that match against real processes, and that reset-dev-env.sh uses the same pattern.
uninstall-spx.sh keeps /var/lib/spinifex and every package by default — uninstalling software should not destroy data, and a curl … | bash install lands on hosts that were already using openvswitch, libvirt and chrony. --purge-data and --purge-deps opt into each, and both confirm separately even under --yes (SPX_PURGE_CONFIRM=destroy for unattended data removal). --dry-run prints the whole plan and touches nothing.
--purge-deps removes with apt and is refused on dnf hosts, where its fixed list cannot tell what setup.sh installed from what the host already had, and dnf would also remove every package that requires them.
It refuses on a multi-node cluster, and --yes does not override that. Uninstalling one node of a running cluster strands its guests and leaves the survivors reconciling against a chassis that will never answer. Run spx admin cluster shutdown first, then the uninstaller on each node.
Dev environment
| Script | What it does |
|---|---|
reset-dev-env.sh |
Single-node reset: stops services, wipes /etc/, /var/lib/, /var/log/ and /run/spinifex, rebuilds, reinstalls via dev-install.sh, and relaunches a smoke-test instance. Preserves the existing node/network topology from spinifex.toml. |
dev-env/*.conf |
Topology profiles for reset-dev-env.sh --profile=NAME, declaring which bridge carries the wan, lan and vpc planes: single-nic (all collapsed onto wan), two-nic (vpc folds onto lan), three-plane (a NIC each). |
Verification and benchmarks
| Script | What it does |
|---|---|
smoke-test.sh |
End-to-end sanity check on a running node: imports an SSH keypair and the Ubuntu AMI, then launches an instance. --create-vpc builds the VPC, gateway, subnet and routes first and tears them down after; --nodes N launches through a spread placement group so each instance lands on a different physical server. TEST_GPU=1 adds GPU passthrough checks. It runs the AWS CLI by absolute path, since sudo's secure_path leaves out /usr/local/bin on Oracle Linux. |
verify-network.sh |
Asserts a node's datapath matches its declared planes — EIP reachability, which plane carries Geneve, and which interfaces services bind. Catches what smoke-test.sh cannot, since none of its traffic leaves the box. |
check-coverage.sh |
Fails the build if total Go coverage falls below the minimum threshold, excluding packages exempt from the requirement. |
diff-coverage.sh |
Coverage check scoped to changed lines only, against an auto-detected base ref (HEAD~1 on main, origin/main on dev, otherwise origin/dev). |
sync-aws-models.sh |
Replaces the embedded AWS Smithy models under internal/awsmodel/models/ with the ten service files from aws/api-models-aws at the given commit, and records that commit in internal/awsmodel/model_source.go. Takes a full 40-character SHA. |
ui-playwright.sh compare|baselines [args...] |
Runs the console's Playwright suite inside the image pinned in the frontend's playwright.image, the one CI's UI End-to-End job uses, with podman or docker. compare checks screenshots as CI does and baselines redraws them; extra arguments go to pnpm e2e. Installs into its own node_modules under ~/.cache/spinifex-ui-playwright, leaving the host's alone. make ui-e2e-container and make ui-baselines run it. |
run-bench.sh |
Launches a benchmark against running Ubuntu instances, auto-detecting the AMI for the host architecture. Tracks nbdkit and perf alongside the run. |
disk-performance.sh |
Runs a fio random 70/30 read-write benchmark swept across 4k/16k/128k/1M. Writes JSON results to /tmp/spinifex-disk-bench; override with BENCH_DIR, OUT_DIR, SIZE, JOBS, BLOCK_SIZES. |
network-performance.sh |
iperf throughput from several clients to one server, driven over SSH, writing a summary.txt of Gbit/s per client. --server-ip sets the address clients dial, so pointing it at instance private addresses measures the VPC overlay rather than the external pool. |
Guest images
| Script | What it does |
|---|---|
build-system-image.sh |
Builds a minimal system AMI (Alpine or Ubuntu) from a manifest. All customisation happens inside the libguestfs appliance, so the build touches no host block device, no host mount, and needs no sudo. --import also registers it as an AMI. |
publish-system-image.sh |
Converts a built raw image to compressed qcow2, writes the .sha256 sidecar the catalog verifier expects, and uploads both to the R2 bucket behind iso.mulgadc.com/system-ami/. |
build-microvm-image.sh |
Builds the minimal Alpine kernel + initramfs used by the QEMU microvm machine type and the lb-agent, carrying haproxy (ALB) and nginx-stream (NLB) data planes. |
build-qboot.sh |
Builds qboot, the firmware the daemon passes to every microvm with -bios, from a pinned upstream commit in a digest-pinned Debian container. Fails unless the output matches the recorded sha256, which is also Debian 13's qboot.rom. Writes qboot.rom and its GPL-2.0 licence beside the microvm kernel, and the matching source tarball, which releases publish, under build/qboot/. make build-microvm-image runs it. |
mkosi-build.sh |
Runs an mkosi system-image build inside the pinned builder container, so a runner needs only Docker and the toolchain cannot drift. --image names an output and expands to the right profile list. |
mkosi-publish.sh |
Publishes an mkosi-built image to R2 from inside the same pinned container — qcow2 conversion, .sha256 sidecar, and upload. |
mkosi-common.sh |
Shared plumbing sourced by both mkosi scripts. Builds the toolchain image once, with the builder account matched to the invoking user so bind-mounted output is not written back as root. |
mkosi-builder.Dockerfile |
The pinned image-build toolchain. Debian-hosted regardless of target distro, since mkosi bootstraps the target from its own archive. |
images/ — per-image build inputs
Each subdirectory holds a manifest.conf, init scripts, and a setup.sh that build-system-image.sh runs inside the libguestfs appliance after packages and files are placed. *_test.sh files are shell unit tests for the helper of the same name.
| Path | What it builds |
|---|---|
ecs-agent/ |
The Alpine spinifex-ecs-node container-instance AMI (containerd + ecs-agent). setup.sh sets OpenRC exec bits, wires the CNI bin path, and applies the serial-console tweaks. |
ecs-agent-gpu/ |
The Ubuntu ECS GPU container-instance AMI. Same CNI and containerd wiring adjusted for Ubuntu's package layout, plus the headless NVIDIA driver and nvidia-container-toolkit. |
eks-node/ |
The unified Alpine EKS node AMI (K3s server or agent, selected at first boot). setup.sh installs the pinned, SHA-verified K3s binary and the role init scripts. |
eks-node-gpu/ |
The Ubuntu EKS GPU worker AMI. Agent role only — GPU nodes never run the control plane — plus the NVIDIA GPU stack. |
rds-postgres/ |
The spinifex-rds-postgres AMI. setup.sh sets exec bits on the OpenRC services and creates the root-only directory cloud-init drops the agent env file and gateway CA into. |
rds-mariadb/ |
The spinifex-rds-mariadb AMI. Same layout as rds-postgres/, plus the /etc/my.cnf.d include drop-in and the conf.d/mariadb dependency that keeps the packaged service off an unbootstrapped datadir. |
ubuntu-gpu-{amd,nvidia}-setup.sh |
Chroot setup for GPU-capable Ubuntu guest images: ROCm CLI tools and AMD firmware, or the DKMS-built headless NVIDIA driver. Guest-side only — host VFIO setup is spx admin gpu setup. |
images/eks-node/ — in-guest helpers
Installed into the EKS node AMI and run at boot or on a timer. All host communication goes through the AWS gateway over SigV4; the VM never speaks NATS directly.
| Script | What it does |
|---|---|
eks-node-role.sh |
First-boot role selector. Reads SPINIFEX_K3S_ROLE from the cloud-init env file and enables the matching services — nothing is enabled at bake time. |
k3s-prestart.sh |
Server prestart: refuses to start on a restore-block marker, derives config.yaml from its skeleton, stages the konnectivity-agent and CoreDNS manifests, and waits for the token-webhook kubeconfig. |
k3s-agent-prestart.sh |
Agent prestart shared by the OpenRC and systemd units. Requires K3S_URL and K3S_TOKEN, then resolves the kubelet provider-id from IMDSv2 for EKS parity. |
k3s-first-boot.sh |
Runs once after the server is healthy: reads the node-token and admin kubeconfig, rewrites the kubeconfig server address to the cluster NLB endpoint, and publishes both to the host. |
konnectivity-server-prestart.sh |
Waits for the K3s CA and admin kubeconfig, mints the konnectivity serving cert, and writes the apiserver replica count for the caller to source. |
mulga-eks-addon-sync.sh |
Pulls staged managed-addon descriptors through the gateway, renders each bundle into the K3s auto-deploy dir, and reports per-addon delivery status back. |
mulga-eks-etcd-snapshot.sh |
Snapshots K3s embedded etcd to predastore. Installed into two crond periodic dirs and picks its tier — frequent or daily — from the dir it was invoked from. |
mulga-eks-k3s-recovery.sh |
Applies a control-plane etcd-quorum recovery directive before k3s starts. Pull-based and a no-op unless the reconciler escalated a wedged control plane. |
mulga-eks-state-report.sh |
Publishes server health (healthz, node count) to the cluster reconciler. The apiserver binds the VPC node-ip, which the host daemon cannot probe directly. |
mulga-eks-provider-id.sh |
Writes the K3s config drop-in supplying the metadata the EBS CSI driver needs from IMDS: provider-id plus the region, zone and instance-type labels. |
mulga-ebs-byid.sh |
Mints the EBS-style /dev/disk/by-id symlink the upstream EBS CSI node plugin resolves, from the virtio-blk serial. Alpine's busybox mdev has no eudev to do it. |
mulga-mgmt-net.sh |
Boot oneshot bringing a multi-NIC system VM's interfaces up by MAC from the QEMU fw_cfg netcfg blob, before cloud-init's network stage. cloud-init cannot reliably pick the right NIC out of two. |
mulga-vpc-mtu.sh |
Boot oneshot pinning the VPC data NIC MTU so flannel sizes its VXLAN overlay to fit the OVN geneve underlay. Otherwise large frames are silently dropped while handshakes pass. |
demo/ — GPU demo
| Script | What it does |
|---|---|
deploy.sh |
Deploys and starts the four-instance GPU demo (dashboard, YOLO, Ollama, qwen3-vl) from the *_IP and VIDEO_PATH env vars. |
dashboard_server.py |
Serves the demo web UI and proxies the streams from the GPU instances. |
yolo_stream.py |
Runs YOLO detection over a video file, serving an MJPEG stream of annotated frames and an SSE stream of qwen3-vl scene descriptions. |