mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-11-06 19:01:10 +00:00
Josh Siviour
b9bce93877
* feat(lambda): add the unregistered RequestResponse Invoke path Invoke resolves the function and qualifier with the existing lookup, authorizes lambda:InvokeFunction on the function ARN, and hands one synchronous attempt to an injected Executor. With no executor it fails closed; a Pending function is ResourceConflictException. An accepted attempt is not cancelled by a caller disconnect: only the wait is bounded. Event, DryRun, LogType Tail and other unimplemented options are refused explicitly. Lambda stays unregistered and no user code runs. Also makes two flaky tests deterministic: the workerboot key-mismatch test could pick an identical key 1 time in 256, and the loadgen first-breach test depended on a 2ms op staying under a 30ms wall-clock p99. * feat: track each Lambda code upload before storing it CreateFunction now writes a per-Region upload record before the name claim and before any code Put, so a failed upload, rollback, expired claim or late original writer cannot leave an untracked copy. Refused or ambiguous outcomes keep the record; nothing is deleted or inferred complete from an expired claim. CodeStored validates the upload record before committing and refuses a missing, unreadable, future-format, mismatched or non-owed record without changing either record. Attach CASes only the checked record. * feat(lambda): run test-link Invoke in a daemon-owned microVM Behind the lambdatestlink build tag only. The gateway executor presigns the exact code digest and sends one NATS request to this node's daemon; the daemon fetches and verifies the code, then runs one fresh microVM per invocation through a runner that holds QEMU's creating thread, kills only through a verified pidfd, and counts a stop only on a recorded exit status. The host enforces the function timeout from dispatch, and a late result is a timeout. Each attempt writes a marker before allocating an address or tap and refuses further test invocations while any marker remains, including after a restart. Taps are created exclusively and removed only while their ifindex and owner alias still match. Anything whose stop or release is not proven stays held. The default build is unchanged: Invoke refuses before presigning, sending or allocating, and Lambda stays unrouted. Limits: functions never leave Pending (the env15 run activated its test record by hand); no grant mechanism; a hung daemon leaves the hard timeout unenforced; held attempts are not recovered after a restart; requests are bounded by the 1 MiB NATS message; a timeout is an unknown outcome rather than AWS's Unhandled error. * feat(lambda): enforce attempt grants and a QEMU hard-kill backstop on the test link Test-only and local: everything here is behind the lambdatestlink build tag except the new domains/lambda/execution owner package. The default gateway still refuses Invoke before presigning, sending or allocating, the normal guest agent stays closed, and Lambda stays unregistered. Attempt grants (domains/lambda/execution): - One owner record per node holds the current attempt; every change is a conditional write, and an unrecognised format is left untouched. - The gateway issues a grant binding request ID, account, Region, function, revision and code digest before anything is sent; a second attempt is refused while one is unresolved. - The daemon starts only under the current matching grant, before it downloads code or allocates anything; stale, replayed or mismatched grants are refused with the record unchanged. - A result counts only if the owner accepts it under the dispatch deadline while the attempt is running; a late or uncertain acceptance is reported as an unknown outcome, never success. - An attempt ends only after the VM stop is proven and resources are released, or when nothing was launched; otherwise it is held as ambiguous. Time alone never releases it. An uncertain stop write is an unknown outcome: it may already have landed. QEMU hard-kill backstop (runtime/compute/lambda/testlink): - A preloaded initializer arms a SIGKILL timer inside QEMU before main and acknowledges with QEMU's PID, the deadline and a per-attempt nonce. Any failure after it loads kills the process before main. - The daemon checks the initializer against a pinned SHA-256 before launch, builds QEMU's environment without inherited LD_* or backstop settings, and sends no invocation frame without the exact acknowledgement; a missing or wrong acknowledgement kills QEMU and keeps state until the stop is proven. Limits, not production readiness: - Not the M1 decision. The grant binds request identity and code digest, not payload, memory or function ARN, and the guest does not yet bind generation or sequence. - If the preload never loads (missing file, static or setuid QEMU), nothing arms; a daemon hung before refusing can leave QEMU alive, and the only guarantee is that no invocation frame was sent. - Hash-then-load has a replacement race and is not a trust boundary; the initializer is built and installed by hand, with no packaging. - A killed QEMU stays a zombie until the daemon reaps it; held attempts are not recovered after restart; duplicate attempt IDs are checked only against the current attempt. * feat(lambda): seal the test-link timer library and bind the guest to its grant Test-link (lambdatestlink) build only. Lambda stays publicly unregistered and the normal guest agent stays fail-closed. Timer library: the daemon reads the hard-kill initializer once, checks those bytes against the pinned digest, copies them into a sealed memfd, verifies the seals and re-hashes the sealed copy, then passes it to QEMU as an inherited fd loaded through /proc/self/fd. The initializer closes that fd before it acknowledges. A kernel that rejects MFD_EXEC, or a memfd_noexec policy that refuses it, refuses the attempt; there is no fallback. Guest binding: the daemon writes a test-grant fw_cfg entry with the owner's grant ID, attempt ID, environment ID and generation, and the guest's scope now comes from that grant. Before installing code or starting the bootstrap, the guest refuses unless the grant names the authenticated environment generation, the first frame carries that grant and attempt ID, and the frame deadline is present and still ahead on the guest clock. Limits: - The digest is only as trustworthy as the build input that pins it; the memfd closes the swap window between check and load, it does not authenticate it. - Not yet shown under the real spinifex-daemon systemd sandbox (MemoryDenyWriteExecute, ProtectProc) or with memfd_noexec set to 1 or 2. - fw_cfg and the request frame both come from the executor, so matching them does not independently authenticate owner authority. This is not the M1 decision. - The guest clock check is a local guard only; it cannot detect clock skew or a delayed, superseded grant. The guest has no channel to the owner and checks the generation only on the first frame. - Held attempts are still not recovered after a daemon restart. * feat(lambda): recover held test-link attempts after a daemon restart Test-link (lambdatestlink) build only. Lambda stays publicly unregistered and the normal guest agent stays fail-closed. Launch identity: attempt.Run passes the session callback its own duplicate of the clone-time pidfd and closes SessionDone before publishing the result. Before the guest is sent anything, the daemon takes the VM's pidfs inode from that exact handle (refusing if the pidfd is not on pidfs or the VM has exited), with the host machine ID, boot ID, PID namespace and the launching daemon's own pidfs identity, and records it in the owner record with a conditional Launched transition. A failed write holds the attempt as ambiguous. Recovery: RecoverLoop retries for a bounded number of rounds on the daemon context, each round holding the node slot so it never runs beside a request. It holds anything it cannot scope to the recorded boot and PID namespace with the launcher gone, and holds an attempt with no launch identity. A PID with no process, or one now held by a process with another pidfs inode, proves the stop without signalling; the recorded VM still running is killed through its own pidfd and counted stopped only once the pidfd reports the exit. Cleanup order: files, tap and address (MgmtIPAllocator.ReleaseChecked, which reports an unconfirmed release; Release is unchanged for its callers), then a new owner state released that still blocks Issue, then the marker, then ended. Recovery finishes from either side of each step. Limits: - Reboot and partition recovery stay held until host identity or fencing is established; a machine ID can be copied, so it is not host proof. - A launch that crashes before its identity is recorded stays held. - No recovery if the daemon cannot restart or hangs. - Not yet run under the real spinifex-daemon systemd unit, and no kernel without pidfs was tested. - This is not the M1 decision. * fix(lambda): refuse ownerless checked mgmt IP releases ReleaseChecked now refuses an entry that names no node, or an allocator with no node bound, and leaves both the cluster and local entries in place. The checked path deletes its local entry only after a confirmed release and only if it still holds the same address. Legacy Release is unchanged. Callers must serialize Allocate and ReleaseChecked for one instance ID; the Lambda test-link slot does. Recovery tests that need real pidfs now skip only when the kernel's pidfds are not on pidfs, and a non-pidfs descriptor is shown to be refused. The runtime still fails closed. Adds an owner test for Store.Launched. * fix(lambda): retry test-link recovery until it settles RecoverLoop now retries on the daemon context until the held attempt settles, starting at 5s and doubling to a 5 minute cap, instead of giving up after ten rounds. Each round still takes the node slot, so it never runs beside Handle. It covers only what a previous daemon left: an attempt a later Handle in the same daemon holds stays held until the next restart. The serving log now says the guest does not verify owner grant authority. * feat(lambda): build the test-link daemon and initializer as one pair Add make build-lambda-testlink. It compiles the hard-kill initializer, checks it against the 2.34 glibc ceiling, pins its SHA-256 into a lambdatestlink spx build, and proves with TestLambdaHardKillPin that the daemon's own pin accepts that file and refuses a changed one. Both files are built in a staging directory and moved together only on success; any earlier pair is removed first, so a failed rebuild leaves no pair. The target refuses to build unless Go targets linux on the host architecture. The initializer now uses _POSIX_C_SOURCE with -std=c17, so it no longer needs __isoc23_strtol (glibc 2.38). The daemon refuses an initializer built for another ELF machine or class. Startup now begins recovery with only the owner store, tap and address cleanup. A missing bridge address, pin, trust store or matching initializer stops new invocations but not recovery. Limits: test-only and unregistered. The pin is only as trustworthy as the build host and toolchain; the initializer is still installed by hand and ships in no release artefact. linux/arm64 is untested. Recovery needs pidfs, which older kernels may lack. This is not the M1 decision. * feat(lambda): fix one CLOCK_BOOTTIME backstop deadline per test-link attempt Test-only (lambdatestlink); the default build and public Lambda route are unchanged. - The daemon reads CLOCK_BOOTTIME once before the grant Start and fixes one absolute deadline: bounded code fetch + function timeout + boot allowance + kill margin. Nothing recomputes or extends it; Handle never retries and recovery never relaunches. - The hard-kill initializer arms its SIGKILL timer at that absolute deadline (SPX_HARDKILL_DEADLINE_NS, TIMER_ABSTIME) instead of relative to QEMU start. - Dispatch is refused once the deadline has passed: before QEMU starts, when the initializer ack is read at or after it (VM killed, stop proven), and immediately before the invocation frame is sent. A clock read failure counts as expired. Limits: tap/IP setup, image read, initializer seal and QEMU start have no caps of their own beyond the boot allowance. An overdue timer is not shown to fire before vCPUs resume after host suspend. Owner-side retention, cross-node ordering, clock-rate assumptions and M1 remain unsolved; nothing is released by time and held attempts still need stop proof. * feat(lambda): boot the test-link runtime image from a pinned digest The test-link daemon now prepares its runtime image once at startup: each boot file is opened without following symlinks or blocking, capped by size, copied into a sealed memfd and hashed there, and the image digest over those sealed copies must equal LAMBDA_IMAGE_DIGEST pinned into the build. QEMU boots only the sealed copies (fds 3-5), and firmware comes from the image directory rather than the shared microvm qboot.rom. The digest identifies the bytes; it does not authenticate where they came from. build-lambda-testlink-image builds a separately tagged guest into its own outputs and prints the digest with the daemon's own file selection. Both test-link targets refuse anything but an x86_64 host building linux/amd64, and the digest input is read raw and validated before it reaches ldflags. * test: cover test-link image load failures and daemon startup release Exercise each sealed-copy failure path through the memfd hook and check partial copies are closed. Extract the daemon's readiness and release steps so startup tests can show the image is released when serving fails and that Handle refuses without a prepared image. * fix: align test-link Invoke errors with AWS and verify guest build tag Malformed JSON payloads return 400 InvalidRequestContentException and oversized requests return 413 with X-Amzn-Errortype RequestEntityTooLargeException, both before the executor runs. Timeouts and lost or failed sends are reported as unknown rather than as a function timeout, and any Sandbox.Timedout result is treated as unknown. Oversized function responses report Unhandled with Function.ResponseSizeTooLarge in the body. Add lambda-image-agent-tag, which reads the agent's build information from each built initramfs and refuses an image whose lambdatestlink tag does not match the target. Add tests that the default gateway cannot invoke and that management IP Release drops the local entry when the address is held elsewhere or the KV is down. * test: use the shared RequireAWSCode helper in Lambda tests The per-package requireCode helpers were replaced by testutil.RequireAWSCode, which makes the same assertions; the Lambda Invoke and upload tests still called the removed helper. * fix: validate and authorize qualified Lambda Invoke requests Invoke now runs every deterministic check before authorization and authorizes the qualified function ARN derived from the same validated target. Qualifiers follow the documented FunctionName and Qualifier patterns and length limits, and a malformed query string is refused. Code download errors no longer carry transport detail, so a presigned credential in a bad redirect cannot reach the reply or the gateway log. A seam built only with lambdatestlink and lambdatestlinkseam drives real session outcomes through the executor and composed gateway in one test. * test: restore the loadgen and workerboot tests to dev * test: drive a bootstrap-posted timeout through the composed Invoke path * fix: carry handler and qualifier to the test-link runtime and refuse function environments The test link now passes the stored handler to the guest as _HANDLER and the validated qualifier into the invoked-function ARN. Functions with environment variables are refused before any presign, grant or dispatch, since delivering them needs a separate secret-transport design. * test: stub QEMU for the test-link package so it runs on hosts without it
Spinifex Scripts
Node installation, dev-environment lifecycle, guest image builds, and verification tooling for Spinifex.
Install and node setup
| Script | What it does |
|---|---|
setup.sh |
The production binary installer, served from install.mulgadc.com. Installs deps, users, sudoers, the host firewall policy, files, systemd units, logrotate and udev rules, and loads openvswitch, geneve, vhost_net and tun, listing them in /etc/modules-load.d/spinifex.conf for later boots. It masks firewalld wherever it is installed, since the Spinifex nftables policy is the only supported node firewall. It writes a needrestart drop-in so an apt upgrade never restarts the Spinifex, OVS or OVN units; restarting them after an upgrade is left to the operator. An unset --firewall keeps the mode recorded in /etc/spinifex/firewall/mode, and otherwise arms the policy when firewalld was active. Where systemd-resolved is installed but not running, resolvconf is absent and NetworkManager runs, it hands host DNS to resolved: it saves /etc/resolv.conf once to /var/lib/spinifex-reset/resolv.conf.orig, writes /etc/NetworkManager/conf.d/90-spinifex-dns.conf (dns=systemd-resolved, rc-manager=unmanaged) so NetworkManager never rewrites /etc/resolv.conf, links /etc/resolv.conf to the resolved stub, and stops unless the link holds, resolved has upstream servers and the package repository host still resolves. A rerun after that handover adds a missing drop-in, reloads NetworkManager and relinks the stub. It links /usr/bin/spx to /usr/local/bin/spx where sudo's PATH leaves out /usr/local/bin, and takes the nbdkit plugin directory from nbdkit --dump-config, stopping when nbdkit cannot say. SETUP_STAGES runs a subset; ISO_BUILD=1 runs it inside the ISO builder's chroot. |
setup-ovn.sh |
Bootstraps a compute node for OVN VPC networking: installs OVN/OVS, creates br-int, configures the WAN bridge (bridged, DHCP, or routed-NAT uplink), sets the chassis identity, and applies overlay sysctls. It runs OVS and OVN as root, setting OVS_USER_ID and OVN_USER_ID where /etc/sysconfig holds them, and hooks the socket-permission helper onto ovs-vswitchd, ovn-controller, openvswitch-ipsec and, on EL, ovn-northd. --management also starts the OVN central services, writes their ovn-ctl flags to /etc/default/ovn-central (Debian) or /etc/sysconfig/ovn-northd (EL), and records the role in /etc/spinifex/ovn-role. With clustered databases it returns only once this node's NB and SB members are back in their RAFT clusters under a known leader, and fails after OVN_RAFT_REJOIN_WAIT seconds (default 60) otherwise, so reruns one node at a time never take two members of a cluster down. A later run without --management refuses while that file, or the OVN_CTL_OPTS line older nodes carry in /etc/default/ovn-central, records the role. It adds maxsize 100M to the OVS and OVN logrotate files, plus rotate 5 where the file sets no count, and rotates as root, matching the daemons. On a host whose IPsec stack is libreswan it enables, stops and masks openvswitch-ipsec.service and ipsec.service for the daemon to unmask when IPsec is on, unless openvswitch-ipsec.service is already running; elsewhere it enables and starts openvswitch-ipsec.service, unless the daemon has masked it to keep IPsec off. The links it creates persist through systemd-networkd where that is enabled, and otherwise through /etc/spinifex/hostnet.json, which spinifex-hostnet.service (spx admin hostnet apply) replays at boot and on every OVS start, starting ovn-controller again when an OVS stop took it down; the first run records the choice in /etc/spinifex/hostnet-backend. Where NetworkManager runs, it leaves the links Spinifex owns unmanaged. |
install-node.sh |
Forms a multi-node cluster across hosts that are already running Spinifex: rebuilds the OVN database in clustered form, runs spx admin init on the first host and spx admin join on the rest, then verifies. Before touching any host it refuses hosts of different OS families (distro_os_family) or spx versions, and --ipsec on when any host uses libreswan. The scripted form of install-multi-node. Destroys the joining nodes' logical network state and crypto — see its --help. |
dev-install.sh |
Full local dev setup through the production installer: builds from source, packs a tarball, runs setup.sh, initialises the cluster, and starts services under systemd. |
build-oracle-appliance.sh |
Turns an Oracle Linux 10.2 system reachable over SSH with sudo into a sealed gold image, then powers it off. It updates OL and reboots into the newest UEK R8, sets SELinux permissive, installs Spinifex without initialising it, and leaves spinifex.target and the NATS watchdog disabled. It creates the spinifex login with a default password that must be changed at first login, installs appliance/firstboot.sh as spinifex-appliance-firstboot.service, and wipes the system's identity (machine-id, SSH host keys, OVS chassis, NetworkManager profiles, logs). On each unit's first boot that service puts the first physical NIC with carrier on a DHCP br-wan, names the host spinifex-<mac>, runs setup-ovn.sh --management and spx admin init --nodes 1, and shows the IP and login on the console and in /etc/motd. --channel, --version and --tarball pick what to install. |
clone-deps.sh |
Clones or updates the viperblock and predastore checkouts alongside spinifex for cross-repo development. |
lib/distro.sh: host-OS capability probes
The node scripts select host-OS behaviour by probing for the capability, not by reading /etc/os-release; only setup.sh's supported-OS gate keys on the OS ID.
Each probe prints its answer, or names every candidate it tried on stderr and returns 1, which it also does when it finds both candidates of a pair.
| Probe | Answers |
|---|---|
distro_pkg_manager |
apt or dnf |
distro_ovs_unit |
openvswitch-switch.service (Debian, Ubuntu) or openvswitch.service (EL) |
distro_ovn_central_layout |
split (Debian's ovn-central plus separate NB and SB units) or lone-northd (EL's single ovn-northd.service) |
distro_ca_trust |
the trust-store update command and its anchor directory |
distro_ovs_logrotate |
/etc/logrotate.d/openvswitch-switch or /etc/logrotate.d/openvswitch |
distro_ovn_logrotate |
/etc/logrotate.d/ovn-common or /etc/logrotate.d/ovn |
distro_os_family |
the OS family of the /etc/os-release text on stdin: debian for Debian and Ubuntu, otherwise the ID |
distro_has_libreswan is a predicate rather than a probe: it succeeds when libreswan's IKE daemon, /usr/libexec/ipsec/pluto, is installed.
EL's openvswitch-ipsec drives libreswan even beside strongSwan, so finding both is not an error.
distro_missing_ovs_deps <unit-file>... is a check rather than a probe: it prints each OVS or OVN unit the given unit files depend on or order against that is not installed, and fails only when a file cannot be read.
setup-ovn.sh runs it against the installed Spinifex units and stops when one is missing, because systemd silently drops ordering against a unit that does not exist; a missing openvswitch-ipsec.service is only a warning.
setup-ovn.sh, uninstall-spx.sh and install-node.sh source it from lib/ beside themselves, then from /usr/local/share/spinifex/lib/distro.sh, where setup.sh installs it, and stop naming both paths when neither exists.
setup.sh is piped from the install endpoint before the tarball is downloaded, and install-node.sh --wipe copies node-reset.sh alone to each host, so those two carry inline copies of the probes they need between # --- BEGIN distro.sh probes --- markers.
lib/distro_test.sh fails when an inline copy differs from the library.
Go code uses spinifex/runtime/host/hostos for the same purpose.
System packages
setup.sh installs APT_RUNTIME_PACKAGES with apt, or on Oracle Linux OL_RUNTIME_PACKAGES with dnf, choosing by distro_pkg_manager.
Before the dnf install it enables ol10_UEKR8 and ol10_kvm_utils, excludes qemu-kvm* and qemu-img* from ol10_appstream so QEMU always comes from KVM Utilities, and writes /etc/yum.repos.d/spinifex-nfv-sig.repo for OVS 3.5 and OVN 25.03 from the CentOS NFV SIG.
That repo carries an includepkgs= allowlist, and its key is refused unless its fingerprint matches NFV_SIG_KEY_FPR.
It then installs kernel-uek-modules for the running kernel, stopping with a reboot instruction if that version has left the repositories, and loads openvswitch, geneve, vhost_net and tun.
make install-system takes the same dnf path, adding the build packages from CodeReady Builder.
| Script | What it does |
|---|---|
lint-apt-packages.sh <image> |
Resolves the apt list against a Debian or Ubuntu image's real indices with apt-get install -s. |
lint-dnf-packages.sh [image] |
Runs setup.sh's Oracle Linux repository setup in an oraclelinux:10 container, resolves the dnf list, and fails unless QEMU comes from KVM Utilities and OVS and OVN from the NFV SIG. |
check-glibc-ceiling.sh <ceiling> <elf>... |
Fails when an ELF file needs a glibc symbol version newer than the ceiling, naming the symbols. build/Dockerfile.distro runs it on spx and the nbdkit plugin with 2.34, so the one release tarball keeps loading on every supported OS. |
Teardown
Both ship to /usr/local/share/spinifex/ on every node, so they are available on a host installed with curl … | bash and not only from a checkout. They differ in what they take away:
| Script | What it does |
|---|---|
node-reset.sh |
Clears state and leaves the node installed: stops services, destroys guests, wipes /etc/spinifex, /var/lib/spinifex, /var/log/spinifex and the OVN databases, and removes the OVS bridges. On a libreswan host it deletes the ovs_cert_* and ovs_certkey_* entries ovs-monitor-ipsec imported into libreswan's NSS database, leaving other identities and /etc/ipsec.d/*.conf alone. It stops OVS and OVN central by the unit names the host's packages use. It leaves the NetworkManager unmanaged-device config, which the next setup-ovn.sh run rewrites, so NetworkManager cannot take a NIC back in between. The node is then ready for a fresh spx admin init. --keep-data preserves /var/lib/spinifex; SPX_RESET_CONFIRM=reset runs it unattended. |
uninstall-spx.sh |
Removes the software — the inverse of setup.sh. Delegates the state teardown above to node-reset.sh rather than reimplementing it, then removes the binary, the nbdkit plugin, the systemd units and slices, the service users, the sudoers grant, the sysctl/logrotate/chrony/udev drop-ins, the systemd-networkd units, the host firewall policy and its loaded nft table, the Spinifex systemd-resolved drop-in, and the Spinifex CA from the host trust store. It unmasks firewalld and leaves it disabled. Where setup.sh handed DNS to systemd-resolved, it restores the saved /etc/resolv.conf and turns resolved off. Verifies afterwards and exits non-zero naming anything that survived, then lists the host settings setup.sh changed that it leaves in place, such as the dnf repositories, OVS_USER_ID, masked IPsec units and the SELinux mode. |
node-reset.sh finds guests by process name, qemu-system-* or qemu-kvm, owned by the daemon's service user, and waits for them to exit before it deletes any state.
node-reset_test.sh checks that match against real processes, and that reset-dev-env.sh uses the same pattern.
uninstall-spx.sh keeps /var/lib/spinifex and every package by default — uninstalling software should not destroy data, and a curl … | bash install lands on hosts that were already using openvswitch, libvirt and chrony. --purge-data and --purge-deps opt into each, and both confirm separately even under --yes (SPX_PURGE_CONFIRM=destroy for unattended data removal). --dry-run prints the whole plan and touches nothing.
--purge-deps removes with apt and is refused on dnf hosts, where its fixed list cannot tell what setup.sh installed from what the host already had, and dnf would also remove every package that requires them.
It refuses on a multi-node cluster, and --yes does not override that. Uninstalling one node of a running cluster strands its guests and leaves the survivors reconciling against a chassis that will never answer. Run spx admin cluster shutdown first, then the uninstaller on each node.
Dev environment
| Script | What it does |
|---|---|
reset-dev-env.sh |
Single-node reset: stops services, wipes /etc/, /var/lib/, /var/log/ and /run/spinifex, rebuilds, reinstalls via dev-install.sh, and relaunches a smoke-test instance. Preserves the existing node/network topology from spinifex.toml. |
dev-env/*.conf |
Topology profiles for reset-dev-env.sh --profile=NAME, declaring which bridge carries the wan, lan and vpc planes: single-nic (all collapsed onto wan), two-nic (vpc folds onto lan), three-plane (a NIC each). |
Verification and benchmarks
| Script | What it does |
|---|---|
smoke-test.sh |
End-to-end sanity check on a running node: imports an SSH keypair and the Ubuntu AMI, then launches an instance. --create-vpc builds the VPC, gateway, subnet and routes first and tears them down after; --nodes N launches through a spread placement group so each instance lands on a different physical server. TEST_GPU=1 adds GPU passthrough checks. It runs the AWS CLI by absolute path, since sudo's secure_path leaves out /usr/local/bin on Oracle Linux. |
verify-network.sh |
Asserts a node's datapath matches its declared planes — EIP reachability, which plane carries Geneve, and which interfaces services bind. Catches what smoke-test.sh cannot, since none of its traffic leaves the box. |
check-coverage.sh |
Fails the build if total Go coverage falls below the minimum threshold, excluding packages exempt from the requirement. |
diff-coverage.sh |
Coverage check scoped to changed lines only, against an auto-detected base ref (HEAD~1 on main, origin/main on dev, otherwise origin/dev). |
sync-aws-models.sh |
Replaces the embedded AWS Smithy models under internal/awsmodel/models/ with the ten service files from aws/api-models-aws at the given commit, and records that commit in internal/awsmodel/model_source.go. Takes a full 40-character SHA. |
ui-playwright.sh compare|baselines [args...] |
Runs the console's Playwright suite inside the image pinned in the frontend's playwright.image, the one CI's UI End-to-End job uses, with podman or docker. compare checks screenshots as CI does and baselines redraws them; extra arguments go to pnpm e2e. Installs into its own node_modules under ~/.cache/spinifex-ui-playwright, leaving the host's alone. make ui-e2e-container and make ui-baselines run it. |
run-bench.sh |
Launches a benchmark against running Ubuntu instances, auto-detecting the AMI for the host architecture. Tracks nbdkit and perf alongside the run. |
disk-performance.sh |
Runs a fio random 70/30 read-write benchmark swept across 4k/16k/128k/1M. Writes JSON results to /tmp/spinifex-disk-bench; override with BENCH_DIR, OUT_DIR, SIZE, JOBS, BLOCK_SIZES. |
network-performance.sh |
iperf throughput from several clients to one server, driven over SSH, writing a summary.txt of Gbit/s per client. --server-ip sets the address clients dial, so pointing it at instance private addresses measures the VPC overlay rather than the external pool. |
Guest images
| Script | What it does |
|---|---|
build-system-image.sh |
Builds a minimal system AMI (Alpine or Ubuntu) from a manifest. All customisation happens inside the libguestfs appliance, so the build touches no host block device, no host mount, and needs no sudo. --import also registers it as an AMI. |
publish-system-image.sh |
Converts a built raw image to compressed qcow2, writes the .sha256 sidecar the catalog verifier expects, and uploads both to the R2 bucket behind iso.mulgadc.com/system-ami/. |
build-microvm-image.sh |
Builds the minimal Alpine kernel + initramfs used by the QEMU microvm machine type and the lb-agent, carrying haproxy (ALB) and nginx-stream (NLB) data planes. |
build-qboot.sh |
Builds qboot, the firmware the daemon passes to every microvm with -bios, from a pinned upstream commit in a digest-pinned Debian container. Fails unless the output matches the recorded sha256, which is also Debian 13's qboot.rom. Writes qboot.rom and its GPL-2.0 licence beside the microvm kernel, and the matching source tarball, which releases publish, under build/qboot/. make build-microvm-image runs it. |
mkosi-build.sh |
Runs an mkosi system-image build inside the pinned builder container, so a runner needs only Docker and the toolchain cannot drift. --image names an output and expands to the right profile list. |
mkosi-publish.sh |
Publishes an mkosi-built image to R2 from inside the same pinned container — qcow2 conversion, .sha256 sidecar, and upload. |
mkosi-common.sh |
Shared plumbing sourced by both mkosi scripts. Builds the toolchain image once, with the builder account matched to the invoking user so bind-mounted output is not written back as root. |
mkosi-builder.Dockerfile |
The pinned image-build toolchain. Debian-hosted regardless of target distro, since mkosi bootstraps the target from its own archive. |
images/ — per-image build inputs
Each subdirectory holds a manifest.conf, init scripts, and a setup.sh that build-system-image.sh runs inside the libguestfs appliance after packages and files are placed. *_test.sh files are shell unit tests for the helper of the same name.
| Path | What it builds |
|---|---|
ecs-agent/ |
The Alpine spinifex-ecs-node container-instance AMI (containerd + ecs-agent). setup.sh sets OpenRC exec bits, wires the CNI bin path, and applies the serial-console tweaks. |
ecs-agent-gpu/ |
The Ubuntu ECS GPU container-instance AMI. Same CNI and containerd wiring adjusted for Ubuntu's package layout, plus the headless NVIDIA driver and nvidia-container-toolkit. |
eks-node/ |
The unified Alpine EKS node AMI (K3s server or agent, selected at first boot). setup.sh installs the pinned, SHA-verified K3s binary and the role init scripts. |
eks-node-gpu/ |
The Ubuntu EKS GPU worker AMI. Agent role only — GPU nodes never run the control plane — plus the NVIDIA GPU stack. |
rds-postgres/ |
The spinifex-rds-postgres AMI. setup.sh sets exec bits on the OpenRC services and creates the root-only directory cloud-init drops the agent env file and gateway CA into. |
rds-mariadb/ |
The spinifex-rds-mariadb AMI. Same layout as rds-postgres/, plus the /etc/my.cnf.d include drop-in and the conf.d/mariadb dependency that keeps the packaged service off an unbootstrapped datadir. |
ubuntu-gpu-{amd,nvidia}-setup.sh |
Chroot setup for GPU-capable Ubuntu guest images: ROCm CLI tools and AMD firmware, or the DKMS-built headless NVIDIA driver. Guest-side only — host VFIO setup is spx admin gpu setup. |
images/eks-node/ — in-guest helpers
Installed into the EKS node AMI and run at boot or on a timer. All host communication goes through the AWS gateway over SigV4; the VM never speaks NATS directly.
| Script | What it does |
|---|---|
eks-node-role.sh |
First-boot role selector. Reads SPINIFEX_K3S_ROLE from the cloud-init env file and enables the matching services — nothing is enabled at bake time. |
k3s-prestart.sh |
Server prestart: refuses to start on a restore-block marker, derives config.yaml from its skeleton, stages the konnectivity-agent and CoreDNS manifests, and waits for the token-webhook kubeconfig. |
k3s-agent-prestart.sh |
Agent prestart shared by the OpenRC and systemd units. Requires K3S_URL and K3S_TOKEN, then resolves the kubelet provider-id from IMDSv2 for EKS parity. |
k3s-first-boot.sh |
Runs once after the server is healthy: reads the node-token and admin kubeconfig, rewrites the kubeconfig server address to the cluster NLB endpoint, and publishes both to the host. |
konnectivity-server-prestart.sh |
Waits for the K3s CA and admin kubeconfig, mints the konnectivity serving cert, and writes the apiserver replica count for the caller to source. |
mulga-eks-addon-sync.sh |
Pulls staged managed-addon descriptors through the gateway, renders each bundle into the K3s auto-deploy dir, and reports per-addon delivery status back. |
eks-etcd-member |
Go helper built from cmd/eks-etcd-member and installed on control-plane (server) images. remove --peer-ip <ip> [--self-ip <ip>] evicts a terminated peer's etcd member over the local client endpoint, because control planes run no agent and k3s never prunes them. First boot calls it for a replacement control plane and treats an absent member as success. |
mulga-eks-etcd-snapshot.sh |
Snapshots K3s embedded etcd to predastore. Installed into two crond periodic dirs and picks its tier — frequent or daily — from the dir it was invoked from. |
mulga-eks-k3s-recovery.sh |
Applies a control-plane etcd-quorum recovery directive before k3s starts. Pull-based and a no-op unless the reconciler escalated a wedged control plane. |
mulga-eks-state-report.sh |
Publishes server health (healthz, node count) to the cluster reconciler. The apiserver binds the VPC node-ip, which the host daemon cannot probe directly. |
mulga-eks-provider-id.sh |
Writes the K3s config drop-in supplying the metadata the EBS CSI driver needs from IMDS: provider-id plus the region, zone and instance-type labels. |
mulga-ebs-byid.sh |
Mints the EBS-style /dev/disk/by-id symlink the upstream EBS CSI node plugin resolves, from the virtio-blk serial. Alpine's busybox mdev has no eudev to do it. |
mulga-mgmt-net.sh |
Boot oneshot bringing a multi-NIC system VM's interfaces up by MAC from the QEMU fw_cfg netcfg blob, before cloud-init's network stage. cloud-init cannot reliably pick the right NIC out of two. |
mulga-vpc-mtu.sh |
Boot oneshot pinning the VPC data NIC MTU so flannel sizes its VXLAN overlay to fit the OVN geneve underlay. Otherwise large frames are silently dropped while handshakes pass. |
demo/ — GPU demo
| Script | What it does |
|---|---|
deploy.sh |
Deploys and starts the four-instance GPU demo (dashboard, YOLO, Ollama, qwen3-vl) from the *_IP and VIDEO_PATH env vars. |
dashboard_server.py |
Serves the demo web UI and proxies the streams from the GPU instances. |
yolo_stream.py |
Runs YOLO detection over a video file, serving an MJPEG stream of annotated frames and an SSE stream of qwen3-vl scene descriptions. |