* feat(oci): separate using an instance principal from creating one
The allocator needs OCI credentials at runtime, and a node with none forms,
passes every health check, then refuses every launch wanting a public address.
Instance principal is the credential route that puts no key material on a node,
but it needs a dynamic group and a policy, and those are tenancy-root resources
the compartment-scoped user the rest of the configuration runs as cannot create.
enable_instance_principal conflated the two, so the better credential route was
only reachable by a tenancy admin. instance_principal now takes off, adopt or
create: adopt selects the credential and creates nothing, because the matching
rule covers the whole compartment and so names no instance to reference.
setup-identity.sh owns the create case, with its own state. validate-topology.sh
destroys its state at the end of every run, so holding tenancy resources there
would let a teardown delete the tenancy's policy.
Also package install-node.sh and smoke-test.sh, so a cluster can be formed from
a curl | bash install with no checkout. smoke-test.sh is not optional: formation
scp's it from its own directory for the post-formation verification.
* feat(oci): take the credential handoff from an operator hook, and widen the workflow name
An API-key deployment needs a credential on each node, and nothing in this
repository can supply one: a key in user-data is readable from instance metadata
for the life of the instance, and this host runs other people's guests. A manual
install does not survive either, because a topology destroys itself on success.
So the harness calls an operator-owned executable after formation and before the
pool, the only window where a node has /etc/spinifex and has not started the
allocator. Instance principal needs no handoff, so the hook is skipped there.
The workflow is renamed for the vendors it will cover rather than the first one.
Nothing is orphaned in the Actions UI: it has never been on the default branch
and has never run, so there is no history attached to the old filename.
* docs(oci): the API-key path needs the config file, not just the PEM
oci_config_file names an ordinary OCI SDK config and nothing creates it, so a node
holding only the key forms and then fails every allocation with no stated cause.
* fix(ci): the runner context is not available in a job-level env block
A workflow referencing it there is rejected before any job starts, which is why
the rename produced a failed run attributed to the push. The reference had never
been validated: a dispatch-only workflow that is never dispatched is never parsed.
* fix(deps): bump containerd to v2.4.1 for GO-2026-6597
The image-pull DoS via crafted OCI index graph amplification is reachable from the ECS agent's containerd puller, so the vulnerability scan fails preflight on v2.4.0. A patch release with no API change.
* fix(e2e): probe guest public addresses from outside the cloud's own network
A VCN does not reflect its own public addresses back into itself, so a node asking for one gets nothing however well the datapath works. The workbook driver runs on the first node, so every assertion that curls or SSHes a guest's public address failed on OCI against a cluster that was serving those addresses correctly to the internet.
The driver now takes E2E_PUBLIC_PROXY, a SOCKS5 proxy used for public addresses only, and validate-topology.sh supplies one with a remote dynamic forward on the connection that launches the driver. Egress is then the host running the validation, which is outside the VCN and is where a customer sits. Unset leaves every other platform on a direct probe, which is already correct there.
Also adds --version TAG for a repeatable install, separates the workflow's auth choice from its external-pool choice so an API key can run with the allocator live, and captures allocator diagnostics before the teardown destroys them.
* ci: read the Go version from go.mod in the release workflows
Both pinned 1.27.0 by hand while every other workflow reads the file, so the toolchain they build with drifts from the one the module declares the day go.mod moves. release.yml was already inconsistent with itself: its second setup-go step reads spinifex/go.mod.
* feat: publish the OCI topology results as tables on the run page
A green tick on one job with 4,000 lines of log behind it is confidence for
whoever wrote the harness and nothing to anybody else. Someone assessing whether
Spinifex works on their cloud should be able to see that it formed and served
guests on a single VM, on three VMs and on bare metal without opening a log.
record now takes <gate> <status> [detail] and writes results.tsv beside the
human block, so a gate's status is a field the script sets rather than a regex
over prose that reads fine and matches wrong. record_workbooks turns the
driver's own go-test-shaped lines into workbooks.tsv, and runs on the failing
branch too: which four passed is most of what the fifth failing means.
e2e-oci.sh --summary-only renders both tables to stdout and to
GITHUB_STEP_SUMMARY, and the workflow calls it under always() before the sweep,
because the sweep destroys the state directories the tables are read from. A
skipped gate gets its own mark, and a topology that never ran says so: the one
way a results table can do real harm is a reader taking an absent row for a
passing one.
* docs(oci): cut the deploy-by-hand appendix, and validate the config for real
The appendix was 444 of 1014 lines restating an install the Terraform already
does, which made the document read as though deploying by hand were a normal
route. It is not: the configuration builds the VCN, the volumes and both VNICs
per node and renders the cloud-init, in the same order every time. What is
genuinely OCI-specific is four things, each of which already has a section, so
the replacement names those and sends everyone else to the single-node and
multi-node install guides.
Verify end to end told the reader to ping a guest's public address from the
host, which cannot work: a VCN does not route its own public addresses back into
itself. That is the finding this branch is built on and the document was
asserting the opposite. It now says where to probe from and why, and carries the
allocator's `resolved the external VNIC` check, which existed only in the cut
appendix and is the one line proving the credentials and compartment are right.
`spx config validate` is not a command -- spx has admin, get, service, top and
version -- so that line could only ever fail and its output and status were both
discarded. It is replaced by parsing the TOML, which is what was wanted: the
remap and the pool append are textual edits, and a broken result would otherwise
surface as every service failing to start with nothing saying why. The remap now
checks its own substitution count for the same reason, since re.sub with no
match rewrites the file unchanged and exits 0.
Publication of the run-page tables moves to one owner. GITHUB_STEP_SUMMARY is
per step, so appending from the run as well as the publish step would have put
two copies of every table on the run page.
* fix(oci): check the credential hook before building anything
The hook is a path from a CI variable, and whether it exists on the runner is
knowable before Terraform runs. Checked where it was used, a wrong path cost the
apply, the install and the formation first -- forty minutes and a bare-metal bill
to be told a file is missing. Nothing about the check depends on the apply, so it
moves to the top, where --dry-run reaches it too.
* fix(oci): the summary must say what was built and what was not tested
The first CI run headed its tables "Published release `latest channel`" over a
build of a feature branch, which is the one error in a results table that would
mislead a reader who trusted it. The header now states a tree build as a tree
build, and names how the allocator authenticated.
A run with no external pool says so in the header rather than leaving it to a
SKIPPED row further down. Public-address allocation is the claim a green table is
most likely to be read as making, and it is exactly the claim such a run cannot
support. Both flags are passed to the publish step too, so the summary rendered
after a failure describes the same run as the one that failed.
Raise the dev prerelease prune from five to twenty. A pinned dev tag is how an
external evaluator installs a specific build, and five is about a week of them --
short enough that the tag someone was handed stops resolving while they are still
using it.
* docs(oci): the workbook table is a snapshot, and the nightly is the current answer
It is dated against a build five releases of work ago, and the nightly now runs
the same workbooks on all three topologies and publishes its own table. Say which
one to read rather than letting a stale table be taken as current.
* ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file
A workflow's displayed name is captured when GitHub first registers it and is not
refreshed afterwards. Both OCI entries were registered from a run that could not
be parsed -- a job-level env block referencing runner.temp -- so both list as
their path, which reads as a missing name: key and is not one. The 2026-10-02 run
was green on all three topologies and the entry still read as a path, with
updated_at unchanged from the broken first run, so fixing the file is necessary
and not sufficient.
A new path is a new registration, and its first run reads a valid file. Dropping
-nightly is also more honest: there is no schedule: here, because GitHub
registers one only from the default branch, so a cron could never run dev and a
feature branch as separate nightlies.
* Revert "ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file"
The rename makes the workflow undispatchable. `gh workflow run` resolves a
filename against the default branch, and the raw dispatch endpoint returns 404
for a path GitHub has not registered -- checked twice, several minutes apart, so
it is not an indexing delay. The old name works only because it is already
registered, and an unrunnable workflow is a worse problem than a mislabelled one.
The sidebar label needs a fresh registration, which a rename cannot provide from
a feature branch. It corrects itself when the file reaches main, where GitHub
re-reads it; the alternative is deleting the dead registrations' runs, which is an
operator's call.
* fix(ci): never publish the credential hook's output as an artifact
The hook is the operator's own executable, and the one thing known about it is
that it handles a private key. Its stdout and stderr are captured to
credential-hook.log, which the artifact's *.log glob was collecting into a
downloadable archive with a fourteen-day retention.
Excluded rather than trusted to have been quiet. A hook that prints its key once
while someone is debugging it should not thereby publish it.
* fix(oci): the hairpin probe measured nothing on one node, and claimed too much
With a single host, HOSTS[0] and HOSTS[-1] are the same address, so the node
connected to itself -- which always succeeds. It reported "present" on vm-single
and bm for that reason and not as a measurement. It now needs two nodes and says
so otherwise.
The claim it carried was also wrong. "A VCN does not reflect its own public
addresses back into itself" was read off a workstation run whose
node_client_cidr_allow_list was narrowed to one office address, which makes a
node's own public address a non-permitted source: the refusal that follows is
identical to an absent hairpin. With the committed 0.0.0.0/0 default the probe
reports the hairpin present between two nodes, so the original reading was
confounded and is retracted.
None of this changes the remedy. A customer reaches a guest from outside the VCN,
which is where the proxy's egress is, and probing from there is correct whichever
of the two causes is in play -- 15 of 15 workbooks pass through it on all three
topologies. What changes is what the documentation may assert: the guide now names
both causes as indistinguishable rather than stating one as fact.
* docs(oci): a reader could not run the first command, because nothing told them to clone
The guide's first code block was `cd scripts/terraform/oci-spx`, into a directory
no step had obtained. It also never mentioned the SSH keypair the driver refuses to
start without, nor that the configuration is not on main or in v1.21.0 yet, so all
three had to be inferred from the deploy failing.
Restructured so the deploy comes first and reads as four steps and one command:
get the configuration, choose one node or three, write the inputs, run it. The
credential decision stays before step 4, because it changes an argument there and a
node without one forms, looks healthy, and then refuses every launch that wants a
public address. Everything else -- why OCI, sizing, the architecture, the IAM
policy, quotas, the oci CLI, what Terraform builds, every variable -- moves under
Reference, unchanged in substance.
Adds what a successful run prints, naming `allocator ready` as the line to look for:
every line above it appears on a node that cannot allocate, and that is the one
failure the run does not announce.
Also drops an empty verdict.txt committed from a local run. The directory the guide
now tells customers to work in should not ship one run's artifacts.
Node installation, dev-environment lifecycle, guest image builds, and verification tooling for Spinifex.
Install and node setup
Script
What it does
setup.sh
The production binary installer, served from install.mulgadc.com. Installs deps, users, sudoers, files, systemd units, logrotate and udev rules. SETUP_STAGES runs a subset; ISO_BUILD=1 runs it inside the ISO builder's chroot.
setup-ovn.sh
Bootstraps a compute node for OVN VPC networking: installs OVN/OVS, creates br-int, configures the WAN bridge (bridged, DHCP, or routed-NAT uplink), sets the chassis identity, and applies overlay sysctls. --management also starts the OVN central services.
install-node.sh
Forms a multi-node cluster across hosts that are already running Spinifex: rebuilds the OVN database in clustered form, runs spx admin init on the first host and spx admin join on the rest, then verifies. The scripted form of install-multi-node. Destroys the joining nodes' logical network state and crypto — see its --help.
dev-install.sh
Full local dev setup through the production installer: builds from source, packs a tarball, runs setup.sh, initialises the cluster, and starts services under systemd.
clone-deps.sh
Clones or updates the viperblock and predastore checkouts alongside spinifex for cross-repo development.
Teardown
Both ship to /usr/local/share/spinifex/ on every node, so they are available on a host installed with curl … | bash and not only from a checkout. They differ in what they take away:
Script
What it does
node-reset.sh
Clears state and leaves the node installed: stops services, destroys guests, wipes /etc/spinifex, /var/lib/spinifex, /var/log/spinifex and the OVN databases, and removes the OVS bridges. The node is then ready for a fresh spx admin init. --keep-data preserves /var/lib/spinifex; SPX_RESET_CONFIRM=reset runs it unattended.
uninstall-spx.sh
Removes the software — the inverse of setup.sh. Delegates the state teardown above to node-reset.sh rather than reimplementing it, then removes the binary, the nbdkit plugin, the systemd units and slices, the service users, the sudoers grant, the sysctl/logrotate/chrony/udev drop-ins, the systemd-networkd units, the host firewall policy and its loaded nft table, and the Spinifex CA from the host trust store. Verifies afterwards and exits non-zero naming anything that survived.
uninstall-spx.sh keeps /var/lib/spinifex and every apt package by default — uninstalling software should not destroy data, and a curl … | bash install lands on hosts that were already using openvswitch, libvirt and chrony. --purge-data and --purge-deps opt into each, and both confirm separately even under --yes (SPX_PURGE_CONFIRM=destroy for unattended data removal). --dry-run prints the whole plan and touches nothing.
It refuses on a multi-node cluster, and --yes does not override that. Uninstalling one node of a running cluster strands its guests and leaves the survivors reconciling against a chassis that will never answer. Run spx admin cluster shutdown first, then the uninstaller on each node.
Dev environment
Script
What it does
reset-dev-env.sh
Single-node reset: stops services, wipes /etc/, /var/lib/, /var/log/ and /run/spinifex, rebuilds, reinstalls via dev-install.sh, and relaunches a smoke-test instance. Preserves the existing node/network topology from spinifex.toml.
dev-env/*.conf
Topology profiles for reset-dev-env.sh --profile=NAME, declaring which bridge carries the wan, lan and vpc planes: single-nic (all collapsed onto wan), two-nic (vpc folds onto lan), three-plane (a NIC each).
Verification and benchmarks
Script
What it does
smoke-test.sh
End-to-end sanity check on a running node: imports an SSH keypair and the Ubuntu AMI, then launches an instance. --create-vpc builds the VPC, gateway, subnet and routes first and tears them down after; --nodes N launches through a spread placement group so each instance lands on a different physical server. TEST_GPU=1 adds GPU passthrough checks.
verify-network.sh
Asserts a node's datapath matches its declared planes — EIP reachability, which plane carries Geneve, and which interfaces services bind. Catches what smoke-test.sh cannot, since none of its traffic leaves the box.
test-ddil-phase1.sh
systemd-level smoke test for daemon-local-autonomy tier 1. Destructive on the local node: stops and starts spinifex-nats and restarts spinifex-daemon, restoring both via a cleanup trap.
check-coverage.sh
Fails the build if total Go coverage falls below the minimum threshold, excluding packages exempt from the requirement.
diff-coverage.sh
Coverage check scoped to changed lines only, against an auto-detected base ref (HEAD~1 on main, origin/main on dev, otherwise origin/dev).
sync-aws-models.sh
Replaces the embedded AWS Smithy models under internal/awsmodel/models/ with the ten service files from aws/api-models-aws at the given commit, and records that commit in internal/awsmodel/model_source.go. Takes a full 40-character SHA.
run-bench.sh
Launches a benchmark against running Ubuntu instances, auto-detecting the AMI for the host architecture. Tracks nbdkit and perf alongside the run.
disk-performance.sh
Runs a fio random 70/30 read-write benchmark swept across 4k/16k/128k/1M. Writes JSON results to /tmp/spinifex-disk-bench; override with BENCH_DIR, OUT_DIR, SIZE, JOBS, BLOCK_SIZES.
network-performance.sh
iperf throughput from several clients to one server, driven over SSH, writing a summary.txt of Gbit/s per client. --server-ip sets the address clients dial, so pointing it at instance private addresses measures the VPC overlay rather than the external pool.
workload-performance.sh
In-guest memory, disk, CPU and network workload test. Installs Go and runs the Badger test suite with jemalloc disabled and reduced parallelism.
Guest images
Script
What it does
build-system-image.sh
Builds a minimal system AMI (Alpine or Ubuntu) from a manifest. All customisation happens inside the libguestfs appliance, so the build touches no host block device, no host mount, and needs no sudo. --import also registers it as an AMI.
publish-system-image.sh
Converts a built raw image to compressed qcow2, writes the .sha256 sidecar the catalog verifier expects, and uploads both to the R2 bucket behind iso.mulgadc.com/system-ami/.
build-microvm-image.sh
Builds the minimal Alpine kernel + initramfs used by the QEMU microvm machine type and the lb-agent, carrying haproxy (ALB) and nginx-stream (NLB) data planes.
mkosi-build.sh
Runs an mkosi system-image build inside the pinned builder container, so a runner needs only Docker and the toolchain cannot drift. --image names an output and expands to the right profile list.
mkosi-publish.sh
Publishes an mkosi-built image to R2 from inside the same pinned container — qcow2 conversion, .sha256 sidecar, and upload.
mkosi-common.sh
Shared plumbing sourced by both mkosi scripts. Builds the toolchain image once, with the builder account matched to the invoking user so bind-mounted output is not written back as root.
mkosi-builder.Dockerfile
The pinned image-build toolchain. Debian-hosted regardless of target distro, since mkosi bootstraps the target from its own archive.
images/ — per-image build inputs
Each subdirectory holds a manifest.conf, init scripts, and a setup.sh that build-system-image.sh runs inside the libguestfs appliance after packages and files are placed. *_test.sh files are shell unit tests for the helper of the same name.
Path
What it builds
ecs-agent/
The Alpine spinifex-ecs-node container-instance AMI (containerd + ecs-agent). setup.sh sets OpenRC exec bits, wires the CNI bin path, and applies the serial-console tweaks.
ecs-agent-gpu/
The Ubuntu ECS GPU container-instance AMI. Same CNI and containerd wiring adjusted for Ubuntu's package layout, plus the headless NVIDIA driver and nvidia-container-toolkit.
eks-node/
The unified Alpine EKS node AMI (K3s server or agent, selected at first boot). setup.sh installs the pinned, SHA-verified K3s binary and the role init scripts.
eks-node-gpu/
The Ubuntu EKS GPU worker AMI. Agent role only — GPU nodes never run the control plane — plus the NVIDIA GPU stack.
rds-postgres/
The spinifex-rds-postgres AMI. setup.sh sets exec bits on the OpenRC services and creates the root-only directory cloud-init drops the agent env file and gateway CA into.
rds-mariadb/
The spinifex-rds-mariadb AMI. Same layout as rds-postgres/, plus the /etc/my.cnf.d include drop-in and the conf.d/mariadb dependency that keeps the packaged service off an unbootstrapped datadir.
ubuntu-gpu-{amd,nvidia}-setup.sh
Chroot setup for GPU-capable Ubuntu guest images: ROCm CLI tools and AMD firmware, or the DKMS-built headless NVIDIA driver. Guest-side only — host VFIO setup is spx admin gpu setup.
images/eks-node/ — in-guest helpers
Installed into the EKS node AMI and run at boot or on a timer. All host communication goes through the AWS gateway over SigV4; the VM never speaks NATS directly.
Script
What it does
eks-node-role.sh
First-boot role selector. Reads SPINIFEX_K3S_ROLE from the cloud-init env file and enables the matching services — nothing is enabled at bake time.
k3s-prestart.sh
Server prestart: refuses to start on a restore-block marker, derives config.yaml from its skeleton, stages the konnectivity-agent and CoreDNS manifests, and waits for the token-webhook kubeconfig.
k3s-agent-prestart.sh
Agent prestart shared by the OpenRC and systemd units. Requires K3S_URL and K3S_TOKEN, then resolves the kubelet provider-id from IMDSv2 for EKS parity.
k3s-first-boot.sh
Runs once after the server is healthy: reads the node-token and admin kubeconfig, rewrites the kubeconfig server address to the cluster NLB endpoint, and publishes both to the host.
konnectivity-server-prestart.sh
Waits for the K3s CA and admin kubeconfig, mints the konnectivity serving cert, and writes the apiserver replica count for the caller to source.
mulga-eks-addon-sync.sh
Pulls staged managed-addon descriptors through the gateway, renders each bundle into the K3s auto-deploy dir, and reports per-addon delivery status back.
mulga-eks-etcd-snapshot.sh
Snapshots K3s embedded etcd to predastore. Installed into two crond periodic dirs and picks its tier — frequent or daily — from the dir it was invoked from.
mulga-eks-k3s-recovery.sh
Applies a control-plane etcd-quorum recovery directive before k3s starts. Pull-based and a no-op unless the reconciler escalated a wedged control plane.
mulga-eks-state-report.sh
Publishes server health (healthz, node count) to the cluster reconciler. The apiserver binds the VPC node-ip, which the host daemon cannot probe directly.
mulga-eks-provider-id.sh
Writes the K3s config drop-in supplying the metadata the EBS CSI driver needs from IMDS: provider-id plus the region, zone and instance-type labels.
mulga-ebs-byid.sh
Mints the EBS-style /dev/disk/by-id symlink the upstream EBS CSI node plugin resolves, from the virtio-blk serial. Alpine's busybox mdev has no eudev to do it.
mulga-mgmt-net.sh
Boot oneshot bringing a multi-NIC system VM's interfaces up by MAC from the QEMU fw_cfg netcfg blob, before cloud-init's network stage. cloud-init cannot reliably pick the right NIC out of two.
mulga-vpc-mtu.sh
Boot oneshot pinning the VPC data NIC MTU so flannel sizes its VXLAN overlay to fit the OVN geneve underlay. Otherwise large frames are silently dropped while handshakes pass.
Launch helpers
Script
What it does
launch-gpu.sh
Launches a GPU instance from the AMD or NVIDIA GPU AMI, selecting the AMI from the instance type's GPU brand. --amd / --nvidia override the detection.
ecs-node-bringup.sh
Launches an ECS container-instance VM. Renders the cloud-init user-data seeding the agent's control-plane config, IAM credentials and gateway CA — there is no ECS-specific launch API, by design.
demo/ — GPU demo
Script
What it does
deploy.sh
Deploys and starts the four-instance GPU demo (dashboard, YOLO, Ollama, qwen3-vl) from the *_IP and VIDEO_PATH env vars.
dashboard_server.py
Serves the demo web UI and proxies the streams from the GPU instances.
yolo_stream.py
Runs YOLO detection over a video file, serving an MJPEG stream of annotated frames and an SSE stream of qwen3-vl scene descriptions.
Ben Duncan
c6841ce1f0