* fix(oci): make a bare-metal node form and mount its data volume
Bare metal presents VNICs differently from a VM: the first VNIC on a physical
port is the port, and a second one sharing that port is a tagged overlay with no
Linux device at all. The external VNIC was landing in that second slot, so no
interface ever carried it and the WAN bridge could not be built. Attach it at
nic_index 1 on a BM shape, which makes it the first VNIC on the second port and
therefore untagged and present at boot, exactly as it is on a VM.
Two consequences of that followed. The bridge script picked the first non-primary
VNIC from instance metadata, which on bare metal can be one with no interface, so
it now selects the VNIC the kernel actually has a device for and waits for that.
And OCI makes a bare-metal volume attachment multipath, where the Oracle agent
refuses to log in unless multipathd is running -- probing the unit name Debian
retired, so on Ubuntu 26.04 it never sees it. Alias the retired name and assert
the alias on every poll rather than once, because the first run had it created and
then vanish.
Finally, the three cloud-init steps become ordered systemd units. runcmd
continues past a failed step, so a node whose volume never appeared still opened
its service ports with nothing behind them; Requires= makes that impossible and
lets a boot that raced the iSCSI login retry on the next one.
* feat(oci): authenticate the address allocator as the instance principal
The OCI allocator could only authenticate from an API-key config file, so every
node needed a private key copied to it and a config file hand-written beside it.
A node without them forms, passes every health check, and then refuses every
launch that wants a public address with InsufficientAddressCapacity -- the cause
appears only in the journal.
NewInstancePrincipalClient already existed and was called from nowhere. Wire it
in behind a new oci_auth key on the pool, so a node can authenticate with the
certificate its own metadata service serves and hold no key material at all.
Nothing to copy, nothing to rotate, and nothing in Terraform state.
The reason recorded for preferring the config file was that it reads no instance
metadata, because Spinifex's own IMDS endpoints claim 169.254.169.254 and take
the cloud's metadata service with them. That no longer distinguishes the two:
oci_vnic_iface is what the guide recommends and it resolves the VNIC through a
metadata lookup already, so such a node depends on the IMDS remap either way.
The two constructors' comments also contradicted each other on which was
preferred; both now say what they cost.
config_file stays the default, because instance principal needs a dynamic group
and a policy only a tenancy admin can write. The Terraform creates both behind
enable_instance_principal, off by default for the same reason the module creates
no other tenancy-root resource, and the policy grants three verbs on vnics,
private-ips and public-ips in one compartment rather than anything broader.
Naming the method explicitly rather than inferring it is deliberate: a node meant
to use its own identity now fails at config load instead of falling back to a key
file nobody installed. Pairing oci_auth="instance_principal" with a config file
is refused for the same reason, since the certificate is already the credential
and the file would be the half silently ignored.
Terraform also stages the pool fragment on each node, which under instance
principal contains no secret at all.
* feat(oci): validate all three topologies from one script, and document both auth routes
validate-topology.sh builds one topology from nothing, installs the published
release, forms the cluster, configures the pool and the IMDS remap, runs a
workbook against it, proves the workbook actually serves traffic, and destroys
everything. Three topologies because each breaks differently: bare metal presents
VNICs unlike a VM, a single node has no Geneve underlay to get wrong, and only a
cluster exercises RAFT, the gateway chassis and cross-node allocation.
--topology has no default, deliberately. A command aimed at the wrong topology is
the easiest expensive mistake available here, and each keeps its own Terraform
state so two can be built from one checkout without either destroying the other's
instances.
The teardown decides the verdict rather than tidying up after it. A topology or a
workbook that cannot be destroyed is half proved, so destroy runs from an EXIT
trap and a teardown failure fails the run. It also checks the two things a green
apply does not: that vpcd logged resolving the external VNIC, which is the only
evidence the credentials work and br-wan matched a real VNIC, and that the load
balancer returns 200 -- a workbook can create cleanly and carry no traffic.
The guide gains the instance-principal route beside the key file, and loses its
claim that bare metal changes nothing else in the document. That claim was wrong
in exactly two places, and both are now handled rather than left to the reader:
the external VNIC's nic_index and the Oracle agent's multipath unit name. It now
also says plainly that the external-address path on bare metal is unproven.
* test(oci): let a topology be validated without an OCI credential
The external pool is the one phase that needs a key or a dynamic group, and
the VM topologies have to be regression-tested regardless. --no-external-pool
forms and verifies without it, and implies --skip-workload so the run cannot
claim a guest path it never exercised.
* test(oci): one entry point that proves a ref works on OCI, and cleans up after itself
e2e-oci.sh drives every topology from environment variables alone, so a
workstation, a Worker dispatch and e2e-oci-nightly.yml all take the same path
rather than drifting into three answers about whether OCI works.
Three things it deliberately does not do. It does not test the published
release: --distro installs the tarball built from the ref under test, because a
nightly that cannot fail on a bad commit is decoration. It does not carry its
own workbook assertions, running tests/e2e/run-tofu-examples-e2e.sh over the
published workbooks instead, so a workbook that passes on a hypervisor and fails
here is a difference in OCI. And it cannot quietly leave cloud resources
running: --sweep-only destroys whatever an earlier run's state still holds, and
it runs before the topologies, after them, and from an always() step.
The credential now reaches Terraform as a value through OCI_PRIVATE_KEY, so no
runner writes key material to disk, and oci_env.py refuses --shell on that route
because in CI stdout is the job log.
Four harness defects the first real runs exposed, each of which had been giving
a wrong answer:
- oci_env.py announced itself on stdout, corrupting every captured
terraform output. That is also why a single node came back as two hosts.
- terraform output was passed -var, which it rejects, and a silent fallback
then read the default state file instead of the topology's.
- grep -c Ready matches NotReady, so a single-node run whose only node was
NotReady on 0.0.0.0 reported PASS. The status column is parsed now, and
membership is waited for rather than sampled once.
- install-node.sh refuses fewer than two hosts, so a single node forms with
spx admin init the way the guide documents.
Also waits for the dpkg lock: cloud-init being finished does not mean apt is.
* fix(oci): resolve mulga's path explicitly, and refuse an unpushed ref
Two assumptions in the new harness that a run would not have caught quickly.
Deriving mulga's root from spinifex's is right for a submodule checkout and
wrong for CI, where the two are siblings under the workspace, so MULGA_ROOT is
explicit and the workflow sets it.
build-install-artifacts.sh clones each repo from GitHub rather than using the
checkout beside it, and falls back to dev when the ref is not there. A nightly
on an unpushed branch would then pass or fail on dev's code while reporting the
branch. Check the ref is on origin first, and skip the Go suite build: the
workbook driver is a shell script, so those binaries prove nothing here.
* fix(oci): do not read $? after a command when set -e is on
The workbook phase read the driver's exit status on the line after the ssh that
produced it, which under set -e is never reached: a failing run aborted before
printing the driver's own per-workbook diagnostics, losing them exactly when
they are the thing you need. An `if !` keeps them, and names which workbooks
passed and which did not.
* feat: let an OCI e2e dispatch form without the external allocator
A tenancy without the nodes' dynamic group cannot authenticate the
allocator, so every topology failed at it rather than reporting what it
had proved. The no_external_pool input forms with no allocator and
records the allocator and the workbooks as SKIPPED, never PASS.
Also rebuilds the dry-run plan message imperatively: it still referenced
a variable renamed to WORKBOOKS, and under set -u the substitution died
and the text silently vanished from the line.
* fix: name the offending input when an OCI credential is unusable
A credential from the environment reached the SDK unvalidated, so a
secret with a trailing newline failed as an uncaught InvalidConfig
traceback naming an SDK config field that nobody set by that name.
Strip the scalar values, since a secret store written with echo carries
a newline, and translate both InvalidConfig and InvalidPrivateKey into
the variable or config key to go and fix.
* fix: make the OCI nightly able to build, and stop it leaking a dispatch input into the shell
Four defects, found reviewing the workflow rather than running it.
The builder clones all five sub-repos from GitHub over https and they are
all private, so a hosted runner had no credential and the tree build
could never have worked. The app token now covers six repositories and
git carries it for our organisation only.
A dispatch input was interpolated into a run block, where a quote ends
the string and the remainder executes. It goes through the environment
now, which is the only place a caller-supplied value belongs.
The artifact upload took the whole directory, which meant the distro
tarball and the test archive, and would have taken terraform.tfstate had
the state directories sat under it. It is a named list now, and it also
collects the per-topology logs that were never uploaded at all.
* fix: make the OCI nightly's runner a setting, and stop widening the app token
The five sub-repos are public, so build-install-artifacts.sh clones them
without a credential and the token has no reason to reach them. Only
mulga is private, which is what it was scoped to before.
The runner is now inputs.runner, then the OCI_E2E_RUNNER variable, then
hosted. A per-provider self-hosted agent is a reasonable thing to want --
it lifts the six-hour cap a hosted job carries and keeps a long cloud
wait off the hypervisor labels -- and it should not need a workflow edit.
Removing the SSH key is load bearing on that path: a self-hosted runner
keeps $HOME between jobs, so the key would sit there between nightlies.
* fix: install the OCI SDK the harness needs, and keep state out of the checkout
Nothing installed the oci Python module, on any runner: validate-topology.sh
called bare python3 and oci_env.py imports oci, so every topology would have
failed at the first terraform call. The harness now uses a system python3 that
has it and otherwise builds its own virtualenv from requirements.txt, which is
the same behaviour on a workstation, a hosted runner and a self-hosted one.
The state directories move under OCI_STATE_ROOT, defaulting to the script
directory. A persistent runner needs them outside the checkout, because
actions/checkout wipes untracked files there and would remove the state of a
killed run before the sweep could read it, leaving OCI instances running that
nothing is able to find.
* fix: drop the OCI SDK dependency, so the harness runs on any host
python3-venv is not installed on banksia, and the harness needed it: the
environment helper re-executed itself from a virtualenv to import the OCI
SDK. It used the SDK for two things -- parsing an INI file, which
configparser already does, and discovering the tenancy home region, which
no Terraform file consumed. Neither justifies a dependency on every host
that runs this, so the helper is standard library only now.
The credential is checked for shape instead, locally and before any OCI
call, which is a better failure than a 401 forty seconds into a run.
Two things the removal uncovered. The home region is a fixed property of
a tenancy rather than something to rediscover every run, so it is
OCI_HOME_REGION. And OCI serves IAM writes from the home region while the
dynamic group and policy were using the default provider for the build
region, which would have failed when the tenancy applied them: they go
through an aliased provider that defaults to the build region, so a
tenancy whose home region is where it builds sees no change.
Node installation, dev-environment lifecycle, guest image builds, and verification tooling for Spinifex.
Install and node setup
Script
What it does
setup.sh
The production binary installer, served from install.mulgadc.com. Installs deps, users, sudoers, files, systemd units, logrotate and udev rules. SETUP_STAGES runs a subset; ISO_BUILD=1 runs it inside the ISO builder's chroot.
setup-ovn.sh
Bootstraps a compute node for OVN VPC networking: installs OVN/OVS, creates br-int, configures the WAN bridge (bridged, DHCP, or routed-NAT uplink), sets the chassis identity, and applies overlay sysctls. --management also starts the OVN central services.
install-node.sh
Forms a multi-node cluster across hosts that are already running Spinifex: rebuilds the OVN database in clustered form, runs spx admin init on the first host and spx admin join on the rest, then verifies. The scripted form of install-multi-node. Destroys the joining nodes' logical network state and crypto — see its --help.
dev-install.sh
Full local dev setup through the production installer: builds from source, packs a tarball, runs setup.sh, initialises the cluster, and starts services under systemd.
clone-deps.sh
Clones or updates the viperblock and predastore checkouts alongside spinifex for cross-repo development.
Teardown
Both ship to /usr/local/share/spinifex/ on every node, so they are available on a host installed with curl … | bash and not only from a checkout. They differ in what they take away:
Script
What it does
node-reset.sh
Clears state and leaves the node installed: stops services, destroys guests, wipes /etc/spinifex, /var/lib/spinifex, /var/log/spinifex and the OVN databases, and removes the OVS bridges. The node is then ready for a fresh spx admin init. --keep-data preserves /var/lib/spinifex; SPX_RESET_CONFIRM=reset runs it unattended.
uninstall-spx.sh
Removes the software — the inverse of setup.sh. Delegates the state teardown above to node-reset.sh rather than reimplementing it, then removes the binary, the nbdkit plugin, the systemd units and slices, the service users, the sudoers grant, the sysctl/logrotate/chrony/udev drop-ins, the systemd-networkd units, the host firewall policy and its loaded nft table, and the Spinifex CA from the host trust store. Verifies afterwards and exits non-zero naming anything that survived.
uninstall-spx.sh keeps /var/lib/spinifex and every apt package by default — uninstalling software should not destroy data, and a curl … | bash install lands on hosts that were already using openvswitch, libvirt and chrony. --purge-data and --purge-deps opt into each, and both confirm separately even under --yes (SPX_PURGE_CONFIRM=destroy for unattended data removal). --dry-run prints the whole plan and touches nothing.
It refuses on a multi-node cluster, and --yes does not override that. Uninstalling one node of a running cluster strands its guests and leaves the survivors reconciling against a chassis that will never answer. Run spx admin cluster shutdown first, then the uninstaller on each node.
Dev environment
Script
What it does
reset-dev-env.sh
Single-node reset: stops services, wipes /etc/, /var/lib/, /var/log/ and /run/spinifex, rebuilds, reinstalls via dev-install.sh, and relaunches a smoke-test instance. Preserves the existing node/network topology from spinifex.toml.
dev-env/*.conf
Topology profiles for reset-dev-env.sh --profile=NAME, declaring which bridge carries the wan, lan and vpc planes: single-nic (all collapsed onto wan), two-nic (vpc folds onto lan), three-plane (a NIC each).
Verification and benchmarks
Script
What it does
smoke-test.sh
End-to-end sanity check on a running node: imports an SSH keypair and the Ubuntu AMI, then launches an instance. --create-vpc builds the VPC, gateway, subnet and routes first and tears them down after; --nodes N launches through a spread placement group so each instance lands on a different physical server. TEST_GPU=1 adds GPU passthrough checks.
verify-network.sh
Asserts a node's datapath matches its declared planes — EIP reachability, which plane carries Geneve, and which interfaces services bind. Catches what smoke-test.sh cannot, since none of its traffic leaves the box.
test-ddil-phase1.sh
systemd-level smoke test for daemon-local-autonomy tier 1. Destructive on the local node: stops and starts spinifex-nats and restarts spinifex-daemon, restoring both via a cleanup trap.
check-coverage.sh
Fails the build if total Go coverage falls below the minimum threshold, excluding packages exempt from the requirement.
diff-coverage.sh
Coverage check scoped to changed lines only, against an auto-detected base ref (HEAD~1 on main, origin/main on dev, otherwise origin/dev).
sync-aws-models.sh
Replaces the embedded AWS Smithy models under internal/awsmodel/models/ with the ten service files from aws/api-models-aws at the given commit, and records that commit in internal/awsmodel/model_source.go. Takes a full 40-character SHA.
run-bench.sh
Launches a benchmark against running Ubuntu instances, auto-detecting the AMI for the host architecture. Tracks nbdkit and perf alongside the run.
disk-performance.sh
Runs a fio random 70/30 read-write benchmark swept across 4k/16k/128k/1M. Writes JSON results to /tmp/spinifex-disk-bench; override with BENCH_DIR, OUT_DIR, SIZE, JOBS, BLOCK_SIZES.
network-performance.sh
iperf throughput from several clients to one server, driven over SSH, writing a summary.txt of Gbit/s per client. --server-ip sets the address clients dial, so pointing it at instance private addresses measures the VPC overlay rather than the external pool.
workload-performance.sh
In-guest memory, disk, CPU and network workload test. Installs Go and runs the Badger test suite with jemalloc disabled and reduced parallelism.
Guest images
Script
What it does
build-system-image.sh
Builds a minimal system AMI (Alpine or Ubuntu) from a manifest. All customisation happens inside the libguestfs appliance, so the build touches no host block device, no host mount, and needs no sudo. --import also registers it as an AMI.
publish-system-image.sh
Converts a built raw image to compressed qcow2, writes the .sha256 sidecar the catalog verifier expects, and uploads both to the R2 bucket behind iso.mulgadc.com/system-ami/.
build-microvm-image.sh
Builds the minimal Alpine kernel + initramfs used by the QEMU microvm machine type and the lb-agent, carrying haproxy (ALB) and nginx-stream (NLB) data planes.
mkosi-build.sh
Runs an mkosi system-image build inside the pinned builder container, so a runner needs only Docker and the toolchain cannot drift. --image names an output and expands to the right profile list.
mkosi-publish.sh
Publishes an mkosi-built image to R2 from inside the same pinned container — qcow2 conversion, .sha256 sidecar, and upload.
mkosi-common.sh
Shared plumbing sourced by both mkosi scripts. Builds the toolchain image once, with the builder account matched to the invoking user so bind-mounted output is not written back as root.
mkosi-builder.Dockerfile
The pinned image-build toolchain. Debian-hosted regardless of target distro, since mkosi bootstraps the target from its own archive.
images/ — per-image build inputs
Each subdirectory holds a manifest.conf, init scripts, and a setup.sh that build-system-image.sh runs inside the libguestfs appliance after packages and files are placed. *_test.sh files are shell unit tests for the helper of the same name.
Path
What it builds
ecs-agent/
The Alpine spinifex-ecs-node container-instance AMI (containerd + ecs-agent). setup.sh sets OpenRC exec bits, wires the CNI bin path, and applies the serial-console tweaks.
ecs-agent-gpu/
The Ubuntu ECS GPU container-instance AMI. Same CNI and containerd wiring adjusted for Ubuntu's package layout, plus the headless NVIDIA driver and nvidia-container-toolkit.
eks-node/
The unified Alpine EKS node AMI (K3s server or agent, selected at first boot). setup.sh installs the pinned, SHA-verified K3s binary and the role init scripts.
eks-node-gpu/
The Ubuntu EKS GPU worker AMI. Agent role only — GPU nodes never run the control plane — plus the NVIDIA GPU stack.
rds-postgres/
The spinifex-rds-postgres AMI. setup.sh sets exec bits on the OpenRC services and creates the root-only directory cloud-init drops the agent env file and gateway CA into.
rds-mariadb/
The spinifex-rds-mariadb AMI. Same layout as rds-postgres/, plus the /etc/my.cnf.d include drop-in and the conf.d/mariadb dependency that keeps the packaged service off an unbootstrapped datadir.
ubuntu-gpu-{amd,nvidia}-setup.sh
Chroot setup for GPU-capable Ubuntu guest images: ROCm CLI tools and AMD firmware, or the DKMS-built headless NVIDIA driver. Guest-side only — host VFIO setup is spx admin gpu setup.
images/eks-node/ — in-guest helpers
Installed into the EKS node AMI and run at boot or on a timer. All host communication goes through the AWS gateway over SigV4; the VM never speaks NATS directly.
Script
What it does
eks-node-role.sh
First-boot role selector. Reads SPINIFEX_K3S_ROLE from the cloud-init env file and enables the matching services — nothing is enabled at bake time.
k3s-prestart.sh
Server prestart: refuses to start on a restore-block marker, derives config.yaml from its skeleton, stages the konnectivity-agent and CoreDNS manifests, and waits for the token-webhook kubeconfig.
k3s-agent-prestart.sh
Agent prestart shared by the OpenRC and systemd units. Requires K3S_URL and K3S_TOKEN, then resolves the kubelet provider-id from IMDSv2 for EKS parity.
k3s-first-boot.sh
Runs once after the server is healthy: reads the node-token and admin kubeconfig, rewrites the kubeconfig server address to the cluster NLB endpoint, and publishes both to the host.
konnectivity-server-prestart.sh
Waits for the K3s CA and admin kubeconfig, mints the konnectivity serving cert, and writes the apiserver replica count for the caller to source.
mulga-eks-addon-sync.sh
Pulls staged managed-addon descriptors through the gateway, renders each bundle into the K3s auto-deploy dir, and reports per-addon delivery status back.
mulga-eks-etcd-snapshot.sh
Snapshots K3s embedded etcd to predastore. Installed into two crond periodic dirs and picks its tier — frequent or daily — from the dir it was invoked from.
mulga-eks-k3s-recovery.sh
Applies a control-plane etcd-quorum recovery directive before k3s starts. Pull-based and a no-op unless the reconciler escalated a wedged control plane.
mulga-eks-state-report.sh
Publishes server health (healthz, node count) to the cluster reconciler. The apiserver binds the VPC node-ip, which the host daemon cannot probe directly.
mulga-eks-provider-id.sh
Writes the K3s config drop-in supplying the metadata the EBS CSI driver needs from IMDS: provider-id plus the region, zone and instance-type labels.
mulga-ebs-byid.sh
Mints the EBS-style /dev/disk/by-id symlink the upstream EBS CSI node plugin resolves, from the virtio-blk serial. Alpine's busybox mdev has no eudev to do it.
mulga-mgmt-net.sh
Boot oneshot bringing a multi-NIC system VM's interfaces up by MAC from the QEMU fw_cfg netcfg blob, before cloud-init's network stage. cloud-init cannot reliably pick the right NIC out of two.
mulga-vpc-mtu.sh
Boot oneshot pinning the VPC data NIC MTU so flannel sizes its VXLAN overlay to fit the OVN geneve underlay. Otherwise large frames are silently dropped while handshakes pass.
Launch helpers
Script
What it does
launch-gpu.sh
Launches a GPU instance from the AMD or NVIDIA GPU AMI, selecting the AMI from the instance type's GPU brand. --amd / --nvidia override the detection.
ecs-node-bringup.sh
Launches an ECS container-instance VM. Renders the cloud-init user-data seeding the agent's control-plane config, IAM credentials and gateway CA — there is no ECS-specific launch API, by design.
demo/ — GPU demo
Script
What it does
deploy.sh
Deploys and starts the four-instance GPU demo (dashboard, YOLO, Ollama, qwen3-vl) from the *_IP and VIDEO_PATH env vars.
dashboard_server.py
Serves the demo web UI and proxies the streams from the GPU instances.
yolo_stream.py
Runs YOLO detection over a video file, serving an MJPEG stream of annotated frames and an SSE stream of qwen3-vl scene descriptions.
Ben Duncan
ade060f384