Ben Duncan
ade060f384
* fix(oci): make a bare-metal node form and mount its data volume
Bare metal presents VNICs differently from a VM: the first VNIC on a physical
port is the port, and a second one sharing that port is a tagged overlay with no
Linux device at all. The external VNIC was landing in that second slot, so no
interface ever carried it and the WAN bridge could not be built. Attach it at
nic_index 1 on a BM shape, which makes it the first VNIC on the second port and
therefore untagged and present at boot, exactly as it is on a VM.
Two consequences of that followed. The bridge script picked the first non-primary
VNIC from instance metadata, which on bare metal can be one with no interface, so
it now selects the VNIC the kernel actually has a device for and waits for that.
And OCI makes a bare-metal volume attachment multipath, where the Oracle agent
refuses to log in unless multipathd is running -- probing the unit name Debian
retired, so on Ubuntu 26.04 it never sees it. Alias the retired name and assert
the alias on every poll rather than once, because the first run had it created and
then vanish.
Finally, the three cloud-init steps become ordered systemd units. runcmd
continues past a failed step, so a node whose volume never appeared still opened
its service ports with nothing behind them; Requires= makes that impossible and
lets a boot that raced the iSCSI login retry on the next one.
* feat(oci): authenticate the address allocator as the instance principal
The OCI allocator could only authenticate from an API-key config file, so every
node needed a private key copied to it and a config file hand-written beside it.
A node without them forms, passes every health check, and then refuses every
launch that wants a public address with InsufficientAddressCapacity -- the cause
appears only in the journal.
NewInstancePrincipalClient already existed and was called from nowhere. Wire it
in behind a new oci_auth key on the pool, so a node can authenticate with the
certificate its own metadata service serves and hold no key material at all.
Nothing to copy, nothing to rotate, and nothing in Terraform state.
The reason recorded for preferring the config file was that it reads no instance
metadata, because Spinifex's own IMDS endpoints claim 169.254.169.254 and take
the cloud's metadata service with them. That no longer distinguishes the two:
oci_vnic_iface is what the guide recommends and it resolves the VNIC through a
metadata lookup already, so such a node depends on the IMDS remap either way.
The two constructors' comments also contradicted each other on which was
preferred; both now say what they cost.
config_file stays the default, because instance principal needs a dynamic group
and a policy only a tenancy admin can write. The Terraform creates both behind
enable_instance_principal, off by default for the same reason the module creates
no other tenancy-root resource, and the policy grants three verbs on vnics,
private-ips and public-ips in one compartment rather than anything broader.
Naming the method explicitly rather than inferring it is deliberate: a node meant
to use its own identity now fails at config load instead of falling back to a key
file nobody installed. Pairing oci_auth="instance_principal" with a config file
is refused for the same reason, since the certificate is already the credential
and the file would be the half silently ignored.
Terraform also stages the pool fragment on each node, which under instance
principal contains no secret at all.
* feat(oci): validate all three topologies from one script, and document both auth routes
validate-topology.sh builds one topology from nothing, installs the published
release, forms the cluster, configures the pool and the IMDS remap, runs a
workbook against it, proves the workbook actually serves traffic, and destroys
everything. Three topologies because each breaks differently: bare metal presents
VNICs unlike a VM, a single node has no Geneve underlay to get wrong, and only a
cluster exercises RAFT, the gateway chassis and cross-node allocation.
--topology has no default, deliberately. A command aimed at the wrong topology is
the easiest expensive mistake available here, and each keeps its own Terraform
state so two can be built from one checkout without either destroying the other's
instances.
The teardown decides the verdict rather than tidying up after it. A topology or a
workbook that cannot be destroyed is half proved, so destroy runs from an EXIT
trap and a teardown failure fails the run. It also checks the two things a green
apply does not: that vpcd logged resolving the external VNIC, which is the only
evidence the credentials work and br-wan matched a real VNIC, and that the load
balancer returns 200 -- a workbook can create cleanly and carry no traffic.
The guide gains the instance-principal route beside the key file, and loses its
claim that bare metal changes nothing else in the document. That claim was wrong
in exactly two places, and both are now handled rather than left to the reader:
the external VNIC's nic_index and the Oracle agent's multipath unit name. It now
also says plainly that the external-address path on bare metal is unproven.
* test(oci): let a topology be validated without an OCI credential
The external pool is the one phase that needs a key or a dynamic group, and
the VM topologies have to be regression-tested regardless. --no-external-pool
forms and verifies without it, and implies --skip-workload so the run cannot
claim a guest path it never exercised.
* test(oci): one entry point that proves a ref works on OCI, and cleans up after itself
e2e-oci.sh drives every topology from environment variables alone, so a
workstation, a Worker dispatch and e2e-oci-nightly.yml all take the same path
rather than drifting into three answers about whether OCI works.
Three things it deliberately does not do. It does not test the published
release: --distro installs the tarball built from the ref under test, because a
nightly that cannot fail on a bad commit is decoration. It does not carry its
own workbook assertions, running tests/e2e/run-tofu-examples-e2e.sh over the
published workbooks instead, so a workbook that passes on a hypervisor and fails
here is a difference in OCI. And it cannot quietly leave cloud resources
running: --sweep-only destroys whatever an earlier run's state still holds, and
it runs before the topologies, after them, and from an always() step.
The credential now reaches Terraform as a value through OCI_PRIVATE_KEY, so no
runner writes key material to disk, and oci_env.py refuses --shell on that route
because in CI stdout is the job log.
Four harness defects the first real runs exposed, each of which had been giving
a wrong answer:
- oci_env.py announced itself on stdout, corrupting every captured
terraform output. That is also why a single node came back as two hosts.
- terraform output was passed -var, which it rejects, and a silent fallback
then read the default state file instead of the topology's.
- grep -c Ready matches NotReady, so a single-node run whose only node was
NotReady on 0.0.0.0 reported PASS. The status column is parsed now, and
membership is waited for rather than sampled once.
- install-node.sh refuses fewer than two hosts, so a single node forms with
spx admin init the way the guide documents.
Also waits for the dpkg lock: cloud-init being finished does not mean apt is.
* fix(oci): resolve mulga's path explicitly, and refuse an unpushed ref
Two assumptions in the new harness that a run would not have caught quickly.
Deriving mulga's root from spinifex's is right for a submodule checkout and
wrong for CI, where the two are siblings under the workspace, so MULGA_ROOT is
explicit and the workflow sets it.
build-install-artifacts.sh clones each repo from GitHub rather than using the
checkout beside it, and falls back to dev when the ref is not there. A nightly
on an unpushed branch would then pass or fail on dev's code while reporting the
branch. Check the ref is on origin first, and skip the Go suite build: the
workbook driver is a shell script, so those binaries prove nothing here.
* fix(oci): do not read $? after a command when set -e is on
The workbook phase read the driver's exit status on the line after the ssh that
produced it, which under set -e is never reached: a failing run aborted before
printing the driver's own per-workbook diagnostics, losing them exactly when
they are the thing you need. An `if !` keeps them, and names which workbooks
passed and which did not.
* feat: let an OCI e2e dispatch form without the external allocator
A tenancy without the nodes' dynamic group cannot authenticate the
allocator, so every topology failed at it rather than reporting what it
had proved. The no_external_pool input forms with no allocator and
records the allocator and the workbooks as SKIPPED, never PASS.
Also rebuilds the dry-run plan message imperatively: it still referenced
a variable renamed to WORKBOOKS, and under set -u the substitution died
and the text silently vanished from the line.
* fix: name the offending input when an OCI credential is unusable
A credential from the environment reached the SDK unvalidated, so a
secret with a trailing newline failed as an uncaught InvalidConfig
traceback naming an SDK config field that nobody set by that name.
Strip the scalar values, since a secret store written with echo carries
a newline, and translate both InvalidConfig and InvalidPrivateKey into
the variable or config key to go and fix.
* fix: make the OCI nightly able to build, and stop it leaking a dispatch input into the shell
Four defects, found reviewing the workflow rather than running it.
The builder clones all five sub-repos from GitHub over https and they are
all private, so a hosted runner had no credential and the tree build
could never have worked. The app token now covers six repositories and
git carries it for our organisation only.
A dispatch input was interpolated into a run block, where a quote ends
the string and the remainder executes. It goes through the environment
now, which is the only place a caller-supplied value belongs.
The artifact upload took the whole directory, which meant the distro
tarball and the test archive, and would have taken terraform.tfstate had
the state directories sat under it. It is a named list now, and it also
collects the per-topology logs that were never uploaded at all.
* fix: make the OCI nightly's runner a setting, and stop widening the app token
The five sub-repos are public, so build-install-artifacts.sh clones them
without a credential and the token has no reason to reach them. Only
mulga is private, which is what it was scoped to before.
The runner is now inputs.runner, then the OCI_E2E_RUNNER variable, then
hosted. A per-provider self-hosted agent is a reasonable thing to want --
it lifts the six-hour cap a hosted job carries and keeps a long cloud
wait off the hypervisor labels -- and it should not need a workflow edit.
Removing the SSH key is load bearing on that path: a self-hosted runner
keeps $HOME between jobs, so the key would sit there between nightlies.
* fix: install the OCI SDK the harness needs, and keep state out of the checkout
Nothing installed the oci Python module, on any runner: validate-topology.sh
called bare python3 and oci_env.py imports oci, so every topology would have
failed at the first terraform call. The harness now uses a system python3 that
has it and otherwise builds its own virtualenv from requirements.txt, which is
the same behaviour on a workstation, a hosted runner and a self-hosted one.
The state directories move under OCI_STATE_ROOT, defaulting to the script
directory. A persistent runner needs them outside the checkout, because
actions/checkout wipes untracked files there and would remove the state of a
killed run before the sweep could read it, leaving OCI instances running that
nothing is able to find.
* fix: drop the OCI SDK dependency, so the harness runs on any host
python3-venv is not installed on banksia, and the harness needed it: the
environment helper re-executed itself from a virtualenv to import the OCI
SDK. It used the SDK for two things -- parsing an INI file, which
configparser already does, and discovering the tenancy home region, which
no Terraform file consumed. Neither justifies a dependency on every host
that runs this, so the helper is standard library only now.
The credential is checked for shape instead, locally and before any OCI
call, which is a better failure than a 401 forty seconds into a run.
Two things the removal uncovered. The home region is a fixed property of
a tenancy rather than something to rediscover every run, so it is
OCI_HOME_REGION. And OCI serves IAM writes from the home region while the
dynamic group and policy were using the default provider for the build
region, which would have failed when the tenancy applied them: they go
through an aliased provider that defaults to the build region, so a
tenancy whose home region is where it builds sees no change.
What is Spinifex? · Platform · Deploy · AWS Compatibility · Live Demo · Components · Architecture · Installation · Docs
Spinifex: the open, AWS-compatible cloud you run yourself
Spinifex, built by Mulga, recreates the AWS services your software already uses (EC2, EBS, S3, VPC, IAM) on infrastructure you control. Move workloads off the hyperscalers, run AI at a fraction of the cost, or take the cloud to the edge, without rewriting a line. The exact build you ship to AWS, Spinifex serves on your own hardware: same APIs, same Terraform, same SDKs.
What is Spinifex?
Spinifex is an open-source infrastructure platform that recreates the AWS service surface (EC2, EBS, S3, VPC, IAM) on bare-metal, edge, and on-premises environments. Run cloud-native software without a hyperscaler.
Built for teams that need:
- The AWS tooling you already use (AWS CLI, SDKs, Terraform), with nothing to rewrite
- Full control of the stack: your hardware, your network, your data, your keys
- A cloud that keeps running offline, through disconnection, with no external control plane
- An open core (AGPL-3.0), auditable and yours to keep, with no lock-in
You change where your software runs, not what it is.
The Platform
From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
Three Ways To Deploy
The same AWS-compatible platform runs in three shapes. Pick the one that matches your hardware.
Neocloud
Lift and shift onto a partner Neocloud without rewriting workloads. GPU capacity (H100 / H200 / B200), cheaper, available now. Change an endpoint, keep the software.
On-premise
Host Spinifex on your own hardware in your own data centre. You get a multi-node HA cluster integrated with your storage and networking, so the stack, the data, and the jurisdiction stay yours.
Edge
Cloud where the cloud can't reach. Air-gapped sites, vehicles, vessels, factories, and clinics. Compute next to the data, running through disconnection, on hardware you own. The same AWS APIs your software already uses.
AWS Compatibility
Spinifex speaks the AWS API surface. The AWS SDKs, AWS CLI, and Terraform all work unchanged, whether you deploy at the edge, on-premise, or on a partner Neocloud.
| Service | What it is | Status |
|---|---|---|
| EC2 | Compute | Available |
| EBS | Block Storage | Available |
| S3 | Object Storage | Available |
| VPC | Networking | Available |
| IAM | Identity & Auth | Available |
| ALB / NLB | Load Balancers | Available |
| EKS | Kubernetes | Available |
| ECR | Container Registry | Available |
| ECS | Container Service | Available |
| RDS | Databases | Available |
| Bedrock | AI Deployment | Q4 2026 |
Roadmap items ship under the same AWS API surface. Code written for AWS today keeps working the moment they land. Track what's shipped in the release notes.
Live Demo
Create a live sandbox in under a minute. The fastest way to experience the stack. Try Spinifex against one of your own workloads on infrastructure we run for you.
- Live in under a minute. No card, no install.
- Yours for 72 hours to test and experiment.
- Real AWS-compatible endpoint, access keys and console.
- EC2, EBS, S3, VPC, IAM, EKS and RDS, ready to call.
- Point your existing AWS CLI or Terraform straight at it.
Sign up at https://mulgadc.com/signup
Core Components
Spinifex (Compute and Platform Services – EC2, EKS, ECS, RDS)
Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing an EC2 compatible API. It manages lifecycle operations like start, stop, and terminate through QEMU's QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs with the AWS CLI, SDKs, or Terraform, without Kubernetes or a heavyweight orchestrator.
The managed services build on that same instance layer. An EKS node group, an ECS container instance, and an RDS database are Spinifex VMs on your own hardware, each driven through its own AWS API.
- EC2 – VM lifecycle on bare metal, cloud-init and IMDS metadata, key pairs, tags, spot instances, capacity reservations, and GPU passthrough
- EKS – Managed Kubernetes on k3s, with HA control planes, managed node groups, cluster add-ons, an OIDC provider, and IAM access entries
- ECS – Task and service scheduling across container instances, with awsvpc ENIs, capacity providers, rolling deployments, and GPU-aware placement
- ECR – Private container registry backed by Predastore, with content-addressed blobs deduplicated per account and lifecycle policies
- RDS – Managed PostgreSQL and MariaDB, with parameter groups, automated backups, and DNS endpoints served by Northstar
- ALB / NLB – Layer 7 load balancing on HAProxy and layer 4 on nginx
stream, with health checks and ACM-issued listener certificates - VPC and IAM – OVN-backed networking and security groups, with roles, policies, and SigV4 auth applied across every service
Viperblock (Block Storage – EBS Alternative)
Viperblock is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It's built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
- Fast, durable virtual disks
- Replication for resilience
- Exposed over NBD or embedded in VMs
- High performance WAL logs on local NVMe drives, reducing IO traffic to S3
- In-memory read/write block cache
Predastore (Object Storage – S3-Compatible)
Predastore is a fully S3-compatible object storage system. It supports the AWS S3 API, including Signature V4 authentication, multipart uploads, and Terraform provisioning. Predastore chunks data and distributes it across nodes using Reed-Solomon erasure coding, so it survives node loss and suits large-scale or low-bandwidth sites.
- S3-compatible API and auth
- Multipart uploads, streaming reads/writes
- Data redundancy with Reed-Solomon encoding
Northstar (Authoritative DNS)
Northstar is a lightweight authoritative DNS server that handles both internal service discovery and public-facing DNS for a Spinifex cluster. Zones are human-readable TOML files, served from the local filesystem or synced from Predastore, keeping DNS configuration in the same object storage as the rest of your infrastructure.
- Authoritative DNS over UDP, TCP, and DNS-over-TLS
- Service discovery for cluster components via SRV records
- Zone files stored locally or in S3-compatible storage
Architecture at a Glance
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless: scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Just systemd units, a NATS cluster, and your hardware. Deep dive: docs/DESIGN.md.
Self-hosted Installation
Installation requires an Ubuntu 26.04 or Debian 13 system. See the detailed documentation at docs.mulgadc.com for installing Spinifex.
Bare Metal ISO
The recommended installation is a bootable x86 installer for bare-metal hardware.
curl -fLO https://iso.mulgadc.com/spinifex.iso
Follow the USB install guide to write the ISO to USB and install on your hardware.
Single Node Install
Prerequisite: Spinifex requires a Linux bridge configured on the host for VM networking. See the single-node install guide for setup details.
curl -fsSL https://install.mulgadc.com | bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management
sudo spx admin init --node node1 --nodes 1
sudo systemctl start spinifex.target
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
Development Setup
For a complete development environment see the Source Install documentation.
Spinifex UI
Spinifex ships with a built-in web console, an optional alternative to the AWS CLI, SDKs, and Terraform. If you know the AWS Management Console, the Spinifex UI fills the same role: a browser-based view of your instances, volumes, buckets, VPCs, and IAM resources, without leaving your own network.
Each node serves the console on port 3000 over TLS, and it comes up with spinifex.target:
- Same API, different surface. The UI issues the same AWS SigV4 calls the CLI does, so RBAC, audit trails, and IAM policies apply the same way.
- Single sign-on against your AWS credentials. Log in with the access keys from
~/.aws/credentialson the node where Spinifex is installed. There is no separate user database. - Self-hosted, works offline. The Spinifex binary embeds the UI and each node serves it directly, with no external CDN or analytics calls.
For the full walkthrough, covering first-time TLS certificate trust, login, and a feature tour, see Launching the Web UI in the cluster setup guide.
Development Philosophy
Built by Engineers, For Engineers
Spinifex is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
AI-Assisted Development
Spinifex is architected and implemented by experienced engineers, with Claude Code accelerating certain development tasks. The split keeps the quality and reliability standards that production infrastructure demands:
How We Use Claude Code:
- Code Generation: Boilerplate AWS API structures and handlers
- Documentation: Comprehensive development guides and API documentation
- Testing: Test case generation and validation scenarios
- Refactoring: Large-scale code restructuring and optimization
What Remains Human-Driven:
- Architecture Decisions: Core system design and scalability choices
- Security Implementation: Authentication, encryption, and threat modeling
- Performance Optimization: Real-world performance tuning and benchmarking
- Production Operations: Deployment strategies and operational procedures
Trademarks
"AWS", "Amazon Web Services", and all related service names (EC2, EBS, S3, VPC, IAM, and others) are trademarks of Amazon.com, Inc. or its affiliates. Mulga Defense Corporation and Spinifex are independent and not affiliated with, endorsed by, or sponsored by Amazon.com, Inc. or Amazon Web Services, Inc. References to AWS services describe interoperability and compatibility only.
License
Spinifex is open source under the GNU Affero General Public License v3.0, with a commercial license available. You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.