Ben Duncan
8409799cbf
* fix: default the OCI helper to the DEFAULT profile oci_env.py shipped our own internal Oracle compartment profile as the default for every reader, with a second internal name as a fallback. A fresh tenancy's config holds only `[DEFAULT]`, so the shipped default was a name nobody else can have, and the fallback chain could land a deployment in whichever tenancy a different profile happened to point at. The profile is now `DEFAULT`, overridable by `OCI_CLI_PROFILE` or `--profile`, and a named profile the config does not hold is an error rather than a fallback. `DEFAULT` needs its own existence test, because configparser treats it as the fallback for every other section and `has_section` denies it exists even when `oci setup config` wrote nothing else. `eip-move-test.sh` and the config table in the README carried the same name. CI reads its credential from the environment rather than from a profile, so nothing there referenced it. * docs: look the OCI compartment OCID up by name The guide asked the reader to paste `ocid1.compartment.oc1..xxx` three times, and the `oci` CLI it already requires can resolve the name instead. The lookup needs its own check. `oci iam compartment list --query ... | [0]` exits 0 and prints nothing when the name matches nothing, so an unset `$CID` would otherwise reach Terraform and fail with an OCI error naming neither the compartment nor the cause. Two compartments in one tenancy can share a name, which the listing command shows and the lookup cannot resolve, so the paste form stays as the answer for that case. Records why the tenancy root is the wrong value: the step 2.1 policy is scoped to whatever `$CID` holds. * Revert "docs: look the OCI compartment OCID up by name" This reverts commit5443149cce. * Reapply "docs: look the OCI compartment OCID up by name" This reverts commit1535d94c8a. * fix: stop validate-topology.sh failing on macOS bash `tf_principal_var` is empty on every run that does not pass `--instance-principal`, and bash before 4.4 treats `"${arr[@]}"` on an empty array as an unbound variable under `set -u`. macOS ships bash 3.2, so the documented deployment command aborted at the tf_vars assembly with `tf_principal_var[@]: unbound variable` before building anything. Assigning the array was not enough, which is why the earlier `HOSTS=()` fix did not cover this: the error is in the expansion, not the reference. Both sites now expand only behind a length test, which is valid on every version. `capture_journals` had the same shape for `HOSTS`, reachable when a run fails before the apply. Reproduced and fixed against bash 3.2.57 in a container, which emits the identical message, and the script now runs to the terraform call on 3.2 for vm-single, vm-single-principal and `--instance-principal`, including the `--keep` cleanup path with no hosts. * docs: record the bash 3.x requirement in the OCI scripts These three are the scripts the OCI guide tells an operator to run, and macOS ships bash 3.2, so a bash 4 construct in any of them breaks a deployment rather than a test. The constraint has now been rediscovered twice, through declare -A and mapfile, and then through an empty array expanded under set -u. Names the three traps at the top of each file so the next change does not have to find them by failing on someone's Mac. * fix: stream the workbook suite instead of hiding it in a file The nine workbooks ran with stdout redirected to `workbooks.log`, so a run wedged inside one printed nothing until the whole suite finished. A CI job then sat silent for hours with no way to tell which workbook it was in, or whether it was working at all. The output is now teed, so the file the per-workbook table is parsed from is unchanged while the run is visible live. Each of the driver's own RUN/PASS/FAIL lines is stamped with elapsed seconds and the workbook name, which is what identifies the slow one without reading the log. `pipefail` carries the ssh exit status through the pipe, so a failed suite still takes the branch that records the table and captures the journals. * fix: grant the OCI policy the subnet verb AssignPrivateIP requires An OCI node with a pool-mode external interface formed, reported healthy, and then refused every launch wanting a public address with a 404 NotAuthorizedOrNotFound from AssignPrivateIP. AssignPrivateIP calls CreatePrivateIp, which Oracle documents as needing PRIVATE_IP_CREATE, PRIVATE_IP_ASSIGN, VNIC_ASSIGN and SUBNET_ATTACH, and DeletePrivateIp as needing SUBNET_DETACH. The policy granted use vnics, manage private-ips and manage public-ips, and no subnet permission at all, so SUBNET_ATTACH was never held. Only the subnets use verb carries it. Measured on a live node as the instance principal: GetVnic and ListPrivateIps succeed, GetSubnet and CreatePrivateIp both 404. Authentication, the dynamic group's matching rule and the compartment were all correct, so the credential was authentic and unauthorised. The documented operator policy for the API-key route carried the same three statements, so neither credential route could allocate. A node now proves its own credential at pool start by making one read in each permission class, and logs the missing verb if one fails. The topology gate waited for "resolved the external VNIC", which comes from the metadata service and needs no IAM policy, so an unauthorised node passed the gate and failed an hour later inside a workbook as capacity. It now waits for the authorisation line and fails immediately on a refusal. Reads rather than a real assign-and-release round trip: a round trip also fails on a full subnet, and reporting that as an authorisation fault sends an operator to IAM for a capacity problem. * docs: say what a principal run costs in a tenancy with no dynamic group No principal topology can pass there, and the only previous hint was one clause about OCI_INSTANCE_PRINCIPAL needing the group to exist. * test: cover the subnet read and the authorisation verdict Diff coverage was 39.1% against a 60% gate. Three of the four gaps were real. GetSubnet on the SDK-backed client had no test, so the request shape was unverified: addressing the collection rather than one subnet by OCID would answer from a different permission and report a credential as working when it is not. Also pins that a sparse body is not an error and that the unauthorised 404 arrives as ErrNotFound, which is what stops the check retrying a verdict that will never change. The fake's GetSubnet was exercised only from the ocinet tests, and coverage is recorded per package under test, so it read as zero-covered. reportAuthorisation now has a test asserting the exact log text, because validate-topology.sh greps for it and a rename would quietly turn the gate into one that never fires. The six lines still uncovered are inside FromPoolConfig, which needs a live JetStream to reach. Diff coverage 87.0%. * fix: refuse a run whose tfvars file contradicts the compartment it was given Terraform ranks *.auto.tfvars above TF_VAR_*, so a leftover file wins silently over an operator targeting a different tenancy by environment, and the guide tells every operator to write one. Hit for real: a stale terraform.auto.tfvars naming a different tenancy's compartment and a bare-metal shape overrode TF_VAR_compartment_ocid, and the plan was for 11 resources in a compartment nobody had chosen. Only a dry run caught it. The run now stops and names both values rather than building. * fix: collect orphaned OCI public IPs and name the limit that was hit A reserved public IP attached to nothing is on no VNIC, so the reconcile that exists to find leaked addresses could not see the ones that matter. It also ran only at startup, while an address is leaked on release and on teardown. Twenty accumulated in one compartment and exhausted the tenancy's limit of 50, after which every launch needing a public address aborted. Reconcile now lists reserved public IPs in the compartment and collects those that carry our prefix, are attached to nothing, are named by no binding and are older than a grace period. All four are needed: an allocated but unassociated Elastic IP is detached too and is the customer's to keep, and a young detached address is what another node's CreatePublicIP looks like until it attaches. A pool with no binding record is not swept at all, since that is indistinguishable from a store that cannot be read. The pass now repeats on a ticker rather than running once at startup. Both OCI ceilings also resolved to InsufficientAddressCapacity, which tells an operator the region is out of addresses when the fix is to release addresses they already hold. AWS separates them, so a tenancy out of reserved public IPs is now AddressLimitExceeded and a full VNIC is PrivateIpAddressLimitExceeded. * feat: run the OCI principal e2e in a tenancy that can hold its dynamic group vm-single-principal needs the tenancy's dynamic group, and creating one is a tenancy-root write that a shared test tenancy does not grant us. The topology could never pass there however correct the code was, so it now takes its own credential from OCI_PRINCIPAL_*, all six or none. A partial set is refused before the build rather than at the apply, because it would build one tenancy's instances against another's compartment. With none set the topology is recorded SKIPPED instead of spending a provision to fail. The sweep takes the same credential, or a leftover principal deployment is destroyed against a tenancy that never held it and reported clean. Two defects found while proving it: --dry-run reported FAIL on every topology with a tree source, because the dry run skips the build and then refuses the tarball it would have produced. The flag operators are told to run first could not pass. The dry run truncated its plan to the last 30 lines, which is the outputs block. That dropped the resource count and every compartment and shape line, so the one check a dry run exists for could not be made from its output. It now writes the whole plan and prints the summary. Removing a deployment is documented as leaving public IPs behind. Spinifex allocates them on the running node, so they are in no Terraform state and destroy cannot see them; twenty accumulated and exhausted a tenancy's limit, which then refused every launch in every compartment. * fix: canonicalise the Opc-Next-Page header in the pagination test CI's linter rejects a non-canonical header name and the local preflight does not, so this passed here and failed there.
What is Spinifex? · Platform · Deploy · AWS Compatibility · Live Demo · Components · Architecture · Installation · Docs
Spinifex: the open, AWS-compatible cloud you run yourself
Spinifex, built by Mulga, recreates the AWS services your software already uses (EC2, EBS, S3, VPC, IAM) on infrastructure you control. Move workloads off the hyperscalers, run AI at a fraction of the cost, or take the cloud to the edge, without rewriting a line. The exact build you ship to AWS, Spinifex serves on your own hardware: same APIs, same Terraform, same SDKs.
What is Spinifex?
Spinifex is an open-source infrastructure platform that recreates the AWS service surface (EC2, EBS, S3, VPC, IAM) on bare-metal, edge, and on-premises environments. Run cloud-native software without a hyperscaler.
Built for teams that need:
- The AWS tooling you already use (AWS CLI, SDKs, Terraform), with nothing to rewrite
- Full control of the stack: your hardware, your network, your data, your keys
- A cloud that keeps running offline, through disconnection, with no external control plane
- An open core (AGPL-3.0), auditable and yours to keep, with no lock-in
You change where your software runs, not what it is.
The Platform
From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
Three Ways To Deploy
The same AWS-compatible platform runs in three shapes. Pick the one that matches your hardware.
Neocloud
Lift and shift onto a partner Neocloud without rewriting workloads. GPU capacity (H100 / H200 / B200), cheaper, available now. Change an endpoint, keep the software.
On-premise
Host Spinifex on your own hardware in your own data centre. You get a multi-node HA cluster integrated with your storage and networking, so the stack, the data, and the jurisdiction stay yours.
Edge
Cloud where the cloud can't reach. Air-gapped sites, vehicles, vessels, factories, and clinics. Compute next to the data, running through disconnection, on hardware you own. The same AWS APIs your software already uses.
AWS Compatibility
Spinifex speaks the AWS API surface. The AWS SDKs, AWS CLI, and Terraform all work unchanged, whether you deploy at the edge, on-premise, or on a partner Neocloud.
| Service | What it is | Status |
|---|---|---|
| EC2 | Compute | Available |
| EBS | Block Storage | Available |
| S3 | Object Storage | Available |
| VPC | Networking | Available |
| IAM | Identity & Auth | Available |
| ALB / NLB | Load Balancers | Available |
| EKS | Kubernetes | Available |
| ECR | Container Registry | Available |
| ECS | Container Service | Available |
| RDS | Databases | Available |
| Bedrock | AI Deployment | Q4 2026 |
Roadmap items ship under the same AWS API surface. Code written for AWS today keeps working the moment they land. Track what's shipped in the release notes.
Live Demo
Create a live sandbox in under a minute. The fastest way to experience the stack. Try Spinifex against one of your own workloads on infrastructure we run for you.
- Live in under a minute. No card, no install.
- Yours for 72 hours to test and experiment.
- Real AWS-compatible endpoint, access keys and console.
- EC2, EBS, S3, VPC, IAM, EKS and RDS, ready to call.
- Point your existing AWS CLI or Terraform straight at it.
Sign up at https://mulgadc.com/signup
Core Components
Spinifex (Compute and Platform Services – EC2, EKS, ECS, RDS)
Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing an EC2 compatible API. It manages lifecycle operations like start, stop, and terminate through QEMU's QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs with the AWS CLI, SDKs, or Terraform, without Kubernetes or a heavyweight orchestrator.
The managed services build on that same instance layer. An EKS node group, an ECS container instance, and an RDS database are Spinifex VMs on your own hardware, each driven through its own AWS API.
- EC2 – VM lifecycle on bare metal, cloud-init and IMDS metadata, key pairs, tags, spot instances, capacity reservations, and GPU passthrough
- EKS – Managed Kubernetes on k3s, with HA control planes, managed node groups, cluster add-ons, an OIDC provider, and IAM access entries
- ECS – Task and service scheduling across container instances, with awsvpc ENIs, capacity providers, rolling deployments, and GPU-aware placement
- ECR – Private container registry backed by Predastore, with content-addressed blobs deduplicated per account and lifecycle policies
- RDS – Managed PostgreSQL and MariaDB, with parameter groups, automated backups, and DNS endpoints served by Northstar
- ALB / NLB – Layer 7 load balancing on HAProxy and layer 4 on nginx
stream, with health checks and ACM-issued listener certificates - VPC and IAM – OVN-backed networking and security groups, with roles, policies, and SigV4 auth applied across every service
Viperblock (Block Storage – EBS Alternative)
Viperblock is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It's built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
- Fast, durable virtual disks
- Replication for resilience
- Exposed over NBD or embedded in VMs
- High performance WAL logs on local NVMe drives, reducing IO traffic to S3
- In-memory read/write block cache
Predastore (Object Storage – S3-Compatible)
Predastore is a fully S3-compatible object storage system. It supports the AWS S3 API, including Signature V4 authentication, multipart uploads, and Terraform provisioning. Predastore chunks data and distributes it across nodes using Reed-Solomon erasure coding, so it survives node loss and suits large-scale or low-bandwidth sites.
- S3-compatible API and auth
- Multipart uploads, streaming reads/writes
- Data redundancy with Reed-Solomon encoding
Northstar (Authoritative DNS)
Northstar is a lightweight authoritative DNS server that handles both internal service discovery and public-facing DNS for a Spinifex cluster. Zones are human-readable TOML files, served from the local filesystem or synced from Predastore, keeping DNS configuration in the same object storage as the rest of your infrastructure.
- Authoritative DNS over UDP, TCP, and DNS-over-TLS
- Service discovery for cluster components via SRV records
- Zone files stored locally or in S3-compatible storage
Architecture at a Glance
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless: scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Just systemd units, a NATS cluster, and your hardware. Deep dive: docs/DESIGN.md.
Self-hosted Installation
Installation requires an Ubuntu 26.04 or Debian 13 system. See the detailed documentation at docs.mulgadc.com for installing Spinifex.
Bare Metal ISO
The recommended installation is a bootable x86 installer for bare-metal hardware.
curl -fLO https://iso.mulgadc.com/spinifex.iso
Follow the USB install guide to write the ISO to USB and install on your hardware.
Single Node Install
Prerequisite: Spinifex requires a Linux bridge configured on the host for VM networking. See the single-node install guide for setup details.
curl -fsSL https://install.mulgadc.com | bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management
sudo spx admin init --node node1 --nodes 1
sudo systemctl start spinifex.target
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
Development Setup
For a complete development environment see the Source Install documentation.
Spinifex UI
Spinifex ships with a built-in web console, an optional alternative to the AWS CLI, SDKs, and Terraform. If you know the AWS Management Console, the Spinifex UI fills the same role: a browser-based view of your instances, volumes, buckets, VPCs, and IAM resources, without leaving your own network.
Each node serves the console on port 3000 over TLS, and it comes up with spinifex.target:
- Same API, different surface. The UI issues the same AWS SigV4 calls the CLI does, so RBAC, audit trails, and IAM policies apply the same way.
- Single sign-on against your AWS credentials. Log in with the access keys from
~/.aws/credentialson the node where Spinifex is installed. There is no separate user database. - Self-hosted, works offline. The Spinifex binary embeds the UI and each node serves it directly, with no external CDN or analytics calls.
For the full walkthrough, covering first-time TLS certificate trust, login, and a feature tour, see Launching the Web UI in the cluster setup guide.
Development Philosophy
Built by Engineers, For Engineers
Spinifex is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
AI-Assisted Development
Spinifex is architected and implemented by experienced engineers, with Claude Code accelerating certain development tasks. The split keeps the quality and reliability standards that production infrastructure demands:
How We Use Claude Code:
- Code Generation: Boilerplate AWS API structures and handlers
- Documentation: Comprehensive development guides and API documentation
- Testing: Test case generation and validation scenarios
- Refactoring: Large-scale code restructuring and optimization
What Remains Human-Driven:
- Architecture Decisions: Core system design and scalability choices
- Security Implementation: Authentication, encryption, and threat modeling
- Performance Optimization: Real-world performance tuning and benchmarking
- Production Operations: Deployment strategies and operational procedures
Trademarks
"AWS", "Amazon Web Services", and all related service names (EC2, EBS, S3, VPC, IAM, and others) are trademarks of Amazon.com, Inc. or its affiliates. Mulga Defense Corporation and Spinifex are independent and not affiliated with, endorsed by, or sponsored by Amazon.com, Inc. or Amazon Web Services, Inc. References to AWS services describe interoperability and compatibility only.
License
Spinifex is open source under the GNU Affero General Public License v3.0, with a commercial license available. You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.