Josh Siviour
d52245001e
* feat: add /v1/embeddings gateway adapter * feat: Ochre vector store pgvector backend, KV index registry, orchestration Adds handlers_ochrevector: the VectorBackend interface + pgx/v5 impl with per-account schema/role isolation enforced via SET LOCAL ROLE + search_path inside every transaction, the KV index registry (CREATING/READY/DELETING/STALE), and the create/delete/list/reconcile orchestration tying them together with crash-safe rollback and re-entrant recovery of stuck records. * feat: singleton appliance-ensure state machine for ochre vector store Adds the encrypted control-plane credential store and re-entrant Ensure state machine for the platform pgvector appliance (D2): a fixed-key KV claim mirroring Registry.Reserve elects exactly one winner to launch the appliance via an injected ApplianceLauncher, losers wait or resume a stale crashed provision, and the master password is generated once, stored only AES-256-GCM encrypted, and decrypted solely inside Connect to build a pgxBackend. * feat: Ochre vector store async ingestion (chunker, job store, worker) Adds Stage 3: a deterministic recursive chunk splitter, a per-account ingestion job-record store mirroring the index registry, the transactional per-document replace on the backend, and the IngestService worker that lists a predastore source, chunks and embeds each object, and replaces its rows idempotently. * feat: Ochre vector store similarity query and metadata filter grammar Adds VectorBackend.Query (cosine kNN, k default/cap, iterative HNSW scan under a filter) and a Bedrock-KB-shaped metadata Filter AST that compiles to parameterized SQL over jsonb, closing off arbitrary SQL by allowlisting jsonb keys and always binding values as params. Also fixes RunJob reporting READY on a job that ingested nothing regardless of failure cause, and stamps SourceSpec.Metadata onto ingested rows so the new filters have data to match against. * feat: Ochre vector store request/response API, VectorService, and CLI Adds the daemon-facing VectorService interface (index CRUD, ingest, describe job, query), its server implementation over the existing registry/backend/embedder, a NATS-forwarding client, and the `spx admin ochre vector` operator CLI. Query resolves and embeds against each index's own pinned model; ingest stamps the same model/dimension onto every source spec server-side. accountID is always taken from the caller context, never the request payload. Daemon subscription wiring and manifest registration are a later change; this is pure request/response plumbing with no construction into daemon.go or awsgw.go. * feat: wire Ochre vector store VectorService into the daemon Constructs the platform Postgres appliance and VectorService behind a new OchreVector config section that defaults off, so existing daemons are unaffected. Adds a real RDS-backed ApplianceLauncher that provisions the appliance over rds.CreateDBInstance/DescribeDBInstances, registers the six ochre.vector.* NATS subjects mirroring the bedrock endpoint service pattern, and degrades to a disabled (nil) service on any boot-time appliance failure rather than blocking daemon startup. * fix: fix Ochre vector store boot ordering and self-register subjects startOchreVector ran before subscribeAll, so its appliance launch fired rds.CreateDBInstance/DescribeDBInstances requests with no responder yet registered, timing out the launch on a first-enable boot. It also ran synchronously, blocking daemon startup up to the appliance launch timeout. Move it to a background goroutine started after subscribeAll returns, and have it register its own ochre.vector.* subjects once the VectorService is actually constructed, via a new registerNatsSubs helper shared with subscribeAll's boot-time table. Guard d.natsSubscriptions with d.mu, since this background registration can now race the shutdown path's unsubscribe-all sweep. * feat: bake pgvector into the shared rds-postgres image * fix: reach the ochre appliance over a routed daemon port on its ENI IP The daemon dialled the RDS appliance over rec.Endpoint, a vanity spx3.net hostname the control-plane host can neither resolve nor route to, so a healthy appliance was never reachable and the vector store disabled itself. Resolve the appliance's customer-ENI private IP read-only via ec2.DescribeNetworkInterfaces filtered on the spinifex:rds-db-instance tag, dial that IP for the DSN instead of the hostname, and give the daemon its own routed ENI in the appliance's subnet bound to an OVS host port so the IP is reachable. Retry appliance startup with bounded backoff instead of hard-disabling on the first transient. * fix: install pgvector at bootstrap for the Ochre vector appliance The daemon connects to the appliance as its NOSUPERUSER master, and the image's pgvector is not a trusted extension, so CREATE EXTENSION vector failed with permission denied. rds-init now installs it via the bootstrap superuser, gated on the appliance's fixed master name so tenant instances are unaffected. * fix: grant the Ochre appliance master CREATE on its database The daemon's EnsureAccount runs CREATE SCHEMA per account as the NOSUPERUSER master, which does not own the postgres database, so bootstrap now grants it CREATE on the database alongside the pgvector install. Both are gated to the appliance master, so tenant instances are unaffected. * fix: make the Ochre appliance's pgvector reachable under per-account isolation The daemon connects as a NOSUPERUSER master and runs every account operation under SET ROLE with a search_path pinned to the account schema. Two gaps blocked that against a real non-superuser Postgres: the master could not SET ROLE into the roles it creates, and the account role could not see the vector type. Bootstrap now grants the appliance master createrole_self_grant and installs pgvector into a read-only extensions schema granted to PUBLIC; the backend carries that schema on each account search_path so the vector type resolves without weakening per-account write isolation. All appliance-scoped. * fix: prune stale VPC host port before installing the daemon appliance port A replaced Ochre appliance is handed the same subnet IP, so a new daemon ENI mints a fresh vhp- port at an address a superseded port still holds. The old port survives at the OS/OVS layer even after its ENI record is wiped, so two interfaces claim one address and the kernel can route to the dead one (EHOSTUNREACH). installVPCHostPort now prunes any other vhp- port holding the target address before installing. * fix: gate VM restore's stale-socket kill on backing-store readiness On daemon start, a live VM whose NBD sockets fail to dial was force-killed as an orphan. While predastore/viperblock is still coming up after a restart those sockets are transiently unreachable, so a healthy VM (the Ochre appliance was the observed victim) was destroyed and its RDS record flipped to failed. The kill now only fires when the backing store is confirmed up; otherwise the live QEMU is kept and QMP reconnected. Genuine orphan reaping under a healthy store is unchanged. * fix: drive ochre ingest jobs from a daemon-side sweep scheduler * feat: add operator teardown verb for the ochre platform appliance Adds Appliance.Teardown to remove the singleton platform Postgres appliance (RDS instance, host port, KV record), an admin-only NATS subject and daemon handler to drive it, and `spx admin ochre appliance teardown --confirm` to invoke it without exposing the action through the tenant awsgw surface. * fix: register ochre appliance teardown before Connect so it survives a broken appliance * fix: purge ochre index and job metadata on appliance teardown * test: mark ochre in-package test files for the external-test gate
What is Spinifex? · Platform · Deploy · AWS Compatibility · Components · Architecture · Installation · Docs
Spinifex: the open, AWS-compatible cloud you run yourself
Spinifex, built by Mulga, recreates the AWS services your software already uses (EC2, EBS, S3, VPC, IAM) on infrastructure you control. Move workloads off the hyperscalers, run AI at a fraction of the cost, or take the cloud to the edge, without rewriting a line. The exact build you ship to AWS, Spinifex serves on your own hardware: same APIs, same Terraform, same SDKs.
What is Spinifex?
Spinifex is an open-source infrastructure platform that recreates the AWS service surface (EC2, EBS, S3, VPC, IAM) on bare-metal, edge, and on-premises environments. Run cloud-native software without a hyperscaler.
Most AWS alternatives wrap an API and call it a cloud. We rebuilt the engineering underneath: distributed object storage with erasure coding, block storage built to survive failure, and compute on bare metal. That depth is the reason your software runs unchanged, instead of rewritten.
It's built for teams that need:
- The AWS tooling they already use (AWS CLI, SDKs, Terraform), with nothing to rewrite
- Full control of the stack: your hardware, your network, your data, your keys
- A cloud that keeps running offline, through disconnection, with no external control plane
- An open core (AGPL-3.0), auditable and yours to keep, with no lock-in
You change where your software runs, not what it is.
The Platform
From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
Three Ways To Deploy
One AWS-compatible surface, three deployment shapes. Pick the one that matches your reality; the platform on top is the same.
Neocloud
Lift and shift onto a partner Neocloud. Move workloads off the hyperscalers and onto our Neocloud partner ecosystem without rewriting them. GPU capacity (H100 / H200 / B200), cheaper, available now. Change an endpoint, keep the software.
On-premise
Bring your own hardware and host Spinifex in your own data centre. A real multi-node HA cluster integrated with your storage and networking, with full control of stack, data, and jurisdiction. A predictable bill, no egress surprises, an auditable open-source core.
Edge
Cloud where the cloud can't reach. Air-gapped sites, vehicles, vessels, factories, and clinics. Compute next to the data, running through disconnection, on hardware you own. The same AWS APIs your software already uses.
AWS Compatibility
Speak the AWS API surface, natively. The AWS SDKs, AWS CLI, and Terraform: everything you deploy on AWS deploys on Spinifex unchanged. At the edge, on-premise, or on a partner Neocloud.
| Service | What it is | Status |
|---|---|---|
| EC2 | Compute | Available |
| EBS | Block Storage | Available |
| S3 | Object Storage | Available |
| VPC | Networking | Available |
| IAM | Identity & Auth | Available |
| ALB / NLB | Load Balancers | Available |
| EKS | Kubernetes | Available |
| ECR | Container Registry | Available |
| ECS | Container Service | Available |
| RDS | Databases | Available |
| Bedrock | AI Deployment | Q3 2026 |
Roadmap items ship under the same AWS API surface. Code written for AWS today keeps working the moment they land. Track what's shipped in the release notes.
Core Components
Spinifex (Compute Service – EC2 Alternative)
Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs similar to EC2. It manages lifecycle operations like start, stop, and terminate, using QEMU's QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs using the AWS CLI, SDKs, or Terraform—without needing Kubernetes or heavyweight orchestrators. Keep in mind, you can also setup a Kubernetes environment using Spinifex with underlying instances.
- EC2-like VM management on bare metal
- Launches with cloud-init metadata support
- Works with standard AWS tooling
Viperblock (Block Storage – EBS Alternative)
Viperblock is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It's built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
- Fast, durable virtual disks
- Replication for resilience
- Exposed over NBD or embedded in VMs
- Supports high performance WAL logs using local NVMe drives to reduce IO traffic to S3.
- In memory read/write block cache for blazing performance.
Predastore (Object Storage – S3-Compatible)
Predastore is a fully S3-compatible object storage system. It supports the AWS S3 API, including Signature V4 authentication, multipart uploads, and Terraform provisioning. Data is chunked and distributed across nodes using Reed-Solomon erasure coding, making it fault-tolerant and ideal for large-scale or low-bandwidth scenarios.
- S3-compatible API and auth
- Multipart uploads, streaming reads/writes
- Data redundancy with Reed-Solomon encoding
Northstar (Authoritative DNS)
Northstar is a lightweight authoritative DNS server that handles both internal service discovery and public-facing DNS for a Spinifex cluster. Zones are human-readable TOML files, served from the local filesystem or synced from Predastore, keeping DNS configuration in the same object storage as the rest of your infrastructure.
- Authoritative DNS over UDP, TCP, and DNS-over-TLS
- Service discovery for cluster components via SRV records
- Zone files stored locally or in S3-compatible storage
Architecture at a Glance
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless: scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Just systemd units, a NATS cluster, and your hardware. Deep dive: docs/DESIGN.md.
Key Features
- AWS-compatible APIs. Use the AWS CLI, SDKs, and Terraform you already know. Repoint your endpoint and ship; nothing to rewrite.
- Zero cloud dependency. Runs entirely on your hardware. No phone-home, no control plane, no external authority. Works fully offline.
- Bare-metal compute. QEMU-based with direct hardware access. No hypervisor tax, no abstraction overhead.
- Built-in storage. Block and object storage included: NVMe caching, Reed-Solomon erasure coding, and replication out of the box.
- Edge-first architecture. Designed for disconnected, contested, and resource-constrained environments from day one.
- Open core, no lock-in. AGPL-3.0 with a commercial option. Inspect it, modify it, deploy it. The platform is yours to keep.
Installation
Installation requires an Ubuntu 26.04 or Debian 13 system. See the detailed documentation at docs.mulgadc.com for installing Spinifex.
Bare Metal ISO
The recommended installation is a bootable x86 installer for bare-metal hardware.
curl -fLO https://iso.mulgadc.com/spinifex.iso
Follow the USB install guide to write the ISO to USB and install on your hardware.
Single Node Install
Prerequisite: Spinifex requires a Linux bridge configured on the host for VM networking. See the single-node install guide for setup details.
curl -fsSL https://install.mulgadc.com | bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management
sudo spx admin init --node node1 --nodes 1
sudo systemctl start spinifex.target
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
Development Setup
For a complete development environment see the Source Install documentation.
Component Repositories
Spinifex coordinates these independent components:
- Predastore - S3-compatible object storage
- Viperblock - EBS-compatible block storage
- Northstar - Authoritative DNS server
Each component can be developed independently. See component-specific documentation for focused development guides.
Spinifex UI
Spinifex ships with a built-in web console — an optional alternative to the AWS CLI, SDKs, and Terraform. If you're familiar with the AWS Management Console, the Spinifex UI fills the same role: a browser-based view of your instances, volumes, buckets, VPCs, and IAM resources, without leaving your own network.
The console is served by each node on port 3000 over TLS, and becomes available as soon as spinifex.target is up:
- Same API, different surface. Every action in the UI is the same AWS SigV4 call the CLI makes — so RBAC, audit trails, and IAM policies apply uniformly.
- Single sign-on against your AWS credentials. Log in with the access keys from
~/.aws/credentialson the node where Spinifex is installed — no separate user database. - Self-hosted, works offline. The UI is embedded in the Spinifex binary and served from the node itself. No external CDN, no analytics calls, no cloud dependency.
For the full walkthrough — first-time TLS certificate trust, login, and feature tour — see Launching the Web UI in the cluster setup guide.
Development Philosophy
Built by Engineers, For Engineers
Spinifex is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
Real-World Experience:
- Production AWS service development and operations
- Large-scale infrastructure deployment and management
- Edge computing and resource-constrained environments
- Enterprise security and compliance requirements
AI-Assisted Development
While Spinifex is architected and implemented by experienced engineers, we leverage Claude Code (Anthropic's AI coding assistant) to accelerate certain development tasks. This approach combines human expertise with AI efficiency:
How We Use Claude Code:
- Code Generation: Boilerplate AWS API structures and handlers
- Documentation: Comprehensive development guides and API documentation
- Testing: Test case generation and validation scenarios
- Refactoring: Large-scale code restructuring and optimization
What Remains Human-Driven:
- Architecture Decisions: Core system design and scalability choices
- Security Implementation: Authentication, encryption, and threat modeling
- Performance Optimization: Real-world performance tuning and benchmarking
- Production Operations: Deployment strategies and operational procedures
This hybrid approach ensures Spinifex benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
Trademarks
"AWS", "Amazon Web Services", and all related service names (EC2, EBS, S3, VPC, IAM, and others) are trademarks of Amazon.com, Inc. or its affiliates. Mulga Defense Corporation and Spinifex are independent and not affiliated with, endorsed by, or sponsored by Amazon.com, Inc. or Amazon Web Services, Inc. References to AWS services describe interoperability and compatibility only.
License
Spinifex is open source under the GNU Affero General Public License v3.0. You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.