Josh Siviour
f877f5edef
* refactor(network/external): introduce Allocator interface + StaticPoolAllocator
Carve external IP allocation out of handlers/ec2/vpc/external_ipam.go into
network/external/static_pool.go behind a typed Allocator interface. The KV
bucket name, schema, CAS retry count, and gateway-IP reservation semantics
are unchanged so existing static-pool data carries over with no manual
migration.
ExternalIPAM stays in handlers/ec2/vpc (L5 cleanup deferred per plan) but
becomes a thin facade that constructs a StaticPoolAllocator and delegates
Allocate / Release / GetPoolRecord through the new interface. Q4 will plug
DHCPPoolAllocator into the same Allocator slot.
Plan: docs/development/improvements/external-pool-dhcp-source.md (Q1).
* feat(network/external/dhcp): package skeleton — Client + Fake + KV store
New network/external/dhcp/ package, restored from the structure deleted
in Slice C (commit 7ff013a) but rehomed under the ADR-0006 layer tree:
- client.go Client interface + AcquireRequest + Lease (with
RenewAt/RebindAt/ExpiresAt helpers).
- nclient4.go NClient4Client — AF_PACKET DORA / Renew / Release
backed by insomniacslk/dhcp/dhcpv4/nclient4.
- fake.go In-memory test fake with per-step hooks for fault
injection and acquire/renew/release call counters.
- kv.go Store over the per-AZ spinifex-dhcp-leases-<az> KV
bucket: Put / Get / Delete / List. Schema matches the
plan (absolute timestamps for acquired/t1/t2/expiry).
- *_test.go Lease-timer arithmetic, fake lifecycle, nclient4 input
validation, KV roundtrip + version-key isolation.
Not yet wired into vpcd, ExternalIPAM, or any allocator — Q3/Q4 plug
it in. Plan: docs/development/improvements/external-pool-dhcp-source.md (Q2).
* feat(network/external/dhcp): DHCPManager + daemon-side NATSClient
Adds the per-vpcd DHCPManager that owns active lease state: KV scan on
startup (re-issues RENEW for non-expired entries, drops the rest),
per-lease renewal goroutines with jittered T1/T2/expiry timers and
post-expiry re-DORA, and NATS request/reply subscribers on
vpc.dhcp.acquire / vpc.dhcp.release. Acquire is idempotent on client-id,
coalesced via singleflight so concurrent retries do not produce
duplicate wire DORAs.
Adds the daemon-side dhcp.NATSClient wrapper (RequestAcquire /
RequestRelease) that ExternalIPAM will route DHCP-sourced allocations
through in Q4. Still no allocator wiring — vpcd does not yet construct
the manager.
* feat(network/external/dhcp): wire DHCP allocators; re-allow source=dhcp
Adds DHCPGatewayLRPAllocator and DHCPPoolAllocator, the two consumers
of the Q3 DHCPManager. The gateway-LRP allocator runs in-process inside
vpcd and DORAs once per VPC (client-id `dhcp-gw-lrp-<vpcID>`, used in
centralized NAT). The pool allocator runs daemon-side and routes
per-EIP / ENI / instance acquisitions through vpcd over the restored
vpc.dhcp.acquire / vpc.dhcp.release topics.
ExternalIPAM now dispatches per-pool: static pools keep the existing
StaticPoolAllocator + KV bucket; DHCP-sourced pools attach a
DHCPPoolAllocator via EnableDHCP. vpcd constructs the dhcp.Manager
when any pool has Source="dhcp" and selects the DHCPGatewayLRPAllocator
for IGWManager; everything else keeps StaticRangeAllocator /
LinkLocalAllocator.
Config validator re-allows source="dhcp", requires bind_bridge, and
rejects range_* / gw_lrp_range_* keys on DHCP pools. Unknown sources
are rejected with the supported list. Allocator.Release uses a
server-side IP→client-id lookup so the existing (pool, ip) signature
keeps working for both static and DHCP pools.
Plan: docs/development/improvements/external-pool-dhcp-source.md (Q4)
* feat(spx admin init): restore --external-source=dhcp + --external-bind-bridge
Source defaults to dhcp when --external-pool is omitted; static remains the
default when a pool range is supplied. Adds PoolBindBridge propagation through
ConfigSettings + formation.NetworkConfig + spinifex.toml template so vpcd's
DHCPManager finds the bridge to DORA on across single-node and joiner paths.
* fix(network/external/dhcp): derive chaddr from client-id when HWAddr unset
DHCPGatewayLRPAllocator and DHCPPoolAllocator (first acquire) own no NIC
and pass an empty HWAddr to the manager. nclient4 rejected the request
with "dhcp acquire: hw_addr is required", breaking nightly cells 1/3/8/10
at AttachIGW. Derive the chaddr from the client-id via utils.HashMAC so
the upstream DHCP server sees a stable 02:xx:xx:xx:xx:xx MAC per AWS
identifier, matching the OVN-side MAC scheme operators already grep for.
* fix(network/external/dhcp): enable IFF_PROMISC on bind_bridge across DORA
Nightly cell 1 timed out at OFFER step with "no matching response packet
received". Upstream router replied unicast to the derived 02:xx chaddr
which the Linux bridge had never learned, so the kernel dropped the
frame before nclient4's AF_PACKET socket could see it.
Reference-counted IFF_PROMISC toggle around every Acquire/Renew/Release
so overlapping leases on the same bridge share one flag flip and the
last release restores prior state. Soft-fails to a warn log when
CAP_NET_ADMIN is unavailable — operator may already have promisc set.
* fix(network/external/dhcp): RFC 2131 retransmission — retry DISCOVER 3×
NClient4 was configured with WithRetry(1) + WithTimeout(60s), meaning a
single DISCOVER per Acquire. When the first frame is lost (STP convergence
on a freshly-up bridge, upstream packet loss) the call blocks the full
60s and fails. Nightly cell 1 hit this at startup: gw-lrp Acquire for two
VPCs timed out back-to-back before any e2e step ran.
Switch to WithRetry(3) with a 5s per-attempt seed; nclient4 doubles the
deadline on each retransmission (5s, 10s, 20s) and sends a fresh DISCOVER
each round. Total budget ~35s, three frames on the wire — recoverable
without the higher layer noticing.
* fix(network/external): three CI-blocking extdhcp bugs
DHCP-source pools failed e2e-nightly cells 1/3/8/10 (net: dhcp) after
the restore-dhcp-source branch landed. Three independent root causes:
* gateway nexthop dropped: GatewayIPAllocator.Allocate had no return
channel for the upstream router learned from lease.Routers. igw fell
back to link-local 169.254.0.2, unreachable on the WAN subnet, so
instance egress and SSH-in died silently. Widen the interface to
return nexthop; DHCP impl pulls Routers[0]; empty Routers is now a
hard error. Static impl returns pool.Gateway as before.
* DORA budget shrunk 6x: WithTimeout(5s) x WithRetry(3) gave a 15 s
wallclock for the full DORA, no outer backoff. Restore the deleted
dhcp_manager.go schedule [4s, 8s, 16s, 32s] under a 90 s budget with
+/- 1 s per-attempt jitter; drop nclient4Retries to 1 so the leaf
stays single-shot and the budget arithmetic lives in one place.
* every vpcd subscribed to vpc.dhcp.acquire: plain nc.Subscribe meant
every vpcd in real-multi cells DORA'd in parallel with the same
derived chaddr, causing upstream-server NAK / lease leak. Switch to
QueueSubscribe with per-AZ queue "vpcd-dhcp-workers-<az>".
Plan: docs/development/bugs/extdhcp-pool-fixes.md (mulga).
Bead: mulga-siv-150.
What is Spinifex? · Platform · Components · Architecture · Installation · Docs
Spinifex: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
Spinifex developed by Mulga Defense Corporation is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
What is Spinifex?
Spinifex replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
It's designed for developers and operators who need:
- Cloud-like infrastructure without the cloud
- Minimal dependencies, full control
- Drop-in compatibility with tools like the AWS CLI, SDKs, and Terraform
- A secure cloud environment you control and own the entire hardware, network and software stack
You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren't feasible.
The Platform
From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
Core Components
Spinifex (Compute Service – EC2 Alternative)
Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs similar to EC2. It manages lifecycle operations like start, stop, and terminate, using QEMU's QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs using the AWS CLI, SDKs, or Terraform—without needing Kubernetes or heavyweight orchestrators. Keep in mind, you can also setup a Kubernetes environment using Spinifex with underlying instances.
- EC2-like VM management on bare metal
- Launches with cloud-init metadata support
- Works with standard AWS tooling
Viperblock (Block Storage – EBS Alternative)
Viperblock is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It's built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
- Fast, durable virtual disks
- Replication for resilience
- Exposed over NBD or embedded in VMs
- Supports high performance WAL logs using local NVMe drives to reduce IO traffic to S3.
- In memory read/write block cache for blazing performance.
Predastore (Object Storage – S3-Compatible)
Predastore is a fully S3-compatible object storage system. It supports the AWS S3 API, including Signature V4 authentication, multipart uploads, and Terraform provisioning. Data is chunked and distributed across nodes using Reed-Solomon erasure coding, making it fault-tolerant and ideal for large-scale or low-bandwidth scenarios.
- S3-compatible API and auth
- Multipart uploads, streaming reads/writes
- Data redundancy with Reed-Solomon encoding
Architecture at a Glance
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless — scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Deep dive: docs/DESIGN.md.
Key Features
AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, or SDKs you already use.
- Designed for Control – Run on your own terms, whether that's in a datacenter, remote site, or sensitive environment.
- Minimal Dependencies – Each service is standalone and avoids complex orchestration layers.
- Works Offline – No reliance on centralized cloud services or external networks.
- Open Source – Licensed under AGPL 3.0. Fork it, extend it, or deploy it as-is.
Installation
Installation requires an Ubuntu / Debian system. See the detailed documentation at docs.mulgadc.com for maintaining and installing Spinifex.
Bare Metal ISO
The recommended installation is a bootable x86 installer for bare-metal hardware.
curl -fLO https://iso.mulgadc.com/spinifex.iso
Follow the USB install guide to write the ISO to USB and install on your hardware. The install guide walks through the full process.
Single Node Install
The installation is straightforward to set up and running on a single node for testing purposes. Debian 13 is currently supported, additional Linux distributions are on the immediate roadmap.
Prerequisite: Linux bridge for networking.
Spinifex requires a Linux bridge configured on the host for VM networking. See the single-node install guide prerequisites for setup details.
curl -fsSL https://install.mulgadc.com | bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management
sudo spx admin init --node node1 --nodes 1
sudo systemctl start spinifex.target
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
Development Setup
For a complete development environment see the Source Install documentation
Component Repositories
Spinifex coordinates these independent components:
- Predastore - S3-compatible object storage
- Viperblock - EBS-compatible block storage
Each component can be developed independently. See component-specific documentation for focused development guides.
Spinifex UI
Spinifex ships with a built-in web console — an optional alternative to the AWS CLI, SDKs, and Terraform. If you're familiar with the AWS Management Console, the Spinifex UI fills the same role: a browser-based view of your instances, volumes, buckets, VPCs, and IAM resources, without leaving your own network.
The console is served by each node on port 3000 over TLS, and becomes available as soon as spinifex.target is up:
open https://YOUR_NODE_IP:3000
- Same API, different surface. Every action in the UI is the same AWS SigV4 call the CLI makes — so RBAC, audit trails, and IAM policies apply uniformly.
- Single sign-on against your AWS credentials. Log in with the access keys from
~/.aws/credentialson the node where Spinifex is installed — no separate user database. - Self-hosted, works offline. The UI is embedded in the Spinifex binary and served from the node itself. No external CDN, no analytics calls, no cloud dependency.
For the full walkthrough — first-time TLS certificate trust, login, and feature tour — see Launching the Web UI in the cluster setup guide.
Development Philosophy
Built by Engineers, For Engineers
Spinifex is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
Real-World Experience:
- Production AWS service development and operations
- Large-scale infrastructure deployment and management
- Edge computing and resource-constrained environments
- Enterprise security and compliance requirements
AI-Assisted Development
While Spinifex is architected and implemented by experienced engineers, we leverage Claude Code (Anthropic's AI coding assistant) to accelerate certain development tasks. This approach combines human expertise with AI efficiency:
How We Use Claude Code:
- Code Generation: Boilerplate AWS API structures and handlers
- Documentation: Comprehensive development guides and API documentation
- Testing: Test case generation and validation scenarios
- Refactoring: Large-scale code restructuring and optimization
What Remains Human-Driven:
- Architecture Decisions: Core system design and scalability choices
- Security Implementation: Authentication, encryption, and threat modeling
- Performance Optimization: Real-world performance tuning and benchmarking
- Production Operations: Deployment strategies and operational procedures
This hybrid approach ensures Spinifex benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
License
Spinifex is open source under the GNU Affero General Public License v3.0. You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.