Josh Siviour
75fb15be2b
* fix: provision br-mgmt as OVS bridge + loopback guard for LB gateway Convert br-mgmt from an externally-provisioned Linux bridge to an OVS bridge owned by scripts/setup-ovn.sh, giving every deployment type (tofu-cluster, production single-node, production multi-node) a single provisioning path. setup-ovn.sh grows --mgmt-bridge / --mgmt-cidr / --mgmt-iface / --no-mgmt-bridge flags. Default creates br-mgmt with fail-mode=standalone (excluded from ovn-bridge-mappings so ovn-controller ignores it) and assigns the mgmt IP via a systemd-networkd drop-in so it survives reboot. SetupMgmtTapDevice / CleanupMgmtTapDevice switch from `ip link master` to `ovs-vsctl add-port` / `del-port` to match the new bridge type. Add a loopback guard to wireLBAgentConfig's fallback branch: on a production single-node with awsgw bound to 127.0.0.1 and no mgmt bridge, the fallback previously injected 127.0.0.1 into LB VM cloud-init — silently unreachable because the VM's loopback is its own. The guard leaves GatewayURL empty and the existing "LB agent gateway URL not configured" error fires instead. Regression covered in daemon_wire_lb_test.go. * fix: auto-detect primary IP for admin init --bind default Bare `spx admin init` used a 127.0.0.1 default that kept awsgw on loopback only, so system VMs on br-mgmt could never reach it. Auto-detect the default-route source IPv4 instead, which keeps TLS cert SANs specific (still not a wildcard). Dev-install scripts now pass --bind 127.0.0.1 explicitly to preserve loopback-only behavior on laptops. Closes mulga-siv-7. * test: refresh wire lb loopback-bind comment for siv-7 * fix: dev-install scripts should let admin init auto-detect --bind Pinning --bind to 127.0.0.1 on dev installs left awsgw loopback-only while setup-ovn.sh still provisions br-mgmt, so LB VMs hit 10.15.8.1:9999 with no listener. Drop the explicit --bind so auto-detect picks the primary IP and the siv-6 mgmtRouteVia path routes LB traffic to a real listener. * fix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template PR #144 (nats TLS) made every service listen on --bind's specific IP. For awsgw and predastore this broke localhost-reachability that the UI proxy (spinifexui/spinifexui.go:193-194) and AWS CLI spinifex profile rely on — latent since #144, surfaced by siv-7's auto-detect default on dev installs (502 from the UI). Hardcode awsgw and predastore host fields to 0.0.0.0 so both listen wildcard; loopback clients (UI proxy, spinifex AWS CLI profile, same-host daemon → predastore) keep working, LAN clients still reach them via the LAN-IP SAN on the cert. daemon.host and nats.host stay as {{.BindIP}} because peer-advertised values must not resolve to 0.0.0.0/loopback. predastore.host normalization at config.go:207 rewrites 0.0.0.0 → 127.0.0.1 for local-node consumers, so existing callers see a usable connect address. * revert: siv-7 --bind auto-detect + siv-6 loopback guard (prep mulga-siv-8) Clears the deck before layering the listen-vs-advertise split (mulga-siv-8). The plan restores --bind=0.0.0.0 as the single-node default and introduces an explicit --advertise flag for off-host dial targets; the reverted approach (auto-detect a specific IP at init, hardcode 0.0.0.0 in the template, guard against loopback in wireLBAgentConfig) becomes redundant once that lands. Reverts: -bb0ad3afix: bind awsgw and predastore on 0.0.0.0 in spinifex.toml template -f20a47bfix: dev-install scripts should let admin init auto-detect --bind -49b0b4etest: refresh wire lb loopback-bind comment for siv-7 -b88dfdffix: auto-detect primary IP for admin init --bind default Also drops the loopback guard from siv-6 (74b6e2d) wireLBAgentConfig fallback and its regression test — subsumed by the GatewayURL precedence rework in mulga-siv-8.5 (br-mgmt → AdvertiseIP → error, no silent empty-string fallthrough). The br-mgmt OVS provisioning from74b6e2dis retained. Plan: docs/development/bugs/spx-admin-init-bind.md * feat: admin.DialTarget + AdvertiseHost helpers (mulga-siv-8.2) Introduce the two small address-resolution helpers the listen-vs- advertise split needs, with full unit coverage. No callers yet — dial-site migration lands in mulga-siv-8.5. - DialTarget(listenAddr): rewrites wildcard listen addresses (0.0.0.0:N, [::]:N, :N) to their loopback equivalents so in-process clients dial a target the daemon's cert already has in its SANs. Specific IPs, loopback, hostnames, and malformed input pass through. - AdvertiseHost(listenAddr, advertiseIP): picks the host off-host clients should dial. Wildcard listen → advertiseIP; specific host → that host. Port is dropped so callers append the protocol-appropriate port. Plan: docs/development/bugs/spx-admin-init-bind.md §Proposed Changes/1 * feat: plumb AdvertiseIP through config/formation (mulga-siv-8.3) Pure data-model change. Add AdvertiseIP to: - admin.ConfigSettings (local node, populated by init). - config.Config (per-node; mapstructure "advertise"). - formation.NodeInfo (json "advertise_ip,omitempty") so the join handshake carries it between peers. buildRemoteNodes now prefers n.AdvertiseIP and falls back to n.BindIP so joined nodes from older clusters / pre-siv-8 joiners keep working. No behaviour change yet — templates, CLI flag, and dial-site migration land in siv-8.4 and siv-8.5. * feat: --advertise flag + resolveAdvertiseIP + templates (mulga-siv-8.4) Introduce the listen-vs-advertise split at the CLI boundary. admin init/join: - --bind default for init is 0.0.0.0 (was 127.0.0.1 under siv-7 revert). - New --advertise <IP> flag on both init and join; optional, auto-detected from DetectNetwork().WAN.IP when --bind is wildcard or loopback. - resolveAdvertiseIP(bindIP, advertiseFlag, detected) implements precedence: flag > specific bindIP > WAN auto-detect > 127.0.0.1 + warning. - Join resolves advertise before building the JoinRequest so peers record a reachable dial target (not 0.0.0.0). - isMultiNode: trigger on --nodes >= 2 regardless of bind literal; operator can now run "init --nodes 3" without typing a specific --bind. - Init success message prints bind / advertise / loopback three-way split. Templates: - spinifex.toml: [nodes.X] gains advertise = "{{.AdvertiseIP}}" rendered only when AdvertiseIP is set. - predastore-multinode.toml: iam.nats_url host wrapped in the same {{if eq .BindIP "0.0.0.0"}}localhost{{...}}{{end}} shim that predastore.toml already uses, so co-resident predastore dials its local NATS via loopback instead of 0.0.0.0. Tests cover the resolveAdvertiseIP matrix, the template advertise field (present + omitted), and the predastore nats_url loopback shim. Dial-site migration and GatewayURL rework land in siv-8.5. * feat: dial-site migration + GatewayURL AdvertiseIP fallback (mulga-siv-8.5) Switch every in-process dial site to admin.DialTarget so the wildcard listen address (0.0.0.0) is rewritten to loopback before Dial(): daemon → NATS and → predastore (boot, reconnect, status probe, S3 store, service manifest write), awsgw → NATS. The TLS cert already SANs 127.0.0.1, so loopback-dial a wildcard-listen service is always valid; the old 0.0.0.0 literal was tripping TLS SAN verification. Rework wireLBAgentConfig GatewayURL precedence (spinifex/daemon/daemon.go): 1. AWSGW on specific WAN IP + br-mgmt present → WAN IP (unchanged) 2. br-mgmt present, AWSGW on 0.0.0.0 → br-mgmt IP (unchanged) 3. d.config.AdvertiseIP (single-node default install) → AdvertiseIP (NEW) 4. DevNetworking shim → 10.0.2.2 (unchanged) 5. AWSGW on specific IP, no bridge → that IP (unchanged) 6. Else → log error, skip assignment Kills the pre-siv-8 silent empty-URL failure mode that made CreateLoadBalancer fail with 'missing system credentials: gatewayURL=' on every fresh single-node install. Supersedes siv-6's loopback guard. Extend SetupAWSCredentials with a wanIP parameter (reserved for a future --operator-endpoint flag) and thread AdvertiseIP through finalizeNodeSetup's three call sites (single-node init, multi-node init, join). Print 'Bind / Advertise / Loopback' on multi-node init and join completion to match the single-node summary. Plan: docs/development/bugs/spx-admin-init-bind.md §2, §3
What is Spinifex? · Platform · Components · Architecture · Installation · Docs
Spinifex: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
Spinifex developed by Mulga Defense Corporation is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
What is Spinifex?
Spinifex replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
It's designed for developers and operators who need:
- Cloud-like infrastructure without the cloud
- Minimal dependencies, full control
- Drop-in compatibility with tools like the AWS CLI, SDKs, and Terraform
- A secure cloud environment you control and own the entire hardware, network and software stack
You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren't feasible.
The Platform
From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
Core Components
Spinifex (Compute Service – EC2 Alternative)
Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs similar to EC2. It manages lifecycle operations like start, stop, and terminate, using QEMU's QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs using the AWS CLI, SDKs, or Terraform—without needing Kubernetes or heavyweight orchestrators. Keep in mind, you can also setup a Kubernetes environment using Spinifex with underlying instances.
- EC2-like VM management on bare metal
- Launches with cloud-init metadata support
- Works with standard AWS tooling
Viperblock (Block Storage – EBS Alternative)
Viperblock is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It's built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
- Fast, durable virtual disks
- Replication for resilience
- Exposed over NBD or embedded in VMs
- Supports high performance WAL logs using local NVMe drives to reduce IO traffic to S3.
- In memory read/write block cache for blazing performance.
Predastore (Object Storage – S3-Compatible)
Predastore is a fully S3-compatible object storage system. It supports the AWS S3 API, including Signature V4 authentication, multipart uploads, and Terraform provisioning. Data is chunked and distributed across nodes using Reed-Solomon erasure coding, making it fault-tolerant and ideal for large-scale or low-bandwidth scenarios.
- S3-compatible API and auth
- Multipart uploads, streaming reads/writes
- Data redundancy with Reed-Solomon encoding
Architecture at a Glance
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless — scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Deep dive: docs/DESIGN.md.
Key Features
AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, or SDKs you already use.
- Designed for Control – Run on your own terms, whether that's in a datacenter, remote site, or sensitive environment.
- Minimal Dependencies – Each service is standalone and avoids complex orchestration layers.
- Works Offline – No reliance on centralized cloud services or external networks.
- Open Source – Licensed under AGPL 3.0. Fork it, extend it, or deploy it as-is.
Installation
Installation requires an Ubuntu / Debian system. See the detailed documentation at https://docs.mulgadc.com for maintaining and installing Spinifex.
Single Node Install
The installation is straightforward to set up and running on a single node for testing purposes.
curl -fsSL https://install.mulgadc.com | bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management
sudo spx admin init --node node1 --nodes 1
sudo systemctl start spinifex.target
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
Development Setup
For a complete development environment see the Source Install documentation
Component Repositories
Spinifex coordinates these independent components:
- Predastore - S3-compatible object storage
- Viperblock - EBS-compatible block storage
Each component can be developed independently. See component-specific documentation for focused development guides.
Spinifex UI
Spinifex ships with a built-in web console — an optional alternative to the AWS CLI, SDKs, and Terraform. If you're familiar with the AWS Management Console, the Spinifex UI fills the same role: a browser-based view of your instances, volumes, buckets, VPCs, and IAM resources, without leaving your own network.
The console is served by each node on port 3000 over TLS, and becomes available as soon as spinifex.target is up:
open https://YOUR_NODE_IP:3000
- Same API, different surface. Every action in the UI is the same AWS SigV4 call the CLI makes — so RBAC, audit trails, and IAM policies apply uniformly.
- Single sign-on against your AWS credentials. Log in with the access keys from
~/.aws/credentialson the node where Spinifex is installed — no separate user database. - Self-hosted, works offline. The UI is embedded in the Spinifex binary and served from the node itself. No external CDN, no analytics calls, no cloud dependency.
For the full walkthrough — first-time TLS certificate trust, login, and feature tour — see Launching the Web UI in the cluster setup guide.
Development Philosophy
Built by Engineers, For Engineers
Spinifex is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
Real-World Experience:
- Production AWS service development and operations
- Large-scale infrastructure deployment and management
- Edge computing and resource-constrained environments
- Enterprise security and compliance requirements
AI-Assisted Development
While Spinifex is architected and implemented by experienced engineers, we leverage Claude Code (Anthropic's AI coding assistant) to accelerate certain development tasks. This approach combines human expertise with AI efficiency:
How We Use Claude Code:
- Code Generation: Boilerplate AWS API structures and handlers
- Documentation: Comprehensive development guides and API documentation
- Testing: Test case generation and validation scenarios
- Refactoring: Large-scale code restructuring and optimization
What Remains Human-Driven:
- Architecture Decisions: Core system design and scalability choices
- Security Implementation: Authentication, encryption, and threat modeling
- Performance Optimization: Real-world performance tuning and benchmarking
- Production Operations: Deployment strategies and operational procedures
This hybrid approach ensures Spinifex benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
License
Spinifex is open source under the GNU Affero General Public License v3.0. You're free to use, modify, and deploy it—anywhere you need reliable infrastructure without depending on centralized cloud platforms.