tomnewton-mulga
4f5fd88c86
* feat(gpu): add H200/RTX Pro 6000 SE models and fix MIG host resource allocation - Fix H100 SXM/PCIe device ID labels swapped in models.go (10de:2330=SXM, 10de:2331=PCIe) - Add H200 SXM (10de:2335) to knownModels, computeModels, and migCapableModels - Add NVIDIArtxPro6000BlackwellSE to definitions.go mapped to g7e family (matching AWS g7e instance type) - Add MIGHostResources() policy table: 4 vCPUs / 32 GiB host RAM per compute slice, covering all known NVIDIA MIG profiles - GenerateMIGTypes now emits non-zero vCPU/memory for EC2 instance type descriptors * fix(daemon): enforce CPU/memory limits for MIG instance types Whole-GPU passthrough remains bypass-only (gpuManager.Claim is the sole gate). MIG types now fall through to canAllocateCount so host CPU and RAM are checked before launch — prevents over-committing a host where one physical GPU backs multiple concurrent slices, each with non-trivial vCPU/memory allocation. * feat: Add snapshot image promotion flow * fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs * fix: test coverage and more consistent prefix check * fixed testing * more tests * fix: added new graceful process killer in utils, bumped NATS timeout for testing * fix: complete at-rest encryption launch + volume lifecycle (#361) * chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against the fix for the encrypted AMI snapshot metadata tag-verify failure (cipher: message authentication failed) on instance launch. Pairs with viperblock PR #22. * fix: forward viperblock master key to nbdkit mount path viperblockd loaded the master key but never passed it to the nbdkit plugin it spawns for volume mounts, so encrypted boot volumes opened in cleartext and failed LoadState with an encryption-state mismatch. Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's encryption_key_file= argument when set. * chore: bump viperblock to fresh-volume VolumeUUID mint fix * chore: bump viperblock to snapshot-link LoadState ordering fix * fix: route encrypted-volume config writes through viperblock keyholder config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived nonce can only be advanced by the master-key holder that owns the volume. The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to refuse encrypted volumes (rewriting would strip the tag) — bricking UpdateVolumeState (attach in-use / detach available) and ModifyVolume. Route encrypted-volume config updates through viperblockd, mirroring the ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config queue group opens the detached volume exclusively and reseals. Cleartext volumes keep the direct-S3 fast path. No crypto leaves viperblockd. mulga-siv-323 * fix: unwrap encryption envelope when snapshot handler reads config.json CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into a VBState. For encrypted volumes config.json is an at-rest envelope {payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so CreateSnapshot 500s on the size guard and the in-use scan silently drops encrypted volumes' snapshots. Unwrap via viperblock.StateBody first, mirroring the volume handler's mergeVolumeConfig path. mulga-siv-324 * test: cover viperblockd ebs.config handlers Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler (ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only size reconcile, live-VB apply, invalid JSON, apply errors, and the detached-volume open path. Clears the diff-coverage gate (68.8%). * chore: pin viperblock to dev tip (#22 merged) * fix(viperblockd): seal volume block-map to predastore on detach + durability e2es (#371) * feat: cloud-init EC2 datasource (#411) * feat: Add snapshot image promotion flow * fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs * fix: test coverage and more consistent prefix check * merge changes * chore: remove dead SnapshotSub field and SIGTERM snapshot remnants from viperblockd SnapshotSub was added during the NATS/SIGTERM snapshot attempts but is never set post-rebase. Removes the field from MountedVolume and the two nil-checked unsubscribe blocks. Re-adds the nbd import which is still needed for NBDKitConfig. * fix: post-rebase gofmt fixes and remove dead snapshot handler code Split gofmt-squashed lines in go.mod, vm_adapters.go, nbd_test.go, viperblockd.go, volumes.go, and volume_durability_test.go. Removes the dead unix-socket snapshot handler tests from viperblockd_handlers_test.go. * test: add viperblock snapshot S3 coverage tests Exercises snapshotVolume, snapshotRunningVolume, and snapshotStoppedVolume past the nil-config guard using an httptest server that returns 403. The AWS SDK credential check fails synchronously without network I/O, keeping the tests fast while covering the S3Config construction, master key load, and viperblock.New code paths. Brings diff coverage from 45% to 69.9%. * fix: cloneAMIToVolume reads config via format-agnostic path LoadStateRequest expects a viperblock binary envelope (encrypted), but CreateImageFromInstance writes the AMI config as plain JSON via putAMIConfig. Replace LoadStateRequest with s.amiLoader.GetAMIConfig which uses viperblock.StateBody to handle both formats, matching the path used by ListImages and all other AMI readers. * fix: pin viperblock to feat/snapshot-ami pseudo-version for LoadLiveCheckpoint go.mod referenced v1.10.0 (released from dev) which pre-dates LoadLiveCheckpoint. CI builds without go.work so it resolved to the published version and failed. Pin to the snapshot-ami HEAD pseudo-version until the branch is merged and tagged. * fix: replace indexOf with slices.Index, revert e2e runner and viperblock version - nbd_test.go: remove local indexOf helper, use slices.Index from stdlib - e2e-suites.yml: revert ubuntu-latest back to ubuntu-26.04 with correct formatting - go.mod: revert viperblock back to v1.10.0 (will resolve once viperblock PR merges) * fix: remove pseudo-version pin from go.sum * feat: drain viperblock via snapshot socket before CreateSnapshot Before LoadLiveCheckpoint(), dial {data_dir}/viperblock/{vol}/snapshot.sock and wait for DrainToBackend() to complete (30s timeout). This ensures WAL chunks and the live checkpoint are current in S3 before the snapshot VB reads them. If the socket is absent the instance is stopped and Close() already drained. fix: socket path used s.config.WalDir which is empty in the daemon config; changed to s.config.DataDir + "/viperblock" to match the path the NBD plugin uses when creating the socket. * feat: wire snapshot/AMI UI actions and add PromoteImage NATS endpoint - Register as AMI from snapshot detail (RegisterImage with BDM pointing at snapshot) - Deregister from image detail (DeregisterImage with navigate-away on success) - Promote to System from image detail (admin-only, irreversible via UI) - NATS subject spinifex.image.promote wired through gateway → daemon → existing admin.PromoteSystemImage - signedFetch extended with optional params to support action-keyed extra fields * refactor: address PR review on PromoteImage types and subscription ordering - Merge PromoteImageInput/PromoteImageOutput into PromoteImageOpts/PromoteImageResult (add JSON tags; drop the duplicate pair) - Inline the input variable in gateway/spx/promote_image.go - Move spinifex.image.promote subscription to the spinifex.* group in daemon.go * test: cover snapshotVolume/Running/Stopped error paths for diff coverage Add encryption-key-load-error and LoadState-failure tests for the three new viperblock snapshot paths (snapshotVolume in snapshot service, snapshotRunningVolume and snapshotStoppedVolume in image service). Diff coverage: 58.8% → 65.5%. --------- Co-authored-by: Josh Siviour <mrk@mulgadc.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
What is Spinifex? · Platform · Components · Architecture · Installation · Docs
Spinifex: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
Spinifex developed by Mulga Defense Corporation is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
What is Spinifex?
Spinifex replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
It's designed for developers and operators who need:
- Cloud-like infrastructure without the cloud
- Minimal dependencies, full control
- Drop-in compatibility with tools like the AWS CLI, SDKs, and Terraform
- A secure cloud environment you control and own the entire hardware, network and software stack
You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren't feasible.
The Platform
From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
Core Components
Spinifex (Compute Service – EC2 Alternative)
Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs similar to EC2. It manages lifecycle operations like start, stop, and terminate, using QEMU's QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs using the AWS CLI, SDKs, or Terraform—without needing Kubernetes or heavyweight orchestrators. Keep in mind, you can also setup a Kubernetes environment using Spinifex with underlying instances.
- EC2-like VM management on bare metal
- Launches with cloud-init metadata support
- Works with standard AWS tooling
Viperblock (Block Storage – EBS Alternative)
Viperblock is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It's built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
- Fast, durable virtual disks
- Replication for resilience
- Exposed over NBD or embedded in VMs
- Supports high performance WAL logs using local NVMe drives to reduce IO traffic to S3.
- In memory read/write block cache for blazing performance.
Predastore (Object Storage – S3-Compatible)
Predastore is a fully S3-compatible object storage system. It supports the AWS S3 API, including Signature V4 authentication, multipart uploads, and Terraform provisioning. Data is chunked and distributed across nodes using Reed-Solomon erasure coding, making it fault-tolerant and ideal for large-scale or low-bandwidth scenarios.
- S3-compatible API and auth
- Multipart uploads, streaming reads/writes
- Data redundancy with Reed-Solomon encoding
Architecture at a Glance
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless — scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Deep dive: docs/DESIGN.md.
Key Features
AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, or SDKs you already use.
- Designed for Control – Run on your own terms, whether that's in a datacenter, remote site, or sensitive environment.
- Minimal Dependencies – Each service is standalone and avoids complex orchestration layers.
- Works Offline – No reliance on centralized cloud services or external networks.
- Open Source – Licensed under AGPL 3.0. Fork it, extend it, or deploy it as-is.
Installation
Installation requires an Ubuntu / Debian system. See the detailed documentation at docs.mulgadc.com for maintaining and installing Spinifex.
Bare Metal ISO
The recommended installation is a bootable x86 installer for bare-metal hardware.
curl -fLO https://iso.mulgadc.com/spinifex.iso
Follow the USB install guide to write the ISO to USB and install on your hardware. The install guide walks through the full process.
Single Node Install
The installation is straightforward to set up and running on a single node for testing purposes. Debian 13 is currently supported, additional Linux distributions are on the immediate roadmap.
Prerequisite: Linux bridge for networking.
Spinifex requires a Linux bridge configured on the host for VM networking. See the single-node install guide prerequisites for setup details.
curl -fsSL https://install.mulgadc.com | bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management
sudo spx admin init --node node1 --nodes 1
sudo systemctl start spinifex.target
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
Development Setup
For a complete development environment see the Source Install documentation
Component Repositories
Spinifex coordinates these independent components:
- Predastore - S3-compatible object storage
- Viperblock - EBS-compatible block storage
Each component can be developed independently. See component-specific documentation for focused development guides.
Spinifex UI
Spinifex ships with a built-in web console — an optional alternative to the AWS CLI, SDKs, and Terraform. If you're familiar with the AWS Management Console, the Spinifex UI fills the same role: a browser-based view of your instances, volumes, buckets, VPCs, and IAM resources, without leaving your own network.
The console is served by each node on port 3000 over TLS, and becomes available as soon as spinifex.target is up:
open https://YOUR_NODE_IP:3000
- Same API, different surface. Every action in the UI is the same AWS SigV4 call the CLI makes — so RBAC, audit trails, and IAM policies apply uniformly.
- Single sign-on against your AWS credentials. Log in with the access keys from
~/.aws/credentialson the node where Spinifex is installed — no separate user database. - Self-hosted, works offline. The UI is embedded in the Spinifex binary and served from the node itself. No external CDN, no analytics calls, no cloud dependency.
For the full walkthrough — first-time TLS certificate trust, login, and feature tour — see Launching the Web UI in the cluster setup guide.
Development Philosophy
Built by Engineers, For Engineers
Spinifex is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
Real-World Experience:
- Production AWS service development and operations
- Large-scale infrastructure deployment and management
- Edge computing and resource-constrained environments
- Enterprise security and compliance requirements
AI-Assisted Development
While Spinifex is architected and implemented by experienced engineers, we leverage Claude Code (Anthropic's AI coding assistant) to accelerate certain development tasks. This approach combines human expertise with AI efficiency:
How We Use Claude Code:
- Code Generation: Boilerplate AWS API structures and handlers
- Documentation: Comprehensive development guides and API documentation
- Testing: Test case generation and validation scenarios
- Refactoring: Large-scale code restructuring and optimization
What Remains Human-Driven:
- Architecture Decisions: Core system design and scalability choices
- Security Implementation: Authentication, encryption, and threat modeling
- Performance Optimization: Real-world performance tuning and benchmarking
- Production Operations: Deployment strategies and operational procedures
This hybrid approach ensures Spinifex benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
License
Spinifex is open source under the GNU Affero General Public License v3.0. You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.