spinifex/docs/DESIGN.md

Spinifex Architecture

Spinifex is an AWS-compatible infrastructure platform for bare-metal, edge, and on-premise deployments. It provides EC2-compatible VM orchestration backed by QEMU/KVM, with storage provided by companion projects Viperblock (EBS) and Predastore (S3).

High-Level Architecture

Request Flow

1. AWS SDK Request

Users interact with Spinifex using standard AWS SDKs or the AWS CLI by using the spinifex profile.

AWS_PROFILE=spinifex aws ec2 run-instances \
    --image-id ami-debian13 \
    --instance-type t3.micro \
    --key-name my-keypair

The AWS SDK formats this as an HTTPS POST with:

• AWS SigV4 authentication headers
• EC2 Query Protocol body (Action=RunInstances&ImageId=ami-debian13&...)

2. AWS Gateway

The gateway (spinifex/services/awsgw/awsgw.go) is the entry point:

// Connect to NATS (retries while the local broker comes up)
natsConn, err := utils.ConnectNATSWithRetry(...)

// Load IAM (master key + JetStream KV) and create the gateway
gw := gateway.GatewayConfig{
    NATSConn:   natsConn,
    Config:     nodeConfig.AWSGW.Config,
    IAMService: iamService,
    // ...
}

// Serve over TLS
server.ListenAndServeTLS("", "")

Request routing (spinifex/gateway/gateway.go):

1. Authentication: SigV4 middleware validates AWS credentials and resolves the account ID
2. Throttling: Per-account+action token bucket rejects bursts post-auth
3. Service Detection: Extracts service name from the Authorization header
4. Action Dispatch: Routes to service-specific handler

switch svc {
case "ec2":
    err = gw.EC2_Request(w, r)
case "account":
    err = gw.Account_Request(w, r)
case "iam":
    err = gw.IAM_Request(w, r)
case "sts":
    err = gw.STS_Request(w, r)
case "elasticloadbalancing":
    err = gw.ELBv2_Request(w, r)
case "eks":
    err = gw.EKS_Request(w, r)
case "ecs":
    err = gw.ECS_Request(w, r)
case "ecr":
    err = gw.ECR_Request(w, r)
case "acm":
    err = gw.ACM_Request(w, r)
case "tagging":
    err = gw.Tagging_Request(w, r)
case "spinifex":
    err = gw.Spinifex_Request(w, r)
}

3. EC2 Handler

The EC2 handler (spinifex/gateway/ec2.go) parses the Action parameter and delegates to specific handlers via a generic ec2Handler wrapper. The account ID resolved by SigV4 auth is threaded into every handler so requests are scoped per IAM principal:

"RunInstances": ec2Handler(func(input *ec2.RunInstancesInput, gw *GatewayConfig, accountID string) (any, error) {
    return gateway_ec2_instance.RunInstances(input, gw.NATSConn, accountID)
}),
"DescribeInstances": ec2Handler(func(input *ec2.DescribeInstancesInput, gw *GatewayConfig, accountID string) (any, error) {
    return gateway_ec2_instance.DescribeInstances(input, gw.NATSConn, gw.DiscoverActiveNodes(), accountID)
}),
// ... + volumes, snapshots, VPCs, subnets, route tables, IGWs, NAT gateways,
// security groups, network interfaces, elastic IPs, placement groups, key pairs,
// images, tags, capacity reservations, spot instances

4. NATS Messaging

The gateway communicates with daemons via NATS request/response. Most calls go through utils.NATSRequest, which marshals the input, attaches the account ID as a NATS header, and unmarshals the typed response:

func (s *NATSInstanceService) RunInstances(input *ec2.RunInstancesInput, accountID string) (*ec2.Reservation, error) {
    topic := fmt.Sprintf("ec2.RunInstances.%s", aws.StringValue(input.InstanceType))
    return utils.NATSRequest[ec2.Reservation](s.natsConn, topic, input, 5*time.Minute, accountID)
}

RunInstances uses a per-instance-type subject so NATS only delivers the request to a node with spare capacity for that type — no application-level reject-and-retry.

5. Daemon Processing

Daemons (spinifex/daemon/daemon.go) subscribe to NATS topics and handle requests. A table-driven subscribeAll() registers the static EC2/ELBv2 surface at startup:

// Queue group → load-balanced (one daemon handles each request)
{"ec2.CreateKeyPair", d.handleEC2CreateKeyPair, "spinifex-workers"},

// No queue group → fan-out (every daemon responds)
{"ec2.DescribeInstanceTypes", d.handleEC2DescribeInstanceTypes, ""},

ec2.RunInstances is dynamic: ResourceManager subscribes to ec2.RunInstances.{type} only while the node has capacity for that type, and unsubscribes when full. The node-targeted subject ec2.RunInstances.{type}.{nodeId} stays subscribed even at full capacity — the launch-time allocate() enforces limits there. Per-instance commands flow over ec2.cmd.{instanceID}, subscribed only by the owning node.

Queue Groups: topics with the spinifex-workers queue group are load-balanced — only one daemon handles each request. Topics without a queue group fan out to all daemons.

6. VM Launch

When a daemon handles RunInstances:

1. Resource Check: Validates CPU/memory availability via ResourceManager
2. Volume Generation: Creates boot and EFI volumes via Viperblock
3. Volume Mount: Sends ebs.mount request to Viperblock, receives NBD URI
4. QEMU Launch: Builds and executes QEMU command with NBD-backed disks
5. QMP Monitoring: Establishes QEMU Machine Protocol connection for VM management
6. Command subscription: Subscribes to ec2.cmd.{instanceID} for subsequent commands on this VM

// Volume mounting via NATS
msg, err := d.natsConn.Request("ebs.mount", ebsMountRequest, 10*time.Second)

// QEMU launch with NBD storage
cmd := exec.Command("qemu-system-x86_64",
    "-drive", fmt.Sprintf("file=nbd:%s,format=raw", nbdURI),
    // ... additional QEMU args
)

Key Components

Daemon Structure

type Daemon struct {
    node          string                  // Node identifier
    clusterConfig *config.ClusterConfig   // Cluster-wide configuration
    config        *config.Config          // Node-specific configuration
    natsConn      *nats.Conn              // NATS connection
    resourceMgr   *ResourceManager        // CPU/Memory tracking + dynamic RunInstances subscriptions
    vmMgr         *vm.Manager             // Local VMs
    // ... + one service struct per EC2/ELBv2 resource
    //       (instance, key, image, volume, snapshot, tags, vpc, subnet,
    //        igw, eigw, natgw, routetable, eip, placementgroup, elbv2, ...)
}

Resource Manager

Tracks available and allocated CPU/memory to prevent overcommit, and drives the dynamic ec2.RunInstances.{type} subscriptions:

type ResourceManager struct {
    mu            sync.RWMutex
    hostVCPU      int                                // raw runtime.NumCPU
    hostMemGB     float64                            // raw /proc/meminfo MemTotal
    reservedVCPU  int                                // held back for spinifex services
    reservedMem   float64                            // held back for spinifex services
    allocatedVCPU int
    allocatedMem  float64
    instanceTypes map[string]*ec2.InstanceTypeInfo   // t3.micro, t3.small, etc.
    // ... + capacity-reservation accounting, GPU manager, nbdkit memory
    //       overhead tracking, and the dynamic-subscription state
}

Multi-Node Aggregation

For operations that need data from all nodes (like DescribeInstances), the gateway uses inbox-based fan-out (spinifex/gateway/ec2/instance/DescribeInstances.go):

func DescribeInstances(...) {
    // Create unique inbox for collecting responses
    inbox := nats.NewInbox()
    sub, _ := natsConn.SubscribeSync(inbox)

    // Publish to all nodes (no queue group = all daemons respond).
    // Account ID is propagated as a NATS header so each daemon scopes its response.
    pubMsg := nats.NewMsg("ec2.DescribeInstances")
    pubMsg.Reply = inbox
    pubMsg.Data = jsonData
    pubMsg.Header.Set(utils.AccountIDHeader, accountID)
    _ = natsConn.PublishMsg(pubMsg)

    // Collect responses, returning early once expectedNodes have replied
    // or the 3s timeout is hit.
    for {
        msg, err := sub.NextMsg(3 * time.Second)
        allReservations = append(allReservations, nodeOutput.Reservations...)
    }

    return allReservations
}

Storage Integration

Viperblock (EBS)

Block storage volumes are managed via NATS messages to Viperblock:

// Volume mount request
type EBSRequest struct {
    Name      string  // Volume name (vol-xxxxx)
    VolType   string  // gp2, io1, etc.
    Boot      bool    // Boot volume flag
    EFI       bool    // EFI boot volume
    NBDURI    string  // NBD URI returned from mount
}

Viperblock responds with an NBD URI that QEMU uses to access the block device.

Predastore (S3)

Object storage used for:

• AMI image storage
• SSH public key storage (/bucket/ec2/{key-name}.pub)
• Volume metadata and configuration

Spinifex runs Predastore in-process: spx service predastore start loads /etc/spinifex/predastore/predastore.toml, selects the [[host]] naming this machine, and serves it until the process is signalled. The topology is written by spx admin init / join from the servers that form the cluster.

One [[host]] is one process. The nodes pinned to it under [[host.node]] carry the roles:

Role	Port	Responsibility
gate	TCP 8443	The S3 API. Holds no cluster state; dials the meta and blob nodes to serve a request.
blob	UDP 6660	Erasure-coded object shards, encrypted at rest under this host's master key.
meta	UDP 7660	Raft replica of global state — buckets and the object index.

Ports are unique within a host but repeat across the cluster, so a multi-node cluster puts one node of each role on every machine and every machine uses the same three ports. Nodes sharing a host talk over an in-process pipe and bind no socket; nodes on other hosts are reached over QUIC, authenticated by the cluster CA.

A single-node install is the same arrangement with one host: seven nodes — a gate, three blob and three meta — colocated in one process, needing seven distinct ports (8443, 6660–6662, 7660–7662) to stay unique within the host even though only the gate opens a listener. It has no code path of its own.

The daemon reports the topology on spinifex.storage.config by parsing the config file. It does not connect to Predastore to do so, and no Predastore node exposes a status endpoint.

Configuration

Cluster configuration (spinifex/config/config.go):

type ClusterConfig struct {
    Epoch     uint64            // Incremented on config changes
    Node      string            // This node's identifier
    Version   string            // Spinifex version
    Network   NetworkConfig     // Cluster-wide external network settings (IP pools, IPsec)
    Bootstrap BootstrapConfig   // Default VPC IDs written by admin init
    AWS       AWSConfig         // Cluster-wide AWS-parity settings (region, endpoint suffix)
    Nodes     map[string]Config // Configuration for all nodes
}

type Config struct {
    Node, Host, AdvertiseIP  string
    Region, AZ               string
    BaseDir, DataDir, WalDir string
    Services   []string         // Which services this node runs locally
    Daemon     DaemonConfig
    NATS       NATSConfig       // Broker address, JetStream, ACL token, CA cert
    AWSGW      AWSGWConfig      // Gateway host, TLS certs, throttle config
    Predastore PredastoreConfig // S3 endpoint + service credentials
    Viperblock ViperblockConfig
    VPCD       VPCDConfig       // OVN/OVS endpoints
}

Configs are generated by spx admin init (leader) or spx admin join (followers).

File Structure

spinifex/
├── cmd/
│   ├── spinifex/            # spx CLI: service entry points and `spx admin`
│   ├── installer/           # Platform installer
│   └── ...                  # In-guest agents (lb, ecs, rds, eks helpers) and dev tools
├── spinifex/
│   ├── services/            # Long-running services (awsgw, spinifex, nats, predastore, ...)
│   ├── gateway/             # AWS API surface: routing, SigV4 auth, per-service dispatch
│   ├── handlers/            # NATS-side service implementations (ec2, iam, eks, rds, ...)
│   ├── daemon/              # Per-node daemon and its NATS subscription table
│   ├── vm/, qmp/            # VM lifecycle and the QEMU Machine Protocol client
│   ├── vpcd/, network/      # VPC daemon and the layered OVN/OVS network stack
│   ├── admin/               # Cluster bootstrap (init/join, master.key, CA, tokens)
│   └── config/              # Configuration structures
├── internal/                # In-guest SigV4 clients and the AWS Smithy models
├── build/                   # systemd units, node helper scripts, microVM build
├── scripts/                 # Install, dev-environment, image-build scripts
└── tests/
    ├── integration/         # Gateway router against embedded NATS JetStream
    └── e2e/                 # Live-cluster suites (see docs/testing/e2e-coverage.md)

Development

Running the Stack

sudo spx admin init --node node1 --nodes 1
sudo systemctl start spinifex.target

spx admin init generates the cluster CA, IAM credentials, TLS certs, and AWS CLI spinifex profile, then writes config to /etc/spinifex/ and data to /var/lib/spinifex/. spinifex.target brings up all services (NATS, Predastore, Viperblock, vpcd, the daemon, awsgw, the UI) under systemd. Use spx admin join for additional nodes.

Testing with AWS CLI

export AWS_PROFILE=spinifex

# Run instance
aws ec2 run-instances --image-id ami-debian13 --instance-type t3.micro

# List instances (queries all nodes)
aws ec2 describe-instances

# Terminate instance
aws ec2 terminate-instances --instance-ids i-xxxxx

For cluster-state inspection that bypasses the AWS API surface, use the spx CLI directly: spx get nodes, spx get vms, spx top nodes.