# Spinifex Architecture Spinifex is an AWS-compatible infrastructure platform for bare-metal, edge, and on-premise deployments. It provides EC2-compatible VM orchestration backed by QEMU/KVM, with storage provided by companion projects Viperblock (EBS) and Predastore (S3). ## High-Level Architecture

Spinifex high-level architecture — AWS SDK, gateway, NATS broker, per-node daemons, QEMU/KVM VMs

## Request Flow ### 1. AWS SDK Request Users interact with Spinifex using standard AWS SDKs or the AWS CLI by using the spinifex profile. ```bash AWS_PROFILE=spinifex aws ec2 run-instances \ --image-id ami-debian13 \ --instance-type t3.micro \ --key-name my-keypair ``` The AWS SDK formats this as an HTTPS POST with: - AWS SigV4 authentication headers - EC2 Query Protocol body (`Action=RunInstances&ImageId=ami-debian13&...`) ### 2. AWS Gateway The gateway (`spinifex/services/awsgw/awsgw.go`) is the entry point: ```go // Connect to NATS (retries while the local broker comes up) natsConn, err := utils.ConnectNATSWithRetry(...) // Load IAM (master key + JetStream KV) and create the gateway gw := gateway.GatewayConfig{ NATSConn: natsConn, Config: nodeConfig.AWSGW.Config, IAMService: iamService, // ... } // Serve over TLS server.ListenAndServeTLS("", "") ``` Request routing (`spinifex/gateway/gateway.go`): 1. **Authentication**: SigV4 middleware validates AWS credentials and resolves the account ID 2. **Throttling**: Per-account+action token bucket rejects bursts post-auth 3. **Service Detection**: Extracts service name from the Authorization header 4. **Action Dispatch**: Routes to service-specific handler ```go switch svc { case "ec2": err = gw.EC2_Request(w, r) case "account": err = gw.Account_Request(w, r) case "iam": err = gw.IAM_Request(w, r) case "sts": err = gw.STS_Request(w, r) case "elasticloadbalancing": err = gw.ELBv2_Request(w, r) case "eks": err = gw.EKS_Request(w, r) case "ecs": err = gw.ECS_Request(w, r) case "ecr": err = gw.ECR_Request(w, r) case "acm": err = gw.ACM_Request(w, r) case "tagging": err = gw.Tagging_Request(w, r) case "spinifex": err = gw.Spinifex_Request(w, r) } ``` ### 3. EC2 Handler The EC2 handler (`spinifex/gateway/ec2.go`) parses the `Action` parameter and delegates to specific handlers via a generic `ec2Handler` wrapper. The account ID resolved by SigV4 auth is threaded into every handler so requests are scoped per IAM principal: ```go "RunInstances": ec2Handler(func(input *ec2.RunInstancesInput, gw *GatewayConfig, accountID string) (any, error) { return gateway_ec2_instance.RunInstances(input, gw.NATSConn, accountID) }), "DescribeInstances": ec2Handler(func(input *ec2.DescribeInstancesInput, gw *GatewayConfig, accountID string) (any, error) { return gateway_ec2_instance.DescribeInstances(input, gw.NATSConn, gw.DiscoverActiveNodes(), accountID) }), // ... + volumes, snapshots, VPCs, subnets, route tables, IGWs, NAT gateways, // security groups, network interfaces, elastic IPs, placement groups, key pairs, // images, tags, capacity reservations, spot instances ``` ### 4. NATS Messaging The gateway communicates with daemons via NATS request/response. Most calls go through `utils.NATSRequest`, which marshals the input, attaches the account ID as a NATS header, and unmarshals the typed response: ```go func (s *NATSInstanceService) RunInstances(input *ec2.RunInstancesInput, accountID string) (*ec2.Reservation, error) { topic := fmt.Sprintf("ec2.RunInstances.%s", aws.StringValue(input.InstanceType)) return utils.NATSRequest[ec2.Reservation](s.natsConn, topic, input, 5*time.Minute, accountID) } ``` `RunInstances` uses a per-instance-type subject so NATS only delivers the request to a node with spare capacity for that type — no application-level reject-and-retry. ### 5. Daemon Processing Daemons (`spinifex/daemon/daemon.go`) subscribe to NATS topics and handle requests. A table-driven `subscribeAll()` registers the static EC2/ELBv2 surface at startup: ```go // Queue group → load-balanced (one daemon handles each request) {"ec2.CreateKeyPair", d.handleEC2CreateKeyPair, "spinifex-workers"}, // No queue group → fan-out (every daemon responds) {"ec2.DescribeInstanceTypes", d.handleEC2DescribeInstanceTypes, ""}, ``` `ec2.RunInstances` is dynamic: `ResourceManager` subscribes to `ec2.RunInstances.{type}` only while the node has capacity for that type, and unsubscribes when full. The node-targeted subject `ec2.RunInstances.{type}.{nodeId}` stays subscribed even at full capacity — the launch-time `allocate()` enforces limits there. Per-instance commands flow over `ec2.cmd.{instanceID}`, subscribed only by the owning node. **Queue Groups**: topics with the `spinifex-workers` queue group are load-balanced — only one daemon handles each request. Topics without a queue group fan out to all daemons. ### 6. VM Launch When a daemon handles `RunInstances`: 1. **Resource Check**: Validates CPU/memory availability via `ResourceManager` 2. **Volume Generation**: Creates boot and EFI volumes via Viperblock 3. **Volume Mount**: Sends `ebs.mount` request to Viperblock, receives NBD URI 4. **QEMU Launch**: Builds and executes QEMU command with NBD-backed disks 5. **QMP Monitoring**: Establishes QEMU Machine Protocol connection for VM management 6. **Command subscription**: Subscribes to `ec2.cmd.{instanceID}` for subsequent commands on this VM ```go // Volume mounting via NATS msg, err := d.natsConn.Request("ebs.mount", ebsMountRequest, 10*time.Second) // QEMU launch with NBD storage cmd := exec.Command("qemu-system-x86_64", "-drive", fmt.Sprintf("file=nbd:%s,format=raw", nbdURI), // ... additional QEMU args ) ``` ## Key Components ### Daemon Structure ```go type Daemon struct { node string // Node identifier clusterConfig *config.ClusterConfig // Cluster-wide configuration config *config.Config // Node-specific configuration natsConn *nats.Conn // NATS connection resourceMgr *ResourceManager // CPU/Memory tracking + dynamic RunInstances subscriptions vmMgr *vm.Manager // Local VMs // ... + one service struct per EC2/ELBv2 resource // (instance, key, image, volume, snapshot, tags, vpc, subnet, // igw, eigw, natgw, routetable, eip, placementgroup, elbv2, ...) } ``` ### Resource Manager Tracks available and allocated CPU/memory to prevent overcommit, and drives the dynamic `ec2.RunInstances.{type}` subscriptions: ```go type ResourceManager struct { mu sync.RWMutex hostVCPU int // raw runtime.NumCPU hostMemGB float64 // raw /proc/meminfo MemTotal reservedVCPU int // held back for spinifex services reservedMem float64 // held back for spinifex services allocatedVCPU int allocatedMem float64 instanceTypes map[string]*ec2.InstanceTypeInfo // t3.micro, t3.small, etc. // ... + capacity-reservation accounting, GPU manager, nbdkit memory // overhead tracking, and the dynamic-subscription state } ``` ### Multi-Node Aggregation For operations that need data from all nodes (like `DescribeInstances`), the gateway uses inbox-based fan-out (`spinifex/gateway/ec2/instance/DescribeInstances.go`): ```go func DescribeInstances(...) { // Create unique inbox for collecting responses inbox := nats.NewInbox() sub, _ := natsConn.SubscribeSync(inbox) // Publish to all nodes (no queue group = all daemons respond). // Account ID is propagated as a NATS header so each daemon scopes its response. pubMsg := nats.NewMsg("ec2.DescribeInstances") pubMsg.Reply = inbox pubMsg.Data = jsonData pubMsg.Header.Set(utils.AccountIDHeader, accountID) _ = natsConn.PublishMsg(pubMsg) // Collect responses, returning early once expectedNodes have replied // or the 3s timeout is hit. for { msg, err := sub.NextMsg(3 * time.Second) allReservations = append(allReservations, nodeOutput.Reservations...) } return allReservations } ``` ## Storage Integration ### Viperblock (EBS) Block storage volumes are managed via NATS messages to Viperblock: ```go // Volume mount request type EBSRequest struct { Name string // Volume name (vol-xxxxx) VolType string // gp2, io1, etc. Boot bool // Boot volume flag EFI bool // EFI boot volume NBDURI string // NBD URI returned from mount } ``` Viperblock responds with an NBD URI that QEMU uses to access the block device. ### Predastore (S3) Object storage used for: - AMI image storage - SSH public key storage (`/bucket/ec2/{key-name}.pub`) - Volume metadata and configuration ## Configuration Cluster configuration (`spinifex/config/config.go`): ```go type ClusterConfig struct { Epoch uint64 // Incremented on config changes Node string // This node's identifier Version string // Spinifex version Network NetworkConfig // Cluster-wide external network settings (IP pools, IPsec) Bootstrap BootstrapConfig // Default VPC IDs written by admin init AWS AWSConfig // Cluster-wide AWS-parity settings (region, endpoint suffix) Nodes map[string]Config // Configuration for all nodes } type Config struct { Node, Host, AdvertiseIP string Region, AZ string BaseDir, DataDir, WalDir string Services []string // Which services this node runs locally Daemon DaemonConfig NATS NATSConfig // Broker address, JetStream, ACL token, CA cert AWSGW AWSGWConfig // Gateway host, TLS certs, throttle config Predastore PredastoreConfig // S3 endpoint + service credentials Viperblock ViperblockConfig VPCD VPCDConfig // OVN/OVS endpoints } ``` Configs are generated by `spx admin init` (leader) or `spx admin join` (followers). ## File Structure ``` spinifex/ ├── spinifex/ │ ├── services/ # Long-running services (awsgw, spinifex, viperblockd, │ │ # predastore, nats, spinifexui) │ ├── gateway/ # AWS API surface │ │ ├── gateway.go # chi router, SigV4 auth, throttling │ │ ├── auth.go # SigV4 verification + IAM lookup │ │ ├── ec2.go # EC2 action dispatcher │ │ ├── ec2/ # Per-resource handlers (instance, volume, vpc, ...) │ │ ├── elbv2/, iam/, policy/ │ ├── handlers/ # NATS-side service implementations (ec2, elbv2, iam, ...) │ ├── daemon/ │ │ ├── daemon.go # Daemon entry, NATS subscription table │ │ └── daemon_handlers_*.go # One file per resource family │ ├── admin/ # Cluster bootstrap (init/join, master.key, CA, tokens) │ ├── config/ # Configuration structures │ ├── instancetypes/ # t3.*, m5.*, sys.* definitions │ ├── lbagent/ # In-VM agent for ELB target health │ ├── vm/ # VM instance representation │ ├── vpcd/ # VPC daemon (OVN/OVS network programming) │ └── qmp/ # QEMU Machine Protocol client └── cmd/ ├── spinifex/ # spx CLI ├── lb-agent/ # ELB target agent binary ├── installer/ # Platform installer └── ... # ECS agent, ECR credential provider, EKS helper binaries ``` ## Development ### Running the Stack ```bash sudo spx admin init --node node1 --nodes 1 sudo systemctl start spinifex.target ``` `spx admin init` generates the cluster CA, IAM credentials, TLS certs, and AWS CLI `spinifex` profile, then writes config to `/etc/spinifex/` and data to `/var/lib/spinifex/`. `spinifex.target` brings up all services (NATS, Predastore, Viperblock, vpcd, the daemon, awsgw, the UI) under systemd. Use `spx admin join` for additional nodes. ### Testing with AWS CLI ```bash export AWS_PROFILE=spinifex # Run instance aws ec2 run-instances --image-id ami-debian13 --instance-type t3.micro # List instances (queries all nodes) aws ec2 describe-instances # Terminate instance aws ec2 terminate-instances --instance-ids i-xxxxx ``` For cluster-state inspection that bypasses the AWS API surface, use the `spx` CLI directly: `spx get nodes`, `spx get vms`, `spx top nodes`.