mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Julian Sommer
340 lines
14 KiB
Markdown
340 lines
14 KiB
Markdown
# Spinifex Architecture
|
||
|
||
Spinifex is an AWS-compatible infrastructure platform for bare-metal, edge, and on-premise deployments. It provides EC2-compatible VM orchestration backed by QEMU/KVM, with storage provided by companion projects Viperblock (EBS) and Predastore (S3).
|
||
|
||
## High-Level Architecture
|
||
|
||
<p align="center">
|
||
<img src="../.github/assets/design-architecture.svg" alt="Spinifex high-level architecture — AWS SDK, gateway, NATS broker, per-node daemons, QEMU/KVM VMs" width="900">
|
||
</p>
|
||
|
||
## Request Flow
|
||
|
||
### 1. AWS SDK Request
|
||
|
||
Users interact with Spinifex using standard AWS SDKs or the AWS CLI by using the spinifex profile.
|
||
|
||
```bash
|
||
AWS_PROFILE=spinifex aws ec2 run-instances \
|
||
--image-id ami-debian13 \
|
||
--instance-type t3.micro \
|
||
--key-name my-keypair
|
||
```
|
||
|
||
The AWS SDK formats this as an HTTPS POST with:
|
||
- AWS SigV4 authentication headers
|
||
- EC2 Query Protocol body (`Action=RunInstances&ImageId=ami-debian13&...`)
|
||
|
||
### 2. AWS Gateway
|
||
|
||
The gateway (`spinifex/services/awsgw/awsgw.go`) is the entry point:
|
||
|
||
```go
|
||
// Connect to NATS (retries while the local broker comes up)
|
||
natsConn, err := utils.ConnectNATSWithRetry(...)
|
||
|
||
// Load IAM (master key + JetStream KV) and create the gateway
|
||
gw := gateway.GatewayConfig{
|
||
NATSConn: natsConn,
|
||
Config: nodeConfig.AWSGW.Config,
|
||
IAMService: iamService,
|
||
// ...
|
||
}
|
||
|
||
// Serve over TLS
|
||
server.ListenAndServeTLS("", "")
|
||
```
|
||
|
||
Request routing (`spinifex/gateway/gateway.go`):
|
||
|
||
1. **Authentication**: SigV4 middleware validates AWS credentials and resolves the account ID
|
||
2. **Throttling**: Per-account+action token bucket rejects bursts post-auth
|
||
3. **Service Detection**: Extracts service name from the Authorization header
|
||
4. **Action Dispatch**: Routes to service-specific handler
|
||
|
||
```go
|
||
switch svc {
|
||
case "ec2":
|
||
err = gw.EC2_Request(w, r)
|
||
case "account":
|
||
err = gw.Account_Request(w, r)
|
||
case "iam":
|
||
err = gw.IAM_Request(w, r)
|
||
case "sts":
|
||
err = gw.STS_Request(w, r)
|
||
case "elasticloadbalancing":
|
||
err = gw.ELBv2_Request(w, r)
|
||
case "eks":
|
||
err = gw.EKS_Request(w, r)
|
||
case "ecs":
|
||
err = gw.ECS_Request(w, r)
|
||
case "ecr":
|
||
err = gw.ECR_Request(w, r)
|
||
case "acm":
|
||
err = gw.ACM_Request(w, r)
|
||
case "tagging":
|
||
err = gw.Tagging_Request(w, r)
|
||
case "spinifex":
|
||
err = gw.Spinifex_Request(w, r)
|
||
}
|
||
```
|
||
|
||
### 3. EC2 Handler
|
||
|
||
The EC2 handler (`spinifex/gateway/ec2.go`) parses the `Action` parameter and delegates to specific handlers via a generic `ec2Handler` wrapper. The account ID resolved by SigV4 auth is threaded into every handler so requests are scoped per IAM principal:
|
||
|
||
```go
|
||
"RunInstances": ec2Handler(func(input *ec2.RunInstancesInput, gw *GatewayConfig, accountID string) (any, error) {
|
||
return gateway_ec2_instance.RunInstances(input, gw.NATSConn, accountID)
|
||
}),
|
||
"DescribeInstances": ec2Handler(func(input *ec2.DescribeInstancesInput, gw *GatewayConfig, accountID string) (any, error) {
|
||
return gateway_ec2_instance.DescribeInstances(input, gw.NATSConn, gw.DiscoverActiveNodes(), accountID)
|
||
}),
|
||
// ... + volumes, snapshots, VPCs, subnets, route tables, IGWs, NAT gateways,
|
||
// security groups, network interfaces, elastic IPs, placement groups, key pairs,
|
||
// images, tags, capacity reservations, spot instances
|
||
```
|
||
|
||
### 4. NATS Messaging
|
||
|
||
The gateway communicates with daemons via NATS request/response. Most calls go through `utils.NATSRequest`, which marshals the input, attaches the account ID as a NATS header, and unmarshals the typed response:
|
||
|
||
```go
|
||
func (s *NATSInstanceService) RunInstances(input *ec2.RunInstancesInput, accountID string) (*ec2.Reservation, error) {
|
||
topic := fmt.Sprintf("ec2.RunInstances.%s", aws.StringValue(input.InstanceType))
|
||
return utils.NATSRequest[ec2.Reservation](s.natsConn, topic, input, 5*time.Minute, accountID)
|
||
}
|
||
```
|
||
|
||
`RunInstances` uses a per-instance-type subject so NATS only delivers the request to a node with spare capacity for that type — no application-level reject-and-retry.
|
||
|
||
### 5. Daemon Processing
|
||
|
||
Daemons (`spinifex/daemon/daemon.go`) subscribe to NATS topics and handle requests. A table-driven `subscribeAll()` registers the static EC2/ELBv2 surface at startup:
|
||
|
||
```go
|
||
// Queue group → load-balanced (one daemon handles each request)
|
||
{"ec2.CreateKeyPair", d.handleEC2CreateKeyPair, "spinifex-workers"},
|
||
|
||
// No queue group → fan-out (every daemon responds)
|
||
{"ec2.DescribeInstanceTypes", d.handleEC2DescribeInstanceTypes, ""},
|
||
```
|
||
|
||
`ec2.RunInstances` is dynamic: `ResourceManager` subscribes to `ec2.RunInstances.{type}` only while the node has capacity for that type, and unsubscribes when full. The node-targeted subject `ec2.RunInstances.{type}.{nodeId}` stays subscribed even at full capacity — the launch-time `allocate()` enforces limits there. Per-instance commands flow over `ec2.cmd.{instanceID}`, subscribed only by the owning node.
|
||
|
||
**Queue Groups**: topics with the `spinifex-workers` queue group are load-balanced — only one daemon handles each request. Topics without a queue group fan out to all daemons.
|
||
|
||
### 6. VM Launch
|
||
|
||
When a daemon handles `RunInstances`:
|
||
|
||
1. **Resource Check**: Validates CPU/memory availability via `ResourceManager`
|
||
2. **Volume Generation**: Creates boot and EFI volumes via Viperblock
|
||
3. **Volume Mount**: Sends `ebs.mount` request to Viperblock, receives NBD URI
|
||
4. **QEMU Launch**: Builds and executes QEMU command with NBD-backed disks
|
||
5. **QMP Monitoring**: Establishes QEMU Machine Protocol connection for VM management
|
||
6. **Command subscription**: Subscribes to `ec2.cmd.{instanceID}` for subsequent commands on this VM
|
||
|
||
```go
|
||
// Volume mounting via NATS
|
||
msg, err := d.natsConn.Request("ebs.mount", ebsMountRequest, 10*time.Second)
|
||
|
||
// QEMU launch with NBD storage
|
||
cmd := exec.Command("qemu-system-x86_64",
|
||
"-drive", fmt.Sprintf("file=nbd:%s,format=raw", nbdURI),
|
||
// ... additional QEMU args
|
||
)
|
||
```
|
||
|
||
## Key Components
|
||
|
||
### Daemon Structure
|
||
|
||
```go
|
||
type Daemon struct {
|
||
node string // Node identifier
|
||
clusterConfig *config.ClusterConfig // Cluster-wide configuration
|
||
config *config.Config // Node-specific configuration
|
||
natsConn *nats.Conn // NATS connection
|
||
resourceMgr *ResourceManager // CPU/Memory tracking + dynamic RunInstances subscriptions
|
||
vmMgr *vm.Manager // Local VMs
|
||
// ... + one service struct per EC2/ELBv2 resource
|
||
// (instance, key, image, volume, snapshot, tags, vpc, subnet,
|
||
// igw, eigw, natgw, routetable, eip, placementgroup, elbv2, ...)
|
||
}
|
||
```
|
||
|
||
### Resource Manager
|
||
|
||
Tracks available and allocated CPU/memory to prevent overcommit, and drives the dynamic `ec2.RunInstances.{type}` subscriptions:
|
||
|
||
```go
|
||
type ResourceManager struct {
|
||
mu sync.RWMutex
|
||
hostVCPU int // raw runtime.NumCPU
|
||
hostMemGB float64 // raw /proc/meminfo MemTotal
|
||
reservedVCPU int // held back for spinifex services
|
||
reservedMem float64 // held back for spinifex services
|
||
allocatedVCPU int
|
||
allocatedMem float64
|
||
instanceTypes map[string]*ec2.InstanceTypeInfo // t3.micro, t3.small, etc.
|
||
// ... + capacity-reservation accounting, GPU manager, nbdkit memory
|
||
// overhead tracking, and the dynamic-subscription state
|
||
}
|
||
```
|
||
|
||
### Multi-Node Aggregation
|
||
|
||
For operations that need data from all nodes (like `DescribeInstances`), the gateway uses inbox-based fan-out (`spinifex/gateway/ec2/instance/DescribeInstances.go`):
|
||
|
||
```go
|
||
func DescribeInstances(...) {
|
||
// Create unique inbox for collecting responses
|
||
inbox := nats.NewInbox()
|
||
sub, _ := natsConn.SubscribeSync(inbox)
|
||
|
||
// Publish to all nodes (no queue group = all daemons respond).
|
||
// Account ID is propagated as a NATS header so each daemon scopes its response.
|
||
pubMsg := nats.NewMsg("ec2.DescribeInstances")
|
||
pubMsg.Reply = inbox
|
||
pubMsg.Data = jsonData
|
||
pubMsg.Header.Set(utils.AccountIDHeader, accountID)
|
||
_ = natsConn.PublishMsg(pubMsg)
|
||
|
||
// Collect responses, returning early once expectedNodes have replied
|
||
// or the 3s timeout is hit.
|
||
for {
|
||
msg, err := sub.NextMsg(3 * time.Second)
|
||
allReservations = append(allReservations, nodeOutput.Reservations...)
|
||
}
|
||
|
||
return allReservations
|
||
}
|
||
```
|
||
|
||
## Storage Integration
|
||
|
||
### Viperblock (EBS)
|
||
|
||
Block storage volumes are managed via NATS messages to Viperblock:
|
||
|
||
```go
|
||
// Volume mount request
|
||
type EBSRequest struct {
|
||
Name string // Volume name (vol-xxxxx)
|
||
VolType string // gp2, io1, etc.
|
||
Boot bool // Boot volume flag
|
||
EFI bool // EFI boot volume
|
||
NBDURI string // NBD URI returned from mount
|
||
}
|
||
```
|
||
|
||
Viperblock responds with an NBD URI that QEMU uses to access the block device.
|
||
|
||
### Predastore (S3)
|
||
|
||
Object storage used for:
|
||
- AMI image storage
|
||
- SSH public key storage (`/bucket/ec2/{key-name}.pub`)
|
||
- Volume metadata and configuration
|
||
|
||
Spinifex runs Predastore in-process: `spx service predastore start` loads `/etc/spinifex/predastore/predastore.toml`, selects the `[[host]]` naming this machine, and serves it until the process is signalled. The topology is written by `spx admin init` / `join` from the servers that form the cluster.
|
||
|
||
One `[[host]]` is one process. The nodes pinned to it under `[[host.node]]` carry the roles:
|
||
|
||
| Role | Port | Responsibility |
|
||
|------|------|----------------|
|
||
| `gate` | TCP 8443 | The S3 API. Holds no cluster state; dials the meta and blob nodes to serve a request. |
|
||
| `blob` | UDP 6660 | Erasure-coded object shards, encrypted at rest under this host's master key. |
|
||
| `meta` | UDP 7660 | Raft replica of global state — buckets and the object index. |
|
||
|
||
Ports are unique within a host but repeat across the cluster, so a multi-node cluster puts one node of each role on every machine and every machine uses the same three ports. Nodes sharing a host talk over an in-process pipe and bind no socket; nodes on other hosts are reached over QUIC, authenticated by the cluster CA.
|
||
|
||
A single-node install is the same arrangement with one host: seven nodes — a gate, three blob and three meta — colocated in one process, needing seven distinct ports (8443, 6660–6662, 7660–7662) to stay unique within the host even though only the gate opens a listener. It has no code path of its own.
|
||
|
||
The daemon reports the topology on `spinifex.storage.config` by parsing the config file. It does not connect to Predastore to do so, and no Predastore node exposes a status endpoint.
|
||
|
||
## Configuration
|
||
|
||
Cluster configuration (`spinifex/config/config.go`):
|
||
|
||
```go
|
||
type ClusterConfig struct {
|
||
Epoch uint64 // Incremented on config changes
|
||
Node string // This node's identifier
|
||
Version string // Spinifex version
|
||
Network NetworkConfig // Cluster-wide external network settings (IP pools, IPsec)
|
||
Bootstrap BootstrapConfig // Default VPC IDs written by admin init
|
||
AWS AWSConfig // Cluster-wide AWS-parity settings (region, endpoint suffix)
|
||
Nodes map[string]Config // Configuration for all nodes
|
||
}
|
||
|
||
type Config struct {
|
||
Node, Host, AdvertiseIP string
|
||
Region, AZ string
|
||
BaseDir, DataDir, WalDir string
|
||
Services []string // Which services this node runs locally
|
||
Daemon DaemonConfig
|
||
NATS NATSConfig // Broker address, JetStream, ACL token, CA cert
|
||
AWSGW AWSGWConfig // Gateway host, TLS certs, throttle config
|
||
Predastore PredastoreConfig // S3 endpoint + service credentials
|
||
Viperblock ViperblockConfig
|
||
VPCD VPCDConfig // OVN/OVS endpoints
|
||
}
|
||
```
|
||
|
||
Configs are generated by `spx admin init` (leader) or `spx admin join` (followers).
|
||
|
||
## File Structure
|
||
|
||
```
|
||
spinifex/
|
||
├── cmd/
|
||
│ ├── spinifex/ # spx CLI: service entry points and `spx admin`
|
||
│ ├── installer/ # Platform installer
|
||
│ └── ... # In-guest agents (lb, ecs, rds, eks helpers) and dev tools
|
||
├── spinifex/
|
||
│ ├── services/ # Long-running services (awsgw, spinifex, nats, predastore, ...)
|
||
│ ├── gateway/ # AWS API surface: routing, SigV4 auth, per-service dispatch
|
||
│ ├── handlers/ # NATS-side service implementations (ec2, iam, eks, rds, ...)
|
||
│ ├── daemon/ # Per-node daemon and its NATS subscription table
|
||
│ ├── vm/, qmp/ # VM lifecycle and the QEMU Machine Protocol client
|
||
│ ├── vpcd/, network/ # VPC daemon and the layered OVN/OVS network stack
|
||
│ ├── admin/ # Cluster bootstrap (init/join, master.key, CA, tokens)
|
||
│ └── config/ # Configuration structures
|
||
├── internal/ # In-guest SigV4 clients and the AWS Smithy models
|
||
├── build/ # systemd units, node helper scripts, microVM build
|
||
├── scripts/ # Install, dev-environment, image-build scripts
|
||
└── tests/
|
||
├── integration/ # Gateway router against embedded NATS JetStream
|
||
└── e2e/ # Live-cluster suites (see docs/testing/e2e-coverage.md)
|
||
```
|
||
|
||
## Development
|
||
|
||
### Running the Stack
|
||
|
||
```bash
|
||
sudo spx admin init --node node1 --nodes 1
|
||
sudo systemctl start spinifex.target
|
||
```
|
||
|
||
`spx admin init` generates the cluster CA, IAM credentials, TLS certs, and AWS CLI `spinifex` profile, then writes config to `/etc/spinifex/` and data to `/var/lib/spinifex/`. `spinifex.target` brings up all services (NATS, Predastore, Viperblock, vpcd, the daemon, awsgw, the UI) under systemd. Use `spx admin join` for additional nodes.
|
||
|
||
### Testing with AWS CLI
|
||
|
||
```bash
|
||
export AWS_PROFILE=spinifex
|
||
|
||
# Run instance
|
||
aws ec2 run-instances --image-id ami-debian13 --instance-type t3.micro
|
||
|
||
# List instances (queries all nodes)
|
||
aws ec2 describe-instances
|
||
|
||
# Terminate instance
|
||
aws ec2 terminate-instances --instance-ids i-xxxxx
|
||
```
|
||
|
||
For cluster-state inspection that bypasses the AWS API surface, use the `spx` CLI directly: `spx get nodes`, `spx get vms`, `spx top nodes`.
|