spinifex/docs/installing-spinifex-multi-node

title, description, category, tags, badge, resources

title	description	category	tags	badge	resources
Multi-Node Installation	Deploy Spinifex across multiple servers to create an availability zone with high availability, data durability, and fault tolerance.	Getting Started	install multi node cluster	availability-zone	title url Spinifex Repository https://github.com/mulgadc/spinifex title url Predastore (S3) https://github.com/mulgadc/predastore title url Viperblock (EBS) https://github.com/mulgadc/viperblock

Multi-Node Installation

Deploy Spinifex across multiple servers to create an availability zone.

Table of Contents

• Overview
• Instructions
• External Networking
• Troubleshooting

---

Overview

A Spinifex cluster operates as a fully distributed infrastructure region — similar to an AWS Region — where multiple nodes provide high availability, data durability, and fault tolerance.

Services distributed across nodes:

• NATS — Clustered message bus for request routing and JetStream replication
• Predastore — Raft consensus with Reed-Solomon erasure coding (RS 2+1)
• Viperblock — Block storage co-located with compute nodes
• Spinifex Daemon — Independent AWS API request serving per node

Cluster formation is automatic. When initializing with --nodes 3, the init node starts a formation server and waits for peers to join. Once all nodes register, each receives the full cluster topology — credentials, CA certificates, NATS routes, Predastore peer lists — no manual configuration synchronization required.

Network requirements:

• Minimum 1 NIC per server (single-NIC uses macvlan for external bridge)
• 2 NICs recommended for production (management + WAN)
• UDP port 6081 open between hosts (Geneve tunnels)
• TCP ports 4222, 4248, 6641, 6642 open between hosts (NATS, OVN)

Instructions

Step 1. Install Spinifex on each server

Run the binary installer on every server in the cluster:

curl https://install.mulgadc.com | bash

Step 2. Set node IP variables

On each server, export the IPs for all nodes. Use the management network IP (the interface that other nodes can reach):

export SPINIFEX_NODE1=192.168.1.10
export SPINIFEX_NODE2=192.168.1.11
export SPINIFEX_NODE3=192.168.1.12

To find your server's IP:

ip -4 route get 8.8.8.8 | awk '/src/{print $7}'

Define the region and AZ:

export AWS_REGION=us-east-1
export AWS_AZ=us-east-1a

Step 3. Setup OVN networking

OVN must be configured on every server before forming the cluster. Server 1 runs OVN central (NB and SB databases) and must be set up first.

Server 1 — OVN central + compute (run first):

sudo /usr/local/share/spinifex/setup-ovn.sh \
  --management \
  --external-bridge --external-iface=eth1 --dhcp \
  --encap-ip=$SPINIFEX_NODE1

Replace eth1 with your WAN interface. The --dhcp flag obtains a gateway IP from your router automatically. For single-NIC servers, add --single-nic.

Verify OVN central is ready before proceeding:

sudo ovn-sbctl show

Server 2 — Compute node (after server 1 is ready):

sudo /usr/local/share/spinifex/setup-ovn.sh \
  --external-bridge --external-iface=eth1 --dhcp \
  --ovn-remote=tcp:$SPINIFEX_NODE1:6642 \
  --encap-ip=$SPINIFEX_NODE2

Server 3 — Compute node (after server 1 is ready):

sudo /usr/local/share/spinifex/setup-ovn.sh \
  --external-bridge --external-iface=eth1 --dhcp \
  --ovn-remote=tcp:$SPINIFEX_NODE1:6642 \
  --encap-ip=$SPINIFEX_NODE3

Verify all chassis have registered (from server 1):

sudo ovn-sbctl show

You should see 3 chassis entries, one per server, each with a Geneve encap IP.

Step 4. Form the cluster

The formation process requires running init and join commands concurrently. The init node blocks until all expected nodes have joined.

Server 1 — Initialize the cluster:

sudo spx admin init \
  --node node1 \
  --nodes 3 \
  --bind $SPINIFEX_NODE1 \
  --cluster-bind $SPINIFEX_NODE1 \
  --port 4432 \
  --region $AWS_REGION \
  --az $AWS_AZ

Note: Save the admin credentials printed during init. They will not be shown again. The CA certificate and cluster configuration are automatically distributed to joining nodes.

Server 2 — Join the cluster (while init is running):

sudo spx admin join \
  --node node2 \
  --bind $SPINIFEX_NODE2 \
  --cluster-bind $SPINIFEX_NODE2 \
  --host $SPINIFEX_NODE1:4432 \
  --region $AWS_REGION \
  --az $AWS_AZ

Server 3 — Join the cluster (while init is running):

sudo spx admin join \
  --node node3 \
  --bind $SPINIFEX_NODE3 \
  --cluster-bind $SPINIFEX_NODE3 \
  --host $SPINIFEX_NODE1:4432 \
  --region $AWS_REGION \
  --az $AWS_AZ

All three processes will exit with a cluster summary once formation is complete.

Step 5. Trust the CA certificate

On each server, trust the CA generated during init. Joining nodes receive the CA automatically during formation:

sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates

This is required for the AWS CLI and inter-service HTTPS communication to work.

Step 6. Start services

Start services on all servers:

sudo systemctl start spinifex.target

Set the AWS profile on each server:

export AWS_PROFILE=spinifex

Add this to ~/.bashrc for persistence.

From any node, verify the cluster:

spx get nodes

NAME    STATUS    IP              REGION      AZ           UPTIME   VMs
node1   Ready     192.168.1.10    us-east-1   us-east-1a   2m       0
node2   Ready     192.168.1.11    us-east-1   us-east-1a   2m       0
node3   Ready     192.168.1.12    us-east-1   us-east-1a   2m       0

Step 7. Import an AMI

From any node, list and import an image:

spx admin images list
spx admin images import --name debian-12-amd64

Find the imported AMI ID:

AMI_ID=$(aws ec2 describe-images --query 'Images[0].ImageId' --output text)

Step 8. Create a VPC and launch instances

# Create a VPC
VPC_ID=$(aws ec2 create-vpc --cidr-block 10.200.0.0/16 \
  --query 'Vpc.VpcId' --output text)

# Create a subnet
SUBNET_ID=$(aws ec2 create-subnet --vpc-id $VPC_ID \
  --cidr-block 10.200.1.0/24 \
  --query 'Subnet.SubnetId' --output text)

# Import your SSH key
aws ec2 import-key-pair --key-name "spinifex-key" \
  --public-key-material fileb://~/.ssh/id_rsa.pub

# Launch 3 instances — Spinifex distributes them across nodes
aws ec2 run-instances \
  --image-id $AMI_ID \
  --instance-type t3.small \
  --key-name spinifex-key \
  --subnet-id $SUBNET_ID \
  --count 3

Check which node each instance landed on:

spx get vms

Step 9. Connect via SSH

Find an instance's private IP and connect:

INSTANCE_ID=$(aws ec2 describe-instances \
  --query 'Reservations[0].Instances[0].InstanceId' --output text)

PRIVATE_IP=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID \
  --query 'Reservations[0].Instances[0].PrivateIpAddress' --output text)

ssh -i ~/.ssh/id_rsa ec2-user@$PRIVATE_IP

Instances are reachable via their private IP from any node in the cluster (OVN Geneve overlay handles cross-host routing).

Shutdown

A graceful shutdown coordinates all nodes — drains VMs, flushes storage, and stops services:

spx admin cluster shutdown

---

External Networking

By default, setup-ovn.sh --dhcp obtains a gateway IP from your router, giving VMs outbound internet via SNAT. To enable public IPs for VMs (inbound connectivity), pass --external-pool during admin init:

sudo spx admin init \
  --node node1 --nodes 3 \
  --bind $SPINIFEX_NODE1 --cluster-bind $SPINIFEX_NODE1 \
  --region $AWS_REGION --az $AWS_AZ \
  --external-pool=192.168.1.150-192.168.1.250

Then exclude that range from your router's DHCP scope. See the VPC Public Subnets guide for details on Elastic IPs, security groups, and public subnet configuration.

Tier	Setup	What VMs get
Auto (default)	--dhcp on setup-ovn.sh	Outbound internet (SNAT)
Pool	--external-pool=start-end on admin init	Public IPs + inbound (1:1 NAT)
Disabled	--no-external on admin init	Overlay only, no internet

---

Troubleshooting

Nodes not joining the cluster

The init command must still be running when you execute join on other servers. If init has already exited, re-run it with --force.

Check that all servers can reach the init node on the formation port:

curl -s http://$SPINIFEX_NODE1:4432/health

OVN chassis not registering

Compute nodes need to reach OVN central on server 1. Verify from server 1:

sudo ovn-sbctl show

If chassis are missing, check firewall rules for port 6642:

sudo ss -tlnp | grep 6642

Re-run the OVN setup on the affected compute node.

CA certificate not trusted

AWS CLI will reject HTTPS connections if the Spinifex CA is not trusted. Re-add the certificate on the affected node:

sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates

Cross-host VMs cannot communicate

This indicates a Geneve tunnel issue. Verify tunnels are established:

sudo ovs-vsctl show | grep -i geneve

Ensure UDP port 6081 is open between all hosts:

sudo ss -ulnp | grep 6081

If tunnels are missing, verify --encap-ip was set correctly during OVN setup.

NATS cluster routing issues

Check NATS logs for connection errors between nodes:

grep -i "route\|cluster" ~/spinifex/logs/nats.log

Ensure --cluster-bind IPs are reachable between all servers.

DHCP failed to obtain IP

If setup-ovn.sh --dhcp reports a failure, verify a DHCP server is reachable on your WAN:

sudo dhcpcd --test enp0s3

As a fallback, set a static gateway IP in ~/spinifex/config/spinifex.toml under [[network.external_pools]]:

gateway_ip = "192.168.1.100"

Then restart services.