Ben Duncantitle, description, category, tags, resources
| title | description | category | tags | resources | |||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Multi-Node Install | Deploy Spinifex across multiple servers to create an availability zone with high availability, data durability, and fault tolerance. | Install |
|
|
Multi-Node Installation
Deploy Spinifex across multiple servers to create an availability zone.
Table of Contents
Overview
A Spinifex cluster distributes services across multiple servers for high availability, data durability, and fault tolerance. Cluster formation is automatic — the init node waits for peers to join, then distributes credentials, CA certificates, and configuration.
Installing on bare metal? Step 1 below can be done either with the binary installer or by booting each server from the Spinifex ISO — see Bootable USB Install. The ISO installs the operating system, disks and network configuration as well as Spinifex itself, which makes it the better option for servers with no existing OS. Either way, install every server first, then return here and continue from Step 2.
Cluster Sizing:
Three servers is the minimum recommended for a multi-server deployment. At three or more, the OVN control-plane databases run clustered across three nodes and the cluster tolerates the loss of any one of them. Below three, OVN runs standalone on the first node, which becomes a single point of failure for control-plane operations.
A standalone OVN outage is less severe than it sounds. ovn-controller has already programmed the forwarding rules into each host, so running instances keep full networking — east-west, north-south, NAT and security groups all continue. What stops is change: creating VPCs, launching instances, and updating security groups all require the control plane.
Servers beyond the third join as compute nodes and do not run a database, which keeps write latency stable as the cluster grows.
Network Requirements:
- Minimum 1 NIC per server (2 recommended for production)
- UDP port 6081 open between hosts (Geneve tunnels)
- TCP ports 4222, 4248, 6641, 6642 open between hosts (NATS, OVN)
- TCP ports 6643, 6644 open between database servers (OVN database clustering)
Prerequisites
Important
Prerequisite — WAN bridge required on every node.
Before running the installer on any server, that server's WAN interface must already be enslaved to a Linux bridge named
br-wan. The host IP, default route, and DHCP must all live on the bridge — not on the bare NIC.The bootstrap installer does not create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of
br-wanwill land in a future release.Verify on every node before continuing:
ip -br link show br-wan— bridge exists and isUPip route— default route'sdevisbr-wanSetup references: VPC Networking → Bridge Setup for the topology.
Instructions
Step 1. Install Spinifex on Each Server
Choose one of the two methods below and apply it to every server in the cluster.
Option A — existing OS. On a server already running Ubuntu 26.04 or Debian 13:
curl -fsSL https://install.mulgadc.com | bash
Option B — bare metal, from the ISO. Boot each server from the Spinifex ISO and follow Bootable USB Install. This installs the operating system, partitions the disks, and configures the hostname and network interfaces alongside Spinifex. The ISO installer does not form a cluster — that is what the remaining steps do.
Complete this step on all servers before continuing. The nodes must be installed and reachable from one another before the cluster is formed, because Step 4 requires every node to be available at the same time.
Shortcut: form the cluster with one command
Steps 2 to 6 are a fixed sequence, and scripts/install-node.sh in the Spinifex repository runs it for you over SSH — from a workstation that can reach every server, not from the servers themselves:
scripts/install-node.sh \
--external-pool 10.0.1.100-10.0.1.150 \
--external-gateway 10.0.1.1 --external-prefix-len 24 \
server1 server2 server3
The first host initializes and the rest join. It resolves each node's plane addresses, rebuilds the OVN databases in clustered form, forms the cluster, restarts services and verifies the result — and --dry-run prints every command it would run without touching anything. It needs passwordless sudo on each host and requires all of them to be on the same Spinifex version.
The remaining steps document what it does, and are the path to follow when installing by hand or when something needs to be adjusted mid-way.
Step 2. Set Node IP Variables
On each server, export the management IPs for all nodes:
export SPINIFEX_NODE1=192.168.1.10
export SPINIFEX_NODE2=192.168.1.11
export SPINIFEX_NODE3=192.168.1.12
export AWS_REGION=us-east-1
export AWS_AZ=us-east-1a
Converting ISO-Installed Nodes
Skip this section if you installed with the binary installer (Option A).
The ISO installs each server as a running standalone single-node cluster — it initializes Spinifex, starts a standalone OVN database, and brings up spinifex.target at first boot. That is the right behaviour for a single server, and it means forming a cluster is a conversion rather than a fresh setup. Two extra things are needed:
Before Step 3, stop services on every server:
sudo systemctl stop spinifex.target
In Step 4, pass --force to spx admin init and spx admin join. Servers 2 and 3 each arrived with their own CA and master key from the single-node install, and joining replaces them with server 1's. --force is how you confirm that.
That discard is safe here — nothing has been sealed under those keys on a freshly installed node — but it is unrecoverable on a server that has been in service. spx admin join refuses without --force for exactly this reason.
Step 3. Setup OVN Networking
All three servers run a clustered OVN database, so the control plane survives the loss of any one of them. Server 1 creates the cluster and must be set up first; servers 2 and 3 join it.
If your WAN interface is already a bridge, setup-ovn.sh auto-detects it. Otherwise use --wan-bridge=br-wan --wan-iface=eth1 (dedicated WAN NIC).
Important
--recreate-dbdestroys all logical network state — every logical switch, router, port and ACL, and so every VPC on that node. It is required here because theovn-centralpackage starts a standalone database on install, and a clustered database can only be created from scratch. That is safe on a freshly installed server and is not safe on one already running workloads. If you are converting a server that has been in service, back up its NB database first.
Server 1 — create the cluster:
sudo /usr/local/share/spinifex/setup-ovn.sh \
--management \
--db-cluster-local-addr=$SPINIFEX_NODE1 \
--recreate-db \
--encap-ip=$SPINIFEX_NODE1
Server 2 (after server 1 is ready):
sudo /usr/local/share/spinifex/setup-ovn.sh \
--management \
--db-cluster-local-addr=$SPINIFEX_NODE2 \
--db-cluster-remote-addr=$SPINIFEX_NODE1 \
--recreate-db \
--encap-ip=$SPINIFEX_NODE2
Server 3 (after server 1 is ready):
sudo /usr/local/share/spinifex/setup-ovn.sh \
--management \
--db-cluster-local-addr=$SPINIFEX_NODE3 \
--db-cluster-remote-addr=$SPINIFEX_NODE1 \
--recreate-db \
--encap-ip=$SPINIFEX_NODE3
Servers 4 and beyond are compute nodes and run no database. Point them at all three database servers so they survive any one of them failing:
sudo /usr/local/share/spinifex/setup-ovn.sh \
--ovn-remote=tcp:$SPINIFEX_NODE1:6642,tcp:$SPINIFEX_NODE2:6642,tcp:$SPINIFEX_NODE3:6642 \
--encap-ip=$SPINIFEX_NODE4
Verify the database cluster formed, then that all chassis registered:
sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
sudo ovn-sbctl show
cluster/status should list three servers with one leader. If it reports a standalone database, the cluster did not form — re-check that --db-cluster-local-addr was passed on every database server.
Step 4. Form the Cluster
Run init and join concurrently — init blocks until all nodes join.
If you installed from the ISO, add --force to every command below (see Converting ISO-Installed Nodes).
Server 1 — Initialize:
sudo spx admin init \
--node node1 --nodes 3 \
--bind $SPINIFEX_NODE1 --cluster-bind $SPINIFEX_NODE1 \
--port 4432 --region $AWS_REGION --az $AWS_AZ
The init output displays the join command including the token:
📡 Formation server started on 10.0.0.1:4432
Waiting for 2 more node(s) to join...
Token expires in 30m0s
Other nodes should run:
sudo spx admin join --host 10.0.0.1:4432 --token spx_join_a8Bf3x9Kz2mN --node <name> --bind <ip>
Server 2 — Join (while init is running):
sudo spx admin join \
--node node2 --bind $SPINIFEX_NODE2 --cluster-bind $SPINIFEX_NODE2 \
--host $SPINIFEX_NODE1:4432 --token <token-from-init-output> \
--region $AWS_REGION --az $AWS_AZ
Server 3 — Join (while init is running):
sudo spx admin join \
--node node3 --bind $SPINIFEX_NODE3 --cluster-bind $SPINIFEX_NODE3 \
--host $SPINIFEX_NODE1:4432 --token <token-from-init-output> \
--region $AWS_REGION --az $AWS_AZ
Note: The join token expires 30 minutes after init by default. For larger deployments with slower provisioning, use --token-ttl 2h
Step 5. Start Services
On all servers:
sudo systemctl start spinifex.target
Step 6. Verify
From any node:
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
If this returns a list of available instance types, your cluster is working.
Congratulations! Your Spinifex cluster is installed.
Continue to Setting Up Your Cluster to import an AMI, create a VPC, and launch your first instance.
Troubleshooting
Nodes Not Joining
The init command must still be running when join executes. If init exited, re-run with --force.
curl -sk https://$SPINIFEX_NODE1:4432/health
Join Refuses: "this node is already initialized"
The node has its own cluster configuration — normal for anything installed from the ISO, which initializes a single-node cluster at first boot. Joining replaces that node's CA and master key with the primary's, so it must be confirmed with --force.
Safe on a freshly installed node. On one that has been in service it orphans every volume and fragment sealed under the old key, so check before forcing.
OVN Database Cluster Not Forming
sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
If this reports a standalone database rather than three servers, the database was created before the cluster flags were supplied. A clustered database can only be created from scratch — re-run Step 3 with --recreate-db, remembering that it discards all logical network state.
OVN Chassis Not Registering
sudo ovn-sbctl show
sudo ss -tlnp | grep 6642
CA Certificate Not Trusted
sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
Cross-Host VMs Cannot Communicate
sudo ovs-vsctl show | grep -i geneve
sudo ss -ulnp | grep 6081