spinifex/docs/install/install-multi-node/README.md

title, seoTitle, description, category, tags, resources

title	seoTitle	description	category	tags	resources
Multi-Node Install	Multi-Node Spinifex Cluster Install — Spinifex Docs	Deploy Spinifex across several servers to form an availability zone with automatic cluster formation, high availability, data durability, and fault tolerance.	Install	install multi node cluster iso bare-metal	title url Bootable USB Install /docs/install-usb title url Spinifex Repository https://github.com/mulgadc/spinifex title url Predastore (S3) https://github.com/mulgadc/predastore title url Viperblock (EBS) https://github.com/mulgadc/viperblock

Multi-Node Installation

Deploy Spinifex across multiple servers to create an availability zone.

Table of Contents

• Overview
• Object Storage Layout
• Prerequisites
• Instructions
• Converting ISO-Installed Nodes
• Firewall and Cluster Membership
• Troubleshooting

---

Overview

A Spinifex cluster distributes services across multiple servers for high availability, data durability, and fault tolerance. Cluster formation is automatic — the init node waits for peers to join, then distributes credentials, CA certificates, and configuration.

Installing on bare metal? Step 1 below can be done either with the binary installer or by booting each server from the Spinifex ISO — see Bootable USB Install. The ISO installs the operating system, disks and network configuration as well as Spinifex itself, which makes it the better option for servers with no existing OS. Either way, install every server first, then return here and continue from Step 2.

Cluster Sizing:

Three servers is the minimum recommended for a multi-server deployment. At three or more, the OVN control-plane databases run clustered across three nodes and the cluster tolerates the loss of any one of them. Below three, OVN runs standalone on the first node, which becomes a single point of failure for control-plane operations.

A standalone OVN outage is less severe than it sounds. ovn-controller has already programmed the forwarding rules into each host, so running instances keep full networking — east-west, north-south, NAT and security groups all continue. What stops is change: creating VPCs, launching instances, and updating security groups all require the control plane.

Servers beyond the third join as compute nodes and do not run a database, which keeps write latency stable as the cluster grows.

Network Requirements:

• Minimum 1 NIC per server (2 recommended for production)
• UDP port 6081 open between hosts (Geneve tunnels)
• TCP ports 4222, 4248, 6641, 6642 open between hosts (NATS, OVN)
• TCP ports 6643, 6644 open between database servers (OVN database clustering)
• TCP port 8443 and UDP ports 6660, 7660 open between hosts (Predastore S3, object shards, metadata consensus)

Predastore uses the same three ports on every server, so the surface does not widen as the cluster grows. See Object Storage Layout below.

Object Storage Layout

Predastore is configured for the whole cluster in /etc/spinifex/predastore/predastore.toml. Each server is one [[host]] — a single Predastore process owning that machine's data directory and TLS identity — carrying three nodes under [[host.node]]:

Role	Port	Purpose
gate	TCP 8443	Serves the S3 API. Every server runs one, so any of them answers an S3 request.
blob	UDP 6660	Holds erasure-coded object shards. One per machine.
meta	UDP 7660	Member of the Raft quorum over global state — buckets and the object index.

Ports have to be unique within a host but not across the cluster, so every machine uses the same three. Blob and meta traffic between hosts runs over QUIC, authenticated by the cluster CA; nodes on the same machine talk over an in-process pipe and open no socket, which is why a single-server install listens on 8443 alone.

Reed-Solomon parameters are chosen from the cluster size, since each machine contributes exactly one blob node: two servers get RS(1,1), three or more get RS(2,1). RS(2,1) survives the loss of any one server's shards.

You do not configure any of this by hand. spx admin init and spx admin join build the topology from the servers that actually form the cluster in Step 4, and each machine gets the same file with its own host ID recorded in spinifex.toml.

Warning

There is no upgrade path onto this layout. A cluster installed before the object storage cutover cannot be migrated — the on-disk directory structure and the configuration schema both changed, and no migration rewrites either. Such a cluster has to be re-initialised from scratch, which discards its stored objects. Export anything you need first.

Prerequisites

Important

Prerequisite — WAN bridge required on every node.

Before running the installer on any server, that server's WAN interface must already be enslaved to a Linux bridge named br-wan. The host IP, default route, and DHCP must all live on the bridge — not on the bare NIC.

The bootstrap installer does not create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of br-wan will land in a future release.

Verify on every node before continuing:

• ip -br link show br-wan — bridge exists and is UP
• ip route — default route's dev is br-wan

Setup references: VPC Networking → Bridge Setup for the topology.

Instructions

Step 1. Install Spinifex on Each Server

Choose one of the two methods below and apply it to every server in the cluster.

Option A — existing OS. On a server already running Ubuntu 26.04 or Debian 13:

curl -fsSL https://install.mulgadc.com | bash

Option B — bare metal, from the ISO. Boot each server from the Spinifex ISO and follow Bootable USB Install. This installs the operating system, partitions the disks, and configures the hostname and network interfaces alongside Spinifex. The ISO installer does not form a cluster — that is what the remaining steps do.

Complete this step on all servers before continuing. The nodes must be installed and reachable from one another before the cluster is formed, because Step 4 requires every node to be available at the same time.

Shortcut: form the cluster with one command

Steps 2 to 6 are a fixed sequence, and scripts/install-node.sh in the Spinifex repository runs it for you over SSH — from a workstation that can reach every server, not from the servers themselves:

scripts/install-node.sh \
  --external-pool 10.0.1.100-10.0.1.150 \
  --external-gateway 10.0.1.1 --external-prefix-len 24 \
  server1 server2 server3

The first host initializes and the rest join. It resolves each node's plane addresses, rebuilds the OVN databases in clustered form, forms the cluster, restarts services and verifies the result — and --dry-run prints every command it would run without touching anything. It needs passwordless sudo on each host and requires all of them to be on the same Spinifex version.

The remaining steps document what it does, and are the path to follow when installing by hand or when something needs to be adjusted mid-way.

Step 2. Set Node IP Variables

On each server, export the management IPs for all nodes:

export SPINIFEX_NODE1=192.168.1.10
export SPINIFEX_NODE2=192.168.1.11
export SPINIFEX_NODE3=192.168.1.12
export AWS_REGION=us-east-1
export AWS_AZ=us-east-1a

Converting ISO-Installed Nodes

Skip this section if you installed with the binary installer (Option A).

The ISO installs each server as a running standalone single-node cluster — it initializes Spinifex, starts a standalone OVN database, brings up spinifex.target, and arms the host firewall at first boot. That is the right behaviour for a single server, and it means forming a cluster is a conversion rather than a fresh setup. Three extra things are needed:

Before Step 3, stop services on every server:

sudo systemctl stop spinifex.target

Before Step 3, also turn the firewall off on every server:

sudo /usr/local/lib/spinifex/spinifex-firewall-apply disable

Each server's firewall currently allows internal cluster traffic only from itself, because that is the whole cluster as far as it knows. Step 3 has servers 2 and 3 connect to server 1's OVN database, and Step 4 has them connect to its formation server — neither is recognised yet, so both are blocked until the firewall is out of the way. Stopping spinifex.target does not do this: the firewall lives in the kernel and outlives the services.

Turn it back on after Step 6 — see Firewall and cluster membership.

In Step 4, pass --force to spx admin init and spx admin join. Servers 2 and 3 each arrived with their own CA and master key from the single-node install, and joining replaces them with server 1's. --force is how you confirm that.

That discard is safe here — nothing has been sealed under those keys on a freshly installed node — but it is unrecoverable on a server that has been in service. spx admin join refuses without --force for exactly this reason.

Step 3. Setup OVN Networking

All three servers run a clustered OVN database, so the control plane survives the loss of any one of them. Server 1 creates the cluster and must be set up first; servers 2 and 3 join it.

If your WAN interface is already a bridge, setup-ovn.sh auto-detects it. Otherwise use --wan-bridge=br-wan --wan-iface=eth1 (dedicated WAN NIC).

Important

--recreate-db destroys all logical network state — every logical switch, router, port and ACL, and so every VPC on that node. It is required here because the ovn-central package starts a standalone database on install, and a clustered database can only be created from scratch. That is safe on a freshly installed server and is not safe on one already running workloads. If you are converting a server that has been in service, back up its NB database first.

Server 1 — create the cluster:

sudo /usr/local/share/spinifex/setup-ovn.sh \
  --management \
  --db-cluster-local-addr=$SPINIFEX_NODE1 \
  --recreate-db \
  --encap-ip=$SPINIFEX_NODE1

Server 2 (after server 1 is ready):

sudo /usr/local/share/spinifex/setup-ovn.sh \
  --management \
  --db-cluster-local-addr=$SPINIFEX_NODE2 \
  --db-cluster-remote-addr=$SPINIFEX_NODE1 \
  --recreate-db \
  --encap-ip=$SPINIFEX_NODE2

Server 3 (after server 1 is ready):

sudo /usr/local/share/spinifex/setup-ovn.sh \
  --management \
  --db-cluster-local-addr=$SPINIFEX_NODE3 \
  --db-cluster-remote-addr=$SPINIFEX_NODE1 \
  --recreate-db \
  --encap-ip=$SPINIFEX_NODE3

Servers 4 and beyond are compute nodes and run no database. Point them at all three database servers so they survive any one of them failing:

sudo /usr/local/share/spinifex/setup-ovn.sh \
  --ovn-remote=tcp:$SPINIFEX_NODE1:6642,tcp:$SPINIFEX_NODE2:6642,tcp:$SPINIFEX_NODE3:6642 \
  --encap-ip=$SPINIFEX_NODE4

Verify the database cluster formed, then that all chassis registered:

sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
sudo ovn-sbctl show

cluster/status should list three servers with one leader. If it reports a standalone database, the cluster did not form — re-check that --db-cluster-local-addr was passed on every database server.

Step 4. Form the Cluster

Run init and join concurrently — init blocks until all nodes join.

If you installed from the ISO, add --force to every command below (see Converting ISO-Installed Nodes).

Server 1 — Initialize:

sudo spx admin init \
  --node node1 --nodes 3 \
  --bind $SPINIFEX_NODE1 --cluster-bind $SPINIFEX_NODE1 \
  --port 4432 --region $AWS_REGION --az $AWS_AZ

The init output displays the join command including the token:

📡 Formation server started on 10.0.0.1:4432
   Waiting for 2 more node(s) to join...
   Token expires in 30m0s

   Other nodes should run:
   sudo spx admin join --host 10.0.0.1:4432 --token spx_join_a8Bf3x9Kz2mN --node <name> --bind <ip>

Server 2 — Join (while init is running):

sudo spx admin join \
  --node node2 --bind $SPINIFEX_NODE2 --cluster-bind $SPINIFEX_NODE2 \
  --host $SPINIFEX_NODE1:4432 --token <token-from-init-output> \
  --region $AWS_REGION --az $AWS_AZ

Server 3 — Join (while init is running):

sudo spx admin join \
  --node node3 --bind $SPINIFEX_NODE3 --cluster-bind $SPINIFEX_NODE3 \
  --host $SPINIFEX_NODE1:4432 --token <token-from-init-output> \
  --region $AWS_REGION --az $AWS_AZ

Note: The join token expires 30 minutes after init by default. For larger deployments with slower provisioning, use --token-ttl 2h

Step 5. Start Services

On all servers:

sudo systemctl start spinifex.target

Step 6. Verify

From any node:

export AWS_PROFILE=spinifex
aws ec2 describe-instance-types

If this returns a list of available instance types, your cluster is working.

Congratulations! Your Spinifex cluster is installed.

Continue to Setting Up Your Cluster to import an AMI, create a VPC, and launch your first instance.

Firewall and Cluster Membership

Spinifex ships an optional host firewall. It divides the node's ports into two groups:

	Ports	Who can reach them
Public	SSH, 443, 3000 (console), 8443 (S3), 9999 (AWS gateway), 53 (DNS)	anyone
Internal	OVN, NATS, formation, Geneve and the rest of the cluster plane	cluster members only

The internal group is the point. Before this existed, OVN and NATS were reachable from the public internet on a WAN-facing node.

"Cluster members" is not a list you maintain. Each node works it out from the cluster it belongs to and rewrites its own rules whenever membership changes — you never edit the peer list by hand.

Is it on?

How the node was installed	Firewall
From the ISO	on
curl | bash or setup.sh	off
setup.sh --firewall=on	on

The binary installer defaults to off deliberately: it runs on servers that already have an operating system and services on them, and switching on a default-deny policy uninvited could cut off something Spinifex knows nothing about. For production, turn it on — either at install time:

curl -fsSL https://install.mulgadc.com | bash -s -- --firewall=on

or afterwards, by setting it in /etc/spinifex/spinifex.toml and restarting the daemon:

[network]
firewall_enabled = true

Before you do, check what else the machine is serving. Anything listening on a port outside the public group above stops accepting new connections.

Turning it off and on around cluster changes

A node only recognises the members of the cluster it currently belongs to, so during formation — when the nodes do not yet know each other — internal traffic between them is blocked. Turn the firewall off while you form the cluster, and on again once it is up:

# Off — before forming or expanding a cluster. Run on every node.
sudo /usr/local/lib/spinifex/spinifex-firewall-apply disable

# On — once the cluster is formed and verified. Run on every node.
sudo systemctl restart spinifex-daemon

Restarting the daemon is what re-arms it: the node rebuilds its peer list from the cluster it is now part of, reloads the rules, and re-enables the boot-time unit so the policy survives a reboot. It also happens on its own within five minutes if you would rather wait.

Checking it

sudo nft list table inet spinifex_filter | grep spinifex_peers

Every node's addresses should be listed — on a multi-NIC node, that is its WAN, LAN and VPC addresses, so expect several entries per node. A missing node means its cluster traffic is being dropped.

Dropped packets are logged, rate-limited, so this tells you whether a connection problem is the firewall or something else:

sudo journalctl -k | grep 'spinifex-fw drop'

Adding a node later

Expanding an existing cluster is the same shape as forming one: the node you are adding is not a member yet. Turn the firewall off on the existing nodes and on the new one, add the node, then re-arm all of them. Verify with the nft list command above that every node now lists the newcomer.

Troubleshooting

Nodes Not Joining

The init command must still be running when join executes. If init exited, re-run with --force.

curl -sk https://$SPINIFEX_NODE1:4432/health

If that command hangs rather than failing quickly, it is the firewall on node 1 — a refused connection means nothing is listening, but a hang means the packets are being dropped. Node 1 does not recognise the joining node as a cluster member yet. Confirm it on node 1:

sudo journalctl -k | grep 'spinifex-fw drop'

Turn the firewall off on every node and retry the join, then re-arm once the cluster is up — see Firewall and cluster membership. The joining node retries for 20 minutes by default, so it is often still waiting while you fix this.

Join Refuses: "this node is already initialized"

The node has its own cluster configuration — normal for anything installed from the ISO, which initializes a single-node cluster at first boot. Joining replaces that node's CA and master key with the primary's, so it must be confirmed with --force.

Safe on a freshly installed node. On one that has been in service it orphans every volume and fragment sealed under the old key, so check before forcing.

OVN Database Cluster Not Forming

sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound

If this reports a standalone database rather than three servers, the database was created before the cluster flags were supplied. A clustered database can only be created from scratch — re-run Step 3 with --recreate-db, remembering that it discards all logical network state.

OVN Chassis Not Registering

sudo ovn-sbctl show
sudo ss -tlnp | grep 6642

CA Certificate Not Trusted

On a node or any host running spx/aws against the cluster:

sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates

Inside a guest VM there is no /etc/spinifex; fetch the CA from IMDS instead:

sudo curl -fsS http://169.254.169.254/spinifex/ca.pem \
  -o /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates

Cross-Host VMs Cannot Communicate

sudo ovs-vsctl show | grep -i geneve
sudo ss -ulnp | grep 6081