Ben Duncan
455c97a2e7
* fix: read NB_Global.ipsec through the NB remote list so followers and compute nodes answer quietly * fix: stop probing and nudging the EIP datapath of a stopped instance's port * fix: log the real client IP, request ID and duration_ms in the awsgw access line * test: mark the UI proxy header test as in-package * fix: discard pre-formation JetStream data when forming a multi-node cluster A node that ran as its own single-node cluster carries KV streams into formation. nats-server adopts a local stream with the same name as the clustered one, and raft catch-up only fetches sequences beyond a follower's last, so the conflicting prefix is never repaired and replicas diverge permanently. - spx admin join removes the local JetStream store once formation succeeds. A store holding streams now needs --force, and the identity-discard warning names it. - spx admin init --nodes >= 2 refuses a store that holds streams unless --discard-jetstream is given, and removes it once every node has joined. Keeping it as the seed is unsafe: stream leadership ignores who holds data, and unassigned local streams are deleted as orphans. - Neither removes a store while `spx service nats` runs, --force included. - install-node.sh passes --discard-jetstream and waits for every `spx service` process to exit before formation. * feat: add spx admin kv digest for per-replica JetStream consistency checks Replicas of a stream can hold different content while raft reports them current, and nothing in the cluster detects it. A direct get over NATS is no way to inspect one replica: a client's request goes to a random member of the direct-get queue group, routed replicas included. kv digest copies this node's $G streams tree (without consumer state) to a temporary directory, opens the copy in a private in-process server with no listener, and prints first/last sequence, message count and a SHA-256 over every stored message per stream. Running it on each node and diffing the output shows which streams diverge; --seqs locates the first differing sequence. The live store is only ever read. * fix: let spx admin kv digest read replicated streams A replicated stream's meta.inf carries num_replicas > 1, which a standalone server refuses to recover, so the digest could not read any production store. Rewrite the copy's config to one replica with no expiry and a recomputed checksum, and test it against a three-node cluster and an expired bucket. * test: use assert.ErrorIs for the removed JetStream store * test: mark the JetStream store and digest tests as in-package * fix: stop the process-group test misreading a reaped child as alive The test polled until the child was gone, then checked once more. A task being reaped shows state X for an instant, which processRunning counted as running, so that second look failed about 1 run in 50 under -race. * fix: wait for the recovery retry loop before removing a test's data dir restoreInstances starts the retry loop untracked, and it rewrites the local state file when it ends. A write landing during t.TempDir cleanup failed TestRestoreInstances_* with RemoveAll: directory not empty.
24 KiB
title, seoTitle, description, category, tags, resources
| title | seoTitle | description | category | tags | resources | ||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Multi-Node Install | Multi-Node Spinifex Cluster Install — Spinifex Docs | Deploy Spinifex across three or more servers to form an availability zone with clustered OVN, replicated object storage, and automatic cluster formation. | Install |
|
|
Multi-Node Installation
Deploy Spinifex across three or more servers to create an availability zone.
Table of Contents
- Overview
- Cluster sizing
- Hardware
- Network requirements
- Prerequisites
- Instructions
- Verifying the cluster
- Firewall and cluster membership
- How multi-node storage works
- Troubleshooting
Overview
Important
This guide builds a three-node cluster — the minimum we recommend for any multi-server deployment. Every step below is written for three servers and uses
$SPINIFEX_NODE1,$SPINIFEX_NODE2and$SPINIFEX_NODE3.Running more than three? Install the first three exactly as described, then repeat Steps 1, 2, 3 and the join in Step 4 for each additional server. The only difference is in Step 3: the OVN database cluster stays at three members, so servers four and beyond point at those three instead of joining them.
A Spinifex cluster distributes services across multiple servers for high availability, data durability, and fault tolerance. Cluster formation is automatic — the init node waits for its peers to join, then distributes credentials, CA certificates, and configuration.
Installing on bare metal? Step 1 can be done either with the binary installer or by booting each server from the Spinifex ISO — see Bootable USB Install. The ISO installs the operating system, disks and network configuration as well as Spinifex itself, which makes it the better option for servers with no existing OS. Either way, install every server first, then return here and continue from Step 2.
Cluster sizing
Three servers is the minimum we recommend. Three is the point at which every distributed layer can lose a node and keep running:
| Layer | What three servers gives you |
|---|---|
| VPC networking — OVN | Control-plane databases run clustered, surviving the loss of any one node. |
| Object storage — Predastore (S3) | Objects are erasure coded RS(2,1), surviving the loss of any one node's shards. |
| Block storage — Viperblock (EBS) | Volumes are stored in Predastore, so they inherit the same durability. |
On one or two servers none of that holds. OVN runs standalone on the first node, and the storage metadata quorum has no majority to lose. If that node goes down, running instances keep full networking — but nothing can change: no new VPCs, no launches, no security group edits. See OVN control plane on multi-node clusters.
Servers beyond the third run the full set of services — storage, gateway and networking agents — and add their capacity to the pool. What they do not do is join the OVN database cluster, which stays at three members, so write latency there stays flat as the cluster grows.
Hardware
Per server, for the three-node minimum:
| Minimum | Recommended | |
|---|---|---|
| Nodes | 3 | 3 or more |
| RAM | 32 GB | 128 GB |
| CPU | 16 cores | 32+ cores |
| OS / Spinifex disk | SSD | NVMe |
| NICs | 2 — WAN 1 GbE, LAN/VPC 10 GbE+ | 2 — WAN 10 GbE, LAN/VPC 25 GbE+ |
Two NICs matters more than the raw numbers. One carries WAN traffic; the other carries LAN and VPC traffic between nodes — Geneve tunnels, object shards and OVN replication all cross it, so it is the interface that wants the bandwidth. A single-NIC server will work, but inter-node storage and tunnel traffic then competes with everything going in and out of the cluster.
Network requirements
Open between all hosts:
| Protocol and port | Used by |
|---|---|
| UDP 6081 | Geneve tunnels |
| TCP 4222, 4248 | NATS |
| TCP 6641, 6642 | OVN northbound and southbound |
| TCP 8443 | Predastore S3 gate |
| UDP 6660, 7660 | Object shards, metadata consensus |
Open between the three OVN database servers (servers 1 to 3) only:
| Protocol and port | Used by |
|---|---|
| TCP 6643, 6644 | OVN database clustering |
Predastore uses the same three ports on every server, so the surface does not widen as the cluster grows. See How multi-node storage works.
Those are the ports your network has to permit between the servers — switches, upstream firewalls, and anything else in the path.
Spinifex's own host firewall is separate, and already carries this policy. It ships armed on the ISO path and off on the binary installer path. Two things follow from that, both of which the ISO box in Step 2 acts on:
- The formation port needs nothing from you.
spx admin initopens 4432 to any source for the length of the formation window and closes it again afterwards, because a node dialling in to join is not a peer yet. The handshake behind it is TLS 1.3 with a bearer token. - The rest of the cluster plane is peer-scoped, and that includes the OVN database ports Step 3 uses. Nodes that do not yet know each other cannot reach them, which is why an ISO-installed node has its firewall taken down before Step 3 and re-armed after Step 6.
Prerequisites
Installed from the ISO (Option B)? The bridge is configured for you — the ISO sets up the host's network interfaces, br-wan included. Skip to Instructions.
Important
Binary installer only — a WAN bridge is required on every node.
Before running the installer on any server, that server's WAN interface must already be enslaved to a Linux bridge named
br-wan. The host IP, default route, and DHCP must all live on the bridge — not on the bare NIC.The binary installer does not create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of
br-wanwill land in a future release.Verify on every node before continuing:
ip -br link show br-wan— bridge exists and isUPip route— default route'sdevisbr-wanSee VPC Networking → Bridge Setup for the topology.
Instructions
Step 1. Install Spinifex on Each Server
Choose one method and apply it to every server in the cluster.
Option A — existing OS. On a server already running Ubuntu 26.04 or Debian 13:
curl -fsSL https://install.mulgadc.com | bash
Option B — bare metal, from the ISO. Boot each server from the Spinifex ISO and follow Bootable USB Install. This installs the operating system, partitions the disks, and configures the hostname and network interfaces alongside Spinifex. The ISO installer does not form a cluster — that is what the remaining steps do.
Complete this step on all three servers before continuing. Step 4 requires every node to be installed, reachable, and available at the same time.
Step 2. Set Node IP Variables
On each server, export the management IPs of all three nodes plus the region and AZ. The same values go on every server:
export SPINIFEX_NODE1=192.168.1.10
export SPINIFEX_NODE2=192.168.1.11
export SPINIFEX_NODE3=192.168.1.12
export AWS_REGION=us-east-1
export AWS_AZ=us-east-1a
Adding a fourth server or more? Export SPINIFEX_NODE4 and so on alongside these — the first three stay as they are, because they remain the OVN database nodes.
Note
ISO installs only — skip this box if you used the binary installer (Option A).
The ISO brings each server up as a running standalone single-node cluster with its firewall armed, so forming a cluster is a conversion rather than a fresh setup. One thing is needed, on every server, before Step 3:
sudo systemctl stop spinifex.target sudo /usr/local/lib/spinifex/spinifex-firewall-apply disableEach node's firewall currently trusts only itself, because that is the whole cluster as far as it knows, and the cluster plane is peer-scoped — so the OVN database connections Step 3 makes between servers are dropped. Stopping
spinifex.targetis not enough: the firewall lives in the kernel and outlives the services.Turn it back on after Step 6 — see Firewall and cluster membership.
Step 3. Set Up OVN Networking
Servers 1 to 3 run a clustered OVN database — the VPC networking control plane — so it survives losing any one of them. This is the only database limited to three members; storage and NATS run on every server. Server 1 creates the cluster and must be set up first; servers 2 and 3 then join it.
--recreate-db appears in each command below because ovn-central starts a standalone OVN database when the package installs, and a clustered one can only be created from scratch. It replaces that standalone database on both the binary and ISO paths.
If your WAN interface is already a bridge, setup-ovn.sh auto-detects it. Otherwise add --wan-bridge=br-wan --wan-iface=eth1 for a dedicated WAN NIC.
Server 1 — create the cluster:
sudo /usr/local/share/spinifex/setup-ovn.sh \
--management \
--db-cluster-local-addr=$SPINIFEX_NODE1 \
--recreate-db \
--encap-ip=$SPINIFEX_NODE1
Server 2 (after server 1 is ready):
sudo /usr/local/share/spinifex/setup-ovn.sh \
--management \
--db-cluster-local-addr=$SPINIFEX_NODE2 \
--db-cluster-remote-addr=$SPINIFEX_NODE1 \
--recreate-db \
--encap-ip=$SPINIFEX_NODE2
Server 3 (after server 1 is ready):
sudo /usr/local/share/spinifex/setup-ovn.sh \
--management \
--db-cluster-local-addr=$SPINIFEX_NODE3 \
--db-cluster-remote-addr=$SPINIFEX_NODE1 \
--recreate-db \
--encap-ip=$SPINIFEX_NODE3
Servers 4 and beyond — repeat for each one, substituting its own address. They point at the three OVN database servers rather than joining the database cluster, so they survive any one of those failing:
sudo /usr/local/share/spinifex/setup-ovn.sh \
--ovn-remote=tcp:$SPINIFEX_NODE1:6642,tcp:$SPINIFEX_NODE2:6642,tcp:$SPINIFEX_NODE3:6642 \
--encap-ip=$SPINIFEX_NODE4
Verify the OVN database cluster formed, then that every chassis registered with it:
sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
sudo ovn-sbctl show
cluster/status should list three servers with one leader, and ovn-sbctl show a chassis for every node in the cluster. If cluster/status reports a standalone database, the OVN cluster did not form — re-check that --db-cluster-local-addr was passed on servers 1, 2 and 3.
Step 4. Form the Cluster
Run init and join concurrently — init blocks until all nodes have joined.
--force is in every command below so the sequence is identical whichever way you installed. It does the work on ISO-installed servers, which arrive as their own single-node cluster: joining replaces that server's CA and master key with server 1's and removes its JetStream store, and --force is the confirmation. On spx admin init it is idempotent — existing keys, credentials and CA are preserved, and only the config files and server certificate are refreshed. On a freshly installed server there is nothing to lose either way.
--discard-jetstream on server 1 removes the streams its single-node cluster wrote, once every server has joined. Without it init refuses to form a cluster over them, because NATS would adopt each server's own copy as a replica and those copies never converge. Stop spinifex.target on every server first and confirm pgrep -af 'spx service' prints nothing: init and join both refuse while NATS is still running.
Warning
Do not point these commands at a server that has already been in service. Joining discards its master key, orphaning every volume and fragment sealed under it. That is what
--forceoverrides, and it is unrecoverable.
Server 1 — initialize:
sudo spx admin init --force --discard-jetstream \
--node node1 --nodes 3 \
--bind $SPINIFEX_NODE1 --cluster-bind $SPINIFEX_NODE1 \
--port 4432 --region $AWS_REGION --az $AWS_AZ
--nodes 3 is the number of servers init waits for. Set it to your total node count if you are building a larger cluster.
The init output displays the join command including the token:
📡 Formation server started on 10.0.0.1:4432
Waiting for 2 more node(s) to join...
Token expires in 30m0s
Other nodes should run:
sudo spx admin join --host 10.0.0.1:4432 --token spx_join_a8Bf3x9Kz2mN --node <name> --bind <ip>
Take the token from that output, but run the commands below rather than the line it prints — they add --force and --cluster-bind.
Server 2 — join (while init is still running):
sudo spx admin join --force \
--node node2 --bind $SPINIFEX_NODE2 --cluster-bind $SPINIFEX_NODE2 \
--host $SPINIFEX_NODE1:4432 --token <token-from-init-output> \
--region $AWS_REGION --az $AWS_AZ
Server 3 — join (while init is still running):
sudo spx admin join --force \
--node node3 --bind $SPINIFEX_NODE3 --cluster-bind $SPINIFEX_NODE3 \
--host $SPINIFEX_NODE1:4432 --token <token-from-init-output> \
--region $AWS_REGION --az $AWS_AZ
Each additional server runs the same join command with its own --node name and --bind address.
Note: the join token expires 30 minutes after init by default. For larger deployments with slower provisioning, use --token-ttl 2h.
Step 5. Start Services
On all servers:
sudo systemctl start spinifex.target
Step 6. Verify the Cluster
Run these from any node. Together they confirm that every server joined, that services are healthy on each, and that capacity is being pooled across the cluster.
1. Every node is present and Ready.
spx get nodes
spinifex@node1:~$ spx get nodes
NAME | STATUS | ROLES | IP | REGION | AZ | UPTIME | VMs | SERVICES
node1 | Ready | nats:follower | 10.2.0.2 | us-east-1 | us-east-1a | 21h27m | 0 | nats,predastore,viperblock,daemon,awsgw,vpcd,ui
node2 | Ready | nats:follower | 10.2.0.3 | us-east-1 | us-east-1a | 21h27m | 1 | nats,predastore,viperblock,daemon,awsgw,vpcd,ui
node3 | Ready | nats:leader | 10.2.0.4 | us-east-1 | us-east-1a | 21h27m | 0 | nats,predastore,viperblock,daemon,awsgw,vpcd,ui
What to check:
- Every server you installed is listed. A missing node never joined — see Nodes not joining.
STATUSisReadyon all of them.NotReadymeans the node is in the cluster configuration but is not answering, so start withspinifex.targeton that host.- Exactly one
nats:leader. The rest are followers. SERVICESlists the same set on every node. A short list means something failed to start there; checksystemctl statusfor the missing unit.
2. Capacity is pooled across the cluster.
spx top nodes
spinifex@node1:~$ spx top nodes
NAME | CPU (used/total) | MEM (used/total) | GPU (used/total) | VMs
node1 | 0/64 | 0Mi/220.2Gi | - | 0
node2 | 2/64 | 2.8Gi/251.7Gi | - | 1
node3 | 0/64 | 0Mi/251.7Gi | - | 0
INSTANCE TYPE | AVAILABLE | VCPU | MEMORY
c6a.12xlarge | 3 | 48 | 96.0Gi
c6a.16xlarge | 0 | 64 | 128.0Gi
c6a.24xlarge | 0 | 96 | 192.0Gi
c6a.2xlarge | 21 | 8 | 16.0Gi
c6a.4xlarge | 9 | 16 | 32.0Gi
c6a.8xlarge | 3 | 32 | 64.0Gi
c6a.large | 92 | 2 | 4.0Gi
c6a.xlarge | 45 | 4 | 8.0Gi
c6i.12xlarge | 3 | 48 | 96.0Gi
c6i.16xlarge | 0 | 64 | 128.0Gi
c6i.24xlarge | 0 | 96 | 192.0Gi
c6i.2xlarge | 21 | 8 | 16.0Gi
c6i.4xlarge | 9 | 16 | 32.0Gi
c6i.8xlarge | 3 | 32 | 64.0Gi
c6i.large | 92 | 2 | 4.0Gi
c6i.xlarge | 45 | 4 | 8.0Gi
m6a.12xlarge | 3 | 48 | 192.0Gi
The top table is per-node CPU, memory and GPU usage. The bottom table is what the cluster can actually launch right now: AVAILABLE is the number of instances of that type that would currently fit across all nodes. A type showing 0 does not fit on any single node — instances are not split across servers, so the largest type you can launch is bounded by your biggest node, not by the cluster total.
If capacity looks like a single server rather than the sum of your nodes, the others have not joined.
3. The AWS API answers.
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
A list of instance types means the gateway, IAM and the cluster behind them are all working.
Congratulations! Your Spinifex cluster is installed.
Continue to Setting Up Your Cluster to import an AMI, create a VPC, and launch your first instance.
Firewall and Cluster Membership
Spinifex ships an optional host firewall. It divides the node's ports into two groups:
| Group | Ports | Who can reach them |
|---|---|---|
| Public | SSH, 443, 3000 (console), 8443 (S3), 9999 (AWS gateway), 53 (DNS) | anyone |
| Internal | OVN, NATS, formation, Geneve and the rest of the cluster plane | cluster members only |
The internal group is the point. Before this existed, OVN and NATS were reachable from the public internet on a WAN-facing node.
"Cluster members" is not a list you maintain. Each node works it out from the cluster it belongs to and rewrites its own rules whenever membership changes — you never edit the peer list by hand.
Is it on?
| How the node was installed | Firewall |
|---|---|
| From the ISO | on |
Binary installer (curl | bash) or setup.sh |
off |
setup.sh --firewall=on |
on |
The binary installer defaults to off deliberately: it runs on servers that already have an operating system and services on them, and switching on a default-deny policy uninvited could cut off something Spinifex knows nothing about. For production, turn it on — either at install time:
curl -fsSL https://install.mulgadc.com | bash -s -- --firewall=on
or afterwards, by setting it in /etc/spinifex/spinifex.toml and restarting the daemon:
[network]
firewall_enabled = true
Before you do, check what else the machine is serving. Anything listening on a port outside the public group above stops accepting new connections.
Turning it off and on around cluster changes
A node only recognises the members of the cluster it currently belongs to, so during formation — when the nodes do not yet know each other — internal traffic between them is blocked. Turn the firewall off while you form the cluster, and on again once it is up:
# Off — before forming or expanding a cluster. Run on every node.
sudo /usr/local/lib/spinifex/spinifex-firewall-apply disable
# On — once the cluster is formed and verified. Run on every node.
sudo systemctl restart spinifex-daemon
Restarting the daemon is what re-arms it: the node rebuilds its peer list from the cluster it is now part of, reloads the rules, and re-enables the boot-time unit so the policy survives a reboot. It also happens on its own within five minutes if you would rather wait.
Checking it
sudo nft list table inet spinifex_filter
The peer list is an nft variable, expanded when the rules load, so it does not appear under a name of its own. Look instead at the ip saddr { ... } addresses on the cluster-plane rules — the ones accepting 4222, 6641, 6642 and the rest.
Every node's addresses should be there. On a multi-NIC node that means its WAN, LAN and VPC addresses, so expect several entries per node. A missing node means its cluster traffic is being dropped.
Dropped packets are logged, rate-limited, so this tells you whether a connection problem is the firewall or something else:
sudo journalctl -k | grep 'spinifex-fw drop'
How Multi-Node Storage Works
Background reading — you do not configure any of this by hand. spx admin init and spx admin join build the topology from the servers that actually form the cluster in Step 4, and each machine gets the same file with its own host ID recorded in spinifex.toml.
Predastore is configured for the whole cluster in /etc/spinifex/predastore/predastore.toml. Each server is one [[host]] — a single Predastore process owning that machine's data directory and TLS identity — carrying three nodes under [[host.node]]:
| Role | Port | Purpose |
|---|---|---|
gate |
TCP 8443 | Serves the S3 API. Every server runs one, so any of them answers an S3 request. |
blob |
UDP 6660 | Holds erasure-coded object shards. One per machine. |
meta |
UDP 7660 | Member of the Raft quorum over global state — buckets and the object index. |
Ports have to be unique within a host but not across the cluster, so every machine uses the same three. Blob and meta traffic between hosts runs over QUIC, authenticated by the cluster CA; nodes on the same machine talk over an in-process pipe and open no socket, which is why a single-server install listens on 8443 alone.
Reed-Solomon parameters are chosen from the cluster size, since each machine contributes exactly one blob node: two servers get RS(1,1), three or more get RS(2,1). RS(2,1) survives the loss of any one server's shards — another reason three nodes is the recommended minimum.
Troubleshooting
Nodes Not Joining
The init command must still be running when join executes. If init exited, re-run with --force.
curl -sk https://$SPINIFEX_NODE1:4432/health
A hang rather than a quick failure means packets are being dropped; a refused connection means nothing is listening.
If it hangs, check node 1's init output before blaming the firewall — spx admin init opens the formation port itself while it waits, so this is usually not the cause. It prints ⚠️ Could not open port 4432 in the host firewall when that fails, which is the case where it is. Confirm on node 1:
sudo journalctl -k | grep 'spinifex-fw drop'
Turn the firewall off on every node and retry the join, then re-arm once the cluster is up — see Firewall and cluster membership. The joining node retries for 20 minutes by default, so it is often still waiting while you fix this.
Join Refuses: "this node is already initialized"
The node has its own cluster configuration — normal for anything installed from the ISO, which initializes a single-node cluster at first boot. Joining replaces that node's CA and master key with the primary's, so it must be confirmed with --force.
Safe on a freshly installed node. On one that has been in service it orphans every volume and fragment sealed under the old key, so check before forcing.
OVN Database Cluster Not Forming
sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
If this reports a standalone database rather than three servers, the OVN database was created before the cluster flags were supplied. A clustered one can only be created from scratch — re-run Step 3 with --recreate-db.
OVN Chassis Not Registering
sudo ovn-sbctl show
sudo ss -tlnp | grep 6642
CA Certificate Not Trusted
On a node or any host running spx/aws against the cluster:
sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
Inside a guest VM there is no /etc/spinifex; fetch the CA from IMDS instead:
sudo curl -fsS http://169.254.169.254/spinifex/ca.pem \
-o /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
Cross-Host VMs Cannot Communicate
sudo ovs-vsctl show | grep -i geneve
sudo ss -ulnp | grep 6081