gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-19 10:52:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: rewrite single + multi-node install guides

Browse Source 
Single-node:
- Add auto-detected network topology + external networking section
- Replace hostfwd SSH with VPC overlay private IP SSH
- Add setup-ovn.sh step with --dhcp flag
- Fix undefined variables ($SPINIFEX_AMI, $SPINIFEX_SUBNET)
- Use bash variable capture for VPC/subnet/AMI IDs
- Add three-tier external networking table
- Fix step headings (## → ###)

Multi-node:
- Move CA certificate trust to Step 5 (before starting services)
- Add --external-bridge --dhcp to all OVN setup commands
- Add AMI import step (was missing entirely)
- Add VPC + subnet creation step with variable capture
- Add SSH via private IP step
- Fix sudo inconsistency (all formation commands use sudo)
- Add external networking section with pool upgrade path
- Add DHCP troubleshooting section
- Add port requirements to overview
This commit is contained in:
Ben Duncan
2026-03-20 05:58:38 +00:00
parent 2af3bf5287
commit a554fe4ecf
2 changed files with 269 additions and 99 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/installing-spinifex-multi-node/README.md
+147 -47
View File
@@ -24,6 +24,7 @@ resources:
- [Overview](#overview)
- [Instructions](#instructions)
- [External Networking](#external-networking)
- [Troubleshooting](#troubleshooting)
---
@@ -43,13 +44,14 @@ Cluster formation is automatic. When initializing with `--nodes 3`, the init nod
**Network requirements:**
- Minimum 1 NIC per server
- 2 NICs recommended for production (management + overlay)
- Minimum 1 NIC per server (single-NIC uses macvlan for external bridge)
- 2 NICs recommended for production (management + WAN)
- UDP port 6081 open between hosts (Geneve tunnels)
- TCP ports 4222, 4248, 6641, 6642 open between hosts (NATS, OVN)
## Instructions
## Step 1. Install Spinifex on each server
### Step 1. Install Spinifex on each server
Run the binary installer on **every server** in the cluster:
@@ -57,9 +59,9 @@ Run the binary installer on **every server** in the cluster:
curl https://install.mulgadc.com | bash
```
## Step 2. Set node IP variables
### Step 2. Set node IP variables
On **each server**, export the IPs for all nodes (replace with your actual IPs):
On **each server**, export the IPs for all nodes. Use the management network IP (the interface that other nodes can reach):
```bash
export SPINIFEX_NODE1=192.168.1.10
@@ -70,26 +72,31 @@ export SPINIFEX_NODE3=192.168.1.12
To find your server's IP:
```bash
ip addr show | grep "inet "
ip -4 route get 8.8.8.8 | awk '/src/{print $7}'
```
Next, define the AWS region and AZ you are deploying locally.
Define the region and AZ:
```bash
export AWS_REGION=us-east-1
export AWS_AZ=us-east-1a
```
## Step 3. Setup OVN networking
### Step 3. Setup OVN networking
OVN must be configured on every server before forming the cluster. Server 1 runs OVN central (NB and SB databases) and must be set up first.
**Server 1 — OVN central + compute (run first):**
```bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management --encap-ip=$SPINIFEX_NODE1
sudo /usr/local/share/spinifex/setup-ovn.sh \
--management \
--external-bridge --external-iface=eth1 --dhcp \
--encap-ip=$SPINIFEX_NODE1
```
Replace `eth1` with your WAN interface. The `--dhcp` flag obtains a gateway IP from your router automatically. For single-NIC servers, add `--single-nic`.
Verify OVN central is ready before proceeding:
```bash
@@ -99,13 +106,19 @@ sudo ovn-sbctl show
**Server 2 — Compute node (after server 1 is ready):**
```bash
sudo /usr/local/share/spinifex/setup-ovn.sh --ovn-remote=tcp:$SPINIFEX_NODE1:6642 --encap-ip=$SPINIFEX_NODE2
sudo /usr/local/share/spinifex/setup-ovn.sh \
--external-bridge --external-iface=eth1 --dhcp \
--ovn-remote=tcp:$SPINIFEX_NODE1:6642 \
--encap-ip=$SPINIFEX_NODE2
```
**Server 3 — Compute node (after server 1 is ready):**
```bash
sudo /usr/local/share/spinifex/setup-ovn.sh --ovn-remote=tcp:$SPINIFEX_NODE1:6642 --encap-ip=$SPINIFEX_NODE3
sudo /usr/local/share/spinifex/setup-ovn.sh \
--external-bridge --external-iface=eth1 --dhcp \
--ovn-remote=tcp:$SPINIFEX_NODE1:6642 \
--encap-ip=$SPINIFEX_NODE3
```
Verify all chassis have registered (from server 1):
@@ -116,7 +129,7 @@ sudo ovn-sbctl show
You should see 3 chassis entries, one per server, each with a Geneve encap IP.
## Step 4. Form the cluster
### Step 4. Form the cluster
The formation process requires running init and join commands concurrently. The init node blocks until all expected nodes have joined.
@@ -133,10 +146,12 @@ sudo spx admin init \
--az $AWS_AZ
```
> **Note:** Save the admin credentials printed during init. They will not be shown again. The CA certificate and cluster configuration are automatically distributed to joining nodes.
**Server 2 — Join the cluster (while init is running):**
```bash
spx admin join \
sudo spx admin join \
--node node2 \
--bind $SPINIFEX_NODE2 \
--cluster-bind $SPINIFEX_NODE2 \
@@ -148,7 +163,7 @@ spx admin join \
**Server 3 — Join the cluster (while init is running):**
```bash
spx admin join \
sudo spx admin join \
--node node3 \
--bind $SPINIFEX_NODE3 \
--cluster-bind $SPINIFEX_NODE3 \
@@ -159,7 +174,18 @@ spx admin join \
All three processes will exit with a cluster summary once formation is complete.
## Step 5. Start services and verify
### Step 5. Trust the CA certificate
On **each server**, trust the CA generated during init. Joining nodes receive the CA automatically during formation:
```bash
sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
```
This is required for the AWS CLI and inter-service HTTPS communication to work.
### Step 6. Start services
Start services on **all servers**:
@@ -167,6 +193,14 @@ Start services on **all servers**:
sudo systemctl start spinifex.target
```
Set the AWS profile on each server:
```bash
export AWS_PROFILE=spinifex
```
Add this to `~/.bashrc` for persistence.
From any node, verify the cluster:
```bash
@@ -180,35 +214,68 @@ node2 Ready 192.168.1.11 us-east-1 us-east-1a 2m 0
node3 Ready 192.168.1.12 us-east-1 us-east-1a 2m 0
```
## Step 6. Trust the CA certificate
### Step 7. Import an AMI
On **each server**, trust the CA generated during init:
From any node, list and import an image:
```bash
sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
export AWS_PROFILE=spinifex
spx admin images list
spx admin images import --name debian-12-amd64
```
## Step 7. Launch instances across the cluster
Connect to any node's AWS Gateway — all nodes serve the same cluster state:
Find the imported AMI ID:
```bash
AMI_ID=$(aws ec2 describe-images --query 'Images[0].ImageId' --output text)
```
### Step 8. Create a VPC and launch instances
```bash
# Create a VPC
VPC_ID=$(aws ec2 create-vpc --cidr-block 10.200.0.0/16 \
--query 'Vpc.VpcId' --output text)
# Create a subnet
SUBNET_ID=$(aws ec2 create-subnet --vpc-id $VPC_ID \
--cidr-block 10.200.1.0/24 \
--query 'Subnet.SubnetId' --output text)
# Import your SSH key
aws ec2 import-key-pair --key-name "spinifex-key" \
--public-key-material fileb://~/.ssh/id_rsa.pub
# Launch 3 instances — Spinifex distributes them across nodes
aws ec2 run-instances \
--image-id $SPINIFEX_AMI \
--image-id $AMI_ID \
--instance-type t3.small \
--key-name spinifex-key \
--subnet-id $SPINIFEX_SUBNET \
--subnet-id $SUBNET_ID \
--count 3
```
Spinifex distributes instances across nodes based on available capacity. Check which node each instance landed on:
Check which node each instance landed on:
```bash
spx get vms
```
### Step 9. Connect via SSH
Find an instance's private IP and connect:
```bash
INSTANCE_ID=$(aws ec2 describe-instances \
--query 'Reservations[0].Instances[0].InstanceId' --output text)
PRIVATE_IP=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID \
--query 'Reservations[0].Instances[0].PrivateIpAddress' --output text)
ssh -i ~/.ssh/id_rsa ec2-user@$PRIVATE_IP
```
Instances are reachable via their private IP from any node in the cluster (OVN Geneve overlay handles cross-host routing).
## Shutdown
A graceful shutdown coordinates all nodes — drains VMs, flushes storage, and stops services:
@@ -217,11 +284,35 @@ A graceful shutdown coordinates all nodes — drains VMs, flushes storage, and s
spx admin cluster shutdown
```
---
## External Networking
By default, `setup-ovn.sh --dhcp` obtains a gateway IP from your router, giving VMs outbound internet via SNAT. To enable public IPs for VMs (inbound connectivity), pass `--external-pool` during `admin init`:
```bash
sudo spx admin init \
--node node1 --nodes 3 \
--bind $SPINIFEX_NODE1 --cluster-bind $SPINIFEX_NODE1 \
--region $AWS_REGION --az $AWS_AZ \
--external-pool=192.168.1.150-192.168.1.250
```
Then exclude that range from your router's DHCP scope. See the [VPC Public Subnets](/docs/vpc-public-subnets) guide for details on Elastic IPs, security groups, and public subnet configuration.
| Tier | Setup | What VMs get |
|---|---|---|
| **Auto** (default) | `--dhcp` on setup-ovn.sh | Outbound internet (SNAT) |
| **Pool** | `--external-pool=start-end` on admin init | Public IPs + inbound (1:1 NAT) |
| **Disabled** | `--no-external` on admin init | Overlay only, no internet |
---
## Troubleshooting
## Nodes not joining the cluster
### Nodes not joining the cluster
The init command must still be running when you execute join on other servers. If init has already exited, re-run it.
The init command must still be running when you execute join on other servers. If init has already exited, re-run it with `--force`.
Check that all servers can reach the init node on the formation port:
@@ -229,7 +320,7 @@ Check that all servers can reach the init node on the formation port:
curl -s http://$SPINIFEX_NODE1:4432/health
```
## OVN chassis not registering
### OVN chassis not registering
Compute nodes need to reach OVN central on server 1. Verify from server 1:
@@ -245,7 +336,16 @@ sudo ss -tlnp | grep 6642
Re-run the OVN setup on the affected compute node.
## Cross-host VMs cannot communicate
### CA certificate not trusted
AWS CLI will reject HTTPS connections if the Spinifex CA is not trusted. Re-add the certificate on the affected node:
```bash
sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
```
### Cross-host VMs cannot communicate
This indicates a Geneve tunnel issue. Verify tunnels are established:
@@ -261,23 +361,7 @@ sudo ss -ulnp | grep 6081
If tunnels are missing, verify `--encap-ip` was set correctly during OVN setup.
## Health check fails on a node
Check if the daemon is running and listening on the correct IP:
```bash
curl -s http://$SPINIFEX_NODE1:4432/health
```
Review the daemon logs for errors:
```bash
cat ~/spinifex/logs/spinifex.log
```
Verify `--bind` was set to the correct IP during cluster formation.
## NATS cluster routing issues
### NATS cluster routing issues
Check NATS logs for connection errors between nodes:
@@ -286,3 +370,19 @@ grep -i "route\|cluster" ~/spinifex/logs/nats.log
```
Ensure `--cluster-bind` IPs are reachable between all servers.
### DHCP failed to obtain IP
If `setup-ovn.sh --dhcp` reports a failure, verify a DHCP server is reachable on your WAN:
```bash
sudo dhcpcd --test enp0s3
```
As a fallback, set a static gateway IP in `~/spinifex/config/spinifex.toml` under `[[network.external_pools]]`:
```toml
gateway_ip = "192.168.1.100"
```
Then restart services.
docs/installing-spinifex/README.md
+122 -52
View File
@@ -24,6 +24,7 @@ resources:
- [Overview](#overview)
- [Instructions](#instructions)
- [External Networking](#external-networking)
- [Troubleshooting](#troubleshooting)
---
@@ -37,7 +38,7 @@ This guide installs Spinifex on a single server using the binary installer. For
**Supported Operating Systems:**
- Ubuntu 22.04 / 24.04 / 25.10
- Debian 12.13
- Debian 12 / 13
**What gets installed:**
@@ -48,9 +49,15 @@ This guide installs Spinifex on a single server using the binary installer. For
- Viperblock (EBS-compatible block storage)
- AWS CLI v2
**Network requirements:**
- Minimum 1 NIC (single-NIC uses macvlan for external bridge, SSH-safe)
- 2 NICs recommended (management + WAN)
- Network auto-detection configures everything by default
## Instructions
## Step 1. Install Spinifex
### Step 1. Install Spinifex
```bash
curl https://install.mulgadc.com | bash
@@ -58,36 +65,71 @@ curl https://install.mulgadc.com | bash
The installer downloads the Spinifex binary and bootstraps all dependencies (QEMU, OVN/OVS, AWS CLI).
## Step 2. Initialize a region
### Step 2. Initialize a region
Create your first region and availability zone, replace `ap-southeast-2` with your desired region:
Create your first region and availability zone:
```bash
spx admin init --region ap-southeast-2 --az ap-southeast-2a --node node1 --nodes 1
```
## Step 3. Trust the CA certificate
This auto-detects your network topology and configures external networking:
Spinifex generates a local CA during initialization. Add it to your system trust store:
```
🔍 Detected network topology:
Interface IP Subnet Gateway Role
enp0s3 192.168.1.31 192.168.0.0/23 192.168.1.1 WAN
enp0s5 10.13.7.1 10.13.7.0/24 — LAN
Mode: multi-NIC (1 LAN + 1 WAN)
📡 External networking: nat
WAN interface: enp0s3
Gateway IP: DHCP (obtained during OVN setup)
VMs get: outbound internet via SNAT
```
The init output shows the exact `setup-ovn.sh` command to run next — copy it from the output.
> **Note:** Save the admin credentials printed during init. They will not be shown again.
### Step 3. Trust the CA certificate
Spinifex generates a local CA during initialization. Add it to your system trust store so the AWS CLI trusts Spinifex's HTTPS endpoints:
```bash
sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
```
## Step 4. Start services
### Step 4. Setup OVN networking
Run the `setup-ovn.sh` command printed by `admin init`. For a typical single-node setup:
```bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management --external-bridge --external-iface=enp0s3 --dhcp
```
Replace `enp0s3` with your WAN interface name (shown in the init output). The `--dhcp` flag automatically obtains an IP from your router for OVN's gateway — no manual IP reservation needed.
> **Single-NIC hosts:** If you only have one NIC, `admin init` adds `--single-nic` to the command. This creates a macvlan sub-interface so OVN gets its own IP without disrupting yours.
### Step 5. Start services
```bash
sudo systemctl start spinifex.target
```
Set the AWS profile to use the default admin account:
Set the AWS profile:
```bash
export AWS_PROFILE=spinifex
```
## Step 5. Import an AMI
Add this to your `~/.bashrc` for persistence.
### Step 6. Import an AMI
List available images and import one matching your architecture:
@@ -96,49 +138,92 @@ spx admin images list
spx admin images import --name debian-12-arm64
```
## Step 6. Create a VPC and launch an instance
Find the imported AMI ID:
```bash
aws ec2 create-vpc --cidr-block 10.200.0.0/16
export SPINIFEX_VPC="vpc-XXX"
aws ec2 describe-images --query 'Images[0].ImageId' --output text
```
aws ec2 create-subnet --vpc-id $SPINIFEX_VPC --cidr-block 10.200.1.0/24
export SPINIFEX_SUBNET="subnet-XXX"
### Step 7. Create a VPC and launch an instance
aws ec2 import-key-pair --key-name "spinifex-key" --public-key-material fileb://~/.ssh/id_rsa.pub
```bash
# Create a VPC
VPC_ID=$(aws ec2 create-vpc --cidr-block 10.200.0.0/16 \
--query 'Vpc.VpcId' --output text)
# Create a subnet
SUBNET_ID=$(aws ec2 create-subnet --vpc-id $VPC_ID \
--cidr-block 10.200.1.0/24 \
--query 'Subnet.SubnetId' --output text)
# Import your SSH key
aws ec2 import-key-pair --key-name "spinifex-key" \
--public-key-material fileb://~/.ssh/id_rsa.pub
# Get the AMI ID
AMI_ID=$(aws ec2 describe-images --query 'Images[0].ImageId' --output text)
# Launch an instance
aws ec2 run-instances \
--image-id $SPINIFEX_AMI \
--image-id $AMI_ID \
--instance-type t3.small \
--key-name spinifex-key \
--subnet-id $SPINIFEX_SUBNET \
--subnet-id $SUBNET_ID \
--count 1
```
## Step 7. Connect via SSH
### Step 8. Connect via SSH
Find the forwarded SSH port and connect:
Find the instance's private IP and connect through the VPC network:
```bash
ps auxw | grep hostfwd
# Look for: hostfwd=tcp:127.0.0.1:<port>-:22
INSTANCE_ID=$(aws ec2 describe-instances \
--query 'Reservations[0].Instances[0].InstanceId' --output text)
ssh -i ~/.ssh/spinifex-key ec2-user@127.0.0.1 -p <port>
PRIVATE_IP=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID \
--query 'Reservations[0].Instances[0].PrivateIpAddress' --output text)
ssh -i ~/.ssh/id_rsa ec2-user@$PRIVATE_IP
```
> **Note:** cloud-init takes 30-60 seconds to configure the instance after boot. If SSH is refused, wait and retry.
---
## External Networking
By default, `admin init` auto-detects your network and enables outbound internet for VMs via SNAT. Three tiers are available:
| Tier | Flags | What VMs get |
|---|---|---|
| **Auto** (default) | None | Outbound internet (SNAT via DHCP) |
| **Pool** | `--external-pool=start-end` | Public IPs + inbound (1:1 NAT) |
| **Disabled** | `--no-external` | Overlay only, no internet |
### Enabling public IPs for VMs
To give VMs routable public IPs, reserve an IP range from your network and pass it during init:
```bash
spx admin init --region ap-southeast-2 --az ap-southeast-2a --node node1 --nodes 1 \
--external-pool=192.168.1.150-192.168.1.250
```
Then exclude that range from your router's DHCP scope (e.g., set your router's DHCP to end at 192.168.1.149).
---
## Troubleshooting
## spx command not found
### spx command not found
The binary is not in your PATH. Run the installer again or add the install directory to your PATH:
The binary is not in your PATH. Run the installer again or add the install directory:
```bash
export PATH=$PATH:~/spinifex/bin/
export PATH=$PATH:/usr/local/bin
```
Add this to your `~/.bashrc` or `~/.zshrc` for persistence.
## OVN services not starting
### OVN services not starting
Check if the OVN controller is active:
@@ -147,53 +232,38 @@ sudo systemctl is-active ovn-controller
sudo ovn-sbctl show
```
If inactive, re-run the OVN setup that the installer performs. Check system logs for details:
If inactive, re-run the OVN setup. Check system logs:
```bash
journalctl -u ovn-controller --no-pager -n 20
```
## CA certificate not trusted
### CA certificate not trusted
AWS CLI will reject HTTPS connections if the Spinifex CA is not trusted. Re-add the certificate:
AWS CLI will reject HTTPS connections if the Spinifex CA is not trusted:
```bash
sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
```
Verify it was added:
### Instance stuck in pending
Check the Spinifex daemon logs:
```bash
ls -la /usr/local/share/ca-certificates/spinifex-ca.crt
```
## Instance stuck in pending
Check the Spinifex daemon logs for errors:
```bash
ls ~/spinifex/logs/
cat ~/spinifex/logs/spinifex.log
```
Verify the AMI was imported successfully:
Verify the AMI was imported:
```bash
aws ec2 describe-images
```
## SSH connection refused
### SSH connection refused
cloud-init takes 30-60 seconds to configure the instance after boot. Wait and retry.
Verify the SSH port forwarding is active:
```bash
ps auxw | grep hostfwd
```
If no `hostfwd` entry appears, the instance may not have started correctly. Check instance state:
cloud-init takes 30-60 seconds after boot. Wait and retry. Check instance state:
```bash
aws ec2 describe-instances --instance-ids $INSTANCE_ID