fix: docs

This commit is contained in:
Julian Sommer
2026-03-26 10:29:48 +10:00
parent fca9d018fc
commit e6b48ce170
13 changed files with 111 additions and 149 deletions
+1 -1
View File
@@ -44,7 +44,7 @@ aws sts get-caller-identity
## Troubleshooting
## Credentials not working
## Credentials Not Working
Verify the AWS CLI configuration files exist and contain the correct profile:
+2 -12
View File
@@ -30,16 +30,6 @@ resources:
This guide walks through the complete workflow for creating a production-ready Spinifex environment.
**Steps:**
1. Create a user account
2. Import or create SSH keys
3. Create a VPC and subnet
4. Select and import an AMI
5. Launch EC2 instances
6. Connect via SSH
7. Monitor instance health
## Instructions
## 1. Import SSH Key
@@ -88,7 +78,7 @@ ssh -i ~/.ssh/spinifex-key ec2-user@127.0.0.1 -p <port>
## Troubleshooting
## Instance stuck in pending
## Instance Stuck in Pending
Check the Spinifex daemon and QEMU logs for boot errors:
@@ -103,7 +93,7 @@ Verify the AMI exists and architecture matches your host:
aws ec2 describe-images --image-ids $SPINIFEX_AMI
```
## SSH connection refused
## SSH Connection Refused
cloud-init takes 30-60 seconds to configure networking and SSH after boot. Wait and retry.
+2 -2
View File
@@ -51,7 +51,7 @@ rsync -avz /data/ user@cloud-server:/backup/spinifex-data/
## Troubleshooting
## Sync fails mid-transfer
## Sync Fails Mid Transfer
S3 sync is idempotent — re-run the same command to resume where it left off:
@@ -62,7 +62,7 @@ aws s3 sync s3://local-bucket/ s3://cloud-bucket/ \
Only changed or missing files will be transferred on subsequent runs.
## Permissions errors during sync
## Permission Errors During Sync
Verify your AWS credentials are configured for both the source and destination:
+4 -5
View File
@@ -8,7 +8,6 @@ tags:
- policies
- access keys
- security
badge: setup
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
@@ -475,7 +474,7 @@ AWS_PROFILE=spinifex-carol aws ec2 terminate-instances --instance-ids i-123
## Troubleshooting
### AccessDenied on IAM commands
### AccessDenied on IAM Commands
The calling user must have IAM permissions. Attach a policy with `iam:*` actions, or use the root account profile:
@@ -501,7 +500,7 @@ aws iam update-access-key --user-name alice \
If the key was deleted, create a new one.
### DeleteConflict when deleting a user
### DeleteConflict When Deleting a User
The user still has access keys or attached policies. Remove them first:
@@ -513,7 +512,7 @@ aws iam list-access-keys --user-name alice
aws iam list-attached-user-policies --user-name alice
```
### DeleteConflict when deleting a policy
### DeleteConflict When Deleting a Policy
The policy is still attached to one or more users. Detach it from all users before deleting:
@@ -526,7 +525,7 @@ aws iam delete-policy \
--policy-arn arn:aws:iam::000000000001:policy/MyPolicy
```
### LimitExceeded when creating access keys
### LimitExceeded When Creating Access Keys
Each user can have at most 2 access keys. Delete an existing key before creating a new one:
+7 -8
View File
@@ -6,7 +6,6 @@ tags:
- install
- air-gapped
- offline
badge: advanced
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
@@ -38,7 +37,7 @@ In air-gapped or disconnected environments, Spinifex can be deployed without int
## Instructions
## Step 1. Prepare packages (on a connected machine)
## Step 1. Prepare Packages (on a connected machine)
Set up a local APT cache to collect all required packages:
@@ -56,7 +55,7 @@ sudo apt install --download-only nbdkit nbdkit-plugin-dev pkg-config \
ovn-central ovn-host openvswitch-switch
```
## Step 2. Create USB deployment media
## Step 2. Create USB Deployment Media
```bash
mkdir -p /media/spinifex-deploy/{apt-packages,go-cache,spinifex-source}
@@ -65,7 +64,7 @@ cp -r ~/go/pkg/mod/cache/ /media/spinifex-deploy/go-cache/
cp -r ~/Development/mulga/spinifex /media/spinifex-deploy/spinifex-source/
```
## Step 3. Verify package integrity
## Step 3. Verify Package Integrity
Sign and verify packages before transferring to the target:
@@ -75,7 +74,7 @@ gpg --verify spinifex-v1.0.tar.gz.sig spinifex-v1.0.tar.gz
sha256sum -c /media/spinifex-deploy/checksums.sha256
```
## Step 4. Install on the target server
## Step 4. Install on the Target Server
Mount the USB and install:
@@ -90,7 +89,7 @@ Then follow the [Source Install](/docs/source-install) guide from Step 3 (Setup
## Troubleshooting
## Missing dependencies after dpkg install
### Missing Dependencies After dpkg Install
Some packages may have unresolved dependencies. Fix with:
@@ -101,7 +100,7 @@ sudo apt-get install -f
The `-f` flag tells apt to fix broken dependencies using what's available locally.
## GPG verification fails
### GPG Verification Fails
The signing key may not match or the download was corrupted. Re-import the key and verify:
@@ -116,7 +115,7 @@ If verification still fails, re-download the package on the connected machine an
sha256sum spinifex-v1.0.tar.gz
```
## Go module cache errors
### Go Module Cache Errors
The Go module cache may have been incompletely copied. On the connected machine, re-export the full cache:
+11 -23
View File
@@ -6,7 +6,6 @@ tags:
- install
- multi node
- cluster
badge: availability-zone
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
@@ -40,13 +39,13 @@ A Spinifex cluster distributes services across multiple servers for high availab
## Instructions
### Step 1. Install Spinifex on each server
## Step 1. Install Spinifex on Each Server
```bash
curl https://install.mulgadc.com | bash
```
### Step 2. Set node IP variables
## Step 2. Set Node IP Variables
On **each server**, export the management IPs for all nodes:
@@ -58,7 +57,7 @@ export AWS_REGION=us-east-1
export AWS_AZ=us-east-1a
```
### Step 3. Setup OVN networking
## Step 3. Setup OVN Networking
Server 1 runs OVN central and must be set up first. If your WAN interface is already a bridge (e.g. `br-wan`), setup-ovn.sh auto-detects it. Otherwise use `--wan-bridge=br-wan --wan-iface=eth1` (dedicated WAN NIC) or `--macvlan --wan-iface=eth0` (single-NIC).
@@ -92,7 +91,7 @@ Verify all 3 chassis registered:
sudo ovn-sbctl show
```
### Step 4. Form the cluster
## Step 4. Form the Cluster
Run init and join concurrently — init blocks until all nodes join.
@@ -123,29 +122,20 @@ sudo spx admin join \
--host $SPINIFEX_NODE1:4432 --region $AWS_REGION --az $AWS_AZ
```
### Step 5. Trust the CA certificate
On **each server**:
```bash
sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
```
### Step 6. Start services
## Step 5. Start Services
On **all servers**:
```bash
sudo systemctl start spinifex.target
export AWS_PROFILE=spinifex
```
### Step 7. Verify
## Step 6. Verify
From any node:
```bash
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
```
@@ -155,11 +145,9 @@ If this returns a list of available instance types, your cluster is working.
Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import an AMI, create a VPC, and launch your first instance.
---
## Troubleshooting
### Nodes not joining
### Nodes Not Joining
The init command must still be running when join executes. If init exited, re-run with `--force`.
@@ -167,21 +155,21 @@ The init command must still be running when join executes. If init exited, re-ru
curl -s http://$SPINIFEX_NODE1:4432/health
```
### OVN chassis not registering
### OVN Chassis Not Registering
```bash
sudo ovn-sbctl show
sudo ss -tlnp | grep 6642
```
### CA certificate not trusted
### CA Certificate Not Trusted
```bash
sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
```
### Cross-host VMs cannot communicate
### Cross-Host VMs Cannot Communicate
```bash
sudo ovs-vsctl show | grep -i geneve
+8 -11
View File
@@ -6,7 +6,6 @@ tags:
- install
- single node
- quickstart
badge: quickstart
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
@@ -50,7 +49,7 @@ This guide installs Spinifex on a single server using the binary installer. For
## Instructions
### Step 1. Install Spinifex
## Step 1. Install Spinifex
```bash
curl https://install.mulgadc.com | bash
@@ -58,7 +57,7 @@ curl https://install.mulgadc.com | bash
The installer downloads the Spinifex binary and bootstraps all dependencies (QEMU, OVN/OVS, AWS CLI).
### Step 2. Setup OVN networking
## Step 2. Setup OVN Networking
If your WAN interface is already a bridge (e.g. `br-wan`), setup-ovn.sh auto-detects it:
@@ -76,7 +75,7 @@ sudo /usr/local/share/spinifex/setup-ovn.sh --management --wan-bridge=br-wan --w
sudo /usr/local/share/spinifex/setup-ovn.sh --management --macvlan --wan-iface=enp0s3
```
### Step 3. Initialize
## Step 3. Initialize
```bash
sudo spx admin init --region ap-southeast-2 --az ap-southeast-2a --node node1 --nodes 1
@@ -84,13 +83,13 @@ sudo spx admin init --region ap-southeast-2 --az ap-southeast-2a --node node1 --
This auto-detects your network topology, generates configuration and TLS certificates, installs the CA into the system trust store, and configures AWS CLI credentials. Save the admin credentials printed during init — they will not be shown again.
### Step 4. Start services
## Step 4. Start Services
```bash
sudo systemctl start spinifex.target
```
### Step 5. Verify
## Step 5. Verify
```bash
export AWS_PROFILE=spinifex
@@ -103,17 +102,15 @@ If this returns a list of available instance types, your installation is working
Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import an AMI, create a VPC, and launch your first instance.
---
## Troubleshooting
### spx command not found
### spx Command Not Found
```bash
export PATH=$PATH:/usr/local/bin
```
### CA certificate not trusted
### CA Certificate Not Trusted
`sudo spx admin init` installs the CA automatically. If you need to re-install it manually:
@@ -122,7 +119,7 @@ sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
```
### OVN services not starting
### OVN Services Not Starting
```bash
sudo systemctl is-active ovn-controller
+2 -2
View File
@@ -69,7 +69,7 @@ ssh -i ~/.ssh/spinifex-key ec2-user@127.0.0.1 -p <port>
## Troubleshooting
## Instance fails to boot
### Instance Fails to Boot
Check QEMU logs for the instance and verify the AMI architecture matches your host:
@@ -85,7 +85,7 @@ If the AMI is for a different architecture (e.g. arm64 on an x86_64 host), impor
./bin/spx admin images import --name debian-12-x86_64
```
## Cannot SSH into instance
### Cannot SSH Into Instance
cloud-init needs time to configure the instance after boot. Wait 30-60 seconds and retry.
+2 -3
View File
@@ -6,7 +6,6 @@ tags:
- migration
- aws
- terraform
badge: new
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
@@ -73,7 +72,7 @@ aws s3 sync s3://local-bucket/ s3://cloud-bucket/ --source-region spinifex --reg
## Troubleshooting
## Terraform provider errors
## Terraform Provider Errors
Ensure all three skip flags are set in your provider configuration:
@@ -85,7 +84,7 @@ skip_requesting_account_id = true
Without these, Terraform will try to validate credentials and metadata against real AWS endpoints.
## S3 signature errors
## S3 Signature Errors
Spinifex uses AWS Signature V4. Ensure your AWS CLI is version 2.0 or higher:
+18 -17
View File
@@ -7,7 +7,6 @@ tags:
- vpc
- ec2
- quickstart
badge: setup
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
@@ -19,11 +18,6 @@ resources:
> Import an AMI, create a VPC with a public subnet, and launch your first instance.
This guide assumes Spinifex is already installed and running. If not, follow one of the installation guides first:
- [Binary Install (Single Node)](/docs/installing-spinifex)
- [Multi-Node Installation](/docs/installing-spinifex-multi-node)
- [Source Install](/docs/source-install)
## Table of Contents
- [1. Import an AMI](#1-import-an-ami)
@@ -36,6 +30,15 @@ This guide assumes Spinifex is already installed and running. If not, follow one
---
## Overview
This guide assumes Spinifex is already installed and running. If not, follow one of the installation guides first:
- [Binary Install (Single Node)](/docs/installing-spinifex)
- [Multi-Node Installation](/docs/installing-spinifex-multi-node)
- [Source Install](/docs/source-install)
## Instructions
## Prerequisites
Ensure the AWS profile is set:
@@ -130,7 +133,7 @@ aws ec2 attach-internet-gateway \
echo "IGW: $IGW_ID (attached to $VPC_ID)"
```
### Create a public subnet
### Create a Public Subnet
A public subnet auto-assigns a routable IP to each instance, making it directly reachable from your network.
@@ -236,7 +239,7 @@ aws ec2 start-instances --instance-ids $INSTANCE_ID
aws ec2 terminate-instances --instance-ids $INSTANCE_ID
```
### Console output
### Console Output
View the instance's serial console log (useful for debugging boot issues):
@@ -245,7 +248,7 @@ aws ec2 get-console-output --instance-id $INSTANCE_ID \
--query 'Output' --output text
```
### Multi-node: check instance placement
### Multi-Node: Check Instance Placement
On a multi-node cluster, instances are distributed across nodes:
@@ -255,7 +258,7 @@ spx get vms
## Additional Options
### Private subnets (no public IP)
### Private Subnets (No Public IP)
Create a subnet without `--map-public-ip-on-launch`:
@@ -268,7 +271,7 @@ PRIVATE_SUBNET=$(aws ec2 create-subnet \
Instances in private subnets get a private IP only. They can still reach the internet via SNAT through the VPC router, but are not directly reachable from your network.
### Multiple accounts
### Multiple Accounts
Create isolated accounts with their own resources:
@@ -281,18 +284,16 @@ export AWS_PROFILE=spinifex-myteam
The Spinifex web interface is available at `https://localhost:3000` when services are running.
---
## Troubleshooting
### Instance stuck in pending
### Instance Stuck in Pending
```bash
cat ~/spinifex/logs/spinifex.log
aws ec2 describe-images
```
### SSH connection refused
### SSH Connection Refused
cloud-init needs 30-60 seconds after boot. Check instance state:
@@ -300,7 +301,7 @@ cloud-init needs 30-60 seconds after boot. Check instance state:
aws ec2 describe-instances --instance-ids $INSTANCE_ID
```
### No public IP assigned
### No Public IP Assigned
Verify the subnet has `MapPublicIpOnLaunch` enabled:
@@ -320,7 +321,7 @@ Also verify an Internet Gateway is attached to the VPC:
aws ec2 describe-internet-gateways
```
### Instance has no internet access
### Instance Has No Internet Access
Check the VPC router's NAT rules (from the host):
+11 -21
View File
@@ -6,7 +6,6 @@ tags:
- install
- source
- development
badge: source
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
@@ -39,7 +38,7 @@ This guide builds Spinifex from source. For production deployments, the [binary
## Instructions
### Step 1. Install dependencies
## Step 1. Install Dependencies
```bash
mkdir -p ~/Development/mulga && cd ~/Development/mulga
@@ -48,7 +47,7 @@ sudo make -C spinifex quickinstall
export PATH=$PATH:/usr/local/go/bin
```
### Step 2. Clone and build
## Step 2. Clone and Build
```bash
cd spinifex
@@ -58,22 +57,15 @@ cd spinifex
Confirm `./bin/spx` exists.
### Step 3. Initialize
## Step 3. Initialize
```bash
./bin/spx admin init --region ap-southeast-2 --az ap-southeast-2a --node node1 --nodes 1
sudo ./bin/spx admin init --region ap-southeast-2 --az ap-southeast-2a --node node1 --nodes 1
```
Save the admin credentials printed during init.
### Step 4. Trust the CA certificate
```bash
sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
```
### Step 5. Setup OVN
## Step 4. Setup OVN
If your WAN interface is already a bridge (e.g. `br-wan`), setup-ovn.sh auto-detects it:
@@ -91,16 +83,16 @@ If your WAN is a physical NIC, choose one:
./scripts/setup-ovn.sh --management --macvlan --wan-iface=enp0s3
```
### Step 6. Start services
## Step 5. Start Services
```bash
./scripts/start-dev.sh
export AWS_PROFILE=spinifex
```
### Step 7. Verify
## Step 6. Verify
```bash
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
```
@@ -110,23 +102,21 @@ If this returns a list of available instance types, your installation is working
Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import an AMI, create a VPC, and launch your first instance.
---
## Troubleshooting
### Go not found in PATH
### Go Not Found in PATH
```bash
export PATH=$PATH:/usr/local/go/bin
```
### ./bin/spx missing after build
### ./bin/spx Missing After Build
```bash
./scripts/dev-setup.sh
```
### CA certificate not trusted
### CA Certificate Not Trusted
```bash
sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
+2 -3
View File
@@ -6,7 +6,6 @@ tags:
- cli
- admin
- reference
badge: cli
resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
@@ -75,7 +74,7 @@ node3 Ready 127.0.0.3 ap-southeast-2 ap-southeast-2a 2m 0
## Troubleshooting
## Permission denied running spinifex
### Permission Denied Running Spinifex
The binary may not be executable. Fix permissions:
@@ -85,7 +84,7 @@ chmod +x ./bin/spx
If you get permission errors during operations, ensure you're running with appropriate privileges. Some OVN and networking commands require `sudo`.
## Services fail to start
### Services Fail to Start
Check the daemon logs for specific errors:
+41 -41
View File
@@ -80,7 +80,7 @@ NIC. Each host runs `ovn-controller` which programs OpenFlow rules on `br-int`
A subnet's behavior depends on two things: whether the VPC has an Internet
Gateway, and whether the subnet has `MapPublicIpOnLaunch` enabled.
## Private Subnet (default)
## Private Subnet (Default)
Instances get a private IP only. They can communicate with other instances in the
same VPC (even across subnets and hosts via the overlay). They cannot reach the
@@ -195,19 +195,19 @@ is identical:
Two DHCP conversations (completely independent):
┌─────────────────────────────────────────────────────────────────────┐
│ HOST ↔ ROUTER DHCP (on br-ext, via veth pair → br-wan) │
│ Spinifex asks router: "give me a public IP for this VM" │
│ Router responds: 192.168.1.75 │
│ Spinifex stores IP, tells OVN: dnat_and_snat(192.168.1.75 ↔ │
│ 172.31.0.4) │
│ (Only used in DHCP source mode. Static mode picks from range.) │
│ HOST ↔ ROUTER DHCP (on br-ext, via veth pair → br-wan) │
│ Spinifex asks router: "give me a public IP for this VM" │
│ Router responds: 192.168.1.75 │
│ Spinifex stores IP, tells OVN: dnat_and_snat(192.168.1.75 ↔ │
│ 172.31.0.4) │
│ (Only used in DHCP source mode. Static mode picks from range.) │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ VM ↔ OVN DHCP (on tap → br-int → OVN logical switch) │
│ cloud-init asks: "what's my IP?" │
│ OVN DHCP responds: "you are 172.31.0.4" │
│ VM never sees 192.168.1.x or the router │
│ VM ↔ OVN DHCP (on tap → br-int → OVN logical switch) │
│ cloud-init asks: "what's my IP?" │
│ OVN DHCP responds: "you are 172.31.0.4" │
│ VM never sees 192.168.1.x or the router │
└─────────────────────────────────────────────────────────────────────┘
```
@@ -250,7 +250,7 @@ gateway_ip = "192.168.1.100" # Single IP for all VM outbound SNAT
prefix_len = 24
```
## Disabled (empty/omitted)
## Disabled (Empty/Omitted)
VPC networking is overlay-only. No external connectivity. Instances can only
communicate within their VPC.
@@ -307,7 +307,7 @@ Spinifex supports two ways to wire `br-wan` to the physical network.
**Direct bridge** is preferred when the host has a dedicated WAN NIC. **Macvlan**
is the fallback for single-NIC hosts where the WAN NIC is also the SSH NIC.
### Direct Bridge (preferred)
### Direct Bridge (Preferred)
The WAN NIC is added directly to `br-wan` as an OVS port. OVS sees all
traffic on the wire — any MAC, any protocol. No filtering, no workarounds.
@@ -332,7 +332,7 @@ public NICs. Homelab hosts with 2+ NICs.
sudo setup-ovn.sh --wan-bridge=br-wan --wan-iface=eth1
```
### Macvlan (fallback for single-NIC)
### Macvlan (Fallback for Single-NIC)
A macvlan sub-interface is created in bridge mode off the WAN NIC, and that
macvlan is added to br-wan. The host keeps its IP on the original NIC —
@@ -363,7 +363,7 @@ edge deployments. Adding the NIC directly to OVS would break host connectivity.
sudo setup-ovn.sh --macvlan --wan-iface=eth0
```
### Mode comparison
### Mode Comparison
| Capability | Direct bridge | Macvlan |
|------------|--------------|---------|
@@ -375,7 +375,7 @@ sudo setup-ovn.sh --macvlan --wan-iface=eth0
| Works on single-NIC | No (breaks SSH) | Yes |
| DHCP lease stability | Clean | Fragile (MAC changes post-lease) |
### How setup-ovn.sh decides
### How setup-ovn.sh Decides
| Flags | Result |
|-------|--------|
@@ -383,7 +383,7 @@ sudo setup-ovn.sh --macvlan --wan-iface=eth0
| `--macvlan --wan-iface=eth0` | Macvlan: sub-interface created, added to br-wan |
| (no WAN bridge) | Only br-int created, no WAN connectivity |
### Setup (macvlan mode)
### Setup (Macvlan Mode)
In environments where the WAN IP comes from a router's DHCP server (homelab,
small office), add `--dhcp` to obtain a gateway IP from the router automatically:
@@ -396,7 +396,7 @@ This requests an IP from the **router's DHCP** (e.g., 192.168.1.1 serving
addresses on the LAN). This is not Spinifex's internal OVN DHCP that assigns
private IPs to VMs — it's your network's existing DHCP server.
### macvlan internals
### Macvlan Internals
macvlan in `bridge` mode creates a virtual interface that shares the parent NIC's
physical wire but has its own MAC address. The Linux kernel blocks direct L2
@@ -450,20 +450,20 @@ ip link show spx-ext-eth0
## Network Flow Diagram
```
┌──────────────────────────────────────────────────┐
│ Bare-Metal Host │
│ │
│ ┌──────────┐ ┌──────────┐ │
│ │ br-int │ │br-wan│ │
│ │ (overlay)│ │ (WAN) │ │
│ │ │ │ │ │
│ │ tap-eni1 │ OVN │spx-ext- │ │
│ │ tap-eni2 ├──NAT───→│{nic} │ │
│ │ tap-eni3 │pipeline │(macvlan) │ │
│ └──────────┘ └────┬─────┘ │
┌───────────────────────────────────────────────────┐
│ Bare-Metal Host │
│ │
│ ┌──────────┐ ┌──────────┐ │
│ │ br-int │ │br-wan. │ │
│ │ (overlay)│ │ (WAN) │ │
│ │ │ │ │ │
│ │ tap-eni1 │ OVN │spx-ext- │ │
│ │ tap-eni2 ├──NAT───→│{nic} │ │
│ │ tap-eni3 │pipeline │(macvlan) │ │
│ └──────────┘ └────┬─────┘ │
│ │ macvlan bridge mode │
│ WAN NIC (host IP unchanged)│
└───────────────────────┼──────────────────────────┘
└───────────────────────┼───────────────────────────┘
│
Physical WAN
```
@@ -981,12 +981,12 @@ aws ec2 describe-instances --instance-ids $INSTANCE \
## Troubleshooting
## Debugging Toolkit
### Debugging Toolkit
These commands are used throughout the troubleshooting sections below. Learn
them — they cover 90% of VPC networking issues.
### OVN Northbound (logical topology)
### OVN Northbound (Logical Topology)
```bash
# Full topology overview (routers, switches, ports)
@@ -1096,7 +1096,7 @@ sudo cat /var/log/ovn/ovn-controller.log | tail -50
cat ~/spinifex/logs/spinifex.log
```
## VPC creation fails
### VPC Creation Fails
Check OVN services and vpcd daemon:
@@ -1105,7 +1105,7 @@ sudo systemctl is-active ovn-controller
cat ~/spinifex/logs/vpcd.log
```
## Instances cannot reach each other
### Instances Cannot Reach Each Other
Geneve tunnels may not be established:
@@ -1121,7 +1121,7 @@ ip addr show
ip route show
```
## Instance has no public IP
### Instance Has No Public IP
1. Check subnet has `MapPublicIpOnLaunch`:
```bash
@@ -1145,7 +1145,7 @@ ip route show
sudo ovn-nbctl lr-nat-list vpc-$VPC
```
## Public IP not reachable from WAN
### Public IP Not Reachable from WAN
Work through these checks in order — each eliminates a class of issues.
@@ -1240,7 +1240,7 @@ sudo ovn-trace --ct=new ext-vpc-{vpcId} \
The output shows every table the packet passes through and what action is
taken. Look for `drop` actions or unexpected paths.
## OVN SB commit failure loop
### OVN SB Commit Failure Loop
**Symptom:** ovn-controller log shows:
@@ -1265,7 +1265,7 @@ sudo systemctl start ovn-central ovn-controller
# vpcd reconcile will recreate the NB topology on next startup
```
## Chassis name mismatch
### Chassis Name Mismatch
**Symptom:** Gateway chassis is set in NB but the SB has no matching chassis.
`ovn-nbctl show` reports a gateway chassis name that doesn't appear in
@@ -1298,7 +1298,7 @@ sudo ovs-vsctl set Open_vSwitch . external_ids:system-id=chassis-node1
sudo systemctl restart ovn-controller
```
## Macvlan unicast filtering (inbound traffic lost)
### Macvlan Unicast Filtering (Inbound Traffic Lost)
**Symptom:** VM gets DHCP, outbound works (ping 8.8.8.8 from VM), but inbound
from the LAN to the VM's public IP fails. ARP replies go out but ICMP never
@@ -1338,7 +1338,7 @@ sudo ovn-nbctl lr-nat-list vpc-{vpcId}
# The dnat_and_snat row should have EMPTY external_mac and logical_port columns
```
## Stale ARP on remote hosts
### Stale ARP on Remote Hosts
**Symptom:** Ping from a LAN host to a VM public IP fails after a reset, but
worked before. The remote host has a stale ARP entry with the old MAC.
@@ -1354,7 +1354,7 @@ ping {public_ip} # should work now
OVN's `nat-addresses=router` option sends periodic gratuitous ARPs that will
eventually update remote ARP caches, but flushing is faster for testing.
## Security group rules not taking effect
### Security Group Rules Not Taking Effect
```bash
# Check port is in correct port group