From e6b48ce170546d8d3921d186e9504bb5b1ecac3c Mon Sep 17 00:00:00 2001 From: Julian Sommer <115516918+juliansommer@users.noreply.github.com> Date: Thu, 26 Mar 2026 10:29:48 +1000 Subject: [PATCH] fix: docs --- docs/creating-accounts/README.md | 2 +- docs/creating-an-environment/README.md | 14 +--- docs/hybrid-sync/README.md | 4 +- docs/iam-users-and-policies/README.md | 9 +- docs/installing-spinifex-airgapped/README.md | 15 ++-- docs/installing-spinifex-multi-node/README.md | 34 +++----- docs/installing-spinifex/README.md | 19 ++--- docs/launching-instances/README.md | 4 +- docs/moving-aws-workload/README.md | 5 +- docs/setting-up-your-cluster/README.md | 35 ++++---- docs/source-install/README.md | 32 +++----- docs/spinifex-admin-cli/README.md | 5 +- docs/vpc-networking/README.md | 82 +++++++++---------- 13 files changed, 111 insertions(+), 149 deletions(-) diff --git a/docs/creating-accounts/README.md b/docs/creating-accounts/README.md index d9a753e81..bc4862379 100644 --- a/docs/creating-accounts/README.md +++ b/docs/creating-accounts/README.md @@ -44,7 +44,7 @@ aws sts get-caller-identity ## Troubleshooting -## Credentials not working +## Credentials Not Working Verify the AWS CLI configuration files exist and contain the correct profile: diff --git a/docs/creating-an-environment/README.md b/docs/creating-an-environment/README.md index a792017e1..9d0d32f70 100644 --- a/docs/creating-an-environment/README.md +++ b/docs/creating-an-environment/README.md @@ -30,16 +30,6 @@ resources: This guide walks through the complete workflow for creating a production-ready Spinifex environment. -**Steps:** - -1. Create a user account -2. Import or create SSH keys -3. Create a VPC and subnet -4. Select and import an AMI -5. Launch EC2 instances -6. Connect via SSH -7. Monitor instance health - ## Instructions ## 1. Import SSH Key @@ -88,7 +78,7 @@ ssh -i ~/.ssh/spinifex-key ec2-user@127.0.0.1 -p ## Troubleshooting -## Instance stuck in pending +## Instance Stuck in Pending Check the Spinifex daemon and QEMU logs for boot errors: @@ -103,7 +93,7 @@ Verify the AMI exists and architecture matches your host: aws ec2 describe-images --image-ids $SPINIFEX_AMI ``` -## SSH connection refused +## SSH Connection Refused cloud-init takes 30-60 seconds to configure networking and SSH after boot. Wait and retry. diff --git a/docs/hybrid-sync/README.md b/docs/hybrid-sync/README.md index 1ff5a920b..f5f9865e0 100644 --- a/docs/hybrid-sync/README.md +++ b/docs/hybrid-sync/README.md @@ -51,7 +51,7 @@ rsync -avz /data/ user@cloud-server:/backup/spinifex-data/ ## Troubleshooting -## Sync fails mid-transfer +## Sync Fails Mid Transfer S3 sync is idempotent — re-run the same command to resume where it left off: @@ -62,7 +62,7 @@ aws s3 sync s3://local-bucket/ s3://cloud-bucket/ \ Only changed or missing files will be transferred on subsequent runs. -## Permissions errors during sync +## Permission Errors During Sync Verify your AWS credentials are configured for both the source and destination: diff --git a/docs/iam-users-and-policies/README.md b/docs/iam-users-and-policies/README.md index 4fa199bd7..1b3d86c78 100644 --- a/docs/iam-users-and-policies/README.md +++ b/docs/iam-users-and-policies/README.md @@ -8,7 +8,6 @@ tags: - policies - access keys - security -badge: setup resources: - title: "Spinifex Repository" url: "https://github.com/mulgadc/spinifex" @@ -475,7 +474,7 @@ AWS_PROFILE=spinifex-carol aws ec2 terminate-instances --instance-ids i-123 ## Troubleshooting -### AccessDenied on IAM commands +### AccessDenied on IAM Commands The calling user must have IAM permissions. Attach a policy with `iam:*` actions, or use the root account profile: @@ -501,7 +500,7 @@ aws iam update-access-key --user-name alice \ If the key was deleted, create a new one. -### DeleteConflict when deleting a user +### DeleteConflict When Deleting a User The user still has access keys or attached policies. Remove them first: @@ -513,7 +512,7 @@ aws iam list-access-keys --user-name alice aws iam list-attached-user-policies --user-name alice ``` -### DeleteConflict when deleting a policy +### DeleteConflict When Deleting a Policy The policy is still attached to one or more users. Detach it from all users before deleting: @@ -526,7 +525,7 @@ aws iam delete-policy \ --policy-arn arn:aws:iam::000000000001:policy/MyPolicy ``` -### LimitExceeded when creating access keys +### LimitExceeded When Creating Access Keys Each user can have at most 2 access keys. Delete an existing key before creating a new one: diff --git a/docs/installing-spinifex-airgapped/README.md b/docs/installing-spinifex-airgapped/README.md index 74dd78adf..b7d34829f 100644 --- a/docs/installing-spinifex-airgapped/README.md +++ b/docs/installing-spinifex-airgapped/README.md @@ -6,7 +6,6 @@ tags: - install - air-gapped - offline -badge: advanced resources: - title: "Spinifex Repository" url: "https://github.com/mulgadc/spinifex" @@ -38,7 +37,7 @@ In air-gapped or disconnected environments, Spinifex can be deployed without int ## Instructions -## Step 1. Prepare packages (on a connected machine) +## Step 1. Prepare Packages (on a connected machine) Set up a local APT cache to collect all required packages: @@ -56,7 +55,7 @@ sudo apt install --download-only nbdkit nbdkit-plugin-dev pkg-config \ ovn-central ovn-host openvswitch-switch ``` -## Step 2. Create USB deployment media +## Step 2. Create USB Deployment Media ```bash mkdir -p /media/spinifex-deploy/{apt-packages,go-cache,spinifex-source} @@ -65,7 +64,7 @@ cp -r ~/go/pkg/mod/cache/ /media/spinifex-deploy/go-cache/ cp -r ~/Development/mulga/spinifex /media/spinifex-deploy/spinifex-source/ ``` -## Step 3. Verify package integrity +## Step 3. Verify Package Integrity Sign and verify packages before transferring to the target: @@ -75,7 +74,7 @@ gpg --verify spinifex-v1.0.tar.gz.sig spinifex-v1.0.tar.gz sha256sum -c /media/spinifex-deploy/checksums.sha256 ``` -## Step 4. Install on the target server +## Step 4. Install on the Target Server Mount the USB and install: @@ -90,7 +89,7 @@ Then follow the [Source Install](/docs/source-install) guide from Step 3 (Setup ## Troubleshooting -## Missing dependencies after dpkg install +### Missing Dependencies After dpkg Install Some packages may have unresolved dependencies. Fix with: @@ -101,7 +100,7 @@ sudo apt-get install -f The `-f` flag tells apt to fix broken dependencies using what's available locally. -## GPG verification fails +### GPG Verification Fails The signing key may not match or the download was corrupted. Re-import the key and verify: @@ -116,7 +115,7 @@ If verification still fails, re-download the package on the connected machine an sha256sum spinifex-v1.0.tar.gz ``` -## Go module cache errors +### Go Module Cache Errors The Go module cache may have been incompletely copied. On the connected machine, re-export the full cache: diff --git a/docs/installing-spinifex-multi-node/README.md b/docs/installing-spinifex-multi-node/README.md index 2e06b98e1..eea38cb32 100644 --- a/docs/installing-spinifex-multi-node/README.md +++ b/docs/installing-spinifex-multi-node/README.md @@ -6,7 +6,6 @@ tags: - install - multi node - cluster -badge: availability-zone resources: - title: "Spinifex Repository" url: "https://github.com/mulgadc/spinifex" @@ -40,13 +39,13 @@ A Spinifex cluster distributes services across multiple servers for high availab ## Instructions -### Step 1. Install Spinifex on each server +## Step 1. Install Spinifex on Each Server ```bash curl https://install.mulgadc.com | bash ``` -### Step 2. Set node IP variables +## Step 2. Set Node IP Variables On **each server**, export the management IPs for all nodes: @@ -58,7 +57,7 @@ export AWS_REGION=us-east-1 export AWS_AZ=us-east-1a ``` -### Step 3. Setup OVN networking +## Step 3. Setup OVN Networking Server 1 runs OVN central and must be set up first. If your WAN interface is already a bridge (e.g. `br-wan`), setup-ovn.sh auto-detects it. Otherwise use `--wan-bridge=br-wan --wan-iface=eth1` (dedicated WAN NIC) or `--macvlan --wan-iface=eth0` (single-NIC). @@ -92,7 +91,7 @@ Verify all 3 chassis registered: sudo ovn-sbctl show ``` -### Step 4. Form the cluster +## Step 4. Form the Cluster Run init and join concurrently — init blocks until all nodes join. @@ -123,29 +122,20 @@ sudo spx admin join \ --host $SPINIFEX_NODE1:4432 --region $AWS_REGION --az $AWS_AZ ``` -### Step 5. Trust the CA certificate - -On **each server**: - -```bash -sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt -sudo update-ca-certificates -``` - -### Step 6. Start services +## Step 5. Start Services On **all servers**: ```bash sudo systemctl start spinifex.target -export AWS_PROFILE=spinifex ``` -### Step 7. Verify +## Step 6. Verify From any node: ```bash +export AWS_PROFILE=spinifex aws ec2 describe-instance-types ``` @@ -155,11 +145,9 @@ If this returns a list of available instance types, your cluster is working. Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import an AMI, create a VPC, and launch your first instance. ---- - ## Troubleshooting -### Nodes not joining +### Nodes Not Joining The init command must still be running when join executes. If init exited, re-run with `--force`. @@ -167,21 +155,21 @@ The init command must still be running when join executes. If init exited, re-ru curl -s http://$SPINIFEX_NODE1:4432/health ``` -### OVN chassis not registering +### OVN Chassis Not Registering ```bash sudo ovn-sbctl show sudo ss -tlnp | grep 6642 ``` -### CA certificate not trusted +### CA Certificate Not Trusted ```bash sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt sudo update-ca-certificates ``` -### Cross-host VMs cannot communicate +### Cross-Host VMs Cannot Communicate ```bash sudo ovs-vsctl show | grep -i geneve diff --git a/docs/installing-spinifex/README.md b/docs/installing-spinifex/README.md index 06c3e722a..f5fcc7ee6 100644 --- a/docs/installing-spinifex/README.md +++ b/docs/installing-spinifex/README.md @@ -6,7 +6,6 @@ tags: - install - single node - quickstart -badge: quickstart resources: - title: "Spinifex Repository" url: "https://github.com/mulgadc/spinifex" @@ -50,7 +49,7 @@ This guide installs Spinifex on a single server using the binary installer. For ## Instructions -### Step 1. Install Spinifex +## Step 1. Install Spinifex ```bash curl https://install.mulgadc.com | bash @@ -58,7 +57,7 @@ curl https://install.mulgadc.com | bash The installer downloads the Spinifex binary and bootstraps all dependencies (QEMU, OVN/OVS, AWS CLI). -### Step 2. Setup OVN networking +## Step 2. Setup OVN Networking If your WAN interface is already a bridge (e.g. `br-wan`), setup-ovn.sh auto-detects it: @@ -76,7 +75,7 @@ sudo /usr/local/share/spinifex/setup-ovn.sh --management --wan-bridge=br-wan --w sudo /usr/local/share/spinifex/setup-ovn.sh --management --macvlan --wan-iface=enp0s3 ``` -### Step 3. Initialize +## Step 3. Initialize ```bash sudo spx admin init --region ap-southeast-2 --az ap-southeast-2a --node node1 --nodes 1 @@ -84,13 +83,13 @@ sudo spx admin init --region ap-southeast-2 --az ap-southeast-2a --node node1 -- This auto-detects your network topology, generates configuration and TLS certificates, installs the CA into the system trust store, and configures AWS CLI credentials. Save the admin credentials printed during init — they will not be shown again. -### Step 4. Start services +## Step 4. Start Services ```bash sudo systemctl start spinifex.target ``` -### Step 5. Verify +## Step 5. Verify ```bash export AWS_PROFILE=spinifex @@ -103,17 +102,15 @@ If this returns a list of available instance types, your installation is working Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import an AMI, create a VPC, and launch your first instance. ---- - ## Troubleshooting -### spx command not found +### spx Command Not Found ```bash export PATH=$PATH:/usr/local/bin ``` -### CA certificate not trusted +### CA Certificate Not Trusted `sudo spx admin init` installs the CA automatically. If you need to re-install it manually: @@ -122,7 +119,7 @@ sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt sudo update-ca-certificates ``` -### OVN services not starting +### OVN Services Not Starting ```bash sudo systemctl is-active ovn-controller diff --git a/docs/launching-instances/README.md b/docs/launching-instances/README.md index 724fbf1a0..c310c19ff 100644 --- a/docs/launching-instances/README.md +++ b/docs/launching-instances/README.md @@ -69,7 +69,7 @@ ssh -i ~/.ssh/spinifex-key ec2-user@127.0.0.1 -p ## Troubleshooting -## Instance fails to boot +### Instance Fails to Boot Check QEMU logs for the instance and verify the AMI architecture matches your host: @@ -85,7 +85,7 @@ If the AMI is for a different architecture (e.g. arm64 on an x86_64 host), impor ./bin/spx admin images import --name debian-12-x86_64 ``` -## Cannot SSH into instance +### Cannot SSH Into Instance cloud-init needs time to configure the instance after boot. Wait 30-60 seconds and retry. diff --git a/docs/moving-aws-workload/README.md b/docs/moving-aws-workload/README.md index 51d76b186..af5614c28 100644 --- a/docs/moving-aws-workload/README.md +++ b/docs/moving-aws-workload/README.md @@ -6,7 +6,6 @@ tags: - migration - aws - terraform -badge: new resources: - title: "Spinifex Repository" url: "https://github.com/mulgadc/spinifex" @@ -73,7 +72,7 @@ aws s3 sync s3://local-bucket/ s3://cloud-bucket/ --source-region spinifex --reg ## Troubleshooting -## Terraform provider errors +## Terraform Provider Errors Ensure all three skip flags are set in your provider configuration: @@ -85,7 +84,7 @@ skip_requesting_account_id = true Without these, Terraform will try to validate credentials and metadata against real AWS endpoints. -## S3 signature errors +## S3 Signature Errors Spinifex uses AWS Signature V4. Ensure your AWS CLI is version 2.0 or higher: diff --git a/docs/setting-up-your-cluster/README.md b/docs/setting-up-your-cluster/README.md index 173df47aa..be575cda1 100644 --- a/docs/setting-up-your-cluster/README.md +++ b/docs/setting-up-your-cluster/README.md @@ -7,7 +7,6 @@ tags: - vpc - ec2 - quickstart -badge: setup resources: - title: "Spinifex Repository" url: "https://github.com/mulgadc/spinifex" @@ -19,11 +18,6 @@ resources: > Import an AMI, create a VPC with a public subnet, and launch your first instance. -This guide assumes Spinifex is already installed and running. If not, follow one of the installation guides first: -- [Binary Install (Single Node)](/docs/installing-spinifex) -- [Multi-Node Installation](/docs/installing-spinifex-multi-node) -- [Source Install](/docs/source-install) - ## Table of Contents - [1. Import an AMI](#1-import-an-ami) @@ -36,6 +30,15 @@ This guide assumes Spinifex is already installed and running. If not, follow one --- +## Overview + +This guide assumes Spinifex is already installed and running. If not, follow one of the installation guides first: +- [Binary Install (Single Node)](/docs/installing-spinifex) +- [Multi-Node Installation](/docs/installing-spinifex-multi-node) +- [Source Install](/docs/source-install) + +## Instructions + ## Prerequisites Ensure the AWS profile is set: @@ -130,7 +133,7 @@ aws ec2 attach-internet-gateway \ echo "IGW: $IGW_ID (attached to $VPC_ID)" ``` -### Create a public subnet +### Create a Public Subnet A public subnet auto-assigns a routable IP to each instance, making it directly reachable from your network. @@ -236,7 +239,7 @@ aws ec2 start-instances --instance-ids $INSTANCE_ID aws ec2 terminate-instances --instance-ids $INSTANCE_ID ``` -### Console output +### Console Output View the instance's serial console log (useful for debugging boot issues): @@ -245,7 +248,7 @@ aws ec2 get-console-output --instance-id $INSTANCE_ID \ --query 'Output' --output text ``` -### Multi-node: check instance placement +### Multi-Node: Check Instance Placement On a multi-node cluster, instances are distributed across nodes: @@ -255,7 +258,7 @@ spx get vms ## Additional Options -### Private subnets (no public IP) +### Private Subnets (No Public IP) Create a subnet without `--map-public-ip-on-launch`: @@ -268,7 +271,7 @@ PRIVATE_SUBNET=$(aws ec2 create-subnet \ Instances in private subnets get a private IP only. They can still reach the internet via SNAT through the VPC router, but are not directly reachable from your network. -### Multiple accounts +### Multiple Accounts Create isolated accounts with their own resources: @@ -281,18 +284,16 @@ export AWS_PROFILE=spinifex-myteam The Spinifex web interface is available at `https://localhost:3000` when services are running. ---- - ## Troubleshooting -### Instance stuck in pending +### Instance Stuck in Pending ```bash cat ~/spinifex/logs/spinifex.log aws ec2 describe-images ``` -### SSH connection refused +### SSH Connection Refused cloud-init needs 30-60 seconds after boot. Check instance state: @@ -300,7 +301,7 @@ cloud-init needs 30-60 seconds after boot. Check instance state: aws ec2 describe-instances --instance-ids $INSTANCE_ID ``` -### No public IP assigned +### No Public IP Assigned Verify the subnet has `MapPublicIpOnLaunch` enabled: @@ -320,7 +321,7 @@ Also verify an Internet Gateway is attached to the VPC: aws ec2 describe-internet-gateways ``` -### Instance has no internet access +### Instance Has No Internet Access Check the VPC router's NAT rules (from the host): diff --git a/docs/source-install/README.md b/docs/source-install/README.md index 240236960..81dab8f3b 100644 --- a/docs/source-install/README.md +++ b/docs/source-install/README.md @@ -6,7 +6,6 @@ tags: - install - source - development -badge: source resources: - title: "Spinifex Repository" url: "https://github.com/mulgadc/spinifex" @@ -39,7 +38,7 @@ This guide builds Spinifex from source. For production deployments, the [binary ## Instructions -### Step 1. Install dependencies +## Step 1. Install Dependencies ```bash mkdir -p ~/Development/mulga && cd ~/Development/mulga @@ -48,7 +47,7 @@ sudo make -C spinifex quickinstall export PATH=$PATH:/usr/local/go/bin ``` -### Step 2. Clone and build +## Step 2. Clone and Build ```bash cd spinifex @@ -58,22 +57,15 @@ cd spinifex Confirm `./bin/spx` exists. -### Step 3. Initialize +## Step 3. Initialize ```bash -./bin/spx admin init --region ap-southeast-2 --az ap-southeast-2a --node node1 --nodes 1 +sudo ./bin/spx admin init --region ap-southeast-2 --az ap-southeast-2a --node node1 --nodes 1 ``` Save the admin credentials printed during init. -### Step 4. Trust the CA certificate - -```bash -sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt -sudo update-ca-certificates -``` - -### Step 5. Setup OVN +## Step 4. Setup OVN If your WAN interface is already a bridge (e.g. `br-wan`), setup-ovn.sh auto-detects it: @@ -91,16 +83,16 @@ If your WAN is a physical NIC, choose one: ./scripts/setup-ovn.sh --management --macvlan --wan-iface=enp0s3 ``` -### Step 6. Start services +## Step 5. Start Services ```bash ./scripts/start-dev.sh -export AWS_PROFILE=spinifex ``` -### Step 7. Verify +## Step 6. Verify ```bash +export AWS_PROFILE=spinifex aws ec2 describe-instance-types ``` @@ -110,23 +102,21 @@ If this returns a list of available instance types, your installation is working Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import an AMI, create a VPC, and launch your first instance. ---- - ## Troubleshooting -### Go not found in PATH +### Go Not Found in PATH ```bash export PATH=$PATH:/usr/local/go/bin ``` -### ./bin/spx missing after build +### ./bin/spx Missing After Build ```bash ./scripts/dev-setup.sh ``` -### CA certificate not trusted +### CA Certificate Not Trusted ```bash sudo cp ~/spinifex/config/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt diff --git a/docs/spinifex-admin-cli/README.md b/docs/spinifex-admin-cli/README.md index 0fb6a7b73..99abdd574 100644 --- a/docs/spinifex-admin-cli/README.md +++ b/docs/spinifex-admin-cli/README.md @@ -6,7 +6,6 @@ tags: - cli - admin - reference -badge: cli resources: - title: "Spinifex Repository" url: "https://github.com/mulgadc/spinifex" @@ -75,7 +74,7 @@ node3 Ready 127.0.0.3 ap-southeast-2 ap-southeast-2a 2m 0 ## Troubleshooting -## Permission denied running spinifex +### Permission Denied Running Spinifex The binary may not be executable. Fix permissions: @@ -85,7 +84,7 @@ chmod +x ./bin/spx If you get permission errors during operations, ensure you're running with appropriate privileges. Some OVN and networking commands require `sudo`. -## Services fail to start +### Services Fail to Start Check the daemon logs for specific errors: diff --git a/docs/vpc-networking/README.md b/docs/vpc-networking/README.md index 0817865ba..1b3b00056 100644 --- a/docs/vpc-networking/README.md +++ b/docs/vpc-networking/README.md @@ -80,7 +80,7 @@ NIC. Each host runs `ovn-controller` which programs OpenFlow rules on `br-int` A subnet's behavior depends on two things: whether the VPC has an Internet Gateway, and whether the subnet has `MapPublicIpOnLaunch` enabled. -## Private Subnet (default) +## Private Subnet (Default) Instances get a private IP only. They can communicate with other instances in the same VPC (even across subnets and hosts via the overlay). They cannot reach the @@ -195,19 +195,19 @@ is identical: Two DHCP conversations (completely independent): ┌─────────────────────────────────────────────────────────────────────┐ - │ HOST ↔ ROUTER DHCP (on br-ext, via veth pair → br-wan) │ - │ Spinifex asks router: "give me a public IP for this VM" │ - │ Router responds: 192.168.1.75 │ - │ Spinifex stores IP, tells OVN: dnat_and_snat(192.168.1.75 ↔ │ - │ 172.31.0.4) │ - │ (Only used in DHCP source mode. Static mode picks from range.) │ + │ HOST ↔ ROUTER DHCP (on br-ext, via veth pair → br-wan) │ + │ Spinifex asks router: "give me a public IP for this VM" │ + │ Router responds: 192.168.1.75 │ + │ Spinifex stores IP, tells OVN: dnat_and_snat(192.168.1.75 ↔ │ + │ 172.31.0.4) │ + │ (Only used in DHCP source mode. Static mode picks from range.) │ └─────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────┐ - │ VM ↔ OVN DHCP (on tap → br-int → OVN logical switch) │ - │ cloud-init asks: "what's my IP?" │ - │ OVN DHCP responds: "you are 172.31.0.4" │ - │ VM never sees 192.168.1.x or the router │ + │ VM ↔ OVN DHCP (on tap → br-int → OVN logical switch) │ + │ cloud-init asks: "what's my IP?" │ + │ OVN DHCP responds: "you are 172.31.0.4" │ + │ VM never sees 192.168.1.x or the router │ └─────────────────────────────────────────────────────────────────────┘ ``` @@ -250,7 +250,7 @@ gateway_ip = "192.168.1.100" # Single IP for all VM outbound SNAT prefix_len = 24 ``` -## Disabled (empty/omitted) +## Disabled (Empty/Omitted) VPC networking is overlay-only. No external connectivity. Instances can only communicate within their VPC. @@ -307,7 +307,7 @@ Spinifex supports two ways to wire `br-wan` to the physical network. **Direct bridge** is preferred when the host has a dedicated WAN NIC. **Macvlan** is the fallback for single-NIC hosts where the WAN NIC is also the SSH NIC. -### Direct Bridge (preferred) +### Direct Bridge (Preferred) The WAN NIC is added directly to `br-wan` as an OVS port. OVS sees all traffic on the wire — any MAC, any protocol. No filtering, no workarounds. @@ -332,7 +332,7 @@ public NICs. Homelab hosts with 2+ NICs. sudo setup-ovn.sh --wan-bridge=br-wan --wan-iface=eth1 ``` -### Macvlan (fallback for single-NIC) +### Macvlan (Fallback for Single-NIC) A macvlan sub-interface is created in bridge mode off the WAN NIC, and that macvlan is added to br-wan. The host keeps its IP on the original NIC — @@ -363,7 +363,7 @@ edge deployments. Adding the NIC directly to OVS would break host connectivity. sudo setup-ovn.sh --macvlan --wan-iface=eth0 ``` -### Mode comparison +### Mode Comparison | Capability | Direct bridge | Macvlan | |------------|--------------|---------| @@ -375,7 +375,7 @@ sudo setup-ovn.sh --macvlan --wan-iface=eth0 | Works on single-NIC | No (breaks SSH) | Yes | | DHCP lease stability | Clean | Fragile (MAC changes post-lease) | -### How setup-ovn.sh decides +### How setup-ovn.sh Decides | Flags | Result | |-------|--------| @@ -383,7 +383,7 @@ sudo setup-ovn.sh --macvlan --wan-iface=eth0 | `--macvlan --wan-iface=eth0` | Macvlan: sub-interface created, added to br-wan | | (no WAN bridge) | Only br-int created, no WAN connectivity | -### Setup (macvlan mode) +### Setup (Macvlan Mode) In environments where the WAN IP comes from a router's DHCP server (homelab, small office), add `--dhcp` to obtain a gateway IP from the router automatically: @@ -396,7 +396,7 @@ This requests an IP from the **router's DHCP** (e.g., 192.168.1.1 serving addresses on the LAN). This is not Spinifex's internal OVN DHCP that assigns private IPs to VMs — it's your network's existing DHCP server. -### macvlan internals +### Macvlan Internals macvlan in `bridge` mode creates a virtual interface that shares the parent NIC's physical wire but has its own MAC address. The Linux kernel blocks direct L2 @@ -450,20 +450,20 @@ ip link show spx-ext-eth0 ## Network Flow Diagram ``` -┌──────────────────────────────────────────────────┐ -│ Bare-Metal Host │ -│ │ -│ ┌──────────┐ ┌──────────┐ │ -│ │ br-int │ │br-wan│ │ -│ │ (overlay)│ │ (WAN) │ │ -│ │ │ │ │ │ -│ │ tap-eni1 │ OVN │spx-ext- │ │ -│ │ tap-eni2 ├──NAT───→│{nic} │ │ -│ │ tap-eni3 │pipeline │(macvlan) │ │ -│ └──────────┘ └────┬─────┘ │ +┌───────────────────────────────────────────────────┐ +│ Bare-Metal Host │ +│ │ +│ ┌──────────┐ ┌──────────┐ │ +│ │ br-int │ │br-wan. │ │ +│ │ (overlay)│ │ (WAN) │ │ +│ │ │ │ │ │ +│ │ tap-eni1 │ OVN │spx-ext- │ │ +│ │ tap-eni2 ├──NAT───→│{nic} │ │ +│ │ tap-eni3 │pipeline │(macvlan) │ │ +│ └──────────┘ └────┬─────┘ │ │ │ macvlan bridge mode │ │ WAN NIC (host IP unchanged)│ -└───────────────────────┼──────────────────────────┘ +└───────────────────────┼───────────────────────────┘ │ Physical WAN ``` @@ -981,12 +981,12 @@ aws ec2 describe-instances --instance-ids $INSTANCE \ ## Troubleshooting -## Debugging Toolkit +### Debugging Toolkit These commands are used throughout the troubleshooting sections below. Learn them — they cover 90% of VPC networking issues. -### OVN Northbound (logical topology) +### OVN Northbound (Logical Topology) ```bash # Full topology overview (routers, switches, ports) @@ -1096,7 +1096,7 @@ sudo cat /var/log/ovn/ovn-controller.log | tail -50 cat ~/spinifex/logs/spinifex.log ``` -## VPC creation fails +### VPC Creation Fails Check OVN services and vpcd daemon: @@ -1105,7 +1105,7 @@ sudo systemctl is-active ovn-controller cat ~/spinifex/logs/vpcd.log ``` -## Instances cannot reach each other +### Instances Cannot Reach Each Other Geneve tunnels may not be established: @@ -1121,7 +1121,7 @@ ip addr show ip route show ``` -## Instance has no public IP +### Instance Has No Public IP 1. Check subnet has `MapPublicIpOnLaunch`: ```bash @@ -1145,7 +1145,7 @@ ip route show sudo ovn-nbctl lr-nat-list vpc-$VPC ``` -## Public IP not reachable from WAN +### Public IP Not Reachable from WAN Work through these checks in order — each eliminates a class of issues. @@ -1240,7 +1240,7 @@ sudo ovn-trace --ct=new ext-vpc-{vpcId} \ The output shows every table the packet passes through and what action is taken. Look for `drop` actions or unexpected paths. -## OVN SB commit failure loop +### OVN SB Commit Failure Loop **Symptom:** ovn-controller log shows: @@ -1265,7 +1265,7 @@ sudo systemctl start ovn-central ovn-controller # vpcd reconcile will recreate the NB topology on next startup ``` -## Chassis name mismatch +### Chassis Name Mismatch **Symptom:** Gateway chassis is set in NB but the SB has no matching chassis. `ovn-nbctl show` reports a gateway chassis name that doesn't appear in @@ -1298,7 +1298,7 @@ sudo ovs-vsctl set Open_vSwitch . external_ids:system-id=chassis-node1 sudo systemctl restart ovn-controller ``` -## Macvlan unicast filtering (inbound traffic lost) +### Macvlan Unicast Filtering (Inbound Traffic Lost) **Symptom:** VM gets DHCP, outbound works (ping 8.8.8.8 from VM), but inbound from the LAN to the VM's public IP fails. ARP replies go out but ICMP never @@ -1338,7 +1338,7 @@ sudo ovn-nbctl lr-nat-list vpc-{vpcId} # The dnat_and_snat row should have EMPTY external_mac and logical_port columns ``` -## Stale ARP on remote hosts +### Stale ARP on Remote Hosts **Symptom:** Ping from a LAN host to a VM public IP fails after a reset, but worked before. The remote host has a stale ARP entry with the old MAC. @@ -1354,7 +1354,7 @@ ping {public_ip} # should work now OVN's `nat-addresses=router` option sends periodic gratuitous ARPs that will eventually update remote ARP caches, but flushing is faster for testing. -## Security group rules not taking effect +### Security Group Rules Not Taking Effect ```bash # Check port is in correct port group