gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-15 17:11:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: document direct bridge vs macvlan modes for br-external

Browse Source 
Direct bridge (WAN NIC added directly to OVS) is preferred when a
dedicated WAN NIC exists. Enables distributed NAT, no MAC workarounds,
host can reach VMs. Macvlan is fallback for single-NIC hosts only.
This commit is contained in:
Ben Duncan
2026-03-23 11:36:15 +00:00
parent 93aaf0f991
commit d6062ed550
1 changed files with 66 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/vpc-networking/README.md
+66 -30
View File
@@ -213,12 +213,42 @@ maps the logical external switch to `br-external` via `ovn-bridge-mappings`.
VM TAP ──→ br-int ──→ OVN logical pipeline ──→ localnet ──→ br-external ──→ WAN
```
## Bridge Setup (macvlan)
## Bridge Modes
Spinifex supports two ways to wire `br-external` to the physical network.
**Direct bridge** is preferred when the host has a dedicated WAN NIC. **Macvlan**
is the fallback for single-NIC hosts where the WAN NIC is also the SSH NIC.
### Direct Bridge (preferred)
The WAN NIC is added directly to `br-external` as an OVS port. OVS sees all
traffic on the wire — any MAC, any protocol. No filtering, no workarounds.
```
Management NIC (host IP, SSH)
WAN NIC ──→ br-external (OVS port) ──→ OVN localnet
```
**Use when:** The host has a NIC dedicated to external/WAN traffic that is NOT
used for SSH or management. Datacenter servers with separate management +
public NICs. Homelab hosts with 2+ NICs.
**Benefits over macvlan:**
- **Distributed NAT**: DNAT processed on the VM's own chassis (no hairpin)
- **No MAC workarounds**: OVS sees all frames, any MAC
- **Host can reach VMs**: No macvlan parent↔child isolation
- **Better multi-node performance**: External traffic stays local
- **DHCP lease stability**: No MAC changes after lease
```bash
sudo setup-ovn.sh --external-bridge --external-iface=eth1 --direct
```
### Macvlan (fallback for single-NIC)
A macvlan sub-interface is created in bridge mode off the WAN NIC, and that
macvlan is added to br-external. The host keeps its IP on the original NIC —
SSH stays up, no risk of losing connectivity. This works for all deployments:
single-NIC homelabs, multi-NIC datacenters, and everything in between.
SSH stays up.
```
WAN NIC (host IP — unchanged)
@@ -232,12 +262,41 @@ WAN NIC (host IP — unchanged)
localnet → OVN ext switch
```
### Setup
**Use when:** The WAN NIC is the same NIC used for SSH. Single-NIC homelabs,
edge deployments. Adding the NIC directly to OVS would break host connectivity.
**Limitations:**
- Centralized NAT only (macvlan filters unicast not matching its MAC)
- Host cannot reach VMs at their public IPs (macvlan isolation)
- vpcd must align macvlan MAC with OVN router MAC on startup
- Multi-node: all external traffic hairpins through the gateway chassis
```bash
sudo setup-ovn.sh --external-bridge --external-iface=eth0
```
### Mode comparison
| Capability | Direct bridge | Macvlan |
|------------|--------------|---------|
| NAT mode | Distributed (local DNAT) | Centralized (gateway hairpin) |
| Multi-node external perf | Optimal | Hairpin through gateway chassis |
| Host → VM public IP | Works | Blocked (kernel isolation) |
| MAC alignment needed | No | Yes (fragile startup step) |
| Gratuitous ARPs needed | No | Yes (`nat-addresses=router`) |
| Works on single-NIC | No (breaks SSH) | Yes |
| DHCP lease stability | Clean | Fragile (MAC changes post-lease) |
### How setup-ovn.sh decides
| Flags | Result |
|-------|--------|
| `--external-bridge --external-iface=eth1 --direct` | Direct bridge: NIC added to br-external |
| `--external-bridge --external-iface=eth0` | Macvlan: sub-interface created, added to br-external |
| (no `--external-bridge`) | Only br-int created, no WAN connectivity |
### Setup (macvlan mode)
In environments where the WAN IP comes from a router's DHCP server (homelab,
small office), add `--dhcp` to obtain a gateway IP from the router automatically:
@@ -249,24 +308,11 @@ This requests an IP from the **router's DHCP** (e.g., 192.168.1.1 serving
addresses on the LAN). This is not Spinifex's internal OVN DHCP that assigns
private IPs to VMs — it's your network's existing DHCP server.
### What Happens
1. Creates macvlan: `ip link add spx-ext-eth0 link eth0 type macvlan mode bridge`
2. Brings it up: `ip link set spx-ext-eth0 up`
3. Creates OVS bridge `br-external`
4. Adds the macvlan (not the parent NIC) to br-external
5. Sets `ovn-bridge-mappings=external:br-external`
6. Host IP on eth0 is completely untouched
7. (If `--dhcp`) Obtains gateway IP from router DHCP for OVN SNAT
### macvlan Behavior
### macvlan internals
macvlan in `bridge` mode creates a virtual interface that shares the parent NIC's
physical wire but has its own MAC address. Frames between the macvlan and external
devices work normally. The Linux kernel blocks direct L2 frames between a parent
interface and its macvlan children.
**What this means in practice:**
physical wire but has its own MAC address. The Linux kernel blocks direct L2
frames between a parent interface and its macvlan children.
| Path | Works? | Why |
|------|--------|-----|
@@ -275,16 +321,6 @@ interface and its macvlan children.
| Host → VM public IP | No | macvlan isolation (kernel blocks parent↔child) |
| Host → VM private IP | Yes | Overlay via br-int (unrelated to br-external) |
The host-to-VM-public-IP limitation is rarely a problem — the host manages VMs
via the overlay (private IPs), not their public IPs.
## How setup-ovn.sh Decides
| Flags | Result |
|-------|--------|
| `--external-bridge --external-iface=eth0` | macvlan created, added to br-external |
| (no `--external-bridge`) | Only br-int created, no WAN connectivity |
## Per-Node Configuration
Different nodes in a cluster can have different WAN NICs: