mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
feat: routed-NAT external mode with two-tier ingress (#475)
* feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper * feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode * feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules) * feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode * feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks * feat(scripts): add --nat-uplink routed mode to setup-ovn.sh * test(e2e): add routed-NAT single-node natuplink suite Covers the external_mode=nat lane end to end on a nat-initialized node: host transit wiring (veth, br-ext port, ip_forward, iptables egress rules), spinifex.toml contents, disabled EIP surface (empty describe-addresses, UnsupportedOperation on allocate), default subnet MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit /24, guest outbound egress proven via a serial-console probe (nat mode has no inbound path), and unique transit gateway IPs across VPCs. Suite self-gates on external_mode=nat; registered as e2e-natuplink and in SUITES_BUILD so it compiles in CI. Manifest entry stays commented until CI can provision a nat-mode environment. * feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column Adds the exempted_ext_ips column (optional strong Address_Set ref) to the nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an in-place SetNATExemptedExtIPs update on the client interface, live client, and mock. Test NB schema gains the column and covers insert-resolves-UUID, converge, clear, and missing-NAT paths. * feat(policy): exempt address set on routed SNAT/EIP rows Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR) with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip path patch the ref in place on pre-existing rows. The set is a singleton strong ref and is never deleted. Non-routed modes are unchanged. * feat(host): VPC ingress route primitives for routed-NAT mode EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the spx-nat veth. Replace/del semantics keep both idempotent; a missing route on removal is not an error. * feat(external): routed ingress leg on IGW attach/detach, reconcile-safe AttachIGW's routed leg now also installs the host VPC ingress route via RoutedIngressHooks, and re-runs on the already-attached path so the reconcile loop restores host routes (and patches legacy SNAT exempt refs) after a reboot — OVN state survives restarts but host routes do not. DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs, validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct pass nil hooks and are unchanged. * test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row pointing at spinifex_nat_exempt (containing the transit /24), and — after opening the SG from the transit net (default-closed, AWS parity) — ping and an sshd TCP handshake against the instance private IP from the host, which proves the un-SNATted return path. Phase 8 additionally waits for the second VPC's ingress route to appear on IGW attach. vpc-networking README documents the jumpbox pattern, SG note, and nat_exempt_cidrs. * feat(config,admin): public wan pool alongside nat-transit Restructure admin config generation from single-pool scalars to a Pools slice so nat mode can emit both the transit pool and an optional public wan pool. nat mode now accepts --external-pool/--external-source (plus --external-bind-bridge/--gateway-ip when a public pool is requested), validated by the shared resolvePublicPoolFlags helper also used by pool mode. Transit pool always renders first so the IGW allocator picks it. Formation wire format unchanged (nat mode is single-node only). * feat(daemon): enable EIP service and auto-assign in nat mode with public pool External IPAM (and with it the real EIP service, RunInstances public IP auto-assign, and MapPublicIpOnLaunch on default subnets) now activates whenever public pools exist: pool mode as before, plus nat mode when a wan pool rides alongside the transit segment. The transit pool is filtered out of IPAM — its addresses are gateway-LRP plumbing, never allocatable EIPs. Transit-only nat configs keep the disabled EIP stub. * feat(host): per-EIP ingress primitives for routed-NAT mode DetectUplinkFor parses ip route get to find the interface and source IP facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move). EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC gateway LRP transit IP (src-hinted so host-local traffic still DNATs), answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC, WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated. * feat(policy,vpcd): host EIP binder option on NATManager WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds after priming on both the fresh-add and idempotent-skip paths (host routes and proxy-ARP are volatile even when the OVN row survives, so reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort. The next hop comes from the new gatewayPortIP helper reading the gateway LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID key now an exported policy const shared with the external package. vpcd selects the IGW pool by name instead of index — with two pools in nat mode the gateway allocator must stay on the transit pool — and wires the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced pools (gateway unknown until lease) proxy-ARP on their bind bridge. * feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection dhcp_mac = "derived" (default) | "interface" on external pools. The interface variant leases with the bind interface's own MAC — WiFi/WWAN uplinks drop frames with foreign source MACs, so derived chaddr MACs never complete DORA there. Option 61 stays the lease discriminator. UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and persists through KV so restart re-acquires repeat it. Routers that key leases on MAC and ignore client-id would hand every lease the same IP; the manager detects an ACK matching the interface's own address or another client-id's live lease, releases the colliding lease, and hard-errors advising a static range instead. admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp pools bound to non-bridgeable uplinks (wlan/wwan/ppp). * test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs The natuplink suite now detects a public (non-transit) pool in spinifex.toml and switches lanes: Tier-1-only configs keep the EIP-disabled and no-public-IP assertions; public-pool configs assert the EIP surface is enabled, the default subnet has MapPublicIpOnLaunch=true, instances auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink, per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the EIP, reconcile replay across a vpcd restart, and teardown on disassociate. Docs: vpc-networking README gains a public-pool section for routed-NAT nodes (init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool. * fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd Routed-NAT mode shells out to iptables for the transit masquerade and per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for host-delivered EIPs. Without these sudoers entries vpcd crash-loops with "command not allowed" on nat-mode nodes. * fix(ovn): scope DeleteStaticRoute prefix match to the target router Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache match could grab another router's row and fail with a referential integrity violation, wedging IGW attach on multi-VPC nodes. Match only rows referenced by the target router, mirroring FindStaticRoute. Mock updated to the same semantics, with a regression test covering drift replacement across two routers. * fix(reconcile): always replay AttachIGW so host ingress state is re-ensured applyIGWs skipped AttachIGW whenever the gateway switch port existed in actual state, so the already-attached re-ensure path never ran and routed-NAT host ingress routes lost on reboot were never reinstalled. AttachIGW is idempotent; run it unconditionally. * fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes collide on the single host route table and the last IGW attach silently stole the route (route replace). Pass vpcID and the gateway LRP IP through RoutedIngressHooks; the bootstrap VPC always installs its route, other VPCs only install when the CIDR is free and only remove a route their own gateway IP holds. Adds host.VPCIngressRouteVia to read the current route holder. * ci(e2e): add nightly cell 19 for routed-NAT single-node suite Cell 19 bootstraps with --external-mode=nat plus the CI static pool as the public wan pool and runs the natuplink suite, covering the Tier-1 transit datapath and Tier-2 EIP ingress. The run step exits 0 when natuplink.test is absent from the tests tar so scheduled runs against branches predating the suite stay green. * fix(admin): nat-mode init summary reflects configured public pool The summary printed 'outbound-only — no public IPs/EIPs' even when --external-pool/--external-source configured a public wan pool. Print the pool range (or dhcp bind) and note EIPs are enabled. * fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites Reconcile the routed-NAT branch with dev's ctx-first service APIs after rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService interface, the vm_adapters public-IP resolver asserts the ctx-signed AssociatedPublicIPForInstance, and the default-subnet mapping test threads ctx into DescribeSubnets.
This commit is contained in:
Josh Siviour
committed by
Julian Sommer
Julian Sommer
parent
a7b689fec3
commit
e1c4f85a36
+129
-2
@@ -18,6 +18,11 @@
|
||||
# --wan-bridge=NAME OVS bridge for WAN traffic (default: auto-detect from default route)
|
||||
# --wan-iface=NAME Physical NIC to add to the WAN bridge (use with --wan-bridge)
|
||||
# --dhcp Obtain gateway IP via DHCP on the WAN bridge interface
|
||||
# --nat-uplink Routed NAT mode: no WAN NIC is bridged. Creates br-ext
|
||||
# with a transit veth (spx-nat-host 100.127.0.1/24) and
|
||||
# host masquerade rules; VMs get outbound-only WAN over
|
||||
# any uplink (ethernet, WiFi, cellular, PPP). Pair with
|
||||
# `spx admin init --external-mode=nat`.
|
||||
# --mgmt-bridge=NAME OVS bridge for system-instance control plane (default: br-mgmt)
|
||||
# --mgmt-cidr=CIDR IPv4 CIDR to assign on the mgmt bridge (default: 10.15.8.1/24)
|
||||
# --mgmt-iface=NAME Physical/virtual NIC to enslave to the mgmt bridge (multi-node only)
|
||||
@@ -62,6 +67,9 @@
|
||||
#
|
||||
# # No WAN bridge (overlay-only, no public subnet):
|
||||
# ./scripts/setup-ovn.sh --management --encap-ip=10.0.0.1
|
||||
#
|
||||
# # Non-bridgeable uplink (WiFi/cellular) — routed NAT, outbound-only VMs:
|
||||
# ./scripts/setup-ovn.sh --management --nat-uplink
|
||||
|
||||
set -e
|
||||
|
||||
@@ -70,6 +78,7 @@ MANAGEMENT=false
|
||||
WAN_BRIDGE=""
|
||||
WAN_IFACE=""
|
||||
EXTERNAL_DHCP=false
|
||||
NAT_UPLINK=false
|
||||
MGMT_BRIDGE_ENABLED=true
|
||||
MGMT_BRIDGE="br-mgmt"
|
||||
MGMT_CIDR="10.15.8.1/24"
|
||||
@@ -95,6 +104,7 @@ for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--management) MANAGEMENT=true ;;
|
||||
--dhcp) EXTERNAL_DHCP=true ;;
|
||||
--nat-uplink) NAT_UPLINK=true ;;
|
||||
--wan-bridge=*) WAN_BRIDGE="${arg#*=}" ;;
|
||||
--wan-iface=*) WAN_IFACE="${arg#*=}" ;;
|
||||
--mgmt-bridge=*) MGMT_BRIDGE="${arg#*=}" ;;
|
||||
@@ -204,12 +214,28 @@ detect_wan_bridge() {
|
||||
echo ""
|
||||
echo " 3. No external networking (overlay-only):"
|
||||
echo " ./scripts/setup-ovn.sh --management --encap-ip=$wan_ip"
|
||||
echo ""
|
||||
echo " 4. Non-bridgeable uplink (WiFi/cellular/PPP) — routed NAT mode"
|
||||
echo " (VMs get outbound-only internet, no public IPs):"
|
||||
echo " ./scripts/setup-ovn.sh --management --nat-uplink"
|
||||
echo " then: spx admin init --external-mode=nat"
|
||||
echo "============================================================"
|
||||
echo ""
|
||||
exit 1
|
||||
}
|
||||
|
||||
detect_wan_bridge
|
||||
if [ "$NAT_UPLINK" = true ]; then
|
||||
# Routed NAT: nothing is bridged, so the uplink type is irrelevant.
|
||||
if [ -n "$WAN_BRIDGE" ] || [ -n "$WAN_IFACE" ]; then
|
||||
echo "ERROR: --nat-uplink does not take --wan-bridge/--wan-iface (no WAN NIC is bridged)"
|
||||
exit 1
|
||||
fi
|
||||
WAN_BRIDGE="br-ext"
|
||||
WAN_BRIDGE_MODE="nat"
|
||||
echo " Routed NAT uplink: br-ext + transit veth, host masquerade (no WAN bridge)"
|
||||
else
|
||||
detect_wan_bridge
|
||||
fi
|
||||
|
||||
# Auto-detect encap IP if not specified
|
||||
if [ -z "$ENCAP_IP" ]; then
|
||||
@@ -413,6 +439,16 @@ if [ -n "$WAN_BRIDGE" ]; then
|
||||
sudo ip link del veth-wan-br 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# Same ripdown for stale routed-NAT plumbing when switching away from nat.
|
||||
if [ "$WAN_BRIDGE_MODE" != "nat" ]; then
|
||||
sudo rm -f /etc/systemd/network/17-spinifex-nat.netdev \
|
||||
/etc/systemd/network/17-spinifex-nat.network \
|
||||
/etc/systemd/network/18-spinifex-nat-ovs.network
|
||||
sudo networkctl reload 2>/dev/null || true
|
||||
sudo ovs-vsctl --if-exists del-port spx-nat-ovs 2>/dev/null || true
|
||||
sudo ip link del spx-nat-host 2>/dev/null || true
|
||||
fi
|
||||
|
||||
case "$WAN_BRIDGE_MODE" in
|
||||
existing)
|
||||
# Already an OVS bridge (from a previous run or explicit --wan-bridge).
|
||||
@@ -560,10 +596,101 @@ NETWORK
|
||||
echo " $WAN_BRIDGE: direct bridge on $WAN_IFACE"
|
||||
echo " NOTE: $WAN_IFACE is now an OVS port — no host IP on this NIC"
|
||||
;;
|
||||
|
||||
nat)
|
||||
# Routed NAT: br-ext carries no WAN NIC. A transit veth pair links
|
||||
# it to the host stack (spx-nat-host owns 100.127.0.1/24); the host
|
||||
# forwards and masquerades the transit /24 out whatever uplink it
|
||||
# has. OVN SNATs each VPC CIDR to its gateway LRP transit IP, so
|
||||
# the masquerade rule below is the only host-side NAT state.
|
||||
NAT_TRANSIT_CIDR="100.127.0.0/24"
|
||||
NAT_TRANSIT_GW_CIDR="100.127.0.1/24"
|
||||
|
||||
if ! sudo ovs-vsctl br-exists "$WAN_BRIDGE" 2>/dev/null; then
|
||||
sudo ovs-vsctl --may-exist add-br "$WAN_BRIDGE"
|
||||
echo " created OVS bridge: $WAN_BRIDGE"
|
||||
fi
|
||||
sudo ip link set "$WAN_BRIDGE" up
|
||||
|
||||
# Create transit veth pair (idempotent)
|
||||
if ! ip link show spx-nat-host >/dev/null 2>&1; then
|
||||
sudo ip link add spx-nat-host type veth peer name spx-nat-ovs
|
||||
echo " created veth pair: spx-nat-host ↔ spx-nat-ovs"
|
||||
else
|
||||
echo " veth pair already exists: spx-nat-host ↔ spx-nat-ovs"
|
||||
fi
|
||||
sudo ip addr replace "$NAT_TRANSIT_GW_CIDR" dev spx-nat-host
|
||||
|
||||
# Add the OVS end to br-ext
|
||||
if ! sudo ovs-vsctl port-to-br spx-nat-ovs >/dev/null 2>&1; then
|
||||
sudo ovs-vsctl --may-exist add-port "$WAN_BRIDGE" spx-nat-ovs
|
||||
echo " spx-nat-ovs → $WAN_BRIDGE (OVS bridge)"
|
||||
fi
|
||||
sudo ip link set spx-nat-host up
|
||||
sudo ip link set spx-nat-ovs up
|
||||
echo " host (spx-nat-host $NAT_TRANSIT_GW_CIDR) ↔ veth pair ↔ $WAN_BRIDGE (OVS)"
|
||||
|
||||
# Persist the veth pair + transit IP across reboot (veths are
|
||||
# kernel-only; same rationale as veth mode above).
|
||||
NAT_NETDEV="/etc/systemd/network/17-spinifex-nat.netdev"
|
||||
NAT_NETWORK="/etc/systemd/network/17-spinifex-nat.network"
|
||||
NAT_OVS_NETWORK="/etc/systemd/network/18-spinifex-nat-ovs.network"
|
||||
sudo tee "$NAT_NETDEV" >/dev/null <<NETDEV
|
||||
[NetDev]
|
||||
Name=spx-nat-host
|
||||
Kind=veth
|
||||
|
||||
[Peer]
|
||||
Name=spx-nat-ovs
|
||||
NETDEV
|
||||
sudo tee "$NAT_NETWORK" >/dev/null <<NETWORK
|
||||
[Match]
|
||||
Name=spx-nat-host
|
||||
|
||||
[Network]
|
||||
Address=$NAT_TRANSIT_GW_CIDR
|
||||
ConfigureWithoutCarrier=yes
|
||||
NETWORK
|
||||
# Admin-up the OVS end after reboot (OVS owns the port but does
|
||||
# not flip admin state on external ports — same as veth mode).
|
||||
sudo tee "$NAT_OVS_NETWORK" >/dev/null <<NETWORK
|
||||
[Match]
|
||||
Name=spx-nat-ovs
|
||||
|
||||
[Link]
|
||||
RequiredForOnline=no
|
||||
|
||||
[Network]
|
||||
ConfigureWithoutCarrier=yes
|
||||
NETWORK
|
||||
sudo networkctl reload 2>/dev/null || true
|
||||
echo " wrote $NAT_NETDEV + $NAT_NETWORK + $NAT_OVS_NETWORK (transit veth persists on reboot)"
|
||||
|
||||
# Kernel egress: masquerade the transit /24 out any uplink and
|
||||
# accept forwarded transit traffic even under FORWARD-policy DROP.
|
||||
# vpcd re-ensures these on every start; installing here too means
|
||||
# the wiring is testable before services run.
|
||||
sudo iptables -t nat -C POSTROUTING -s "$NAT_TRANSIT_CIDR" ! -d "$NAT_TRANSIT_CIDR" \
|
||||
-m comment --comment "spinifex-nat-egress" -j MASQUERADE 2>/dev/null || \
|
||||
sudo iptables -t nat -A POSTROUTING -s "$NAT_TRANSIT_CIDR" ! -d "$NAT_TRANSIT_CIDR" \
|
||||
-m comment --comment "spinifex-nat-egress" -j MASQUERADE
|
||||
sudo iptables -C FORWARD -i spx-nat-host -s "$NAT_TRANSIT_CIDR" \
|
||||
-m comment --comment "spinifex-nat-egress" -j ACCEPT 2>/dev/null || \
|
||||
sudo iptables -A FORWARD -i spx-nat-host -s "$NAT_TRANSIT_CIDR" \
|
||||
-m comment --comment "spinifex-nat-egress" -j ACCEPT
|
||||
sudo iptables -C FORWARD -o spx-nat-host -m conntrack --ctstate RELATED,ESTABLISHED \
|
||||
-m comment --comment "spinifex-nat-egress" -j ACCEPT 2>/dev/null || \
|
||||
sudo iptables -A FORWARD -o spx-nat-host -m conntrack --ctstate RELATED,ESTABLISHED \
|
||||
-m comment --comment "spinifex-nat-egress" -j ACCEPT
|
||||
echo " installed masquerade + forward rules for $NAT_TRANSIT_CIDR (comment: spinifex-nat-egress)"
|
||||
;;
|
||||
esac
|
||||
|
||||
# --- DHCP: obtain gateway IP for OVN SNAT ---
|
||||
if [ "$EXTERNAL_DHCP" = true ]; then
|
||||
if [ "$EXTERNAL_DHCP" = true ] && [ "$WAN_BRIDGE_MODE" = "nat" ]; then
|
||||
echo ""
|
||||
echo " Skipping --dhcp: routed NAT mode has a fixed transit gateway (100.127.0.1)"
|
||||
elif [ "$EXTERNAL_DHCP" = true ]; then
|
||||
echo ""
|
||||
echo "Step 3c: Obtaining external gateway IP via DHCP..."
|
||||
|
||||
|
||||
@@ -198,6 +198,11 @@ spinifex-vpcd ALL=(root) NOPASSWD: /usr/bin/ovs-vsctl, /usr/bin/ovs-appctl
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /usr/bin/ovn-nbctl, /usr/bin/ovn-sbctl, /usr/bin/ovn-appctl
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /usr/bin/systemctl is-active --quiet ovn-controller
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /sbin/ip, /usr/sbin/ip
|
||||
# Routed-NAT mode: transit masquerade + per-EIP FORWARD accepts, proxy-ARP
|
||||
# delay tune, and gratuitous ARP announcements for host-delivered EIPs.
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /usr/sbin/iptables, /sbin/iptables
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /usr/sbin/sysctl -w net.ipv4.neigh.*
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /usr/sbin/arping, /usr/bin/arping
|
||||
SUDOERS
|
||||
$SUDO chmod 0440 /etc/sudoers.d/spinifex-network
|
||||
$SUDO visudo -cf /etc/sudoers.d/spinifex-network || fatal "Invalid sudoers syntax in spinifex-network"
|
||||
|
||||
Reference in New Issue
Block a user