The dnat_and_snat rule is removed asynchronously via NATS after instance
termination. A 2s sleep was insufficient. Now polls for up to 30s and
downgrades to WARN if the rule persists (non-fatal — cleanup happens
when the VPC is deleted).
Same fix as single-node: detect PublicIpAddress from describe-instances,
SSH via public IP on port 22. Falls back to QEMU hostfwd on the hosting
node when no public IP is available.
GenerateVolumes did not set TenantID on the VolumeMetadata for root
volumes cloned from AMIs. After the pre-Phase4 tenant fallback was
removed (ae316db), describe-volumes filters volumes by exact tenantID
match — root volumes with empty TenantID became invisible to the
owning account, breaking E2E Phase 5b volume lifecycle tests.
When external_mode is set, dev_networking=false (no QEMU hostfwd NIC).
E2E tests now detect the instance's public IP from describe-instances
and SSH via port 22 on the public IP. Falls back to hostfwd only when
no public IP is available (overlay-only mode).
Applies to Phase 5a-ii, 7b, 7c-pre, and 8b SSH tests.
Verify full networking path: host → public IP → OVN DNAT → VM.
Test outbound internet from VM via OVN SNAT (ping 8.8.8.8).
Falls back to API-only verification if public IP unreachable.
- Add Phase 8b to run-e2e.sh: public subnet instance gets PublicIpAddress
and dnat_and_snat rule; private subnet instance gets neither; no blanket
VPC CIDR SNAT (validates mulga-754 fix)
- Remove legacy external_mode="nat" — only "pool" with source=static or
source=dhcp. Admin init defaults to pool/dhcp when WAN detected.
- Enable dev_networking always (was disabled with external mode, breaking
E2E SSH tests)
- Update bootstrap.sh + e2e.yml: SPX_EXTERNAL_POOL env var configures
static IP pool for CI (230-240 single-node, 240-250 multi-node)
- Fix run-vpc-e2e.sh SNAT assertion for mulga-754 (no blanket SNAT)
- Add unit tests for parseDHCPCDLeasedIP, ReconcileFromKV edge cases,
DHCP pool config validation to meet 60% diff coverage threshold
Replace all ListObjects (V1) usage with ListObjectsV2 across the
objectstore interface, both implementations, all 10 call sites, and
tests. V2 uses ContinuationToken instead of Marker for pagination
and adds KeyCount to responses. The frontend and Predastore already
speak V2 — this aligns the Go backend.
DHCP pool mode requires dhcpcd-base (/usr/sbin/dhcpcd) to obtain
leases from the router. Added to Makefile install-system, setup.sh
binary installer, and source-install docs.
Closes: mulga-753
handleIGWAttach and reconcileIGW created a blanket snat rule for the
entire VPC CIDR, allowing private subnet instances (no public IP) to
route outbound through the IGW. AWS behavior: only instances with
public IPs route through IGW via per-VM dnat_and_snat rules. Private
subnets require a NAT Gateway for outbound internet.
Also fixes pre-existing lint issues in external_ipam_dhcp.go (gofmt,
modernize, nolintlint).
Closes: mulga-754
Adds source="dhcp" option to external pool config. When set, ExternalIPAM
obtains public IPs from the upstream router's DHCP server (via dhcpcd on
br-ext) instead of allocating from a static range_start..range_end.
The VM behavior is identical to static pool mode: private VPC IP from OVN
DHCP, public IP via DNAT+SNAT, full DescribeInstances support. The only
difference is where the public IP comes from.
Two separate DHCP conversations:
- Host ↔ Router DHCP (br-ext): "give me a public IP for this VM's NAT"
- VM ↔ OVN DHCP (br-int): "give me my private VPC IP"
Tested: DHCP pool mode on single-node (ben-banksia). VM got public IP
192.168.1.77 from router DHCP, SSH works, outbound internet works.
Explains static vs DHCP IP sources in pool mode, the two-DHCP-conversations
architecture, when to use each, and adds a DHCP homelab deployment example.
Also updates bridge setup section for veth pair approach.
setup-ovn.sh now detects whether the default route is on a Linux bridge
or OVS bridge. When it's a Linux bridge (e.g. br-wan from cloud-init),
instead of destructively converting it, the script creates an OVS bridge
(br-ext) and links them with a veth pair. The Linux bridge keeps its IP,
routes, and ports untouched — no WAN interruption.
Also fixes all pre-existing lint issues (gofmt, ineffassign, errchkjson,
gosec, modernize, whitespace) to bring golangci-lint to 0 issues.
Tested: pool mode with static IP range on single-node (ben-banksia).
VM gets public IP 192.168.1.201, SSH works, outbound internet works.
Simplify all 6 tenant-check sites in volume handlers from
`if accountID != "" && tenantID != "" && tenantID != accountID`
to `if tenantID != accountID`. No pre-Phase4 volumes with empty
TenantID exist in any environment — this code was unreachable.
Remove vol-legacy test fixture and empty-accountID-sees-all assertion.
The second admin init --force was called without external networking
flags, overwriting the pool/nat config from the first call with
auto-detected NAT+DHCP defaults.
In NAT mode, all VMs share a single gateway IP for SNAT. No per-VM
public IPs are allocated. Previously, external IPAM was initialized
for any external_mode != "", causing noisy AllocateIP failures (empty
pool range) on every instance launch.
Now IPAM is only initialized when external_mode == "pool". In NAT mode,
d.externalIPAM stays nil, cleanly skipping the auto-assign public IP
path. SNAT topology is still created by vpcd via the pool's gateway_ip.
Part of mulga-737 (Phase 16).
Replace all br-external references with br-wan. Update setup-ovn.sh
command examples to use --wan-bridge/--wan-iface/--macvlan instead of
--external-bridge/--external-iface/--direct.
Part of mulga-742.
- All guides: OVN setup before admin init (consistent ordering)
- Single-node + source-install: Replace --external-bridge/--external-iface
with auto-detection + --wan-bridge/--wan-iface/--macvlan options
- Multi-node: Simplify examples (auto-detect WAN bridge), show override
options in prose
- setup.sh: Update "next steps" to match single-node guide, add
--wan-bridge/--macvlan guidance for physical NIC setups
Part of mulga-742.
- config.go: Add WanBridge field to VPCDConfig (default "br-wan")
- spinifex.toml template: Add wan_bridge under [nodes.X.vpcd]
- admin.go: Auto-detect WAN bridge name from default route interface,
add detectedWanBridge helper
- service.go: Wire WanBridge from cluster config to vpcd Config
- vpcd.go: Add WanBridge to Config, log it on startup, update comments
- network_detect.go: Fix isVirtualInterface to accept the default route
device as a parameter instead of hardcoding br-external. Bridges like
br-wan are now correctly treated as physical when they carry the
default route.
Part of mulga-741.
reset-dev-env.sh: Replace --external-bridge/--external-iface/--direct
with --wan-bridge/--wan-iface/--macvlan. Auto-detect when WAN interface
is already a bridge (tofu-cluster) vs physical NIC.
stop-dev.sh: Replace hardcoded br-external check with dynamic scan of
all OVS bridges — any non-br-int bridge with an IP means OVS is
infrastructure and should not be stopped.
Part of mulga-740.
Replace --external-bridge/--external-iface/--direct with:
- --wan-bridge=NAME: OVS bridge for WAN traffic (auto-detected from
default route if it's already a bridge)
- --wan-iface=NAME: physical NIC to add to the WAN bridge
- --macvlan: SSH-safe macvlan mode for single-NIC hosts
Auto-detection logic:
- If default route goes through a bridge (br-wan) → use it directly
- If default route is a physical NIC → stop with guidance (never
auto-move a NIC into OVS — may drop SSH)
Bridge-mappings updated: external:$WAN_BRIDGE (was external:br-external).
OVN logical network name stays "external" (no topology.go change needed).
Encap IP now prefers br-vpc IP when available (dedicated VPC data plane).
Part of mulga-739. Blocks VPC E2E work.
Phase 12b: vpcd now reconciles OVN topology from NATS KV on startup
(Pass 2). After subscribing to topics, it reads all VPCs, subnets,
IGW attachments, and ENIs from KV and ensures matching OVN objects
exist. Handles reboots, OVN DB loss, and missed events. All operations
are idempotent.
Phase 16 prep: Add --gateway-ip flag to admin init for NAT mode
(specifies the OVN gateway router's external IP without DHCP).
Update reset-dev-env.sh to support EXTERNAL_MODE=nat env var for
NAT/DHCP mode testing alongside the default pool mode.
Auto-detect bridge topology: if a dedicated WAN NIC exists (not used for
SSH), add it directly to br-external instead of creating a macvlan.
Direct bridge enables distributed NAT (DNAT on VM's own chassis) and
eliminates macvlan workarounds (MAC alignment, gratuitous ARPs,
centralized NAT bottleneck).
- topology.go: bridge-mode-aware NAT and localnet port options
- vpcd.go: auto-detect bridge mode, skip MAC alignment for direct bridge
- setup-ovn.sh: --direct flag for physical NIC on br-external
- reset-dev-env.sh: SSH NIC detection via SSH_CONNECTION
Ben Duncan