setup-ovn.sh's veth branch creates veth-wan-br ↔ veth-wan-ovs but nothing
recreates the kernel interfaces on reboot. OVSDB remembers veth-wan-ovs
as a port on br-ext but the peer is gone — the external path silently
breaks and vpcd falls through to direct mode.
- setup-ovn.sh: write /etc/systemd/network/15-spinifex-veth-wan.{netdev,network}
in the veth branch so systemd-networkd recreates the pair on boot. Add
a single pre-case guard that rips down stale veth persistence + the OVS
port + the kernel veth whenever WAN_BRIDGE_MODE != veth, so mode
transitions are idempotent (D17).
- reset-dev-env.sh: delete the netdev/network units during cleanup so
`make reset` gives a truly clean slate.
- spinifex-vpcd.service: After=systemd-networkd.service openvswitch-switch.service
so vpcd's sanity checks see a populated kernel + OVSDB (D5).
Auto-discover all node IPs (WAN, mgmt, VPC) and hostname for server
cert SANs — fixes TLS failures when accessing Spinifex via non-localhost
IPs. Adds CA download endpoint (/api/ca.pem) with install instructions
on the login page. Injects CA cert into guest VM cloud-init so EC2
instances trust Spinifex services without --insecure. Adds 'spx admin
cert renew' command for cert regeneration after IP changes.
Includes certificate validation E2E test suite (run-cert-e2e.sh) with
system trust store, SAN, and in-guest TLS verification across single,
pseudo multi-node, and real multi-node modes.
Closes: mulga-777, mulga-778, mulga-779, mulga-780, mulga-781
Plan: docs/development/feature/cert-manager.md
Previously hardcoded CHASSIS_ID="chassis-node1" which could mismatch
the OVS system-id from prior setup-ovn.sh runs, creating stale chassis
entries in the SBDB. Now defers to setup-ovn.sh's auto-detect
(chassis-$(hostname -s)) so the name is always consistent.
E2E: associate subnet with route table BEFORE adding the NAT GW route.
CreateRoute publishes SNAT events for associated subnets — if the
association doesn't exist yet, no SNAT rule is created in OVN.
Validated locally: bastion SSH hop works, ping 8.8.8.8 passes with
NAT GW, fails without. All 5 steps confirmed.
reset-dev-env.sh: trust Spinifex CA cert before starting services,
fixing TLS cert verify failures on fresh resets.
setup-ovn.sh now detects whether the default route is on a Linux bridge
or OVS bridge. When it's a Linux bridge (e.g. br-wan from cloud-init),
instead of destructively converting it, the script creates an OVS bridge
(br-ext) and links them with a veth pair. The Linux bridge keeps its IP,
routes, and ports untouched — no WAN interruption.
Also fixes all pre-existing lint issues (gofmt, ineffassign, errchkjson,
gosec, modernize, whitespace) to bring golangci-lint to 0 issues.
Tested: pool mode with static IP range on single-node (ben-banksia).
VM gets public IP 192.168.1.201, SSH works, outbound internet works.
The second admin init --force was called without external networking
flags, overwriting the pool/nat config from the first call with
auto-detected NAT+DHCP defaults.
reset-dev-env.sh: Replace --external-bridge/--external-iface/--direct
with --wan-bridge/--wan-iface/--macvlan. Auto-detect when WAN interface
is already a bridge (tofu-cluster) vs physical NIC.
stop-dev.sh: Replace hardcoded br-external check with dynamic scan of
all OVS bridges — any non-br-int bridge with an IP means OVS is
infrastructure and should not be stopped.
Part of mulga-740.
Phase 12b: vpcd now reconciles OVN topology from NATS KV on startup
(Pass 2). After subscribing to topics, it reads all VPCs, subnets,
IGW attachments, and ENIs from KV and ensures matching OVN objects
exist. Handles reboots, OVN DB loss, and missed events. All operations
are idempotent.
Phase 16 prep: Add --gateway-ip flag to admin init for NAT mode
(specifies the OVN gateway router's external IP without DHCP).
Update reset-dev-env.sh to support EXTERNAL_MODE=nat env var for
NAT/DHCP mode testing alongside the default pool mode.
Auto-detect bridge topology: if a dedicated WAN NIC exists (not used for
SSH), add it directly to br-external instead of creating a macvlan.
Direct bridge enables distributed NAT (DNAT on VM's own chassis) and
eliminates macvlan workarounds (MAC alignment, gratuitous ARPs,
centralized NAT bottleneck).
- topology.go: bridge-mode-aware NAT and localnet port options
- vpcd.go: auto-detect bridge mode, skip MAC alignment for direct bridge
- setup-ovn.sh: --direct flag for physical NIC on br-external
- reset-dev-env.sh: SSH NIC detection via SSH_CONNECTION
Three blockers fixed for VPC public subnet E2E on single-node:
1. OVN SB stale data: reset-dev-env.sh now deletes both NB+SB DB files
instead of individual NB cleanup. Eliminates ovn-controller commit
failure loop from stale chassis/port binding entries.
2. Chassis name mismatch: reset-dev-env.sh passes --chassis-id=chassis-node1
to setup-ovn.sh, aligning OVS system-id with vpcd's chassis-{nodeName}.
3. Macvlan unicast filtering: switched to centralized NAT (no external_mac
on DNAT rules), set nat-addresses=router on localnet ports, and vpcd
aligns macvlan MAC with OVN gateway router MAC on startup. Macvlan only
delivers unicast matching its own MAC — this ensures inbound traffic
from the LAN reaches OVS/OVN.
Also fixed: DNS for VMs. admin init auto-detects host DNS servers via
resolvectl, stores them in spinifex.toml dns_servers field, and vpcd
propagates them to OVN DHCP options. Fallback: 8.8.8.8 + 1.1.1.1.
Validated E2E: inbound ping, SSH, outbound ping, DNS, curl, apt update.
The single-nic IP migration path was fragile and caused connectivity
issues during setup. macvlan is now the only bridge strategy — always
creates spx-ext-{nic} on the WAN NIC. Removed --single-nic flag,
SingleNIC config/detection fields, and updated all docs.
joshsiv-mulga