Direct bridge (WAN NIC added directly to OVS) is preferred when a
dedicated WAN NIC exists. Enables distributed NAT, no MAC workarounds,
host can reach VMs. Macvlan is fallback for single-NIC hosts only.
Three blockers fixed for VPC public subnet E2E on single-node:
1. OVN SB stale data: reset-dev-env.sh now deletes both NB+SB DB files
instead of individual NB cleanup. Eliminates ovn-controller commit
failure loop from stale chassis/port binding entries.
2. Chassis name mismatch: reset-dev-env.sh passes --chassis-id=chassis-node1
to setup-ovn.sh, aligning OVS system-id with vpcd's chassis-{nodeName}.
3. Macvlan unicast filtering: switched to centralized NAT (no external_mac
on DNAT rules), set nat-addresses=router on localnet ports, and vpcd
aligns macvlan MAC with OVN gateway router MAC on startup. Macvlan only
delivers unicast matching its own MAC — this ensures inbound traffic
from the LAN reaches OVS/OVN.
Also fixed: DNS for VMs. admin init auto-detects host DNS servers via
resolvectl, stores them in spinifex.toml dns_servers field, and vpcd
propagates them to OVN DHCP options. Fallback: 8.8.8.8 + 1.1.1.1.
Validated E2E: inbound ping, SSH, outbound ping, DNS, curl, apt update.
- admin init now pre-generates VPC/subnet/IGW IDs and writes [bootstrap]
to spinifex.toml before services ever start. No race condition.
- EnsureDefaultVPC accepts optional BootstrapIDs to use pre-determined
IDs instead of generating random ones.
- CreateInternetGatewayWithID allows daemon to use the bootstrap IGW ID.
- Removed writeBootstrapConfig from daemon (no longer needed).
- Fixed bootstrap account_id: uses admin account (000000000001) not
system account (000000000000).
- reconcileIGW now sets spinifex:igw_id in OVN external IDs.
Phase 11b: RunInstances now auto-resolves the default subnet when no
SubnetId is specified, matching AWS behavior. Previously fell back to
SLIRP networking.
Phase 12: vpcd reconciles OVN topology from [bootstrap] config on startup.
The daemon writes a [bootstrap] section to spinifex.toml with the default
VPC/subnet IDs on first run. vpcd reads this on startup and creates any
missing OVN objects (router, switch, DHCP, IGW topology, SNAT, gateway
chassis). All operations are idempotent — safe on every restart.
This solves the bootstrap ordering problem where admin init creates VPC
records before vpcd is running, so NATS events are missed.
EnsureDefaultVPC now returns DefaultVPCInfo with VPC/subnet IDs.
The single-nic IP migration path was fragile and caused connectivity
issues during setup. macvlan is now the only bridge strategy — always
creates spx-ext-{nic} on the WAN NIC. Removed --single-nic flag,
SingleNIC config/detection fields, and updated all docs.
Phase 8a: Wire all 5 EIP gateway actions (AllocateAddress, ReleaseAddress,
AssociateAddress, DisassociateAddress, DescribeAddresses) end-to-end through
gateway → NATS → daemon → EIP service.
Phase 8b: Wire all 7 SG gateway actions (Create/Delete/Describe SecurityGroup,
Authorize/Revoke Ingress/Egress) through the same pipeline.
Critical fix: handleIGWAttach was not calling SetGatewayChassis on the gateway
router port. Without this, OVN has no chassis binding for external traffic —
SNAT/default route exist logically but packets have nowhere to go physically.
This was the root cause of outbound routing failure (can't ping 8.8.8.8).
Default VPC now matches AWS behavior: auto-creates IGW (attached),
MapPublicIpOnLaunch=true on default subnet, and a default security group
with intra-SG inbound allow + all outbound allow.
Comprehensive VPC networking docs rewrite covering modes, bridge setup,
configuration, security groups, EIPs, OVN reference, and troubleshooting.
EnsureDefaultVPC was only called for GlobalAccountID (000000000000).
The admin account (000000000001) from admin init never got a default
VPC because seedAdminAccount didn't publish iam.account.created.
Fix: daemon startup now ensures default VPC for both system and admin
accounts. Also publish iam.account.created from seedAdminAccount so
any future account-creation listeners fire correctly.
All 3 install guides (binary, multi-node, source) now end at the same
verification point: aws ec2 describe-instance-types. Installation is
complete when that command returns instance types.
New docs/setting-up-your-cluster/README.md is the common guide all 3
install paths point to. Covers: AMI import, SSH key creation, VPC with
public subnet (IGW + modify-subnet-attribute), instance launch with
public IP, SSH via public IP, instance lifecycle management.
Removed SSH-via-hostfwd from all guides — public subnet is the default
connectivity path. Private subnet documented as an alternative.
When setup-ovn.sh migrates the WAN IP from a physical NIC to
br-external, systemd-resolved loses its DNS association (DNS servers
are per-interface). This breaks hostname resolution for everything
on the host — git, apt, curl, SSH all fail.
Now captures DNS config from the original interface before migration
and re-associates it with br-external via resolvectl. Falls back to
8.8.8.8 if no DNS was configured on the original interface.
When setup-ovn.sh migrates the host's WAN IP to br-external, stopping
openvswitch-switch destroys the bridge and the host loses all network
connectivity — unreachable via SSH, requires console reboot.
Now checks if br-external has an IP before stopping OVS. If it does,
skip the OVS stop and only stop ovn-controller + ovn-central. OVS is
infrastructure — bridges must survive service stop/start cycles.
Discovered during 3-node E2E testing: stop-dev.sh killed all 3 nodes'
WAN connectivity simultaneously.
Admin init output was duplicating install guide steps. Simplified to
just show completion message. Install guides already have all steps.
Set dev_networking=false when external_mode is configured — the hostfwd
NIC adds a confusing second interface inside VMs and is not needed when
public subnets provide WAN-routable IPs.
Wire public IP auto-assignment end-to-end:
- RunInstances checks MapPublicIpOnLaunch on subnet, allocates external
IP from ExternalIPAM, updates ENI record, publishes vpc.add-nat for
dnat_and_snat rule, returns PublicIpAddress in response
- TerminateInstances releases public IP back to pool, publishes
vpc.delete-nat to remove NAT rule before ENI deletion
- DescribeInstances populates PublicIpAddress from VM struct
Add ExternalIPAM initialization in daemon startup from cluster
NetworkConfig. Add GetSubnet and UpdateENIPublicIP helpers. Add
PublicIP/PublicIPPool fields to vm.VM struct. Fix network auto-detect
to recognize br-external as WAN interface.
spx admin init now works with zero external networking flags:
- Auto-detects WAN interface, gateway, prefix from ip route
- Auto-detects single-NIC vs multi-NIC mode
- Defaults to nat mode with DHCP for gateway IP
- Prints detected topology with interface roles
setup-ovn.sh gains --dhcp flag: obtains a DHCP lease on the
macvlan/bridge interface for the OVN gateway SNAT address, then
writes gateway_ip into spinifex.toml. Tested on VM: macvlan gets
separate IP from router DHCP, host keeps its original IP.
Three tiers of external networking:
- Zero flags: auto nat+DHCP (VMs get outbound internet)
- --external-pool=start-end: pool mode (VMs get public IPs)
- --no-external: overlay only (no internet for VMs)
Add --single-nic flag to setup-ovn.sh. When used with --external-bridge,
creates a macvlan sub-interface (spx-ext-{iface}) in bridge mode off the
host's NIC instead of adding the NIC directly to br-external. The host
keeps its IP — no migration, SSH-safe.
Strategy selection:
- Dedicated NIC: --external-iface=eth1 (add directly)
- Single NIC: --external-iface=eth0 --single-nic (macvlan)
- IP migration: --external-iface=eth0 (legacy, no --single-nic)
Add single_nic bool field to VPCDConfig for per-node config. Different
nodes in a cluster can use different strategies. Unit test verifies
single_nic parsing alongside external_interface.
Add security group CRUD handlers: CreateSecurityGroup, DeleteSecurityGroup,
DescribeSecurityGroups, AuthorizeSecurityGroupIngress/Egress,
RevokeSecurityGroupIngress/Egress. SG records stored in NATS KV.
Add ACL match expression compiler (acl.go) translating AWS SG rules to
OVN ACL match strings for TCP/UDP port/ranges, ICMP, all-protocols, CIDR
filters, and intra-SG references.
Add vpcd security.go handlers for OVN Port Group lifecycle and ACL
management via vpc.create-sg/delete-sg/update-sg NATS topics. Extend
OVNClient interface with CreatePortGroup, DeletePortGroup,
SetPortGroupPorts, AddACL, ClearACLs. Add PortGroup and ACL to nbdb
model.
Add EIP service with AllocateAddress, ReleaseAddress, AssociateAddress,
DisassociateAddress, and DescribeAddresses. Uses ExternalIPAM for IP
allocation from admin-defined pools. Publishes vpc.add-nat/delete-nat
events for OVN dnat_and_snat rules.
EIP records stored in spinifex-vpc-elastic-ips NATS KV bucket with
account-scoped keys. AssociateAddress replaces auto-assigned public IPs
when present.
Add MapPublicIpOnLaunch field to SubnetRecord with ModifySubnetAttribute
API handler. Add PublicIpAddress/PublicIpPool fields to ENIRecord for
tracking auto-assigned public IPs.
Add vpc.add-nat / vpc.delete-nat NATS topics and handlers in vpcd
topology for creating distributed dnat_and_snat NAT rules (with
LogicalPort + ExternalMAC for chassis-local NAT processing).
Wire ModifySubnetAttribute through gateway → daemon → VPC service.
Enable PR-triggered CI so required status checks can gate merges.
Spinifex runs on all PRs (no path filter) to avoid blocking non-Go PRs.
UI workflow keeps path filters as it's not a required check.
Add external IPAM module (CAS-based IP allocation from admin-defined
pools in NATS KV). Supports multi-pool with region/AZ scoping, pool
fallback logic, gateway IP reservation, and concurrent allocation via
CAS retries. 12 unit tests covering allocation, release, exhaustion,
CAS conflict, multi-pool, fallback, validation, and idempotent init.
Add --external-bridge flag to setup-ovn.sh: creates OVS bridge
br-external, adds WAN NIC, migrates host IP to bridge, and sets
ovn-bridge-mappings=external:br-external for OVN localnet ports.
Add commented [network] section and external_interface to spinifex.toml
config template for discoverability.
Redistribute E2E tests between single-node and multi-node suites, slim
down pseudo multi-node suite, and remove directory creation from
dev-setup.sh so spx admin init is the single source of truth for
~/spinifex/ setup.
Add cluster-wide [network] config with external_mode and multi-pool
support (ExternalPool with region/AZ scoping). Per-node vpcd gets
external_interface for WAN NIC name.
Fix handleIGWAttach to use real external IPs from the configured pool
instead of link-local 169.254.0.1/30. Gateway router port now gets
the pool's gateway IP with correct prefix length, SNAT uses the real
external IP, and the default route points to the actual WAN gateway.
Includes pool selection logic (AZ-scoped → region-scoped → unscoped
fallback) and unit tests for config parsing, IGW with external pool,
explicit gateway IP, and pool selection.
dependabot[bot]