Root cause: when terminating instances, the vpc.delete-nat event was
published with an empty logicalIP. vpcd's DeleteNAT looked for a
dnat_and_snat rule with logicalIP="" — found nothing — and the real
rule was left as a stale entry in OVN.
When the IPAM later recycled that public IP for an EIP/NAT Gateway,
OVN had conflicting rules: stale dnat_and_snat AND new snat, both
using the same external IP.
Fix: extract PrivateIpAddress from instance.Instance before publishing
vpc.delete-nat, in both running and stopped termination paths.
Also adds 14 unit tests for NAT Gateway service.
E2E: associate subnet with route table BEFORE adding the NAT GW route.
CreateRoute publishes SNAT events for associated subnets — if the
association doesn't exist yet, no SNAT rule is created in OVN.
Validated locally: bastion SSH hop works, ping 8.8.8.8 passes with
NAT GW, fails without. All 5 steps confirmed.
reset-dev-env.sh: trust Spinifex CA cert before starting services,
fixing TLS cert verify failures on fresh resets.
The bastion SSH hop failed because the private instance's cloud-init
hadn't completed (SSH daemon not ready). The instance was "running"
from the hypervisor perspective but cloud-init takes ~30-40s.
- Replace 2-attempt retry with 30-attempt poll (5s intervals, 150s max)
- Hard failure with OVN debug output instead of silent skip
- Matches how wait_for_ssh works for public instances
- getRouteTable: distinguish ErrKeyNotFound (→ not found) from other
errors (→ log + internal error) instead of mapping all to "not found"
- putRouteTable: log underlying error on marshal/write failures
- KV scan loops (allRouteTablesForVPC, DescribeRouteTables,
DisassociateRouteTable, ReplaceRouteTableAssociation): fail on
transient NATS errors instead of silently skipping entries. Only
skip ErrKeyNotFound (deleted between Keys() and Get())
- ReplaceRouteTableAssociation: compensating write if second KV put
fails — restores association to old table to prevent data loss
- createMainRouteTable: returns error instead of swallowing it
- ensureDefaultIGWRoute: log actual errors at Warn level, not Debug
- E2E test: fail on missing default VPC instead of skipping
The pseudo multi-node test was not passing --external-mode/--external-pool
to spx admin init, so instances never got public IPs. SSH then failed
trying to use hostfwd which doesn't exist with tap networking.
- init_leader_node: pass SPX_EXTERNAL_POOL as --external-mode=pool flags
- e2e.yml: set SPX_EXTERNAL_POOL=192.168.1.220-229 for pseudo test run
After enabling tap networking (public subnet), QEMU no longer uses
hostfwd bindings. The pseudo multi-node test was still grepping for
hostfwd to detect instance distribution and SSH endpoints.
- check_instance_distribution: count nbdkit root-volume processes per
node IP instead of hostfwd-bound QEMU processes
- get_instance_node: match QEMU nbd socket → nbdkit host= IP to
determine which simulated node hosts each instance
- count_instances_per_node: use get_instance_node instead of hostfwd
- SSH connectivity: check PublicIpAddress first (port 22), fall back
to hostfwd — same pattern as single-node and multi-node E2E
admin init builds a NetworkConfig from external networking flags and
passes it to the formation server. When joining nodes poll /formation/status,
they receive the cluster-wide [network] section (external mode, pools,
gateway, DNS, bootstrap VPC IDs). Each joining node auto-detects its
own external_interface and wan_bridge locally.
Fixes multi-node clusters having no [network] section in spinifex.toml
on secondary nodes, which caused no public IP assignment and broke
guest SSH in E2E tests.
The dnat_and_snat rule is removed asynchronously via NATS after instance
termination. A 2s sleep was insufficient. Now polls for up to 30s and
downgrades to WARN if the rule persists (non-fatal — cleanup happens
when the VPC is deleted).
Same fix as single-node: detect PublicIpAddress from describe-instances,
SSH via public IP on port 22. Falls back to QEMU hostfwd on the hosting
node when no public IP is available.
GenerateVolumes did not set TenantID on the VolumeMetadata for root
volumes cloned from AMIs. After the pre-Phase4 tenant fallback was
removed (ae316db), describe-volumes filters volumes by exact tenantID
match — root volumes with empty TenantID became invisible to the
owning account, breaking E2E Phase 5b volume lifecycle tests.
When external_mode is set, dev_networking=false (no QEMU hostfwd NIC).
E2E tests now detect the instance's public IP from describe-instances
and SSH via port 22 on the public IP. Falls back to hostfwd only when
no public IP is available (overlay-only mode).
Applies to Phase 5a-ii, 7b, 7c-pre, and 8b SSH tests.
Verify full networking path: host → public IP → OVN DNAT → VM.
Test outbound internet from VM via OVN SNAT (ping 8.8.8.8).
Falls back to API-only verification if public IP unreachable.
- Add Phase 8b to run-e2e.sh: public subnet instance gets PublicIpAddress
and dnat_and_snat rule; private subnet instance gets neither; no blanket
VPC CIDR SNAT (validates mulga-754 fix)
- Remove legacy external_mode="nat" — only "pool" with source=static or
source=dhcp. Admin init defaults to pool/dhcp when WAN detected.
- Enable dev_networking always (was disabled with external mode, breaking
E2E SSH tests)
- Update bootstrap.sh + e2e.yml: SPX_EXTERNAL_POOL env var configures
static IP pool for CI (230-240 single-node, 240-250 multi-node)
- Fix run-vpc-e2e.sh SNAT assertion for mulga-754 (no blanket SNAT)
- Add unit tests for parseDHCPCDLeasedIP, ReconcileFromKV edge cases,
DHCP pool config validation to meet 60% diff coverage threshold
Replace all ListObjects (V1) usage with ListObjectsV2 across the
objectstore interface, both implementations, all 10 call sites, and
tests. V2 uses ContinuationToken instead of Marker for pagination
and adds KeyCount to responses. The frontend and Predastore already
speak V2 — this aligns the Go backend.
DHCP pool mode requires dhcpcd-base (/usr/sbin/dhcpcd) to obtain
leases from the router. Added to Makefile install-system, setup.sh
binary installer, and source-install docs.
Closes: mulga-753
Julian Sommer