Phase 11 in run-multinode-e2e.sh:
- Create VPC with public + private subnets + IGW
- Launch bastion in public subnet, verify SSH
- Create spread placement group, launch 3 VMs in private subnet
- Validate spread distribution via spx get vms (3 nodes)
- Verify no internet pre-NAT (ping 8.8.8.8 fails from all 3)
- Create NAT GW + EIP + route, poll for OVN SNAT rule
- Verify internet via NAT GW from all 3 nodes (different physical hosts)
- Full cleanup: NAT GW, instances, placement group, VPC resources
In single-node mode, cert E2E discovered all host IPs (WAN, mgmt, VPC)
and tried openssl s_client on each. But awsgw only binds to the
configured host IP, not all interfaces. Management (10.11.x) and VPC
(10.x) IPs have no listener on port 9999, causing false failures.
Split NODE_IPS (all IPs for SAN validation) from SERVICE_IPS (IPs
where awsgw listens) parsed from spinifex.toml. Connection-based
tests (3, 4c, 5, 8) now use SERVICE_IPS.
The NAT GW E2E tests used 2-3 second sleeps before verifying SNAT
rule creation, outbound connectivity, and rule removal. This caused
flaky CI failures when OVN took longer to propagate flows to the
datapath (varies by host load and OVN version).
Replace with retry loops:
- SNAT rule check: poll OVN NB every 1s for up to 30s
- Outbound ping: retry up to 10 attempts with 5s backoff (~50s max)
- SNAT removal: poll up to 15 attempts with 2s backoff (~30s max)
Also dump OVN routes alongside NAT rules on failure for debugging.
Auto-discover all node IPs (WAN, mgmt, VPC) and hostname for server
cert SANs — fixes TLS failures when accessing Spinifex via non-localhost
IPs. Adds CA download endpoint (/api/ca.pem) with install instructions
on the login page. Injects CA cert into guest VM cloud-init so EC2
instances trust Spinifex services without --insecure. Adds 'spx admin
cert renew' command for cert regeneration after IP changes.
Includes certificate validation E2E test suite (run-cert-e2e.sh) with
system trust store, SAN, and in-guest TLS verification across single,
pseudo multi-node, and real multi-node modes.
Closes: mulga-777, mulga-778, mulga-779, mulga-780, mulga-781
Plan: docs/development/feature/cert-manager.md
E2E: associate subnet with route table BEFORE adding the NAT GW route.
CreateRoute publishes SNAT events for associated subnets — if the
association doesn't exist yet, no SNAT rule is created in OVN.
Validated locally: bastion SSH hop works, ping 8.8.8.8 passes with
NAT GW, fails without. All 5 steps confirmed.
reset-dev-env.sh: trust Spinifex CA cert before starting services,
fixing TLS cert verify failures on fresh resets.
The bastion SSH hop failed because the private instance's cloud-init
hadn't completed (SSH daemon not ready). The instance was "running"
from the hypervisor perspective but cloud-init takes ~30-40s.
- Replace 2-attempt retry with 30-attempt poll (5s intervals, 150s max)
- Hard failure with OVN debug output instead of silent skip
- Matches how wait_for_ssh works for public instances
- getRouteTable: distinguish ErrKeyNotFound (→ not found) from other
errors (→ log + internal error) instead of mapping all to "not found"
- putRouteTable: log underlying error on marshal/write failures
- KV scan loops (allRouteTablesForVPC, DescribeRouteTables,
DisassociateRouteTable, ReplaceRouteTableAssociation): fail on
transient NATS errors instead of silently skipping entries. Only
skip ErrKeyNotFound (deleted between Keys() and Get())
- ReplaceRouteTableAssociation: compensating write if second KV put
fails — restores association to old table to prevent data loss
- createMainRouteTable: returns error instead of swallowing it
- ensureDefaultIGWRoute: log actual errors at Warn level, not Debug
- E2E test: fail on missing default VPC instead of skipping
The pseudo multi-node test was not passing --external-mode/--external-pool
to spx admin init, so instances never got public IPs. SSH then failed
trying to use hostfwd which doesn't exist with tap networking.
- init_leader_node: pass SPX_EXTERNAL_POOL as --external-mode=pool flags
- e2e.yml: set SPX_EXTERNAL_POOL=192.168.1.220-229 for pseudo test run
After enabling tap networking (public subnet), QEMU no longer uses
hostfwd bindings. The pseudo multi-node test was still grepping for
hostfwd to detect instance distribution and SSH endpoints.
- check_instance_distribution: count nbdkit root-volume processes per
node IP instead of hostfwd-bound QEMU processes
- get_instance_node: match QEMU nbd socket → nbdkit host= IP to
determine which simulated node hosts each instance
- count_instances_per_node: use get_instance_node instead of hostfwd
- SSH connectivity: check PublicIpAddress first (port 22), fall back
to hostfwd — same pattern as single-node and multi-node E2E
The dnat_and_snat rule is removed asynchronously via NATS after instance
termination. A 2s sleep was insufficient. Now polls for up to 30s and
downgrades to WARN if the rule persists (non-fatal — cleanup happens
when the VPC is deleted).
Same fix as single-node: detect PublicIpAddress from describe-instances,
SSH via public IP on port 22. Falls back to QEMU hostfwd on the hosting
node when no public IP is available.
When external_mode is set, dev_networking=false (no QEMU hostfwd NIC).
E2E tests now detect the instance's public IP from describe-instances
and SSH via port 22 on the public IP. Falls back to hostfwd only when
no public IP is available (overlay-only mode).
Applies to Phase 5a-ii, 7b, 7c-pre, and 8b SSH tests.
Verify full networking path: host → public IP → OVN DNAT → VM.
Test outbound internet from VM via OVN SNAT (ping 8.8.8.8).
Falls back to API-only verification if public IP unreachable.
- Add Phase 8b to run-e2e.sh: public subnet instance gets PublicIpAddress
and dnat_and_snat rule; private subnet instance gets neither; no blanket
VPC CIDR SNAT (validates mulga-754 fix)
- Remove legacy external_mode="nat" — only "pool" with source=static or
source=dhcp. Admin init defaults to pool/dhcp when WAN detected.
- Enable dev_networking always (was disabled with external mode, breaking
E2E SSH tests)
- Update bootstrap.sh + e2e.yml: SPX_EXTERNAL_POOL env var configures
static IP pool for CI (230-240 single-node, 240-250 multi-node)
- Fix run-vpc-e2e.sh SNAT assertion for mulga-754 (no blanket SNAT)
- Add unit tests for parseDHCPCDLeasedIP, ReconcileFromKV edge cases,
DHCP pool config validation to meet 60% diff coverage threshold
Redistribute E2E tests between single-node and multi-node suites, slim
down pseudo multi-node suite, and remove directory creation from
dev-setup.sh so spx admin init is the single source of truth for
~/spinifex/ setup.
Julian Sommer