Auto-discover all node IPs (WAN, mgmt, VPC) and hostname for server
cert SANs — fixes TLS failures when accessing Spinifex via non-localhost
IPs. Adds CA download endpoint (/api/ca.pem) with install instructions
on the login page. Injects CA cert into guest VM cloud-init so EC2
instances trust Spinifex services without --insecure. Adds 'spx admin
cert renew' command for cert regeneration after IP changes.
Includes certificate validation E2E test suite (run-cert-e2e.sh) with
system trust store, SAN, and in-guest TLS verification across single,
pseudo multi-node, and real multi-node modes.
Closes: mulga-777, mulga-778, mulga-779, mulga-780, mulga-781
Plan: docs/development/feature/cert-manager.md
Previously hardcoded CHASSIS_ID="chassis-node1" which could mismatch
the OVS system-id from prior setup-ovn.sh runs, creating stale chassis
entries in the SBDB. Now defers to setup-ovn.sh's auto-detect
(chassis-$(hostname -s)) so the name is always consistent.
Two stacked bugs prevented SSH to VPC instances:
1. The default route (0.0.0.0/0) was added without OutputPort. For
DHCP-sourced pools the gateway router port uses link-local addressing
(169.254.0.1/30) so OVN northd cannot resolve the nexthop (e.g.
192.168.1.1) to an output port — it silently drops the route from
the southbound DB.
2. discoverChassis returned all chassis from the SBDB including stale
entries from previous OVN setups on the same host. When a stale
chassis got highest gateway priority, the chassisredirect port
stayed unbound (up: false), so no ARP/NAT traffic was processed.
Fix: add OutputPort to static routes in both handleIGWAttach and
reconcileIGW, and filter stale chassis by cross-referencing the local
OVS system-id with SBDB hostnames.
Root cause: when terminating instances, the vpc.delete-nat event was
published with an empty logicalIP. vpcd's DeleteNAT looked for a
dnat_and_snat rule with logicalIP="" — found nothing — and the real
rule was left as a stale entry in OVN.
When the IPAM later recycled that public IP for an EIP/NAT Gateway,
OVN had conflicting rules: stale dnat_and_snat AND new snat, both
using the same external IP.
Fix: extract PrivateIpAddress from instance.Instance before publishing
vpc.delete-nat, in both running and stopped termination paths.
Also adds 14 unit tests for NAT Gateway service.
E2E: associate subnet with route table BEFORE adding the NAT GW route.
CreateRoute publishes SNAT events for associated subnets — if the
association doesn't exist yet, no SNAT rule is created in OVN.
Validated locally: bastion SSH hop works, ping 8.8.8.8 passes with
NAT GW, fails without. All 5 steps confirmed.
reset-dev-env.sh: trust Spinifex CA cert before starting services,
fixing TLS cert verify failures on fresh resets.
The bastion SSH hop failed because the private instance's cloud-init
hadn't completed (SSH daemon not ready). The instance was "running"
from the hypervisor perspective but cloud-init takes ~30-40s.
- Replace 2-attempt retry with 30-attempt poll (5s intervals, 150s max)
- Hard failure with OVN debug output instead of silent skip
- Matches how wait_for_ssh works for public instances
- getRouteTable: distinguish ErrKeyNotFound (→ not found) from other
errors (→ log + internal error) instead of mapping all to "not found"
- putRouteTable: log underlying error on marshal/write failures
- KV scan loops (allRouteTablesForVPC, DescribeRouteTables,
DisassociateRouteTable, ReplaceRouteTableAssociation): fail on
transient NATS errors instead of silently skipping entries. Only
skip ErrKeyNotFound (deleted between Keys() and Get())
- ReplaceRouteTableAssociation: compensating write if second KV put
fails — restores association to old table to prevent data loss
- createMainRouteTable: returns error instead of swallowing it
- ensureDefaultIGWRoute: log actual errors at Warn level, not Debug
- E2E test: fail on missing default VPC instead of skipping
The pseudo multi-node test was not passing --external-mode/--external-pool
to spx admin init, so instances never got public IPs. SSH then failed
trying to use hostfwd which doesn't exist with tap networking.
- init_leader_node: pass SPX_EXTERNAL_POOL as --external-mode=pool flags
- e2e.yml: set SPX_EXTERNAL_POOL=192.168.1.220-229 for pseudo test run
After enabling tap networking (public subnet), QEMU no longer uses
hostfwd bindings. The pseudo multi-node test was still grepping for
hostfwd to detect instance distribution and SSH endpoints.
- check_instance_distribution: count nbdkit root-volume processes per
node IP instead of hostfwd-bound QEMU processes
- get_instance_node: match QEMU nbd socket → nbdkit host= IP to
determine which simulated node hosts each instance
- count_instances_per_node: use get_instance_node instead of hostfwd
- SSH connectivity: check PublicIpAddress first (port 22), fall back
to hostfwd — same pattern as single-node and multi-node E2E
admin init builds a NetworkConfig from external networking flags and
passes it to the formation server. When joining nodes poll /formation/status,
they receive the cluster-wide [network] section (external mode, pools,
gateway, DNS, bootstrap VPC IDs). Each joining node auto-detects its
own external_interface and wan_bridge locally.
Fixes multi-node clusters having no [network] section in spinifex.toml
on secondary nodes, which caused no public IP assignment and broke
guest SSH in E2E tests.
The dnat_and_snat rule is removed asynchronously via NATS after instance
termination. A 2s sleep was insufficient. Now polls for up to 30s and
downgrades to WARN if the rule persists (non-fatal — cleanup happens
when the VPC is deleted).
Same fix as single-node: detect PublicIpAddress from describe-instances,
SSH via public IP on port 22. Falls back to QEMU hostfwd on the hosting
node when no public IP is available.
Ben Duncan