feat: routed-NAT external mode with two-tier ingress (#475)

* feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper

* feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode

* feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules)

* feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode

* feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks

* feat(scripts): add --nat-uplink routed mode to setup-ovn.sh

* test(e2e): add routed-NAT single-node natuplink suite

Covers the external_mode=nat lane end to end on a nat-initialized node:
host transit wiring (veth, br-ext port, ip_forward, iptables egress
rules), spinifex.toml contents, disabled EIP surface (empty
describe-addresses, UnsupportedOperation on allocate), default subnet
MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit
/24, guest outbound egress proven via a serial-console probe (nat mode
has no inbound path), and unique transit gateway IPs across VPCs.

Suite self-gates on external_mode=nat; registered as e2e-natuplink and
in SUITES_BUILD so it compiles in CI. Manifest entry stays commented
until CI can provision a nat-mode environment.

* feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column

Adds the exempted_ext_ips column (optional strong Address_Set ref) to the
nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an
in-place SetNATExemptedExtIPs update on the client interface, live client,
and mock. Test NB schema gains the column and covers insert-resolves-UUID,
converge, clear, and missing-NAT paths.

* feat(policy): exempt address set on routed SNAT/EIP rows

Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the
spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW
snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows
un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR)
with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip
path patch the ref in place on pre-existing rows. The set is a singleton
strong ref and is never deleted. Non-routed modes are unchanged.

* feat(host): VPC ingress route primitives for routed-NAT mode

EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that
sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the
spx-nat veth. Replace/del semantics keep both idempotent; a missing route
on removal is not an error.

* feat(external): routed ingress leg on IGW attach/detach, reconcile-safe

AttachIGW's routed leg now also installs the host VPC ingress route via
RoutedIngressHooks, and re-runs on the already-attached path so the
reconcile loop restores host routes (and patches legacy SNAT exempt refs)
after a reboot — OVN state survives restarts but host routes do not.
DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and
the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs,
validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct
pass nil hooks and are unchanged.

* test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern

New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR
via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row
pointing at spinifex_nat_exempt (containing the transit /24), and — after
opening the SG from the transit net (default-closed, AWS parity) — ping and
an sshd TCP handshake against the instance private IP from the host, which
proves the un-SNATted return path. Phase 8 additionally waits for the
second VPC's ingress route to appear on IGW attach. vpc-networking README
documents the jumpbox pattern, SG note, and nat_exempt_cidrs.

* feat(config,admin): public wan pool alongside nat-transit

Restructure admin config generation from single-pool scalars to a Pools
slice so nat mode can emit both the transit pool and an optional public
wan pool. nat mode now accepts --external-pool/--external-source (plus
--external-bind-bridge/--gateway-ip when a public pool is requested),
validated by the shared resolvePublicPoolFlags helper also used by pool
mode. Transit pool always renders first so the IGW allocator picks it.
Formation wire format unchanged (nat mode is single-node only).

* feat(daemon): enable EIP service and auto-assign in nat mode with public pool

External IPAM (and with it the real EIP service, RunInstances public IP
auto-assign, and MapPublicIpOnLaunch on default subnets) now activates
whenever public pools exist: pool mode as before, plus nat mode when a
wan pool rides alongside the transit segment. The transit pool is
filtered out of IPAM — its addresses are gateway-LRP plumbing, never
allocatable EIPs. Transit-only nat configs keep the disabled EIP stub.

* feat(host): per-EIP ingress primitives for routed-NAT mode

DetectUplinkFor parses ip route get to find the interface and source IP
facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move).
EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC
gateway LRP transit IP (src-hinted so host-local traffic still DNATs),
answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC,
WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for
DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated.

* feat(policy,vpcd): host EIP binder option on NATManager

WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds
after priming on both the fresh-add and idempotent-skip paths (host
routes and proxy-ARP are volatile even when the OVN row survives, so
reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort.
The next hop comes from the new gatewayPortIP helper reading the gateway
LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID
key now an exported policy const shared with the external package.

vpcd selects the IGW pool by name instead of index — with two pools in
nat mode the gateway allocator must stay on the transit pool — and wires
the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public
pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced
pools (gateway unknown until lease) proxy-ARP on their bind bridge.

* feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection

dhcp_mac = "derived" (default) | "interface" on external pools. The
interface variant leases with the bind interface's own MAC — WiFi/WWAN
uplinks drop frames with foreign source MACs, so derived chaddr MACs
never complete DORA there. Option 61 stays the lease discriminator.
UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and
persists through KV so restart re-acquires repeat it.

Routers that key leases on MAC and ignore client-id would hand every
lease the same IP; the manager detects an ACK matching the interface's
own address or another client-id's live lease, releases the colliding
lease, and hard-errors advising a static range instead.

admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp
pools bound to non-bridgeable uplinks (wlan/wwan/ppp).

* test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs

The natuplink suite now detects a public (non-transit) pool in spinifex.toml
and switches lanes: Tier-1-only configs keep the EIP-disabled and
no-public-IP assertions; public-pool configs assert the EIP surface is
enabled, the default subnet has MapPublicIpOnLaunch=true, instances
auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress
lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink,
per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the
EIP, reconcile replay across a vpcd restart, and teardown on disassociate.

Docs: vpc-networking README gains a public-pool section for routed-NAT nodes
(init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated
mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool.

* fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd

Routed-NAT mode shells out to iptables for the transit masquerade and
per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for
host-delivered EIPs. Without these sudoers entries vpcd crash-loops with
"command not allowed" on nat-mode nodes.

* fix(ovn): scope DeleteStaticRoute prefix match to the target router

Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache
match could grab another router's row and fail with a referential
integrity violation, wedging IGW attach on multi-VPC nodes. Match only
rows referenced by the target router, mirroring FindStaticRoute. Mock
updated to the same semantics, with a regression test covering drift
replacement across two routers.

* fix(reconcile): always replay AttachIGW so host ingress state is re-ensured

applyIGWs skipped AttachIGW whenever the gateway switch port existed in
actual state, so the already-attached re-ensure path never ran and
routed-NAT host ingress routes lost on reboot were never reinstalled.
AttachIGW is idempotent; run it unconditionally.

* fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins

Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes
collide on the single host route table and the last IGW attach silently
stole the route (route replace). Pass vpcID and the gateway LRP IP
through RoutedIngressHooks; the bootstrap VPC always installs its route,
other VPCs only install when the CIDR is free and only remove a route
their own gateway IP holds. Adds host.VPCIngressRouteVia to read the
current route holder.

* ci(e2e): add nightly cell 19 for routed-NAT single-node suite

Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.

* fix(admin): nat-mode init summary reflects configured public pool

The summary printed 'outbound-only — no public IPs/EIPs' even when
--external-pool/--external-source configured a public wan pool. Print
the pool range (or dhcp bind) and note EIPs are enabled.

* fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites

Reconcile the routed-NAT branch with dev's ctx-first service APIs after
rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService
interface, the vm_adapters public-IP resolver asserts the ctx-signed
AssociatedPublicIPForInstance, and the default-subnet mapping test threads
ctx into DescribeSubnets.

docs/COMMANDS.md

@@ -54,7 +54,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Flags | Description |
 |---------|-------|-------------|
-| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--token-ttl`, `--force` | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` |
+| `spx admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; add `--external-pool start-end --external-gateway <ip>` or `--external-source=dhcp [--external-bind-bridge <iface>]` for a public pool with full EIP support), `--no-external` | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` |
 | `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services` | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations |
 
 ### Version
@@ -385,7 +385,7 @@ termination. Standalone attach/detach API is internal-only.
 
 ### EC2 — Elastic IP
 
-EIP commands are only registered when an external IPAM pool is configured.
+EIP handlers are always registered. Without a public IPAM pool (external mode disabled, or `nat` without a public pool), `describe-addresses` returns an empty list and mutating commands return `UnsupportedOperation`. In `nat` mode a public pool (`--external-pool` or `--external-source=dhcp` at init) enables the full EIP surface with host-delivered ingress.
 
 | Command | Implemented Flags | Missing Flags | Status |
 |---------|-------------------|---------------|--------|

docs/compute/vpc-networking/README.md

@@ -191,11 +191,12 @@ is identical:
 Both support the same AWS features: public subnets, Elastic IPs, security groups,
 DescribeInstances showing public IPs.
 
-## `nat` — Outbound Only (Simple)
+## `nat` — Shared SNAT (Simple)
 
-All VMs share a single external IP for outbound SNAT. No public IPs, no Elastic
-IPs, no inbound from WAN. All subnets behave as private subnets with internet
-access.
+All VMs share a single external IP for outbound SNAT. By default there are no
+public IPs, no Elastic IPs, and no inbound from WAN — all subnets behave as
+private subnets with internet access. On routed-NAT nodes, adding a public pool
+restores full public IP parity (see below).
 
 The `gateway_ip` is the IP that OVN uses for SNAT. You can set it statically or
 use `setup-ovn.sh --dhcp` to obtain one from the router. This is the router's
@@ -215,6 +216,90 @@ gateway_ip = "192.168.1.100"     # Single IP for all VM outbound SNAT
 prefix_len = 24
 ```
 
+### Host access to instances (jumpbox pattern)
+
+The spinifex host automatically reaches every instance's **private IP**: IGW
+attach installs a host route into OVN (`<vpc-cidr> via <gateway-transit-ip>
+dev spx-nat-host`) and exempts the transit net from SNAT, so replies to
+host-initiated connections come back un-NATted. No per-instance setup.
+
+Security groups still apply and the default SG is closed to the host, same as
+AWS — open SSH/ICMP from the transit net first:
+
+```bash
+aws ec2 authorize-security-group-ingress --group-id $SG \
+  --protocol tcp --port 22 --cidr 100.127.0.0/24
+```
+
+Then use the host as a jumpbox for remote access:
+
+```bash
+ssh -J admin@<spinifex-host> ubuntu@<instance-private-ip>
+```
+
+Extra networks that must reach instances without SNAT (e.g. a management LAN)
+can be added via `[network] nat_exempt_cidrs = ["192.168.50.0/24"]`.
+
+### Public IPs in NAT mode (public pool)
+
+A routed-NAT node (`setup-ovn.sh --nat-uplink` + `spx admin init
+--external-mode=nat`) can carry a public pool alongside the internal
+`nat-transit` pool. With one configured, nat mode behaves like pool mode for
+public IPs: `MapPublicIpOnLaunch` on the default subnet, auto-assigned public
+IPs, and Elastic IPs all work. Spinifex delivers each public IP at the host —
+a `/32` route steers it into OVN and a proxy-ARP neighbor entry answers for it
+on the uplink (L3 only, same MAC, so it works on WiFi and other non-bridgeable
+uplinks).
+
+Static range carved out of the router's DHCP scope:
+
+```bash
+spx admin init --external-mode=nat \
+  --external-pool 192.168.1.150-192.168.1.250 \
+  --external-gateway 192.168.1.1
+```
+
+Or lease public IPs from the upstream router's DHCP:
+
+```bash
+spx admin init --external-mode=nat --external-source=dhcp \
+  --external-bind-bridge wlan0
+```
+
+On WiFi/WWAN uplinks the leases are requested with the interface's own MAC
+(`dhcp_mac = "interface"`, written automatically) and distinguished by DHCP
+client-id. Some routers key leases by MAC and ignore the client-id — Spinifex
+detects this (the router hands the same IP to two client-ids) and fails the
+allocation with advice to switch to a static range.
+
+Resulting config:
+
+```toml
+[network]
+external_mode = "nat"
+bridge_mode   = "nat"
+
+[[network.external_pools]]
+name       = "nat-transit"        # internal transit net (auto-generated)
+gateway    = "100.127.0.1"
+prefix_len = 24
+
+[[network.external_pools]]
+name        = "wan"               # public pool
+range_start = "192.168.1.150"
+range_end   = "192.168.1.250"
+gateway     = "192.168.1.1"
+prefix_len  = 24
+```
+
+**Caveat — reaching an EIP from the spinifex host itself.** Host-sourced
+traffic enters OVN from the transit net, which is exempt from NAT (that is what
+makes the jumpbox pattern work) — so a host connection to an EIP that carries
+the transit source IP would skip DNAT. Spinifex stamps the EIP route with the
+uplink's LAN IP as source to avoid this, but if no uplink address can be
+determined, connect to the instance's **private IP** from the host instead.
+Other machines on the LAN are unaffected.
+
 ## Disabled (Empty/Omitted)
 
 VPC networking is overlay-only. No external connectivity. Instances can only
@@ -222,19 +307,21 @@ communicate within their VPC.
 
 ## Mode Comparison
 
-| Capability                        | `pool` (static) | `pool` (dhcp) | `nat`    | Disabled |
-| --------------------------------- | --------------- | ------------- | -------- | -------- |
-| Outbound internet                 | Yes             | Yes           | Yes      | No       |
-| Inbound from WAN                  | Yes (1:1 NAT)   | Yes (1:1 NAT) | No       | No       |
-| Public subnets                    | Yes             | Yes           | No       | No       |
-| Auto-assign public IPs            | Yes             | Yes           | No       | No       |
-| Elastic IPs                       | Yes             | Yes           | No       | No       |
-| DescribeInstances shows public IP | Yes             | Yes           | No       | No       |
-| Admin must reserve IP range       | Yes             | No            | No       | No       |
-| Needs router DHCP                 | No              | Yes           | Optional | No       |
+| Capability                        | `pool` (static) | `pool` (dhcp) | `nat`             | Disabled |
+| --------------------------------- | --------------- | ------------- | ----------------- | -------- |
+| Outbound internet                 | Yes             | Yes           | Yes               | No       |
+| Host reaches instance private IPs | No              | No            | Yes (routed)      | No       |
+| Inbound from WAN                  | Yes (1:1 NAT)   | Yes (1:1 NAT) | With public pool  | No       |
+| Public subnets                    | Yes             | Yes           | With public pool  | No       |
+| Auto-assign public IPs            | Yes             | Yes           | With public pool  | No       |
+| Elastic IPs                       | Yes             | Yes           | With public pool  | No       |
+| DescribeInstances shows public IP | Yes             | Yes           | With public pool  | No       |
+| Admin must reserve IP range       | Yes             | No            | Only static pool  | No       |
+| Needs router DHCP                 | No              | Yes           | Optional          | No       |
 
-If you start with `nat` and later need public subnets, switch to `pool` and
-define a range (or use `source = "dhcp"`) — no data migration needed.
+If you start with `nat` and later need public subnets: on a bridgeable uplink
+switch to `pool` and define a range (or use `source = "dhcp"`); on a routed-NAT
+node just add a public pool alongside `nat-transit` — no data migration needed.
 
 ## Bridge Setup — Physical Network Wiring
 

docs/service-interfaces.yaml

@@ -332,6 +332,12 @@ suites:
   # e2e-bm:
   #   path: spinifex/tests/e2e/baremetal
   #   covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+  #
+  # e2e-natuplink activates once CI can provision an external_mode=nat
+  # node (setup-ovn.sh --nat-uplink + spx admin init --external-mode=nat).
+  # e2e-natuplink:
+  #   path: spinifex/tests/e2e/natuplink
+  #   covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
 
 
 # Special-case rules consumed by the selector (Bead 4). Hard-coded in