Ben Duncan
828b5df4aa
* fix: fail DRAIN when a volume seal failed
shutdownAndUnmount logged the volume unmount error and returned nothing, so
StopAll returned nil and the DRAIN phase ACKed success with vms_stopped set.
The coordinator then advanced to STORAGE and stopped viperblock while the seal
was still in flight, tearing the VBState envelope and leaving the guest unable
to mount its root volume on the next boot. The deploy reported success.
shutdownAndUnmount now returns ErrVolumeSealFailed when the unmount did not
seal; QMP, PID-file, fw_cfg and telemetry cleanup stay best-effort. stopCleanup
and stopOne carry it without aborting the rest of the teardown, since QEMU is
already down by then, and StopAll aggregates the per-VM failures so
handleShutdownDrain ACKs with Error set and never reports vms_stopped.
volumeMounterAdapter.Unmount was the second swallow site: it logged each failed
seal and returned nil unconditionally. It now attempts every volume and returns
the aggregate, which is what the VolumeMounter contract already described.
Terminate still tolerates a failed seal — the volumes are deleted next, so an
unsealed block map loses nothing a caller can ask for.
This does not fix the seal itself. It converts silent data loss into a refused
upgrade that an operator can act on.
* test: add the storagefault e2e suite for backend-outage guest survival
Freezes predastore with SIGSTOP to hold connections open while answering
nothing, which is the fault a timeout-free path cannot distinguish from a
slow peer. On one node that is a total outage; on three it freezes every
node but the guest's own, breaking the RS(2,1) shard floor.
The suite writes and flushes a verifiable region before the fault and
churns a separate unverified region during it, so an interrupted write is
never mistaken for lost data. The root-filesystem load runs from the
user's home rather than /tmp, which is tmpfs on this image and would move
no bytes on the root device.
* test: verify the root device is under write pressure, and size the outage
The previous guard matched /^ *vda / against /proc/diskstats, where the
device name is the third field, and read $7 which is ms_reading rather
than sectors_written. It printed empty and asserted nothing, so a green
run could not distinguish a surviving guest from an idle disk. It now
samples sectors_written twice and fails unless it advances.
The outage is sized from viperblock's absorption rather than guessed:
a workload must fill DefaultMaxPendingBytes before backpressure engages
at all, so a short freeze measures the retry budget rather than the fault.
* test: add nbdkit fault injection to the storage-fault e2e suite
The predastore freeze takes the whole storage cluster away from every
guest on it, which is too broad to run beside anything else. Removing a
single volume's NBD server is the same fault at one-volume scope: one
host, one guest, nothing cluster-wide stopped.
Both signals are here because they sit on opposite sides of the defect.
SIGSTOP holds the socket open and answers nothing, so the guest blocks
and no error reaches ext4. SIGKILL closes it, and QEMU's nbd driver at
the default reconnect-delay of 0 fails requests at once, which under
werror=report aborts the journal.
Signalling is guarded: a pid is matched on comm, our plugin and the
volume argument together, and re-confirmed immediately before the
signal, because these hosts run nbdkit for other volumes and pids are
reused.
* fix: pause a guest on a backend I/O error instead of handing it EIO
Every virtio-blk device was created with werror=report,rerror=report, so a
backend outage was delivered to the guest as EIO. ext4 aborts its journal on
that and remounts read-only, and the write that failed is simply gone. Under
werror=stop QEMU pauses the VM and holds the failed request, so a backend that
comes back resumes the guest with nothing lost.
The boot drive, the cold-boot data volume and the hot-plugged data volume all
move to stop. Alongside it the NBD transport gets reconnect-delay=30, which
pauses requests rather than failing them while a storage process restarts on
the same socket; on the legacy -drive form it has to be file.reconnect-delay,
because the bare name reaches the raw format driver, which has no such option.
The QMP heartbeat now recognises run state io-error and issues cont. After ten
attempts the instance is reported impaired and the cadence drops to one attempt
per ten heartbeats, but it never stops: QEMU holds the request for as long as
the guest exists, so a backend that returns an hour later can still be resumed.
The storagefault suite gains the fault this was written for — kill the volume's
own nbdkit, hold, then put it back on the same socket — and it now verifies a
sha256 region across the outage rather than trusting the guest to be reachable.
Three harness defects went with it: the socket-liveness poll ran unprivileged
against a 0770 directory and could never succeed, the relaunched socket was
left 0755 where QEMU needs group write, and the two variants shared one guest
so a fault in the first was diagnosed in the second.
* test: cover the io-error resume policy
* test: stop two parallel tests racing on the terminate backoff variable
* fix: report a guest paused on a backend I/O error as impaired
A guest held by werror=stop kept reporting ok on both status checks. QMP stays
responsive while the guest is paused -- cont is being sent and answered -- so
the QMP failure gate that DescribeInstanceStatus keys on never trips, and the
one state a customer most needs to see looked healthy.
Instance status now reports impaired/failed while the resume budget is
non-zero, timestamped from IOErrorSince so impairedSince is when the guest
paused rather than when it was escalated. System status reports impaired too:
a backend I/O error is an infrastructure fault, not a guest one, and that
distinction is what the two checks exist to draw. The counter is cleared by
the first poll that sees the guest running, so both clear on recovery.
* test: reap the relaunched nbdkit when the fault test ends
relaunchNbdkit detaches the server with setsid so it outlives the SSH session
that started it, and viperblockd never learned it exists. Nothing reaped it, so
every run left an nbdkit holding a socket and a cache for a volume that was
later detached or deleted -- four were found running on dev-prod against two
detached and two deleted volumes.
The PID is resolved from the socket path rather than the volume id, so a
legitimate viperblockd server for the same volume is never the one killed.
* test: prove the backend is serving before a fault is called cleared
Thawing the process a fault stopped is not the same as restoring the service.
The outage test treated SIGCONT as the end of the fault and went straight to
judging the guest, so it passed while the cluster it ran on stayed unable to
serve a single object for the next 35 minutes, and the next test would have
inherited that.
thawPredastore now puts and gets an object through every gate, with a budget,
and fails the test if the backend does not come back.
* fix: do not hand back a mount entry whose export is gone
A volume whose unmount failed kept its entry in MountedVolumes while its nbdkit
was gone. Every later ebs.mount then matched that entry and returned its socket
path, so vmMgr waited for a socket nothing would ever create and the instance
could not be started again — on dev-prod a guest stopped against a frozen
backend was left permanently stopped, four start attempts failing identically
on a socket the node had already removed.
Drop the entry when there is positive evidence the export is dead: the socket
is absent, or the recorded pid is not running. Neither can be true of a live
export, so the double-writer hazard the guard exists to prevent is unaffected.
Also make the durability test's pattern helpers report failures. The read dd sat
in a pipeline, so its exit status was discarded and the harness hashed nothing
and returned the empty-string sha256 as a checksum.
* test: report the guest console when a restart never answers SSH
A restart that reaches running but never comes back is indistinguishable from
a dead root filesystem without the serial console, and the log is gone once
the fixture terminates the instance.
* fix: converge guest port datapaths concurrently, not one deadline after another
Each guest port that misses its SB binding burns the full convergence deadline,
and applyEIPs spent them in series. Three ports belonging to long-stopped
instances therefore held a newly launched guest's EIP dark for the sum of them
before its own probe started.
AddEIP stays serial since it mutates NB. Only the waits fan out, bounded, and
sb-cluster-state-reset is now serialised with a minimum interval so concurrent
probes cannot each fire a cluster-wide resync.
* fix: refuse a stale cross-node start after the owning node is gone
The volume lease already refused a cross-node mount, but it is a liveness
claim: 45s TTL, renewed every 15s. A node that dies holding un-uploaded
writes stops renewing, and 45 seconds later nothing refuses a start elsewhere
that loads a stale backend checkpoint.
A failed seal now writes a durable marker naming the node holding the only
current copy, cleared on a seal that reaches the backend and on delete. The
lease covers a live holder, the marker a dead one.
The refusal also reaches the caller instead of being flattened to
ServerInternal, which told an operator to retry something that cannot succeed
while the holder is down.
* fix: keep the daemon's message when a start is refused
Rebuilding the error from the code alone rendered the generic ErrorLookup
text, so a refusal naming the node holding the volume reached the caller as
boilerplate about attaching volumes.
* fix(ec2): terminate detaches every DeleteOnTermination=false volume
TerminateStoppedInstance skipped the detach for a non-Boot DoT=false volume
on the assumption that Stop's Unmount had already cleared its attachment.
Unmount only clears it when the seal succeeded, so a volume whose seal failed
stayed in-use pointing at an instance that terminate then deleted, leaving it
unreachable with no owner to release it.
The running-instance path already detaches every DoT=false volume; the stopped
path now matches it, and tolerates an already-gone volume the same way.
DeleteVolume also keeps its provider error on the way out. The terminate reaper
retries the delete every two minutes and logged a bare ServerInternal, which
made a blob node being down indistinguishable from a transient fault.
* fix(ec2): DetachVolume --force releases a volume whose instance is gone
The ordinary detach routes to ec2.cmd.{instanceID} for the guest unplug, so a
volume left attached by a part-way teardown could never be detached: the
instance no longer runs anywhere and nothing answers. That also made it
undeletable, since DeleteVolume refuses an attached volume.
ForceDetachVolume already existed for this deadlock but was reachable only from
account teardown. Force now falls back to it when the instance has no responder,
which is the one case where clearing the attachment cannot strand a live guest.
* fix(viperblock): unsealed writes steer placement instead of pinning a volume
The dirty marker refused any mount from a node other than the one holding
unsealed writes. Instance start already forwards to the node that last ran the
instance and falls back after a 30s window, so that refusal turned a working
fallback into a volume that could run nowhere once its owner was gone. It
traded a rare data-loss risk for a permanent availability failure.
The marker is now a record and a warning, not a gate. A takeover logs the
previous holder and why it could not seal, then adopts the marker so the old
node is no longer treated as the better source. Live exclusion is unchanged and
still enforced by the lease.
It is also written on open rather than on seal failure. A node killed mid-write
never reaches its seal, so the old trigger left no trace in exactly the case the
marker exists to describe.
Adds 'spx admin volumes unsealed' to see which volumes would cost something if
started elsewhere.
* fix(viperblock): fence a volume whose lease moved instead of logging it
The lease was checked once, at mount, and never again. A node that lost it
kept its export up and kept writing into a volume another node owned, so
both copies advanced and the backend settled it by arrival order.
Losing a renewal now decides: another holder means tear the export down,
no holder means the entry merely aged out, so reclaim it and leave the
guest running. Fencing never seals — this node's copy is the stale one —
and leaves the dirty marker for the winner to take over.
This is not an epoch fence. It narrows the window from the 45s TTL to one
renewal interval at no write-path cost; the backend still cannot refuse a
stale writer.
* fix(network): advertise EIPs so a recycled address is reachable
The gateway switch port only asked ovn-northd to publish the router's NAT
addresses in centralised and routed modes, so on a distributed cluster nothing
ever announced an elastic IP. OVN still answers ARP for it, which is why a
first-time allocation works and hid this — but no gratuitous ARP is sent when
the address moves to a new ENI, so every host on the segment keeps resolving it
to the previous holder's MAC until its cache ages out. Reusing an address from
the pool then lands on a dead MAC and the instance is unreachable from outside.
Distributed mode needs the option as much as the others: northd scopes each
entry with is_chassis_resident(<vif port>), so the announcement leaves from the
chassis running the instance rather than the gateway, which is what distributed
NAT wants.
Converge on the already-attached path too. AttachIGW returns early once the
gateway port exists and nothing else rewrites it, so without that an existing
VPC would stay silent for its whole life.
Confirmed on dev-prod: setting the option moved three EIPs from unadvertised to
carrying their correct external_mac, each scoped to its own ENI's port.
* fix(viperblock): make the volume fence and its dirty marker ownership-safe
Four corrections to the lease-loss fence, all found in review.
Drop the reacquire path. Losing a lease and finding no successor was treated as
a lapsed entry worth reclaiming, but leases.acquire returns the cached entry —
already flagged lost, with a dead renewal goroutine — so the "reclaimed" lease
renewed nothing and the node went on writing a volume it could not prove it
owned. The two errors are not symmetric: a false fence costs one guest a
restart, a false reclaim leaves two nodes writing one volume. Fence in both
cases, and report which happened through the outcome attribute.
Surrender a lease this node cannot confirm. Renewal only noticed a loss when
the server rejected an update, so a node partitioned from NATS kept its export
up while the TTL expired and another node took the volume. Track the last
confirmed renewal and give the volume up once that ages past the point where
the server may admit another owner. Bound the renewal call too, so a hung
JetStream request cannot hold the goroutine past that point.
Do not act on a kill that did not take. Both the fence and ebs.unmount removed
the mount entry, released the lease and announced the result before knowing
nbdkit had exited — the fence then reported a writer stopped that was still
running, and the unmount sealed into a directory a live process was writing.
Wait for the exit, and on failure leave the volume mounted and say so. An
already-exited process now counts as success, which is what the caller asked.
Give the dirty marker a generation. It recorded an owner and nothing to order
two nodes' claims, so a node returning after a takeover could clear or overwrite
the winner's marker and point placement back at the stale copy. Stamp the lease
generation, refuse a write from an older one, and only clear a marker this node
still owns. A marker that cannot be written now fails the open rather than
leaving writes nothing records.
* test(viperblock): cover the stall surrender and the unmount kill guard
The two ownership cases the fence work left untested.
A holder partitioned from NATS while the winner keeps it never gets an update
rejected, so revision checking cannot see it — it has to give the volume up on
its own clock. Pins that validity sits below the server TTL with room for a
check in between, that a lapsed holder reports the loss as a stall rather than
as a takeover, and that releasing a surrendered lease leaves the entry for
whoever holds it now.
The unmount side of the kill guard: sealing under a live nbdkit writes a state
torn between two writers, so a kill that did not take has to fail the unmount
and leave the entry mounted for a retry.
* fix(e2e): keep the guest private key for as long as the key pair
EnsureKeyPair computed the PEM path from the caller's directory but wrote
the file inside the ensureOnce closure, which runs only on the first call.
Callers passed a per-test artifact dir, and ArtifactDir removes that dir
when the test passes — so the first test's success deleted the only copy
of the private key while the key pair stayed registered, and every later
call returned a path nothing had ever been written to.
Guests after the first were then launched against a public key whose
private half no longer existed, so the harness could never authenticate
and gave up after the full five-minute budget. The guests were healthy
and reachable on tcp/22 throughout, which is why this read as a storage
or network fault for so long.
The directory parameter was the defect, so it is gone: the PEM now lives
in a directory the fixture owns and removes with the key pair, and no
caller can supply one that is shorter-lived.
Also reorder the killed-nbdkit teardown. The reaper registered its own
cleanup after the detach registered one, so LIFO killed the block server
before the platform detached the disk and DetachVolume then failed with
"Node ... is in use". The detach cleanup now reaps once it is done.
* fix: stop leaking an nbdkit export when recovery cannot adopt it
Restart recovery adopts a guest's boot volume, then routinely fails the
-efi volume with "claim lease: context deadline exceeded" -- JetStream is
still catching up at daemon start and the KV claim runs on the client's
5s default. Recovery gave up on the first error and never retried, so the
export stayed live with no registry entry. At terminate the unmount found
nothing to seal, returned NotFound and killed nothing, and the process
outlived its guest holding the volume and socket open. EFI is always the
second volume adopted, which is why it was always the one that leaked.
Two changes, because either alone leaves a hole:
- rebuildMountedVolume retries the build (4 attempts, 3s apart) so a
JetStream that is merely slow no longer orphans an export for the life
of the process.
- unmountVolume reaps a corroborated export for the volume it was asked
to unmount when the registry has no entry for it, so an export orphaned
by any means is still cleaned up. The reap goes through
reapOrphanedNbdkit, which declines anything a live process references:
at unmount the guest's blockdev may not be torn down yet, and killing
the server underneath it strands QEMU.
* test: hold the backend-serving probe and assert predastore readiness
The backend-outage test declared the backend restored on one successful
round trip through each gate. A gate can answer the request that arrives
just after a thaw and then lose its blob peers under the load that
follows, which is a backend that never recovered reported as one that did
-- and that is what happened: the probe reported all three gates serving
minutes before the guest was still taking BLOCK_IO_ERROR on every resume.
The round trip has to hold across three rounds now, and the count restarts
rather than decays when one fails, so a gate that serves and then stops is
the failure the hold is looking for rather than noise it averages out.
It also checks /readyz on each host and reports any failed check by name.
The round trip only proves this workstation can write through a gate; it
says nothing about which blob peers that gate can reach, and a gate one
node short of the write floor serves a small object and fails a guest. A
host that serves no admin listener is logged and skipped, since the port
is optional configuration.
* fix: scope predastore's admin port to the cluster plane in the host policy
predastore's admin listener binds the host's cluster bind_addr and serves
/healthz and /readyz unauthenticated. Readiness now names which blob peers
a gate cannot reach, which is exactly the kind of topology detail 4432 is
peer-scoped for, so 8660 joins it in the peer-scoped rule rather than the
public one.
* fix: keep resuming a paused guest at heartbeat rate after a long outage
Past ten attempts the cont dropped to one per ten heartbeats, so a guest
whose backend had come back could sit paused for another five minutes
before anything retried it. Measured on dev-prod: the backend was proved
serving through all three gates while the guest was still waiting on
attempt 60, five minutes after attempt 50.
The slowdown was there to avoid paying a failed I/O every thirty seconds
on a backend still down. That is nearly free, and five minutes of downtime
after recovery is not, so only the log line is throttled now.
* test: fail the run when a key pair name resolves to more than one pair
A create that lands twice leaves two key pairs with one name and different
material. RunInstances resolves the name to one of them while the harness
holds the private key of the other, and the only symptom is every guest in
the run refusing the key at the SSH gate — five minutes of waiting per
guest, four guests, and nothing anywhere naming the cause. Observed on
dev-prod: two pairs created a millisecond apart, both named
e2e-key-30de10ba.
One DescribeKeyPairs after the create says so outright instead.
* test: tolerate an already-released volume in the detach teardown
Teardown runs DetachVolumeWait after a terminate that releases the volume
itself, so the detach can arrive at a volume already on its way to
available. That is the outcome it wanted, and treating the IncorrectState
refusal as fatal failed a backend-outage run whose assertions had all
passed — the guest came back, every acknowledged write verified, and the
test still went red on the way out.
VolumeInUse is still fatal: that one means the volume is attached
somewhere it should not be.
* test: carry the merged suites onto the reworked key-pair and account-scoped seams
The diskperf suite arrived on dev calling EnsureKeyPair with the artifact dir
this branch removed, and DeleteVolume's metadata lookup is account-scoped now,
so the provider-failure test has to seed and delete under the same account.
* refactor: make the fence outcome a type in one const block
Four separate exported string constants with a paragraph of comment each,
passed to RecordVolumeFence as a bare string. Group them, give them a
FenceOutcome type, and take that type at both call sites so an arbitrary
value can no longer reach the metric attribute.
* refactor: move viperblockd's wire contract into a leaf vbwire package
The daemon imported viperblockd for exactly two symbols -- the fenced
volume subject and its event -- which put the storage service back on the
daemon's dependency graph to name something it only listens for.
vbwire holds the contract instead: the subject and event, the dirty
bucket name and record shape, and the read-only listing an operator tool
needs. It imports nothing from spinifex, so both the daemon and the admin
CLI drop their edge to viperblockd entirely. cmd/service.go keeps its
one, which is genuine: it constructs the service.
* docs: drop a specific host name from the start-refusal comment
Named a real dev-prod host in a comment and in test fixtures. Say what
the name is instead.
* ci: give storagefault a dedicated nightly cell
The suite is eligible for both topologies and runs on the PR gate, but no
nightly cell ran it: the permutation cells run a deliberately narrow
subset and none of the dedicated suite cells named it.
Cell 29, single-node, following eks/ecs/rds. It cannot join a permutation
cell -- it SIGSTOPs a cluster-wide service, so every suite sharing that
environment would report the outage as its own failure.
* ci: run storagefault nightly on three nodes, not one
The previous commit put its dedicated cell on a single-node runner. That was
the wrong topology and the wrong justification: single-node is RS(1,0), where
the shard floor cannot be broken and a cross-node takeover after a failed seal
does not exist, so the two assertions worth having a dedicated cell for were
the two it could not make. Cell 29 is now real-multi on ci-multi.
It also corrects the claim that the PR gate already covers the multi-node case.
The suite is eligible for both topologies, but e2e.yml's default suite list
never names it, so it has never run there. A full pass is ~63 minutes on
dev-prod bare metal, almost all of it deliberate waiting -- one test holds
predastore SIGSTOPped for seven minutes -- which is too long to add to a gate
that currently finishes well inside its budget. Nightly is where it belongs.
Consequences: a per-cell timeout_minutes, defaulting to the previous 50, since
90 minutes of suite cannot run in a 50-minute job; the permutation steps gated
on matrix.suite == '' so the dedicated cells do not also run them; and a Run
Multi Suite E2E step that builds NODES_CSV from the tofu inventory and asserts
the suite against E2E_SUITES_MULTI before running it.
* fix: stop a startup KV race leaving vpcd serving keyless guests
vpcd opens five KV buckets while NATS is still coming up, each attempt
carrying a ~5s JetStream API deadline, and applied three different policies to
the same failure: spinifex-vpc-enis was fatal, while instance-state,
security-groups, subnets and vpcs each degraded silently for the lifetime of
the process. A five-second blip during a cold start therefore decided how a
node served metadata until someone restarted it.
On dev-prod all three nodes hit this after a deploy. Two were saved by
accident: their ENI bucket also timed out, so they exited 1 and systemd
restarted them cleanly. The third's ENI bucket opened in time, so it never
crashed and ran for hours with no instance record loader. With the local state
file unreadable as well, every instance-backed IMDS leaf 404'd, cloud-init's
crawl died on ami-id and fell back to DataSourceNone, and every guest launched
on that node booted with no authorized_keys.
kvstore.OpenWithRetry retries for a bounded window before any policy applies,
logs the cause per attempt and the wait on a late success, wraps the last error
with the bucket name, and returns at once on a cancelled context. All five
opens use it. Failing to open the instance record space is now fatal, matching
the ENI bucket: the dependency whose loss stops guests logging in should not be
treated more leniently than the one whose loss stops ENI resolution.
Also fixes the reason the fallback was load bearing at all. The local state
file was written 0600 and vpcd reads it as a different user in the shared
spinifex group, so the local-first path had never once worked and its
mtime+size cache had never fired. It is 0640 now, set with an explicit Chmod so
an inherited umask cannot undo it.
Closes mulga-gtpzz, mulga-dhqxr. Still open: mulga-mm2oc, IMDS serving an
unresolvable instance as "no key pair" with nothing logged.
* fix: scope the ime-* firewall accept to IMDS and VPC DNS only
The per-tap OVS demux delivers any guest packet addressed to
169.254.169.254 or .253 onto an ime-* host port regardless of L4 port,
and iifname "ime-*" accept then admitted all of it. Every wildcard-bound
host service — sshd, the AWS gateway, northstar, the console — answered
guest packets on those addresses; only asymmetric reply routing kept the
handshake from completing, and a guest could still drive northstar to
recurse unthrottled. Scope the accept to IMDS (.254:80) and VPC DNS
(.253:53), letting everything else fall through to policy drop.
* docs: document the instance-to-host plane and how to protect it without the ISO
The metadata/VPC-DNS link-local plane (169.254.169.254/.253) is served by
host services and is invisible to a NIC-level or cloud firewall, since the
traffic is intra-host on the OVS bridge. Explain that the ISO ships the
scoping rule armed, that the binary installer ships the firewall off, and
that a non-ISO install must enable it with setup.sh --firewall=on or
firewall_enabled=true. Include verification from an instance and the host.
* fix: withhold the IMDS meta-data listing when the instance does not resolve
Every omission from the meta-data root is a positive claim about the instance —
public-keys/ absent means "launched without a key pair", and cloud-init believes
it, writes no authorized_keys and boots the guest keyless. So resolving the
instance was best-effort for a listing whose whole meaning depends on it: a
backend error or a miss shortened the listing instead of failing it, and nothing
anywhere recorded that the answer was a guess.
Withhold the listing with a 500 instead. cloud-init retries a 5xx and cannot
retry a 200 that is merely wrong, so this converts a permanent silent failure
into a transient loud one. servePublicKeys already had this policy for the leaf;
the root is what contradicted it.
A profile lookup that fails still serves the listing minus iam/, and now logs.
The key pair is the field a guest cannot recover from losing, while a missing
role surfaces through the credentials leaf, which reports the miss itself.
Also stops the storagefault cross-node test failing on its own access path. It
tolerates the pattern changing across a takeover, but readPattern needs sudo,
and a volume opened from the backend checkpoint may predate cloud-init writing
its sudoers drop-in — indistinguishable from an unreadable device at the dd call
site. It now checks sudo first, and reads cloud-init's recorded datasource to
tell a rolled-back root filesystem from IMDS never having answered, failing
loudly on the latter rather than accepting it as storage loss.
---------
Co-authored-by: brynmailer-mdc <bdm@mulgadc.com>
title, seoTitle, description, category, tags
| title | seoTitle | description | category | tags | |||||
|---|---|---|---|---|---|---|---|---|---|
| VPC Networking | AWS-Compatible VPC Networking with OVN — Spinifex Docs | How Spinifex implements AWS-compatible VPC networking on bare metal with OVN: public and private subnets, security groups, route tables, and Elastic IPs. | Compute |
|
VPC Networking
How Spinifex implements AWS-compatible VPC networking with public and private subnets, security groups, and Elastic IPs using OVN.
Overview
Spinifex provides AWS-compatible VPC networking on bare-metal. Every EC2 instance runs inside an isolated virtual network backed by OVN (Open Virtual Network). Instances can operate in two modes: private (overlay-only, no WAN access) or public (routable from the WAN with a unique public IP).
Instructions
How It Works
Spinifex maps AWS VPC concepts directly to OVN constructs:
| AWS Concept | OVN Construct | What It Does |
|---|---|---|
| VPC | Logical Router | Isolates tenant networks, routes between subnets |
| Subnet | Logical Switch + DHCP | L2 broadcast domain with automatic IP assignment |
| ENI | Logical Switch Port | Per-instance network interface with MAC/IP binding |
| Internet Gateway | External Switch + NAT | Connects VPC router to physical WAN |
| Security Group | Port Group + ACLs | Stateful firewall rules enforced in OVS datapath |
| Elastic IP | dnat_and_snat NAT rule |
Static 1:1 NAT between public and private IP |
Network Path
Cross-host traffic uses Geneve tunnels (UDP 6081) over the management/overlay NIC. Each host runs ovn-controller which programs OpenFlow rules on br-int (the integration bridge where all VM TAP devices connect).
Private vs Public Subnets
A subnet's behavior depends on three things: whether the VPC has an Internet Gateway, whether the subnet's route table has a default route to that IGW, and whether the subnet has MapPublicIpOnLaunch enabled.
Private Subnet (Default)
Instances get a private IP only. They can communicate with other instances in the same VPC (even across subnets and hosts via the overlay). They cannot reach the internet or be reached from the WAN.
Private subnet instances reach the internet only if their route table has a default route to the IGW (shared SNAT, outbound only — they share the gateway IP) or to a NAT gateway. With no default route, egress is dropped. Either way they cannot be reached from the WAN because they have no public IP.
Public Subnet
Instances get both a private IP and a public IP. The public IP is a 1:1 NAT managed by OVN — the instance OS only sees its private IP.
Requirements for a public subnet:
- VPC has an Internet Gateway attached
- A route table associated with the subnet has a
0.0.0.0/0route to the IGW - Subnet has
MapPublicIpOnLaunch = true - External IP pool configured in
spinifex.toml
Spinifex follows AWS route-table semantics: a subnet is only "public" if its effective route table carries a default route to the IGW. A new VPC's main route table has the local route only — Spinifex does not add the IGW route for you. Without it, the subnet's egress is gated with a drop policy, so instances cannot reach the internet (and inbound connections cannot complete because return traffic is dropped) even with a public IP and an attached IGW. Add the route explicitly — either to the main route table, or to a custom route table associated with the subnet (see Quick Start).
Comparison
| Private Subnet | Public Subnet | |
|---|---|---|
| Private IP | Yes | Yes |
| Public IP | No | Auto-assigned from pool |
| Outbound internet | Only with a default route to IGW/NAT GW | Yes (own public IP via SNAT) |
| Inbound from WAN | No | Yes (via 1:1 NAT to public IP) |
| Instance sees public IP? | N/A | No — only sees private IP |
| Elastic IP support | Only if explicitly associated | Yes |
External Connectivity Modes
The [network] section in spinifex.toml controls how VMs reach the outside world. There are three modes, and pool mode has two IP sources (static or DHCP).
pool — Full Public Networking (Recommended)
Each VM in a public subnet gets its own public IP with bidirectional 1:1 NAT. Supports the full AWS feature set: public subnets, auto-assign public IPs, Elastic IPs, and security groups.
Pool mode supports two ways to obtain public IPs:
Static Range (default)
The admin defines a range of routable IPs that Spinifex manages exclusively.
Use when: You have a block of IPs you control — datacenter ISP allocation, homelab range carved out of your router's DHCP scope, enterprise DMZ range.
Requirement: The IP range must NOT be served by any other DHCP server. In a homelab, shrink your router's DHCP scope to exclude the Spinifex range.
[network]
external_mode = "pool"
[[network.external_pools]]
name = "wan"
range_start = "192.168.1.150"
range_end = "192.168.1.250"
gateway = "192.168.1.1" # Router / next-hop IP
prefix_len = 24
dns_servers = ["192.168.1.1", "8.8.8.8"]
DHCP Source
Instead of a static range, public IPs come from the upstream router's DHCP server. When a VM launches, Spinifex requests a DHCP lease from the router on behalf of the VM. When the VM terminates, the lease is released.
The VM itself never talks to the router's DHCP — it only sees its private VPC IP (from OVN's internal DHCP). The host-side DHCP conversation is invisible to the guest.
Use when: You don't control a static IP block but the router's DHCP server has enough leases. Homelabs where you don't want to carve out a range. Environments where IPs are managed centrally by the network team's DHCP.
Requirement: dhclient or dhcpcd-base installed on the host.
[network]
external_mode = "pool"
[[network.external_pools]]
name = "wan"
source = "dhcp" # "static" (default) or "dhcp"
gateway = "192.168.1.1" # Router / next-hop IP
prefix_len = 24
dns_servers = ["192.168.1.1", "8.8.8.8"]
# No range_start/range_end — IPs come from router DHCP
How Pool Mode Works (Both Sources)
Regardless of whether IPs come from a static range or DHCP, the OVN behavior is identical:
Choosing Static vs DHCP
| Static Range | DHCP Source | |
|---|---|---|
| Public IPs from | Admin-defined range_start..range_end |
Router's DHCP server |
| IP predictability | You know the exact range | Router assigns whatever is available |
| Setup effort | Must reserve range, shrink router DHCP scope | Just set source = "dhcp" |
| Dependency | None | Requires dhclient on host, working router DHCP |
| Best for | Datacenters, ISP blocks, production | Homelabs, dev environments, shared networks |
| Capacity | Exact: range_end - range_start IPs |
Limited by router's DHCP pool size |
Both support the same AWS features: public subnets, Elastic IPs, security groups, DescribeInstances showing public IPs.
nat — Shared SNAT (Simple)
All VMs share a single external IP for outbound SNAT. By default there are no public IPs, no Elastic IPs, and no inbound from WAN — all subnets behave as private subnets with internet access. On routed-NAT nodes, adding a public pool restores full public IP parity (see below).
Limitation (routed-NAT v1): System instances (ECS/EKS/load-balancer agents) source egress from ExternalIPAM pool IPs, which do not exist in
external_mode=nat. Those features requireexternal_mode=pool. A v2 will either allocate transit IPs for system instances or reject the feature at the API level in nat mode.
The gateway_ip is the IP that OVN uses for SNAT. You can set it statically or use setup-ovn.sh --dhcp to obtain one from the router. This is the router's DHCP — not Spinifex's internal OVN DHCP for VMs.
Use when: VMs only need outbound access (apt update, pulling images). Edge deployments behind ISP NAT. Single WAN IP available.
[network]
external_mode = "nat"
[[network.external_pools]]
name = "wan"
gateway = "192.168.1.1"
gateway_ip = "192.168.1.100" # Single IP for all VM outbound SNAT
prefix_len = 24
Host access to instances (jumpbox pattern)
The spinifex host automatically reaches every instance's private IP: IGW attach installs a host route into OVN (<vpc-cidr> via <gateway-transit-ip> dev spx-nat-host) and exempts the transit net from SNAT, so replies to host-initiated connections come back un-NATted. No per-instance setup.
Security groups still apply and the default SG is closed to the host, same as AWS — open SSH/ICMP from the transit net first:
aws ec2 authorize-security-group-ingress --group-id $SG \
--protocol tcp --port 22 --cidr 100.127.0.0/24
Then use the host as a jumpbox for remote access:
ssh -J admin@<spinifex-host> ubuntu@<instance-private-ip>
Extra networks that must reach instances without SNAT (e.g. a management LAN) can be added via [network] nat_exempt_cidrs = ["192.168.50.0/24"].
Public IPs in NAT mode (public pool)
A routed-NAT node (setup-ovn.sh --nat-uplink + spx admin init --external-mode=nat) can carry a public pool alongside the internal nat-transit pool. With one configured, nat mode behaves like pool mode for public IPs: MapPublicIpOnLaunch on the default subnet, auto-assigned public IPs, and Elastic IPs all work. Spinifex delivers each public IP at the host — a /32 route steers it into OVN and a proxy-ARP neighbor entry answers for it on the uplink (L3 only, same MAC, so it works on WiFi and other non-bridgeable uplinks).
Static range carved out of the router's DHCP scope:
spx admin init --external-mode=nat \
--external-pool 192.168.1.150-192.168.1.250 \
--external-gateway 192.168.1.1
Or lease public IPs from the upstream router's DHCP:
spx admin init --external-mode=nat --external-source=dhcp \
--external-bind-bridge wlan0
On WiFi/WWAN uplinks the leases are requested with the interface's own MAC (dhcp_mac = "interface", written automatically) and distinguished by DHCP client-id. Some routers key leases by MAC and ignore the client-id — Spinifex detects this (the router hands the same IP to two client-ids) and fails the allocation with advice to switch to a static range.
Resulting config:
[network]
external_mode = "nat"
bridge_mode = "nat"
[[network.external_pools]]
name = "nat-transit" # internal transit net (auto-generated)
gateway = "100.127.0.1"
prefix_len = 24
[[network.external_pools]]
name = "wan" # public pool
range_start = "192.168.1.150"
range_end = "192.168.1.250"
gateway = "192.168.1.1"
prefix_len = 24
Caveat — reaching an EIP from the spinifex host itself. Host-sourced traffic enters OVN from the transit net, which is exempt from NAT (that is what makes the jumpbox pattern work) — so a host connection to an EIP that carries the transit source IP would skip DNAT. Spinifex stamps the EIP route with the uplink's LAN IP as source to avoid this, but if no uplink address can be determined, connect to the instance's private IP from the host instead. Other machines on the LAN are unaffected.
Disabled (Empty/Omitted)
VPC networking is overlay-only. No external connectivity. Instances can only communicate within their VPC.
Mode Comparison
| Capability | pool (static) |
pool (dhcp) |
nat |
Disabled |
|---|---|---|---|---|
| Outbound internet | Yes | Yes | Yes | No |
| Host reaches instance private IPs | No | No | Yes (routed) | No |
| Inbound from WAN | Yes (1:1 NAT) | Yes (1:1 NAT) | With public pool | No |
| Public subnets | Yes | Yes | With public pool | No |
| Auto-assign public IPs | Yes | Yes | With public pool | No |
| Elastic IPs | Yes | Yes | With public pool | No |
| DescribeInstances shows public IP | Yes | Yes | With public pool | No |
| Admin must reserve IP range | Yes | No | Only static pool | No |
| Needs router DHCP | No | Yes | Optional | No |
If you start with nat and later need public subnets: on a bridgeable uplink switch to pool and define a range (or use source = "dhcp"); on a routed-NAT node just add a public pool alongside nat-transit — no data migration needed.
Bridge Setup — Physical Network Wiring
The WAN NIC must be enslaved to a Linux bridge. This is a hard requirement — setup-ovn.sh will not attach a physical NIC directly to OVS, and macvlan is no longer supported. The Linux bridge owns the host IP, default route, and any DHCP lease, so SSH and management traffic stay up while OVS/OVN are configured underneath.
The full datapath chain looks like this:
physical NIC (e.g. `wan`)
└─ enslaved to ─▶ br-wan (Linux bridge — host IP, default route, DHCP)
│
└─ veth pair ─▶ br-ext (OVS bridge — OVN external uplink)
│
└─ localnet ─▶ br-int (OVS integration bridge)
│
└─▶ TAP devices (VM NICs)
setup-ovn.sh auto-detects the Linux bridge that owns the default route (typically br-wan, provisioned by cloud-init / netplan / systemd-networkd). You can override the detection with --wan-bridge=<name>. Once detected, the script creates the OVS bridge br-ext and links it to the WAN bridge with a veth pair. The Linux bridge keeps its IP and routes — no interruption. Bridge-mapping is set to external:br-ext.
If the default route is on a bare physical NIC (no bridge), setup-ovn.sh stops and prints guidance on how to convert the NIC to a bridge before re-running.
Example: Required br-wan State
The host must have something resembling this before setup-ovn.sh is run:
7: br-wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 26:df:3c:de:d0:c2 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.31/23 brd 192.168.1.255 scope global br-wan
valid_lft forever preferred_lft forever
inet6 fe80::24df:3cff:fede:d0c2/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
The physical NIC (e.g. wan, eth0, eno1) is enslaved to br-wan and has no IP of its own — all L3 state lives on the bridge.
Example netplan that produces this:
network:
version: 2
ethernets:
wan:
dhcp4: false
bridges:
br-wan:
interfaces: [wan]
dhcp4: true
Three Bridges, Three Jobs
Every Spinifex node has three bridges in the datapath:
| Bridge | Type | Created By | Purpose | Ports |
|---|---|---|---|---|
br-wan |
Linux bridge | Host (cloud-init / netplan) | Host WAN uplink — owns host IP and default route | Physical WAN NIC, veth peer |
br-ext |
OVS bridge | setup-ovn.sh |
OVN external uplink (localnet) |
veth peer to br-wan |
br-int |
OVS bridge | setup-ovn.sh |
VM overlay traffic (Geneve tunnels) | VM TAP devices, tunnel ports |
br-wan is provisioned by your distro's network configuration (cloud-init, netplan, systemd-networkd, ifupdown). The name is configurable; br-wan is the convention. br-int and br-ext are always created by setup-ovn.sh.
The link between br-ext and the VM datapath is logical, not physical: OVN's localnet port type maps the logical external switch to br-ext via ovn-bridge-mappings. Frames egressing a VM travel TAP → br-int → OVN pipeline → br-ext → veth → br-wan → physical NIC → wire.
Running setup-ovn.sh
# Auto-detect the WAN bridge (recommended)
sudo setup-ovn.sh
# Explicitly specify the WAN bridge name
sudo setup-ovn.sh --wan-bridge=br-wan
In environments where the WAN IP comes from a router's DHCP server (homelab, small office), add --dhcp to obtain a gateway IP from the router automatically:
sudo setup-ovn.sh --dhcp
This requests an IP from the router's DHCP (e.g., 192.168.1.1 serving addresses on the LAN). This is not Spinifex's internal OVN DHCP that assigns private IPs to VMs — it's your network's existing DHCP server.
| Flags | Result |
|---|---|
| (no flags) | Auto-detect WAN bridge from default route, create br-int + br-ext |
--wan-bridge=<name> |
Use the specified Linux bridge as the WAN uplink |
--dhcp |
Obtain the OVN gateway IP from the router's DHCP |
If no Linux bridge owns the default route, setup-ovn.sh exits with guidance rather than silently breaking host connectivity.
OVN Control Plane on Multi-Node Clusters
How OVN is deployed depends on the size of the cluster, and this is the main reason three servers is the recommended minimum for a multi-server deployment.
| Cluster size | OVN databases | Tolerates |
|---|---|---|
| 1–2 servers | standalone, on the first node | nothing — that node is a single point of failure for the control plane |
| 3 or more | clustered (RAFT) across the first three nodes | loss of any one database node |
Servers beyond the third run the full set of Spinifex services, but do not join the OVN database cluster — they connect to it as clients. The quorum stays at three however large the cluster gets, which is what keeps control-plane write latency stable.
What a control-plane outage actually costs
Less than it sounds. ovn-controller has already programmed the forwarding rules into each host, so running instances keep full networking — east-west, north-south, NAT and security groups all continue to work with the databases down.
What stops is change. Creating a VPC, launching an instance, and updating a security group all need the control plane, because each has to write new logical topology before anything can program it.
So a standalone OVN deployment is a reasonable choice for a lab or a single-server install. It is a poor one for production, where the inability to launch an instance during an outage is usually as bad as the instances being down.
Clustering the databases
The three database nodes are set up with --db-cluster-local-addr, and nodes 2 and 3 additionally point at node 1 with --db-cluster-remote-addr. Compute nodes take --ovn-remote listing all three, so they survive any one of them failing. See Multi-Node Install for the exact commands in sequence.
A clustered database can only be created from scratch — the ovn-central package starts a standalone one on install, so the cluster setup passes --recreate-db to replace it. Confirm the result with:
sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
Three servers should be listed with exactly one leader. A report of a standalone database means the cluster flags did not take.
Per-Node Configuration
Different nodes in a cluster can have different WAN bridges and NICs:
[nodes.node1.vpcd]
external_interface = "br-wan"
[nodes.node2.vpcd]
external_interface = "br-public"
[nodes.node3.vpcd]
external_interface = "br-wan" # br-wan enslaving bond0
external_interface is the Linux bridge that owns the WAN uplink on this node — not the physical NIC. The physical NIC lives underneath the bridge. Each node runs setup-ovn.sh with its own WAN bridge name (or relies on auto-detection). OVN only requires ovn-bridge-mappings to point at br-ext.
Bridge Verification
# OVS bridges created by setup-ovn.sh
sudo ovs-vsctl br-exists br-int && echo "br-int OK" || echo "br-int MISSING"
sudo ovs-vsctl br-exists br-ext && echo "br-ext OK" || echo "br-ext MISSING"
# Linux WAN bridge owns the host IP and default route
ip -br addr show br-wan
ip route show default
# Default route's dev should be br-wan (or your WAN bridge name)
# br-ext should have one veth port linking it to br-wan
sudo ovs-vsctl list-ports br-ext
# Expect: a veth name (e.g. "veth-wan-ovs")
# Confirm the matching peer is enslaved to the Linux WAN bridge
sudo bridge link show | grep br-wan
# Bridge mappings must point at br-ext
sudo ovs-vsctl get Open_vSwitch . external-ids:ovn-bridge-mappings
# Output: "external:br-ext"
# Physical NIC is enslaved to br-wan (master should be br-wan)
ip -d link show wan
Network Flow Diagram
Configuration Reference
All network configuration lives in spinifex.toml. Settings are split into three levels: cluster-wide mode, IP pool definitions, and per-node NIC settings.
Configuration Levels
Cluster-Wide: external_mode
[network]
external_mode = "pool" # "pool", "nat", or "" (disabled)
| Value | Behavior |
|---|---|
"pool" |
Full public networking — public subnets, auto-assign, Elastic IPs |
"nat" |
Outbound-only SNAT — all VMs share one external IP |
"" / omitted |
Overlay-only — no external connectivity |
IP Pools: network.external_pools
Each pool defines where external IPs come from. You can have one pool (homelab) or many (multi-region datacenter).
[[network.external_pools]]
name = "wan" # Pool identifier (unique within cluster)
source = "static" # "static" (default) or "dhcp"
range_start = "192.168.1.150" # First allocatable IP (static source only)
range_end = "192.168.1.250" # Last allocatable IP (static source only)
gateway = "192.168.1.1" # WAN default gateway (next hop for 0.0.0.0/0)
gateway_ip = "" # OVN router SNAT address (defaults to range_start)
prefix_len = 24 # Subnet mask length
region = "" # Scope to region (optional)
az = "" # Scope to AZ (optional)
dns_servers = ["8.8.8.8"] # DNS for VMs (optional)
Field Details
| Field | Required | Description |
|---|---|---|
name |
Yes | Unique pool name. Referenced by AllocateAddress to target a specific pool. |
source |
No | IP source: "static" (default) uses range_start/range_end. "dhcp" obtains IPs from the router's DHCP server on each VM launch. |
range_start |
Static only | First IP in the range. First IP is reserved for OVN gateway SNAT (unless gateway_ip overrides). |
range_end |
Static only | Last IP in the range. |
gateway |
Yes | Physical router/switch — the WAN default gateway. OVN sets 0.0.0.0/0 → gateway. |
gateway_ip |
NAT mode | Static IP for OVN router SNAT. In pool mode, defaults to range_start (static) or first DHCP lease (dhcp). In NAT mode, this is the single external IP all VMs share. |
prefix_len |
Yes | Subnet mask for the external network (e.g., 24 = /24). |
region |
No | Scopes pool to a region. Instances in this region prefer this pool. |
az |
No | Scopes pool to an AZ. More specific than region. |
dns_servers |
No | DNS servers propagated to VMs via OVN DHCP. |
gw_lrp_range_start / gw_lrp_range_end |
No | Reserve gateway-LRP IPs for per-VPC OVN routers. When unset, the allocator auto-derives the top 16 host IPs of the pool subnet (~15 concurrent VPCs). Widen to raise the concurrent-VPC ceiling — the nat-transit pool defaults to 100.127.0.16-100.127.0.254 (239 VPCs). Must not overlap range_start/range_end. |
Why range_start/range_end Instead of CIDR?
Customer IP ranges rarely align to CIDR boundaries. A datacenter might have 203.0.113.10-203.0.113.200 from their ISP. Start/end avoids forcing admins to calculate CIDR blocks.
Gateway vs Gateway_IP
These are different things:
gateway= Your network's default gateway (e.g., 192.168.1.1). This is where OVN sends packets destined for the internet. It's your router.gateway_ip= The IP that OVN uses for outbound SNAT. In pool mode, defaults to the first IP in the range. In NAT mode, set this explicitly. Must be on the same subnet as the gateway.
Per-Node: nodes.NAME.vpcd
[nodes.spx1.vpcd]
# Comma-separated list of the OVN NB/SB quorum endpoints (3 DB nodes). vpcd and
# ovn-controller fail over across them and follow the RAFT leader.
ovn_nb_addr = "tcp:10.1.3.181:6641,tcp:10.1.3.182:6641,tcp:10.1.3.183:6641"
ovn_sb_addr = "tcp:10.1.3.181:6642,tcp:10.1.3.182:6642,tcp:10.1.3.183:6642"
external_interface = "br-wan" # WAN Linux bridge name
| Field | Description |
|---|---|
ovn_nb_addr / ovn_sb_addr |
OVN Northbound / Southbound DB endpoint(s). The NB and SB databases run clustered via OVSDB RAFT across the first three nodes (client ports 6641/6642, RAFT ports 6643/6644). Each node's config lists all three quorum endpoints so the loss of one DB node does not stall the control plane. A single tcp:IP:6641 string remains valid for single-node dev. |
external_interface |
Linux bridge that owns the WAN uplink on this node (e.g. br-wan, br-public). The physical NIC is enslaved to this bridge — not configured here. Different nodes may differ. |
Pool Selection Logic
When an instance needs a public IP:
- AZ-scoped pool first: Pool with matching
region+az - Region-scoped fallback: Pool with matching
region, noaz(overflow) - Unscoped fallback: Pool with no
region/az(global, homelab configs) - Exhausted: All pools full →
InsufficientAddressCapacityerror
AllocateAddress accepts optional pool name to target a specific block (maps to AWS PublicIpv4Pool).
IPAM Storage
Pool allocation state is stored durably in the cluster (NATS KV bucket spinifex-external-ipam, one entry per pool) and survives restarts. Each allocation records the ENI and instance holding the address. Pools are initialized from spinifex.toml on vpcd startup (idempotent).
Deployment Examples
Homelab / Dev (Single Pool)
Network: 192.168.1.0/24
Router: 192.168.1.1 (DHCP .2–.149)
Spinifex: 192.168.1.150–.250 (100 IPs)
[network]
external_mode = "pool"
[[network.external_pools]]
name = "wan"
range_start = "192.168.1.150"
range_end = "192.168.1.250"
gateway = "192.168.1.1"
prefix_len = 24
[nodes.homelab.vpcd]
external_interface = "br-wan"
Setup: Configure br-wan to enslave your physical WAN NIC (netplan, cloud-init, or systemd-networkd). Change your router's DHCP range to end at .149. Run sudo setup-ovn.sh — it auto-detects the WAN bridge from the default route, or specify it with --wan-bridge=br-wan.
Homelab / Dev (DHCP Pool — No Range Reservation)
Network: 192.168.1.0/24
Router: 192.168.1.1 (DHCP serves full range, no carve-out needed)
Spinifex: gets IPs from router DHCP on demand
[network]
external_mode = "pool"
[[network.external_pools]]
name = "wan"
source = "dhcp"
gateway = "192.168.1.1"
prefix_len = 24
dns_servers = ["192.168.1.1", "8.8.8.8"]
[nodes.homelab.vpcd]
external_interface = "br-wan"
Setup: No router changes needed. Spinifex requests IPs from the router's DHCP server when VMs launch and releases them on terminate. Requires dhclient on the host (apt install isc-dhcp-client).
Host-Local Subnet (No Upstream Router)
Network: 192.168.10.0/24 — host-local, reachable from the host only
Host WAN: 198.51.100.10/24 on br-wan (existing address — unchanged)
Gateway: 192.168.10.1 — second address added to br-wan
Add the VM pool gateway as a second address on br-wan alongside the existing WAN IP. The host acts as the gateway for the pool — no upstream router or DHCP server needed for this range.
# /etc/netplan/…
bridges:
br-wan:
addresses:
- 192.168.10.1/24 # VM pool gateway — host-local
- 198.51.100.10/24 # existing WAN IP — unchanged
routes:
- to: default
via: 198.51.100.1
[network]
external_mode = "pool"
[[network.external_pools]]
name = "wan"
source = "static" # required — no upstream DHCP for this range
range_start = "192.168.10.2"
range_end = "192.168.10.100"
gateway = "192.168.10.1" # second address on br-wan
prefix_len = 24
dns_servers = ["8.8.8.8"]
Setup: Apply with sudo netplan apply. VMs are reachable from the host at 192.168.10.x. For internet access through the host's WAN interface:
sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o br-wan -j MASQUERADE
Persist via /etc/sysctl.d/99-ip-forward.conf and netfilter-persistent save.
Datacenter / Colo (ISP Block)
ISP-assigned: 203.0.113.0/28 (14 usable IPs)
ISP gateway: 203.0.113.1
Servers: 3x with separate mgmt NIC (eth0) and public NIC enslaved to br-wan
[network]
external_mode = "pool"
[[network.external_pools]]
name = "public"
range_start = "203.0.113.2"
range_end = "203.0.113.14"
gateway = "203.0.113.1"
prefix_len = 28
[nodes.dc1.vpcd]
external_interface = "br-wan" # br-wan enslaves eth1
[nodes.dc2.vpcd]
external_interface = "br-wan" # br-wan enslaves eth1
[nodes.dc3.vpcd]
external_interface = "br-public" # br-public enslaves eno1
Enterprise On-Prem (VLAN)
The Linux WAN bridge enslaves a VLAN sub-interface (e.g. eth1.200) instead of a raw NIC. From OVN's perspective nothing changes — external_interface still points at the bridge.
[network]
external_mode = "pool"
[[network.external_pools]]
name = "dmz"
range_start = "172.16.0.100"
range_end = "172.16.0.200"
gateway = "172.16.0.1"
prefix_len = 24
[nodes.srv1.vpcd]
external_interface = "br-dmz" # br-dmz enslaves eth1.200
[nodes.srv2.vpcd]
external_interface = "br-dmz" # br-dmz enslaves bond0.200
Edge / Branch (Outbound Only)
[network]
external_mode = "nat"
[[network.external_pools]]
name = "wan"
gateway = "10.0.0.1"
gateway_ip = "10.0.0.50"
prefix_len = 24
[nodes.edge1.vpcd]
external_interface = "br-wan"
Multi-Region (Multiple Pools)
[network]
external_mode = "pool"
# US East — AZ-scoped
[[network.external_pools]]
name = "us-east-1a"
range_start = "203.0.113.2"
range_end = "203.0.113.254"
gateway = "203.0.113.1"
prefix_len = 24
region = "us-east-1"
az = "us-east-1a"
# US East — overflow (any AZ in region)
[[network.external_pools]]
name = "us-east-overflow"
range_start = "192.0.2.2"
range_end = "192.0.3.254"
gateway = "192.0.2.1"
prefix_len = 23
region = "us-east-1"
# EU West
[[network.external_pools]]
name = "eu-west"
range_start = "213.189.1.2"
range_end = "213.189.2.254"
gateway = "213.189.1.1"
prefix_len = 23
region = "eu-west-1"
Spinifex allocates from the correct pool based on where the instance launches. An instance in us-east-1a gets an IP from us-east-1a first; if exhausted, falls back to us-east-overflow.
Security Groups
Security groups are stateful firewalls enforced at the OVS datapath level on each hypervisor. Traffic is filtered before it reaches the wire — equivalent to AWS Nitro card enforcement. The VM never sees dropped packets.
How Security Groups Work
Each security group maps to an OVN Port Group. When an instance launches, its ENI port is added to the port group(s) for its security groups. ACL rules on the port group control traffic:
- Default deny: All inbound traffic dropped at priority 900
- Allow rules: Specific ports/protocols allowed at priority 1000 (overrides deny)
- Stateful: All allow rules use
allow-related— return traffic is automatically permitted
Default Security Group
Every VPC gets a default security group that:
- Allows all inbound from instances in the same security group
- Allows all outbound
- Denies all other inbound
AWS Rule → OVN ACL Translation
| AWS Security Group Rule | OVN ACL Match |
|---|---|
| Ingress TCP/22 from 0.0.0.0/0 | outport == @sg && ip4 && tcp.dst == 22 |
| Ingress TCP/443 from 10.0.0.0/8 | outport == @sg && ip4 && tcp.dst == 443 && ip4.src == 10.0.0.0/8 |
| Ingress ALL from sg-other | outport == @sg && ip4 && ip4.src == $sg_other_ip4 |
| Ingress ICMP from anywhere | outport == @sg && ip4 && icmp4 |
| Egress ALL to 0.0.0.0/0 | inport == @sg && ip4 |
| Default deny inbound | outport == @sg && ip4 (priority 900, action=drop) |
Example: Allow SSH + HTTP
# Create security group
SG=$(aws ec2 create-security-group --group-name web \
--description "Web servers" --vpc-id $VPC \
--query GroupId --output text)
# Allow SSH from anywhere
aws ec2 authorize-security-group-ingress --group-id $SG \
--protocol tcp --port 22 --cidr 0.0.0.0/0
# Allow HTTP from anywhere
aws ec2 authorize-security-group-ingress --group-id $SG \
--protocol tcp --port 80 --cidr 0.0.0.0/0
# Launch instance with this SG
aws ec2 run-instances --image-id $AMI --instance-type t3.small \
--subnet-id $SUBNET --security-group-ids $SG --key-name mykey
Rule changes take effect immediately — no instance restart needed.
The Instance-to-Host Plane (Metadata and VPC DNS)
Security groups govern traffic between instances and to the outside. There is a third path they do not touch, and it is the one most easily left exposed: traffic from an instance to the host it runs on.
Every instance reaches two link-local addresses served by the hypervisor:
| Address | Service | Port |
|---|---|---|
169.254.169.254 |
Instance metadata (IMDS) — cloud-init, instance-role credentials | TCP 80 |
169.254.169.253 |
VPC DNS resolver | UDP/TCP 53 |
These are not guest addresses and not OVN routed. Each instance's tap has a
capture rule on the hypervisor that intercepts any packet addressed to .254 or
.253 and delivers it to a per-ENI internal port in the host network namespace
(named ime-*). That interception is by destination address only — it does not
match the port. A packet to 169.254.169.254:22 is delivered to the host just
as readily as one to 169.254.169.254:80.
That matters because host services — SSH, the AWS gateway on 9999, the console on
3000, the DNS server — bind the wildcard address, so they also answer on these
link-local addresses. Without a control on the ime-* path, an instance can
reach every one of them, with none of the source scoping that protects the same
services on the network. A resolver query sent straight to the host's DNS port
this way also bypasses the per-instance DNS rate limit.
This plane is invisible to a network firewall
The critical point for anyone securing a deployment: this traffic never crosses
the physical NIC. It is intra-host, on the OVS bridge between the instance's tap
and the ime-* port. A cloud security group, an upstream firewall, ufw, or any
rule written against the WAN interface is not in this path and cannot filter it.
The only place to enforce it is on the ime-* interface in the host itself.
How it is protected
Spinifex's host firewall carries a rule that scopes the ime-* path to exactly
the two legitimate endpoints and ports — IMDS on .254:80, DNS on .253:53 —
and drops everything else. Whether you have that rule depends entirely on whether
the host firewall is on:
| Install path | Host firewall | This plane |
|---|---|---|
| From the ISO | on | protected |
Binary installer (curl | bash) or setup.sh |
off | exposed |
Binary installer with --firewall=on |
on | protected |
The ISO ships the firewall armed, so ISO deployments are protected out of the box. The binary installer ships it off — deliberately, because it runs on servers that may already have services the installer knows nothing about, and a default-deny policy could cut them off. The consequence is that a binary install left at its default has this plane wide open, and so does every host service on the WAN besides.
If you install any way other than the ISO, turn the host firewall on. At install time:
curl -fsSL https://install.mulgadc.com | bash -s -- --firewall=on
# or, from a source checkout:
sudo /usr/local/share/spinifex/setup.sh --firewall=on
or afterwards, in /etc/spinifex/spinifex.toml, followed by a daemon restart:
[network]
firewall_enabled = true
Before enabling it, check what else the machine is serving — anything listening outside the public port group stops accepting new connections. See Firewall and Cluster Membership for the full port policy and the cluster-formation steps.
Verifying it
On the hypervisor, the ime-* accept should be scoped, not blanket:
sudo nft list chain inet spinifex_filter input | grep ime
# Expect three rules naming 169.254.169.254 tcp dport 80 and 169.254.169.253
# udp/tcp dport 53 — not a bare `iifname "ime-*" accept`.
From an instance, the two service ports work and nothing else does:
curl -s -o /dev/null -w '%{http_code}\n' http://169.254.169.254/latest/meta-data/ # 401 (IMDSv2)
dig +short @169.254.169.253 example.com A # resolves
nc -vz 169.254.169.254 22 # must fail
A missing spinifex_filter table means the host firewall is off and this plane
is unprotected regardless of the rule above.
Elastic IPs
Elastic IPs are static public IPs that persist across instance stop/start cycles. Unlike auto-assigned public IPs (which change on stop/start), an Elastic IP stays with your instance.
# Allocate
EIP=$(aws ec2 allocate-address --query AllocationId --output text)
# Associate with instance
aws ec2 associate-address --allocation-id $EIP --instance-id $INSTANCE
# Stop/start instance — same Elastic IP
# Disassociate
aws ec2 disassociate-address --association-id $ASSOC_ID
# Release back to pool
aws ec2 release-address --allocation-id $EIP
When you associate an Elastic IP with an instance that already has an auto-assigned public IP, the auto-assigned IP is released and replaced.
OVN Reference
For operators debugging or verifying the OVN topology.
IGW Attach Creates
# External logical switch with localnet port
ovn-nbctl ls-add ext-{vpcId}
ovn-nbctl lsp-add ext-{vpcId} ext-localnet-{vpcId}
ovn-nbctl lsp-set-type ext-localnet-{vpcId} localnet
ovn-nbctl lsp-set-addresses ext-localnet-{vpcId} unknown
ovn-nbctl lsp-set-options ext-localnet-{vpcId} network_name=external
# Gateway router port with real external IP
ovn-nbctl lrp-add vpc-{vpcId} gw-{vpcId} {mac} 192.168.1.150/24
# Connect external switch to router
ovn-nbctl lsp-add ext-{vpcId} ext-rtr-{vpcId}
ovn-nbctl lsp-set-type ext-rtr-{vpcId} router
ovn-nbctl lsp-set-options ext-rtr-{vpcId} router-port=gw-{vpcId}
# Gateway chassis HA
ovn-nbctl lrp-set-gateway-chassis gw-{vpcId} chassis-1 20
ovn-nbctl lrp-set-gateway-chassis gw-{vpcId} chassis-2 15
# SNAT for all VPC traffic
ovn-nbctl lr-nat-add vpc-{vpcId} snat 192.168.1.150 10.0.0.0/16
# Default route to WAN
ovn-nbctl lr-route-add vpc-{vpcId} 0.0.0.0/0 192.168.1.1
Per-Instance Public IP
# 1:1 NAT — distributed (DNAT processed on the VM's own chassis)
ovn-nbctl lr-nat-add vpc-{vpcId} dnat_and_snat {public_ip} {private_ip} \
port-{eniId} {vm_mac}
With the WAN NIC on a Linux bridge wired to OVS via veth, OVS sees every frame on the wire regardless of MAC, so OVN can use distributed NAT. The DNAT is processed on the chassis hosting the VM rather than hairpinning through a single gateway chassis.
Security Group
# Create port group
ovn-nbctl pg-add sg-{groupId}
# Add VM ports
ovn-nbctl pg-set-ports sg-{groupId} port-{eniId1} port-{eniId2}
# Allow SSH inbound (stateful)
ovn-nbctl acl-add sg-{groupId} to-lport 1000 \
'outport == @sg_{groupId} && ip4 && tcp.dst == 22' allow-related
# Allow all egress
ovn-nbctl acl-add sg-{groupId} from-lport 1000 \
'inport == @sg_{groupId} && ip4' allow-related
# Default deny inbound
ovn-nbctl acl-add sg-{groupId} to-lport 900 \
'outport == @sg_{groupId} && ip4' drop
Useful Debug Commands
# List all logical routers (VPCs)
sudo ovn-nbctl lr-list
# List all logical switches (subnets + external)
sudo ovn-nbctl ls-list
# Show NAT rules for a VPC
sudo ovn-nbctl lr-nat-list vpc-{vpcId}
# Show routes for a VPC
sudo ovn-nbctl lr-route-list vpc-{vpcId}
# Show chassis (nodes) in the cluster
sudo ovn-sbctl show
# Show port bindings (which VM is on which host)
sudo ovn-sbctl find Port_Binding type="" | grep -E "logical_port|chassis"
# Check ACLs on a security group
sudo ovn-nbctl acl-list sg-{groupId}
# Check port group membership
sudo ovn-nbctl pg-get-ports sg-{groupId}
Quick Start
1. Set Up OVN Bridges
Make sure the WAN NIC is enslaved to a Linux bridge (e.g. br-wan) and that bridge owns the default route. Then run:
sudo setup-ovn.sh # auto-detect WAN bridge
# or
sudo setup-ovn.sh --wan-bridge=br-wan # explicit
2. Configure External IP Pool
spx admin init
# Follow prompts — auto-detects NICs, suggests IP pool range
# Or edit spinifex.toml manually
3. Create VPC with Public Subnet
VPC=$(aws ec2 create-vpc --cidr-block 10.200.0.0/16 \
--query Vpc.VpcId --output text)
IGW=$(aws ec2 create-internet-gateway \
--query InternetGateway.InternetGatewayId --output text)
aws ec2 attach-internet-gateway \
--internet-gateway-id $IGW --vpc-id $VPC
SUBNET=$(aws ec2 create-subnet --vpc-id $VPC \
--cidr-block 10.200.1.0/24 \
--query Subnet.SubnetId --output text)
# Route table — give the subnet a default route to the IGW. Spinifex does NOT
# add this automatically; without it the subnet's egress is dropped.
RT=$(aws ec2 create-route-table --vpc-id $VPC \
--query RouteTable.RouteTableId --output text)
aws ec2 create-route --route-table-id $RT \
--destination-cidr-block 0.0.0.0/0 --gateway-id $IGW
aws ec2 associate-route-table --route-table-id $RT --subnet-id $SUBNET
# Auto-assign a public IP to instances launched into the subnet
aws ec2 modify-subnet-attribute \
--subnet-id $SUBNET --map-public-ip-on-launch
# Allow SSH + ICMP on the VPC's default security group (blocks inbound by default)
SG=$(aws ec2 describe-security-groups \
--filters Name=vpc-id,Values=$VPC \
--query 'SecurityGroups[0].GroupId' --output text)
aws ec2 authorize-security-group-ingress --group-id $SG \
--protocol tcp --port 22 --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-id $SG \
--protocol icmp --port -1 --cidr 0.0.0.0/0
4. Launch Instance
INSTANCE=$(aws ec2 run-instances \
--image-id $AMI --instance-type t3.small \
--subnet-id $SUBNET --key-name mykey \
--query Instances[0].InstanceId --output text)
aws ec2 describe-instances --instance-ids $INSTANCE \
--query 'Reservations[0].Instances[0].[PrivateIpAddress,PublicIpAddress]'
Troubleshooting
Debugging Toolkit
These commands are used throughout the troubleshooting sections below. Learn them — they cover 90% of VPC networking issues.
OVN Northbound (Logical Topology)
# Full topology overview (routers, switches, ports)
sudo ovn-nbctl show
# List all VPC routers
sudo ovn-nbctl lr-list
# List all switches (subnets + external)
sudo ovn-nbctl ls-list
# NAT rules for a VPC
sudo ovn-nbctl lr-nat-list vpc-{vpcId}
# Routes for a VPC
sudo ovn-nbctl lr-route-list vpc-{vpcId}
# Port details (check "up" field for DHCP status)
sudo ovn-nbctl list Logical_Switch_Port port-eni-{eniId}
# Gateway chassis assignment
sudo ovn-nbctl list Logical_Router_Port gw-vpc-{vpcId}
# Localnet port options (network_name should be "external")
sudo ovn-nbctl get Logical_Switch_Port ext-port-vpc-{vpcId} options
OVN Southbound (runtime state)
# Chassis list + port bindings (which VM is on which host)
sudo ovn-sbctl show
# Detailed chassis info (check name matches expectations)
sudo ovn-sbctl list Chassis
# MAC binding table (shows ARP resolution for external traffic)
sudo ovn-sbctl list MAC_Binding
# Trace a packet through the OVN pipeline (invaluable for debugging)
sudo ovn-trace ext-vpc-{vpcId} \
'inport=="ext-port-vpc-{vpcId}" && eth.dst==ff:ff:ff:ff:ff:ff && \
arp.op==1 && arp.spa==192.168.1.13 && arp.tpa==192.168.1.201'
OVN DB RAFT cluster (NB/SB replication)
The NB and SB databases run clustered across the first three nodes via native OVSDB RAFT. A 3-node quorum tolerates the loss of one DB node; check status when the control plane stalls.
# NB cluster status: expect 3 servers and exactly one "Role: leader"
sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
# SB cluster status
sudo ovn-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound
# Drive a CLI at the whole quorum (fails over past a dead node)
sudo ovn-nbctl --db=tcp:10.1.3.181:6641,tcp:10.1.3.182:6641,tcp:10.1.3.183:6641 show
A node showing only itself under Servers: never joined the cluster — confirm it was bootstrapped with --db-cluster-remote-addr pointing at the creator and that RAFT ports 6643/6644 are reachable between DB nodes.
OVS (datapath / physical wiring)
# Full bridge + port topology
sudo ovs-vsctl show
# Kernel datapath ports and stats
sudo ovs-dpctl show
# Kernel datapath flow cache (actual forwarding rules)
sudo ovs-dpctl dump-flows
# OpenFlow rules installed by ovn-controller
sudo ovs-ofctl dump-flows br-int | grep {pattern}
# Conntrack entries (shows active NAT sessions)
sudo ovs-appctl dpctl/dump-conntrack | grep {ip}
# FDB (MAC address table) for a bridge
sudo ovs-appctl fdb/show br-wan
# OVS external_ids (system-id, bridge-mappings, encap-ip)
sudo ovs-vsctl get Open_vSwitch . external_ids
Network Interfaces
# Physical NIC should be enslaved to br-wan ("master br-wan")
ip -d link show {nic}
# Linux WAN bridge — owns the host IP and default route
ip -br addr show br-wan
ip route show default
# OVS bridges and the veth linking br-ext to br-wan
sudo ovs-vsctl list-ports br-ext
bridge link show | grep br-wan
# Interface traffic stats (RX/TX counts, drops)
ip -s link show {nic}
ip -s link show br-wan
Packet Capture
# Capture on the OVS uplink bridge (frames between OVN and br-wan)
sudo tcpdump -i br-ext -n -e arp
# Capture on the Linux WAN bridge (frames between br-ext veth and the NIC)
sudo tcpdump -i br-wan -n -e "host {public_ip}"
# Capture on the physical NIC (sees everything on the wire)
sudo tcpdump -i {nic} -n "host {public_ip}"
Service Logs
# vpcd log (reconcile, NAT, topology operations)
journalctl -u spinifex-vpcd -f
# ovn-controller log (port binding, commit failures)
sudo cat /var/log/ovn/ovn-controller.log | tail -50
# Daemon log (instance launch, network setup)
journalctl -u spinifex-daemon -f
VPC Creation Fails
Check OVN services and vpcd daemon:
sudo systemctl is-active ovn-controller
journalctl -u spinifex-vpcd -f
Instances Cannot Reach Each Other
Geneve tunnels may not be established:
sudo ovs-vsctl show | grep -i geneve
sudo ss -ulnp | grep 6081
From inside a VM:
ip addr show
ip route show
Instance Has No Public IP
-
Check subnet has
MapPublicIpOnLaunch:aws ec2 describe-subnets --subnet-ids $SUBNET \ --query 'Subnets[0].MapPublicIpOnLaunch' -
Check IGW is attached:
aws ec2 describe-internet-gateways \ --filters Name=attachment.vpc-id,Values=$VPC -
Check external IP pool:
nats kv get spinifex-external-ipam wan -
Check OVN NAT rules:
sudo ovn-nbctl lr-nat-list vpc-$VPC
Instance Has Public IP But No Internet
The instance shows a public IP in describe-instances but cannot reach the internet, and inbound connections hang. The usual cause is a missing default route: the subnet's effective route table has no 0.0.0.0/0 route to the IGW, so Spinifex gates the subnet with a drop policy.
# Find the route table that applies to the subnet and check its routes
aws ec2 describe-route-tables \
--filters Name=association.subnet-id,Values=$SUBNET \
--query 'RouteTables[0].Routes'
# Expect a route with DestinationCidrBlock 0.0.0.0/0 and a GatewayId of igw-...
If the subnet has no explicit association it falls back to the VPC's main route table — check that one too, then add the route to whichever table applies:
aws ec2 create-route --route-table-id $RT \
--destination-cidr-block 0.0.0.0/0 --gateway-id $IGW
On the host, the drop policy installed when a subnet lacks an IGW route appears as a Logical_Router_Policy on the VPC router:
sudo ovn-nbctl lr-policy-list vpc-$VPC
# A drop policy matching the subnet CIDR means the subnet is gated.
Public IP Not Reachable from WAN
Work through these checks in order — each eliminates a class of issues.
1. Verify OVS wiring
# br-int and br-ext must exist
sudo ovs-vsctl show | grep -E "Bridge (br-int|br-ext)"
# br-ext should have a veth port linking it to br-wan
sudo ovs-vsctl list-ports br-ext
# The Linux WAN bridge should have the physical NIC and the veth peer
bridge link show | grep br-wan
# Bridge mappings must point at br-ext
sudo ovs-vsctl get Open_vSwitch . external-ids:ovn-bridge-mappings
# Expected: "external:br-ext"
2. Verify chassis and gateway scheduling
# What OVS thinks the chassis name is
sudo ovs-vsctl get Open_vSwitch . external-ids:system-id
# What OVN SB registered (must match the system-id above)
sudo ovn-sbctl show
# Look for: Chassis {name}
# What vpcd scheduled as gateway chassis (must match SB chassis name)
sudo ovn-nbctl list Logical_Router_Port gw-vpc-{vpcId} | grep gateway_chassis
If these don't match, see "Chassis name mismatch" below.
3. Verify NAT rule
sudo ovn-nbctl lr-nat-list vpc-$VPC | grep dnat_and_snat
# Must show the public IP → private IP mapping
# For distributed NAT, external_mac and logical_port should be set
# (the VM's MAC and ENI port name)
4. Verify ARP resolution
From another host on the same LAN, check if OVN responds to ARP:
# On the remote host:
ping -c 1 {public_ip}
ip neigh show {public_ip}
# Should show the VM's ENI MAC (distributed NAT) or the OVN router MAC
If ARP fails, confirm the WAN bridge is forwarding:
# Physical NIC must be enslaved to br-wan
ip -d link show {nic} | grep "master br-wan"
# br-wan must be UP and have an IP
ip -br addr show br-wan
# br-ext must have a veth port whose peer is enslaved to br-wan
sudo ovs-vsctl list-ports br-ext
bridge link show | grep br-wan
5. Verify packet flow with tcpdump
Capture at each layer to find where packets stop:
# Layer 1: Does the ARP/ICMP arrive on the physical NIC?
sudo tcpdump -i {nic} -n "host {public_ip}"
# Layer 2: Does it cross the Linux WAN bridge?
sudo tcpdump -i br-wan -n "host {public_ip}"
# Layer 3: Does it reach the OVS uplink bridge?
sudo tcpdump -i br-ext -n -e "host {public_ip}"
If traffic arrives on the physical NIC but not br-wan, the NIC is not enslaved to the bridge. If it reaches br-wan but not br-ext, the veth pair between them is missing or down — re-run setup-ovn.sh.
6. Use ovn-trace for pipeline debugging
Simulate a packet through the entire OVN pipeline:
sudo ovn-trace --ct=new ext-vpc-{vpcId} \
'inport=="ext-port-vpc-{vpcId}" && eth.dst==ff:ff:ff:ff:ff:ff && \
arp.op==1 && arp.sha=={remote_mac} && arp.spa=={remote_ip} && \
arp.tpa=={public_ip}'
The output shows every table the packet passes through and what action is taken. Look for drop actions or unexpected paths.
OVN SB Commit Failure Loop
Symptom: ovn-controller log shows:
OVNSB commit failed, force recompute next time.
Repeated millions of times. Port binding never happens (up: false).
Cause: Stale entries in the OVN Southbound DB (old chassis records, port bindings, datapath bindings) conflict with ovn-controller's expected state, typically after an ungraceful shutdown.
Fix: Delete both OVN DB files and restart:
sudo systemctl stop ovn-central ovn-controller
sudo rm -f /var/lib/ovn/ovnnb_db.db /var/lib/ovn/ovnsb_db.db
sudo systemctl start ovn-central ovn-controller
# vpcd reconcile will recreate the NB topology on next startup
WAN NIC Not Enslaved to a Bridge
Symptom: setup-ovn.sh exits with an error like "default route is on a physical NIC, not a bridge" and refuses to continue.
Cause: Spinifex requires the WAN NIC to be enslaved to a Linux bridge (typically br-wan). Macvlan is no longer supported, and attaching the NIC directly to OVS would break SSH and any other host services using the NIC.
Fix: Move the host IP and default route onto a Linux bridge. Example netplan:
network:
version: 2
ethernets:
wan:
dhcp4: false
bridges:
br-wan:
interfaces: [wan]
dhcp4: true
Apply (sudo netplan apply), confirm the host IP is now on br-wan (ip -br addr show br-wan), and re-run setup-ovn.sh.
Stale ARP on Remote Hosts
Symptom: Ping from a LAN host to a VM public IP fails after a reset, but worked before. The remote host has a stale ARP entry with the old MAC.
Fix: Flush the ARP entry on the remote host:
# On the remote host:
sudo ip neigh flush dev {nic} {public_ip}
ping {public_ip} # should work now
OVN sends periodic gratuitous ARPs from the chassis hosting the VM that will eventually update remote ARP caches, but flushing is faster for testing.
Security Group Rules Not Taking Effect
# Check port is in correct port group
sudo ovn-nbctl pg-get-ports sg-$SG_ID
# Check ACLs
sudo ovn-nbctl acl-list sg-$SG_ID