* feat(security): add an nftables host firewall scoping cluster ports to peers
Nodes shipped no host firewall, so every cluster listener — OVN NB/SB, the
formation port, NATS, predastore — was reachable from anything that could route
to the node. On a single-NIC node the planes collapse onto the public address,
which made that the default rather than the exception.
setup.sh installs a policy and an apply helper but does not enable them: the
peer set is unknown until the cluster's planes resolve, and a drop policy with
an empty set would cut the first node that applied it out of its own formation.
install-node.sh writes the peers and enables the unit after formation.
The policy owns only table inet spinifex_filter and replaces it in one
transaction. vpcd writes MASQUERADE and per-EIP FORWARD rules into the ip
nat/filter tables and reinstalls them only on start, so a whole-ruleset flush —
what Debian's nftables.service does on both start and stop — would break
elastic IPs silently. INPUT only, for the same reason: a forward-hook drop here
could not be rescued by an accept in the other table.
Guest traffic keeps working: IMDS and VPC DNS terminate on ime-* ports in the
host netns, and every guest agent reaches its control plane over the AWS
gateway rather than NATS, so no cluster port needs to face the guests.
* feat: reconcile the host firewall peer set from the daemon
The peer set was written by install-node.sh and by cluster-bootstrap.yml. That
missed two of the four formation paths (bm-bootstrap.sh and the ISO firstboot),
duplicated the same logic in each, and went stale as soon as a node was added or
replaced. host.ReconcileFirewall now owns it: every formation path starts the
daemon, and it re-runs on each startup.
A node's own config names its peers by their wan address only, so a set built
from config alone drops every peer's NATS, OVSDB and formation traffic. The
lan plane comes from the generated NATS cluster routes and the OVN remote lists,
and the vpc plane from the Southbound Encap table; an empty encap answer is an
error rather than a narrower set.
ProtectSystem=full applies to the sudo child too, so the root helper could not
write /etc/spinifex/firewall either — it is now in the daemon's ReadWritePaths,
optional so a node installed before the firewall stage still starts.
The apply helper gains set-peers and disable verbs, re-validates every address
as a dotted quad with in-range octets, and rolls the peer file back if the
ruleset will not load.
* fix: retry the host firewall reconcile until every chassis registers
The peer set is only safe to write once OVN reports an encap address for
every node. On a fresh bootstrap the daemon routinely wins the race against
ovn-controller, so a single startup attempt saw a partial chassis list and
either wrote a set that dropped Geneve from the nodes missing from it, or
failed outright and left the node with no policy until a restart.
Gate on the full chassis count and schedule the reconcile instead of running
it once: back off from 15s to 5m while it fails, then re-run every 5m so a
node added or replaced after formation is picked up without a daemon restart.
An unchanged peer set short-circuits before the helper runs.
* test: cover the firewall chassis quorum gate in the multinode e2e suite
A partial chassis list is the normal state early in a bootstrap, and writing a
peer set from one produces a firewall that drops Geneve from every node missing
from it. The gate that refuses that write had no live coverage: a 5-node
bootstrap converges too fast for any daemon to observe a partial list, so the
verification run only ever exercised the OVN-unreachable path.
Reach the branch by inflating the expected count instead of breaking OVN: a
phantom node in one node's config makes the cluster one chassis short, with
nothing in the Southbound DB disturbed. Asserts the gate logs, that no peer
file is written while it holds, that the loaded ruleset survives, and that the
node recovers to a byte-identical peer set once the phantom is removed.
setup-ovn.sh set the OVN Northbound and Southbound client remotes with a bare
set-connection ptcp:6641 / ptcp:6642. The address-free ptcp: form binds every
interface, so on a multi-plane node the unauthenticated, unencrypted OVSDB
control plane was reachable from the public interface. Write access to the
Southbound database is control of the whole tenant network fabric.
ovsdb-server takes remotes from two additive sources: the Connection table,
which RAFT replicates cluster-wide, and local ovn-ctl command-line flags. That
split is what the fix uses. Loopback goes in the replicated row, because
127.0.0.1 is correct on every node, so replication is harmless and local
clients keep working. The lan address goes on the per-node command line via
--db-nb-addr and --db-nb-create-insecure-remote, which never replicates, so
each node binds its own address without pushing it to its peers.
A new --lan-addr flag carries that address, falling back to
--db-cluster-local-addr so clustered nodes stay correct without caller changes.
The standalone branch now writes /etc/default/ovn-central too, since a two-node
install still has a remote ovn-controller that needs the lan listener. The
guard on --db-nb-addr is load-bearing: it defaults to 0.0.0.0, so emitting
create-insecure-remote without a resolved address would recreate the bug.
install-node.sh passes the lan address on both branches and warns when the lan
plane has collapsed onto the public address, since the listener then lands on
the public interface and a firewall is the only remaining control.
Josh Siviour