mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
Josh Siviour
91d22d9f12
* feat(security): add an nftables host firewall scoping cluster ports to peers Nodes shipped no host firewall, so every cluster listener — OVN NB/SB, the formation port, NATS, predastore — was reachable from anything that could route to the node. On a single-NIC node the planes collapse onto the public address, which made that the default rather than the exception. setup.sh installs a policy and an apply helper but does not enable them: the peer set is unknown until the cluster's planes resolve, and a drop policy with an empty set would cut the first node that applied it out of its own formation. install-node.sh writes the peers and enables the unit after formation. The policy owns only table inet spinifex_filter and replaces it in one transaction. vpcd writes MASQUERADE and per-EIP FORWARD rules into the ip nat/filter tables and reinstalls them only on start, so a whole-ruleset flush — what Debian's nftables.service does on both start and stop — would break elastic IPs silently. INPUT only, for the same reason: a forward-hook drop here could not be rescued by an accept in the other table. Guest traffic keeps working: IMDS and VPC DNS terminate on ime-* ports in the host netns, and every guest agent reaches its control plane over the AWS gateway rather than NATS, so no cluster port needs to face the guests. * feat: reconcile the host firewall peer set from the daemon The peer set was written by install-node.sh and by cluster-bootstrap.yml. That missed two of the four formation paths (bm-bootstrap.sh and the ISO firstboot), duplicated the same logic in each, and went stale as soon as a node was added or replaced. host.ReconcileFirewall now owns it: every formation path starts the daemon, and it re-runs on each startup. A node's own config names its peers by their wan address only, so a set built from config alone drops every peer's NATS, OVSDB and formation traffic. The lan plane comes from the generated NATS cluster routes and the OVN remote lists, and the vpc plane from the Southbound Encap table; an empty encap answer is an error rather than a narrower set. ProtectSystem=full applies to the sudo child too, so the root helper could not write /etc/spinifex/firewall either — it is now in the daemon's ReadWritePaths, optional so a node installed before the firewall stage still starts. The apply helper gains set-peers and disable verbs, re-validates every address as a dotted quad with in-range octets, and rolls the peer file back if the ruleset will not load. * fix: retry the host firewall reconcile until every chassis registers The peer set is only safe to write once OVN reports an encap address for every node. On a fresh bootstrap the daemon routinely wins the race against ovn-controller, so a single startup attempt saw a partial chassis list and either wrote a set that dropped Geneve from the nodes missing from it, or failed outright and left the node with no policy until a restart. Gate on the full chassis count and schedule the reconcile instead of running it once: back off from 15s to 5m while it fails, then re-run every 5m so a node added or replaced after formation is picked up without a daemon restart. An unchanged peer set short-circuits before the helper runs. * test: cover the firewall chassis quorum gate in the multinode e2e suite A partial chassis list is the normal state early in a bootstrap, and writing a peer set from one produces a firewall that drops Geneve from every node missing from it. The gate that refuses that write had no live coverage: a 5-node bootstrap converges too fast for any daemon to observe a partial list, so the verification run only ever exercised the OVN-unreachable path. Reach the branch by inflating the expected count instead of breaking OVN: a phantom node in one node's config makes the cluster one chassis short, with nothing in the Southbound DB disturbed. Asserts the gate logs, that no peer file is written while it holds, that the loaded ruleset survives, and that the node recovers to a byte-identical peer set once the phantom is removed.