mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-20 19:25:52 +00:00
91d22d9f12
* feat(security): add an nftables host firewall scoping cluster ports to peers Nodes shipped no host firewall, so every cluster listener — OVN NB/SB, the formation port, NATS, predastore — was reachable from anything that could route to the node. On a single-NIC node the planes collapse onto the public address, which made that the default rather than the exception. setup.sh installs a policy and an apply helper but does not enable them: the peer set is unknown until the cluster's planes resolve, and a drop policy with an empty set would cut the first node that applied it out of its own formation. install-node.sh writes the peers and enables the unit after formation. The policy owns only table inet spinifex_filter and replaces it in one transaction. vpcd writes MASQUERADE and per-EIP FORWARD rules into the ip nat/filter tables and reinstalls them only on start, so a whole-ruleset flush — what Debian's nftables.service does on both start and stop — would break elastic IPs silently. INPUT only, for the same reason: a forward-hook drop here could not be rescued by an accept in the other table. Guest traffic keeps working: IMDS and VPC DNS terminate on ime-* ports in the host netns, and every guest agent reaches its control plane over the AWS gateway rather than NATS, so no cluster port needs to face the guests. * feat: reconcile the host firewall peer set from the daemon The peer set was written by install-node.sh and by cluster-bootstrap.yml. That missed two of the four formation paths (bm-bootstrap.sh and the ISO firstboot), duplicated the same logic in each, and went stale as soon as a node was added or replaced. host.ReconcileFirewall now owns it: every formation path starts the daemon, and it re-runs on each startup. A node's own config names its peers by their wan address only, so a set built from config alone drops every peer's NATS, OVSDB and formation traffic. The lan plane comes from the generated NATS cluster routes and the OVN remote lists, and the vpc plane from the Southbound Encap table; an empty encap answer is an error rather than a narrower set. ProtectSystem=full applies to the sudo child too, so the root helper could not write /etc/spinifex/firewall either — it is now in the daemon's ReadWritePaths, optional so a node installed before the firewall stage still starts. The apply helper gains set-peers and disable verbs, re-validates every address as a dotted quad with in-range octets, and rolls the peer file back if the ruleset will not load. * fix: retry the host firewall reconcile until every chassis registers The peer set is only safe to write once OVN reports an encap address for every node. On a fresh bootstrap the daemon routinely wins the race against ovn-controller, so a single startup attempt saw a partial chassis list and either wrote a set that dropped Geneve from the nodes missing from it, or failed outright and left the node with no policy until a restart. Gate on the full chassis count and schedule the reconcile instead of running it once: back off from 15s to 5m while it fails, then re-run every 5m so a node added or replaced after formation is picked up without a daemon restart. An unchanged peer set short-circuits before the helper runs. * test: cover the firewall chassis quorum gate in the multinode e2e suite A partial chassis list is the normal state early in a bootstrap, and writing a peer set from one produces a firewall that drops Geneve from every node missing from it. The gate that refuses that write had no live coverage: a 5-node bootstrap converges too fast for any daemon to observe a partial list, so the verification run only ever exercised the OVN-unreachable path. Reach the branch by inflating the expected count instead of breaking OVN: a phantom node in one node's config makes the cluster one chassis short, with nothing in the Southbound DB disturbed. Asserts the gate logs, that no peer file is written while it holds, that the loaded ruleset survives, and that the node recovers to a byte-identical peer set once the phantom is removed.
104 lines
3.6 KiB
Go
104 lines
3.6 KiB
Go
//go:build e2e
|
|
|
|
package multinode
|
|
|
|
import (
|
|
"testing"
|
|
)
|
|
|
|
// Top-level Test* wrappers, each delegating to a runX function. Parallel tests
|
|
// share the read-only singleton trio; sequential tests mutate cluster state
|
|
// (stop/start nodes, predastore, EIP pool) and are ordered so the cluster is
|
|
// fully restabilised before later tests probe the trio.
|
|
//
|
|
// Pre-flight (/dev/kvm writability, peer SSH reachability) runs inside
|
|
// requireMultiNodeFixture itself, so the package fixture singleton fails
|
|
// fast with a clear message before any Test* body executes.
|
|
|
|
// Baseline test owns a dedicated SG and self-cleaning instances; it runs
|
|
// before needInstanceTrio so the default SG/subnet/route table are in
|
|
// pristine state.
|
|
func TestMultinodeSameSGCrossHostComms(t *testing.T) {
|
|
runMultinodeSameSGCrossHostComms(t, requireMultiNodeFixture(t))
|
|
}
|
|
|
|
func TestMultinodeClusterHealth(t *testing.T) {
|
|
t.Parallel()
|
|
runClusterHealth(t, requireMultiNodeFixture(t))
|
|
}
|
|
|
|
// TestMultinodeDNS is sequential because it launches guests and briefly stops
|
|
// one Northstar unit while exercising resolver failover.
|
|
func TestMultinodeDNS(t *testing.T) {
|
|
runMultinodeDNS(t, requireMultiNodeFixture(t))
|
|
}
|
|
|
|
// TestMultinodeJetStreamReplicas is read-only over NATS; parallel so it resumes
|
|
// after the sequential node-failure/recovery tests have restabilised the cluster.
|
|
func TestMultinodeJetStreamReplicas(t *testing.T) {
|
|
t.Parallel()
|
|
runJetStreamReplicas(t, requireMultiNodeFixture(t))
|
|
}
|
|
|
|
// TestMultinodeVolumeDurability is sequential and declared before CrossNodeOps
|
|
// (which destabilises trio[0]'s sshd) so its guest-SSH probes hit settled VMs.
|
|
// Touches predastore state.
|
|
func TestMultinodeVolumeDurability(t *testing.T) {
|
|
runVolumeDurability(t, requireMultiNodeFixture(t))
|
|
}
|
|
|
|
// TestMultinodeCrossNodeGateway is sequential: asserts a stable per-gateway instance count
|
|
// that concurrent launches/terminates would break.
|
|
func TestMultinodeCrossNodeGateway(t *testing.T) {
|
|
runCrossNodeGateway(t, requireMultiNodeFixture(t))
|
|
}
|
|
|
|
// TestMultinodeInstanceStatusFanout is sequential and declared before
|
|
// CrossNodeOps (which stops trio[0]) so the trio is still spread across nodes
|
|
// when the fan-out spread is measured.
|
|
func TestMultinodeInstanceStatusFanout(t *testing.T) {
|
|
runInstanceStatusFanout(t, requireMultiNodeFixture(t))
|
|
}
|
|
|
|
// TestMultinodeCrossNodeOps is sequential: stops/starts trio[0].
|
|
func TestMultinodeCrossNodeOps(t *testing.T) {
|
|
runCrossNodeOps(t, requireMultiNodeFixture(t))
|
|
}
|
|
|
|
func TestMultinodeNodeFailure(t *testing.T) {
|
|
runNodeFailure(t, requireMultiNodeFixture(t))
|
|
}
|
|
|
|
func TestMultinodeNodeRecovery(t *testing.T) {
|
|
runNodeRecovery(t, requireMultiNodeFixture(t))
|
|
}
|
|
|
|
// TestMultinodeOVNRaft is sequential: stops ovn-central on the NB leader to
|
|
// prove DB failover, restoring it in cleanup before later tests run.
|
|
func TestMultinodeOVNRaft(t *testing.T) {
|
|
runOVNRaft(t, requireMultiNodeFixture(t))
|
|
}
|
|
|
|
// TestMultinodeFirewallChassisQuorum is sequential: it edits one node's config
|
|
// and restarts that node's daemon, restoring both before it returns.
|
|
func TestMultinodeFirewallChassisQuorum(t *testing.T) {
|
|
runFirewallChassisQuorum(t, requireMultiNodeFixture(t))
|
|
}
|
|
|
|
// TestMultinodeSpread is sequential after NodeRecovery; owns 10.100.0.0/16 + EIP pool.
|
|
func TestMultinodeSpread(t *testing.T) {
|
|
runSpread(t, requireMultiNodeFixture(t))
|
|
}
|
|
|
|
// TestMultinodeVPCNetworking owns 10.200.0.0/16 (no EIP use); safe to run in parallel.
|
|
func TestMultinodeVPCNetworking(t *testing.T) {
|
|
t.Parallel()
|
|
runVPCNetworking(t, requireMultiNodeFixture(t))
|
|
}
|
|
|
|
// TestMultinodeIPSec is read-only over SSH; launches no AWS resources.
|
|
func TestMultinodeIPSec(t *testing.T) {
|
|
t.Parallel()
|
|
runIPSec(t, requireMultiNodeFixture(t))
|
|
}
|