* fix(e2e): resolve the listener inventory without a Go toolchain on the target node
tests/e2e/multinode/listener_invariant_test.go resolved
docs/security/network-connections/README.md at runtime via `go env
GOMOD`, which works under `go test` but not inside a compiled .test
binary shipped to a node with no Go toolchain. Embed the doc in a new
docs/security/network-connections package instead, so the content is
compiled into the binary and every caller parses bytes rather than a
resolved path. RepoRoot and DocPath are now unused and removed.
* docs: trim the embed package comment to the three-line standard
* test: add the listener invariant static guard and e2e runtime check
Adds both halves the plan called for: a static test in
spinifex/network/invariants that scans install scripts and config
templates for a bare OVSDB ptcp remote, a wildcard bind on a
Cluster/Encap port, or a bind port absent from the inventory, and an
e2e test in tests/e2e/multinode that asserts the same on live ss
-tulnp output. Both read
docs/security/network-connections/README.md "## 1. Inbound
Listeners" as their fixture through a shared, non-test parser package
(spinifex/network/listenerinventory), so a wildcard exception has to
be declared in the doc's own prose rather than hardcoded in either
test.
* fix: make the listener wildcard exception fail closed on an explicit marker
The prior heuristic granted an exception for any unnegated "wildcard"
or "0.0.0.0" mention within 40 characters of the port's row. That
fails open: a future row phrased as "avoids the wildcard" or with a
negation further upstream than the window would silently exempt
itself. Replace it with a single exact literal phrase, "binds the
wildcard by design", that a row must contain to grant the exception —
no other wording counts, so an unrecognized mention denies rather
than grants. Update the two rows that need it (5300, 500/4500) and
document the marker's role in the inventory's own prose.
* docs: record that the Geneve tunnel socket binds the wildcard by design
The inventory was missing northstar (53 on the advertise address, 5300 on the
wildcard and dialled cross-node by every node's per-instance DNS shim), charon
(500/4500 plus ESP), Geneve 6081, the per-instance IMDS and VPC-DNS listeners,
and chronyd. Section 6 asks the operator to compare the table against ss -tulnp,
which was not possible while six listeners were absent.
Two classifications were wrong. 8443 was Cluster, but predastore binds the S3
gate to 0.0.0.0 deliberately because S3 is a public plane. 4432 was External,
but it binds the node's --bind address and is transient.
The scope vocabulary predated the three-plane model, so it could not express the
encap plane or record that a node with fewer NICs collapses the planes onto its
public address. Cluster and Encap now state intent rather than a guarantee.
Replaces the reference nftables ruleset. The previous one had no conntrack,
loopback, ICMP or ime-* rules and ended in a blanket drop, so an operator who
applied it lost guest metadata, guest DNS, all egress return traffic and their
own SSH session. The replacement is input-only in a dedicated table, because
nftables verdicts are per-table and a default-deny forward chain would override
the accepts vpcd writes to ip filter and break every routed-NAT instance.
setup-ovn.sh set the OVN Northbound and Southbound client remotes with a bare
set-connection ptcp:6641 / ptcp:6642. The address-free ptcp: form binds every
interface, so on a multi-plane node the unauthenticated, unencrypted OVSDB
control plane was reachable from the public interface. Write access to the
Southbound database is control of the whole tenant network fabric.
ovsdb-server takes remotes from two additive sources: the Connection table,
which RAFT replicates cluster-wide, and local ovn-ctl command-line flags. That
split is what the fix uses. Loopback goes in the replicated row, because
127.0.0.1 is correct on every node, so replication is harmless and local
clients keep working. The lan address goes on the per-node command line via
--db-nb-addr and --db-nb-create-insecure-remote, which never replicates, so
each node binds its own address without pushing it to its peers.
A new --lan-addr flag carries that address, falling back to
--db-cluster-local-addr so clustered nodes stay correct without caller changes.
The standalone branch now writes /etc/default/ovn-central too, since a two-node
install still has a remote ovn-controller that needs the lan listener. The
guard on --db-nb-addr is load-bearing: it defaults to 0.0.0.0, so emitting
create-insecure-remote without a resolved address would recreate the bug.
install-node.sh passes the lan address on both branches and warns when the lan
plane has collapsed onto the public address, since the listener then lands on
the public interface and a firewall is the only remaining control.
The port tables were wrong in both directions: inter-node traffic is QUIC over
UDP rather than TCP, and the shard range 9991-9993 no longer exists. Ports are
unique within a host rather than across the cluster, so every machine uses the
same three, and a single-node install opens only the S3 port because colocated
nodes talk over an in-process pipe.
The reference architectures scripted a move of node data out of a directory
layout that no longer exists, which would have broken any install following
them. The update guide now says object storage has no migration path.
* match new predastore config surface area
* fix(migrate): carry existing installs onto the predastore v5 topology
The cluster topology refactor breaks an install three ways at once. The old
config carries a top-level host key, which the new schema binds to the [[host]]
table array, so TOML rejects the document and the daemon dies before it reads
any topology. Node state moved from distributed/{db,nodes}/node-N to a per-host
data directory keyed by a node id that is now unique across roles. And because
object metadata records which node holds each shard, the shard nodes had to
keep their ids, so the state replicas took new ones and every replica's
persisted raft configuration stopped listing it.
predastore.toml gains a 4 to 5 migration covering all three: it rewrites the
tables, relocates whichever node directories exist on this machine, and
recovers each relocated state replica onto the new member set. The rewrite is
line-oriented rather than a re-render from the template, so buckets, auth,
rate limits and iam settings an operator edited survive.
spinifex.toml gains a 3 to 4 migration renaming the predastore node_id to
host_id, which predastore-start.sh greps to select the host this machine runs.
The predastore migration lives in its own package rather than in migrate,
which every handler that stamps a KV bucket version imports; reaching the
cluster runtime from there would pull raft, badger and the QUIC client into all
of them. admin_upgrade.go blank-imports it, so only the binary that runs the
upgrade links it, and a test pins both registrations.
* test(predastore): cover the service runtime, gateway and shutdown paths
The service wrapper's new seams had no unit tests: buildRuntime, serve and
the errgroup Start runs the cluster and the S3 gateway under. CI's diff
coverage gate measured the PR at 34.6% against a 60% threshold as a result.
Cover every buildRuntime error wrap and its host selection, both ways the
gateway can fail serve, and the full lifecycle: Start brings the nodes and
the gateway up together, Shutdown drains both by cancelling the context
they share. Topologies are written per test as real config, with node ids
from a process-wide counter — the pipe transport's listener registry is
global, and this package's integration tests hold a fixture daemon on the
low ids for the life of the test binary.
* feat: add OVSDB RAFT cluster primitives to setup-ovn.sh
Add --db-cluster-local-addr / --db-cluster-remote-addr flags so a
management node can run the OVN NB/SB databases as a native OVSDB RAFT
cluster instead of a single non-clustered ovn-central. When a local addr
is given, write a shared OVN_CTL_OPTS to /etc/default/ovn-central (NB
RAFT 6643 / SB RAFT 6644) and restart the per-DB units directly, since
the ovn-central aggregator is ExecStart=/bin/true and won't restart its
children. Empty remote addr creates the cluster (init node); a set remote
addr joins it. NB/SB client set-connection runs only on the create node
(RAFT replicates it cluster-wide).
The single-node --management path is unchanged when no cluster addrs are
supplied, so dev boxes and existing single-node clusters keep working.
--ovn-remote already passes through verbatim, so a comma-separated SB
endpoint list works for compute nodes; documented in usage.
* feat: OVN client multi-endpoint failover for RAFT clusters
Accept a comma-separated OVN NB/SB endpoint list so vpcd can target a
clustered DB. config.ParseEndpoints splits the list (trimming whitespace,
dropping empties); live.go Connect passes one libovsdb WithEndpoint per
endpoint, so the client tries them in order, fails over on disconnect, and
follows RAFT leader redirects. Single-endpoint strings still yield one
endpoint, so single-node dev is unchanged.
ovn-sbctl --db= already accepts the raw comma list, so the SB CLI
consumers (discoverChassis, gateway_claim) need no change.
* feat: write clustered OVN NB/SB endpoints across the RAFT quorum
spx admin init/join now emit comma-separated OVN NB/SB endpoint lists for
the first three quorum nodes (formation.BuildOVNDBAddrs) instead of loopback
or a single leader address, so vpcd fails over across the cluster.
firstboot defers OVN bring-up to the provisioning controller when it owns
formation (SkipFormation): a standalone single-node .db would otherwise block
the controller's clustered create/join.
* test: add multinode OVN RAFT cluster + failover e2e; document clustered topology
ovn_raft_test.go asserts NB/SB cluster/status reports a 3-server quorum with
one leader on every DB node, then stops the NB leader's ovn-central and proves
an NB write still lands via a surviving endpoint.
Docs: vpc-networking gains the multi-endpoint vpcd config and a RAFT
cluster-status debugging section; network-connections lists the 6643/6644 RAFT
ports.
Josh Siviour