spinifex/docs/security/network-connections

title, description, category, sections, tags, resources

title	description	category	sections	tags	resources
External Connection Inventory	Operator inventory of inbound listeners and outbound connections on Spinifex nodes	Security	overview	security compliance cmmc network connections boundary	title url NIST SP 800-171 Rev 3 https://csrc.nist.gov/pubs/sp/800/171/r3/final title url CMMC Level 1 Self-Assessment Guide v2.0 https://dodcio.defense.gov/CMMC/Documentation title url NIST SP 800-41 Rev 1 — Guidelines on Firewalls and Firewall Policy https://csrc.nist.gov/pubs/sp/800/41/r1/final

External Connection Inventory

Operator inventory of inbound listeners and outbound connections on Spinifex nodes

Table of Contents

• Overview
• CMMC Practices Covered
• Approach
• 1. Inbound Listeners
• 2. Outbound Connections
• 3. Cross-Node (Internal) Connections
• 4. Limiting Controls
• 5. Configuration Surface
• 6. Operator Checklist

---

Overview

Audience: Operators deploying Spinifex into environments subject to CMMC Level 1, or any site that requires a documented inventory of system connections.

Scope: Network connections originated by or terminated at the Spinifex nodes — the Linux hosts running spinifex-daemon, spinifex-awsgw, spinifex-nats, spinifex-predastore, spinifex-viperblock, spinifex-vpcd, spinifex-ui, and the OVN control plane. Guest VM traffic is the workload owner's responsibility and out of scope.

Boundary definition. For the purposes of this document:

• External means outside the Spinifex cluster's trusted network perimeter — the public internet, the operator's corporate network, tenant users of the AWS API, and guest VMs.
• Internal means between Spinifex nodes inside the cluster subnet(s) defined in spinifex.toml.

AC.L1-3.1.20 applies specifically to external connections. Internal cluster connections are documented here as well so operators can build an accurate firewall policy.

CMMC Practices Covered

This guide addresses AC.L1-3.1.20. The related boundary-protection practice SC.L1-3.13.1 is covered by OVN ACL and security-group enforcement in vpcd and is documented separately.

Practice	Title	Objective
AC.L1-3.1.20	External Connections	[a] Connections to external systems are identified. [b] The use of external systems is identified. [c] Connections to external systems are verified. [d] The use of external systems is verified. [e] Connections to external systems are controlled/limited. [f] The use of external systems is controlled/limited.

Approach

Spinifex has a small, enumerable set of network surfaces:

1. Inbound listeners — the TCP/UDP ports each node binds. These are the attack surface exposed to whoever can reach the node.
2. Outbound connections — the destinations the node's services reach out to. Today this is a short list: peer Spinifex nodes, OS image mirrors, and install telemetry.
3. Cross-node connections — inter-node control- and data-plane traffic inside the cluster subnet.

The inventory in §1–§2 satisfies objectives [a]/[b]. The Auth / Verification columns throughout satisfy [c]/[d]. §4 and §5 satisfy [e]/[f]. The default install meets [c]–[f] for every listed connection; the operator's remaining work is to record the inventory in the system security plan, apply host/network firewall rules per §4, and audit §5 on a recurring schedule.

1. Inbound Listeners

"Scope" classifies intended reach:

• External — reachable by tenant/operator networks. Authenticated and TLS-protected.
• Cluster — reachable only from peer Spinifex nodes. Operator must restrict via host or network firewall.
• Localhost — bound to 127.0.0.1; not reachable off-node.

Port	Service	Protocol	Scope	Purpose	Auth / Verification
9999	spinifex-awsgw	HTTPS	External	AWS-compatible API (EC2, S3, ELBv2, IAM) — customer endpoint	AWS SigV4 + TLS (cluster CA)
3000	spinifex-ui	HTTPS	External	Operator web dashboard	Session cookie + TLS
22	OpenSSH	SSH	External	Operator administration	Key-based auth (operator-managed)
4432	Formation server	HTTPS	External (bootstrap only)	Cluster join coordination; active only while a join token is valid. See Formation port lifecycle below.	Short-lived bearer token + TLS¹
4222	spinifex-nats (client)	NATS + TLS	Cluster	Internal service bus for EC2/EBS/VPC/S3 handlers	Token + mutual TLS (cluster CA)
4248	spinifex-nats (cluster)	NATS + TLS	Cluster	Inter-node NATS federation	Token + mutual TLS (cluster CA)
8443	spinifex-predastore (gate)	HTTPS	Cluster	S3-compatible object storage (AMIs, snapshots, user objects)	AWS SigV4 + TLS
6660	predastore (blob node)	QUIC / UDP	Cluster	Erasure-coded object shard transport between hosts. Multi-node clusters only — see Predastore ports below.	TLS 1.3, server certificate verified against the cluster CA
7660	predastore (meta node)	QUIC / UDP	Cluster	Raft consensus over global state — buckets and the object index — between hosts. Multi-node clusters only.	TLS 1.3, server certificate verified against the cluster CA
6641	OVN Northbound DB (client)	OVSDB/TCP	Cluster	Logical network topology consumed by vpcd	Cluster network only; TLS planned
6642	OVN Southbound DB (client)	OVSDB/TCP	Cluster	Chassis / port / MAC binding state	Cluster network only; TLS planned
6643	OVN Northbound DB (RAFT)	OVSDB/TCP	Cluster	NB database RAFT replication between the 3 quorum nodes	Cluster network only; TLS planned
6644	OVN Southbound DB (RAFT)	OVSDB/TCP	Cluster	SB database RAFT replication between the 3 quorum nodes	Cluster network only; TLS planned
8222	spinifex-nats (monitoring)	HTTP	Localhost	varz/subsz metrics consumed by the daemon	Loopback only
socket / dynamic TCP	nbdkit (Viperblock)	NBD	Host-local / cluster	Block device transport for guest EBS volumes	Unix socket by default; TCP only in remote/DPU mode

¹ Formation port lifecycle. 4432 opens during spx admin init / spx admin join while a bootstrap token is outstanding and closes once the cluster is formed (token TTL default 30 min, --token-ttl). The server presents an ephemeral self-signed cert that pre-dates trust bootstrap, so the joining node does not verify the certificate chain for this single dial. Authenticity rests on the operator supplying the leader address out-of-band plus possession of the bearer token. Document in the security plan so reviewers do not flag 4432 as a persistent open port.

Predastore ports. A Predastore cluster is described in /etc/spinifex/predastore/predastore.toml as a set of [[host]] blocks — one per machine, each running a single process — with the nodes pinned to it declared under [[host.node]]. There are three roles: a gate serving the S3 API, a blob node holding erasure-coded object shards, and a meta node in the Raft quorum over global state. Ports must be unique within a host but are not unique across the cluster, so every machine uses the same three ports — 8443, 6660 and 7660. It is three fixed ports per machine, not a range, and adding machines does not widen it.

Nodes on the same host talk over an in-process pipe and bind no socket at all, so a single-node install opens only 8443; 6660 and 7660 appear only once a second host exists. The gate is dialled by S3 clients but never by peer nodes, so it binds no QUIC socket of its own — it takes an ephemeral UDP port to dial out with.

Development-only listeners. When dev_networking=true, QEMU opens arbitrary host TCP ports for SSH port-forwarding into guest VMs. Production installs (the /etc/spinifex layout) do not enable this; it must not appear on compliance nodes.

2. Outbound Connections

Spinifex nodes initiate a small, fixed set of outbound connections.

To external destinations:

Destination	Purpose	Protocol	Verification
https://cloud.debian.org/images/cloud/trixie/latest/	Debian 13 cloud image download	HTTPS	TLS + checksum verification
https://cloud-images.ubuntu.com/releases/resolute/release/	Ubuntu 26.04 LTS cloud image download	HTTPS	TLS + checksum verification
https://dl.rockylinux.org/pub/rocky/10/images/	Rocky Linux 10 cloud image download	HTTPS	TLS + checksum verification
https://dl-cdn.alpinelinux.org/alpine/	Alpine Linux cloud image download	HTTPS	TLS + checksum verification
https://d2yp8ipz5jfqcw.cloudfront.net	Alpine image for managed HAProxy load-balancer	HTTPS	TLS + checksum verification
https://install.mulgadc.com/install	One-shot install telemetry POST on spx admin init / join.	HTTPS	TLS

To peer nodes (cluster-internal): NATS federation (4248), Predastore S3 (8443), OVN NB/SB (6641/6642) — see §3 for encryption and verification of each. The daemon also polls local NATS monitoring at 127.0.0.1:8222/varz (loopback HTTP). It opens no connection to Predastore for status: the storage topology it reports comes from reading predastore.toml, and no Predastore node serves a status endpoint.

Update checks and metadata. Spinifex does not check for updates and does not consume a cloud metadata service (169.254.169.254 is served by the cluster to guest VMs). Node software updates come from the operator's OS package channel. The install-telemetry endpoint above is the only vendor-operated destination contacted by a node; closed-egress deployments should disable it and record the opt-out in the security plan.

Air-gapped deployments. The image URLs above are the only destinations needed for the standard image catalogue. Mirror them locally and use spx admin images import --file with pre-staged files. Telemetry must also be disabled. See Air-Gapped Install.

3. Cross-Node (Internal) Connections

Control-plane and data-plane traffic between Spinifex nodes, for completeness and firewall planning:

Connection	Port(s)	Encryption / Auth	Notes
NATS cluster routes	4248	Mutual TLS + cluster token	Full mesh between NATS servers
Predastore S3 (gate)	TCP 8443	TLS + AWS SigV4	Cross-node object reads/writes
Predastore blob	UDP 6660	QUIC with TLS 1.3; server certificate verified against the cluster CA	Erasure-coded object shards. Same port on every machine.
Predastore meta	UDP 7660	QUIC with TLS 1.3; server certificate verified against the cluster CA	Raft consensus over buckets and the object index. Same port on every machine.
OVN NB/SB (client)	6641 / 6642	Cluster network only (TLS planned)	Network control plane; vpcd and ovn-controller dial the quorum
OVN NB/SB (RAFT)	6643 / 6644	Cluster network only (TLS planned)	NB/SB database replication across the 3 quorum nodes
OVN tunnels (Geneve)	UDP 6081	None	Tenant traffic overlay between chassis, inside the cluster subnet

Nodes must sit on a network segment that is not routed to tenant/guest VMs or to the internet. The Predastore blob and meta transports and the OVN DBs are cluster-internal and must not be reachable from anywhere else.

4. Limiting Controls

Default external surface is three listeners — 9999 (AWS API), 3000 (UI), 22 (SSH) — plus 4432 transiently during bootstrap. Every other listener is cluster-only and the operator must enforce this with a host firewall (nftables/iptables/firewalld) or an upstream network ACL. Minimal nftables reference:

# External: from anywhere
tcp dport { 22, 9999, 3000 } accept

# Cluster-only: replace 10.0.1.0/24 with your cluster CIDR
ip saddr 10.0.1.0/24 tcp dport { 4222, 4248, 8443, 6641-6644 } accept
ip saddr 10.0.1.0/24 udp dport { 6081, 6660, 7660 } accept

# Default deny
tcp dport 0-65535 drop
udp dport 0-65535 drop

Port 4432 must be closed outside the bootstrap window; spx admin join opens it transiently. Outbound egress can be limited to the image-catalogue hostnames in §2 plus the operator's OS package repositories; on air-gapped nodes, block all outbound HTTPS and use spx admin images import --file.

5. Configuration Surface

Every listener and outbound destination is controlled by one of these files. Changes require a service restart.

File	Keys	Controls
/etc/spinifex/spinifex.toml	nodes.<node>.{awsgw,nats,predastore,daemon}.host, nodes.<node>.vpcd.ovn_{nb,sb}_addr, nodes.<node>.daemon.dev_networking	Per-service bind addresses/ports; dev-mode QEMU port forwarding.
/etc/spinifex/nats.conf	listen, cluster.listen, cluster.routes, http, tls, cluster.authorization	NATS client/cluster/monitoring listeners, peer routes, TLS, cluster token.
/etc/spinifex/predastore/predastore.toml	[[host]].bind_addr, [[host]].addr, [[host]].tls_cert, [[host]].tls_key, [[host.node]].role, [[host.node]].port	Predastore host and node layout: bind_addr is the address the host's sockets bind, addr is the address peer hosts dial it on, and both carry no port — the nodes pinned to the host supply their own. The service's --host/--port override the bind address and the gate's S3 port.
OVN packages (ovn-central, ovn-host)	ovn-nb-db, ovn-sb-db (via ovs-vsctl set open_vswitch …)	OVN DB bind addresses.
Spinifex UI service	Built-in defaults: host = "0.0.0.0", port = 3000. No spinifex.toml block today.	UI listener.
spx admin init / spx admin join	--port, --token-ttl, --no-telemetry (or SPX_NO_TELEMETRY=1)	Formation port, token TTL, telemetry opt-out.
Image catalogue (built-in)	Fixed URLs listed in §2; not operator-configurable	Outbound HTTPS destinations for image downloads.

6. Operator Checklist

• Inventory recorded in the system security plan — inbound (§1), outbound (§2), cross-node (§3) — matches what is observed on the node (ss -tlnp, ss -unlp).
• Host firewall enforces the external/cluster/localhost split in §4: external surface limited to 9999, 3000, 22 (and 4432 only during bootstrap).
• Cluster subnet is isolated from tenant guest VM networks and from the public internet.
• Formation port 4432 is closed on nodes not actively running a bootstrap token.
• Outbound HTTPS restricted to the §2 image-catalogue hosts, or replaced with air-gapped import.
• Install telemetry (install.mulgadc.com) is either permitted and recorded in the security plan, or disabled via SPX_NO_TELEMETRY=1 / --no-telemetry.
• OVN NB/SB client and RAFT ports (6641–6644) exposure limited to the cluster subnet pending the L2 TLS work.
• SSH (22) configured to operator-managed keys only; password auth disabled in sshd_config.
• Periodic review (at least annually, and after any topology change) confirms this inventory still matches the deployed configuration.