Bryn Mailer
c2b80a9b93
* match new predastore config surface area
* fix(migrate): carry existing installs onto the predastore v5 topology
The cluster topology refactor breaks an install three ways at once. The old
config carries a top-level host key, which the new schema binds to the [[host]]
table array, so TOML rejects the document and the daemon dies before it reads
any topology. Node state moved from distributed/{db,nodes}/node-N to a per-host
data directory keyed by a node id that is now unique across roles. And because
object metadata records which node holds each shard, the shard nodes had to
keep their ids, so the state replicas took new ones and every replica's
persisted raft configuration stopped listing it.
predastore.toml gains a 4 to 5 migration covering all three: it rewrites the
tables, relocates whichever node directories exist on this machine, and
recovers each relocated state replica onto the new member set. The rewrite is
line-oriented rather than a re-render from the template, so buckets, auth,
rate limits and iam settings an operator edited survive.
spinifex.toml gains a 3 to 4 migration renaming the predastore node_id to
host_id, which predastore-start.sh greps to select the host this machine runs.
The predastore migration lives in its own package rather than in migrate,
which every handler that stamps a KV bucket version imports; reaching the
cluster runtime from there would pull raft, badger and the QUIC client into all
of them. admin_upgrade.go blank-imports it, so only the binary that runs the
upgrade links it, and a test pins both registrations.
* test(predastore): cover the service runtime, gateway and shutdown paths
The service wrapper's new seams had no unit tests: buildRuntime, serve and
the errgroup Start runs the cluster and the S3 gateway under. CI's diff
coverage gate measured the PR at 34.6% against a 60% threshold as a result.
Cover every buildRuntime error wrap and its host selection, both ways the
gateway can fail serve, and the full lifecycle: Start brings the nodes and
the gateway up together, Shutdown drains both by cancelling the context
they share. Topologies are written per test as real config, with node ids
from a process-wide counter — the pipe transport's listener registry is
global, and this package's integration tests hold a fixture daemon on the
low ids for the life of the test binary.
title, description, category, sections, tags, resources
| title | description | category | sections | tags | resources | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| External Connection Inventory | Operator inventory of inbound listeners and outbound connections on Spinifex nodes | Security |
|
|
|
External Connection Inventory
Operator inventory of inbound listeners and outbound connections on Spinifex nodes
Table of Contents
- Overview
- CMMC Practices Covered
- Approach
- 1. Inbound Listeners
- 2. Outbound Connections
- 3. Cross-Node (Internal) Connections
- 4. Limiting Controls
- 5. Configuration Surface
- 6. Operator Checklist
Overview
Audience: Operators deploying Spinifex into environments subject to CMMC Level 1, or any site that requires a documented inventory of system connections.
Scope: Network connections originated by or terminated at the Spinifex nodes — the Linux hosts running spinifex-daemon, spinifex-awsgw, spinifex-nats, spinifex-predastore, spinifex-viperblock, spinifex-vpcd, spinifex-ui, and the OVN control plane. Guest VM traffic is the workload owner's responsibility and out of scope.
Boundary definition. For the purposes of this document:
- External means outside the Spinifex cluster's trusted network perimeter — the public internet, the operator's corporate network, tenant users of the AWS API, and guest VMs.
- Internal means between Spinifex nodes inside the cluster subnet(s) defined in
spinifex.toml.
AC.L1-3.1.20 applies specifically to external connections. Internal cluster connections are documented here as well so operators can build an accurate firewall policy.
CMMC Practices Covered
This guide addresses AC.L1-3.1.20. The related boundary-protection practice SC.L1-3.13.1 is covered by OVN ACL and security-group enforcement in vpcd and is documented separately.
| Practice | Title | Objective |
|---|---|---|
| AC.L1-3.1.20 | External Connections | [a] Connections to external systems are identified. [b] The use of external systems is identified. [c] Connections to external systems are verified. [d] The use of external systems is verified. [e] Connections to external systems are controlled/limited. [f] The use of external systems is controlled/limited. |
Approach
Spinifex has a small, enumerable set of network surfaces:
- Inbound listeners — the TCP/UDP ports each node binds. These are the attack surface exposed to whoever can reach the node.
- Outbound connections — the destinations the node's services reach out to. Today this is a short list: peer Spinifex nodes, OS image mirrors, and install telemetry.
- Cross-node connections — inter-node control- and data-plane traffic inside the cluster subnet.
The inventory in §1–§2 satisfies objectives [a]/[b]. The Auth / Verification columns throughout satisfy [c]/[d]. §4 and §5 satisfy [e]/[f]. The default install meets [c]–[f] for every listed connection; the operator's remaining work is to record the inventory in the system security plan, apply host/network firewall rules per §4, and audit §5 on a recurring schedule.
1. Inbound Listeners
"Scope" classifies intended reach:
- External — reachable by tenant/operator networks. Authenticated and TLS-protected.
- Cluster — reachable only from peer Spinifex nodes. Operator must restrict via host or network firewall.
- Localhost — bound to
127.0.0.1; not reachable off-node.
| Port | Service | Protocol | Scope | Purpose | Auth / Verification |
|---|---|---|---|---|---|
| 9999 | spinifex-awsgw | HTTPS | External | AWS-compatible API (EC2, S3, ELBv2, IAM) — customer endpoint | AWS SigV4 + TLS (cluster CA) |
| 3000 | spinifex-ui | HTTPS | External | Operator web dashboard | Session cookie + TLS |
| 22 | OpenSSH | SSH | External | Operator administration | Key-based auth (operator-managed) |
| 4432 | Formation server | HTTPS | External (bootstrap only) | Cluster join coordination; active only while a join token is valid. See Formation port lifecycle below. | Short-lived bearer token + TLS¹ |
| 4222 | spinifex-nats (client) | NATS + TLS | Cluster | Internal service bus for EC2/EBS/VPC/S3 handlers | Token + mutual TLS (cluster CA) |
| 4248 | spinifex-nats (cluster) | NATS + TLS | Cluster | Inter-node NATS federation | Token + mutual TLS (cluster CA) |
| 8443 | spinifex-predastore | HTTPS | Cluster | S3-compatible object storage (AMIs, snapshots, user objects) | AWS SigV4 + TLS |
| 6660–6662 | predastore (Raft) | TCP | Cluster | Metadata consensus (3 nodes) | Cluster network only |
| 9991–9993 | predastore (data shards) | TCP | Cluster | Erasure-coded data shard transport | Cluster network only |
| 6641 | OVN Northbound DB (client) | OVSDB/TCP | Cluster | Logical network topology consumed by vpcd | Cluster network only; TLS planned |
| 6642 | OVN Southbound DB (client) | OVSDB/TCP | Cluster | Chassis / port / MAC binding state | Cluster network only; TLS planned |
| 6643 | OVN Northbound DB (RAFT) | OVSDB/TCP | Cluster | NB database RAFT replication between the 3 quorum nodes | Cluster network only; TLS planned |
| 6644 | OVN Southbound DB (RAFT) | OVSDB/TCP | Cluster | SB database RAFT replication between the 3 quorum nodes | Cluster network only; TLS planned |
| 8222 | spinifex-nats (monitoring) | HTTP | Localhost | varz/subsz metrics consumed by the daemon |
Loopback only |
| socket / dynamic TCP | nbdkit (Viperblock) | NBD | Host-local / cluster | Block device transport for guest EBS volumes | Unix socket by default; TCP only in remote/DPU mode |
¹ Formation port lifecycle. 4432 opens during spx admin init / spx admin join while a bootstrap token is outstanding and closes once the cluster is formed (token TTL default 30 min, --token-ttl). The server presents an ephemeral self-signed cert that pre-dates trust bootstrap, so the joining node does not verify the certificate chain for this single dial. Authenticity rests on the operator supplying the leader address out-of-band plus possession of the bearer token. Document in the security plan so reviewers do not flag 4432 as a persistent open port.
Development-only listeners. When dev_networking=true, QEMU opens arbitrary host TCP ports for SSH port-forwarding into guest VMs. Production installs (the /etc/spinifex layout) do not enable this; it must not appear on compliance nodes.
2. Outbound Connections
Spinifex nodes initiate a small, fixed set of outbound connections.
To external destinations:
| Destination | Purpose | Protocol | Verification |
|---|---|---|---|
https://cloud.debian.org/images/cloud/trixie/latest/ |
Debian 13 cloud image download | HTTPS | TLS + checksum verification |
https://cloud-images.ubuntu.com/releases/resolute/release/ |
Ubuntu 26.04 LTS cloud image download | HTTPS | TLS + checksum verification |
https://dl.rockylinux.org/pub/rocky/10/images/ |
Rocky Linux 10 cloud image download | HTTPS | TLS + checksum verification |
https://dl-cdn.alpinelinux.org/alpine/ |
Alpine Linux cloud image download | HTTPS | TLS + checksum verification |
https://d2yp8ipz5jfqcw.cloudfront.net |
Alpine image for managed HAProxy load-balancer | HTTPS | TLS + checksum verification |
https://install.mulgadc.com/install |
One-shot install telemetry POST on spx admin init / join. |
HTTPS | TLS |
To peer nodes (cluster-internal): NATS federation (4248), Predastore S3 (8443), OVN NB/SB (6641/6642) — see §3 for encryption and verification of each. The daemon also probes local Predastore Raft status at <bind IP>:6660/status (TLS, cluster CA) and local NATS monitoring at 127.0.0.1:8222/varz (loopback HTTP).
Update checks and metadata. Spinifex does not check for updates and does not consume a cloud metadata service (169.254.169.254 is served by the cluster to guest VMs). Node software updates come from the operator's OS package channel. The install-telemetry endpoint above is the only vendor-operated destination contacted by a node; closed-egress deployments should disable it and record the opt-out in the security plan.
Air-gapped deployments. The image URLs above are the only destinations needed for the standard image catalogue. Mirror them locally and use spx admin images import --file with pre-staged files. Telemetry must also be disabled. See Air-Gapped Install.
3. Cross-Node (Internal) Connections
Control-plane and data-plane traffic between Spinifex nodes, for completeness and firewall planning:
| Connection | Port(s) | Encryption / Auth | Notes |
|---|---|---|---|
| NATS cluster routes | 4248 | Mutual TLS + cluster token | Full mesh between NATS servers |
| Predastore S3 | 8443 | TLS + AWS SigV4 | Cross-node object reads/writes |
| Predastore Raft | 6660–6662 | Cluster network only | Metadata consensus |
| Predastore shards | 9991–9993 | Cluster network only | Erasure-coded data shards |
| OVN NB/SB (client) | 6641 / 6642 | Cluster network only (TLS planned) | Network control plane; vpcd and ovn-controller dial the quorum |
| OVN NB/SB (RAFT) | 6643 / 6644 | Cluster network only (TLS planned) | NB/SB database replication across the 3 quorum nodes |
| OVN tunnels (Geneve) | UDP 6081 | None | Tenant traffic overlay between chassis, inside the cluster subnet |
Nodes must sit on a network segment that is not routed to tenant/guest VMs or to the internet. Predastore Raft/shards and OVN DBs are cluster-internal and must not be reachable from anywhere else.
4. Limiting Controls
Default external surface is three listeners — 9999 (AWS API), 3000 (UI), 22 (SSH) — plus 4432 transiently during bootstrap. Every other listener is cluster-only and the operator must enforce this with a host firewall (nftables/iptables/firewalld) or an upstream network ACL. Minimal nftables reference:
# External: from anywhere
tcp dport { 22, 9999, 3000 } accept
# Cluster-only: replace 10.0.1.0/24 with your cluster CIDR
ip saddr 10.0.1.0/24 tcp dport { 4222, 4248, 8443, 6641-6644, 6660-6662, 9991-9993 } accept
ip saddr 10.0.1.0/24 udp dport 6081 accept
# Default deny
tcp dport 0-65535 drop
udp dport 0-65535 drop
Port 4432 must be closed outside the bootstrap window; spx admin join opens it transiently. Outbound egress can be limited to the image-catalogue hostnames in §2 plus the operator's OS package repositories; on air-gapped nodes, block all outbound HTTPS and use spx admin images import --file.
5. Configuration Surface
Every listener and outbound destination is controlled by one of these files. Changes require a service restart.
| File | Keys | Controls |
|---|---|---|
/etc/spinifex/spinifex.toml |
nodes.<node>.{awsgw,nats,predastore,daemon}.host, nodes.<node>.vpcd.ovn_{nb,sb}_addr, nodes.<node>.daemon.dev_networking |
Per-service bind addresses/ports; dev-mode QEMU port forwarding. |
/etc/spinifex/nats.conf |
listen, cluster.listen, cluster.routes, http, tls, cluster.authorization |
NATS client/cluster/monitoring listeners, peer routes, TLS, cluster token. |
/etc/spinifex/predastore.toml |
[[host]].bind_addr, [[host]].public_addr, [[node]].role, tls.* |
Predastore intra-cluster listener (one socket per host, carrying every node pinned to it), node placement, TLS certs. The S3 listener itself is set by the service's --host/--port. |
OVN packages (ovn-central, ovn-host) |
ovn-nb-db, ovn-sb-db (via ovs-vsctl set open_vswitch …) |
OVN DB bind addresses. |
| Spinifex UI service | Built-in defaults: host = "0.0.0.0", port = 3000. No spinifex.toml block today. |
UI listener. |
spx admin init / spx admin join |
--port, --token-ttl, --no-telemetry (or SPX_NO_TELEMETRY=1) |
Formation port, token TTL, telemetry opt-out. |
| Image catalogue (built-in) | Fixed URLs listed in §2; not operator-configurable | Outbound HTTPS destinations for image downloads. |
6. Operator Checklist
- Inventory recorded in the system security plan — inbound (§1), outbound (§2), cross-node (§3) — matches what is observed on the node (
ss -tlnp,ss -unlp). - Host firewall enforces the external/cluster/localhost split in §4: external surface limited to 9999, 3000, 22 (and 4432 only during bootstrap).
- Cluster subnet is isolated from tenant guest VM networks and from the public internet.
- Formation port 4432 is closed on nodes not actively running a bootstrap token.
- Outbound HTTPS restricted to the §2 image-catalogue hosts, or replaced with air-gapped import.
- Install telemetry (
install.mulgadc.com) is either permitted and recorded in the security plan, or disabled viaSPX_NO_TELEMETRY=1/--no-telemetry. - OVN NB/SB client and RAFT ports (6641–6644) exposure limited to the cluster subnet pending the L2 TLS work.
- SSH (22) configured to operator-managed keys only; password auth disabled in
sshd_config. - Periodic review (at least annually, and after any topology change) confirms this inventory still matches the deployed configuration.