gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-15 09:01:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: separate the install guides from the reference material they had absorbed

Browse Source 
Both install guides had grown sections that were not install steps, which is what made the multi-node one hard to follow. Each now ends on a step that hands off to Setting Up Your Cluster, and the reference material moves under Administration where it can be read on its own.

- Single-node and multi-node both end on a "Set Up Your Cluster" step rather than a trailing sentence.
- The multi-node cluster-state check via `spx admin kv digest` is gone. Verification is now every node Ready, capacity pooled, and the AWS API answering.
- "How Multi-Node Storage Works" becomes docs/admin/predastore-distributed-storage, expanded with the host and node model, the write and read paths, degraded writes, hinted handoff, repair, encryption at rest, and what each class of failure costs.
- "Firewall and Cluster Membership" becomes docs/admin/host-firewall, expanded with the full public, cluster and encap port groups, how to narrow SSH, and which files under /etc/spinifex/firewall survive an upgrade.
- Setting Up Your Cluster gains a closing step that arms the firewall and narrows SSH, so the securing work lands after a cluster that works rather than during the install.

install-usb and vpc-networking followed the two moved sections to their new paths.
This commit is contained in:
Ben Duncan
2026-09-18 10:58:25 -05:00
parent e035842503
commit 2ba8f5e4ab
7 changed files with 517 additions and 118 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/admin/host-firewall/README.md
+233
View File
@@ -0,0 +1,233 @@
---
title: "Host Firewall"
seoTitle: "Host Firewall and Cluster Membership — Spinifex Docs"
description: "Arm the Spinifex host firewall, narrow SSH to your management networks, and learn how the peer-scoped cluster plane tracks membership changes on its own."
category: "Admin"
tags:
- security
- firewall
- nftables
- hardening
resources:
- title: "Network Connections"
url: "/docs/network-connections"
- title: "Multi-Node Install"
url: "/docs/install-multi-node"
- title: "Setting Up Your Cluster"
url: "/docs/setting-up-your-cluster"
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
---
# Host Firewall
> Arm the host firewall, narrow SSH, and keep the cluster plane off the public internet.
## Table of Contents
- [Overview](#overview)
- [What the policy allows](#what-the-policy-allows)
- [Cluster membership](#cluster-membership)
- [Is it on?](#is-it-on)
- [Narrowing SSH access](#narrowing-ssh-access)
- [Turning it off and on around cluster changes](#turning-it-off-and-on-around-cluster-changes)
- [The files](#the-files)
- [Checking it](#checking-it)
- [Troubleshooting](#troubleshooting)
---
## Overview
Spinifex ships an optional host firewall — an nftables input policy that defaults to drop and divides the node's ports into two groups:
| Group | Ports | Who can reach them |
|---|---|---|
| **Public** | SSH, 443, 3000 (console), 8443 (S3), 9999 (AWS gateway), 53 (DNS) | anyone |
| **Internal** | OVN, NATS, formation, Geneve and the rest of the cluster plane | **cluster members only** |
The internal group is the point. Before this existed, OVN and NATS were reachable from the public internet on a WAN-facing node.
**"Cluster members" is not a list you maintain.** Each node works it out from the cluster it belongs to and rewrites its own rules whenever membership changes — you never edit the peer list by hand.
The policy governs inbound traffic only. Outbound is untouched, because the metadata reply path egresses under per-tap policy routing and a filter there would break it.
## Instructions
## What the Policy Allows
The public plane is open to any source, IPv4 only — an accept with no source scope would otherwise also serve on any global IPv6 address the node picks up from an upstream router advertisement:
| Ports | Serving |
|---|---|
| TCP 443 | Held for the TLS edge; no listener yet, so the rollout needs no firewall change |
| TCP 3000 | Web console |
| TCP 8443 | Predastore S3 gate |
| TCP 9999 | AWS gateway — also how every guest agent (load balancer, EKS, ECS, RDS) reaches its control plane, since they speak SigV4 rather than NATS |
| TCP and UDP 53 | Northstar authoritative DNS. Open on every node deliberately: a node serving no public zone still answers here, which is the accepted cost of not templating the policy per node |
SSH is public too, but scoped by a variable you own — see [Narrowing SSH access](#narrowing-ssh-access).
The cluster plane is peer-scoped. On a single-NIC node the planes collapse onto one address, which is why these rules match peer addresses rather than an interface or a CIDR:
| Ports | Serving |
|---|---|
| TCP 4222, 4248 | NATS client and cluster routes |
| TCP 4432 | Formation, then the daemon cluster manager |
| TCP and UDP 5300 | Internal DNS |
| TCP 6641, 6642 | OVN northbound and southbound |
| TCP 6643, 6644 | OVN database clustering |
| TCP 8660 | Predastore's admin listener |
| UDP 6660, 7660 | Object shards and metadata consensus |
| UDP 6081, 500, 4500, and ESP | Geneve, IKE, NAT-T and IPsec |
Two of those are peer-scoped for reasons worth knowing. Outside a formation window, 4432 is the daemon cluster manager, whose routes report node topology, service inventory and running instances to anyone who asks. And 8660 is Predastore's unauthenticated `/healthz` and `/readyz`, which name the peers a gate cannot reach.
**The formation port needs nothing from you.** `spx admin init` opens 4432 to any source for the length of the formation window and closes it again afterwards, because a node dialling in to join is not a peer yet. It is one slot rather than a list, so it cannot be grown into a general-purpose hole. The handshake behind it is TLS 1.3 with a bearer token.
Beyond those groups the policy accepts what the platform cannot work without: established and related connections, loopback, the ICMP types PMTUD needs under a Geneve overlay, broadcast DHCP replies for the uplink and external-pool leases, and guest traffic to the metadata and VPC DNS endpoints on `169.254.169.254` and `169.254.169.253`. The guest rules are scoped to those two addresses and their ports rather than accepting everything arriving on a guest port, which would expose every wildcard-bound host service to any tenant.
## Cluster Membership
The daemon resolves who the node's peers are from the cluster it belongs to, writes them into `peers.nft`, and reloads the policy. A node that has not resolved membership yet is left reachable rather than cut off from a cluster it cannot name: the apply helper fails without touching the ruleset when the peer file is missing.
One consequence matters in practice. **A node only recognises the members of the cluster it currently belongs to.** During formation the nodes do not yet know each other, so internal traffic between them is dropped — which is why forming or expanding a cluster starts by turning the policy off.
## Is It On?
| How the node was installed | Firewall |
|---|---|
| From the ISO | **on** |
| Binary installer (`curl \| bash`) or `setup.sh` | **off** |
| `setup.sh --firewall=on` | **on** |
The binary installer defaults to off deliberately: it runs on servers that already have an operating system and services on them, and switching on a default-deny policy uninvited could cut off something Spinifex knows nothing about. The ISO is an appliance on hardware we define, so there the policy is the product.
**For production, turn it on** — either at install time:
```bash
curl -fsSL https://install.mulgadc.com | bash -s -- --firewall=on
```
or afterwards, by setting it in `/etc/spinifex/spinifex.toml` and restarting the daemon:
```toml
[network]
firewall_enabled = true
```
Before you do, check what else the machine is serving. Anything listening on a port outside the public group above stops accepting new connections.
## Narrowing SSH Access
SSH is accepted from **every source** by default. That is deliberate — arming the policy should not lock an operator out of the node they are arming it from — but it is the first thing to change on a production cluster.
The sources live in `/etc/spinifex/firewall/custom.nft`, which the installer creates once and never rewrites, so an edit here survives upgrades, re-installs and node resets:
```
define trusted_ssh_peers = { 10.0.0.0/8, 192.168.1.0/24 }
```
Check the change before applying it, and check the policy file rather than this one — on its own `custom.nft` parses as a bare define and proves nothing, because the rule that uses the variable lives elsewhere:
```bash
sudo nft -c -f /etc/spinifex/firewall/spinifex.nft
sudo nft -f /etc/spinifex/firewall/spinifex.nft
```
Never leave the variable empty. An undefined or empty set fails the whole ruleset, not just its own rule.
The ports come from sshd's own configuration, re-read on every `setup.sh` run, so moving sshd to a non-standard port is picked up by an upgrade rather than needing an edit here. IPv4 only: sshd reached over IPv6 is not matched by this rule.
## Turning It Off and On Around Cluster Changes
Turn the firewall off while you form or expand a cluster, and on again once it is up:
```bash
# Off — before forming or expanding a cluster. Run on every node.
sudo /usr/local/lib/spinifex/spinifex-firewall-apply disable
# On — once the cluster is formed and verified. Run on every node.
sudo systemctl restart spinifex-daemon
```
Restarting the daemon is what re-arms it: the node rebuilds its peer list from the cluster it is now part of, reloads the rules, and re-enables the boot-time unit so the policy survives a reboot. It also happens on its own within five minutes if you would rather wait.
`disable` removes Spinifex's own table and the peer file and leaves every other nftables table alone. It is idempotent, and it is the recovery path — it runs even on a node with no policy installed, so it cannot leave you with a node you can neither arm nor disarm.
> [!WARNING]
> Do not flush the whole nftables ruleset to clear the policy. The VPC daemon writes SNAT and per-elastic-IP forward rules into the `ip nat` and `ip filter` tables and reinstalls them only when it starts, so `nft flush ruleset` breaks elastic IPs silently until the next `systemctl restart spinifex-vpcd`. Use `disable`, which only ever touches `inet spinifex_filter`.
## The Files
Everything lives under `/etc/spinifex/firewall/`. Which files are safe to edit follows from who writes them:
| File | Written by | Survives an upgrade |
|---|---|---|
| `spinifex.nft` | `setup.sh` — the policy itself | No. Edits are overwritten |
| `local.nft` | `setup.sh` — the sshd ports it detected | No. Refreshed on every run |
| `peers.nft` | The daemon — cluster and encap peer addresses | No. Rewritten on every membership change |
| `open-ports.nft` | The apply helper — the formation window only | No. Reset closed on install and upgrade |
| `custom.nft` | **You** — `trusted_ssh_peers` and anything else you add | **Yes.** Created once, never rewritten |
| `mode` | `setup.sh` — whether the install path armed the policy | No |
The daemon cannot write `/etc` itself, and a blanket `nft` grant would be root-equivalent, so changes go through `/usr/local/lib/spinifex/spinifex-firewall-apply` — a root helper that accepts a fixed set of verbs and re-validates every address handed to it.
## Checking It
```bash
sudo nft list table inet spinifex_filter
```
The peer list is an nft variable, expanded when the rules load, so it does not appear under a name of its own. Look instead at the `ip saddr { ... }` addresses on the cluster-plane rules — the ones accepting 4222, 6641, 6642 and the rest.
Every node's addresses should be there. On a multi-NIC node that means its WAN, LAN and VPC addresses, so expect several entries per node. A missing node means its cluster traffic is being dropped.
Dropped packets are logged, rate-limited to five a minute, so this tells you whether a connection problem is the firewall or something else:
```bash
sudo journalctl -k | grep 'spinifex-fw drop'
```
An empty log with a connection still failing means the policy is not the cause. A node with no table loaded at all is not armed:
```bash
systemctl is-enabled spinifex-firewall.service
```
## Troubleshooting
### A Join Hangs Instead of Failing
A hang rather than a quick refusal means packets are being dropped. Check the drop log on the node running init, and confirm the formation port opened — `spx admin init` prints `⚠️ Could not open port 4432 in the host firewall` when it could not:
```bash
sudo journalctl -k | grep 'spinifex-fw drop'
```
On ISO-installed nodes the usual cause is that each node's policy still trusts only itself, because that is the whole cluster as far as it knows. Turn the firewall off on **every** node, retry, and re-arm once the cluster is verified.
### A Node Is Missing from the Peer List
The daemon rewrites `peers.nft` from the cluster it belongs to, so a missing node means that node is not in the membership this one resolved. Confirm what the cluster thinks first, then restart the daemon to force a rebuild:
```bash
spx get nodes
sudo systemctl restart spinifex-daemon
```
### One of My Own Services Stopped Answering
Arming the policy closes every port outside the public group. Add what you need to `custom.nft`, which is the only file an upgrade leaves alone, and check the policy before applying it:
```bash
sudo nft -c -f /etc/spinifex/firewall/spinifex.nft
```
### Elastic IPs Stopped Working
Something flushed the whole nftables ruleset, taking the VPC daemon's SNAT and forward rules with it. They are reinstalled on start:
```bash
sudo systemctl restart spinifex-vpcd
```