gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-15 17:11:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: separate the install guides from the reference material they had absorbed

Browse Source 
Both install guides had grown sections that were not install steps, which is what made the multi-node one hard to follow. Each now ends on a step that hands off to Setting Up Your Cluster, and the reference material moves under Administration where it can be read on its own.

- Single-node and multi-node both end on a "Set Up Your Cluster" step rather than a trailing sentence.
- The multi-node cluster-state check via `spx admin kv digest` is gone. Verification is now every node Ready, capacity pooled, and the AWS API answering.
- "How Multi-Node Storage Works" becomes docs/admin/predastore-distributed-storage, expanded with the host and node model, the write and read paths, degraded writes, hinted handoff, repair, encryption at rest, and what each class of failure costs.
- "Firewall and Cluster Membership" becomes docs/admin/host-firewall, expanded with the full public, cluster and encap port groups, how to narrow SSH, and which files under /etc/spinifex/firewall survive an upgrade.
- Setting Up Your Cluster gains a closing step that arms the firewall and narrows SSH, so the securing work lands after a cluster that works rather than during the install.

install-usb and vpc-networking followed the two moved sections to their new paths.
This commit is contained in:
Ben Duncan
2026-09-18 10:58:25 -05:00
parent e035842503
commit 2ba8f5e4ab
7 changed files with 517 additions and 118 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/admin/host-firewall/README.md
+233
View File
@@ -0,0 +1,233 @@
---
title: "Host Firewall"
seoTitle: "Host Firewall and Cluster Membership — Spinifex Docs"
description: "Arm the Spinifex host firewall, narrow SSH to your management networks, and learn how the peer-scoped cluster plane tracks membership changes on its own."
category: "Admin"
tags:
- security
- firewall
- nftables
- hardening
resources:
- title: "Network Connections"
url: "/docs/network-connections"
- title: "Multi-Node Install"
url: "/docs/install-multi-node"
- title: "Setting Up Your Cluster"
url: "/docs/setting-up-your-cluster"
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
---
# Host Firewall
> Arm the host firewall, narrow SSH, and keep the cluster plane off the public internet.
## Table of Contents
- [Overview](#overview)
- [What the policy allows](#what-the-policy-allows)
- [Cluster membership](#cluster-membership)
- [Is it on?](#is-it-on)
- [Narrowing SSH access](#narrowing-ssh-access)
- [Turning it off and on around cluster changes](#turning-it-off-and-on-around-cluster-changes)
- [The files](#the-files)
- [Checking it](#checking-it)
- [Troubleshooting](#troubleshooting)
---
## Overview
Spinifex ships an optional host firewall — an nftables input policy that defaults to drop and divides the node's ports into two groups:
| Group | Ports | Who can reach them |
|---|---|---|
| **Public** | SSH, 443, 3000 (console), 8443 (S3), 9999 (AWS gateway), 53 (DNS) | anyone |
| **Internal** | OVN, NATS, formation, Geneve and the rest of the cluster plane | **cluster members only** |
The internal group is the point. Before this existed, OVN and NATS were reachable from the public internet on a WAN-facing node.
**"Cluster members" is not a list you maintain.** Each node works it out from the cluster it belongs to and rewrites its own rules whenever membership changes — you never edit the peer list by hand.
The policy governs inbound traffic only. Outbound is untouched, because the metadata reply path egresses under per-tap policy routing and a filter there would break it.
## Instructions
## What the Policy Allows
The public plane is open to any source, IPv4 only — an accept with no source scope would otherwise also serve on any global IPv6 address the node picks up from an upstream router advertisement:
| Ports | Serving |
|---|---|
| TCP 443 | Held for the TLS edge; no listener yet, so the rollout needs no firewall change |
| TCP 3000 | Web console |
| TCP 8443 | Predastore S3 gate |
| TCP 9999 | AWS gateway — also how every guest agent (load balancer, EKS, ECS, RDS) reaches its control plane, since they speak SigV4 rather than NATS |
| TCP and UDP 53 | Northstar authoritative DNS. Open on every node deliberately: a node serving no public zone still answers here, which is the accepted cost of not templating the policy per node |
SSH is public too, but scoped by a variable you own — see [Narrowing SSH access](#narrowing-ssh-access).
The cluster plane is peer-scoped. On a single-NIC node the planes collapse onto one address, which is why these rules match peer addresses rather than an interface or a CIDR:
| Ports | Serving |
|---|---|
| TCP 4222, 4248 | NATS client and cluster routes |
| TCP 4432 | Formation, then the daemon cluster manager |
| TCP and UDP 5300 | Internal DNS |
| TCP 6641, 6642 | OVN northbound and southbound |
| TCP 6643, 6644 | OVN database clustering |
| TCP 8660 | Predastore's admin listener |
| UDP 6660, 7660 | Object shards and metadata consensus |
| UDP 6081, 500, 4500, and ESP | Geneve, IKE, NAT-T and IPsec |
Two of those are peer-scoped for reasons worth knowing. Outside a formation window, 4432 is the daemon cluster manager, whose routes report node topology, service inventory and running instances to anyone who asks. And 8660 is Predastore's unauthenticated `/healthz` and `/readyz`, which name the peers a gate cannot reach.
**The formation port needs nothing from you.** `spx admin init` opens 4432 to any source for the length of the formation window and closes it again afterwards, because a node dialling in to join is not a peer yet. It is one slot rather than a list, so it cannot be grown into a general-purpose hole. The handshake behind it is TLS 1.3 with a bearer token.
Beyond those groups the policy accepts what the platform cannot work without: established and related connections, loopback, the ICMP types PMTUD needs under a Geneve overlay, broadcast DHCP replies for the uplink and external-pool leases, and guest traffic to the metadata and VPC DNS endpoints on `169.254.169.254` and `169.254.169.253`. The guest rules are scoped to those two addresses and their ports rather than accepting everything arriving on a guest port, which would expose every wildcard-bound host service to any tenant.
## Cluster Membership
The daemon resolves who the node's peers are from the cluster it belongs to, writes them into `peers.nft`, and reloads the policy. A node that has not resolved membership yet is left reachable rather than cut off from a cluster it cannot name: the apply helper fails without touching the ruleset when the peer file is missing.
One consequence matters in practice. **A node only recognises the members of the cluster it currently belongs to.** During formation the nodes do not yet know each other, so internal traffic between them is dropped — which is why forming or expanding a cluster starts by turning the policy off.
## Is It On?
| How the node was installed | Firewall |
|---|---|
| From the ISO | **on** |
| Binary installer (`curl \| bash`) or `setup.sh` | **off** |
| `setup.sh --firewall=on` | **on** |
The binary installer defaults to off deliberately: it runs on servers that already have an operating system and services on them, and switching on a default-deny policy uninvited could cut off something Spinifex knows nothing about. The ISO is an appliance on hardware we define, so there the policy is the product.
**For production, turn it on** — either at install time:
```bash
curl -fsSL https://install.mulgadc.com | bash -s -- --firewall=on
```
or afterwards, by setting it in `/etc/spinifex/spinifex.toml` and restarting the daemon:
```toml
[network]
firewall_enabled = true
```
Before you do, check what else the machine is serving. Anything listening on a port outside the public group above stops accepting new connections.
## Narrowing SSH Access
SSH is accepted from **every source** by default. That is deliberate — arming the policy should not lock an operator out of the node they are arming it from — but it is the first thing to change on a production cluster.
The sources live in `/etc/spinifex/firewall/custom.nft`, which the installer creates once and never rewrites, so an edit here survives upgrades, re-installs and node resets:
```
define trusted_ssh_peers = { 10.0.0.0/8, 192.168.1.0/24 }
```
Check the change before applying it, and check the policy file rather than this one — on its own `custom.nft` parses as a bare define and proves nothing, because the rule that uses the variable lives elsewhere:
```bash
sudo nft -c -f /etc/spinifex/firewall/spinifex.nft
sudo nft -f /etc/spinifex/firewall/spinifex.nft
```
Never leave the variable empty. An undefined or empty set fails the whole ruleset, not just its own rule.
The ports come from sshd's own configuration, re-read on every `setup.sh` run, so moving sshd to a non-standard port is picked up by an upgrade rather than needing an edit here. IPv4 only: sshd reached over IPv6 is not matched by this rule.
## Turning It Off and On Around Cluster Changes
Turn the firewall off while you form or expand a cluster, and on again once it is up:
```bash
# Off — before forming or expanding a cluster. Run on every node.
sudo /usr/local/lib/spinifex/spinifex-firewall-apply disable
# On — once the cluster is formed and verified. Run on every node.
sudo systemctl restart spinifex-daemon
```
Restarting the daemon is what re-arms it: the node rebuilds its peer list from the cluster it is now part of, reloads the rules, and re-enables the boot-time unit so the policy survives a reboot. It also happens on its own within five minutes if you would rather wait.
`disable` removes Spinifex's own table and the peer file and leaves every other nftables table alone. It is idempotent, and it is the recovery path — it runs even on a node with no policy installed, so it cannot leave you with a node you can neither arm nor disarm.
> [!WARNING]
> Do not flush the whole nftables ruleset to clear the policy. The VPC daemon writes SNAT and per-elastic-IP forward rules into the `ip nat` and `ip filter` tables and reinstalls them only when it starts, so `nft flush ruleset` breaks elastic IPs silently until the next `systemctl restart spinifex-vpcd`. Use `disable`, which only ever touches `inet spinifex_filter`.
## The Files
Everything lives under `/etc/spinifex/firewall/`. Which files are safe to edit follows from who writes them:
| File | Written by | Survives an upgrade |
|---|---|---|
| `spinifex.nft` | `setup.sh` — the policy itself | No. Edits are overwritten |
| `local.nft` | `setup.sh` — the sshd ports it detected | No. Refreshed on every run |
| `peers.nft` | The daemon — cluster and encap peer addresses | No. Rewritten on every membership change |
| `open-ports.nft` | The apply helper — the formation window only | No. Reset closed on install and upgrade |
| `custom.nft` | **You** — `trusted_ssh_peers` and anything else you add | **Yes.** Created once, never rewritten |
| `mode` | `setup.sh` — whether the install path armed the policy | No |
The daemon cannot write `/etc` itself, and a blanket `nft` grant would be root-equivalent, so changes go through `/usr/local/lib/spinifex/spinifex-firewall-apply` — a root helper that accepts a fixed set of verbs and re-validates every address handed to it.
## Checking It
```bash
sudo nft list table inet spinifex_filter
```
The peer list is an nft variable, expanded when the rules load, so it does not appear under a name of its own. Look instead at the `ip saddr { ... }` addresses on the cluster-plane rules — the ones accepting 4222, 6641, 6642 and the rest.
Every node's addresses should be there. On a multi-NIC node that means its WAN, LAN and VPC addresses, so expect several entries per node. A missing node means its cluster traffic is being dropped.
Dropped packets are logged, rate-limited to five a minute, so this tells you whether a connection problem is the firewall or something else:
```bash
sudo journalctl -k | grep 'spinifex-fw drop'
```
An empty log with a connection still failing means the policy is not the cause. A node with no table loaded at all is not armed:
```bash
systemctl is-enabled spinifex-firewall.service
```
## Troubleshooting
### A Join Hangs Instead of Failing
A hang rather than a quick refusal means packets are being dropped. Check the drop log on the node running init, and confirm the formation port opened — `spx admin init` prints `⚠️ Could not open port 4432 in the host firewall` when it could not:
```bash
sudo journalctl -k | grep 'spinifex-fw drop'
```
On ISO-installed nodes the usual cause is that each node's policy still trusts only itself, because that is the whole cluster as far as it knows. Turn the firewall off on **every** node, retry, and re-arm once the cluster is verified.
### A Node Is Missing from the Peer List
The daemon rewrites `peers.nft` from the cluster it belongs to, so a missing node means that node is not in the membership this one resolved. Confirm what the cluster thinks first, then restart the daemon to force a rebuild:
```bash
spx get nodes
sudo systemctl restart spinifex-daemon
```
### One of My Own Services Stopped Answering
Arming the policy closes every port outside the public group. Add what you need to `custom.nft`, which is the only file an upgrade leaves alone, and check the policy before applying it:
```bash
sudo nft -c -f /etc/spinifex/firewall/spinifex.nft
```
### Elastic IPs Stopped Working
Something flushed the whole nftables ruleset, taking the VPC daemon's SNAT and forward rules with it. They are reinstalled on start:
```bash
sudo systemctl restart spinifex-vpcd
```
docs/admin/predastore-distributed-storage/README.md
+250
View File
@@ -0,0 +1,250 @@
---
title: "Predastore Distributed Storage"
seoTitle: "How Predastore Distributed Storage Works — Spinifex Docs"
description: "How Spinifex stores objects across a cluster: gate, blob and meta nodes, Reed-Solomon erasure coding, placement, degraded writes, repair and encryption at rest."
category: "Admin"
tags:
- storage
- s3
- predastore
- erasure coding
- durability
resources:
- title: "Predastore Repository"
url: "https://github.com/mulgadc/predastore"
- title: "Viperblock (EBS)"
url: "https://github.com/mulgadc/viperblock"
- title: "Multi-Node Install"
url: "/docs/install-multi-node"
- title: "Network Connections"
url: "/docs/network-connections"
---
# Predastore Distributed Storage
> How Spinifex stores objects across a cluster, and what a server failure costs.
## Table of Contents
- [Overview](#overview)
- [Hosts and nodes](#hosts-and-nodes)
- [The configuration file](#the-configuration-file)
- [Erasure coding and durability](#erasure-coding-and-durability)
- [Placement](#placement)
- [The write path](#the-write-path)
- [The read path](#the-read-path)
- [Repair](#repair)
- [Encryption at rest](#encryption-at-rest)
- [What a server failure costs](#what-a-server-failure-costs)
- [Checking it](#checking-it)
- [Troubleshooting](#troubleshooting)
---
## Overview
Predastore is the S3-compatible object store underneath a Spinifex cluster, and almost everything durable ends up in it:
- **Machine images** — the AMIs instances launch from
- **Block storage** — EBS volume data and snapshots, written through [Viperblock](https://github.com/mulgadc/viperblock), which uses Predastore as its S3 backend
- **User data** — cloud-init configuration and system artifacts
- **DNS zone files** — the zones Northstar serves
- **Anything your own workloads put in a bucket** through the S3 API
This is background reading. You do not configure any of it by hand: `spx admin init` and `spx admin join` build the topology from the servers that actually form the cluster, and every machine gets the same file with its own host ID recorded in `spinifex.toml`.
## Instructions
## Hosts and Nodes
A cluster is two levels.
A **host** is one machine: one Predastore process owning that machine's data directory, TLS identity and encryption key. A **node** is a role pinned to a host, running inside that process on its own port. Every server in a Spinifex cluster is one host carrying three nodes:
| Role | Port | Purpose |
|---|---|---|
| `gate` | TCP 8443 | Serves the S3 API — SigV4, IAM, erasure coding, placement. Every server runs one, so any of them answers an S3 request. |
| `blob` | UDP 6660 | Holds erasure-coded object shards. One per machine. |
| `meta` | UDP 7660 | Member of the Raft quorum over global state — buckets and the object index. |
Ports have to be unique within a host but not across the cluster, so every machine uses the same three. That is why the port surface does not widen as the cluster grows — see [Network requirements](/docs/install-multi-node#network-requirements).
Which transport a pair of nodes uses follows from where they sit, not from a mode flag. Nodes on the same machine talk over an in-process pipe and open no socket at all; nodes on different machines talk over QUIC, TLS 1.3, authenticated by the cluster CA. A single-server install is therefore one process listening on 8443 and nothing else.
Nothing dials a gate, so a gate binds no cluster-plane listener. Its port is the S3 port and nothing more.
## The Configuration File
The whole cluster's storage topology lives in `/etc/spinifex/predastore/predastore.toml`, identical on every server apart from which host the local process is told to be.
Each server is one `[[host]]` block carrying three `[[host.node]]` blocks:
```toml
[[host]]
id = 1
bind_addr = "10.2.0.2" # cluster plane: raft and blob
addr = "10.2.0.2" # what the other hosts dial
data_dir = "/var/lib/spinifex/predastore"
tls_cert = "/etc/spinifex/server.pem"
tls_key = "/etc/spinifex/server.key"
encryption_key = "/etc/spinifex/predastore/encryption.key"
admin_port = 8660 # /healthz, /readyz, /repair
[[host.node]]
id = 1
role = "gate"
port = 8443
bind_addr = "0.0.0.0" # public plane; only a gate may set this
```
The two bind addresses are the point. `bind_addr` on the host is the cluster plane — Raft consensus and shard traffic listen there and nowhere else, at the address peers dial rather than a wildcard, so on a multi-homed machine replication never reaches the public interface. The gate binds the public plane separately, because S3 is meant to be reachable.
## Erasure Coding and Durability
The erasure code is cluster configuration, not a per-request choice, and it is chosen at formation from the number of servers — each machine contributes exactly one blob node:
| Servers | Code | What it survives |
|---|---|---|
| 1 | `RS(1,0)` | Nothing at this layer. One shard on one node; redundancy is the pool's job — ZFS raidz or a mirror under the data directory. |
| 2 | `RS(1,1)` | One server's shards, but the metadata quorum has no majority to lose. |
| 3 or more | `RS(2,1)` | Any one server's shards, with the metadata quorum intact. |
An object is encoded end to end into `K` data and `M` parity shards with no intermediate chunking. Each shard is roughly `⌈size / K⌉` bytes, and any `K` of the `K+M` shards rebuild the object.
`data + parity` must not exceed the number of blob nodes, because a stripe is spread over distinct nodes and a narrower cluster would fail every write. `RS(2,1)` surviving the loss of any one server is the reason [three servers is the recommended minimum](/docs/install-multi-node).
## Placement
Which nodes hold an object's shards comes from a consistent hash ring over the blob nodes, keyed by the object hash. `Nodes(hash, K+M)` returns the placement in shard order — shard `i` goes to the node at position `i` — so every gate in the cluster derives the same placement from the same object without consulting anything.
The gate then **records** that placement in the metadata plane rather than trusting the ring at read time. A ring that has since changed shape does not lose objects placed under the old one.
## The Write Path
The gate authenticates the request, resolves the bucket, hashes the object name, places it on the ring, encodes the body into data and parity shards, streams each shard to its blob node in parallel, and records the placement.
Three properties matter operationally:
- **The write is two-phase.** Each shard put makes the bytes durable and invisible. The placement record landing in Raft is the commit point, and the shards are published only afterwards — so an interrupted writer costs a round trip rather than the object. Shards are always durable before metadata: a crash between the two leaves shards nothing points at, rather than metadata pointing at data that was never written.
- **Every write mints an epoch**, stamped on every shard of the stripe and into the record. The record names one generation, and a shard either belongs to it or does not. That is what makes a half-completed overwrite a reconstruction rather than an object spliced from two generations.
- **One server down does not refuse writes.** The write is acknowledged once `K` shards are durable, and the response carries `X-Spx-Degraded-Write: <n>`. `K` is the floor rather than `K+1` because any `K` of the `K+M` reconstruct — at `RS(2,1)` one more buys no redundancy and would refuse three quarters of writes with a single server down.
A shard its owner will not take is written to the next node along the ring instead of being given up on — **hinted handoff** — and the response carries `X-Spx-Handoff: <n>`. The stripe is then complete and the object is as redundant as a full-width write; what is outstanding is only that the shards are not all where the record says. The holder is derived rather than recorded, so there is no hint to store, replicate or lose. Handoff is skipped when the cluster has no node to spare, since a second shard of one object on a node that already holds one is not redundancy.
If any node reported storage pressure the response also carries `X-Predastore-Pool-Pressure: nearfull`; a node that refused outright surfaces as `507 InsufficientStorage`.
## The Read Path
The gate reads the recorded placement, fetches the data shards, and joins them. Parity is never touched on the happy path. Only if the join fails does it refetch including parity and reconstruct, and a read that had to reconstruct says so in `X-Spx-Degraded: <n>` — a cost report, not an error.
Every shard read names the record's epoch, and a node holding another generation answers `epoch-mismatch`. **The gate counts that exactly as a missing shard**, so the read either rebuilds the object the record names or fails. It never returns plausible nonsense.
A shard its owner cannot serve is looked for on the handoff node before it is counted as failed, since a write that could not reach the owner will have put it there.
A `Range` request that lands entirely inside one data shard takes a fast path — a single ranged shard read — because Reed-Solomon splits data sequentially. Anything wider reconstructs the object and slices it.
## Repair
Degraded writes trade redundancy for availability, and repair is what pays that back. It restores shards a blob node owns but does not hold at the generation its object's record names, and it runs on every server with a local blob node.
Two things are worth knowing about it:
- **It carries no correctness weight.** A read compares each shard's epoch against the record it has already loaded and discards a stale one whether or not repair has ever run. What repair restores is redundancy — the number of further losses an object survives — and never the object.
- **It rebuilds onto the owner**, so it closes the gap only once the missing server returns. At `RS(2,1)` an object written while a server is down holds no parity for the duration of the outage. That is the accepted position: refusing writes meanwhile is the worse failure.
Every pass scans the placement records from the beginning, asks each holder which generation it has, and rebuilds the ones that disagree. A shard that was handed off is returned rather than recomputed — cheaper by a decode — and the holder's copy is released only after the owner's is published. If fewer than `K` peers hold the record's generation the rebuild fails and says so, rather than inventing a shard from a mixture of generations.
## Encryption at Rest
Every fragment is sealed independently under AES-256-GCM, with a deterministic nonce and additional data reconstructed at read time from the on-disk header plus per-data-directory state. There is no unencrypted code path: the process refuses to start without an encryption key.
| Property | What it means |
|---|---|
| **Confidentiality** | Disk-level access yields ciphertext and tags. Plaintext is never written. |
| **Authenticated integrity** | GCM is the sole integrity authority; there is no separate checksum. Corruption, tampering and a wrong key all surface as one integrity error. |
| **Position binding** | The additional data binds each fragment to its slot, so swapping fragments between objects fails the tag. |
| **Cross-directory defence** | A per-data-directory store ID enters the nonce, so splicing a fragment from one data directory into another fails the tag. |
The key is one 32-byte AES-256 key per cluster, and **every server must be given the same file** — fragments sealed under one key cannot be opened under another. `spx admin init` generates it and `spx admin join` distributes it, which is also why joining a server that has already been in service discards its own key and orphans everything sealed under it. The loader is fail-closed on permissions: a group- or other-readable key file is rejected outright, with no override.
## What a Server Failure Costs
| What fails | Reads | Writes |
|---|---|---|
| One gate (of three or more) | Unaffected — any gate answers for the cluster | Unaffected |
| One blob node, `RS(2,1)` | Served by reconstruction, reported in `X-Spx-Degraded` | Accepted at the `K` floor, reported in `X-Spx-Degraded-Write`, or placed complete via handoff |
| More than `M` blob nodes | Objects placed on them are lost | Refused |
| Metadata quorum | Serve whatever the surviving replicas last applied | Stopped entirely |
Reads try the cached Raft leader first, then every replica, and report not-found only once every replica has answered not-found — a replica that has not applied a key yet must not be mistaken for absence. Writes go through the leader, and a replica that cannot commit redirects the client to the one that can.
Replacing a server restores capacity, not the shards that were on the old one: repair rebuilds onto the owner it expects, so plan a replacement as a return of that node rather than as re-replication onto a new one.
## Checking It
Each host runs an unauthenticated admin listener on port 8660, bound to the cluster plane rather than the S3 address. It serves counters and readiness, never addresses or keys:
```bash
curl -s http://$(hostname -I | awk '{print $1}'):8660/readyz
curl -s http://$(hostname -I | awk '{print $1}'):8660/repair
```
`/readyz` reports the most recent probe cycle for the nodes in that process, and answers `503` until one has completed — reporting ready on no information would send traffic to a process that has never answered the question. `/healthz` answers for the process itself: reaching the handler is the whole check.
That the gate is serving is an S3 request like any other:
```bash
export AWS_PROFILE=spinifex
aws s3 ls
```
To see whether a write or read was degraded, ask for the response headers:
```bash
aws s3api put-object --bucket predastore --key probe --body /etc/hostname --debug 2>&1 | grep -i 'x-spx-\|x-predastore-'
```
And `spx get nodes` should list `predastore` in `SERVICES` on every node — a short service list on one server means something failed to start there.
## Troubleshooting
### S3 Answers but Every Write Fails
The gate is up and the metadata quorum is not. Check that a majority of servers are running `spinifex-predastore`:
```bash
spx get nodes
sudo systemctl is-active spinifex-predastore
```
Reads keep serving what the surviving replicas last applied, which is why this presents as a write-only outage.
### 507 InsufficientStorage, or `nearfull` on Every Response
A blob node is out of space, or close enough that compaction has been kicked. Free space in the data directory on each server:
```bash
df -h /var/lib/spinifex/predastore
```
Capacity here is per server, not pooled: a stripe needs its shard on a specific node, so one full server refuses writes for the objects the ring places on it.
### Reads Fail with an Integrity Error
GCM is the only integrity authority, so corruption, tampering and a wrong encryption key are indistinguishable. Confirm the key first, since it is by far the most likely of the three — every server must hold the identical file:
```bash
sudo sha256sum /etc/spinifex/predastore/encryption.key
```
Compare the digest across servers. A mismatch means that server was initialized separately rather than joined, and the fragments it sealed cannot be opened by the cluster.
### Degraded Reads That Never Clear
`X-Spx-Degraded` on reads long after an outage means repair has not restored the shards. Repair rebuilds onto the node that owns the shard, so it cannot finish while that node is absent:
```bash
curl -s http://$(hostname -I | awk '{print $1}'):8660/repair
```
`"enabled": false` means the sweep is off on that host. Otherwise bring the missing server back and let a pass complete — the sweep runs every five minutes by default.
docs/admin/setting-up-your-cluster/README.md
+17
View File
@@ -28,6 +28,7 @@ resources:
- [5. Connect via SSH](#5-connect-via-ssh)
- [6. Managing Instances](#6-managing-instances)
- [7. Launching the Web UI](#7-launching-the-web-ui)
- [8. Secure the Install](#8-secure-the-install)
- [Troubleshooting](#troubleshooting)
---
@@ -340,6 +341,22 @@ At the Spinifex login screen, paste the **Access Key ID** and **Secret Access Ke
Once logged in, you have browser-based access to every Spinifex feature: launch and manage instances, attach EBS volumes, browse S3 buckets, configure VPCs and security groups, and manage IAM users and keys — all backed by the same AWS-compatible control plane the CLI uses.
## 8. Secure the Install
Your cluster works. Before it carries anything real, close the host down.
**Arm the host firewall.** A binary install leaves it off, because it lands on a machine that was already doing something. Armed, it keeps OVN, NATS and the rest of the cluster plane reachable by cluster members only, and everything outside the public group closed:
```toml
# /etc/spinifex/spinifex.toml
[network]
firewall_enabled = true
```
**Narrow SSH.** It is accepted from every source by default, so that arming the policy cannot lock you out of the node you are arming it from. Replace `0.0.0.0/0` with your management networks in `/etc/spinifex/firewall/custom.nft` — the one file an upgrade never overwrites.
[Host Firewall](/docs/host-firewall) covers both, what the policy allows, and how each node tracks cluster membership without a peer list you maintain.
## Additional Options
### Private Subnets (No Public IP)