gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-15 09:01:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: rework the multi-node install guide, fix cross-doc links

Browse Source 
Multi-node install
- Lead with the fact that the guide builds a three-node cluster, and say
  how to go beyond three. The callout sits inside ## Overview because the
  docs site discards everything above the first section heading.
- Replace the OVN-only sizing prose with a table covering all three
  distributed layers, and move the OVN control-plane detail into the VPC
  networking guide, linked from here.
- Add minimum and recommended hardware.
- Move the object storage layout to the end as background; it was the
  second thing a reader hit.
- Collapse the ISO conversion notes from a section into one skippable
  box, now that --force is unconditional in Step 4 and the formation
  port needs no manual intervention.
- Expand verification with spx get nodes and spx top nodes, including
  what to look for in each.
- Drop the install-node.sh shortcut (internal tooling) and the
  "adding a node later" section (no good story yet).

Corrections found while writing
- Servers 4+ were described as running no database. They run predastore
  and viperblock like every other node; only the OVN database cluster is
  capped at three. Fixed in four places.
- Step 4 never needed the firewall down: spx admin init opens the
  formation port itself for the formation window. Only Step 3's
  peer-scoped OVN ports are affected.
- `nft list table inet spinifex_filter | grep spinifex_peers` could never
  match — the peer list is an nft define, expanded at load time. Anyone
  following it would have seen empty output and assumed the worst.
- Disambiguate "database": with predastore, NATS and OVN all in play it
  needed qualifying throughout.
- Align the example output region with the AWS_REGION the guide exports.

Cross-doc links
- Rewrite relative and category-prefixed links to the canonical
  /docs/<slug> form. The site publishes docs flat, so ../foo/README.md
  and /docs/category/foo both 404.
- Point the six links to unpublished docs at the source repo instead.
- Fix four fragments written against GitHub's anchor algorithm.
This commit is contained in:
Ben Duncan
2026-08-21 14:14:28 -05:00
parent b834910e64
commit 8b3c706ea3
16 changed files with 268 additions and 142 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/admin/host-lifecycle/README.md
+1 -1
View File
@@ -11,7 +11,7 @@ resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "Updating Spinifex"
url: "/docs/admin/update"
url: "/docs/update"
---
# Host and Guest Lifecycle
docs/admin/spinifex-admin-cli/README.md
+1 -1
View File
@@ -77,7 +77,7 @@ ACCOUNT ID NAME STATUS CREATED
000000000002 myteam ACTIVE 2026-07-03 03:32
```
Deleting an account, and creating or removing accounts from your own signup or provisioning system over the private admin API, are covered in [Account Management](/docs/admin/account-management).
Deleting an account, and creating or removing accounts from your own signup or provisioning system over the private admin API, are covered in [Account Management](https://github.com/mulgadc/spinifex/blob/main/docs/admin/account-management/README.md).
## Node Management
docs/admin/update/README.md
+3 -3
View File
@@ -43,7 +43,7 @@ For operators who want to review migrations before they are applied, a manual up
> [!WARNING]
> **AMI metadata is exempt when upgrading from v1.16.0 or earlier to the release carrying the EBS-provider decoupling.** AMI metadata moved to `ebsmetadata` documents, and the legacy path that read it from `ami-<id>/config.json` was removed rather than migrated. There is no prefix scan, no fallback and no backfill at daemon start, so an AMI imported before the change becomes invisible to the control plane afterwards: `describe-images --image-ids` answers `InvalidAMIID.NotFound` and launches fail with `AMI has no snapshot ID, cannot perform zero-copy clone`. As with object storage above, `spx admin upgrade` reports nothing pending, because the gap is in stored data rather than in a config file.
>
> **Re-import affected AMIs after upgrading**, using [`spx admin images import`](/docs/admin/spinifex-admin-cli), which writes metadata in the new location. If the source images are no longer available, the installation has to be re-initialised. Verify before you rely on it — an AMI is only healthy if it resolves by ID, not merely if it appears in the list:
> **Re-import affected AMIs after upgrading**, using [`spx admin images import`](/docs/spinifex-admin-cli), which writes metadata in the new location. If the source images are no longer available, the installation has to be re-initialised. Verify before you rely on it — an AMI is only healthy if it resolves by ID, not merely if it appears in the list:
>
> ```bash
> aws ec2 describe-images --image-ids <ami-id>
@@ -113,7 +113,7 @@ Migrations modify config files on disk but do not restart running services, and
sudo systemctl restart spinifex.target
```
A restart preserves any running guests — they are not rebooted, and storage returns within seconds. See [Host and Guest Lifecycle](/docs/admin/host-lifecycle) for the full contract.
A restart preserves any running guests — they are not rebooted, and storage returns within seconds. See [Host and Guest Lifecycle](https://github.com/mulgadc/spinifex/blob/main/docs/admin/host-lifecycle/README.md) for the full contract.
## Checking for Unit Drift
@@ -165,7 +165,7 @@ Migrations edit config files on disk but the running daemons continue to use the
sudo systemctl restart spinifex.target
```
A restart preserves any running guests — they are not rebooted, and storage returns within seconds. See [Host and Guest Lifecycle](/docs/admin/host-lifecycle) for the full contract.
A restart preserves any running guests — they are not rebooted, and storage returns within seconds. See [Host and Guest Lifecycle](https://github.com/mulgadc/spinifex/blob/main/docs/admin/host-lifecycle/README.md) for the full contract.
### A Node Answers Some Requests as if It Were Still on the Old Build
docs/compute/vpc-networking/README.md
+31
View File
@@ -360,6 +360,37 @@ This requests an IP from the **router's DHCP** (e.g., 192.168.1.1 serving addres
If no Linux bridge owns the default route, `setup-ovn.sh` exits with guidance rather than silently breaking host connectivity.
## OVN Control Plane on Multi-Node Clusters
How OVN is deployed depends on the size of the cluster, and this is the main reason three servers is the recommended minimum for a multi-server deployment.
| Cluster size | OVN databases | Tolerates |
|---|---|---|
| 1–2 servers | standalone, on the first node | nothing — that node is a single point of failure for the control plane |
| 3 or more | clustered (RAFT) across the first three nodes | loss of any one database node |
Servers beyond the third run the full set of Spinifex services, but do not join the OVN database cluster — they connect to it as clients. The quorum stays at three however large the cluster gets, which is what keeps control-plane write latency stable.
### What a control-plane outage actually costs
Less than it sounds. `ovn-controller` has already programmed the forwarding rules into each host, so **running instances keep full networking** — east-west, north-south, NAT and security groups all continue to work with the databases down.
What stops is *change*. Creating a VPC, launching an instance, and updating a security group all need the control plane, because each has to write new logical topology before anything can program it.
So a standalone OVN deployment is a reasonable choice for a lab or a single-server install. It is a poor one for production, where the inability to launch an instance during an outage is usually as bad as the instances being down.
### Clustering the databases
The three database nodes are set up with `--db-cluster-local-addr`, and nodes 2 and 3 additionally point at node 1 with `--db-cluster-remote-addr`. Compute nodes take `--ovn-remote` listing all three, so they survive any one of them failing. See [Multi-Node Install](/docs/install-multi-node) for the exact commands in sequence.
A clustered database can only be created from scratch — the `ovn-central` package starts a standalone one on install, so the cluster setup passes `--recreate-db` to replace it. Confirm the result with:
```bash
sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
```
Three servers should be listed with exactly one leader. A report of a standalone database means the cluster flags did not take.
## Per-Node Configuration
Different nodes in a cluster can have different WAN bridges and NICs:
docs/install/install-airgapped/README.md
+1 -1
View File
@@ -11,7 +11,7 @@ resources:
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "External Connection Inventory"
url: "/docs/security/network-connections"
url: "/docs/network-connections"
---
# Air-Gapped Installation
docs/install/install-multi-node/README.md
+198 -103
View File
@@ -1,7 +1,7 @@
---
title: "Multi-Node Install"
seoTitle: "Multi-Node Spinifex Cluster Install — Spinifex Docs"
description: "Deploy Spinifex across several servers to form an availability zone with automatic cluster formation, high availability, data durability, and fault tolerance."
description: "Deploy Spinifex across three or more servers to form an availability zone with clustered OVN, replicated object storage, and automatic cluster formation."
category: "Install"
tags:
- install
@@ -12,6 +12,8 @@ tags:
resources:
- title: "Bootable USB Install"
url: "/docs/install-usb"
- title: "VPC Networking"
url: "/docs/vpc-networking"
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
- title: "Predastore (S3)"
@@ -22,84 +24,117 @@ resources:
# Multi-Node Installation
> Deploy Spinifex across multiple servers to create an availability zone.
> Deploy Spinifex across three or more servers to create an availability zone.
## Table of Contents
- [Overview](#overview)
- [Object Storage Layout](#object-storage-layout)
- [Cluster sizing](#cluster-sizing)
- [Hardware](#hardware)
- [Network requirements](#network-requirements)
- [Prerequisites](#prerequisites)
- [Instructions](#instructions)
- [Converting ISO-Installed Nodes](#converting-iso-installed-nodes)
- [Firewall and Cluster Membership](#firewall-and-cluster-membership)
- [Verifying the cluster](#step-6-verify-the-cluster)
- [Firewall and cluster membership](#firewall-and-cluster-membership)
- [How multi-node storage works](#how-multi-node-storage-works)
- [Troubleshooting](#troubleshooting)
---
## Overview
A Spinifex cluster distributes services across multiple servers for high availability, data durability, and fault tolerance. Cluster formation is automatic — the init node waits for peers to join, then distributes credentials, CA certificates, and configuration.
> [!IMPORTANT]
> **This guide builds a three-node cluster** — the minimum we recommend for any multi-server
> deployment. Every step below is written for three servers and uses `$SPINIFEX_NODE1`,
> `$SPINIFEX_NODE2` and `$SPINIFEX_NODE3`.
>
> **Running more than three?** Install the first three exactly as described, then repeat
> Steps 1, 2, 3 and the join in Step 4 for each additional server. The only difference is in
> Step 3: the OVN database cluster stays at three members, so servers four and beyond point at
> those three instead of joining them.
**Installing on bare metal?** Step 1 below can be done either with the binary installer or by booting each server from the Spinifex ISO — see [Bootable USB Install](/docs/install-usb). The ISO installs the operating system, disks and network configuration as well as Spinifex itself, which makes it the better option for servers with no existing OS. Either way, install every server first, then return here and continue from Step 2.
A Spinifex cluster distributes services across multiple servers for high availability, data durability, and fault tolerance. Cluster formation is automatic — the init node waits for its peers to join, then distributes credentials, CA certificates, and configuration.
**Cluster Sizing:**
**Installing on bare metal?** Step 1 can be done either with the binary installer or by booting each server from the Spinifex ISO — see [Bootable USB Install](/docs/install-usb). The ISO installs the operating system, disks and network configuration as well as Spinifex itself, which makes it the better option for servers with no existing OS. Either way, install every server first, then return here and continue from Step 2.
**Three servers is the minimum recommended for a multi-server deployment.** At three or more, the OVN control-plane databases run clustered across three nodes and the cluster tolerates the loss of any one of them. Below three, OVN runs standalone on the first node, which becomes a single point of failure for control-plane operations.
### Cluster sizing
A standalone OVN outage is less severe than it sounds. `ovn-controller` has already programmed the forwarding rules into each host, so **running instances keep full networking** — east-west, north-south, NAT and security groups all continue. What stops is change: creating VPCs, launching instances, and updating security groups all require the control plane.
**Three servers is the minimum we recommend.** Three is the point at which every distributed layer can lose a node and keep running:
Servers beyond the third join as compute nodes and do not run a database, which keeps write latency stable as the cluster grows.
| Layer | What three servers gives you |
|---|---|
| **VPC networking** — OVN | Control-plane databases run clustered, surviving the loss of any one node. |
| **Object storage** — Predastore (S3) | Objects are erasure coded `RS(2,1)`, surviving the loss of any one node's shards. |
| **Block storage** — Viperblock (EBS) | Volumes are stored in Predastore, so they inherit the same durability. |
**Network Requirements:**
On one or two servers none of that holds. OVN runs standalone on the first node, and the storage metadata quorum has no majority to lose. If that node goes down, running instances keep full networking — but nothing can *change*: no new VPCs, no launches, no security group edits. See [OVN control plane on multi-node clusters](/docs/vpc-networking#ovn-control-plane-on-multi-node-clusters).
- Minimum 1 NIC per server (2 recommended for production)
- UDP port 6081 open between hosts (Geneve tunnels)
- TCP ports 4222, 4248, 6641, 6642 open between hosts (NATS, OVN)
- TCP ports 6643, 6644 open between database servers (OVN database clustering)
- TCP port 8443 and UDP ports 6660, 7660 open between hosts (Predastore S3, object shards, metadata consensus)
Servers beyond the third run the full set of services — storage, gateway and networking agents — and add their capacity to the pool. What they do not do is join the OVN database cluster, which stays at three members, so write latency there stays flat as the cluster grows.
Predastore uses the same three ports on every server, so the surface does not widen as the cluster grows. See [Object Storage Layout](#object-storage-layout) below.
### Hardware
## Object Storage Layout
Per server, for the three-node minimum:
Predastore is configured for the whole cluster in `/etc/spinifex/predastore/predastore.toml`. Each server is one `[[host]]` — a single Predastore process owning that machine's data directory and TLS identity — carrying three nodes under `[[host.node]]`:
| | Minimum | Recommended |
|---|---|---|
| **Nodes** | 3 | 3 or more |
| **RAM** | 32 GB | 128 GB |
| **CPU** | 16 cores | 32+ cores |
| **OS / Spinifex disk** | SSD | NVMe |
| **NICs** | 2 — WAN 1 GbE, LAN/VPC 10 GbE+ | 2 — WAN 10 GbE, LAN/VPC 25 GbE+ |
| Role | Port | Purpose |
|------|------|---------|
| `gate` | TCP 8443 | Serves the S3 API. Every server runs one, so any of them answers an S3 request. |
| `blob` | UDP 6660 | Holds erasure-coded object shards. One per machine. |
| `meta` | UDP 7660 | Member of the Raft quorum over global state — buckets and the object index. |
Two NICs matters more than the raw numbers. One carries WAN traffic; the other carries LAN and VPC traffic between nodes — Geneve tunnels, object shards and OVN replication all cross it, so it is the interface that wants the bandwidth. A single-NIC server will work, but inter-node storage and tunnel traffic then competes with everything going in and out of the cluster.
Ports have to be unique within a host but not across the cluster, so every machine uses the same three. Blob and meta traffic between hosts runs over QUIC, authenticated by the cluster CA; nodes on the same machine talk over an in-process pipe and open no socket, which is why a single-server install listens on 8443 alone.
### Network requirements
Reed-Solomon parameters are chosen from the cluster size, since each machine contributes exactly one blob node: two servers get `RS(1,1)`, three or more get `RS(2,1)`. `RS(2,1)` survives the loss of any one server's shards.
Open between all hosts:
You do not configure any of this by hand. `spx admin init` and `spx admin join` build the topology from the servers that actually form the cluster in Step 4, and each machine gets the same file with its own host ID recorded in `spinifex.toml`.
| Protocol and port | Used by |
|---|---|
| UDP 6081 | Geneve tunnels |
| TCP 4222, 4248 | NATS |
| TCP 6641, 6642 | OVN northbound and southbound |
| TCP 8443 | Predastore S3 gate |
| UDP 6660, 7660 | Object shards, metadata consensus |
> [!WARNING]
> **There is no upgrade path onto this layout.** A cluster installed before the object storage cutover cannot be migrated — the on-disk directory structure and the configuration schema both changed, and no migration rewrites either. Such a cluster has to be re-initialised from scratch, which discards its stored objects. Export anything you need first.
Open between the three OVN database servers (servers 1 to 3) only:
| Protocol and port | Used by |
|---|---|
| TCP 6643, 6644 | OVN database clustering |
Predastore uses the same three ports on every server, so the surface does not widen as the cluster grows. See [How multi-node storage works](#how-multi-node-storage-works).
Those are the ports your **network** has to permit between the servers — switches, upstream firewalls, and anything else in the path.
Spinifex's own **host** firewall is separate, and already carries this policy. It ships armed on the ISO path and off on the binary installer path. Two things follow from that, both of which the ISO box in Step 2 acts on:
- **The formation port needs nothing from you.** `spx admin init` opens 4432 to any source for the length of the formation window and closes it again afterwards, because a node dialling in to join is not a peer yet. The handshake behind it is TLS 1.3 with a bearer token.
- **The rest of the cluster plane is peer-scoped**, and that includes the OVN database ports Step 3 uses. Nodes that do not yet know each other cannot reach them, which is why an ISO-installed node has its firewall taken down before Step 3 and re-armed after Step 6.
## Prerequisites
**Installed from the ISO (Option B)?** The bridge is configured for you — the ISO sets up the host's network interfaces, `br-wan` included. Skip to [Instructions](#instructions).
> [!IMPORTANT]
> **Prerequisite — WAN bridge required on every node.**
> **Binary installer only — a WAN bridge is required on every node.**
>
> Before running the installer on any server, that server's WAN interface **must** already be enslaved to a Linux bridge named `br-wan`. The host IP, default route, and DHCP must all live on the bridge — not on the bare NIC.
>
> The bootstrap installer does **not** create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of `br-wan` will land in a future release.
> The binary installer does **not** create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of `br-wan` will land in a future release.
>
> **Verify on every node before continuing:**
>
> - `ip -br link show br-wan` — bridge exists and is `UP`
> - `ip route` — default route's `dev` is `br-wan`
>
> **Setup references:** [VPC Networking → Bridge Setup](/docs/vpc-networking#bridge-setup-physical-network-wiring) for the topology.
> See [VPC Networking → Bridge Setup](/docs/vpc-networking#bridge-setup-physical-network-wiring) for the topology.
## Instructions
## Step 1. Install Spinifex on Each Server
Choose one of the two methods below and apply it to **every** server in the cluster.
Choose one method and apply it to **every** server in the cluster.
**Option A — existing OS.** On a server already running Ubuntu 26.04 or Debian 13:
@@ -109,26 +144,11 @@ curl -fsSL https://install.mulgadc.com | bash
**Option B — bare metal, from the ISO.** Boot each server from the Spinifex ISO and follow [Bootable USB Install](/docs/install-usb). This installs the operating system, partitions the disks, and configures the hostname and network interfaces alongside Spinifex. The ISO installer does not form a cluster — that is what the remaining steps do.
Complete this step on all servers before continuing. The nodes must be installed and reachable from one another before the cluster is formed, because Step 4 requires every node to be available at the same time.
### Shortcut: form the cluster with one command
Steps 2 to 6 are a fixed sequence, and `scripts/install-node.sh` in the Spinifex repository runs it for you over SSH — from a workstation that can reach every server, not from the servers themselves:
```bash
scripts/install-node.sh \
--external-pool 10.0.1.100-10.0.1.150 \
--external-gateway 10.0.1.1 --external-prefix-len 24 \
server1 server2 server3
```
The first host initializes and the rest join. It resolves each node's plane addresses, rebuilds the OVN databases in clustered form, forms the cluster, restarts services and verifies the result — and `--dry-run` prints every command it would run without touching anything. It needs passwordless `sudo` on each host and requires all of them to be on the same Spinifex version.
The remaining steps document what it does, and are the path to follow when installing by hand or when something needs to be adjusted mid-way.
Complete this step on all three servers before continuing. Step 4 requires every node to be installed, reachable, and available at the same time.
## Step 2. Set Node IP Variables
On **each server**, export the management IPs for all nodes:
On **each server**, export the management IPs of all three nodes plus the region and AZ. The same values go on every server:
```bash
export SPINIFEX_NODE1=192.168.1.10
@@ -138,40 +158,29 @@ export AWS_REGION=us-east-1
export AWS_AZ=us-east-1a
```
## Converting ISO-Installed Nodes
Adding a fourth server or more? Export `SPINIFEX_NODE4` and so on alongside these — the first three stay as they are, because they remain the OVN database nodes.
Skip this section if you installed with the binary installer (Option A).
> [!NOTE]
> **ISO installs only — skip this box if you used the binary installer (Option A).**
>
> The ISO brings each server up as a **running standalone single-node cluster** with its firewall armed, so forming a cluster is a conversion rather than a fresh setup. One thing is needed, on **every** server, before Step 3:
>
> ```bash
> sudo systemctl stop spinifex.target
> sudo /usr/local/lib/spinifex/spinifex-firewall-apply disable
> ```
>
> Each node's firewall currently trusts only itself, because that is the whole cluster as far as it knows, and the cluster plane is peer-scoped — so the OVN database connections Step 3 makes between servers are dropped. Stopping `spinifex.target` is not enough: the firewall lives in the kernel and outlives the services.
>
> Turn it back on after Step 6 — see [Firewall and cluster membership](#firewall-and-cluster-membership).
The ISO installs each server as a **running standalone single-node cluster** — it initializes Spinifex, starts a standalone OVN database, brings up `spinifex.target`, and arms the host firewall at first boot. That is the right behaviour for a single server, and it means forming a cluster is a conversion rather than a fresh setup. Three extra things are needed:
## Step 3. Set Up OVN Networking
**Before Step 3**, stop services on every server:
Servers 1 to 3 run a clustered **OVN database** — the VPC networking control plane — so it survives losing any one of them. This is the only database limited to three members; storage and NATS run on every server. **Server 1 creates the cluster and must be set up first**; servers 2 and 3 then join it.
```bash
sudo systemctl stop spinifex.target
```
`--recreate-db` appears in each command below because `ovn-central` starts a standalone OVN database when the package installs, and a clustered one can only be created from scratch. It replaces that standalone database on both the binary and ISO paths.
**Before Step 3**, also turn the firewall off on every server:
```bash
sudo /usr/local/lib/spinifex/spinifex-firewall-apply disable
```
Each server's firewall currently allows internal cluster traffic only from itself, because that is the whole cluster as far as it knows. Step 3 has servers 2 and 3 connect to server 1's OVN database, and Step 4 has them connect to its formation server — neither is recognised yet, so both are blocked until the firewall is out of the way. Stopping `spinifex.target` does not do this: the firewall lives in the kernel and outlives the services.
Turn it back on after Step 6 — see [Firewall and cluster membership](#firewall-and-cluster-membership).
**In Step 4**, pass `--force` to `spx admin init` and `spx admin join`. Servers 2 and 3 each arrived with their own CA and master key from the single-node install, and joining replaces them with server 1's. `--force` is how you confirm that.
That discard is safe here — nothing has been sealed under those keys on a freshly installed node — but it is unrecoverable on a server that has been in service. `spx admin join` refuses without `--force` for exactly this reason.
## Step 3. Setup OVN Networking
All three servers run a clustered OVN database, so the control plane survives the loss of any one of them. Server 1 creates the cluster and must be set up first; servers 2 and 3 join it.
If your WAN interface is already a bridge, setup-ovn.sh auto-detects it. Otherwise use `--wan-bridge=br-wan --wan-iface=eth1` (dedicated WAN NIC).
> [!IMPORTANT]
> **`--recreate-db` destroys all logical network state** — every logical switch, router, port and ACL, and so every VPC on that node. It is required here because the `ovn-central` package starts a standalone database on install, and a clustered database can only be created from scratch. That is safe on a freshly installed server and is **not** safe on one already running workloads. If you are converting a server that has been in service, back up its NB database first.
If your WAN interface is already a bridge, `setup-ovn.sh` auto-detects it. Otherwise add `--wan-bridge=br-wan --wan-iface=eth1` for a dedicated WAN NIC.
**Server 1 — create the cluster:**
@@ -205,7 +214,7 @@ sudo /usr/local/share/spinifex/setup-ovn.sh \
--encap-ip=$SPINIFEX_NODE3
```
**Servers 4 and beyond** are compute nodes and run no database. Point them at all three database servers so they survive any one of them failing:
**Servers 4 and beyond** — repeat for each one, substituting its own address. They point at the three OVN database servers rather than joining the database cluster, so they survive any one of those failing:
```bash
sudo /usr/local/share/spinifex/setup-ovn.sh \
@@ -213,30 +222,35 @@ sudo /usr/local/share/spinifex/setup-ovn.sh \
--encap-ip=$SPINIFEX_NODE4
```
Verify the database cluster formed, then that all chassis registered:
Verify the OVN database cluster formed, then that every chassis registered with it:
```bash
sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
sudo ovn-sbctl show
```
`cluster/status` should list three servers with one leader. If it reports a standalone database, the cluster did not form — re-check that `--db-cluster-local-addr` was passed on every database server.
`cluster/status` should list three servers with one leader, and `ovn-sbctl show` a chassis for every node in the cluster. If `cluster/status` reports a standalone database, the OVN cluster did not form — re-check that `--db-cluster-local-addr` was passed on servers 1, 2 and 3.
## Step 4. Form the Cluster
Run init and join concurrently — init blocks until all nodes join.
Run init and join **concurrently** — init blocks until all nodes have joined.
If you installed from the ISO, add `--force` to every command below (see [Converting ISO-Installed Nodes](#converting-iso-installed-nodes)).
`--force` is in every command below so the sequence is identical whichever way you installed. It does the work on ISO-installed servers, which arrive as their own single-node cluster: joining replaces that server's CA and master key with server 1's, and `--force` is the confirmation. On `spx admin init` it is idempotent — existing keys, credentials and CA are preserved, and only the config files and server certificate are refreshed. On a freshly installed server there is nothing to lose either way.
**Server 1 — Initialize:**
> [!WARNING]
> Do not point these commands at a server that has already been in service. Joining discards its master key, orphaning every volume and fragment sealed under it. That is what `--force` overrides, and it is unrecoverable.
**Server 1 — initialize:**
```bash
sudo spx admin init \
sudo spx admin init --force \
--node node1 --nodes 3 \
--bind $SPINIFEX_NODE1 --cluster-bind $SPINIFEX_NODE1 \
--port 4432 --region $AWS_REGION --az $AWS_AZ
```
`--nodes 3` is the number of servers init waits for. Set it to your total node count if you are building a larger cluster.
The init output displays the join command including the token:
```
@@ -248,25 +262,29 @@ The init output displays the join command including the token:
sudo spx admin join --host 10.0.0.1:4432 --token spx_join_a8Bf3x9Kz2mN --node <name> --bind <ip>
```
**Server 2 — Join** (while init is running):
Take the **token** from that output, but run the commands below rather than the line it prints — they add `--force` and `--cluster-bind`.
**Server 2 — join** (while init is still running):
```bash
sudo spx admin join \
sudo spx admin join --force \
--node node2 --bind $SPINIFEX_NODE2 --cluster-bind $SPINIFEX_NODE2 \
--host $SPINIFEX_NODE1:4432 --token <token-from-init-output> \
--region $AWS_REGION --az $AWS_AZ
```
**Server 3 — Join** (while init is running):
**Server 3 — join** (while init is still running):
```bash
sudo spx admin join \
sudo spx admin join --force \
--node node3 --bind $SPINIFEX_NODE3 --cluster-bind $SPINIFEX_NODE3 \
--host $SPINIFEX_NODE1:4432 --token <token-from-init-output> \
--region $AWS_REGION --az $AWS_AZ
```
**Note:** The join token expires 30 minutes after init by default. For larger deployments with slower provisioning, use `--token-ttl 2h`
Each additional server runs the same join command with its own `--node` name and `--bind` address.
**Note:** the join token expires 30 minutes after init by default. For larger deployments with slower provisioning, use `--token-ttl 2h`.
## Step 5. Start Services
@@ -276,16 +294,77 @@ On **all servers**:
sudo systemctl start spinifex.target
```
## Step 6. Verify
## Step 6. Verify the Cluster
From any node:
Run these from any node. Together they confirm that every server joined, that services are healthy on each, and that capacity is being pooled across the cluster.
**1. Every node is present and Ready.**
```bash
spx get nodes
```
```
spinifex@node1:~$ spx get nodes
NAME | STATUS | ROLES | IP | REGION | AZ | UPTIME | VMs | SERVICES
node1 | Ready | nats:follower | 10.2.0.2 | us-east-1 | us-east-1a | 21h27m | 0 | nats,predastore,viperblock,daemon,awsgw,vpcd,ui
node2 | Ready | nats:follower | 10.2.0.3 | us-east-1 | us-east-1a | 21h27m | 1 | nats,predastore,viperblock,daemon,awsgw,vpcd,ui
node3 | Ready | nats:leader | 10.2.0.4 | us-east-1 | us-east-1a | 21h27m | 0 | nats,predastore,viperblock,daemon,awsgw,vpcd,ui
```
What to check:
- **Every server you installed is listed.** A missing node never joined — see [Nodes not joining](#nodes-not-joining).
- **`STATUS` is `Ready`** on all of them. `NotReady` means the node is in the cluster configuration but is not answering, so start with `spinifex.target` on that host.
- **Exactly one `nats:leader`.** The rest are followers.
- **`SERVICES` lists the same set on every node.** A short list means something failed to start there; check `systemctl status` for the missing unit.
**2. Capacity is pooled across the cluster.**
```bash
spx top nodes
```
```
spinifex@node1:~$ spx top nodes
NAME | CPU (used/total) | MEM (used/total) | GPU (used/total) | VMs
node1 | 0/64 | 0Mi/220.2Gi | - | 0
node2 | 2/64 | 2.8Gi/251.7Gi | - | 1
node3 | 0/64 | 0Mi/251.7Gi | - | 0
INSTANCE TYPE | AVAILABLE | VCPU | MEMORY
c6a.12xlarge | 3 | 48 | 96.0Gi
c6a.16xlarge | 0 | 64 | 128.0Gi
c6a.24xlarge | 0 | 96 | 192.0Gi
c6a.2xlarge | 21 | 8 | 16.0Gi
c6a.4xlarge | 9 | 16 | 32.0Gi
c6a.8xlarge | 3 | 32 | 64.0Gi
c6a.large | 92 | 2 | 4.0Gi
c6a.xlarge | 45 | 4 | 8.0Gi
c6i.12xlarge | 3 | 48 | 96.0Gi
c6i.16xlarge | 0 | 64 | 128.0Gi
c6i.24xlarge | 0 | 96 | 192.0Gi
c6i.2xlarge | 21 | 8 | 16.0Gi
c6i.4xlarge | 9 | 16 | 32.0Gi
c6i.8xlarge | 3 | 32 | 64.0Gi
c6i.large | 92 | 2 | 4.0Gi
c6i.xlarge | 45 | 4 | 8.0Gi
m6a.12xlarge | 3 | 48 | 192.0Gi
```
The top table is per-node CPU, memory and GPU usage. The bottom table is what the cluster can actually launch right now: `AVAILABLE` is the number of instances of that type that would currently fit across all nodes. A type showing `0` does not fit on any single node — instances are not split across servers, so the largest type you can launch is bounded by your biggest node, not by the cluster total.
If capacity looks like a single server rather than the sum of your nodes, the others have not joined.
**3. The AWS API answers.**
```bash
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
```
If this returns a list of available instance types, your cluster is working.
A list of instance types means the gateway, IAM and the cluster behind them are all working.
**Congratulations! Your Spinifex cluster is installed.**
@@ -295,7 +374,7 @@ Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import a
Spinifex ships an optional host firewall. It divides the node's ports into two groups:
| | Ports | Who can reach them |
| Group | Ports | Who can reach them |
|---|---|---|
| **Public** | SSH, 443, 3000 (console), 8443 (S3), 9999 (AWS gateway), 53 (DNS) | anyone |
| **Internal** | OVN, NATS, formation, Geneve and the rest of the cluster plane | **cluster members only** |
@@ -309,7 +388,7 @@ The internal group is the point. Before this existed, OVN and NATS were reachabl
| How the node was installed | Firewall |
|---|---|
| From the ISO | **on** |
| `curl \| bash` or `setup.sh` | **off** |
| Binary installer (`curl \| bash`) or `setup.sh` | **off** |
| `setup.sh --firewall=on` | **on** |
The binary installer defaults to off deliberately: it runs on servers that already have an operating system and services on them, and switching on a default-deny policy uninvited could cut off something Spinifex knows nothing about. **For production, turn it on** — either at install time:
@@ -344,10 +423,12 @@ Restarting the daemon is what re-arms it: the node rebuilds its peer list from t
### Checking it
```bash
sudo nft list table inet spinifex_filter | grep spinifex_peers
sudo nft list table inet spinifex_filter
```
Every node's addresses should be listed — on a multi-NIC node, that is its WAN, LAN and VPC addresses, so expect several entries per node. A missing node means its cluster traffic is being dropped.
The peer list is an nft variable, expanded when the rules load, so it does not appear under a name of its own. Look instead at the `ip saddr { ... }` addresses on the cluster-plane rules — the ones accepting 4222, 6641, 6642 and the rest.
Every node's addresses should be there. On a multi-NIC node that means its WAN, LAN and VPC addresses, so expect several entries per node. A missing node means its cluster traffic is being dropped.
Dropped packets are logged, rate-limited, so this tells you whether a connection problem is the firewall or something else:
@@ -355,9 +436,21 @@ Dropped packets are logged, rate-limited, so this tells you whether a connection
sudo journalctl -k | grep 'spinifex-fw drop'
```
### Adding a node later
## How Multi-Node Storage Works
Expanding an existing cluster is the same shape as forming one: the node you are adding is not a member yet. Turn the firewall off on the existing nodes and on the new one, add the node, then re-arm all of them. Verify with the `nft list` command above that every node now lists the newcomer.
Background reading — you do not configure any of this by hand. `spx admin init` and `spx admin join` build the topology from the servers that actually form the cluster in Step 4, and each machine gets the same file with its own host ID recorded in `spinifex.toml`.
Predastore is configured for the whole cluster in `/etc/spinifex/predastore/predastore.toml`. Each server is one `[[host]]` — a single Predastore process owning that machine's data directory and TLS identity — carrying three nodes under `[[host.node]]`:
| Role | Port | Purpose |
|---|---|---|
| `gate` | TCP 8443 | Serves the S3 API. Every server runs one, so any of them answers an S3 request. |
| `blob` | UDP 6660 | Holds erasure-coded object shards. One per machine. |
| `meta` | UDP 7660 | Member of the Raft quorum over global state — buckets and the object index. |
Ports have to be unique within a host but not across the cluster, so every machine uses the same three. Blob and meta traffic between hosts runs over QUIC, authenticated by the cluster CA; nodes on the same machine talk over an in-process pipe and open no socket, which is why a single-server install listens on 8443 alone.
Reed-Solomon parameters are chosen from the cluster size, since each machine contributes exactly one blob node: two servers get `RS(1,1)`, three or more get `RS(2,1)`. `RS(2,1)` survives the loss of any one server's shards — another reason three nodes is the recommended minimum.
## Troubleshooting
@@ -369,7 +462,9 @@ The init command must still be running when join executes. If init exited, re-ru
curl -sk https://$SPINIFEX_NODE1:4432/health
```
If that command hangs rather than failing quickly, it is the firewall on node 1 — a refused connection means nothing is listening, but a hang means the packets are being dropped. Node 1 does not recognise the joining node as a cluster member yet. Confirm it on node 1:
A hang rather than a quick failure means packets are being dropped; a refused connection means nothing is listening.
If it hangs, check node 1's init output before blaming the firewall — `spx admin init` opens the formation port itself while it waits, so this is usually not the cause. It prints `⚠️ Could not open port 4432 in the host firewall` when that fails, which is the case where it is. Confirm on node 1:
```bash
sudo journalctl -k | grep 'spinifex-fw drop'
@@ -389,7 +484,7 @@ Safe on a freshly installed node. On one that has been in service it orphans eve
sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
```
If this reports a standalone database rather than three servers, the database was created before the cluster flags were supplied. A clustered database can only be created from scratch — re-run Step 3 with `--recreate-db`, remembering that it discards all logical network state.
If this reports a standalone database rather than three servers, the OVN database was created before the cluster flags were supplied. A clustered one can only be created from scratch — re-run Step 3 with `--recreate-db`.
### OVN Chassis Not Registering
docs/install/install-usb/README.md
+3 -3
View File
@@ -135,7 +135,7 @@ AWS_PROFILE=spinifex aws ec2 describe-instance-types
That profile is the operator account, with administrator access. Copy it to your workstation to drive the node remotely — you will also need the cluster CA, available unauthenticated from `https://<node-address>:3000/api/ca.pem`.
From here, [Setting Up Your Cluster](/docs/admin/setting-up-your-cluster) walks through importing an AMI, creating a key pair and a VPC, and launching your first instance.
From here, [Setting Up Your Cluster](/docs/setting-up-your-cluster) walks through importing an AMI, creating a key pair and a VPC, and launching your first instance.
### Building a Cluster
@@ -180,10 +180,10 @@ sudo systemctl restart spinifex-daemon
The node rewrites its peer list from the cluster it is now part of and re-arms itself, including at boot. Confirm every server can see the others:
```bash
sudo nft list table inet spinifex_filter | grep spinifex_peers
sudo nft list table inet spinifex_filter
```
The addresses of **all** your servers should appear. If one is missing, that server's cluster traffic will be blocked — see [Firewall and cluster membership](/docs/install-multi-node#firewall-and-cluster-membership).
Check the `ip saddr { ... }` addresses on the cluster-plane rules — the peer list is an nft variable expanded at load time, so it has no name of its own in the output. The addresses of **all** your servers should appear. If one is missing, that server's cluster traffic will be blocked — see [Firewall and cluster membership](/docs/install-multi-node#firewall-and-cluster-membership).
### Setup Complete
docs/reference-architectures/cisco-ucs-llm-serving/README.md
+3 -3
View File
@@ -19,9 +19,9 @@ resources:
- title: "GPU Passthrough"
url: "/docs/gpu-passthrough"
- title: "Cisco UCS: AWS-compatible cloud at the edge (companion document)"
url: "../cisco-ucs-platform-benchmark/README.md"
url: "/docs/cisco-ucs-platform-benchmark"
- title: "Vision pipeline (companion document)"
url: "../cisco-ucs-vision-pipeline/README.md"
url: "/docs/cisco-ucs-vision-pipeline"
- title: "Benchmark and workload materials (CISCO-refarch)"
url: "https://github.com/tomnewton-mulga/CISCO-refarch"
- title: "vLLM"
@@ -45,7 +45,7 @@ cluster, comparing Intel AMX-accelerated CPU serving on an `m8i.2xlarge`-class i
against NVIDIA L4 GPU serving on a `g6.2xlarge`-class instance — a real,
production-representative LLM-serving stack, with the two accelerators measured independently at matched model, version, and serving configuration.
**Companion architectures:** [Cisco UCS: AWS-compatible cloud at the edge](../cisco-ucs-platform-benchmark/README.md) · [Spinifex Vision Pipeline on Cisco UCS](../cisco-ucs-vision-pipeline/README.md)
**Companion architectures:** [Cisco UCS: AWS-compatible cloud at the edge](/docs/cisco-ucs-platform-benchmark) · [Spinifex Vision Pipeline on Cisco UCS](/docs/cisco-ucs-vision-pipeline)
### Platform
docs/reference-architectures/cisco-ucs-platform-benchmark/README.md
+3 -3
View File
@@ -39,7 +39,7 @@ EBS, S3 and EKS-compatible APIs on the cluster, so the operational model is
familiar: use the AWS CLI, SDKs, Terraform or OpenTofu, Kubernetes manifests and
CI/CD systems already used for AWS. The change is the endpoint, not the workflow.
**Companion architectures:** [Vision Pipeline on Cisco UCS](../cisco-ucs-vision-pipeline/README.md) shows a local AI pipeline that streams inputs from S3; [vLLM Serving on Cisco UCS](../cisco-ucs-llm-serving/README.md) shows CPU and GPU model serving on the same cluster.
**Companion architectures:** [Vision Pipeline on Cisco UCS](/docs/cisco-ucs-vision-pipeline) shows a local AI pipeline that streams inputs from S3; [vLLM Serving on Cisco UCS](/docs/cisco-ucs-llm-serving) shows CPU and GPU model serving on the same cluster.
### Platform
@@ -219,8 +219,8 @@ kubectl get pods -n default
```
Kubernetes pod rescheduling is the platform's primary answer to node failure for
scheduled workloads. The companion [vision pipeline](../cisco-ucs-vision-pipeline/README.md)
and [vLLM serving](../cisco-ucs-llm-serving/README.md) architectures use raw EC2 instances;
scheduled workloads. The companion [vision pipeline](/docs/cisco-ucs-vision-pipeline)
and [vLLM serving](/docs/cisco-ucs-llm-serving) architectures use raw EC2 instances;
EKS is the pattern to reach for when automatic rescheduling matters over manual placement.
### 5. Storage benchmark
docs/reference-architectures/cisco-ucs-vision-pipeline/README.md
+2 -2
View File
@@ -25,7 +25,7 @@ resources:
- title: "GPU Passthrough"
url: "/docs/gpu-passthrough"
- title: "Cisco UCS: AWS-compatible cloud at the edge (companion document)"
url: "../cisco-ucs-platform-benchmark/README.md"
url: "/docs/cisco-ucs-platform-benchmark"
- title: "Ultralytics YOLO"
url: "https://docs.ultralytics.com"
- title: "AWS: Accelerate CPU-based AI inference with Intel AMX on EC2"
@@ -48,7 +48,7 @@ cluster running Spinifex: real-time object detection plus a vision-language mode
producing plain-English scene descriptions, both streaming from the same shared Predastore
(S3-compatible) bucket as two entirely independent EC2-compatible instances.
**Companion architectures:** [Cisco UCS: AWS-compatible cloud at the edge](../cisco-ucs-platform-benchmark/README.md) · [vLLM Serving on Cisco UCS: Intel AMX vs NVIDIA L4](../cisco-ucs-llm-serving/README.md)
**Companion architectures:** [Cisco UCS: AWS-compatible cloud at the edge](/docs/cisco-ucs-platform-benchmark) · [vLLM Serving on Cisco UCS: Intel AMX vs NVIDIA L4](/docs/cisco-ucs-llm-serving)
### Platform
docs/security/flaw-remediation-policy/README.md
+5 -5
View File
@@ -39,7 +39,7 @@ resources:
- [3. Remediation SLAs](#3-remediation-slas)
- [4. Reporting Workflow](#4-reporting-workflow)
- [5. Operator Responsibilities](#5-operator-responsibilities)
- [6. Evidence](#6-evidence)
- [6. Evidence](#6-evidence-and-record-keeping)
- [7. Operator Checklist](#7-operator-checklist)
---
@@ -60,7 +60,7 @@ resources:
Spinifex ships a short dependency chain and identification is already automated: Dependabot raises security PRs the moment GitHub links a CVE to a shipped dependency, `govulncheck` runs on every PR, and GitHub Security Advisories watch all three `mulgadc/*` repositories.
Sections map to the three scored objectives: [§1](#1-identification) names the identification sources (objectives [a]/[b]); [§3](#3-remediation-slas) and [§4](#4-reporting-workflow) set the report and correct timeframes (objectives [c]–[f]). [§5](#5-operator-responsibilities) delineates layers the operator owns, and [§6](#6-evidence) covers the records needed to demonstrate compliance at assessment.
Sections map to the three scored objectives: [§1](#1-identification) names the identification sources (objectives [a]/[b]); [§3](#3-remediation-slas) and [§4](#4-reporting-workflow) set the report and correct timeframes (objectives [c]–[f]). [§5](#5-operator-responsibilities) delineates layers the operator owns, and [§6](#6-evidence-and-record-keeping) covers the records needed to demonstrate compliance at assessment.
## 1. Identification
@@ -93,7 +93,7 @@ CVSS v3.1 base score from the upstream advisory; otherwise scored by maintainers
## 3. Remediation SLAs
Clocks start at the moment of identification ([§1](#1-identification-sources)). All three objectives — identify [a], report [c], correct [e] — are time-bound.
Clocks start at the moment of identification ([§1](#1-identification)). All three objectives — identify [a], report [c], correct [e] — are time-bound.
| Severity | Identify | Report | Correct |
|----------|----------|--------|---------|
@@ -130,7 +130,7 @@ Operators own remediation for layers Spinifex does not ship:
| Host OS + kernel | Track distribution advisories (Debian DSA, Ubuntu USN) at matching cadence. |
| OVN / OVS | Patched via the operator's package channel alongside the OS. |
| QEMU / KVM / libvirt | As above — hypervisor CVEs affect tenant isolation. |
| Host AV / EDR agent | See [Malware Protection §2](/docs/security/malware-protection#2-update-requirements-sil1-3144). |
| Host AV / EDR agent | See [Malware Protection §2](/docs/malware-protection#2-update-requirements-sil1-3144). |
Operators should subscribe to the `mulgadc/spinifex` release feed (Watch → Custom → Releases + Security Advisories) so Critical/High advisories arrive via GitHub's notification channel rather than polling.
@@ -151,4 +151,4 @@ These demonstrate objectives [b], [d], [f] — "within the specified time" — i
- Change-management records patch-apply timestamps for every Spinifex release.
- Operator subscribes to `mulgadc/spinifex`, `mulgadc/predastore`, `mulgadc/viperblock` releases and advisories.
- OS / kernel / hypervisor patching cadence documented and at least as strict as [§3](#3-remediation-slas).
- Annual review confirms at least one patch cycle completed within SLA in the prior 12 months, with evidence per [§6](#6-evidence).
- Annual review confirms at least one patch cycle completed within SLA in the prior 12 months, with evidence per [§6](#6-evidence-and-record-keeping).
docs/security/media-sanitization/README.md
+8 -8
View File
@@ -37,8 +37,8 @@ resources:
- [Approach](#approach)
- [1. Media in Scope](#1-media-in-scope)
- [2. Sanitization Method Selection](#2-sanitization-method-selection)
- [3. Volume-Level Sanitization — Before Reuse (MP.L1-3.8.3 [b])](#3-volume-level-sanitization--before-reuse-mpl1-383-b)
- [4. Whole-Drive and Node Decommissioning — Before Disposal (MP.L1-3.8.3 [a])](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a)
- [3. Volume-Level Sanitization — Before Reuse (MP.L1-3.8.3 [b])](#3-volume-level-sanitization-before-reuse-mpl1-383-b)
- [4. Whole-Drive and Node Decommissioning — Before Disposal (MP.L1-3.8.3 [a])](#4-whole-drive-and-node-decommissioning-before-disposal-mpl1-383-a)
- [5. Key Destruction](#5-key-destruction)
- [6. Removable and Backup Media](#6-removable-and-backup-media)
- [7. Evidence and Record Keeping](#7-evidence-and-record-keeping)
@@ -92,7 +92,7 @@ The method depends on (a) media type and (b) whether the media remains inside th
| Scenario | Minimum method | Reference |
|----------|----------------|-----------|
| Encrypted volume returning to free pool, DEK deletable | **Cryptographic Erase** (Purge) via DEK deletion | [§3](#3-volume-level-sanitization--before-reuse-mpl1-383-b) |
| Encrypted volume returning to free pool, DEK deletable | **Cryptographic Erase** (Purge) via DEK deletion | [§3](#3-volume-level-sanitization-before-reuse-mpl1-383-b) |
| SSD / NVMe leaving the node | **Cryptographic Erase** + **Purge** (SANITIZE block-erase or ATA Secure Erase) | [§4.1](#41-nvme-ssd) |
| Magnetic HDD leaving the node, operational | **Purge** (single-pass overwrite with verify, or ATA Secure Erase on drives that support it) | [§4.3](#43-hdd) |
| Magnetic HDD leaving the node, faulty (not writable) | **Destroy** (degauss followed by shred/incinerate) | [§4.3](#43-hdd) |
@@ -121,7 +121,7 @@ This is the sanitization-before-reuse path for every in-platform object. Operato
### 3.2 What Happens to the Physical Storage
Cryptographic erase leaves ciphertext chunks in Predastore. These chunks are eventually overwritten as new volumes reuse the space. For CMMC purposes, the ciphertext without the DEK satisfies Purge. The underlying drive still requires whole-drive sanitization when it eventually leaves the cluster — see [§4](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a).
Cryptographic erase leaves ciphertext chunks in Predastore. These chunks are eventually overwritten as new volumes reuse the space. For CMMC purposes, the ciphertext without the DEK satisfies Purge. The underlying drive still requires whole-drive sanitization when it eventually leaves the cluster — see [§4](#4-whole-drive-and-node-decommissioning-before-disposal-mpl1-383-a).
## 4. Whole-Drive and Node Decommissioning — Before Disposal (MP.L1-3.8.3 [a])
@@ -211,11 +211,11 @@ Cryptographic erase relies on the key being unrecoverable. A drive that previous
| `/etc/spinifex/master.key` | Node system disk (every node) | Overwrite the file (`shred -u /etc/spinifex/master.key`) before the drive is sanitized, in addition to drive sanitization. Once the master key is gone cluster-wide, every DEK it wrapped is effectively destroyed. |
| Cluster CA private key (`/etc/spinifex/ca.key`) | Leader node system disk | Same treatment. Loss of the CA key does not sanitize data, but prevents impersonation of the cluster identity post-disposal. |
| Per-node TLS private key (`/etc/spinifex/server.key`) | Each node system disk | `shred -u` before drive sanitization. |
| Wrapped DEKs in NATS KV | `/var/lib/spinifex/nats/` on NATS-carrying nodes | Deleted via the `DeleteVolume` API; the KV record is compacted. Sanitization of the NATS JetStream disk per [§4](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a) ensures no recoverable copies. |
| Wrapped DEKs in NATS KV | `/var/lib/spinifex/nats/` on NATS-carrying nodes | Deleted via the `DeleteVolume` API; the KV record is compacted. Sanitization of the NATS JetStream disk per [§4](#4-whole-drive-and-node-decommissioning-before-disposal-mpl1-383-a) ensures no recoverable copies. |
| Removable master-key media (USB) | USB flash token | Physically destroy the token when the cluster is decommissioned; alternatively retain inside a secured location as evidence of cluster-wide cryptographic erase. Do not reuse for another cluster without full sanitization (see [§6](#6-removable-and-backup-media)). |
| Backup-copy keys | Any off-site or escrow copy | Destroy simultaneously with the primary. A surviving backup defeats the cryptographic erase. |
If even one copy of the master key survives and the encrypted drives are recoverable, cryptographic erase has **not** been achieved. Track every key copy in the device register ([Physical Security Guide §5](/docs/security/physical-security-guide#5-manage-physical-access-devices-pel1-3105)).
If even one copy of the master key survives and the encrypted drives are recoverable, cryptographic erase has **not** been achieved. Track every key copy in the device register ([Physical Security Guide §5](/docs/physical-security-guide#5-manage-physical-access-devices-pel1-3105)).
## 6. Removable and Backup Media
@@ -228,7 +228,7 @@ If even one copy of the master key survives and the encrypted drives are recover
| Off-site cloud backup | Issue the cloud provider's delete-and-purge API; retain the provider's deletion-confirmation receipt. For cryptographically wrapped backups, destroy the wrapping key cluster-side and retain that deletion as evidence. |
| Printed material (console photos, recovery codes, admin handover sheets) | Cross-cut shred. |
Any removable medium that has entered the protected boundary and been written to must be tracked in the device register (see [Physical Security Guide §5](/docs/security/physical-security-guide#5-manage-physical-access-devices-pel1-3105)) and accounted for at decommissioning.
Any removable medium that has entered the protected boundary and been written to must be tracked in the device register (see [Physical Security Guide §5](/docs/physical-security-guide#5-manage-physical-access-devices-pel1-3105)) and accounted for at decommissioning.
## 7. Evidence and Record Keeping
@@ -255,5 +255,5 @@ Use this list to confirm a Spinifex deployment meets MP.L1-3.8.3:
- Every backup copy of any key is destroyed at the same time as the primary; surviving backup copies are tracked and accounted for.
- Removable media (USB, tape, optical) is destroyed rather than erased unless the medium is an SED with documented cryptographic erase.
- Decommissioning records capture method, tool output, verification, final disposition, and operator; certificates of destruction are reconciled against the asset register.
- Drives awaiting sanitization or awaiting pickup by a destruction vendor are held inside the physical protection boundary ([Physical Security Guide §1](/docs/security/physical-security-guide#1-protected-assets)) with chain of custody logged.
- Drives awaiting sanitization or awaiting pickup by a destruction vendor are held inside the physical protection boundary ([Physical Security Guide §1](/docs/physical-security-guide#1-protected-assets)) with chain of custody logged.
- System security plan references this guide and names the sanitization tooling, destruction vendor (if any), retention period for records, and the security owner attesting annually.
docs/security/physical-security-guide/README.md
+2 -2
View File
@@ -145,7 +145,7 @@ Every entry into a protected space must produce a log entry capturing:
### 4.3 Integration with Spinifex Logs
Forward physical-access events to the same SIEM or log collector used for Spinifex service logs (see [Malware Protection §3](/docs/security/malware-protection#3-scan-schedule-sil1-3145)) so a `master.key` access attempt can be correlated with a physical entry into the rack.
Forward physical-access events to the same SIEM or log collector used for Spinifex service logs (see [Malware Protection §3](/docs/malware-protection#3-scan-schedule-sil1-3145)) so a `master.key` access attempt can be correlated with a physical entry into the rack.
## 5. Manage Physical Access Devices (PE.L1-3.10.5)
@@ -200,7 +200,7 @@ Use this list to confirm a site meets the four CMMC practices before admitting S
- CCTV or equivalent monitoring covers the rack fronts, rears, and console positions, with 30-day minimum retention.
- Physical access logs are produced for every entry, retained 12 months, and reviewed on the cadence in [§4.2](#42-retention-and-review).
- Door-forced, door-held, and out-of-hours events alert the on-call operator.
- Physical access events forwarded to the SIEM used for Spinifex service logs (see [Malware Protection §3](/docs/security/malware-protection#3-scan-schedule-sil1-3145)).
- Physical access events forwarded to the SIEM used for Spinifex service logs (see [Malware Protection §3](/docs/malware-protection#3-scan-schedule-sil1-3145)).
- Device register reconciles quarterly against the access-control system and physical key count; discrepancies are closed before sign-off.
- Lost-device and termination procedures trigger immediate revocation per [§5.2](#52-control-b).
- System security plan references this guide and records the facility, access-control product, CCTV retention, and log-forwarding destination.
docs/terraform-workbooks/eks-gitops-argocd/README.md
+3 -3
View File
@@ -27,7 +27,7 @@ resources:
# Terraform: GitOps on EKS (Argo CD + EBS-CSI)
> The top rung of the EKS ladder. It extends [EKS HTTPS Ingress](../eks-https-ingress) and changes how the app is delivered: the **Argo CD** addon syncs a more elaborate Spinifex-themed app from a **git repo**, an **EBS-CSI** PersistentVolume (Viperblock-backed) holds its state, and it is still served over **HTTPS** through the AWS Load Balancer Controller + ACM.
> The top rung of the EKS ladder. It extends [EKS HTTPS Ingress](/docs/eks-https-ingress) and changes how the app is delivered: the **Argo CD** addon syncs a more elaborate Spinifex-themed app from a **git repo**, an **EBS-CSI** PersistentVolume (Viperblock-backed) holds its state, and it is still served over **HTTPS** through the AWS Load Balancer Controller + ACM.
## Table of Contents
@@ -39,7 +39,7 @@ resources:
## Overview
This workbook keeps everything from [EKS HTTPS Ingress](../eks-https-ingress) — the public/private VPC, the NAT gateway, the LBC addon, the ACM cert, the HTTPS Ingress — and adds the parts you reach for once an app is real: **GitOps delivery** and **persistent storage**.
This workbook keeps everything from [EKS HTTPS Ingress](/docs/eks-https-ingress) — the public/private VPC, the NAT gateway, the LBC addon, the ACM cert, the HTTPS Ingress — and adds the parts you reach for once an app is real: **GitOps delivery** and **persistent storage**.
Three changes over the previous rung:
@@ -86,7 +86,7 @@ Terraform manages the cluster, the addons, the ACM cert, and the HTTPS Ingress;
- Spinifex running with an `eks-node` image carrying the LBC, Argo CD, and EBS-CSI bundles
- OpenTofu or Terraform, plus `kubectl`, the AWS CLI, and Docker
- The demo image built and pushed to ECR (see [`../demo-app/README.md`](../demo-app/README.md))
- The demo image built and pushed to ECR (see [`docs/terraform-workbooks/demo-app`](https://github.com/mulgadc/spinifex/blob/main/docs/terraform-workbooks/demo-app/README.md))
- A git repo with the app manifests ([`mulgadc/eks-demo-app`](https://github.com/mulgadc/eks-demo-app)); a read-only PAT if it is private
## Instructions
docs/terraform-workbooks/eks-https-ingress/README.md
+3 -3
View File
@@ -37,7 +37,7 @@ resources:
## Overview
This is the second rung of the EKS ladder. It takes the cluster + demo app from [EKS Quickstart](../eks-quickstart) and publishes the app the way you would in production: a Kubernetes `Ingress` reconciled by the **AWS Load Balancer Controller (LBC)** into an internet-facing **Application Load Balancer** that terminates TLS with an **ACM** certificate.
This is the second rung of the EKS ladder. It takes the cluster + demo app from [EKS Quickstart](/docs/eks-quickstart) and publishes the app the way you would in production: a Kubernetes `Ingress` reconciled by the **AWS Load Balancer Controller (LBC)** into an internet-facing **Application Load Balancer** that terminates TLS with an **ACM** certificate.
To make room for an internet-facing ALB with private workers, the network grows from the quickstart's two public subnets to a public/private split: the ALB and a **NAT gateway** sit in the public subnets, the workers sit in the private subnets and reach ECR through the NAT. The cluster is tagged `spinifex.io/managed-ingress = "false"`, which disables K3s' built-in traefik/servicelb so the LBC owns ingress.
@@ -74,13 +74,13 @@ The LBC is installed as a managed **addon** (`aws_eks_addon`). Spinifex wires th
- **`spinifex.io/managed-ingress = "false"`** disables the K3s built-in traefik/servicelb so the LBC owns ingress, matching AWS parity. Omit the tag (or set it `true`) and the built-in ingress stays on.
- The ALB targets the workers' **NodePort** (`target-type: instance`). The one SG rule admits the VPC CIDR on that port so the in-VPC ALB can reach the workers; the worker SG is otherwise auto-managed.
- Spinifex ACM supports **ImportCertificate only** — the cert is generated by the `tls` provider and imported, not requested.
- **Leave `addon_version` unset** (see the [quickstart/addons notes](../eks-quickstart)) — the AWS provider and the Spinifex catalog disagree on the version string format, so let the server choose.
- **Leave `addon_version` unset** (see the [quickstart/addons notes](/docs/eks-quickstart)) — the AWS provider and the Spinifex catalog disagree on the version string format, so let the server choose.
**Prerequisites:**
- Spinifex installed and running, with the `eks-node` image (carrying the LBC bundle) available
- OpenTofu or Terraform, plus `kubectl`, the AWS CLI, and Docker
- The demo image built and pushed to ECR (see [`../demo-app/README.md`](../demo-app/README.md))
- The demo image built and pushed to ECR (see [`docs/terraform-workbooks/demo-app`](https://github.com/mulgadc/spinifex/blob/main/docs/terraform-workbooks/demo-app/README.md))
## Instructions
docs/terraform-workbooks/eks-quickstart/README.md
+1 -1
View File
@@ -109,7 +109,7 @@ Set the worker count with `node_desired_size` (`1` for a single node, `3` for an
tofu apply -var node_desired_size=3
```
Once the cluster is `ACTIVE`, **build and push the demo image** to the ECR repository this created (full commands in [`../demo-app/README.md`](../demo-app/README.md)):
Once the cluster is `ACTIVE`, **build and push the demo image** to the ECR repository this created (full commands in [`docs/terraform-workbooks/demo-app`](https://github.com/mulgadc/spinifex/blob/main/docs/terraform-workbooks/demo-app/README.md)):
```bash
cd ../demo-app