mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
chore: cleanup docs
This commit is contained in:
Julian Sommer
@@ -27,7 +27,7 @@ resources:
|
||||
|
||||
The `spx` binary is the central administration tool for managing your Spinifex infrastructure. It provides commands for cluster initialization, account management, node operations, VM lifecycle, and service control.
|
||||
|
||||
**Binary location:** `/usr/local/bin/spx` (binary install) or `~/Development/mulga/spinifex/bin/spx` (source install)
|
||||
**Binary location:** `/usr/local/bin/spx`
|
||||
|
||||
All services in the Spinifex platform are managed through this single binary.
|
||||
|
||||
@@ -35,7 +35,7 @@ All services in the Spinifex platform are managed through this single binary.
|
||||
|
||||
## Account Management
|
||||
|
||||
Create a new isolated account. This provisions a sequential 12-digit account ID, an `admin` user with an `AdministratorAccess` policy attached, and an access key pair. The credentials are written to `~/.aws/credentials` and `~/.aws/config` under a `spinifex-<name>` profile automatically. Requires a running cluster; run from a cluster node:
|
||||
Create a new isolated account. This provisions a sequential 12-digit account ID, an `admin` user with an `AdministratorAccess` policy attached, and an access key pair. The credentials are written to `~/.aws/credentials` and `~/.aws/config` under a `spinifex-<name>` profile automatically.
|
||||
|
||||
```bash
|
||||
spx admin account create --name myteam
|
||||
@@ -125,7 +125,7 @@ configuration is needed.
|
||||
|
||||
## Cluster Shutdown
|
||||
|
||||
Coordinated, phased shutdown of the entire cluster (API/UI → VMs → viperblock → predastore → NATS/daemon):
|
||||
Coordinated, phased shutdown of the entire cluster — running VMs are stopped before storage and control-plane services:
|
||||
|
||||
```bash
|
||||
spx admin cluster shutdown
|
||||
|
||||
@@ -72,7 +72,7 @@ After setup and reboot, enable GPU passthrough:
|
||||
sudo spx admin gpu enable
|
||||
```
|
||||
|
||||
The command writes the configuration, signals the daemon (`SIGHUP`), and waits up to 30 seconds for the daemon to confirm the new state. On success it prints the current GPU status.
|
||||
The command writes the configuration, notifies the daemon and waits up to 30 seconds for the daemon to confirm the new state. On success it prints the current GPU status.
|
||||
|
||||
<img src="../../../.github/assets/images/gpu-enabled.png" alt="GPU enabled">
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@ Spinifex provides EC2-compatible VM management built on QEMU/KVM. Instances supp
|
||||
- `run-instances` — Launch new VMs
|
||||
- `describe-instances` — Query state
|
||||
- `stop-instances` / `start-instances` — Lifecycle
|
||||
- `reboot-instances` — In-place restart (QMP reset)
|
||||
- `reboot-instances` — In-place restart
|
||||
- `terminate-instances` — Permanent removal
|
||||
- `modify-instance-attribute` — Change type, user data, or source/dest check
|
||||
- `get-console-output` — Retrieve serial console log
|
||||
@@ -131,16 +131,9 @@ aws ec2 describe-instance-types \
|
||||
--filters Name=capacity,Values=true
|
||||
```
|
||||
|
||||
## SSH (Development)
|
||||
## SSH
|
||||
|
||||
In development mode, find the QEMU port forward and connect via localhost:
|
||||
|
||||
```bash
|
||||
ps auxw | grep $INSTANCE_ID
|
||||
ssh -i ~/.ssh/spinifex-key ubuntu@127.0.0.1 -p <port>
|
||||
```
|
||||
|
||||
For production SSH via public IPs, see [Setting Up Your Cluster](/docs/setting-up-your-cluster).
|
||||
To SSH into instances via their public IPs, see [Setting Up Your Cluster](/docs/setting-up-your-cluster).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
||||
@@ -452,7 +452,7 @@ dns_servers = ["8.8.8.8"] # DNS for VMs (optional)
|
||||
|
||||
| Field | Required | Description |
|
||||
| ------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `name` | Yes | Unique pool name. Used as NATS KV key and in `AllocateAddress`. |
|
||||
| `name` | Yes | Unique pool name. Referenced by `AllocateAddress` to target a specific pool. |
|
||||
| `source` | No | IP source: `"static"` (default) uses `range_start`/`range_end`. `"dhcp"` obtains IPs from the router's DHCP server on each VM launch. |
|
||||
| `range_start` | Static only | First IP in the range. First IP is reserved for OVN gateway SNAT (unless `gateway_ip` overrides). |
|
||||
| `range_end` | Static only | Last IP in the range. |
|
||||
@@ -509,26 +509,10 @@ When an instance needs a public IP:
|
||||
|
||||
## IPAM Storage
|
||||
|
||||
Each pool gets a NATS KV entry in bucket `spinifex-external-ipam`, keyed by pool
|
||||
name. Allocation uses CAS (Compare-And-Set) for lock-free concurrent access:
|
||||
|
||||
```json
|
||||
{
|
||||
"pool_name": "wan",
|
||||
"range_start": "192.168.1.150",
|
||||
"range_end": "192.168.1.250",
|
||||
"allocated": {
|
||||
"192.168.1.150": { "type": "gateway" },
|
||||
"192.168.1.151": {
|
||||
"type": "auto_assign",
|
||||
"eni_id": "eni-abc",
|
||||
"instance_id": "i-123"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Pools are initialized from `spinifex.toml` on vpcd startup (idempotent).
|
||||
Pool allocation state is stored durably in the cluster (NATS KV bucket
|
||||
`spinifex-external-ipam`, one entry per pool) and survives restarts. Each
|
||||
allocation records the ENI and instance holding the address. Pools are
|
||||
initialized from `spinifex.toml` on vpcd startup (idempotent).
|
||||
|
||||
## Deployment Examples
|
||||
|
||||
@@ -1338,12 +1322,10 @@ OVNSB commit failed, force recompute next time.
|
||||
Repeated millions of times. Port binding never happens (`up: false`).
|
||||
|
||||
**Cause:** Stale entries in the OVN Southbound DB (old chassis records, port
|
||||
bindings, datapath bindings) conflict with ovn-controller's expected state.
|
||||
Happens when `reset-dev-env.sh` cleans NB but not SB, or after ungraceful
|
||||
shutdowns.
|
||||
bindings, datapath bindings) conflict with ovn-controller's expected state,
|
||||
typically after an ungraceful shutdown.
|
||||
|
||||
**Fix:** Delete both OVN DB files and restart. `reset-dev-env.sh` does this
|
||||
automatically:
|
||||
**Fix:** Delete both OVN DB files and restart:
|
||||
|
||||
```bash
|
||||
sudo systemctl stop ovn-central ovn-controller
|
||||
@@ -1352,21 +1334,6 @@ sudo systemctl start ovn-central ovn-controller
|
||||
# vpcd reconcile will recreate the NB topology on next startup
|
||||
```
|
||||
|
||||
### Chassis Name Mismatch
|
||||
|
||||
After mulga-999, vpcd uses the OVS-managed UUID (persisted at
|
||||
`/etc/openvswitch/system-id.conf` by the `openvswitch-switch` package and
|
||||
re-applied on every boot) as the canonical chassis identity. The chassis name
|
||||
across OVS `external_ids:system-id`, OVN SB `Chassis.name`, and NB
|
||||
`Gateway_Chassis.chassis_name` is always the same value, so the mismatch
|
||||
class that this section previously documented can no longer recur.
|
||||
|
||||
`ReconcileFromKV` runs a `reconcileGatewayChassis` pre-step on every vpcd
|
||||
startup that deletes any stale `Gateway_Chassis` row left over from earlier
|
||||
installs (where `setup-ovn.sh` fabricated `chassis-$(hostname -s)`) and
|
||||
re-binds every gateway LRP against the live SBDB chassis. Pre-mulga-999
|
||||
brokenness recovers automatically on the first vpcd restart after upgrade —
|
||||
no manual `ovn-nbctl destroy` required.
|
||||
|
||||
### WAN NIC Not Enslaved to a Bridge
|
||||
|
||||
|
||||
@@ -45,7 +45,7 @@ A Spinifex cluster distributes services across multiple servers for high availab
|
||||
>
|
||||
> Before running the installer on any server, that server's WAN interface **must** already be enslaved to a Linux bridge named `br-wan`. The host IP, default route, and DHCP must all live on the bridge — not on the bare NIC.
|
||||
>
|
||||
> The bootstrap installer does **not** create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of `br-wan` will land in the next release.
|
||||
> The bootstrap installer does **not** create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of `br-wan` will land in a future release.
|
||||
>
|
||||
> **Verify on every node before continuing:**
|
||||
>
|
||||
|
||||
@@ -55,7 +55,7 @@ This guide installs Spinifex on a single server using the binary installer. For
|
||||
>
|
||||
> Before running the installer, the host's WAN interface **must** already be enslaved to a Linux bridge named `br-wan`. The host IP, default route, and DHCP must all live on the bridge — not on the bare NIC.
|
||||
>
|
||||
> The bootstrap installer does **not** create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of `br-wan` will land in the next release.
|
||||
> The bootstrap installer does **not** create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of `br-wan` will land in a future release.
|
||||
>
|
||||
> **Verify before continuing:**
|
||||
>
|
||||
|
||||
@@ -267,8 +267,8 @@ Rejected settings:
|
||||
Useful background for host-side troubleshooting:
|
||||
|
||||
- The IMDS HTTP server runs inside the `spinifex-vpcd` service on each host, listening on `169.254.169.254:80` with one listener per instance's primary interface — attribution is structural, with no source-IP trust.
|
||||
- vpcd reconciles listeners against live interfaces every **15 seconds**, so a freshly launched guest's metadata service is serving within one reconcile tick (cloud-init's built-in retries absorb this window).
|
||||
- Session tokens and cached role credentials live in memory only. A vpcd restart drops them; SDKs and cloud-init transparently reissue. Datapath state on the host survives service restarts.
|
||||
- A freshly launched guest's metadata service comes up within a few seconds of launch; cloud-init's built-in retries absorb this window.
|
||||
- Session tokens and cached role credentials are not persisted. A vpcd restart drops them; SDKs and cloud-init transparently reissue. Datapath state on the host survives service restarts.
|
||||
|
||||
## Limits and Defaults
|
||||
|
||||
|
||||
@@ -97,8 +97,8 @@ iface br-wan inet dhcp
|
||||
|
||||
Walkthrough of changes:
|
||||
|
||||
* Adds eno3 into the bridge
|
||||
* eno3 becomes a slave interface
|
||||
* Adds enp0s1 into the bridge
|
||||
* enp0s1 becomes a slave interface
|
||||
* All traffic goes through br-wan
|
||||
* Spinifex EC2 (VMs) in a public subnet, obtain an IP address from the WAN upstream
|
||||
|
||||
|
||||
@@ -266,7 +266,7 @@ aideinit
|
||||
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
|
||||
```
|
||||
|
||||
Daily check — a systemd timer invoking `aide --check`. Non-zero exit must alert. Re-baseline after every Spinifex upgrade (`make reinstall`, package upgrade, config change) and record the new hash.
|
||||
Daily check — a systemd timer invoking `aide --check`. Non-zero exit must alert. Re-baseline after every Spinifex upgrade, package upgrade, or config change and record the new hash.
|
||||
|
||||
### 4.3 Wazuh (centralised AV + FIM + alerting)
|
||||
|
||||
|
||||
@@ -77,7 +77,7 @@ Any medium in any of these roles on a Spinifex node has held, or may have held,
|
||||
| Viperblock chunk cache / local backing | NVMe/SATA SSD or HDD | Encrypted volume chunks. |
|
||||
| Predastore object store | NVMe/SATA SSD or HDD | All S3 object data — AMIs, snapshots, user-uploaded artifacts, IAM state files, tenant data. |
|
||||
| NATS JetStream disk | NVMe/SATA SSD | IAM NATS token, cluster metadata, pending job state, NATS KV entries (including wrapped DEKs). |
|
||||
| Removable master-key media | USB flash token, HSM, smart card | Master encryption key (Phase 4 USB-mount path). Destroying this sanitizes every DEK-encrypted volume cluster-wide. |
|
||||
| Removable master-key media | USB flash token, HSM, smart card | Master encryption key (USB-mount path). Destroying this sanitizes every DEK-encrypted volume cluster-wide. |
|
||||
| Backup media | LTO tape, removable HDD/SSD, off-site cloud backup | Any of the above, point-in-time. |
|
||||
| BMC / iDRAC / iLO storage | Embedded flash | Console recordings, SEL logs, BMC credentials, cached operator certificates. Sanitize via the BMC "Reset to defaults" / "Erase user data" command before disposal. |
|
||||
| Switch / router config storage | Embedded flash | Cluster VLAN, management IPs, ACLs. Operator network kit, out of Spinifex scope but noted for completeness. |
|
||||
@@ -193,7 +193,7 @@ Third-party destruction vendors must provide a certificate of destruction listin
|
||||
When retiring a whole node:
|
||||
|
||||
1. **Drain** — migrate or terminate all instances hosted on the node; confirm Predastore and Viperblock roles are reassigned if the node held one. Volume DEKs of terminated instances are already cryptographically erased per [§3.1](#31-cryptographic-erase-via-dek-deletion).
|
||||
2. **Deregister** — `bd`-linked change ticket; the operator removes the node from the cluster (`spx admin node remove`) so NATS routes, OVN chassis, and predastore distributed membership drop it.
|
||||
2. **Deregister** — raise a change ticket; the operator removes the node from the cluster (`spx admin node remove`) so NATS routes, OVN chassis, and predastore distributed membership drop it.
|
||||
3. **Stop services and unmount** — `systemctl stop 'spinifex-*'`, `nbdkit` sessions, OVN agent. Confirm `/run/spinifex/nbd/` is empty.
|
||||
4. **Destroy on-disk keys** — see [§5](#5-key-destruction). This is the single most important step: sanitization of volume data reduces to sanitization of keys once data is encrypted.
|
||||
5. **Sanitize each drive** by media type per [§4.1](#41-nvme-ssd) – [§4.3](#43-hdd). Every drive bay is sanitized, including any unpopulated cache or WAL devices.
|
||||
@@ -211,7 +211,7 @@ Cryptographic erase relies on the key being unrecoverable. A drive that previous
|
||||
| Cluster CA private key (`/etc/spinifex/ca.key`) | Leader node system disk | Same treatment. Loss of the CA key does not sanitize data, but prevents impersonation of the cluster identity post-disposal. |
|
||||
| Per-node TLS private key (`/etc/spinifex/server.key`) | Each node system disk | `shred -u` before drive sanitization. |
|
||||
| Wrapped DEKs in NATS KV | `/var/lib/spinifex/nats/` on NATS-carrying nodes | Deleted via the `DeleteVolume` API; the KV record is compacted. Sanitization of the NATS JetStream disk per [§4](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a) ensures no recoverable copies. |
|
||||
| Removable master-key media (Phase 4 USB) | USB flash token | Physically destroy the token when the cluster is decommissioned; alternatively retain inside a secured location as evidence of cluster-wide cryptographic erase. Do not reuse for another cluster without full sanitization (see [§6](#6-removable-and-backup-media)). |
|
||||
| Removable master-key media (USB) | USB flash token | Physically destroy the token when the cluster is decommissioned; alternatively retain inside a secured location as evidence of cluster-wide cryptographic erase. Do not reuse for another cluster without full sanitization (see [§6](#6-removable-and-backup-media)). |
|
||||
| Backup-copy keys | Any off-site or escrow copy | Destroy simultaneously with the primary. A surviving backup defeats the cryptographic erase. |
|
||||
|
||||
If even one copy of the master key survives and the encrypted drives are recoverable, cryptographic erase has **not** been achieved. Track every key copy in the device register ([Physical Security Guide §5](/docs/security/physical-security-guide#5-manage-physical-access-devices-pel1-3105)).
|
||||
|
||||
@@ -95,7 +95,7 @@ The inventory in [§1](#1-inbound-listeners)–[§2](#2-outbound-connections) sa
|
||||
| 8222 | spinifex-nats (monitoring) | HTTP | Localhost | `varz`/`subsz` metrics consumed by the daemon | Loopback only |
|
||||
| socket / dynamic TCP | nbdkit (Viperblock) | NBD | Host-local / cluster | Block device transport for guest EBS volumes | Unix socket by default; TCP only in remote/DPU mode |
|
||||
|
||||
¹ **Formation port lifecycle.** 4432 opens during `spx admin init` / `spx admin join` while a bootstrap token is outstanding and closes once the cluster is formed (token TTL default 30 min, `--token-ttl`). The server presents an ephemeral self-signed cert that pre-dates trust bootstrap, so the joining node dials with `InsecureSkipVerify` — the only production dial that does. Authenticity rests on the operator supplying the leader address out-of-band plus possession of the bearer token. Document in the security plan so reviewers do not flag 4432 as a persistent open port.
|
||||
¹ **Formation port lifecycle.** 4432 opens during `spx admin init` / `spx admin join` while a bootstrap token is outstanding and closes once the cluster is formed (token TTL default 30 min, `--token-ttl`). The server presents an ephemeral self-signed cert that pre-dates trust bootstrap, so the joining node does not verify the certificate chain for this single dial. Authenticity rests on the operator supplying the leader address out-of-band plus possession of the bearer token. Document in the security plan so reviewers do not flag 4432 as a persistent open port.
|
||||
|
||||
**Development-only listeners.** When `dev_networking=true`, QEMU opens arbitrary host TCP ports for SSH port-forwarding into guest VMs. Production installs (the `/etc/spinifex` layout) do not enable this; it must not appear on compliance nodes.
|
||||
|
||||
@@ -165,7 +165,7 @@ Every listener and outbound destination is controlled by one of these files. Cha
|
||||
| OVN packages (`ovn-central`, `ovn-host`) | `ovn-nb-db`, `ovn-sb-db` (via `ovs-vsctl set open_vswitch …`) | OVN DB bind addresses. |
|
||||
| Spinifex UI service | Built-in defaults: `host = "0.0.0.0"`, `port = 3000`. No `spinifex.toml` block today. | UI listener. |
|
||||
| `spx admin init` / `spx admin join` | `--port`, `--token-ttl`, `--no-telemetry` (or `SPX_NO_TELEMETRY=1`) | Formation port, token TTL, telemetry opt-out. |
|
||||
| `utils/images.go` `AvailableImages` | Image catalogue URLs | Outbound HTTPS destinations for image downloads. |
|
||||
| Image catalogue (built-in) | Fixed URLs listed in [§2](#2-outbound-connections); not operator-configurable | Outbound HTTPS destinations for image downloads. |
|
||||
|
||||
## 6. Operator Checklist
|
||||
|
||||
|
||||
@@ -77,7 +77,7 @@ Terraform manages the cluster, the addons, the ACM cert, and the HTTPS Ingress;
|
||||
|
||||
- The `argocd` and `aws-ebs-csi-driver` bundles must be baked into the `eks-node` AMI — `describe-addon` returns `no baked bundle` otherwise.
|
||||
- The EBS-CSI default StorageClass uses `provisioner: ebs.csi.aws.com`; a PVC against it provisions a **Viperblock-backed EBS volume**.
|
||||
- **Verify dynamic provisioning on a live cluster.** EBS-CSI dynamic provisioning depends on the running Spinifex EBS/Viperblock data plane and may not be available on every build yet. If the PVC stays `Pending`, the app still serves (the counter falls back to in-memory) — confirm the volume binds on a stack where the driver is healthy.
|
||||
- If the PVC stays `Pending`, the app still serves (the counter falls back to in-memory) — see [Troubleshooting](#troubleshooting).
|
||||
- Leave addon versions unset (the AWS provider and the catalog disagree on the version-string format).
|
||||
- The Argo CD `Application` is a `kubernetes_manifest`, so the `argoproj.io` CRDs must exist before `apply` — apply the parent and let the addon reach `ACTIVE` first.
|
||||
|
||||
@@ -256,7 +256,7 @@ kubectl describe pvc <name>
|
||||
kubectl get storageclass
|
||||
```
|
||||
|
||||
Confirm the `aws-ebs-csi-driver` addon is `ACTIVE` and the default StorageClass exists. EBS-CSI dynamic provisioning depends on the live Spinifex EBS/Viperblock data plane — verify on a stack where the driver is healthy. The app still serves with an in-memory counter if the volume never binds.
|
||||
Confirm the `aws-ebs-csi-driver` addon is `ACTIVE` and the default StorageClass exists. The app still serves with an in-memory counter if the volume never binds.
|
||||
|
||||
### The Ingress Has No Address
|
||||
|
||||
|
||||
@@ -176,8 +176,6 @@ aws eks describe-addon --cluster-name eks-https --addon-name aws-load-balancer-c
|
||||
|
||||
Confirm the addon is `ACTIVE` and the cluster carries `spinifex.io/managed-ingress = "false"` (otherwise the built-in ingress competes).
|
||||
|
||||
> **Verify on a live cluster.** The LBC reconcile path (ALB creation, target registration, listener wiring) exercises the running Spinifex EKS data plane end to end. Stand this up on a live stack to confirm the ALB serves the page; the manifests here are the AWS-parity shape of that flow.
|
||||
|
||||
### HTTPS Loads but Returns 502 / 504
|
||||
|
||||
The ALB reached the workers but the backend didn't answer. Check the demo pods and the NodePort SG rule:
|
||||
|
||||
Reference in New Issue
Block a user