diff --git a/docs/administration/spinifex-admin-cli/README.md b/docs/administration/spinifex-admin-cli/README.md index 594c29c0a..20fdf42a9 100644 --- a/docs/administration/spinifex-admin-cli/README.md +++ b/docs/administration/spinifex-admin-cli/README.md @@ -27,7 +27,7 @@ resources: The `spx` binary is the central administration tool for managing your Spinifex infrastructure. It provides commands for cluster initialization, account management, node operations, VM lifecycle, and service control. -**Binary location:** `/usr/local/bin/spx` (binary install) or `~/Development/mulga/spinifex/bin/spx` (source install) +**Binary location:** `/usr/local/bin/spx` All services in the Spinifex platform are managed through this single binary. @@ -35,7 +35,7 @@ All services in the Spinifex platform are managed through this single binary. ## Account Management -Create a new isolated account. This provisions a sequential 12-digit account ID, an `admin` user with an `AdministratorAccess` policy attached, and an access key pair. The credentials are written to `~/.aws/credentials` and `~/.aws/config` under a `spinifex-` profile automatically. Requires a running cluster; run from a cluster node: +Create a new isolated account. This provisions a sequential 12-digit account ID, an `admin` user with an `AdministratorAccess` policy attached, and an access key pair. The credentials are written to `~/.aws/credentials` and `~/.aws/config` under a `spinifex-` profile automatically. ```bash spx admin account create --name myteam @@ -125,7 +125,7 @@ configuration is needed. ## Cluster Shutdown -Coordinated, phased shutdown of the entire cluster (API/UI → VMs → viperblock → predastore → NATS/daemon): +Coordinated, phased shutdown of the entire cluster — running VMs are stopped before storage and control-plane services: ```bash spx admin cluster shutdown diff --git a/docs/compute/gpu-passthrough/README.md b/docs/compute/gpu-passthrough/README.md index 23c66240c..9a9c78db5 100644 --- a/docs/compute/gpu-passthrough/README.md +++ b/docs/compute/gpu-passthrough/README.md @@ -72,7 +72,7 @@ After setup and reboot, enable GPU passthrough: sudo spx admin gpu enable ``` -The command writes the configuration, signals the daemon (`SIGHUP`), and waits up to 30 seconds for the daemon to confirm the new state. On success it prints the current GPU status. +The command writes the configuration, notifies the daemon and waits up to 30 seconds for the daemon to confirm the new state. On success it prints the current GPU status. GPU enabled diff --git a/docs/compute/launching-instances/README.md b/docs/compute/launching-instances/README.md index 949a25007..fe6635234 100644 --- a/docs/compute/launching-instances/README.md +++ b/docs/compute/launching-instances/README.md @@ -42,7 +42,7 @@ Spinifex provides EC2-compatible VM management built on QEMU/KVM. Instances supp - `run-instances` — Launch new VMs - `describe-instances` — Query state - `stop-instances` / `start-instances` — Lifecycle -- `reboot-instances` — In-place restart (QMP reset) +- `reboot-instances` — In-place restart - `terminate-instances` — Permanent removal - `modify-instance-attribute` — Change type, user data, or source/dest check - `get-console-output` — Retrieve serial console log @@ -131,16 +131,9 @@ aws ec2 describe-instance-types \ --filters Name=capacity,Values=true ``` -## SSH (Development) +## SSH -In development mode, find the QEMU port forward and connect via localhost: - -```bash -ps auxw | grep $INSTANCE_ID -ssh -i ~/.ssh/spinifex-key ubuntu@127.0.0.1 -p -``` - -For production SSH via public IPs, see [Setting Up Your Cluster](/docs/setting-up-your-cluster). +To SSH into instances via their public IPs, see [Setting Up Your Cluster](/docs/setting-up-your-cluster). ## Troubleshooting diff --git a/docs/compute/vpc-networking/README.md b/docs/compute/vpc-networking/README.md index 1f3998db3..49926239e 100644 --- a/docs/compute/vpc-networking/README.md +++ b/docs/compute/vpc-networking/README.md @@ -452,7 +452,7 @@ dns_servers = ["8.8.8.8"] # DNS for VMs (optional) | Field | Required | Description | | ------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `name` | Yes | Unique pool name. Used as NATS KV key and in `AllocateAddress`. | +| `name` | Yes | Unique pool name. Referenced by `AllocateAddress` to target a specific pool. | | `source` | No | IP source: `"static"` (default) uses `range_start`/`range_end`. `"dhcp"` obtains IPs from the router's DHCP server on each VM launch. | | `range_start` | Static only | First IP in the range. First IP is reserved for OVN gateway SNAT (unless `gateway_ip` overrides). | | `range_end` | Static only | Last IP in the range. | @@ -509,26 +509,10 @@ When an instance needs a public IP: ## IPAM Storage -Each pool gets a NATS KV entry in bucket `spinifex-external-ipam`, keyed by pool -name. Allocation uses CAS (Compare-And-Set) for lock-free concurrent access: - -```json -{ - "pool_name": "wan", - "range_start": "192.168.1.150", - "range_end": "192.168.1.250", - "allocated": { - "192.168.1.150": { "type": "gateway" }, - "192.168.1.151": { - "type": "auto_assign", - "eni_id": "eni-abc", - "instance_id": "i-123" - } - } -} -``` - -Pools are initialized from `spinifex.toml` on vpcd startup (idempotent). +Pool allocation state is stored durably in the cluster (NATS KV bucket +`spinifex-external-ipam`, one entry per pool) and survives restarts. Each +allocation records the ENI and instance holding the address. Pools are +initialized from `spinifex.toml` on vpcd startup (idempotent). ## Deployment Examples @@ -1338,12 +1322,10 @@ OVNSB commit failed, force recompute next time. Repeated millions of times. Port binding never happens (`up: false`). **Cause:** Stale entries in the OVN Southbound DB (old chassis records, port -bindings, datapath bindings) conflict with ovn-controller's expected state. -Happens when `reset-dev-env.sh` cleans NB but not SB, or after ungraceful -shutdowns. +bindings, datapath bindings) conflict with ovn-controller's expected state, +typically after an ungraceful shutdown. -**Fix:** Delete both OVN DB files and restart. `reset-dev-env.sh` does this -automatically: +**Fix:** Delete both OVN DB files and restart: ```bash sudo systemctl stop ovn-central ovn-controller @@ -1352,21 +1334,6 @@ sudo systemctl start ovn-central ovn-controller # vpcd reconcile will recreate the NB topology on next startup ``` -### Chassis Name Mismatch - -After mulga-999, vpcd uses the OVS-managed UUID (persisted at -`/etc/openvswitch/system-id.conf` by the `openvswitch-switch` package and -re-applied on every boot) as the canonical chassis identity. The chassis name -across OVS `external_ids:system-id`, OVN SB `Chassis.name`, and NB -`Gateway_Chassis.chassis_name` is always the same value, so the mismatch -class that this section previously documented can no longer recur. - -`ReconcileFromKV` runs a `reconcileGatewayChassis` pre-step on every vpcd -startup that deletes any stale `Gateway_Chassis` row left over from earlier -installs (where `setup-ovn.sh` fabricated `chassis-$(hostname -s)`) and -re-binds every gateway LRP against the live SBDB chassis. Pre-mulga-999 -brokenness recovers automatically on the first vpcd restart after upgrade — -no manual `ovn-nbctl destroy` required. ### WAN NIC Not Enslaved to a Bridge diff --git a/docs/getting-started/install-multi-node/README.md b/docs/getting-started/install-multi-node/README.md index 3f7d05b03..714e0783d 100644 --- a/docs/getting-started/install-multi-node/README.md +++ b/docs/getting-started/install-multi-node/README.md @@ -45,7 +45,7 @@ A Spinifex cluster distributes services across multiple servers for high availab > > Before running the installer on any server, that server's WAN interface **must** already be enslaved to a Linux bridge named `br-wan`. The host IP, default route, and DHCP must all live on the bridge — not on the bare NIC. > -> The bootstrap installer does **not** create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of `br-wan` will land in the next release. +> The bootstrap installer does **not** create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of `br-wan` will land in a future release. > > **Verify on every node before continuing:** > diff --git a/docs/getting-started/install/README.md b/docs/getting-started/install/README.md index 7a92e560b..07df0d920 100644 --- a/docs/getting-started/install/README.md +++ b/docs/getting-started/install/README.md @@ -55,7 +55,7 @@ This guide installs Spinifex on a single server using the binary installer. For > > Before running the installer, the host's WAN interface **must** already be enslaved to a Linux bridge named `br-wan`. The host IP, default route, and DHCP must all live on the bridge — not on the bare NIC. > -> The bootstrap installer does **not** create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of `br-wan` will land in the next release. +> The bootstrap installer does **not** create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of `br-wan` will land in a future release. > > **Verify before continuing:** > diff --git a/docs/identity/imds/README.md b/docs/identity/imds/README.md index 3e89ad652..bcc61cb72 100644 --- a/docs/identity/imds/README.md +++ b/docs/identity/imds/README.md @@ -267,8 +267,8 @@ Rejected settings: Useful background for host-side troubleshooting: - The IMDS HTTP server runs inside the `spinifex-vpcd` service on each host, listening on `169.254.169.254:80` with one listener per instance's primary interface — attribution is structural, with no source-IP trust. -- vpcd reconciles listeners against live interfaces every **15 seconds**, so a freshly launched guest's metadata service is serving within one reconcile tick (cloud-init's built-in retries absorb this window). -- Session tokens and cached role credentials live in memory only. A vpcd restart drops them; SDKs and cloud-init transparently reissue. Datapath state on the host survives service restarts. +- A freshly launched guest's metadata service comes up within a few seconds of launch; cloud-init's built-in retries absorb this window. +- Session tokens and cached role credentials are not persisted. A vpcd restart drops them; SDKs and cloud-init transparently reissue. Datapath state on the host survives service restarts. ## Limits and Defaults diff --git a/docs/install-vm-eval/README.md b/docs/install-vm-eval/README.md index b69e422c0..32ee9db4f 100644 --- a/docs/install-vm-eval/README.md +++ b/docs/install-vm-eval/README.md @@ -97,8 +97,8 @@ iface br-wan inet dhcp Walkthrough of changes: -* Adds eno3 into the bridge -* eno3 becomes a slave interface +* Adds enp0s1 into the bridge +* enp0s1 becomes a slave interface * All traffic goes through br-wan * Spinifex EC2 (VMs) in a public subnet, obtain an IP address from the WAN upstream diff --git a/docs/security/malware-protection/README.md b/docs/security/malware-protection/README.md index e40b87359..6e890c39b 100644 --- a/docs/security/malware-protection/README.md +++ b/docs/security/malware-protection/README.md @@ -266,7 +266,7 @@ aideinit mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db ``` -Daily check — a systemd timer invoking `aide --check`. Non-zero exit must alert. Re-baseline after every Spinifex upgrade (`make reinstall`, package upgrade, config change) and record the new hash. +Daily check — a systemd timer invoking `aide --check`. Non-zero exit must alert. Re-baseline after every Spinifex upgrade, package upgrade, or config change and record the new hash. ### 4.3 Wazuh (centralised AV + FIM + alerting) diff --git a/docs/security/media-sanitization/README.md b/docs/security/media-sanitization/README.md index 950f9adab..0666d1bb5 100644 --- a/docs/security/media-sanitization/README.md +++ b/docs/security/media-sanitization/README.md @@ -77,7 +77,7 @@ Any medium in any of these roles on a Spinifex node has held, or may have held, | Viperblock chunk cache / local backing | NVMe/SATA SSD or HDD | Encrypted volume chunks. | | Predastore object store | NVMe/SATA SSD or HDD | All S3 object data — AMIs, snapshots, user-uploaded artifacts, IAM state files, tenant data. | | NATS JetStream disk | NVMe/SATA SSD | IAM NATS token, cluster metadata, pending job state, NATS KV entries (including wrapped DEKs). | -| Removable master-key media | USB flash token, HSM, smart card | Master encryption key (Phase 4 USB-mount path). Destroying this sanitizes every DEK-encrypted volume cluster-wide. | +| Removable master-key media | USB flash token, HSM, smart card | Master encryption key (USB-mount path). Destroying this sanitizes every DEK-encrypted volume cluster-wide. | | Backup media | LTO tape, removable HDD/SSD, off-site cloud backup | Any of the above, point-in-time. | | BMC / iDRAC / iLO storage | Embedded flash | Console recordings, SEL logs, BMC credentials, cached operator certificates. Sanitize via the BMC "Reset to defaults" / "Erase user data" command before disposal. | | Switch / router config storage | Embedded flash | Cluster VLAN, management IPs, ACLs. Operator network kit, out of Spinifex scope but noted for completeness. | @@ -193,7 +193,7 @@ Third-party destruction vendors must provide a certificate of destruction listin When retiring a whole node: 1. **Drain** — migrate or terminate all instances hosted on the node; confirm Predastore and Viperblock roles are reassigned if the node held one. Volume DEKs of terminated instances are already cryptographically erased per [§3.1](#31-cryptographic-erase-via-dek-deletion). -2. **Deregister** — `bd`-linked change ticket; the operator removes the node from the cluster (`spx admin node remove`) so NATS routes, OVN chassis, and predastore distributed membership drop it. +2. **Deregister** — raise a change ticket; the operator removes the node from the cluster (`spx admin node remove`) so NATS routes, OVN chassis, and predastore distributed membership drop it. 3. **Stop services and unmount** — `systemctl stop 'spinifex-*'`, `nbdkit` sessions, OVN agent. Confirm `/run/spinifex/nbd/` is empty. 4. **Destroy on-disk keys** — see [§5](#5-key-destruction). This is the single most important step: sanitization of volume data reduces to sanitization of keys once data is encrypted. 5. **Sanitize each drive** by media type per [§4.1](#41-nvme-ssd) – [§4.3](#43-hdd). Every drive bay is sanitized, including any unpopulated cache or WAL devices. @@ -211,7 +211,7 @@ Cryptographic erase relies on the key being unrecoverable. A drive that previous | Cluster CA private key (`/etc/spinifex/ca.key`) | Leader node system disk | Same treatment. Loss of the CA key does not sanitize data, but prevents impersonation of the cluster identity post-disposal. | | Per-node TLS private key (`/etc/spinifex/server.key`) | Each node system disk | `shred -u` before drive sanitization. | | Wrapped DEKs in NATS KV | `/var/lib/spinifex/nats/` on NATS-carrying nodes | Deleted via the `DeleteVolume` API; the KV record is compacted. Sanitization of the NATS JetStream disk per [§4](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a) ensures no recoverable copies. | -| Removable master-key media (Phase 4 USB) | USB flash token | Physically destroy the token when the cluster is decommissioned; alternatively retain inside a secured location as evidence of cluster-wide cryptographic erase. Do not reuse for another cluster without full sanitization (see [§6](#6-removable-and-backup-media)). | +| Removable master-key media (USB) | USB flash token | Physically destroy the token when the cluster is decommissioned; alternatively retain inside a secured location as evidence of cluster-wide cryptographic erase. Do not reuse for another cluster without full sanitization (see [§6](#6-removable-and-backup-media)). | | Backup-copy keys | Any off-site or escrow copy | Destroy simultaneously with the primary. A surviving backup defeats the cryptographic erase. | If even one copy of the master key survives and the encrypted drives are recoverable, cryptographic erase has **not** been achieved. Track every key copy in the device register ([Physical Security Guide §5](/docs/security/physical-security-guide#5-manage-physical-access-devices-pel1-3105)). diff --git a/docs/security/network-connections/README.md b/docs/security/network-connections/README.md index 883743dee..3fec2da89 100644 --- a/docs/security/network-connections/README.md +++ b/docs/security/network-connections/README.md @@ -95,7 +95,7 @@ The inventory in [§1](#1-inbound-listeners)–[§2](#2-outbound-connections) sa | 8222 | spinifex-nats (monitoring) | HTTP | Localhost | `varz`/`subsz` metrics consumed by the daemon | Loopback only | | socket / dynamic TCP | nbdkit (Viperblock) | NBD | Host-local / cluster | Block device transport for guest EBS volumes | Unix socket by default; TCP only in remote/DPU mode | -¹ **Formation port lifecycle.** 4432 opens during `spx admin init` / `spx admin join` while a bootstrap token is outstanding and closes once the cluster is formed (token TTL default 30 min, `--token-ttl`). The server presents an ephemeral self-signed cert that pre-dates trust bootstrap, so the joining node dials with `InsecureSkipVerify` — the only production dial that does. Authenticity rests on the operator supplying the leader address out-of-band plus possession of the bearer token. Document in the security plan so reviewers do not flag 4432 as a persistent open port. +¹ **Formation port lifecycle.** 4432 opens during `spx admin init` / `spx admin join` while a bootstrap token is outstanding and closes once the cluster is formed (token TTL default 30 min, `--token-ttl`). The server presents an ephemeral self-signed cert that pre-dates trust bootstrap, so the joining node does not verify the certificate chain for this single dial. Authenticity rests on the operator supplying the leader address out-of-band plus possession of the bearer token. Document in the security plan so reviewers do not flag 4432 as a persistent open port. **Development-only listeners.** When `dev_networking=true`, QEMU opens arbitrary host TCP ports for SSH port-forwarding into guest VMs. Production installs (the `/etc/spinifex` layout) do not enable this; it must not appear on compliance nodes. @@ -165,7 +165,7 @@ Every listener and outbound destination is controlled by one of these files. Cha | OVN packages (`ovn-central`, `ovn-host`) | `ovn-nb-db`, `ovn-sb-db` (via `ovs-vsctl set open_vswitch …`) | OVN DB bind addresses. | | Spinifex UI service | Built-in defaults: `host = "0.0.0.0"`, `port = 3000`. No `spinifex.toml` block today. | UI listener. | | `spx admin init` / `spx admin join` | `--port`, `--token-ttl`, `--no-telemetry` (or `SPX_NO_TELEMETRY=1`) | Formation port, token TTL, telemetry opt-out. | -| `utils/images.go` `AvailableImages` | Image catalogue URLs | Outbound HTTPS destinations for image downloads. | +| Image catalogue (built-in) | Fixed URLs listed in [§2](#2-outbound-connections); not operator-configurable | Outbound HTTPS destinations for image downloads. | ## 6. Operator Checklist diff --git a/docs/terraform-workbooks/eks-gitops-argocd/README.md b/docs/terraform-workbooks/eks-gitops-argocd/README.md index 3e8d89b71..f31af486f 100644 --- a/docs/terraform-workbooks/eks-gitops-argocd/README.md +++ b/docs/terraform-workbooks/eks-gitops-argocd/README.md @@ -77,7 +77,7 @@ Terraform manages the cluster, the addons, the ACM cert, and the HTTPS Ingress; - The `argocd` and `aws-ebs-csi-driver` bundles must be baked into the `eks-node` AMI — `describe-addon` returns `no baked bundle` otherwise. - The EBS-CSI default StorageClass uses `provisioner: ebs.csi.aws.com`; a PVC against it provisions a **Viperblock-backed EBS volume**. -- **Verify dynamic provisioning on a live cluster.** EBS-CSI dynamic provisioning depends on the running Spinifex EBS/Viperblock data plane and may not be available on every build yet. If the PVC stays `Pending`, the app still serves (the counter falls back to in-memory) — confirm the volume binds on a stack where the driver is healthy. +- If the PVC stays `Pending`, the app still serves (the counter falls back to in-memory) — see [Troubleshooting](#troubleshooting). - Leave addon versions unset (the AWS provider and the catalog disagree on the version-string format). - The Argo CD `Application` is a `kubernetes_manifest`, so the `argoproj.io` CRDs must exist before `apply` — apply the parent and let the addon reach `ACTIVE` first. @@ -256,7 +256,7 @@ kubectl describe pvc kubectl get storageclass ``` -Confirm the `aws-ebs-csi-driver` addon is `ACTIVE` and the default StorageClass exists. EBS-CSI dynamic provisioning depends on the live Spinifex EBS/Viperblock data plane — verify on a stack where the driver is healthy. The app still serves with an in-memory counter if the volume never binds. +Confirm the `aws-ebs-csi-driver` addon is `ACTIVE` and the default StorageClass exists. The app still serves with an in-memory counter if the volume never binds. ### The Ingress Has No Address diff --git a/docs/terraform-workbooks/eks-https-ingress/README.md b/docs/terraform-workbooks/eks-https-ingress/README.md index 4e312f6bc..8ebea2d93 100644 --- a/docs/terraform-workbooks/eks-https-ingress/README.md +++ b/docs/terraform-workbooks/eks-https-ingress/README.md @@ -176,8 +176,6 @@ aws eks describe-addon --cluster-name eks-https --addon-name aws-load-balancer-c Confirm the addon is `ACTIVE` and the cluster carries `spinifex.io/managed-ingress = "false"` (otherwise the built-in ingress competes). -> **Verify on a live cluster.** The LBC reconcile path (ALB creation, target registration, listener wiring) exercises the running Spinifex EKS data plane end to end. Stand this up on a live stack to confirm the ALB serves the page; the manifests here are the AWS-parity shape of that flow. - ### HTTPS Loads but Returns 502 / 504 The ALB reached the workers but the backend didn't answer. Check the demo pods and the NodePort SG rule: