docs: describe the predastore host and node model

The port tables were wrong in both directions: inter-node traffic is QUIC over UDP rather than TCP, and the shard range 9991-9993 no longer exists. Ports are unique within a host rather than across the cluster, so every machine uses the same three, and a single-node install opens only the S3 port because colocated nodes talk over an in-process pipe. The reference architectures scripted a move of node data out of a directory layout that no longer exists, which would have broken any install following them. The update guide now says object storage has no migration path.
2026-10-15 09:01:31 +00:00 · 2026-08-11 15:54:31 +10:00
parent a7a62addeb
commit 1394fbf488
9 changed files with 76 additions and 33 deletions
@@ -127,10 +127,8 @@ bridge_mode = "{{.BridgeMode}}"

 [nodes.{{.Node}}.predastore]
 # Wildcard rather than the bind IP: S3 is a public service and has to answer on
-# the wan plane, and predastore-start.sh scrapes this line into
-# SPINIFEX_PREDASTORE_HOST, which outranks the host set in predastore.toml. Only
-# the S3 API moves — the [[db]] and [[nodes]] peer addresses in
-# predastore-multinode.toml stay on the bind address.
+# the wan plane. The service binds this address and serves S3 on this port,
+# leaving [[host]].addr — the address peer hosts dial — as predastore.toml has it.
 #
 # This doubles as the local dial target, and 0.0.0.0 is not one: config loading
 # rewrites it to 127.0.0.1 for this node, and admin.DialTarget does the same for
@@ -10,7 +10,7 @@ Service lifecycle commands for starting, stopping, and checking status of all Sp

 | Command | Flags | Description |
 |---------|-------|-------------|
-| `spx service predastore start` | `--port` (default: 8443), `--host` (default: 0.0.0.0), `--base-path` (directory holding the service pid file), `--config-path`, `--tls-cert`, `--tls-key`, `--encryption-key-file`, `--host-id` (required; names a `[[host]]` in the predastore config) | Creates predastore service instance with S3-compatible storage backend → starts service, serving that host's nodes and the S3 gate among them until the process is signalled. When `--config`/`SPINIFEX_CONFIG_PATH` points at a cluster spinifex.toml, `--host`/`--port`/`--host-id` default to that node's `[nodes.<node>.predastore]` section instead of the flag defaults above; an explicit flag or `SPINIFEX_PREDASTORE_*` env var still overrides it. `host_id` absent from spinifex.toml resolves to 0, which names no host and is rejected. |
+| `spx service predastore start` | `--port` (default: 8443, the host's S3 gate port), `--host` (default: 0.0.0.0, the address the host's sockets bind), `--base-path` (directory holding the service pid file), `--config-path` (required), `--tls-cert` (required), `--tls-key` (required), `--encryption-key-file` (required), `--host-id` (required; names a `[[host]]` in the predastore config) | Creates predastore service instance with S3-compatible storage backend → starts service, serving that host's nodes and the S3 gate among them until the process is signalled. When `--config`/`SPINIFEX_CONFIG_PATH` points at a cluster spinifex.toml, `--host`/`--port`/`--host-id` default to that node's `[nodes.<node>.predastore]` section instead of the flag defaults above; an explicit flag or `SPINIFEX_PREDASTORE_*` env var still overrides it. `host_id` absent from spinifex.toml resolves to 0, which names no host and is rejected. |
 | `spx service predastore stop` | — | Stops the predastore service |
 | `spx service predastore status` | — | Reports predastore service status |
 | `spx service viperblock start` | `--s3-host` (default: 0.0.0.0:8443), `--s3-bucket` (default: predastore), `--s3-region` (default: ap-southeast-2), `--plugin-path` (auto-detected via `nbdkit --dump-config plugindir`; typically `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` on amd64, overridable via `SPINIFEX_VIPERBLOCK_PLUGIN_PATH` in `/etc/spinifex/systemd.env`) | Loads cluster config → connects to NATS and Predastore → starts viperblock block storage service with NBD plugin |
@@ -238,6 +238,22 @@ Object storage used for:
 - SSH public key storage (`/bucket/ec2/{key-name}.pub`)
 - Volume metadata and configuration

+Spinifex runs Predastore in-process: `spx service predastore start` loads `/etc/spinifex/predastore/predastore.toml`, selects the `[[host]]` naming this machine, and serves it until the process is signalled. The topology is written by `spx admin init` / `join` from the servers that form the cluster.
+
+One `[[host]]` is one process. The nodes pinned to it under `[[host.node]]` carry the roles:
+
+| Role | Port | Responsibility |
+|------|------|----------------|
+| `gate` | TCP 8443 | The S3 API. Holds no cluster state; dials the meta and blob nodes to serve a request. |
+| `blob` | UDP 6660 | Erasure-coded object shards, encrypted at rest under this host's master key. |
+| `meta` | UDP 7660 | Raft replica of global state — buckets and the object index. |
+
+Ports are unique within a host but repeat across the cluster, so a multi-node cluster puts one node of each role on every machine and every machine uses the same three ports. Nodes sharing a host talk over an in-process pipe and bind no socket; nodes on other hosts are reached over QUIC, authenticated by the cluster CA.
+
+A single-node install is the same arrangement with one host: seven nodes — a gate, three blob and three meta — colocated in one process, needing seven distinct ports (8443, 6660–6662, 7660–7662) to stay unique within the host even though only the gate opens a listener. It has no code path of its own.
+
+The daemon reports the topology on `spinifex.storage.config` by parsing the config file. It does not connect to Predastore to do so, and no Predastore node exposes a status endpoint.
+
 ## Configuration

 Cluster configuration (`spinifex/config/config.go`):
@@ -32,6 +32,9 @@ Updating Spinifex is the same command used to install it. The installer detects

 For operators who want to review migrations before they are applied, a manual upgrade path is also supported.

+> [!WARNING]
+> **Object storage is exempt.** Predastore's configuration schema and on-disk layout changed with the object storage cutover, and no migration converts an installation from before it. `spx admin upgrade` will not report anything pending for `predastore.toml`, and updating the binary over such an installation leaves Predastore unable to start. Those clusters have to be re-initialised from scratch, which discards their stored objects — export anything you need first.
+
 ## Instructions

 ## Step 1. Re-run the Installer
@@ -26,6 +26,7 @@ resources:
 ## Table of Contents

 - [Overview](#overview)
+- [Object Storage Layout](#object-storage-layout)
 - [Prerequisites](#prerequisites)
 - [Instructions](#instructions)
 - [Converting ISO-Installed Nodes](#converting-iso-installed-nodes)
@@ -53,6 +54,28 @@ Servers beyond the third join as compute nodes and do not run a database, which
 - UDP port 6081 open between hosts (Geneve tunnels)
 - TCP ports 4222, 4248, 6641, 6642 open between hosts (NATS, OVN)
 - TCP ports 6643, 6644 open between database servers (OVN database clustering)
+- TCP port 8443 and UDP ports 6660, 7660 open between hosts (Predastore S3, object shards, metadata consensus)
+
+Predastore uses the same three ports on every server, so the surface does not widen as the cluster grows. See [Object Storage Layout](#object-storage-layout) below.
+
+## Object Storage Layout
+
+Predastore is configured for the whole cluster in `/etc/spinifex/predastore/predastore.toml`. Each server is one `[[host]]` — a single Predastore process owning that machine's data directory and TLS identity — carrying three nodes under `[[host.node]]`:
+
+| Role | Port | Purpose |
+|------|------|---------|
+| `gate` | TCP 8443 | Serves the S3 API. Every server runs one, so any of them answers an S3 request. |
+| `blob` | UDP 6660 | Holds erasure-coded object shards. One per machine. |
+| `meta` | UDP 7660 | Member of the Raft quorum over global state — buckets and the object index. |
+
+Ports have to be unique within a host but not across the cluster, so every machine uses the same three. Blob and meta traffic between hosts runs over QUIC, authenticated by the cluster CA; nodes on the same machine talk over an in-process pipe and open no socket, which is why a single-server install listens on 8443 alone.
+
+Reed-Solomon parameters are chosen from the cluster size, since each machine contributes exactly one blob node: two servers get `RS(1,1)`, three or more get `RS(2,1)`. `RS(2,1)` survives the loss of any one server's shards.
+
+You do not configure any of this by hand. `spx admin init` and `spx admin join` build the topology from the servers that actually form the cluster in Step 4, and each machine gets the same file with its own host ID recorded in `spinifex.toml`.
+
+> [!WARNING]
+> **There is no upgrade path onto this layout.** A cluster installed before the object storage cutover cannot be migrated — the on-disk directory structure and the configuration schema both changed, and no migration rewrites either. Such a cluster has to be re-initialised from scratch, which discards its stored objects. Export anything you need first.

 ## Prerequisites

@@ -137,19 +137,19 @@ lspci -d 10de: -nn

 **4. Attach Predastore storage**

-The X14 has four NVMe drives: two 1.5 TB SSDs (one carries the OS) and two ~880 GB SSDs. The OS occupies its own dedicated NVMe; the remaining three drives are pre-formatted and already mounted at `/mnt/nvme-1`, `/mnt/nvme-2`, and `/mnt/nvme-3`. Predastore is distributed across these three drives — one storage node per physical drive, with Reed–Solomon redundancy so a single drive failure is recoverable.
+The X14 has four NVMe drives: two 1.5 TB SSDs (one carries the OS) and two ~880 GB SSDs. The OS occupies its own dedicated NVMe; the remaining three drives are pre-formatted and already mounted at `/mnt/nvme-1`, `/mnt/nvme-2`, and `/mnt/nvme-3`. Predastore spreads object shards across these three drives — one blob node per physical drive, with Reed–Solomon redundancy so a single drive failure is recoverable.

-Relocate the Predastore data directories onto the mounted drives:
+A single-node install is one Predastore host running seven nodes, each with a directory named for its node ID under `/var/lib/spinifex/predastore/cluster`. The three blob nodes are `node-2`, `node-3` and `node-4`, so `nvme-$i` takes `node-$((i + 1))`. The gate keeps no data at all, and the three meta nodes hold only the Raft log for buckets and the object index, which is small enough to leave on the OS drive.
+
+Relocate the blob node directories onto the mounted drives:

 ```bash
 sudo systemctl stop spinifex.target

 for i in 1 2 3; do
-  sudo mkdir -p /mnt/nvme-$i/nodes /mnt/nvme-$i/db
-  sudo mv /var/lib/spinifex/predastore/distributed/nodes/node-$i /mnt/nvme-$i/nodes/node-$i
-  sudo mv /var/lib/spinifex/predastore/distributed/db/node-$i    /mnt/nvme-$i/db/node-$i
-  sudo ln -s /mnt/nvme-$i/nodes/node-$i /var/lib/spinifex/predastore/distributed/nodes/node-$i
-  sudo ln -s /mnt/nvme-$i/db/node-$i    /var/lib/spinifex/predastore/distributed/db/node-$i
+  node="node-$((i + 1))"
+  sudo mv /var/lib/spinifex/predastore/cluster/$node /mnt/nvme-$i/$node
+  sudo ln -s /mnt/nvme-$i/$node /var/lib/spinifex/predastore/cluster/$node
 done

 sudo systemctl start spinifex.target
@@ -138,7 +138,9 @@ sudo systemctl status spinifex.target

 ### 4. Attach Predastore storage

-Spinifex distributes the Predastore object storage volume across multiple nodes using Reed–Solomon encoding for data redundancy. This chassis has four NVMe drives — one for the OS, three dedicated to data — so we back each Predastore storage node with its own physical drive, giving us fault-tolerant distributed storage on a single machine — conceptually similar to RAID 5, where data and parity are spread across drives so a single drive failure is recoverable.
+Spinifex spreads Predastore's object shards across multiple blob nodes using Reed–Solomon encoding for data redundancy. This chassis has four NVMe drives — one for the OS, three dedicated to data — so we back each blob node with its own physical drive, giving us fault-tolerant storage on a single machine — conceptually similar to RAID 5, where data and parity are spread across drives so a single drive failure is recoverable.
+
+A single-node install is one Predastore host running seven nodes, each with a directory named for its node ID under `/var/lib/spinifex/predastore/cluster`. The three blob nodes are `node-2`, `node-3` and `node-4`, and they are the ones worth their own drive: the gate keeps no data, and the three meta nodes hold only the Raft log for buckets and the object index.

 Confirm drive assignments with `lsblk` before proceeding, as device names vary between systems.

@@ -148,20 +150,17 @@ lsblk   # identify the OS drive and the three data drives
 # Stop services so Predastore isn't writing while we relocate its data directories
 sudo systemctl stop spinifex.target

-# --- Repeat the block below for each data drive (node-1/nvme-1, node-2/nvme-2, node-3/nvme-3) ---
+# --- Repeat the block below for each data drive (node-2/nvme-1, node-3/nvme-2, node-4/nvme-3) ---

 # Mount the physical drive at a stable path
 sudo mkdir -p /mnt/nvme-1
 sudo mount /dev/nvme1n1 /mnt/nvme-1
-sudo mkdir -p /mnt/nvme-1/nodes /mnt/nvme-1/db

-# Move Predastore's node data and metadata off the OS drive onto the physical NVMe
-sudo mv /var/lib/spinifex/predastore/distributed/nodes/node-1 /mnt/nvme-1/nodes/node-1
-sudo mv /var/lib/spinifex/predastore/distributed/db/node-1    /mnt/nvme-1/db/node-1
+# Move the blob node's data off the OS drive onto the physical NVMe
+sudo mv /var/lib/spinifex/predastore/cluster/node-2 /mnt/nvme-1/node-2

-# Symlink the original paths back so Predastore finds its data unchanged
-sudo ln -s /mnt/nvme-1/nodes/node-1 /var/lib/spinifex/predastore/distributed/nodes/node-1
-sudo ln -s /mnt/nvme-1/db/node-1    /var/lib/spinifex/predastore/distributed/db/node-1
+# Symlink the original path back so Predastore finds its data unchanged
+sudo ln -s /mnt/nvme-1/node-2 /var/lib/spinifex/predastore/cluster/node-2

 # Persist the mount across reboots
 echo "/dev/nvme1n1  /mnt/nvme-1  auto  defaults  0  2" | sudo tee -a /etc/fstab
@@ -80,7 +80,7 @@ All Spinifex nodes must include the following in their malware protection scope.

 | Path | Contents |
 |------|----------|
-| `/etc/spinifex/` | Cluster config directory: `spinifex.toml`, `predastore.toml`, `nats.conf`. |
+| `/etc/spinifex/` | Cluster config directory: `spinifex.toml`, `awsgw.toml`, `nats.conf`, and `predastore/predastore.toml`. |
 | `/etc/spinifex/ca.pem`, `/etc/spinifex/ca.key` | Cluster root CA certificate and private key (leader only). |
 | `/etc/spinifex/server.pem`, `/etc/spinifex/server.key` | Per-node TLS certificate and key. |
 | `/etc/spinifex/master.key` | Master encryption key that wraps per-volume DEKs and IAM secrets. **Highest-value artifact on the node.** |
@@ -85,9 +85,9 @@ The inventory in [§1](#1-inbound-listeners)–[§2](#2-outbound-connections) sa
 | 4432 | Formation server | HTTPS | External (bootstrap only) | Cluster join coordination; active only while a join token is valid. See *Formation port lifecycle* below. | Short-lived bearer token + TLS¹ |
 | 4222 | spinifex-nats (client) | NATS + TLS | Cluster | Internal service bus for EC2/EBS/VPC/S3 handlers | Token + mutual TLS (cluster CA) |
 | 4248 | spinifex-nats (cluster) | NATS + TLS | Cluster | Inter-node NATS federation | Token + mutual TLS (cluster CA) |
-| 8443 | spinifex-predastore | HTTPS | Cluster | S3-compatible object storage (AMIs, snapshots, user objects) | AWS SigV4 + TLS |
-| 6660–6662 | predastore (Raft) | TCP | Cluster | Metadata consensus (3 nodes) | Cluster network only |
-| 9991–9993 | predastore (data shards) | TCP | Cluster | Erasure-coded data shard transport | Cluster network only |
+| 8443 | spinifex-predastore (gate) | HTTPS | Cluster | S3-compatible object storage (AMIs, snapshots, user objects) | AWS SigV4 + TLS |
+| 6660 | predastore (blob node) | QUIC / UDP | Cluster | Erasure-coded object shard transport between hosts. Multi-node clusters only — see *Predastore ports* below. | TLS 1.3, server certificate verified against the cluster CA |
+| 7660 | predastore (meta node) | QUIC / UDP | Cluster | Raft consensus over global state — buckets and the object index — between hosts. Multi-node clusters only. | TLS 1.3, server certificate verified against the cluster CA |
 | 6641 | OVN Northbound DB (client) | OVSDB/TCP | Cluster | Logical network topology consumed by vpcd | Cluster network only; TLS planned |
 | 6642 | OVN Southbound DB (client) | OVSDB/TCP | Cluster | Chassis / port / MAC binding state | Cluster network only; TLS planned |
 | 6643 | OVN Northbound DB (RAFT) | OVSDB/TCP | Cluster | NB database RAFT replication between the 3 quorum nodes | Cluster network only; TLS planned |
@@ -97,6 +97,10 @@ The inventory in [§1](#1-inbound-listeners)–[§2](#2-outbound-connections) sa

 ¹ **Formation port lifecycle.** 4432 opens during `spx admin init` / `spx admin join` while a bootstrap token is outstanding and closes once the cluster is formed (token TTL default 30 min, `--token-ttl`). The server presents an ephemeral self-signed cert that pre-dates trust bootstrap, so the joining node does not verify the certificate chain for this single dial. Authenticity rests on the operator supplying the leader address out-of-band plus possession of the bearer token. Document in the security plan so reviewers do not flag 4432 as a persistent open port.

+**Predastore ports.** A Predastore cluster is described in `/etc/spinifex/predastore/predastore.toml` as a set of `[[host]]` blocks — one per machine, each running a single process — with the nodes pinned to it declared under `[[host.node]]`. There are three roles: a `gate` serving the S3 API, a `blob` node holding erasure-coded object shards, and a `meta` node in the Raft quorum over global state. Ports must be unique within a host but are not unique across the cluster, so **every machine uses the same three ports** — 8443, 6660 and 7660. It is three fixed ports per machine, not a range, and adding machines does not widen it.
+
+Nodes on the same host talk over an in-process pipe and bind no socket at all, so a single-node install opens only 8443; 6660 and 7660 appear only once a second host exists. The gate is dialled by S3 clients but never by peer nodes, so it binds no QUIC socket of its own — it takes an ephemeral UDP port to dial out with.
+
 **Development-only listeners.** When `dev_networking=true`, QEMU opens arbitrary host TCP ports for SSH port-forwarding into guest VMs. Production installs (the `/etc/spinifex` layout) do not enable this; it must not appear on compliance nodes.

 ## 2. Outbound Connections
@@ -114,7 +118,7 @@ Spinifex nodes initiate a small, fixed set of outbound connections.
 | `https://d2yp8ipz5jfqcw.cloudfront.net` | Alpine image for managed HAProxy load-balancer | HTTPS | TLS + checksum verification |
 | `https://install.mulgadc.com/install` | One-shot install telemetry POST on `spx admin init` / `join`. | HTTPS | TLS |

-**To peer nodes (cluster-internal):** NATS federation (4248), Predastore S3 (8443), OVN NB/SB (6641/6642) — see [§3](#3-cross-node-internal-connections) for encryption and verification of each. The daemon also probes local Predastore Raft status at `<bind IP>:6660/status` (TLS, cluster CA) and local NATS monitoring at `127.0.0.1:8222/varz` (loopback HTTP).
+**To peer nodes (cluster-internal):** NATS federation (4248), Predastore S3 (8443), OVN NB/SB (6641/6642) — see [§3](#3-cross-node-internal-connections) for encryption and verification of each. The daemon also polls local NATS monitoring at `127.0.0.1:8222/varz` (loopback HTTP). It opens no connection to Predastore for status: the storage topology it reports comes from reading `predastore.toml`, and no Predastore node serves a status endpoint.

 **Update checks and metadata.** Spinifex does not check for updates and does not consume a cloud metadata service (`169.254.169.254` is served *by* the cluster to guest VMs). Node software updates come from the operator's OS package channel. The install-telemetry endpoint above is the only vendor-operated destination contacted by a node; closed-egress deployments should disable it and record the opt-out in the security plan.

@@ -127,14 +131,14 @@ Control-plane and data-plane traffic between Spinifex nodes, for completeness an
 | Connection | Port(s) | Encryption / Auth | Notes |
 |-----------|---------|-------------------|-------|
 | NATS cluster routes | 4248 | Mutual TLS + cluster token | Full mesh between NATS servers |
-| Predastore S3 | 8443 | TLS + AWS SigV4 | Cross-node object reads/writes |
-| Predastore Raft | 6660–6662 | Cluster network only | Metadata consensus |
-| Predastore shards | 9991–9993 | Cluster network only | Erasure-coded data shards |
+| Predastore S3 (gate) | TCP 8443 | TLS + AWS SigV4 | Cross-node object reads/writes |
+| Predastore blob | UDP 6660 | QUIC with TLS 1.3; server certificate verified against the cluster CA | Erasure-coded object shards. Same port on every machine. |
+| Predastore meta | UDP 7660 | QUIC with TLS 1.3; server certificate verified against the cluster CA | Raft consensus over buckets and the object index. Same port on every machine. |
 | OVN NB/SB (client) | 6641 / 6642 | Cluster network only (TLS planned) | Network control plane; vpcd and ovn-controller dial the quorum |
 | OVN NB/SB (RAFT) | 6643 / 6644 | Cluster network only (TLS planned) | NB/SB database replication across the 3 quorum nodes |
 | OVN tunnels (Geneve) | UDP 6081 | None | Tenant traffic overlay between chassis, inside the cluster subnet |

-Nodes **must** sit on a network segment that is not routed to tenant/guest VMs or to the internet. Predastore Raft/shards and OVN DBs are cluster-internal and must not be reachable from anywhere else.
+Nodes **must** sit on a network segment that is not routed to tenant/guest VMs or to the internet. The Predastore blob and meta transports and the OVN DBs are cluster-internal and must not be reachable from anywhere else.

 ## 4. Limiting Controls

@@ -145,8 +149,8 @@ Default external surface is three listeners — **9999** (AWS API), **3000** (UI
 tcp dport { 22, 9999, 3000 } accept

 # Cluster-only: replace 10.0.1.0/24 with your cluster CIDR
-ip saddr 10.0.1.0/24 tcp dport { 4222, 4248, 8443, 6641-6644, 6660-6662, 9991-9993 } accept
-ip saddr 10.0.1.0/24 udp dport 6081 accept
+ip saddr 10.0.1.0/24 tcp dport { 4222, 4248, 8443, 6641-6644 } accept
+ip saddr 10.0.1.0/24 udp dport { 6081, 6660, 7660 } accept

 # Default deny
 tcp dport 0-65535 drop
@@ -163,7 +167,7 @@ Every listener and outbound destination is controlled by one of these files. Cha
 |------|------|----------|
 | `/etc/spinifex/spinifex.toml` | `nodes.<node>.{awsgw,nats,predastore,daemon}.host`, `nodes.<node>.vpcd.ovn_{nb,sb}_addr`, `nodes.<node>.daemon.dev_networking` | Per-service bind addresses/ports; dev-mode QEMU port forwarding. |
 | `/etc/spinifex/nats.conf` | `listen`, `cluster.listen`, `cluster.routes`, `http`, `tls`, `cluster.authorization` | NATS client/cluster/monitoring listeners, peer routes, TLS, cluster token. |
-| `/etc/spinifex/predastore.toml` | `[[host]].bind_addr`, `[[host]].public_addr`, `[[node]].role`, `tls.*` | Predastore intra-cluster listener (one socket per host, carrying every node pinned to it), node placement, TLS certs. The S3 listener itself is set by the service's `--host`/`--port`. |
+| `/etc/spinifex/predastore/predastore.toml` | `[[host]].bind_addr`, `[[host]].addr`, `[[host]].tls_cert`, `[[host]].tls_key`, `[[host.node]].role`, `[[host.node]].port` | Predastore host and node layout: `bind_addr` is the address the host's sockets bind, `addr` is the address peer hosts dial it on, and both carry no port — the nodes pinned to the host supply their own. The service's `--host`/`--port` override the bind address and the gate's S3 port. |
 | OVN packages (`ovn-central`, `ovn-host`) | `ovn-nb-db`, `ovn-sb-db` (via `ovs-vsctl set open_vswitch …`) | OVN DB bind addresses. |
 | Spinifex UI service | Built-in defaults: `host = "0.0.0.0"`, `port = 3000`. No `spinifex.toml` block today. | UI listener. |
 | `spx admin init` / `spx admin join` | `--port`, `--token-ttl`, `--no-telemetry` (or `SPX_NO_TELEMETRY=1`) | Formation port, token TTL, telemetry opt-out. |