diff --git a/INSTALL.md b/INSTALL.md index 1310aaf22..31325dc55 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -560,19 +560,32 @@ When deploying a multi-node cluster, you define a **region** (e.g., `ap-southeas Cluster formation is automatic. When initializing with `--nodes 3`, the init node starts a formation server and waits for other nodes to join. Once all nodes have registered, each node receives the full cluster topology — credentials, CA certificates, NATS routes, Predastore peer lists — and generates its own configuration files. No manual configuration synchronization is required. -## Simulated Multi-Node (Single Server) +## Choose a Deployment Mode -For development and testing, a 3-node cluster can be simulated on a single machine using loopback addresses. Linux handles the full `127.0.0.0/8` range natively, so `127.0.0.1`, `127.0.0.2`, and `127.0.0.3` all work without any network configuration. +| | Simulated (Option A) | Real Multi-Server (Option B) | +|---|---|---| +| **Servers** | 1 machine, loopback IPs | 3+ physical/virtual servers | +| **Use case** | Development & testing | Production & staging | +| **Network** | `127.0.0.1/2/3` (no setup) | Real LAN IPs | +| **Data dirs** | `~/node1/`, `~/node2/`, `~/node3/` | `~/hive/` on each server | -Each node uses a separate data directory (`~/node1/`, `~/node2/`, `~/node3/`) to isolate configuration, logs, and state. +Both options use `HIVE_NODE1`, `HIVE_NODE2`, and `HIVE_NODE3` environment variables. Set them in the option you choose, then all subsequent sections (verification, usage, shutdown) reference them. -### 1. Build +### Option A: Simulated Multi-Node (Single Server) + +For development and testing, a 3-node cluster can be simulated on a single machine using loopback addresses. Linux handles the full `127.0.0.0/8` range natively, so `127.0.0.1`, `127.0.0.2`, and `127.0.0.3` all work without any network configuration. Each node uses a separate data directory to isolate configuration, logs, and state. + +#### 1. Build and Set Variables ```bash make build + +export HIVE_NODE1=127.0.0.1 +export HIVE_NODE2=127.0.0.2 +export HIVE_NODE3=127.0.0.3 ``` -### 2. Form the Cluster +#### 2. Form the Cluster The formation process requires running init and join commands concurrently. The init node blocks until all expected nodes have joined. @@ -582,8 +595,8 @@ The formation process requires running init and join commands concurrently. The ./bin/hive admin init \ --node node1 \ --nodes 3 \ - --bind 127.0.0.1 \ - --cluster-bind 127.0.0.1 \ + --bind $HIVE_NODE1 \ + --cluster-bind $HIVE_NODE1 \ --port 4432 \ --hive-dir ~/node1/ \ --config-dir ~/node1/config/ \ @@ -591,16 +604,16 @@ The formation process requires running init and join commands concurrently. The --az ap-southeast-2a ``` -This starts a formation server on `127.0.0.1:4432` and waits for 2 more nodes to join. +This starts a formation server on `$HIVE_NODE1:4432` and waits for 2 more nodes to join. **Terminal 2 — Join node 2 (while init is running):** ```bash ./bin/hive admin join \ --node node2 \ - --bind 127.0.0.2 \ - --cluster-bind 127.0.0.2 \ - --host 127.0.0.1:4432 \ + --bind $HIVE_NODE2 \ + --cluster-bind $HIVE_NODE2 \ + --host $HIVE_NODE1:4432 \ --data-dir ~/node2/ \ --config-dir ~/node2/config/ \ --region ap-southeast-2 \ @@ -612,9 +625,9 @@ This starts a formation server on `127.0.0.1:4432` and waits for 2 more nodes to ```bash ./bin/hive admin join \ --node node3 \ - --bind 127.0.0.3 \ - --cluster-bind 127.0.0.3 \ - --host 127.0.0.1:4432 \ + --bind $HIVE_NODE3 \ + --cluster-bind $HIVE_NODE3 \ + --host $HIVE_NODE1:4432 \ --data-dir ~/node3/ \ --config-dir ~/node3/config/ \ --region ap-southeast-2 \ @@ -623,7 +636,7 @@ This starts a formation server on `127.0.0.1:4432` and waits for 2 more nodes to All three processes will exit with a cluster summary once formation is complete. -### 3. Start Services +#### 3. Start Services Start services on each node (in separate terminals or background): @@ -633,88 +646,11 @@ HIVE_SKIP_BUILD=true UI=false ./scripts/start-dev.sh ~/node2/ HIVE_SKIP_BUILD=true UI=false ./scripts/start-dev.sh ~/node3/ ``` -### 4. Verify the Cluster - -Check that all daemons are healthy: - -```bash -curl -s http://127.0.0.1:4432/health -curl -s http://127.0.0.2:4432/health -curl -s http://127.0.0.3:4432/health -``` - -Check NATS cluster routing (from any node): - -```bash -grep -i "route\|cluster" ~/node1/logs/nats.log -``` - -Check Predastore Raft consensus: - -```bash -grep -i "leader\|election" ~/node1/logs/predastore.log -``` - -### 5. Using the Cluster - -When using the AWS CLI, specify the endpoint URL pointing to any node's gateway: - -```bash -AWS_ENDPOINT_URL=https://127.0.0.1:9999 aws ec2 describe-instances -AWS_ENDPOINT_URL=https://127.0.0.2:9999 aws ec2 describe-instances -``` - -Each node has its own logs for debugging: - -```bash -tail -f ~/node1/logs/hive.log -tail -f ~/node2/logs/hive.log -tail -f ~/node3/logs/hive.log -``` - -### 6. Stop the Cluster - -Stop services on each node in reverse order: - -```bash -./scripts/stop-dev.sh ~/node3/ -./scripts/stop-dev.sh ~/node2/ -./scripts/stop-dev.sh ~/node1/ -``` - -### 7. Clean Up - -To fully remove the simulated cluster: - -```bash -rm -rf ~/node1/ ~/node2/ ~/node3/ -``` - -### Alternative: IP Aliases - -For a more production-like simulation using dedicated IP addresses on a network interface: - -```bash -sudo ip addr add 10.11.12.1/24 dev eth0 -sudo ip addr add 10.11.12.2/24 dev eth0 -sudo ip addr add 10.11.12.3/24 dev eth0 -``` - -Replace `eth0` with your network interface name (e.g., `enp0s3`, `ens33`). Then use these IPs instead of `127.0.0.x` in the formation commands above. - -To remove the aliases: - -```bash -sudo ip addr del 10.11.12.1/24 dev eth0 -sudo ip addr del 10.11.12.2/24 dev eth0 -sudo ip addr del 10.11.12.3/24 dev eth0 -``` - -## Real Multi-Node (Physical Servers) +### Option B: Real Multi-Node (Physical Servers) For production or production-like deployments across multiple physical servers or VMs. -### Prerequisites +#### Prerequisites On **each server**: @@ -730,21 +666,23 @@ cd hive make build ``` -3. Identify the bind IP for each server (e.g., the primary network interface IP): +3. Identify the bind IP for each server: ```bash ip addr show | grep "inet " ``` -### Example: 3-Node Cluster +#### 1. Set Variables -Assume three servers with these IPs: +Export the IPs for your three servers (replace with your actual IPs): -| Node | IP Address | Role | -|------|------------|------| -| node1 | 192.168.1.10 | Init node | -| node2 | 192.168.1.11 | Join node | -| node3 | 192.168.1.12 | Join node | +```bash +export HIVE_NODE1=192.168.1.10 +export HIVE_NODE2=192.168.1.11 +export HIVE_NODE3=192.168.1.12 +``` + +#### 2. Form the Cluster **Server 1 — Initialize the cluster:** @@ -754,8 +692,8 @@ cd ~/Development/mulga/hive ./bin/hive admin init \ --node node1 \ --nodes 3 \ - --bind 192.168.1.10 \ - --cluster-bind 192.168.1.10 \ + --bind $HIVE_NODE1 \ + --cluster-bind $HIVE_NODE1 \ --port 4432 \ --hive-dir ~/hive/ \ --config-dir ~/hive/config/ \ @@ -770,9 +708,9 @@ cd ~/Development/mulga/hive ./bin/hive admin join \ --node node2 \ - --bind 192.168.1.11 \ - --cluster-bind 192.168.1.11 \ - --host 192.168.1.10:4432 \ + --bind $HIVE_NODE2 \ + --cluster-bind $HIVE_NODE2 \ + --host $HIVE_NODE1:4432 \ --data-dir ~/hive/ \ --config-dir ~/hive/config/ \ --region us-east-1 \ @@ -786,9 +724,9 @@ cd ~/Development/mulga/hive ./bin/hive admin join \ --node node3 \ - --bind 192.168.1.12 \ - --cluster-bind 192.168.1.12 \ - --host 192.168.1.10:4432 \ + --bind $HIVE_NODE3 \ + --cluster-bind $HIVE_NODE3 \ + --host $HIVE_NODE1:4432 \ --data-dir ~/hive/ \ --config-dir ~/hive/config/ \ --region us-east-1 \ @@ -797,13 +735,7 @@ cd ~/Development/mulga/hive All three processes will exit with a cluster summary once formation is complete. -### Service Co-location - -Every node that runs EC2 instances (compute) **must** also run the viperblock service. EBS volumes are mounted locally via Unix domain sockets (NBD) — the daemon publishes to a node-specific NATS topic (`ebs.{nodeName}.mount`) which is handled by the viperblock instance on the same server. Nodes without viperblock cannot run EC2 instances. - -In the default deployment, every node runs all services (NATS, Predastore, Viperblock, Daemon, Gateway), which satisfies this requirement. - -### Start Services on Each Server +#### 3. Start Services After formation completes, start services on **all servers**: @@ -811,45 +743,89 @@ After formation completes, start services on **all servers**: ./scripts/start-dev.sh ``` -### Verify the Cluster +## Verify the Cluster -From any server, check daemon health: +From any node, view cluster status and resource capacity: ```bash -curl -s http://192.168.1.10:4432/health -curl -s http://192.168.1.11:4432/health -curl -s http://192.168.1.12:4432/health +./bin/hive get nodes +# NAME STATUS IP REGION AZ UPTIME VMs SERVICES +# node1 Ready $HIVE_NODE1 ... ... 2m 0 nats,predastore,viperblock,daemon,... +# node2 Ready $HIVE_NODE2 ... ... 2m 0 nats,predastore,viperblock,daemon +# node3 Ready $HIVE_NODE3 ... ... 2m 0 nats,predastore,viperblock,daemon + +./bin/hive top nodes +# NAME CPU (used/total) MEM (used/total) VMs +# node1 0/16 0.0Gi/30.6Gi 0 +# node2 0/16 0.0Gi/30.6Gi 0 +# node3 0/16 0.0Gi/30.6Gi 0 ``` -Check NATS cluster mesh: +All nodes should show `Ready` status. Use `hive get vms` to see running instances across the cluster. + +Check individual daemon health endpoints: ```bash -grep -i "route\|cluster" ~/hive/logs/nats.log +curl -s http://$HIVE_NODE1:4432/health +curl -s http://$HIVE_NODE2:4432/health +curl -s http://$HIVE_NODE3:4432/health ``` -Check Predastore Raft: +Check NATS cluster routing (from any node): ```bash -grep -i "leader\|election" ~/hive/logs/predastore.log +grep -i "route\|cluster" ~/node1/logs/nats.log # simulated +grep -i "route\|cluster" ~/hive/logs/nats.log # real multi-server ``` -### Trust CA Certificate +Check Predastore Raft consensus: -The CA certificate is generated by the init node and distributed to all joining nodes during formation. On **each server**, trust the CA: +```bash +grep -i "leader\|election" ~/node1/logs/predastore.log # simulated +grep -i "leader\|election" ~/hive/logs/predastore.log # real multi-server +``` + +## Trust CA Certificate + +The CA certificate is generated by the init node and distributed to all joining nodes during formation. On **each node** (or the single machine for simulated mode), trust the CA: ```bash sudo cp ~/hive/config/ca.pem /usr/local/share/ca-certificates/hive-ca.crt sudo update-ca-certificates ``` -### Using the Cluster +For simulated mode, the config directory is under each node's data directory — copy from any node since the CA is the same: + +```bash +sudo cp ~/node1/config/ca.pem /usr/local/share/ca-certificates/hive-ca.crt +sudo update-ca-certificates +``` + +Set the AWS profile: + +```bash +export AWS_PROFILE=hive +``` + +## Using the Cluster Connect to any node's AWS Gateway: ```bash -AWS_ENDPOINT_URL=https://192.168.1.10:9999 AWS_PROFILE=hive aws ec2 describe-instances +AWS_ENDPOINT_URL=https://$HIVE_NODE1:9999 aws ec2 describe-instances +AWS_ENDPOINT_URL=https://$HIVE_NODE2:9999 aws ec2 describe-instances ``` +Check logs on each node for debugging. Log locations depend on your deployment mode — `~/node{1,2,3}/logs/` for simulated, `~/hive/logs/` for real multi-server. + +## Advanced Configuration + +### Service Co-location + +Every node that runs EC2 instances (compute) **must** also run the viperblock service. EBS volumes are mounted locally via Unix domain sockets (NBD) — the daemon publishes to a node-specific NATS topic (`ebs.{nodeName}.mount`) which is handled by the viperblock instance on the same server. Nodes without viperblock cannot run EC2 instances. + +In the default deployment, every node runs all services (NATS, Predastore, Viperblock, Daemon, Gateway), which satisfies this requirement. + ### Dual NIC Configuration If servers have separate management and cluster network interfaces, use `--cluster-bind` to specify the cluster network IP: @@ -870,12 +846,44 @@ If servers have separate management and cluster network interfaces, use `--clust The `--bind` IP is used for the formation server, daemon, and AWS gateway. The `--cluster-bind` IP is used for NATS cluster routing and Predastore Raft consensus. If `--cluster-bind` is not specified, it defaults to the `--bind` IP. -### Shutdown +### IP Aliases (Simulated Mode) -Stop services on each server: +For a more production-like simulation using dedicated IP addresses on a network interface: ```bash +sudo ip addr add 10.11.12.1/24 dev eth0 +sudo ip addr add 10.11.12.2/24 dev eth0 +sudo ip addr add 10.11.12.3/24 dev eth0 +``` + +Replace `eth0` with your network interface name (e.g., `enp0s3`, `ens33`). Then export these as your `HIVE_NODE` variables instead of the loopback addresses. + +To remove the aliases: + +```bash +sudo ip addr del 10.11.12.1/24 dev eth0 +sudo ip addr del 10.11.12.2/24 dev eth0 +sudo ip addr del 10.11.12.3/24 dev eth0 +``` + +## Shutdown + +Stop services on each node: + +```bash +# Simulated — stop in reverse order: +./scripts/stop-dev.sh ~/node3/ +./scripts/stop-dev.sh ~/node2/ +./scripts/stop-dev.sh ~/node1/ + +# Real multi-server — run on each server: ./scripts/stop-dev.sh ``` -Note: if EC2 instances are running, the stop process will gracefully terminate them, unmount attached EBS volumes (via NBD), and flush the write-ahead-log (WAL) to the S3 server (Predastore). This may take a few minutes depending on instance volume sizes. +If EC2 instances are running, the stop process will gracefully terminate them, unmount attached EBS volumes (via NBD), and flush the write-ahead-log (WAL) to the S3 server (Predastore). This may take a few minutes depending on instance volume sizes. + +To fully remove a simulated cluster's data: + +```bash +rm -rf ~/node1/ ~/node2/ ~/node3/ +``` diff --git a/cmd/hive/cmd/admin.go b/cmd/hive/cmd/admin.go index 562d28a55..ea1f143a5 100644 --- a/cmd/hive/cmd/admin.go +++ b/cmd/hive/cmd/admin.go @@ -831,6 +831,7 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID, PredastoreNodeID: predastoreNodeID, Services: services, + RemoteNodes: buildRemoteNodes(allNodes, node), } // Generate config files @@ -1134,6 +1135,7 @@ func runAdminJoin(cmd *cobra.Command, args []string) { PredastoreNodeID: predastoreNodeID, Services: services, + RemoteNodes: buildRemoteNodes(statusResp.Nodes, node), } // Generate config files @@ -1177,3 +1179,26 @@ func runAdminJoin(cmd *cobra.Command, args []string) { fmt.Println(" ./scripts/start-dev.sh") fmt.Println() } + +// buildRemoteNodes converts formation NodeInfo into RemoteNode entries, +// excluding the local node. This puts all cluster members into hive.toml +// so config is the source of truth for expected cluster membership. +func buildRemoteNodes(allNodes map[string]formation.NodeInfo, localNode string) []admin.RemoteNode { + var remote []admin.RemoteNode + for name, n := range allNodes { + if name == localNode { + continue + } + remote = append(remote, admin.RemoteNode{ + Name: name, + Host: n.BindIP, + Region: n.Region, + AZ: n.AZ, + Services: n.Services, + }) + } + sort.Slice(remote, func(i, j int) bool { + return remote[i].Name < remote[j].Name + }) + return remote +} diff --git a/cmd/hive/cmd/get.go b/cmd/hive/cmd/get.go index b3e959929..958f3eb28 100644 --- a/cmd/hive/cmd/get.go +++ b/cmd/hive/cmd/get.go @@ -162,14 +162,21 @@ func runGetNodes(cmd *cobra.Command, args []string) { respondedNodes[resp.Node] = resp } - // Build table: include all known nodes from config, mark non-responders as NotReady + // Build table: union of config-known nodes + NATS responders tableData := pterm.TableData{ {"NAME", "STATUS", "IP", "REGION", "AZ", "UPTIME", "VMs", "SERVICES"}, } - // Collect and sort node names for stable output - nodeNames := make([]string, 0, len(cfg.Nodes)) + // Collect all node names: config + responded (union) + nodeSet := make(map[string]struct{}) for name := range cfg.Nodes { + nodeSet[name] = struct{}{} + } + for name := range respondedNodes { + nodeSet[name] = struct{}{} + } + nodeNames := make([]string, 0, len(nodeSet)) + for name := range nodeSet { nodeNames = append(nodeNames, name) } sort.Strings(nodeNames) diff --git a/cmd/hive/cmd/templates/hive.toml b/cmd/hive/cmd/templates/hive.toml index 2e918c141..3214acf39 100644 --- a/cmd/hive/cmd/templates/hive.toml +++ b/cmd/hive/cmd/templates/hive.toml @@ -5,6 +5,7 @@ node = "{{.Node}}" [nodes.{{.Node}}] node = "{{.Node}}" +host = "{{.BindIP}}" region = "{{.Region}}" az = "{{.Az}}" {{- if .Services}} @@ -43,4 +44,15 @@ accesskey = "{{.AccessKey}}" secretkey = "{{.SecretKey}}" {{- if gt .PredastoreNodeID 0}} node_id = {{.PredastoreNodeID}} +{{- end}} +{{- range .RemoteNodes}} + +[nodes.{{.Name}}] +node = "{{.Name}}" +host = "{{.Host}}" +region = "{{.Region}}" +az = "{{.AZ}}" +{{- if .Services}} +services = [{{range $i, $s := .Services}}{{if $i}}, {{end}}"{{$s}}"{{end}}] +{{- end}} {{- end}} \ No newline at end of file diff --git a/cmd/hive/cmd/top.go b/cmd/hive/cmd/top.go index 9aabfc78e..449b14797 100644 --- a/cmd/hive/cmd/top.go +++ b/cmd/hive/cmd/top.go @@ -71,9 +71,16 @@ func runTopNodes(cmd *cobra.Command, args []string) { respondedNodes[resp.Node] = resp } - // Node resource table - nodeNames := make([]string, 0, len(cfg.Nodes)) + // Node resource table: union of config-known nodes + NATS responders + nodeSet := make(map[string]struct{}) for name := range cfg.Nodes { + nodeSet[name] = struct{}{} + } + for name := range respondedNodes { + nodeSet[name] = struct{}{} + } + nodeNames := make([]string, 0, len(nodeSet)) + for name := range nodeSet { nodeNames = append(nodeNames, name) } sort.Strings(nodeNames) diff --git a/docs/development/improvements/hive-cluster-stats.md b/docs/development/improvements/hive-cluster-stats.md index 5a486aab9..059559b56 100644 --- a/docs/development/improvements/hive-cluster-stats.md +++ b/docs/development/improvements/hive-cluster-stats.md @@ -62,6 +62,15 @@ Bottom table aggregates free capacity across all Ready nodes. ## Design +### Config as Source of Truth + +During multi-node formation, each node's `hive.toml` includes ALL cluster nodes — the local node with full config (daemon, nats, predastore subsections) and remote nodes with basic info (host, region, AZ, services). This makes config the authoritative list of expected cluster members. + +The CLI builds its node list from the union of config nodes + NATS responders: +- **Config nodes that respond** → shown as `Ready` with live data from NATS +- **Config nodes that don't respond** → shown as `NotReady` with host/region/AZ from config +- **NATS responders not in config** → shown as `Ready` (safety net for dynamic additions) + ### NATS Topics Two fan-out topics (no queue group — all daemons respond): @@ -95,8 +104,11 @@ New method that returns total/allocated vCPU and memory plus per-instance-type a | File | Change | |------|--------| | `hive/config/config.go` | Added `NodeStatusResponse`, `InstanceTypeCap`, `VMInfo`, `NodeVMsResponse` types | +| `hive/admin/admin.go` | Added `RemoteNode` type and `RemoteNodes` field to `ConfigSettings` | | `hive/daemon/daemon_handlers.go` | Added `daemonIP()`, `handleNodeStatus()`, `handleNodeVMs()` handlers | | `hive/daemon/daemon.go` | Added `hive.node.status` and `hive.node.vms` subscriptions in `subscribeAll()`, added `GetResourceStats()` method to `ResourceManager` | +| `cmd/hive/cmd/admin.go` | Added `buildRemoteNodes()` helper; populate `RemoteNodes` in init and join flows | +| `cmd/hive/cmd/templates/hive.toml` | Added `host` to local node, template loop for remote nodes | | `cmd/hive/cmd/get.go` | **New file** — `get nodes`, `get vms` commands with pterm table output | | `cmd/hive/cmd/top.go` | **New file** — `top nodes` command with node resource + instance type capacity tables | | `tests/e2e/run-e2e.sh` | Added Phase 1b (get nodes, top nodes, get vms empty) and Phase 5a-pre (get vms with running VM) | diff --git a/hive/admin/admin.go b/hive/admin/admin.go index c6fad9331..a27356fba 100644 --- a/hive/admin/admin.go +++ b/hive/admin/admin.go @@ -21,6 +21,15 @@ import ( "gopkg.in/ini.v1" ) +// RemoteNode holds basic info about a remote cluster node for config generation. +type RemoteNode struct { + Name string + Host string + Region string + AZ string + Services []string +} + type ConfigSettings struct { AccessKey string SecretKey string @@ -45,6 +54,9 @@ type ConfigSettings struct { // Node capabilities Services []string + + // Other nodes in the cluster (for config source of truth) + RemoteNodes []RemoteNode } // PredastoreNodeConfig describes a single Predastore node for multi-node config generation.