chore: delete legacy scripts/iac

scripts/iac/README.md

@@ -1,9 +0,0 @@
-# IAC
-
-Collection of OpenTofu/Terraform templates for creating Spinifex instances.
-
-| Template | Description |
-|---|---|
-| [proxmox/](proxmox/) | Provisions a 3-node Spinifex cluster on Proxmox VE bare-metal hypervisors |
-| [aws/](aws/) | Provisions infrastructure using the Terraform AWS provider, with MulgaOS AWS-compatible API as the local endpoint |
-| [proxmox-template/](proxmox-template/) | Builds a Proxmox VM template with all Spinifex dependencies pre-installed (QEMU, Go, repos, modules) |

scripts/iac/aws/README.md

@@ -1,206 +0,0 @@
-# MulgaOS AWS Infrastructure as Code
-
-This directory contains OpenTofu/Terraform configuration files for deploying infrastructure on MulgaOS using AWS-compatible APIs.
-
-## Prerequisites
-
-1. **OpenTofu or Terraform installed**
-   ```bash
-   # Install OpenTofu (recommended)
-   # See: https://opentofu.org/docs/intro/install/
-
-   # Or use Terraform
-   # See: https://developer.hashicorp.com/terraform/install
-   ```
-
-2. **MulgaOS services running**
-   - `spinifex daemon` - Core daemon with NATS subscriptions
-   - `awsd` - AWS API gateway (default: http://localhost:8080)
-   - `predastore` - S3-compatible storage
-   - `viperblock` - EBS-compatible block storage
-
-3. **AWS credentials configured**
-   ```bash
-   # Option 1: Environment variables
-   export AWS_ACCESS_KEY_ID=your_access_key
-   export AWS_SECRET_ACCESS_KEY=your_secret_key
-
-   # Option 2: AWS CLI profile
-   aws configure --profile spinifex
-   export AWS_PROFILE=spinifex
-   ```
-
-4. **AMI available**
-   Before running, ensure you have a Debian 13 AMI imported:
-   ```bash
-   # Import an AMI using the spinifex CLI or aws CLI
-   aws --endpoint-url http://localhost:8080 ec2 describe-images
-   ```
-
-## Configuration
-
-### Environment Variables
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `TF_VAR_mulgaos_endpoint` | `http://localhost:8080` | MulgaOS API endpoint |
-| `TF_VAR_region` | `ap-southeast-2` | AWS region (for compatibility) |
-| `AWS_ACCESS_KEY_ID` | - | MulgaOS access key |
-| `AWS_SECRET_ACCESS_KEY` | - | MulgaOS secret key |
-
-### terraform.tfvars (optional)
-
-Create a `terraform.tfvars` file for custom settings:
-
-```hcl
-mulgaos_endpoint = "http://192.168.1.100:8080"
-region           = "us-west-2"
-```
-
-## Usage
-
-### Initialize
-
-```bash
-cd iac/aws
-tofu init
-# or: terraform init
-```
-
-### Validate
-
-```bash
-tofu validate
-# or: terraform validate
-```
-
-### Plan
-
-```bash
-tofu plan
-# or: terraform plan
-```
-
-### Apply
-
-```bash
-tofu apply
-# or: terraform apply
-```
-
-### Destroy
-
-```bash
-tofu destroy
-# or: terraform destroy
-```
-
-## What Gets Created
-
-The `main.tf` creates:
-
-1. **VPC** (`10.1.1.0/16`) - Virtual Private Cloud
-2. **Internet Gateway** - For public internet access
-3. **Public Subnet** (`10.1.2.0/24`) - With auto-assign public IP
-4. **Route Table** - Routes `0.0.0.0/0` through IGW
-5. **Security Group** - Allows SSH (22), HTTP (80), HTTPS (443) inbound
-6. **Placement Group** - Spread strategy for HA
-7. **3 EC2 Instances** - t3.small Debian 13 VMs
-8. **SSH Key Pair** - ED25519 key saved to `spinifex-test.pem`
-
-## SSH Access
-
-After `tofu apply`, SSH into instances:
-
-```bash
-# Using the generated key
-ssh -i spinifex-test.pem admin@<public_ip>
-
-# For local development with QEMU port forwarding
-# Find the forwarded port from QEMU args (-netdev user,id=net0,hostfwd=tcp:127.0.0.1:PORT-:22)
-ssh -p PORT -i spinifex-test.pem admin@127.0.0.1
-```
-
-## Troubleshooting
-
-### "No valid credential sources found"
-
-Ensure AWS credentials are set:
-```bash
-export AWS_ACCESS_KEY_ID=your_key
-export AWS_SECRET_ACCESS_KEY=your_secret
-```
-
-### "Error: No AMI found"
-
-Import a Debian 13 AMI first, or modify `main.tf` to use a different AMI filter.
-
-### Connection refused
-
-Ensure MulgaOS services are running:
-```bash
-# Check if awsd gateway is running
-curl http://localhost:8080/health
-
-# Check daemon status
-spinifex daemon status
-```
-
-### "InvalidAction" errors
-
-Some AWS operations may not be implemented yet. Check `TODO.md` for the current implementation status.
-
-## Customization
-
-### Using a different AMI
-
-Modify the `data "aws_ami"` block in `main.tf`:
-
-```hcl
-data "aws_ami" "custom" {
-  most_recent = true
-  owners      = ["self"]  # Your imported AMIs
-
-  filter {
-    name   = "name"
-    values = ["my-custom-ami-*"]
-  }
-}
-```
-
-### Changing instance count
-
-Modify the `count` parameter:
-
-```hcl
-resource "aws_instance" "vm" {
-  count = 5  # Change from 3 to 5
-  # ...
-}
-```
-
-### Different instance type
-
-```hcl
-resource "aws_instance" "vm" {
-  instance_type = "t3.medium"  # Change from t3.small
-  # ...
-}
-```
-
-## Files
-
-| File | Description |
-|------|-------------|
-| `main.tf` | Main Terraform/OpenTofu configuration |
-| `spinifex-test.pem` | Generated SSH private key (after apply) |
-| `.terraform/` | Provider plugins (after init) |
-| `terraform.tfstate` | State file (after apply) |
-| `terraform.tfvars` | Custom variables (optional) |
-
-## Related Documentation
-
-- [MulgaOS DEV.md](../../DEV.md) - Development guide
-- [MulgaOS TODO.md](../../TODO.md) - Implementation status
-- [OpenTofu Documentation](https://opentofu.org/docs/)
-- [AWS Provider Documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs)

scripts/iac/aws/basic/main.tf

@@ -1,307 +0,0 @@
-# Spinifex IaC Compatibility Test — "basic" suite
-#
-# Exercises every AWS resource Spinifex currently supports via tofu apply,
-# WITHOUT route tables, NACLs, or other unimplemented networking.
-#
-# Usage:
-#   cd spinifex/scripts/iac/aws/basic
-#   tofu init
-#   tofu plan
-#   tofu apply        # full round-trip against a running Spinifex cluster
-#   tofu destroy      # clean teardown
-
-terraform {
-  required_version = ">= 1.6.0"
-
-  required_providers {
-    aws = {
-      source  = "hashicorp/aws"
-      version = ">= 5.0"
-    }
-    tls = {
-      source  = "hashicorp/tls"
-      version = ">= 4.0"
-    }
-    local = {
-      source  = "hashicorp/local"
-      version = ">= 2.0"
-    }
-  }
-}
-
-# ---------------------------------------------------------------------------
-# Variables
-# ---------------------------------------------------------------------------
-
-variable "region" {
-  type    = string
-  default = "ap-southeast-2"
-}
-
-variable "mulgaos_endpoint" {
-  type        = string
-  default     = "https://localhost:9999"
-  description = "Spinifex AWS gateway endpoint"
-}
-
-# ---------------------------------------------------------------------------
-# Provider
-# ---------------------------------------------------------------------------
-
-provider "aws" {
-  region = var.region
-
-  endpoints {
-    ec2 = var.mulgaos_endpoint
-    s3  = var.mulgaos_endpoint
-    iam = var.mulgaos_endpoint
-    sts = var.mulgaos_endpoint
-  }
-
-  skip_credentials_validation = true
-  skip_metadata_api_check     = true
-  skip_requesting_account_id  = true
-  skip_region_validation      = true
-}
-
-# ---------------------------------------------------------------------------
-# Data sources
-# ---------------------------------------------------------------------------
-
-data "aws_availability_zones" "available" {
-  state = "available"
-}
-
-data "aws_ami" "debian13" {
-  most_recent = true
-  owners      = ["000000000000"]
-
-  filter {
-    name   = "name"
-    values = ["*debian-13*"]
-  }
-
-  filter {
-    name   = "virtualization-type"
-    values = ["hvm"]
-  }
-
-  filter {
-    name   = "root-device-type"
-    values = ["ebs"]
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 1. VPC + Subnet (no route table needed)
-# ---------------------------------------------------------------------------
-
-resource "aws_vpc" "test" {
-  cidr_block           = "10.99.0.0/16"
-  enable_dns_hostnames = true
-  enable_dns_support   = true
-
-  tags = {
-    Name = "spx-basic-test-vpc"
-  }
-}
-
-resource "aws_subnet" "test" {
-  vpc_id                  = aws_vpc.test.id
-  cidr_block              = "10.99.1.0/24"
-  availability_zone       = data.aws_availability_zones.available.names[0]
-  map_public_ip_on_launch = true
-
-  tags = {
-    Name = "spx-basic-test-subnet"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 2. Internet Gateway (attach to VPC, but no route table referencing it)
-# ---------------------------------------------------------------------------
-
-resource "aws_internet_gateway" "test" {
-  vpc_id = aws_vpc.test.id
-
-  tags = {
-    Name = "spx-basic-test-igw"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 3. Security Group — inbound SSH + HTTP, all outbound
-# ---------------------------------------------------------------------------
-
-resource "aws_security_group" "test" {
-  name        = "spx-basic-test-sg"
-  description = "Basic test: SSH + HTTP inbound, all outbound"
-  vpc_id      = aws_vpc.test.id
-
-  ingress {
-    description = "SSH"
-    from_port   = 22
-    to_port     = 22
-    protocol    = "tcp"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  ingress {
-    description = "HTTP"
-    from_port   = 80
-    to_port     = 80
-    protocol    = "tcp"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  egress {
-    description = "All outbound"
-    from_port   = 0
-    to_port     = 0
-    protocol    = "-1"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  tags = {
-    Name = "spx-basic-test-sg"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 4. Key Pair
-# ---------------------------------------------------------------------------
-
-resource "tls_private_key" "test" {
-  algorithm = "ED25519"
-}
-
-resource "aws_key_pair" "test" {
-  key_name   = "spx-basic-test-key"
-  public_key = tls_private_key.test.public_key_openssh
-}
-
-resource "local_file" "test_pem" {
-  filename        = "${path.module}/spx-basic-test.pem"
-  content         = tls_private_key.test.private_key_openssh
-  file_permission = "0600"
-}
-
-# ---------------------------------------------------------------------------
-# 5. Placement Group
-# ---------------------------------------------------------------------------
-
-resource "aws_placement_group" "test" {
-  name     = "spx-basic-test-spread"
-  strategy = "spread"
-}
-
-# ---------------------------------------------------------------------------
-# 6. EC2 Instance
-# ---------------------------------------------------------------------------
-
-resource "aws_instance" "test" {
-  ami           = data.aws_ami.debian13.id
-  instance_type = "t3.small"
-
-  subnet_id              = aws_subnet.test.id
-  vpc_security_group_ids = [aws_security_group.test.id]
-  key_name               = aws_key_pair.test.key_name
-  placement_group        = aws_placement_group.test.name
-
-  associate_public_ip_address = true
-
-  tags = {
-    Name        = "spx-basic-test-vm"
-    Environment = "test"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 7. EBS Volume + Attachment
-# ---------------------------------------------------------------------------
-
-resource "aws_ebs_volume" "data" {
-  availability_zone = data.aws_availability_zones.available.names[0]
-  size              = 10
-  type              = "gp3"
-
-  tags = {
-    Name = "spx-basic-test-data-vol"
-  }
-}
-
-
-# ---------------------------------------------------------------------------
-# 8. EBS Snapshot (of the standalone volume)
-# ---------------------------------------------------------------------------
-
-resource "aws_ebs_snapshot" "data" {
-  volume_id = aws_ebs_volume.data.id
-
-  tags = {
-    Name = "spx-basic-test-snapshot"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# 9. Elastic IP + Association
-#    Requires external IPAM (pool mode) — uncomment when available.
-# ---------------------------------------------------------------------------
-
-# resource "aws_eip" "test" {
-#   domain = "vpc"
-#
-#   tags = {
-#     Name = "spx-basic-test-eip"
-#   }
-# }
-#
-# resource "aws_eip_association" "test" {
-#   allocation_id = aws_eip.test.id
-#   instance_id   = aws_instance.test.id
-# }
-
-# ---------------------------------------------------------------------------
-# 10. Standalone Network Interface (not attached — just tests CRUD)
-# ---------------------------------------------------------------------------
-
-resource "aws_network_interface" "extra" {
-  subnet_id       = aws_subnet.test.id
-  security_groups = [aws_security_group.test.id]
-
-  tags = {
-    Name = "spx-basic-test-eni"
-  }
-}
-
-# ---------------------------------------------------------------------------
-# Outputs
-# ---------------------------------------------------------------------------
-
-output "vpc_id" {
-  value = aws_vpc.test.id
-}
-
-output "instance_id" {
-  value = aws_instance.test.id
-}
-
-output "instance_public_ip" {
-  value = aws_instance.test.public_ip
-}
-
-# output "eip_public_ip" {
-#   value = aws_eip.test.public_ip
-# }
-
-output "volume_id" {
-  value = aws_ebs_volume.data.id
-}
-
-output "snapshot_id" {
-  value = aws_ebs_snapshot.data.id
-}
-
-output "ssh_command" {
-  value = "ssh -i spx-basic-test.pem admin@${aws_instance.test.public_ip}"
-}

scripts/iac/aws/main.tf

@@ -1,217 +0,0 @@
-terraform {
-  required_version = ">= 1.6.0"
-
-  required_providers {
-    aws = {
-      source  = "hashicorp/aws"
-      version = ">= 5.0"
-    }
-    tls = {
-      source  = "hashicorp/tls"
-      version = ">= 4.0"
-    }
-    local = {
-      source  = "hashicorp/local"
-      version = ">= 2.0"
-    }
-  }
-}
-
-variable "region" {
-  type    = string
-  default = "ap-southeast-2" # Sydney, change if you want
-}
-
-# MulgaOS AWS-compatible endpoints
-# Set these environment variables or use terraform.tfvars:
-#   export AWS_ACCESS_KEY_ID=your_access_key
-#   export AWS_SECRET_ACCESS_KEY=your_secret_key
-#   export TF_VAR_mulgaos_endpoint=https://localhost:9999
-variable "mulgaos_endpoint" {
-  type        = string
-  default     = "https://localhost:9999"
-  description = "MulgaOS API endpoint (awsd gateway)"
-}
-
-provider "aws" {
-  region = var.region
-
-  # MulgaOS custom endpoints
-  endpoints {
-    ec2 = var.mulgaos_endpoint
-    s3  = var.mulgaos_endpoint
-    iam = var.mulgaos_endpoint
-    sts = var.mulgaos_endpoint
-  }
-
-  # Skip AWS credential validation (MulgaOS handles auth)
-  skip_credentials_validation = true
-  skip_metadata_api_check     = true
-  skip_requesting_account_id  = true
-
-  # For local development, skip region validation
-  skip_region_validation = true
-}
-
-data "aws_availability_zones" "available" {
-  state = "available"
-}
-
-# Debian 13 AMI
-# For MulgaOS, need to import a local AMI first
-data "aws_ami" "debian13" {
-  most_recent = true
-  owners      = ["000000000000"] # Spinifex system images (on AWS use "136693071363" for Debian official)
-
-  filter {
-    name   = "name"
-    values = ["*debian-13*"]
-  }
-
-  filter {
-    name   = "virtualization-type"
-    values = ["hvm"]
-  }
-
-  filter {
-    name   = "root-device-type"
-    values = ["ebs"]
-  }
-}
-
-# Keypair: spinifex-test
-resource "tls_private_key" "hive_test" {
-  algorithm = "ED25519"
-}
-
-resource "aws_key_pair" "hive_test" {
-  key_name   = "spinifex-test"
-  public_key = tls_private_key.hive_test.public_key_openssh
-}
-
-# Save private key locally for SSH
-resource "local_file" "hive_test_pem" {
-  filename        = "${path.module}/spinifex-test.pem"
-  content         = tls_private_key.hive_test.private_key_openssh
-  file_permission = "0600"
-}
-
-# 1: VPC 10.1.0.0/16
-resource "aws_vpc" "main" {
-  cidr_block           = "10.1.0.0/16"
-  enable_dns_hostnames = true
-  enable_dns_support   = true
-
-  tags = {
-    Name = "spinifex-test-vpc"
-  }
-}
-
-# Internet access for a public subnet
-resource "aws_internet_gateway" "igw" {
-  vpc_id = aws_vpc.main.id
-
-  tags = {
-    Name = "spinifex-test-igw"
-  }
-}
-
-# 2: Subnet 10.1.2.0/24 (public)
-resource "aws_subnet" "public" {
-  vpc_id                  = aws_vpc.main.id
-  cidr_block              = "10.1.2.0/24"
-  availability_zone       = data.aws_availability_zones.available.names[0]
-  map_public_ip_on_launch = true
-
-  tags = {
-    Name = "spinifex-test-public-subnet"
-  }
-}
-
-resource "aws_route_table" "public" {
-  vpc_id = aws_vpc.main.id
-
-  route {
-    cidr_block = "0.0.0.0/0"
-    gateway_id = aws_internet_gateway.igw.id
-  }
-
-  tags = {
-    Name = "spinifex-test-public-rt"
-  }
-}
-
-resource "aws_route_table_association" "public" {
-  subnet_id      = aws_subnet.public.id
-  route_table_id = aws_route_table.public.id
-}
-
-# 3: Security group
-resource "aws_security_group" "web_ssh" {
-  name        = "spinifex-test-sg"
-  description = "Allow inbound 22,80,443; allow all outbound"
-  vpc_id      = aws_vpc.main.id
-
-  ingress {
-    description = "SSH"
-    from_port   = 22
-    to_port     = 22
-    protocol    = "tcp"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  ingress {
-    description = "HTTP"
-    from_port   = 80
-    to_port     = 80
-    protocol    = "tcp"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  ingress {
-    description = "HTTPS"
-    from_port   = 443
-    to_port     = 443
-    protocol    = "tcp"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  egress {
-    description = "All outbound"
-    from_port   = 0
-    to_port     = 0
-    protocol    = "-1"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
-
-  tags = {
-    Name = "spinifex-test-sg"
-  }
-}
-
-resource "aws_instance" "vm" {
-  count         = 3
-  ami           = data.aws_ami.debian13.id
-  instance_type = "t3.small"
-
-  subnet_id              = aws_subnet.public.id
-  vpc_security_group_ids = [aws_security_group.web_ssh.id]
-  key_name               = aws_key_pair.hive_test.key_name
-
-  associate_public_ip_address = true
-
-  tags = {
-    Name = "spinifex-test-vm-${count.index + 1}"
-  }
-}
-
-output "instance_public_ips" {
-  value = aws_instance.vm[*].public_ip
-}
-
-output "ssh_commands" {
-  value = [
-    for ip in aws_instance.vm[*].public_ip :
-    "ssh -i spinifex-test.pem admin@${ip}"
-  ]
-}

scripts/iac/configure-cluster.sh

@@ -1,204 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-# Configure a provisioned Spinifex cluster: clone repo, build, form cluster, start services.
-# Usage: configure-cluster.sh <cluster_name> <state_file>
-
-CLUSTER_NAME="${1:?Usage: configure-cluster.sh <cluster_name> <state_file>}"
-STATE_FILE="${2:?Usage: configure-cluster.sh <cluster_name> <state_file>}"
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-TOFU_DIR="$SCRIPT_DIR/proxmox"
-
-SSH_OPTS="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10"
-
-REGION="ap-southeast-2"
-AZ="ap-southeast-2a"
-
-inventory=$(tofu -chdir="$TOFU_DIR" output -json -state="$STATE_FILE" inventory)
-NODE_COUNT=$(echo "$inventory" | jq -r '.node_count')
-SSH_KEY=$(eval echo "$(echo "$inventory" | jq -r '.ssh_key_path')")
-SSH_USER=$(echo "$inventory" | jq -r '.ssh_user')
-
-# Build arrays of node IPs
-declare -a MGMT_IPS
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    MGMT_IPS+=("$(echo "$inventory" | jq -r ".nodes[$i].management")")
-done
-
-remote() {
-    local ip="$1"
-    shift
-    ssh -i "$SSH_KEY" $SSH_OPTS "$SSH_USER@$ip" "$@"
-}
-
-remote_bg() {
-    local ip="$1"
-    shift
-    ssh -i "$SSH_KEY" $SSH_OPTS "$SSH_USER@$ip" "$@" &
-}
-
-echo "==> Cluster '$CLUSTER_NAME': $NODE_COUNT nodes"
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    echo "    node$((i + 1)): ${MGMT_IPS[$i]}"
-done
-
-# --- Step 1: Wait for cloud-init ---
-echo ""
-echo "==> Waiting for cloud-init to complete on all nodes..."
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    ip="${MGMT_IPS[$i]}"
-    printf "    node%d (%s): " "$((i + 1))" "$ip"
-    attempts=0
-    while ! remote "$ip" "test -f /tmp/vendor-cloud-init-done" 2>/dev/null; do
-        attempts=$((attempts + 1))
-        if [ $attempts -ge 60 ]; then
-            echo "TIMEOUT"
-            echo "Error: cloud-init did not complete on $ip after 5 minutes"
-            exit 1
-        fi
-        sleep 5
-    done
-    echo "done"
-done
-
-# --- Step 2: Clone repo and build on all nodes ---
-echo ""
-echo "==> Cloning Spinifex repo and building on all nodes..."
-PIDS=()
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    ip="${MGMT_IPS[$i]}"
-    (
-        echo "    node$((i + 1)) ($ip): cloning..."
-        remote "$ip" "mkdir -p ~/Development/mulga && cd ~/Development/mulga && git clone https://github.com/mulgadc/spinifex.git && export PATH=\$PATH:/usr/local/go/bin && sudo make -C spinifex quickinstall"
-        echo "    node$((i + 1)) ($ip): clone-deps + dev-setup..."
-        remote "$ip" "export PATH=\$PATH:/usr/local/go/bin && cd ~/Development/mulga/spinifex && ./scripts/clone-deps.sh && ./scripts/dev-setup.sh"
-        echo "    node$((i + 1)) ($ip): build complete"
-    ) &
-    PIDS+=($!)
-done
-
-for pid in "${PIDS[@]}"; do
-    wait "$pid"
-done
-echo "    All nodes built successfully"
-
-# --- Step 3: Form cluster ---
-echo ""
-echo "==> Forming cluster..."
-
-# Build comma-separated IP lists
-ALL_MGMT_IPS=$(IFS=,; echo "${MGMT_IPS[*]}")
-NODE1_IP="${MGMT_IPS[0]}"
-
-# Node 1: init
-echo "    Starting init on node1 ($NODE1_IP)..."
-remote_bg "$NODE1_IP" "export PATH=\$PATH:/usr/local/go/bin && cd ~/Development/mulga/spinifex && ./bin/spx admin init \
-    --node node1 \
-    --bind $NODE1_IP \
-    --cluster-bind $NODE1_IP \
-    --cluster-routes $NODE1_IP:4248 \
-    --predastore-nodes $ALL_MGMT_IPS \
-    --port 4432 \
-    --nodes $NODE_COUNT \
-    --region $REGION \
-    --az $AZ \
-    --spinifex-dir ~/spinifex/ \
-    --config-dir ~/spinifex/config/"
-INIT_PID=$!
-
-# Wait for formation server to be ready
-echo "    Waiting for formation server on $NODE1_IP..."
-attempts=0
-while ! curl -sk --connect-timeout 3 "https://$NODE1_IP:4432/formation/health" >/dev/null 2>&1; do
-    attempts=$((attempts + 1))
-    if [ $attempts -ge 60 ]; then
-        echo "Error: Formation server did not start on $NODE1_IP after 5 minutes"
-        kill $INIT_PID 2>/dev/null || true
-        exit 1
-    fi
-    sleep 5
-done
-echo "    Formation server ready"
-
-# Read join token from init node
-JOIN_TOKEN=$(remote "$NODE1_IP" "cat ~/spinifex/config/join-token")
-if [ -z "$JOIN_TOKEN" ]; then
-    echo "Error: Failed to read join token from $NODE1_IP"
-    exit 1
-fi
-echo "    Join token loaded from node1"
-
-# Nodes 2..N: join
-if [ "$NODE_COUNT" -gt 1 ]; then
-    JOIN_PIDS=()
-    for i in $(seq 1 $((NODE_COUNT - 1))); do
-        ip="${MGMT_IPS[$i]}"
-        node_num=$((i + 1))
-        echo "    Starting join on node$node_num ($ip)..."
-        remote_bg "$ip" "export PATH=\$PATH:/usr/local/go/bin && cd ~/Development/mulga/spinifex && ./bin/spx admin join \
-            --node node$node_num \
-            --bind $ip \
-            --cluster-bind $ip \
-            --cluster-routes $NODE1_IP:4248 \
-            --host $NODE1_IP:4432 \
-            --token $JOIN_TOKEN \
-            --data-dir ~/spinifex/ \
-            --config-dir ~/spinifex/config/ \
-            --region $REGION \
-            --az $AZ"
-        JOIN_PIDS+=($!)
-    done
-
-    for pid in "${JOIN_PIDS[@]}"; do
-        wait "$pid"
-    done
-fi
-
-# Wait for init process to complete
-wait $INIT_PID
-echo "    Cluster formation complete"
-
-# --- Step 4: Start services ---
-echo ""
-echo "==> Starting services on all nodes..."
-PIDS=()
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    ip="${MGMT_IPS[$i]}"
-    (
-        echo "    node$((i + 1)) ($ip): starting services..."
-        remote "$ip" "export PATH=\$PATH:/usr/local/go/bin && cd ~/Development/mulga/spinifex && ./scripts/start-dev.sh"
-        echo "    node$((i + 1)) ($ip): services started"
-    ) &
-    PIDS+=($!)
-done
-
-for pid in "${PIDS[@]}"; do
-    wait "$pid"
-done
-
-# --- Step 5: Health check ---
-echo ""
-echo "==> Waiting for all nodes to become healthy..."
-for i in $(seq 0 $((NODE_COUNT - 1))); do
-    ip="${MGMT_IPS[$i]}"
-    printf "    node%d (%s): " "$((i + 1))" "$ip"
-    attempts=0
-    while true; do
-        health=$(curl -sk --connect-timeout 3 "https://$ip:4432/health" 2>/dev/null || echo "")
-        if [ -n "$health" ]; then
-            echo "healthy"
-            break
-        fi
-        attempts=$((attempts + 1))
-        if [ $attempts -ge 30 ]; then
-            echo "TIMEOUT"
-            echo "Warning: node$((i + 1)) did not become healthy after 150 seconds"
-            break
-        fi
-        sleep 5
-    done
-done
-
-echo ""
-echo "==> Cluster '$CLUSTER_NAME' configured successfully"

scripts/iac/proxmox-template/.env.example

@@ -1,60 +0,0 @@
-# Proxmox Template Builder — Environment Configuration
-#
-# Copy to .env and fill in your values:
-#   cp .env.example .env
-#
-# Then source it before running tofu:
-#   source .env && tofu plan
-
-# --- Proxmox API ---
-
-# API token for Proxmox VE (read by the provider directly)
-export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN_HERE"
-
-# Proxmox VE API endpoint URL
-export TF_VAR_proxmox_endpoint="https://pve-host.example.com:8006/"
-
-# --- Proxmox Node ---
-# Build the template on ONE node. After creation, use vzdump to copy to others.
-#
-#   name    - Proxmox node name (as shown in the Proxmox UI)
-#   address - SSH-reachable hostname or IP of the Proxmox host
-
-export TF_VAR_proxmox_node_name="node1"
-export TF_VAR_proxmox_node_address="node1.example.com"
-
-# --- SSH Access ---
-
-# SSH username on Proxmox hosts (used by provider for file uploads + qm commands)
-export TF_VAR_proxmox_ssh_username="terraform"
-
-# Path to SSH private key for Proxmox host access
-export TF_VAR_ssh_private_key_path="~/.ssh/proxmox-host-tf"
-
-# Path to SSH private key for VM access (must be the pair of the public key below)
-export TF_VAR_vm_ssh_private_key_path="~/.ssh/cloud-init"
-
-# Path to SSH public key injected into VMs via cloud-init (for tf-user)
-export TF_VAR_ssh_public_key_path="~/.ssh/cloud-init.pub"
-
-# --- Template ---
-
-# VM ID for the template (pick a high number to avoid conflicts)
-export TF_VAR_template_vmid=9000
-
-# Short tag for the base image (used in Proxmox template tags)
-export TF_VAR_base_image_tag="debian-12"
-
-# Storage and network for the build node
-export TF_VAR_datastore_id="local-lvm"
-export TF_VAR_bridge="vmbr0"
-
-# --- Optional ---
-
-# Base image — change this when testing a newer Debian/Ubuntu version:
-# export TF_VAR_base_image="local:iso/debian-13-genericcloud-amd64.img"
-# export TF_VAR_base_image_tag="debian-13"
-# export TF_VAR_template_vmid=9010
-
-# Clone a specific branch instead of main:
-# export TF_VAR_git_branch="feature/my-branch"

scripts/iac/proxmox-template/.gitignore

@@ -1,6 +0,0 @@
-.env
-.terraform/
-.terraform.lock.hcl
-*.tfstate
-*.tfstate.backup
-*.tfplan

scripts/iac/proxmox-template/README.md

@@ -1,124 +0,0 @@
-# Spinifex Node Template Builder
-
-Builds a Proxmox VM template with all Spinifex dependencies pre-installed. Cloning from this template reduces cluster spin-up from ~15 minutes to ~1-2 minutes (`git pull` + `make build`).
-
-## What's in the template
-
-| Component | Details |
-|---|---|
-| System packages | QEMU, nbdkit, libvirt, gcc, jq, curl, etc. (`make install-system`) |
-| Go | 1.26.4 (`make install-go`) |
-| AWS CLI | v2 (`make install-aws`) |
-| Repositories | spinifex, viperblock, predastore (with `go.work` configured) |
-| Module cache | `go mod download` completed for all repos |
-| Cloud images | Ubuntu 26.04 for nested VMs (`~/images/ubuntu-26.04.img`) |
-| Tuning | sysctl (rmem_max/wmem_max), kvm group, ufw disabled |
-| Generalized | Clean machine-id, SSH host keys, cloud-init — each clone boots fresh |
-
-## Quick Start
-
-```bash
-cd scripts/iac/proxmox-template
-
-# 1. Configure
-cp .env.example .env
-# Edit .env — set your Proxmox endpoint, API token, node, and SSH key paths.
-
-# 2. Set API token (not stored in .env for security)
-export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN"
-
-# 3. Build
-source .env
-tofu init
-tofu plan
-tofu apply
-```
-
-Build takes ~10-15 minutes (apt packages, Go, cloud image download). When done, you'll have a template on the build node at the configured VMID (default 9000).
-
-## Distribute to Other Nodes
-
-After `tofu apply`, the output prints ready-to-run commands. Example for a 3-node cluster:
-
-```bash
-# Export from build node
-ssh -i ~/.ssh/proxmox-host-tf terraform@<BUILD_NODE> \
-  'vzdump 9000 --dumpdir /tmp --mode stop --compress zstd'
-
-# Copy to other nodes
-scp -3 -i ~/.ssh/proxmox-host-tf \
-  terraform@<BUILD_NODE>:/tmp/vzdump-qemu-9000-*.vma.zst \
-  terraform@<TARGET_NODE>:/tmp/
-
-# Restore on each target node (use a unique VMID and the node's datastore)
-ssh -i ~/.ssh/proxmox-host-tf terraform@<TARGET_NODE> \
-  'qmrestore /tmp/vzdump-qemu-9000-*.vma.zst <NEW_VMID> --storage <DATASTORE> && qm template <NEW_VMID>'
-```
-
-After distributing, each node has its own copy of the template with a unique VMID.
-
-## Using the Template
-
-Update `scripts/iac/proxmox/vms.tf` to clone from the template instead of booting from a raw cloud image. Or clone manually from the Proxmox UI.
-
-On each cloned VM after boot:
-
-```bash
-# Pull latest code
-cd ~/Development/mulga/spinifex && git pull
-cd ~/Development/mulga/viperblock && git pull
-cd ~/Development/mulga/predastore && git pull
-
-# Build (~30s with cached modules)
-cd ~/Development/mulga/spinifex && export PATH=/usr/local/go/bin:$PATH && make build
-```
-
-## Testing a Different Base Image
-
-Upload the new image to Proxmox, then build a separate template:
-
-```bash
-# Debian 13 example
-export TF_VAR_base_image="local:iso/debian-13-genericcloud-amd64.img"
-export TF_VAR_base_image_tag="debian-13"
-export TF_VAR_template_vmid=9010   # Keep the debian-12 template at 9000
-
-source .env
-tofu apply
-```
-
-Templates are tagged in the Proxmox UI with the base image name (`debian-12`, `debian-13`, etc.) so you can tell them apart.
-
-## Rebuilding
-
-When dependencies change (new Go version, new system packages):
-
-```bash
-source .env
-tofu destroy     # Remove the old template
-tofu apply       # Build fresh
-# Then re-distribute with vzdump
-```
-
-## SSH Key Setup
-
-This template uses **two separate key pairs** (matching the `scripts/iac/proxmox/` setup):
-
-| Key | Used for | Variable |
-|---|---|---|
-| Proxmox host key | SSH to Proxmox hosts (provider, `qm template` command) | `ssh_private_key_path` |
-| VM / cloud-init key | SSH to VMs (cloud-init injection, provisioning) | `vm_ssh_private_key_path` + `ssh_public_key_path` |
-
-If you use the same key for both, set all three variables to the same key pair.
-
-## Files
-
-| File | Description |
-|------|-------------|
-| `main.tf` | Proxmox provider configuration |
-| `variables.tf` | All configurable variables with defaults |
-| `template-builder.tf` | VM creation, provisioning, template conversion |
-| `outputs.tf` | Template info + vzdump/restore commands |
-| `scripts/provision.sh` | Install deps, clone repos, cache modules, generalize |
-| `.env.example` | Environment variable template |
-| `.gitignore` | Ignores .env, state files, provider plugins |

scripts/iac/proxmox-template/main.tf

@@ -1,27 +0,0 @@
-terraform {
-  required_version = ">= 1.6.0"
-
-  required_providers {
-    proxmox = {
-      source  = "bpg/proxmox"
-      version = "0.52.0"
-    }
-  }
-}
-
-# API token is read from PROXMOX_VE_API_TOKEN env var automatically
-provider "proxmox" {
-  endpoint = var.proxmox_endpoint
-  insecure = true
-
-  ssh {
-    agent       = false
-    username    = var.proxmox_ssh_username
-    private_key = file(pathexpand(var.ssh_private_key_path))
-
-    node {
-      name    = var.proxmox_node_name
-      address = var.proxmox_node_address
-    }
-  }
-}

scripts/iac/proxmox-template/outputs.tf

@@ -1,43 +0,0 @@
-output "template_vmid" {
-  description = "VM ID of the created template"
-  value       = proxmox_virtual_environment_vm.builder.vm_id
-}
-
-output "template_node" {
-  description = "Proxmox node where the template lives"
-  value       = var.proxmox_node_name
-}
-
-output "base_image_tag" {
-  description = "Base image tag for this template"
-  value       = var.base_image_tag
-}
-
-# Ready-to-run command: export the template for distribution to other Proxmox nodes
-output "vzdump_export_command" {
-  description = "Run this to export the template (vzdump) for copying to other nodes"
-  value       = <<-EOT
-    ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@${var.proxmox_node_address} \
-      'vzdump ${var.template_vmid} --dumpdir /tmp --mode stop --compress zstd'
-  EOT
-}
-
-# After exporting, copy to each target node and restore
-output "distribute_commands" {
-  description = "Copy and restore the template on other Proxmox nodes"
-  value       = <<-EOT
-    # 1. Export from build node (${var.proxmox_node_name}):
-    ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@${var.proxmox_node_address} \
-      'vzdump ${var.template_vmid} --dumpdir /tmp --mode stop --compress zstd'
-
-    # 2. Copy to each target node:
-    scp -3 \
-      -i ${var.ssh_private_key_path} \
-      ${var.proxmox_ssh_username}@${var.proxmox_node_address}:/tmp/vzdump-qemu-${var.template_vmid}-*.vma.zst \
-      ${var.proxmox_ssh_username}@<TARGET_NODE_ADDRESS>:/tmp/
-
-    # 3. On each target node — restore and convert to template:
-    ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@<TARGET_NODE_ADDRESS> \
-      'qmrestore /tmp/vzdump-qemu-${var.template_vmid}-*.vma.zst <NEW_VMID> --storage <DATASTORE> && qm template <NEW_VMID>'
-  EOT
-}

scripts/iac/proxmox-template/scripts/provision.sh

@@ -1,176 +0,0 @@
-#!/bin/bash
-# provision.sh — Installs all Spinifex dependencies, clones repos, caches Go modules,
-# and generalizes the VM for use as a Proxmox template.
-#
-# Runs as tf-user via Terraform remote-exec. Uses sudo for system operations.
-# Environment: GIT_BRANCH (default: dev)
-
-set -euo pipefail
-
-GIT_BRANCH="${GIT_BRANCH:-dev}"
-WORK_DIR="$HOME/Development/mulga"
-
-echo "========================================"
-echo "  Spinifex Template Provisioning"
-echo "  Branch: ${GIT_BRANCH}"
-echo "  Arch:   $(uname -m)"
-echo "========================================"
-
-# ── 1. Wait for cloud-init ──────────────────────────────────────────────────
-echo ""
-echo "=== [1/8] Waiting for cloud-init to complete ==="
-cloud-init status --wait
-
-# ── 2. Clone repositories ───────────────────────────────────────────────────
-echo ""
-echo "=== [2/8] Cloning repositories ==="
-mkdir -p "$WORK_DIR"
-cd "$WORK_DIR"
-
-for repo in spinifex viperblock predastore; do
-    if [ ! -d "$repo" ]; then
-        echo "  Cloning $repo (branch: $GIT_BRANCH)..."
-        git clone -b "$GIT_BRANCH" "https://github.com/mulgadc/${repo}.git"
-    else
-        echo "  $repo already exists, skipping"
-    fi
-done
-
-# ── 3. Install system dependencies via Makefile ─────────────────────────────
-echo ""
-echo "=== [3/8] Installing system dependencies ==="
-cd "$WORK_DIR/spinifex"
-sudo make quickinstall
-
-# Ensure Go is in PATH for the rest of the script
-export PATH="/usr/local/go/bin:$HOME/go/bin:$PATH"
-echo "  Go version: $(go version)"
-echo "  AWS CLI:    $(aws --version)"
-
-# ── 4. Set up go.work ───────────────────────────────────────────────────────
-echo ""
-echo "=== [4/8] Creating go.work ==="
-
-# Detect Go version from go.mod
-GO_VERSION=$(grep '^go ' "$WORK_DIR/spinifex/go.mod" | awk '{print $2}')
-echo "  Using Go version from go.mod: ${GO_VERSION}"
-
-cat > "$WORK_DIR/spinifex/go.work" << GOWORK
-go ${GO_VERSION}
-
-use (
-    .
-    ../viperblock
-    ../viperblock/nbd
-    ../viperblock/nbd/libguestfs.org/nbdkit
-    ../predastore
-)
-GOWORK
-
-echo "  Created $WORK_DIR/spinifex/go.work"
-
-# ── 5. Download Go modules ──────────────────────────────────────────────────
-echo ""
-echo "=== [5/8] Downloading Go modules ==="
-cd "$WORK_DIR/spinifex"    && go mod download && echo "  spinifex: done"
-cd "$WORK_DIR/viperblock"  && go mod download && echo "  viperblock: done"
-cd "$WORK_DIR/predastore"  && go mod download && echo "  predastore: done"
-
-# Sub-modules
-if [ -f "$WORK_DIR/viperblock/nbd/go.mod" ]; then
-    cd "$WORK_DIR/viperblock/nbd" && go mod download && echo "  viperblock/nbd: done"
-fi
-if [ -f "$WORK_DIR/viperblock/nbd/libguestfs.org/nbdkit/go.mod" ]; then
-    cd "$WORK_DIR/viperblock/nbd/libguestfs.org/nbdkit" && go mod download && echo "  viperblock/nbd/nbdkit: done"
-fi
-
-# ── 6. Pre-download cloud images for nested VMs ─────────────────────────────
-echo ""
-echo "=== [6/8] Pre-downloading cloud images ==="
-mkdir -p "$HOME/images"
-
-ARCH=$(dpkg --print-architecture)
-IMAGE_URL=""
-if [ "$ARCH" = "amd64" ]; then
-    IMAGE_URL="https://cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-amd64.img"
-elif [ "$ARCH" = "arm64" ]; then
-    IMAGE_URL="https://cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-arm64.img"
-fi
-
-if [ -n "$IMAGE_URL" ]; then
-    echo "  Downloading Ubuntu 26.04 cloud image ($ARCH)..."
-    curl -L -o "$HOME/images/ubuntu-26.04.img" "$IMAGE_URL"
-    echo "  Saved to ~/images/ubuntu-26.04.img"
-else
-    echo "  Unknown arch ($ARCH), skipping cloud image download"
-fi
-
-# ── 7. System tuning ────────────────────────────────────────────────────────
-echo ""
-echo "=== [7/8] Applying system tuning ==="
-
-# Persistent sysctl for NATS / high-throughput networking
-sudo tee /etc/sysctl.d/99-spinifex.conf > /dev/null << 'SYSCTL'
-net.core.rmem_max=4194304
-net.core.wmem_max=4194304
-SYSCTL
-echo "  sysctl: net.core.rmem_max/wmem_max = 4MB"
-
-# Add user to kvm group for QEMU access
-sudo usermod -aG kvm "$(whoami)"
-echo "  Added $(whoami) to kvm group"
-
-# Disable ufw if present
-if command -v ufw >/dev/null 2>&1; then
-    sudo ufw disable 2>/dev/null || true
-    echo "  ufw disabled"
-fi
-
-# ── 8. Generalize for templating ────────────────────────────────────────────
-echo ""
-echo "=== [8/8] Cleaning up for template ==="
-
-# Apt caches
-sudo apt-get clean
-sudo rm -rf /var/lib/apt/lists/*
-echo "  Cleared apt cache"
-
-# Logs
-sudo find /var/log -type f -name "*.log" -exec truncate -s 0 {} \; 2>/dev/null || true
-sudo rm -rf /var/log/journal/*
-echo "  Truncated logs"
-
-# Temp files
-sudo rm -rf /tmp/* /var/tmp/*
-
-# Machine ID — systemd regenerates on next boot.
-# This is critical: without this, all clones get the same DHCP client ID → IP conflicts.
-sudo truncate -s 0 /etc/machine-id
-sudo rm -f /var/lib/dbus/machine-id
-echo "  Cleared machine-id (will regenerate on boot)"
-
-# SSH host keys — regenerated on boot by cloud-init or systemd
-sudo rm -f /etc/ssh/ssh_host_*
-echo "  Removed SSH host keys (will regenerate on boot)"
-
-# Cloud-init state — forces cloud-init to run again on next boot.
-# This ensures each clone gets its own hostname, SSH keys, user config, etc.
-sudo cloud-init clean
-echo "  Cleaned cloud-init state (will re-run on boot)"
-
-# Shell history
-history -c 2>/dev/null || true
-cat /dev/null > "$HOME/.bash_history" 2>/dev/null || true
-
-# Provisioning script itself
-rm -f /tmp/provision.sh
-
-# Sync filesystem
-sync
-
-echo ""
-echo "========================================"
-echo "  Template provisioning complete!"
-echo "  VM is ready to be shut down and"
-echo "  converted to a Proxmox template."
-echo "========================================"

scripts/iac/proxmox-template/template-builder.tf

@@ -1,130 +0,0 @@
-data "local_file" "ssh_public_key" {
-  filename = pathexpand(var.ssh_public_key_path)
-}
-
-# Minimal cloud-init: just enough to bootstrap SSH + guest agent.
-# Heavy provisioning is done via remote-exec after boot.
-resource "proxmox_virtual_environment_file" "cloud_config" {
-  content_type = "snippets"
-  datastore_id = "local"
-  node_name    = var.proxmox_node_name
-
-  source_raw {
-    # NOTE: #cloud-config MUST start at column 0 — cloud-init uses it as a magic header.
-    data = <<EOF
-#cloud-config
-package_update: true
-packages:
-  - qemu-guest-agent
-  - git
-  - make
-  - curl
-  - sudo
-  - ca-certificates
-runcmd:
-  - systemctl enable qemu-guest-agent
-  - systemctl start qemu-guest-agent
-EOF
-
-    file_name = "hive-template-cloud-config.yaml"
-  }
-}
-
-resource "proxmox_virtual_environment_vm" "builder" {
-  name      = "${var.template_name}-builder"
-  node_name = var.proxmox_node_name
-  vm_id     = var.template_vmid
-
-  tags = sort(["terraform", "hive-template", var.base_image_tag])
-
-  initialization {
-    datastore_id        = "local"
-    vendor_data_file_id = proxmox_virtual_environment_file.cloud_config.id
-
-    user_account {
-      username = var.vm_ssh_user
-      keys     = [trimspace(data.local_file.ssh_public_key.content)]
-    }
-
-    ip_config {
-      ipv4 {
-        address = "dhcp"
-      }
-    }
-  }
-
-  cpu {
-    cores = var.cpu_cores
-    type  = "host"
-  }
-
-  memory {
-    dedicated = var.memory_mb
-  }
-
-  agent {
-    enabled = true
-  }
-
-  network_device {
-    bridge = var.bridge
-  }
-
-  disk {
-    file_id      = var.base_image
-    datastore_id = var.datastore_id
-    interface    = "virtio0"
-    iothread     = true
-    discard      = "on"
-    size         = var.disk_size_gb
-  }
-
-  # SSH connection to the VM for provisioning.
-  # Uses the VM key (vm_ssh_private_key_path), NOT the Proxmox host key.
-  # Waits for guest agent to report an IP, then connects.
-  connection {
-    type        = "ssh"
-    user        = var.vm_ssh_user
-    private_key = file(pathexpand(var.vm_ssh_private_key_path))
-    host        = element([for addr in flatten(self.ipv4_addresses) : addr if !startswith(addr, "127.")], 0)
-    timeout     = "10m"
-  }
-
-  # Copy provisioning script to VM
-  provisioner "file" {
-    source      = "${path.module}/scripts/provision.sh"
-    destination = "/tmp/provision.sh"
-  }
-
-  # Run provisioning (installs deps, clones repos, caches modules, cleans up)
-  provisioner "remote-exec" {
-    inline = [
-      "chmod +x /tmp/provision.sh",
-      "GIT_BRANCH='${var.git_branch}' /tmp/provision.sh",
-    ]
-  }
-}
-
-# After provisioning completes, shut down the VM and convert to a Proxmox template.
-# This runs on the Proxmox host via SSH (not on the VM — it's being shut down).
-resource "null_resource" "convert_template" {
-  depends_on = [proxmox_virtual_environment_vm.builder]
-
-  triggers = {
-    builder_id = proxmox_virtual_environment_vm.builder.vm_id
-  }
-
-  provisioner "local-exec" {
-    command = <<-EOT
-      ssh -i '${pathexpand(var.ssh_private_key_path)}' \
-        -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
-        '${var.proxmox_ssh_username}@${var.proxmox_node_address}' \
-        "sudo qm shutdown ${proxmox_virtual_environment_vm.builder.vm_id} --timeout 60 && \
-         echo 'Waiting for VM to stop...' && \
-         while sudo qm status ${proxmox_virtual_environment_vm.builder.vm_id} | grep -q running; do sleep 2; done && \
-         echo 'Converting to template...' && \
-         sudo qm template ${proxmox_virtual_environment_vm.builder.vm_id} && \
-         echo 'Template ${var.template_vmid} created successfully'"
-    EOT
-  }
-}

scripts/iac/proxmox-template/variables.tf

@@ -1,117 +0,0 @@
-# --- Proxmox Connection ---
-
-variable "proxmox_endpoint" {
-  type        = string
-  description = "Proxmox VE API endpoint URL (e.g. https://pve.example.com:8006/)"
-}
-
-variable "proxmox_node_name" {
-  type        = string
-  description = "Proxmox node name to build the template on (as shown in the Proxmox UI)"
-}
-
-variable "proxmox_node_address" {
-  type        = string
-  description = "SSH-reachable hostname or IP of the Proxmox node"
-}
-
-variable "proxmox_ssh_username" {
-  type        = string
-  description = "SSH username for Proxmox host access"
-  default     = "terraform"
-}
-
-# --- SSH Keys ---
-# Proxmox host and VM typically use different key pairs.
-# - ssh_private_key_path  → authenticates to the Proxmox host (provider + qm commands)
-# - vm_ssh_private_key_path → authenticates to the VM for provisioning (remote-exec)
-# - ssh_public_key_path → injected into the VM via cloud-init
-
-variable "ssh_private_key_path" {
-  type        = string
-  description = "Path to SSH private key for Proxmox host access"
-}
-
-variable "vm_ssh_private_key_path" {
-  type        = string
-  description = "Path to SSH private key for VM access (must match ssh_public_key_path)"
-}
-
-variable "ssh_public_key_path" {
-  type        = string
-  description = "Path to SSH public key injected into the VM via cloud-init"
-}
-
-# --- Base Image ---
-
-variable "base_image" {
-  type        = string
-  description = "Proxmox image file ID for the base OS (e.g. local:iso/debian-12-genericcloud-amd64-20240211-1654.img)"
-  default     = "local:iso/debian-12-genericcloud-amd64-20240211-1654.img"
-}
-
-variable "base_image_tag" {
-  type        = string
-  description = "Short tag identifying the base image, used in template tags for tracking"
-  default     = "debian-12"
-}
-
-# --- Template Settings ---
-
-variable "template_name" {
-  type        = string
-  description = "Name for the resulting Proxmox template"
-  default     = "hive-node-template"
-}
-
-variable "template_vmid" {
-  type        = number
-  description = "VM ID for the template (pick a high number to avoid conflicts, e.g. 9000)"
-  default     = 9000
-}
-
-variable "vm_ssh_user" {
-  type        = string
-  description = "Username created in the VM via cloud-init"
-  default     = "tf-user"
-}
-
-# --- VM Resources ---
-
-variable "cpu_cores" {
-  type        = number
-  description = "CPU cores for the builder VM (more = faster builds)"
-  default     = 4
-}
-
-variable "memory_mb" {
-  type        = number
-  description = "Memory in MB for the builder VM"
-  default     = 8192
-}
-
-variable "disk_size_gb" {
-  type        = number
-  description = "Disk size in GB (needs room for OS + Go + modules + cloud images)"
-  default     = 32
-}
-
-variable "datastore_id" {
-  type        = string
-  description = "Proxmox storage for the VM disk"
-  default     = "local-lvm"
-}
-
-variable "bridge" {
-  type        = string
-  description = "Network bridge for the VM"
-  default     = "vmbr0"
-}
-
-# --- Git ---
-
-variable "git_branch" {
-  type        = string
-  description = "Git branch to clone for spinifex, viperblock, and predastore"
-  default     = "dev"
-}

scripts/iac/proxmox/.env.example

@@ -1,68 +0,0 @@
-# Proxmox IaC Environment Configuration
-# Copy this file to .env and fill in your values:
-#   cp .env.example .env
-#
-# Then source it before running tofu:
-#   source .env && tofu plan
-
-# --- Proxmox API ---
-
-# API token for Proxmox VE (read by the provider directly)
-export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN_HERE"
-
-# Proxmox VE API endpoint URL
-export TF_VAR_proxmox_endpoint="https://pve-host.example.com:8006/"
-
-# --- SSH Access ---
-
-# SSH username on Proxmox hosts (used by provider for file uploads)
-export TF_VAR_proxmox_ssh_username="terraform"
-
-# Path to SSH private key for Proxmox host access
-export TF_VAR_ssh_private_key_path="~/.ssh/proxmox-host-tf"
-
-# Path to SSH public key injected into VMs via cloud-init (for tf-user)
-export TF_VAR_ssh_public_key_path="~/.ssh/cloud-init.pub"
-
-# --- Proxmox Nodes ---
-# Physical Proxmox hosts where VMs are placed (round-robin).
-# You can define 1 or more nodes — VMs are distributed across them.
-#
-# Each node object:
-#   name          - Proxmox node name (as shown in the Proxmox UI)
-#   address       - SSH-reachable hostname or IP of the Proxmox host
-#   bridge        - Network bridge for VM interfaces (e.g. vmbr0, vmbr3)
-#   datastore_id  - Proxmox storage ID for VM disks (e.g. local, local-lvm)
-#   template_vmid - (optional) VMID of a pre-built template on this node.
-#                   Build with scripts/iac/proxmox-template/, then distribute
-#                   with vzdump. When set, VMs clone from the template (fast).
-#                   When omitted, VMs boot from os_image (slower).
-
-export TF_VAR_nodes='[
-  {
-    "name": "node1",
-    "address": "node1.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local"
-  },
-  {
-    "name": "node2",
-    "address": "node2.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local"
-  },
-  {
-    "name": "node3",
-    "address": "node3.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local"
-  }
-]'
-
-# Example with templates (VMIDs may differ per node after vzdump restore):
-#
-# export TF_VAR_nodes='[
-#   { "name": "node1", "address": "node1.example.com", "bridge": "vmbr0", "datastore_id": "local", "template_vmid": 9999 },
-#   { "name": "node2", "address": "node2.example.com", "bridge": "vmbr0", "datastore_id": "local-lvm", "template_vmid": 9997 },
-#   { "name": "node3", "address": "node3.example.com", "bridge": "vmbr0", "datastore_id": "ssd", "template_vmid": 9998 }
-# ]'

scripts/iac/proxmox/README.md

@@ -1,191 +0,0 @@
-# Proxmox IaC
-
-OpenTofu/Terraform templates for provisioning Spinifex development clusters on Proxmox VE.
-
-## Prerequisites
-
-- [OpenTofu](https://opentofu.org/) (or Terraform >= 1.6.0)
-- 1 or more Proxmox VE nodes with:
-  - A Terraform API token (`Datacenter > Permissions > API Tokens`)
-  - SSH access for a `terraform` user on each node
-  - Debian 13 cloud image uploaded to `local:iso/` on each node
-- SSH keypair for cloud-init VM access
-- A hardcoded Debian ISO is used, which needs to be pre-installed on each Proxmox host ("debian-13-genericcloud-amd64-20260518-2482.img") https://cloud.debian.org/images/cloud/trixie/20260518-2482/debian-13-genericcloud-amd64-20260518-2482.qcow2
-
-## Quick Start (spinifex-test.sh)
-
-The `spinifex-test.sh` wrapper script manages the full lifecycle: provision, configure, test, and destroy.
-
-```sh
-# Source environment
-source scripts/iac/proxmox/.env
-source ~/prox   # Proxmox API token
-
-# Provision a 3-node cluster
-./scripts/iac/spinifex-test.sh up my-cluster
-
-# Configure: clone repo, build, form cluster, start services
-./scripts/iac/spinifex-test.sh configure my-cluster
-
-# Run smoke tests
-./scripts/iac/spinifex-test.sh test my-cluster
-
-# Check cluster health
-./scripts/iac/spinifex-test.sh status my-cluster
-
-# SSH to a specific node
-./scripts/iac/spinifex-test.sh ssh my-cluster 2
-
-# Destroy when done
-./scripts/iac/spinifex-test.sh down my-cluster
-
-# Or run the full lifecycle (up → configure → test → down)
-./scripts/iac/spinifex-test.sh full my-cluster
-```
-
-### Options
-
-```
---node-count=N       Number of VMs (default: 3)
---memory-mb=N        Memory per VM in MB (default: 16384)
---cpu-cores=N        CPU cores per VM (default: 4)
---disk-size-gb=N     Disk size per VM in GB (default: 32)
-```
-
-Example with custom sizing:
-
-```sh
-./scripts/iac/spinifex-test.sh up my-cluster --node-count=5 --memory-mb=8192 --cpu-cores=2
-```
-
-Per-cluster state is stored in `scripts/iac/proxmox/clusters/<name>/terraform.tfstate`, allowing multiple clusters to coexist.
-
-## Manual Setup
-
-### 1. Configure environment
-
-```sh
-cp .env.example .env
-# Edit .env with your Proxmox endpoint, SSH keys, and node configuration
-```
-
-### 2. Set API token
-
-```sh
-export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN_SECRET"
-```
-
-### 3. Deploy
-
-```sh
-source .env
-tofu init
-tofu plan -var="cluster_name=dev1"
-tofu apply -var="cluster_name=dev1"
-```
-
-### Example `.env` file
-
-```sh
-# --- Proxmox API ---
-
-# API token - set manually via: export PROXMOX_VE_API_TOKEN="terraform@pve!provider=..."
-# Format: <user>@<realm>!<token-name>=<token-secret>
-
-# Proxmox VE API endpoint (the URL you use to access the web UI)
-export TF_VAR_proxmox_endpoint="https://pve1.lab.example.com:8006/"
-
-# --- SSH Access ---
-
-# SSH username on the Proxmox hosts (used by the provider for file uploads via SCP)
-export TF_VAR_proxmox_ssh_username="terraform"
-
-# Path to SSH private key that authenticates as the above user on each Proxmox host
-export TF_VAR_ssh_private_key_path="~/.ssh/proxmox-tf"
-
-# Path to SSH public key injected into VMs via cloud-init (the tf-user account)
-export TF_VAR_ssh_public_key_path="~/.ssh/proxmox-tf-cloudinit.pub"
-
-# --- Proxmox Nodes ---
-#
-# Physical Proxmox hosts where VMs are placed (round-robin).
-# Each node object:
-#   name         - Proxmox node name exactly as shown in the Proxmox UI sidebar
-#   address      - SSH-reachable hostname or IP of the Proxmox host
-#   bridge       - Linux bridge for VM network interfaces (check Proxmox UI > Node > Network)
-#   datastore_id - Proxmox storage pool for VM disks (check Proxmox UI > Node > Storage)
-
-export TF_VAR_nodes='[
-  {
-    "name": "pve1",
-    "address": "pve1.lab.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local-lvm"
-  },
-  {
-    "name": "pve2",
-    "address": "pve2.lab.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local-lvm"
-  },
-  {
-    "name": "pve3",
-    "address": "pve3.lab.example.com",
-    "bridge": "vmbr0",
-    "datastore_id": "local-lvm"
-  }
-]'
-```
-
-### Environment variable reference
-
-| Variable | Required | Description |
-|---|---|---|
-| `PROXMOX_VE_API_TOKEN` | Yes | Proxmox API token (read by provider directly) |
-| `TF_VAR_proxmox_endpoint` | Yes | Proxmox VE API URL (e.g. `https://host:8006/`) |
-| `TF_VAR_proxmox_ssh_username` | No | SSH user on Proxmox hosts (default: `terraform`) |
-| `TF_VAR_ssh_private_key_path` | Yes | SSH private key for Proxmox host access |
-| `TF_VAR_ssh_public_key_path` | Yes | SSH public key injected into VMs via cloud-init |
-| `TF_VAR_nodes` | Yes | JSON array of Proxmox node objects |
-
-Each node object in `TF_VAR_nodes`:
-
-| Field | Example | Description |
-|---|---|---|
-| `name` | `pve1` | Proxmox node name (as shown in UI sidebar) |
-| `address` | `pve1.lab.example.com` | SSH-reachable hostname or IP |
-| `bridge` | `vmbr0` | Network bridge for VM interfaces |
-| `datastore_id` | `local-lvm` | Storage pool for VM disks |
-
-### Cluster variables (passed via `-var` or `TF_VAR_`)
-
-| Variable | Default | Description |
-|---|---|---|
-| `cluster_name` | (required) | Cluster name — used in VM names and tags |
-| `node_count` | `3` | Number of VMs to create (1-10) |
-| `cpu_cores` | `4` | CPU cores per VM |
-| `memory_mb` | `16384` | Memory per VM in MB |
-| `disk_size_gb` | `32` | Disk size per VM in GB |
-| `os_image` | `local:iso/debian-12-...` | Proxmox image for boot disk |
-
-## Destroy
-
-```sh
-source .env
-tofu destroy -var="cluster_name=dev1"
-```
-
-## SSH access
-
-Connect to provisioned VMs using the cloud-init public key:
-
-```sh
-ssh -i ~/.ssh/your-cloud-init-key tf-user@<VM_IP>
-```
-
-## Known issues
-
-- `~/spinifex/config/spinifex.toml` - Does not add node1, node2, node3 from config automatically
-- `~/spinifex/config/predastore/predastore.toml` - Uses previous static local node config, needs to use the IPs for each node in the cluster
-- On multi-node deployments, NATS on the primary can timeout waiting for other nodes to start, causing a race condition where NATS fails and all dependent services fail
-- When adding 3 nodes, `nats.conf` is not updated with the 3rd node. Each node's cluster name is hardcoded to `C1` instead of per-node names

scripts/iac/proxmox/cloud-config.tf

@@ -1,48 +0,0 @@
-data "local_file" "ssh_public_key" {
-  filename = pathexpand(var.ssh_public_key_path)
-}
-
-# When cloning from a template, packages are pre-installed — skip apt update/upgrade.
-locals {
-  cloud_config_template = <<-EOF
-    #cloud-config
-    runcmd:
-      - timedatectl set-timezone America/Los_Angeles
-      - systemctl enable qemu-guest-agent
-      - systemctl start qemu-guest-agent
-      - echo "done" > /tmp/vendor-cloud-init-done
-  EOF
-
-  cloud_config_image = <<-EOF
-    #cloud-config
-    package_update: true
-    package_upgrade: true
-    packages:
-      - qemu-guest-agent
-      - net-tools
-      - git
-      - make
-    runcmd:
-      - timedatectl set-timezone America/Los_Angeles
-      - systemctl enable qemu-guest-agent
-      - systemctl start qemu-guest-agent
-      - ufw disable
-      - sysctl -w net.core.rmem_max=4194304
-      - sysctl -w net.core.wmem_max=4194304
-      - usermod -aG kvm tf-user
-      - echo "done" > /tmp/vendor-cloud-init-done
-  EOF
-}
-
-resource "proxmox_virtual_environment_file" "cloud_config" {
-  count = var.node_count
-
-  content_type = "snippets"
-  datastore_id = "local"
-  node_name    = var.nodes[count.index % length(var.nodes)].name
-
-  source_raw {
-    data      = var.nodes[count.index % length(var.nodes)].template_vmid != null ? local.cloud_config_template : local.cloud_config_image
-    file_name = "${var.cluster_name}-${count.index + 1}-cloud-config.yaml"
-  }
-}

scripts/iac/proxmox/files.tf

@@ -1,17 +0,0 @@
-# Files already exist, skip
-
-#resource "proxmox_virtual_environment_download_file" "debian_cloud_image_neon" {
-#  content_type = "iso"
-#  datastore_id = "local"
-#  node_name    = "neon"
-#  url          = "https://cdimage.debian.org/images/cloud/bookworm/20240211-1654/debian-12-genericcloud-amd64-20240211-1654.qcow2"
-#  file_name    = "debian-12-genericcloud-amd64-20240211-1654.img"
-#}
-
-#resource "proxmox_virtual_environment_download_file" "debian_cloud_image_radon" {
-#  content_type = "iso"
-#  datastore_id = "local"
-#  node_name    = "radon"
-#  url          = "https://cdimage.debian.org/images/cloud/bookworm/20240211-1654/debian-12-genericcloud-amd64-20240211-1654.qcow2"
-#  file_name    = "debian-12-genericcloud-amd64-20240211-1654.img"
-#}

scripts/iac/proxmox/main.tf

@@ -1,28 +0,0 @@
-terraform {
-  required_providers {
-    proxmox = {
-      source  = "bpg/proxmox"
-      version = "0.52.0"
-    }
-  }
-}
-
-# API token is read from PROXMOX_VE_API_TOKEN env var automatically
-provider "proxmox" {
-  endpoint = var.proxmox_endpoint
-  insecure = true
-
-  ssh {
-    agent       = false
-    username    = var.proxmox_ssh_username
-    private_key = file(pathexpand(var.ssh_private_key_path))
-
-    dynamic "node" {
-      for_each = var.nodes
-      content {
-        name    = node.value.name
-        address = node.value.address
-      }
-    }
-  }
-}

scripts/iac/proxmox/outputs.tf

@@ -1,14 +0,0 @@
-output "inventory" {
-  value = {
-    cluster_name = var.cluster_name
-    node_count   = var.node_count
-    nodes = [for i, vm in proxmox_virtual_environment_vm.hive_node : {
-      name       = "${var.cluster_name}-${i + 1}"
-      index      = i + 1
-      management = vm.ipv4_addresses[1][0]
-      data       = vm.ipv4_addresses[2][0]
-    }]
-    ssh_user     = "tf-user"
-    ssh_key_path = trimsuffix(var.ssh_public_key_path, ".pub")
-  }
-}

scripts/iac/proxmox/variables.tf

@@ -1,106 +0,0 @@
-variable "proxmox_endpoint" {
-  type        = string
-  description = "Proxmox VE API endpoint URL (e.g. https://pve.example.com:8006/)"
-
-  validation {
-    condition     = can(regex("^https?://", var.proxmox_endpoint))
-    error_message = "proxmox_endpoint must be a valid URL starting with http:// or https://"
-  }
-}
-
-variable "proxmox_ssh_username" {
-  type        = string
-  description = "SSH username for Proxmox host access"
-  default     = "terraform"
-}
-
-variable "ssh_private_key_path" {
-  type        = string
-  description = "Path to SSH private key for Proxmox host access"
-
-  validation {
-    condition     = length(var.ssh_private_key_path) > 0
-    error_message = "ssh_private_key_path must not be empty."
-  }
-}
-
-variable "ssh_public_key_path" {
-  type        = string
-  description = "Path to SSH public key injected into VMs via cloud-init"
-
-  validation {
-    condition     = length(var.ssh_public_key_path) > 0
-    error_message = "ssh_public_key_path must not be empty."
-  }
-}
-
-variable "nodes" {
-  type = list(object({
-    name          = string
-    address       = string
-    bridge        = string
-    datastore_id  = string
-    template_vmid = optional(number)
-  }))
-  description = "Proxmox hosts for VM placement (VMs are distributed round-robin). Set template_vmid to clone from a pre-built template instead of booting from os_image."
-
-  validation {
-    condition     = length(var.nodes) > 0
-    error_message = "At least one Proxmox node must be defined."
-  }
-
-  validation {
-    condition     = alltrue([for n in var.nodes : length(n.name) > 0 && length(n.address) > 0])
-    error_message = "Each node must have a non-empty name and address."
-  }
-
-  validation {
-    condition     = length(distinct([for n in var.nodes : n.name])) == length(var.nodes)
-    error_message = "Each node must have a unique name."
-  }
-}
-
-variable "cluster_name" {
-  type        = string
-  description = "Name for the Spinifex cluster (used in VM names and tags)"
-
-  validation {
-    condition     = can(regex("^[a-z0-9-]+$", var.cluster_name))
-    error_message = "cluster_name must contain only lowercase letters, numbers, and hyphens."
-  }
-}
-
-variable "node_count" {
-  type        = number
-  description = "Number of Spinifex VMs to create"
-  default     = 3
-
-  validation {
-    condition     = var.node_count >= 1 && var.node_count <= 10
-    error_message = "node_count must be between 1 and 10."
-  }
-}
-
-variable "cpu_cores" {
-  type        = number
-  description = "Number of CPU cores per VM"
-  default     = 4
-}
-
-variable "memory_mb" {
-  type        = number
-  description = "Memory in MB per VM"
-  default     = 16384
-}
-
-variable "disk_size_gb" {
-  type        = number
-  description = "Disk size in GB per VM"
-  default     = 32
-}
-
-variable "os_image" {
-  type        = string
-  description = "Proxmox image file ID for VM boot disk (used when node has no template_vmid)"
-  default     = "local:iso/debian-12-genericcloud-amd64-20240211-1654.img"
-}

scripts/iac/proxmox/vms.tf

@@ -1,74 +0,0 @@
-resource "proxmox_virtual_environment_vm" "hive_node" {
-  count = var.node_count
-
-  name        = "${var.cluster_name}-${count.index + 1}"
-  description = "Managed by Terraform"
-  tags        = ["terraform", "spinifex", var.cluster_name]
-  node_name   = var.nodes[count.index % length(var.nodes)].name
-
-  initialization {
-    datastore_id        = "local"
-    vendor_data_file_id = proxmox_virtual_environment_file.cloud_config[count.index].id
-
-    user_account {
-      username = "tf-user"
-      keys     = [trimspace(data.local_file.ssh_public_key.content)]
-    }
-
-    # Management
-    ip_config {
-      ipv4 {
-        address = "dhcp"
-      }
-    }
-
-    # Data/VM
-    ip_config {
-      ipv4 {
-        address = "dhcp"
-      }
-    }
-  }
-
-  cpu {
-    cores = var.cpu_cores
-    type  = "host"
-  }
-
-  memory {
-    dedicated = var.memory_mb
-  }
-
-  agent {
-    enabled = true
-  }
-
-  network_device {
-    bridge = var.nodes[count.index % length(var.nodes)].bridge
-  }
-
-  network_device {
-    bridge = var.nodes[count.index % length(var.nodes)].bridge
-  }
-
-  # Clone from template if the target node has template_vmid set, otherwise boot from cloud image.
-  dynamic "clone" {
-    for_each = var.nodes[count.index % length(var.nodes)].template_vmid != null ? [1] : []
-    content {
-      vm_id = var.nodes[count.index % length(var.nodes)].template_vmid
-      full  = true
-    }
-  }
-
-  dynamic "disk" {
-    for_each = var.nodes[count.index % length(var.nodes)].template_vmid == null ? [1] : []
-    content {
-      file_id      = var.os_image
-      datastore_id = var.nodes[count.index % length(var.nodes)].datastore_id
-      interface    = "virtio0"
-      iothread     = true
-      discard      = "on"
-      size         = var.disk_size_gb
-    }
-  }
-}

scripts/iac/spinifex-test.sh

@@ -1,223 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-TOFU_DIR="$SCRIPT_DIR/proxmox"
-CLUSTERS_DIR="$TOFU_DIR/clusters"
-
-usage() {
-    cat <<'USAGE'
-Usage: spinifex-test.sh <command> <cluster_name> [options]
-
-Commands:
-  up          Provision VMs with OpenTofu
-  configure   Clone repo, build, form cluster, start services
-  test        Run smoke tests against the cluster
-  down        Destroy VMs and clean up state
-  full        up → configure → test → down (preserves test exit code)
-  status      Show cluster health and node IPs
-  ssh <N>     SSH to node N (1-based)
-
-Options:
-  --node-count=N     Number of VMs (default: 3)
-  --memory-mb=N      Memory per VM in MB (default: 16384)
-  --cpu-cores=N      CPU cores per VM (default: 4)
-  --disk-size-gb=N   Disk size per VM in GB (default: 32)
-
-Examples:
-  ./scripts/iac/spinifex-test.sh up spinifex-test1
-  ./scripts/iac/spinifex-test.sh configure spinifex-test1
-  ./scripts/iac/spinifex-test.sh test spinifex-test1
-  ./scripts/iac/spinifex-test.sh status spinifex-test1
-  ./scripts/iac/spinifex-test.sh ssh spinifex-test1 2
-  ./scripts/iac/spinifex-test.sh down spinifex-test1
-  ./scripts/iac/spinifex-test.sh full spinifex-test1 --node-count=3
-USAGE
-    exit 1
-}
-
-[ $# -lt 2 ] && usage
-
-COMMAND="$1"
-CLUSTER_NAME="$2"
-shift 2
-
-# Parse options
-NODE_COUNT=3
-MEMORY_MB=16384
-CPU_CORES=4
-DISK_SIZE_GB=32
-SSH_NODE=""
-
-for arg in "$@"; do
-    case "$arg" in
-        --node-count=*) NODE_COUNT="${arg#*=}" ;;
-        --memory-mb=*)  MEMORY_MB="${arg#*=}" ;;
-        --cpu-cores=*)  CPU_CORES="${arg#*=}" ;;
-        --disk-size-gb=*) DISK_SIZE_GB="${arg#*=}" ;;
-        [0-9]*)         SSH_NODE="$arg" ;;
-        *) echo "Unknown option: $arg"; usage ;;
-    esac
-done
-
-STATE_DIR="$CLUSTERS_DIR/$CLUSTER_NAME"
-STATE_FILE="$STATE_DIR/terraform.tfstate"
-
-tofu_cmd() {
-    tofu -chdir="$TOFU_DIR" "$@"
-}
-
-get_inventory() {
-    tofu_cmd output -json -state="$STATE_FILE" inventory 2>/dev/null
-}
-
-get_node_ip() {
-    local index=$1
-    get_inventory | jq -r ".nodes[$((index - 1))].management"
-}
-
-get_ssh_key() {
-    get_inventory | jq -r ".ssh_key_path"
-}
-
-get_node_count() {
-    get_inventory | jq -r ".node_count"
-}
-
-cmd_up() {
-    echo "==> Provisioning cluster '$CLUSTER_NAME' ($NODE_COUNT nodes, ${MEMORY_MB}MB RAM, ${CPU_CORES} cores)"
-    mkdir -p "$STATE_DIR"
-
-    tofu_cmd init
-
-    tofu_cmd apply \
-        -state="$STATE_FILE" \
-        -var="cluster_name=$CLUSTER_NAME" \
-        -var="node_count=$NODE_COUNT" \
-        -var="memory_mb=$MEMORY_MB" \
-        -var="cpu_cores=$CPU_CORES" \
-        -var="disk_size_gb=$DISK_SIZE_GB" \
-        -auto-approve
-
-    echo ""
-    echo "==> Cluster '$CLUSTER_NAME' provisioned"
-    echo ""
-    get_inventory | jq '.nodes[] | {name, management, data}'
-}
-
-cmd_configure() {
-    echo "==> Configuring cluster '$CLUSTER_NAME'"
-    if [ ! -f "$STATE_FILE" ]; then
-        echo "Error: No state file found at $STATE_FILE. Run 'up' first."
-        exit 1
-    fi
-
-    "$SCRIPT_DIR/configure-cluster.sh" "$CLUSTER_NAME" "$STATE_FILE"
-}
-
-cmd_test() {
-    echo "==> Testing cluster '$CLUSTER_NAME'"
-    if [ ! -f "$STATE_FILE" ]; then
-        echo "Error: No state file found at $STATE_FILE. Run 'up' first."
-        exit 1
-    fi
-
-    "$SCRIPT_DIR/test-cluster.sh" "$CLUSTER_NAME" "$STATE_FILE"
-}
-
-cmd_down() {
-    echo "==> Destroying cluster '$CLUSTER_NAME'"
-    if [ ! -f "$STATE_FILE" ]; then
-        echo "Error: No state file found at $STATE_FILE."
-        exit 1
-    fi
-
-    tofu_cmd destroy \
-        -state="$STATE_FILE" \
-        -var="cluster_name=$CLUSTER_NAME" \
-        -var="node_count=$NODE_COUNT" \
-        -var="memory_mb=$MEMORY_MB" \
-        -var="cpu_cores=$CPU_CORES" \
-        -var="disk_size_gb=$DISK_SIZE_GB" \
-        -auto-approve
-
-    rm -rf "$STATE_DIR"
-    echo "==> Cluster '$CLUSTER_NAME' destroyed"
-}
-
-cmd_full() {
-    cmd_up
-
-    cmd_configure
-
-    local test_exit=0
-    cmd_test || test_exit=$?
-
-    cmd_down
-
-    exit $test_exit
-}
-
-cmd_status() {
-    if [ ! -f "$STATE_FILE" ]; then
-        echo "Error: No state file found at $STATE_FILE."
-        exit 1
-    fi
-
-    local inventory
-    inventory=$(get_inventory)
-    local count
-    count=$(echo "$inventory" | jq -r '.node_count')
-    local ssh_key
-    ssh_key=$(echo "$inventory" | jq -r '.ssh_key_path')
-
-    echo "==> Cluster '$CLUSTER_NAME' status ($count nodes)"
-    echo ""
-
-    for i in $(seq 1 "$count"); do
-        local ip
-        ip=$(echo "$inventory" | jq -r ".nodes[$((i - 1))].management")
-        local data_ip
-        data_ip=$(echo "$inventory" | jq -r ".nodes[$((i - 1))].data")
-        local name
-        name=$(echo "$inventory" | jq -r ".nodes[$((i - 1))].name")
-
-        printf "  %-20s mgmt=%-16s data=%-16s " "$name" "$ip" "$data_ip"
-
-        local health
-        if health=$(curl -sk --connect-timeout 3 "https://$ip:4432/health" 2>/dev/null); then
-            echo "health=$health"
-        else
-            echo "health=unreachable"
-        fi
-    done
-}
-
-cmd_ssh() {
-    if [ -z "$SSH_NODE" ]; then
-        echo "Usage: spinifex-test.sh ssh <cluster_name> <node_number>"
-        exit 1
-    fi
-
-    local ip
-    ip=$(get_node_ip "$SSH_NODE")
-    local ssh_key
-    ssh_key=$(get_ssh_key)
-
-    echo "==> SSH to $CLUSTER_NAME node $SSH_NODE ($ip)"
-    ssh -i "$(eval echo "$ssh_key")" \
-        -o StrictHostKeyChecking=no \
-        -o UserKnownHostsFile=/dev/null \
-        "tf-user@$ip"
-}
-
-case "$COMMAND" in
-    up)        cmd_up ;;
-    configure) cmd_configure ;;
-    test)      cmd_test ;;
-    down)      cmd_down ;;
-    full)      cmd_full ;;
-    status)    cmd_status ;;
-    ssh)       cmd_ssh ;;
-    *)         echo "Unknown command: $COMMAND"; usage ;;
-esac

scripts/iac/test-cluster.sh

@@ -1,71 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-# Run smoke tests against a configured Spinifex cluster.
-# Usage: test-cluster.sh <cluster_name> <state_file>
-
-CLUSTER_NAME="${1:?Usage: test-cluster.sh <cluster_name> <state_file>}"
-STATE_FILE="${2:?Usage: test-cluster.sh <cluster_name> <state_file>}"
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-TOFU_DIR="$SCRIPT_DIR/proxmox"
-
-inventory=$(tofu -chdir="$TOFU_DIR" output -json -state="$STATE_FILE" inventory)
-NODE1_IP=$(echo "$inventory" | jq -r '.nodes[0].management')
-
-AWS_ENDPOINT="https://$NODE1_IP:9999"
-PASS=0
-FAIL=0
-
-run_test() {
-    local name="$1"
-    shift
-    printf "  %-40s " "$name"
-    if output=$("$@" 2>&1); then
-        echo "PASS"
-        PASS=$((PASS + 1))
-    else
-        echo "FAIL"
-        echo "    $output" | head -5
-        FAIL=$((FAIL + 1))
-    fi
-}
-
-echo "==> Smoke tests for cluster '$CLUSTER_NAME'"
-echo "    Endpoint: $AWS_ENDPOINT"
-echo ""
-
-# Wait for gateway to be reachable
-echo "==> Waiting for AWS gateway..."
-attempts=0
-while ! curl -sk --connect-timeout 3 "$AWS_ENDPOINT" >/dev/null 2>&1; do
-    attempts=$((attempts + 1))
-    if [ $attempts -ge 30 ]; then
-        echo "Error: Gateway not reachable at $AWS_ENDPOINT after 150 seconds"
-        exit 1
-    fi
-    sleep 5
-done
-echo "    Gateway reachable"
-echo ""
-
-echo "==> Running tests..."
-
-run_test "describe-regions" \
-    aws --endpoint-url "$AWS_ENDPOINT" --no-verify-ssl --region ap-southeast-2 \
-    ec2 describe-regions
-
-run_test "describe-instance-types" \
-    aws --endpoint-url "$AWS_ENDPOINT" --no-verify-ssl --region ap-southeast-2 \
-    ec2 describe-instance-types --query 'InstanceTypes[0].InstanceType'
-
-run_test "describe-instances" \
-    aws --endpoint-url "$AWS_ENDPOINT" --no-verify-ssl --region ap-southeast-2 \
-    ec2 describe-instances
-
-echo ""
-echo "==> Results: $PASS passed, $FAIL failed"
-
-if [ $FAIL -gt 0 ]; then
-    exit 1
-fi