diff --git a/scripts/iac/README.md b/scripts/iac/README.md deleted file mode 100644 index 1aa7ba89b..000000000 --- a/scripts/iac/README.md +++ /dev/null @@ -1,9 +0,0 @@ -# IAC - -Collection of OpenTofu/Terraform templates for creating Spinifex instances. - -| Template | Description | -|---|---| -| [proxmox/](proxmox/) | Provisions a 3-node Spinifex cluster on Proxmox VE bare-metal hypervisors | -| [aws/](aws/) | Provisions infrastructure using the Terraform AWS provider, with MulgaOS AWS-compatible API as the local endpoint | -| [proxmox-template/](proxmox-template/) | Builds a Proxmox VM template with all Spinifex dependencies pre-installed (QEMU, Go, repos, modules) | diff --git a/scripts/iac/aws/README.md b/scripts/iac/aws/README.md deleted file mode 100644 index 0191f371a..000000000 --- a/scripts/iac/aws/README.md +++ /dev/null @@ -1,206 +0,0 @@ -# MulgaOS AWS Infrastructure as Code - -This directory contains OpenTofu/Terraform configuration files for deploying infrastructure on MulgaOS using AWS-compatible APIs. - -## Prerequisites - -1. **OpenTofu or Terraform installed** - ```bash - # Install OpenTofu (recommended) - # See: https://opentofu.org/docs/intro/install/ - - # Or use Terraform - # See: https://developer.hashicorp.com/terraform/install - ``` - -2. **MulgaOS services running** - - `spinifex daemon` - Core daemon with NATS subscriptions - - `awsd` - AWS API gateway (default: http://localhost:8080) - - `predastore` - S3-compatible storage - - `viperblock` - EBS-compatible block storage - -3. **AWS credentials configured** - ```bash - # Option 1: Environment variables - export AWS_ACCESS_KEY_ID=your_access_key - export AWS_SECRET_ACCESS_KEY=your_secret_key - - # Option 2: AWS CLI profile - aws configure --profile spinifex - export AWS_PROFILE=spinifex - ``` - -4. **AMI available** - Before running, ensure you have a Debian 13 AMI imported: - ```bash - # Import an AMI using the spinifex CLI or aws CLI - aws --endpoint-url http://localhost:8080 ec2 describe-images - ``` - -## Configuration - -### Environment Variables - -| Variable | Default | Description | -|----------|---------|-------------| -| `TF_VAR_mulgaos_endpoint` | `http://localhost:8080` | MulgaOS API endpoint | -| `TF_VAR_region` | `ap-southeast-2` | AWS region (for compatibility) | -| `AWS_ACCESS_KEY_ID` | - | MulgaOS access key | -| `AWS_SECRET_ACCESS_KEY` | - | MulgaOS secret key | - -### terraform.tfvars (optional) - -Create a `terraform.tfvars` file for custom settings: - -```hcl -mulgaos_endpoint = "http://192.168.1.100:8080" -region = "us-west-2" -``` - -## Usage - -### Initialize - -```bash -cd iac/aws -tofu init -# or: terraform init -``` - -### Validate - -```bash -tofu validate -# or: terraform validate -``` - -### Plan - -```bash -tofu plan -# or: terraform plan -``` - -### Apply - -```bash -tofu apply -# or: terraform apply -``` - -### Destroy - -```bash -tofu destroy -# or: terraform destroy -``` - -## What Gets Created - -The `main.tf` creates: - -1. **VPC** (`10.1.1.0/16`) - Virtual Private Cloud -2. **Internet Gateway** - For public internet access -3. **Public Subnet** (`10.1.2.0/24`) - With auto-assign public IP -4. **Route Table** - Routes `0.0.0.0/0` through IGW -5. **Security Group** - Allows SSH (22), HTTP (80), HTTPS (443) inbound -6. **Placement Group** - Spread strategy for HA -7. **3 EC2 Instances** - t3.small Debian 13 VMs -8. **SSH Key Pair** - ED25519 key saved to `spinifex-test.pem` - -## SSH Access - -After `tofu apply`, SSH into instances: - -```bash -# Using the generated key -ssh -i spinifex-test.pem admin@ - -# For local development with QEMU port forwarding -# Find the forwarded port from QEMU args (-netdev user,id=net0,hostfwd=tcp:127.0.0.1:PORT-:22) -ssh -p PORT -i spinifex-test.pem admin@127.0.0.1 -``` - -## Troubleshooting - -### "No valid credential sources found" - -Ensure AWS credentials are set: -```bash -export AWS_ACCESS_KEY_ID=your_key -export AWS_SECRET_ACCESS_KEY=your_secret -``` - -### "Error: No AMI found" - -Import a Debian 13 AMI first, or modify `main.tf` to use a different AMI filter. - -### Connection refused - -Ensure MulgaOS services are running: -```bash -# Check if awsd gateway is running -curl http://localhost:8080/health - -# Check daemon status -spinifex daemon status -``` - -### "InvalidAction" errors - -Some AWS operations may not be implemented yet. Check `TODO.md` for the current implementation status. - -## Customization - -### Using a different AMI - -Modify the `data "aws_ami"` block in `main.tf`: - -```hcl -data "aws_ami" "custom" { - most_recent = true - owners = ["self"] # Your imported AMIs - - filter { - name = "name" - values = ["my-custom-ami-*"] - } -} -``` - -### Changing instance count - -Modify the `count` parameter: - -```hcl -resource "aws_instance" "vm" { - count = 5 # Change from 3 to 5 - # ... -} -``` - -### Different instance type - -```hcl -resource "aws_instance" "vm" { - instance_type = "t3.medium" # Change from t3.small - # ... -} -``` - -## Files - -| File | Description | -|------|-------------| -| `main.tf` | Main Terraform/OpenTofu configuration | -| `spinifex-test.pem` | Generated SSH private key (after apply) | -| `.terraform/` | Provider plugins (after init) | -| `terraform.tfstate` | State file (after apply) | -| `terraform.tfvars` | Custom variables (optional) | - -## Related Documentation - -- [MulgaOS DEV.md](../../DEV.md) - Development guide -- [MulgaOS TODO.md](../../TODO.md) - Implementation status -- [OpenTofu Documentation](https://opentofu.org/docs/) -- [AWS Provider Documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs) diff --git a/scripts/iac/aws/basic/main.tf b/scripts/iac/aws/basic/main.tf deleted file mode 100644 index dc62cab64..000000000 --- a/scripts/iac/aws/basic/main.tf +++ /dev/null @@ -1,307 +0,0 @@ -# Spinifex IaC Compatibility Test — "basic" suite -# -# Exercises every AWS resource Spinifex currently supports via tofu apply, -# WITHOUT route tables, NACLs, or other unimplemented networking. -# -# Usage: -# cd spinifex/scripts/iac/aws/basic -# tofu init -# tofu plan -# tofu apply # full round-trip against a running Spinifex cluster -# tofu destroy # clean teardown - -terraform { - required_version = ">= 1.6.0" - - required_providers { - aws = { - source = "hashicorp/aws" - version = ">= 5.0" - } - tls = { - source = "hashicorp/tls" - version = ">= 4.0" - } - local = { - source = "hashicorp/local" - version = ">= 2.0" - } - } -} - -# --------------------------------------------------------------------------- -# Variables -# --------------------------------------------------------------------------- - -variable "region" { - type = string - default = "ap-southeast-2" -} - -variable "mulgaos_endpoint" { - type = string - default = "https://localhost:9999" - description = "Spinifex AWS gateway endpoint" -} - -# --------------------------------------------------------------------------- -# Provider -# --------------------------------------------------------------------------- - -provider "aws" { - region = var.region - - endpoints { - ec2 = var.mulgaos_endpoint - s3 = var.mulgaos_endpoint - iam = var.mulgaos_endpoint - sts = var.mulgaos_endpoint - } - - skip_credentials_validation = true - skip_metadata_api_check = true - skip_requesting_account_id = true - skip_region_validation = true -} - -# --------------------------------------------------------------------------- -# Data sources -# --------------------------------------------------------------------------- - -data "aws_availability_zones" "available" { - state = "available" -} - -data "aws_ami" "debian13" { - most_recent = true - owners = ["000000000000"] - - filter { - name = "name" - values = ["*debian-13*"] - } - - filter { - name = "virtualization-type" - values = ["hvm"] - } - - filter { - name = "root-device-type" - values = ["ebs"] - } -} - -# --------------------------------------------------------------------------- -# 1. VPC + Subnet (no route table needed) -# --------------------------------------------------------------------------- - -resource "aws_vpc" "test" { - cidr_block = "10.99.0.0/16" - enable_dns_hostnames = true - enable_dns_support = true - - tags = { - Name = "spx-basic-test-vpc" - } -} - -resource "aws_subnet" "test" { - vpc_id = aws_vpc.test.id - cidr_block = "10.99.1.0/24" - availability_zone = data.aws_availability_zones.available.names[0] - map_public_ip_on_launch = true - - tags = { - Name = "spx-basic-test-subnet" - } -} - -# --------------------------------------------------------------------------- -# 2. Internet Gateway (attach to VPC, but no route table referencing it) -# --------------------------------------------------------------------------- - -resource "aws_internet_gateway" "test" { - vpc_id = aws_vpc.test.id - - tags = { - Name = "spx-basic-test-igw" - } -} - -# --------------------------------------------------------------------------- -# 3. Security Group — inbound SSH + HTTP, all outbound -# --------------------------------------------------------------------------- - -resource "aws_security_group" "test" { - name = "spx-basic-test-sg" - description = "Basic test: SSH + HTTP inbound, all outbound" - vpc_id = aws_vpc.test.id - - ingress { - description = "SSH" - from_port = 22 - to_port = 22 - protocol = "tcp" - cidr_blocks = ["0.0.0.0/0"] - } - - ingress { - description = "HTTP" - from_port = 80 - to_port = 80 - protocol = "tcp" - cidr_blocks = ["0.0.0.0/0"] - } - - egress { - description = "All outbound" - from_port = 0 - to_port = 0 - protocol = "-1" - cidr_blocks = ["0.0.0.0/0"] - } - - tags = { - Name = "spx-basic-test-sg" - } -} - -# --------------------------------------------------------------------------- -# 4. Key Pair -# --------------------------------------------------------------------------- - -resource "tls_private_key" "test" { - algorithm = "ED25519" -} - -resource "aws_key_pair" "test" { - key_name = "spx-basic-test-key" - public_key = tls_private_key.test.public_key_openssh -} - -resource "local_file" "test_pem" { - filename = "${path.module}/spx-basic-test.pem" - content = tls_private_key.test.private_key_openssh - file_permission = "0600" -} - -# --------------------------------------------------------------------------- -# 5. Placement Group -# --------------------------------------------------------------------------- - -resource "aws_placement_group" "test" { - name = "spx-basic-test-spread" - strategy = "spread" -} - -# --------------------------------------------------------------------------- -# 6. EC2 Instance -# --------------------------------------------------------------------------- - -resource "aws_instance" "test" { - ami = data.aws_ami.debian13.id - instance_type = "t3.small" - - subnet_id = aws_subnet.test.id - vpc_security_group_ids = [aws_security_group.test.id] - key_name = aws_key_pair.test.key_name - placement_group = aws_placement_group.test.name - - associate_public_ip_address = true - - tags = { - Name = "spx-basic-test-vm" - Environment = "test" - } -} - -# --------------------------------------------------------------------------- -# 7. EBS Volume + Attachment -# --------------------------------------------------------------------------- - -resource "aws_ebs_volume" "data" { - availability_zone = data.aws_availability_zones.available.names[0] - size = 10 - type = "gp3" - - tags = { - Name = "spx-basic-test-data-vol" - } -} - - -# --------------------------------------------------------------------------- -# 8. EBS Snapshot (of the standalone volume) -# --------------------------------------------------------------------------- - -resource "aws_ebs_snapshot" "data" { - volume_id = aws_ebs_volume.data.id - - tags = { - Name = "spx-basic-test-snapshot" - } -} - -# --------------------------------------------------------------------------- -# 9. Elastic IP + Association -# Requires external IPAM (pool mode) — uncomment when available. -# --------------------------------------------------------------------------- - -# resource "aws_eip" "test" { -# domain = "vpc" -# -# tags = { -# Name = "spx-basic-test-eip" -# } -# } -# -# resource "aws_eip_association" "test" { -# allocation_id = aws_eip.test.id -# instance_id = aws_instance.test.id -# } - -# --------------------------------------------------------------------------- -# 10. Standalone Network Interface (not attached — just tests CRUD) -# --------------------------------------------------------------------------- - -resource "aws_network_interface" "extra" { - subnet_id = aws_subnet.test.id - security_groups = [aws_security_group.test.id] - - tags = { - Name = "spx-basic-test-eni" - } -} - -# --------------------------------------------------------------------------- -# Outputs -# --------------------------------------------------------------------------- - -output "vpc_id" { - value = aws_vpc.test.id -} - -output "instance_id" { - value = aws_instance.test.id -} - -output "instance_public_ip" { - value = aws_instance.test.public_ip -} - -# output "eip_public_ip" { -# value = aws_eip.test.public_ip -# } - -output "volume_id" { - value = aws_ebs_volume.data.id -} - -output "snapshot_id" { - value = aws_ebs_snapshot.data.id -} - -output "ssh_command" { - value = "ssh -i spx-basic-test.pem admin@${aws_instance.test.public_ip}" -} diff --git a/scripts/iac/aws/main.tf b/scripts/iac/aws/main.tf deleted file mode 100644 index 886e8c6ec..000000000 --- a/scripts/iac/aws/main.tf +++ /dev/null @@ -1,217 +0,0 @@ -terraform { - required_version = ">= 1.6.0" - - required_providers { - aws = { - source = "hashicorp/aws" - version = ">= 5.0" - } - tls = { - source = "hashicorp/tls" - version = ">= 4.0" - } - local = { - source = "hashicorp/local" - version = ">= 2.0" - } - } -} - -variable "region" { - type = string - default = "ap-southeast-2" # Sydney, change if you want -} - -# MulgaOS AWS-compatible endpoints -# Set these environment variables or use terraform.tfvars: -# export AWS_ACCESS_KEY_ID=your_access_key -# export AWS_SECRET_ACCESS_KEY=your_secret_key -# export TF_VAR_mulgaos_endpoint=https://localhost:9999 -variable "mulgaos_endpoint" { - type = string - default = "https://localhost:9999" - description = "MulgaOS API endpoint (awsd gateway)" -} - -provider "aws" { - region = var.region - - # MulgaOS custom endpoints - endpoints { - ec2 = var.mulgaos_endpoint - s3 = var.mulgaos_endpoint - iam = var.mulgaos_endpoint - sts = var.mulgaos_endpoint - } - - # Skip AWS credential validation (MulgaOS handles auth) - skip_credentials_validation = true - skip_metadata_api_check = true - skip_requesting_account_id = true - - # For local development, skip region validation - skip_region_validation = true -} - -data "aws_availability_zones" "available" { - state = "available" -} - -# Debian 13 AMI -# For MulgaOS, need to import a local AMI first -data "aws_ami" "debian13" { - most_recent = true - owners = ["000000000000"] # Spinifex system images (on AWS use "136693071363" for Debian official) - - filter { - name = "name" - values = ["*debian-13*"] - } - - filter { - name = "virtualization-type" - values = ["hvm"] - } - - filter { - name = "root-device-type" - values = ["ebs"] - } -} - -# Keypair: spinifex-test -resource "tls_private_key" "hive_test" { - algorithm = "ED25519" -} - -resource "aws_key_pair" "hive_test" { - key_name = "spinifex-test" - public_key = tls_private_key.hive_test.public_key_openssh -} - -# Save private key locally for SSH -resource "local_file" "hive_test_pem" { - filename = "${path.module}/spinifex-test.pem" - content = tls_private_key.hive_test.private_key_openssh - file_permission = "0600" -} - -# 1: VPC 10.1.0.0/16 -resource "aws_vpc" "main" { - cidr_block = "10.1.0.0/16" - enable_dns_hostnames = true - enable_dns_support = true - - tags = { - Name = "spinifex-test-vpc" - } -} - -# Internet access for a public subnet -resource "aws_internet_gateway" "igw" { - vpc_id = aws_vpc.main.id - - tags = { - Name = "spinifex-test-igw" - } -} - -# 2: Subnet 10.1.2.0/24 (public) -resource "aws_subnet" "public" { - vpc_id = aws_vpc.main.id - cidr_block = "10.1.2.0/24" - availability_zone = data.aws_availability_zones.available.names[0] - map_public_ip_on_launch = true - - tags = { - Name = "spinifex-test-public-subnet" - } -} - -resource "aws_route_table" "public" { - vpc_id = aws_vpc.main.id - - route { - cidr_block = "0.0.0.0/0" - gateway_id = aws_internet_gateway.igw.id - } - - tags = { - Name = "spinifex-test-public-rt" - } -} - -resource "aws_route_table_association" "public" { - subnet_id = aws_subnet.public.id - route_table_id = aws_route_table.public.id -} - -# 3: Security group -resource "aws_security_group" "web_ssh" { - name = "spinifex-test-sg" - description = "Allow inbound 22,80,443; allow all outbound" - vpc_id = aws_vpc.main.id - - ingress { - description = "SSH" - from_port = 22 - to_port = 22 - protocol = "tcp" - cidr_blocks = ["0.0.0.0/0"] - } - - ingress { - description = "HTTP" - from_port = 80 - to_port = 80 - protocol = "tcp" - cidr_blocks = ["0.0.0.0/0"] - } - - ingress { - description = "HTTPS" - from_port = 443 - to_port = 443 - protocol = "tcp" - cidr_blocks = ["0.0.0.0/0"] - } - - egress { - description = "All outbound" - from_port = 0 - to_port = 0 - protocol = "-1" - cidr_blocks = ["0.0.0.0/0"] - } - - tags = { - Name = "spinifex-test-sg" - } -} - -resource "aws_instance" "vm" { - count = 3 - ami = data.aws_ami.debian13.id - instance_type = "t3.small" - - subnet_id = aws_subnet.public.id - vpc_security_group_ids = [aws_security_group.web_ssh.id] - key_name = aws_key_pair.hive_test.key_name - - associate_public_ip_address = true - - tags = { - Name = "spinifex-test-vm-${count.index + 1}" - } -} - -output "instance_public_ips" { - value = aws_instance.vm[*].public_ip -} - -output "ssh_commands" { - value = [ - for ip in aws_instance.vm[*].public_ip : - "ssh -i spinifex-test.pem admin@${ip}" - ] -} diff --git a/scripts/iac/configure-cluster.sh b/scripts/iac/configure-cluster.sh deleted file mode 100755 index 48ee7d263..000000000 --- a/scripts/iac/configure-cluster.sh +++ /dev/null @@ -1,204 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -# Configure a provisioned Spinifex cluster: clone repo, build, form cluster, start services. -# Usage: configure-cluster.sh - -CLUSTER_NAME="${1:?Usage: configure-cluster.sh }" -STATE_FILE="${2:?Usage: configure-cluster.sh }" - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -TOFU_DIR="$SCRIPT_DIR/proxmox" - -SSH_OPTS="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10" - -REGION="ap-southeast-2" -AZ="ap-southeast-2a" - -inventory=$(tofu -chdir="$TOFU_DIR" output -json -state="$STATE_FILE" inventory) -NODE_COUNT=$(echo "$inventory" | jq -r '.node_count') -SSH_KEY=$(eval echo "$(echo "$inventory" | jq -r '.ssh_key_path')") -SSH_USER=$(echo "$inventory" | jq -r '.ssh_user') - -# Build arrays of node IPs -declare -a MGMT_IPS -for i in $(seq 0 $((NODE_COUNT - 1))); do - MGMT_IPS+=("$(echo "$inventory" | jq -r ".nodes[$i].management")") -done - -remote() { - local ip="$1" - shift - ssh -i "$SSH_KEY" $SSH_OPTS "$SSH_USER@$ip" "$@" -} - -remote_bg() { - local ip="$1" - shift - ssh -i "$SSH_KEY" $SSH_OPTS "$SSH_USER@$ip" "$@" & -} - -echo "==> Cluster '$CLUSTER_NAME': $NODE_COUNT nodes" -for i in $(seq 0 $((NODE_COUNT - 1))); do - echo " node$((i + 1)): ${MGMT_IPS[$i]}" -done - -# --- Step 1: Wait for cloud-init --- -echo "" -echo "==> Waiting for cloud-init to complete on all nodes..." -for i in $(seq 0 $((NODE_COUNT - 1))); do - ip="${MGMT_IPS[$i]}" - printf " node%d (%s): " "$((i + 1))" "$ip" - attempts=0 - while ! remote "$ip" "test -f /tmp/vendor-cloud-init-done" 2>/dev/null; do - attempts=$((attempts + 1)) - if [ $attempts -ge 60 ]; then - echo "TIMEOUT" - echo "Error: cloud-init did not complete on $ip after 5 minutes" - exit 1 - fi - sleep 5 - done - echo "done" -done - -# --- Step 2: Clone repo and build on all nodes --- -echo "" -echo "==> Cloning Spinifex repo and building on all nodes..." -PIDS=() -for i in $(seq 0 $((NODE_COUNT - 1))); do - ip="${MGMT_IPS[$i]}" - ( - echo " node$((i + 1)) ($ip): cloning..." - remote "$ip" "mkdir -p ~/Development/mulga && cd ~/Development/mulga && git clone https://github.com/mulgadc/spinifex.git && export PATH=\$PATH:/usr/local/go/bin && sudo make -C spinifex quickinstall" - echo " node$((i + 1)) ($ip): clone-deps + dev-setup..." - remote "$ip" "export PATH=\$PATH:/usr/local/go/bin && cd ~/Development/mulga/spinifex && ./scripts/clone-deps.sh && ./scripts/dev-setup.sh" - echo " node$((i + 1)) ($ip): build complete" - ) & - PIDS+=($!) -done - -for pid in "${PIDS[@]}"; do - wait "$pid" -done -echo " All nodes built successfully" - -# --- Step 3: Form cluster --- -echo "" -echo "==> Forming cluster..." - -# Build comma-separated IP lists -ALL_MGMT_IPS=$(IFS=,; echo "${MGMT_IPS[*]}") -NODE1_IP="${MGMT_IPS[0]}" - -# Node 1: init -echo " Starting init on node1 ($NODE1_IP)..." -remote_bg "$NODE1_IP" "export PATH=\$PATH:/usr/local/go/bin && cd ~/Development/mulga/spinifex && ./bin/spx admin init \ - --node node1 \ - --bind $NODE1_IP \ - --cluster-bind $NODE1_IP \ - --cluster-routes $NODE1_IP:4248 \ - --predastore-nodes $ALL_MGMT_IPS \ - --port 4432 \ - --nodes $NODE_COUNT \ - --region $REGION \ - --az $AZ \ - --spinifex-dir ~/spinifex/ \ - --config-dir ~/spinifex/config/" -INIT_PID=$! - -# Wait for formation server to be ready -echo " Waiting for formation server on $NODE1_IP..." -attempts=0 -while ! curl -sk --connect-timeout 3 "https://$NODE1_IP:4432/formation/health" >/dev/null 2>&1; do - attempts=$((attempts + 1)) - if [ $attempts -ge 60 ]; then - echo "Error: Formation server did not start on $NODE1_IP after 5 minutes" - kill $INIT_PID 2>/dev/null || true - exit 1 - fi - sleep 5 -done -echo " Formation server ready" - -# Read join token from init node -JOIN_TOKEN=$(remote "$NODE1_IP" "cat ~/spinifex/config/join-token") -if [ -z "$JOIN_TOKEN" ]; then - echo "Error: Failed to read join token from $NODE1_IP" - exit 1 -fi -echo " Join token loaded from node1" - -# Nodes 2..N: join -if [ "$NODE_COUNT" -gt 1 ]; then - JOIN_PIDS=() - for i in $(seq 1 $((NODE_COUNT - 1))); do - ip="${MGMT_IPS[$i]}" - node_num=$((i + 1)) - echo " Starting join on node$node_num ($ip)..." - remote_bg "$ip" "export PATH=\$PATH:/usr/local/go/bin && cd ~/Development/mulga/spinifex && ./bin/spx admin join \ - --node node$node_num \ - --bind $ip \ - --cluster-bind $ip \ - --cluster-routes $NODE1_IP:4248 \ - --host $NODE1_IP:4432 \ - --token $JOIN_TOKEN \ - --data-dir ~/spinifex/ \ - --config-dir ~/spinifex/config/ \ - --region $REGION \ - --az $AZ" - JOIN_PIDS+=($!) - done - - for pid in "${JOIN_PIDS[@]}"; do - wait "$pid" - done -fi - -# Wait for init process to complete -wait $INIT_PID -echo " Cluster formation complete" - -# --- Step 4: Start services --- -echo "" -echo "==> Starting services on all nodes..." -PIDS=() -for i in $(seq 0 $((NODE_COUNT - 1))); do - ip="${MGMT_IPS[$i]}" - ( - echo " node$((i + 1)) ($ip): starting services..." - remote "$ip" "export PATH=\$PATH:/usr/local/go/bin && cd ~/Development/mulga/spinifex && ./scripts/start-dev.sh" - echo " node$((i + 1)) ($ip): services started" - ) & - PIDS+=($!) -done - -for pid in "${PIDS[@]}"; do - wait "$pid" -done - -# --- Step 5: Health check --- -echo "" -echo "==> Waiting for all nodes to become healthy..." -for i in $(seq 0 $((NODE_COUNT - 1))); do - ip="${MGMT_IPS[$i]}" - printf " node%d (%s): " "$((i + 1))" "$ip" - attempts=0 - while true; do - health=$(curl -sk --connect-timeout 3 "https://$ip:4432/health" 2>/dev/null || echo "") - if [ -n "$health" ]; then - echo "healthy" - break - fi - attempts=$((attempts + 1)) - if [ $attempts -ge 30 ]; then - echo "TIMEOUT" - echo "Warning: node$((i + 1)) did not become healthy after 150 seconds" - break - fi - sleep 5 - done -done - -echo "" -echo "==> Cluster '$CLUSTER_NAME' configured successfully" diff --git a/scripts/iac/proxmox-template/.env.example b/scripts/iac/proxmox-template/.env.example deleted file mode 100644 index 0f542cc31..000000000 --- a/scripts/iac/proxmox-template/.env.example +++ /dev/null @@ -1,60 +0,0 @@ -# Proxmox Template Builder — Environment Configuration -# -# Copy to .env and fill in your values: -# cp .env.example .env -# -# Then source it before running tofu: -# source .env && tofu plan - -# --- Proxmox API --- - -# API token for Proxmox VE (read by the provider directly) -export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN_HERE" - -# Proxmox VE API endpoint URL -export TF_VAR_proxmox_endpoint="https://pve-host.example.com:8006/" - -# --- Proxmox Node --- -# Build the template on ONE node. After creation, use vzdump to copy to others. -# -# name - Proxmox node name (as shown in the Proxmox UI) -# address - SSH-reachable hostname or IP of the Proxmox host - -export TF_VAR_proxmox_node_name="node1" -export TF_VAR_proxmox_node_address="node1.example.com" - -# --- SSH Access --- - -# SSH username on Proxmox hosts (used by provider for file uploads + qm commands) -export TF_VAR_proxmox_ssh_username="terraform" - -# Path to SSH private key for Proxmox host access -export TF_VAR_ssh_private_key_path="~/.ssh/proxmox-host-tf" - -# Path to SSH private key for VM access (must be the pair of the public key below) -export TF_VAR_vm_ssh_private_key_path="~/.ssh/cloud-init" - -# Path to SSH public key injected into VMs via cloud-init (for tf-user) -export TF_VAR_ssh_public_key_path="~/.ssh/cloud-init.pub" - -# --- Template --- - -# VM ID for the template (pick a high number to avoid conflicts) -export TF_VAR_template_vmid=9000 - -# Short tag for the base image (used in Proxmox template tags) -export TF_VAR_base_image_tag="debian-12" - -# Storage and network for the build node -export TF_VAR_datastore_id="local-lvm" -export TF_VAR_bridge="vmbr0" - -# --- Optional --- - -# Base image — change this when testing a newer Debian/Ubuntu version: -# export TF_VAR_base_image="local:iso/debian-13-genericcloud-amd64.img" -# export TF_VAR_base_image_tag="debian-13" -# export TF_VAR_template_vmid=9010 - -# Clone a specific branch instead of main: -# export TF_VAR_git_branch="feature/my-branch" diff --git a/scripts/iac/proxmox-template/.gitignore b/scripts/iac/proxmox-template/.gitignore deleted file mode 100644 index 537e90ed3..000000000 --- a/scripts/iac/proxmox-template/.gitignore +++ /dev/null @@ -1,6 +0,0 @@ -.env -.terraform/ -.terraform.lock.hcl -*.tfstate -*.tfstate.backup -*.tfplan diff --git a/scripts/iac/proxmox-template/README.md b/scripts/iac/proxmox-template/README.md deleted file mode 100644 index fa9f0ddf8..000000000 --- a/scripts/iac/proxmox-template/README.md +++ /dev/null @@ -1,124 +0,0 @@ -# Spinifex Node Template Builder - -Builds a Proxmox VM template with all Spinifex dependencies pre-installed. Cloning from this template reduces cluster spin-up from ~15 minutes to ~1-2 minutes (`git pull` + `make build`). - -## What's in the template - -| Component | Details | -|---|---| -| System packages | QEMU, nbdkit, libvirt, gcc, jq, curl, etc. (`make install-system`) | -| Go | 1.26.4 (`make install-go`) | -| AWS CLI | v2 (`make install-aws`) | -| Repositories | spinifex, viperblock, predastore (with `go.work` configured) | -| Module cache | `go mod download` completed for all repos | -| Cloud images | Ubuntu 26.04 for nested VMs (`~/images/ubuntu-26.04.img`) | -| Tuning | sysctl (rmem_max/wmem_max), kvm group, ufw disabled | -| Generalized | Clean machine-id, SSH host keys, cloud-init — each clone boots fresh | - -## Quick Start - -```bash -cd scripts/iac/proxmox-template - -# 1. Configure -cp .env.example .env -# Edit .env — set your Proxmox endpoint, API token, node, and SSH key paths. - -# 2. Set API token (not stored in .env for security) -export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN" - -# 3. Build -source .env -tofu init -tofu plan -tofu apply -``` - -Build takes ~10-15 minutes (apt packages, Go, cloud image download). When done, you'll have a template on the build node at the configured VMID (default 9000). - -## Distribute to Other Nodes - -After `tofu apply`, the output prints ready-to-run commands. Example for a 3-node cluster: - -```bash -# Export from build node -ssh -i ~/.ssh/proxmox-host-tf terraform@ \ - 'vzdump 9000 --dumpdir /tmp --mode stop --compress zstd' - -# Copy to other nodes -scp -3 -i ~/.ssh/proxmox-host-tf \ - terraform@:/tmp/vzdump-qemu-9000-*.vma.zst \ - terraform@:/tmp/ - -# Restore on each target node (use a unique VMID and the node's datastore) -ssh -i ~/.ssh/proxmox-host-tf terraform@ \ - 'qmrestore /tmp/vzdump-qemu-9000-*.vma.zst --storage && qm template ' -``` - -After distributing, each node has its own copy of the template with a unique VMID. - -## Using the Template - -Update `scripts/iac/proxmox/vms.tf` to clone from the template instead of booting from a raw cloud image. Or clone manually from the Proxmox UI. - -On each cloned VM after boot: - -```bash -# Pull latest code -cd ~/Development/mulga/spinifex && git pull -cd ~/Development/mulga/viperblock && git pull -cd ~/Development/mulga/predastore && git pull - -# Build (~30s with cached modules) -cd ~/Development/mulga/spinifex && export PATH=/usr/local/go/bin:$PATH && make build -``` - -## Testing a Different Base Image - -Upload the new image to Proxmox, then build a separate template: - -```bash -# Debian 13 example -export TF_VAR_base_image="local:iso/debian-13-genericcloud-amd64.img" -export TF_VAR_base_image_tag="debian-13" -export TF_VAR_template_vmid=9010 # Keep the debian-12 template at 9000 - -source .env -tofu apply -``` - -Templates are tagged in the Proxmox UI with the base image name (`debian-12`, `debian-13`, etc.) so you can tell them apart. - -## Rebuilding - -When dependencies change (new Go version, new system packages): - -```bash -source .env -tofu destroy # Remove the old template -tofu apply # Build fresh -# Then re-distribute with vzdump -``` - -## SSH Key Setup - -This template uses **two separate key pairs** (matching the `scripts/iac/proxmox/` setup): - -| Key | Used for | Variable | -|---|---|---| -| Proxmox host key | SSH to Proxmox hosts (provider, `qm template` command) | `ssh_private_key_path` | -| VM / cloud-init key | SSH to VMs (cloud-init injection, provisioning) | `vm_ssh_private_key_path` + `ssh_public_key_path` | - -If you use the same key for both, set all three variables to the same key pair. - -## Files - -| File | Description | -|------|-------------| -| `main.tf` | Proxmox provider configuration | -| `variables.tf` | All configurable variables with defaults | -| `template-builder.tf` | VM creation, provisioning, template conversion | -| `outputs.tf` | Template info + vzdump/restore commands | -| `scripts/provision.sh` | Install deps, clone repos, cache modules, generalize | -| `.env.example` | Environment variable template | -| `.gitignore` | Ignores .env, state files, provider plugins | diff --git a/scripts/iac/proxmox-template/main.tf b/scripts/iac/proxmox-template/main.tf deleted file mode 100644 index 461399faa..000000000 --- a/scripts/iac/proxmox-template/main.tf +++ /dev/null @@ -1,27 +0,0 @@ -terraform { - required_version = ">= 1.6.0" - - required_providers { - proxmox = { - source = "bpg/proxmox" - version = "0.52.0" - } - } -} - -# API token is read from PROXMOX_VE_API_TOKEN env var automatically -provider "proxmox" { - endpoint = var.proxmox_endpoint - insecure = true - - ssh { - agent = false - username = var.proxmox_ssh_username - private_key = file(pathexpand(var.ssh_private_key_path)) - - node { - name = var.proxmox_node_name - address = var.proxmox_node_address - } - } -} diff --git a/scripts/iac/proxmox-template/outputs.tf b/scripts/iac/proxmox-template/outputs.tf deleted file mode 100644 index 25d9fc1a5..000000000 --- a/scripts/iac/proxmox-template/outputs.tf +++ /dev/null @@ -1,43 +0,0 @@ -output "template_vmid" { - description = "VM ID of the created template" - value = proxmox_virtual_environment_vm.builder.vm_id -} - -output "template_node" { - description = "Proxmox node where the template lives" - value = var.proxmox_node_name -} - -output "base_image_tag" { - description = "Base image tag for this template" - value = var.base_image_tag -} - -# Ready-to-run command: export the template for distribution to other Proxmox nodes -output "vzdump_export_command" { - description = "Run this to export the template (vzdump) for copying to other nodes" - value = <<-EOT - ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@${var.proxmox_node_address} \ - 'vzdump ${var.template_vmid} --dumpdir /tmp --mode stop --compress zstd' - EOT -} - -# After exporting, copy to each target node and restore -output "distribute_commands" { - description = "Copy and restore the template on other Proxmox nodes" - value = <<-EOT - # 1. Export from build node (${var.proxmox_node_name}): - ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@${var.proxmox_node_address} \ - 'vzdump ${var.template_vmid} --dumpdir /tmp --mode stop --compress zstd' - - # 2. Copy to each target node: - scp -3 \ - -i ${var.ssh_private_key_path} \ - ${var.proxmox_ssh_username}@${var.proxmox_node_address}:/tmp/vzdump-qemu-${var.template_vmid}-*.vma.zst \ - ${var.proxmox_ssh_username}@:/tmp/ - - # 3. On each target node — restore and convert to template: - ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@ \ - 'qmrestore /tmp/vzdump-qemu-${var.template_vmid}-*.vma.zst --storage && qm template ' - EOT -} diff --git a/scripts/iac/proxmox-template/scripts/provision.sh b/scripts/iac/proxmox-template/scripts/provision.sh deleted file mode 100755 index e8924e052..000000000 --- a/scripts/iac/proxmox-template/scripts/provision.sh +++ /dev/null @@ -1,176 +0,0 @@ -#!/bin/bash -# provision.sh — Installs all Spinifex dependencies, clones repos, caches Go modules, -# and generalizes the VM for use as a Proxmox template. -# -# Runs as tf-user via Terraform remote-exec. Uses sudo for system operations. -# Environment: GIT_BRANCH (default: dev) - -set -euo pipefail - -GIT_BRANCH="${GIT_BRANCH:-dev}" -WORK_DIR="$HOME/Development/mulga" - -echo "========================================" -echo " Spinifex Template Provisioning" -echo " Branch: ${GIT_BRANCH}" -echo " Arch: $(uname -m)" -echo "========================================" - -# ── 1. Wait for cloud-init ────────────────────────────────────────────────── -echo "" -echo "=== [1/8] Waiting for cloud-init to complete ===" -cloud-init status --wait - -# ── 2. Clone repositories ─────────────────────────────────────────────────── -echo "" -echo "=== [2/8] Cloning repositories ===" -mkdir -p "$WORK_DIR" -cd "$WORK_DIR" - -for repo in spinifex viperblock predastore; do - if [ ! -d "$repo" ]; then - echo " Cloning $repo (branch: $GIT_BRANCH)..." - git clone -b "$GIT_BRANCH" "https://github.com/mulgadc/${repo}.git" - else - echo " $repo already exists, skipping" - fi -done - -# ── 3. Install system dependencies via Makefile ───────────────────────────── -echo "" -echo "=== [3/8] Installing system dependencies ===" -cd "$WORK_DIR/spinifex" -sudo make quickinstall - -# Ensure Go is in PATH for the rest of the script -export PATH="/usr/local/go/bin:$HOME/go/bin:$PATH" -echo " Go version: $(go version)" -echo " AWS CLI: $(aws --version)" - -# ── 4. Set up go.work ─────────────────────────────────────────────────────── -echo "" -echo "=== [4/8] Creating go.work ===" - -# Detect Go version from go.mod -GO_VERSION=$(grep '^go ' "$WORK_DIR/spinifex/go.mod" | awk '{print $2}') -echo " Using Go version from go.mod: ${GO_VERSION}" - -cat > "$WORK_DIR/spinifex/go.work" << GOWORK -go ${GO_VERSION} - -use ( - . - ../viperblock - ../viperblock/nbd - ../viperblock/nbd/libguestfs.org/nbdkit - ../predastore -) -GOWORK - -echo " Created $WORK_DIR/spinifex/go.work" - -# ── 5. Download Go modules ────────────────────────────────────────────────── -echo "" -echo "=== [5/8] Downloading Go modules ===" -cd "$WORK_DIR/spinifex" && go mod download && echo " spinifex: done" -cd "$WORK_DIR/viperblock" && go mod download && echo " viperblock: done" -cd "$WORK_DIR/predastore" && go mod download && echo " predastore: done" - -# Sub-modules -if [ -f "$WORK_DIR/viperblock/nbd/go.mod" ]; then - cd "$WORK_DIR/viperblock/nbd" && go mod download && echo " viperblock/nbd: done" -fi -if [ -f "$WORK_DIR/viperblock/nbd/libguestfs.org/nbdkit/go.mod" ]; then - cd "$WORK_DIR/viperblock/nbd/libguestfs.org/nbdkit" && go mod download && echo " viperblock/nbd/nbdkit: done" -fi - -# ── 6. Pre-download cloud images for nested VMs ───────────────────────────── -echo "" -echo "=== [6/8] Pre-downloading cloud images ===" -mkdir -p "$HOME/images" - -ARCH=$(dpkg --print-architecture) -IMAGE_URL="" -if [ "$ARCH" = "amd64" ]; then - IMAGE_URL="https://cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-amd64.img" -elif [ "$ARCH" = "arm64" ]; then - IMAGE_URL="https://cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-arm64.img" -fi - -if [ -n "$IMAGE_URL" ]; then - echo " Downloading Ubuntu 26.04 cloud image ($ARCH)..." - curl -L -o "$HOME/images/ubuntu-26.04.img" "$IMAGE_URL" - echo " Saved to ~/images/ubuntu-26.04.img" -else - echo " Unknown arch ($ARCH), skipping cloud image download" -fi - -# ── 7. System tuning ──────────────────────────────────────────────────────── -echo "" -echo "=== [7/8] Applying system tuning ===" - -# Persistent sysctl for NATS / high-throughput networking -sudo tee /etc/sysctl.d/99-spinifex.conf > /dev/null << 'SYSCTL' -net.core.rmem_max=4194304 -net.core.wmem_max=4194304 -SYSCTL -echo " sysctl: net.core.rmem_max/wmem_max = 4MB" - -# Add user to kvm group for QEMU access -sudo usermod -aG kvm "$(whoami)" -echo " Added $(whoami) to kvm group" - -# Disable ufw if present -if command -v ufw >/dev/null 2>&1; then - sudo ufw disable 2>/dev/null || true - echo " ufw disabled" -fi - -# ── 8. Generalize for templating ──────────────────────────────────────────── -echo "" -echo "=== [8/8] Cleaning up for template ===" - -# Apt caches -sudo apt-get clean -sudo rm -rf /var/lib/apt/lists/* -echo " Cleared apt cache" - -# Logs -sudo find /var/log -type f -name "*.log" -exec truncate -s 0 {} \; 2>/dev/null || true -sudo rm -rf /var/log/journal/* -echo " Truncated logs" - -# Temp files -sudo rm -rf /tmp/* /var/tmp/* - -# Machine ID — systemd regenerates on next boot. -# This is critical: without this, all clones get the same DHCP client ID → IP conflicts. -sudo truncate -s 0 /etc/machine-id -sudo rm -f /var/lib/dbus/machine-id -echo " Cleared machine-id (will regenerate on boot)" - -# SSH host keys — regenerated on boot by cloud-init or systemd -sudo rm -f /etc/ssh/ssh_host_* -echo " Removed SSH host keys (will regenerate on boot)" - -# Cloud-init state — forces cloud-init to run again on next boot. -# This ensures each clone gets its own hostname, SSH keys, user config, etc. -sudo cloud-init clean -echo " Cleaned cloud-init state (will re-run on boot)" - -# Shell history -history -c 2>/dev/null || true -cat /dev/null > "$HOME/.bash_history" 2>/dev/null || true - -# Provisioning script itself -rm -f /tmp/provision.sh - -# Sync filesystem -sync - -echo "" -echo "========================================" -echo " Template provisioning complete!" -echo " VM is ready to be shut down and" -echo " converted to a Proxmox template." -echo "========================================" diff --git a/scripts/iac/proxmox-template/template-builder.tf b/scripts/iac/proxmox-template/template-builder.tf deleted file mode 100644 index 294ff124b..000000000 --- a/scripts/iac/proxmox-template/template-builder.tf +++ /dev/null @@ -1,130 +0,0 @@ -data "local_file" "ssh_public_key" { - filename = pathexpand(var.ssh_public_key_path) -} - -# Minimal cloud-init: just enough to bootstrap SSH + guest agent. -# Heavy provisioning is done via remote-exec after boot. -resource "proxmox_virtual_environment_file" "cloud_config" { - content_type = "snippets" - datastore_id = "local" - node_name = var.proxmox_node_name - - source_raw { - # NOTE: #cloud-config MUST start at column 0 — cloud-init uses it as a magic header. - data = <= 1.6.0) -- 1 or more Proxmox VE nodes with: - - A Terraform API token (`Datacenter > Permissions > API Tokens`) - - SSH access for a `terraform` user on each node - - Debian 13 cloud image uploaded to `local:iso/` on each node -- SSH keypair for cloud-init VM access -- A hardcoded Debian ISO is used, which needs to be pre-installed on each Proxmox host ("debian-13-genericcloud-amd64-20260518-2482.img") https://cloud.debian.org/images/cloud/trixie/20260518-2482/debian-13-genericcloud-amd64-20260518-2482.qcow2 - -## Quick Start (spinifex-test.sh) - -The `spinifex-test.sh` wrapper script manages the full lifecycle: provision, configure, test, and destroy. - -```sh -# Source environment -source scripts/iac/proxmox/.env -source ~/prox # Proxmox API token - -# Provision a 3-node cluster -./scripts/iac/spinifex-test.sh up my-cluster - -# Configure: clone repo, build, form cluster, start services -./scripts/iac/spinifex-test.sh configure my-cluster - -# Run smoke tests -./scripts/iac/spinifex-test.sh test my-cluster - -# Check cluster health -./scripts/iac/spinifex-test.sh status my-cluster - -# SSH to a specific node -./scripts/iac/spinifex-test.sh ssh my-cluster 2 - -# Destroy when done -./scripts/iac/spinifex-test.sh down my-cluster - -# Or run the full lifecycle (up → configure → test → down) -./scripts/iac/spinifex-test.sh full my-cluster -``` - -### Options - -``` ---node-count=N Number of VMs (default: 3) ---memory-mb=N Memory per VM in MB (default: 16384) ---cpu-cores=N CPU cores per VM (default: 4) ---disk-size-gb=N Disk size per VM in GB (default: 32) -``` - -Example with custom sizing: - -```sh -./scripts/iac/spinifex-test.sh up my-cluster --node-count=5 --memory-mb=8192 --cpu-cores=2 -``` - -Per-cluster state is stored in `scripts/iac/proxmox/clusters//terraform.tfstate`, allowing multiple clusters to coexist. - -## Manual Setup - -### 1. Configure environment - -```sh -cp .env.example .env -# Edit .env with your Proxmox endpoint, SSH keys, and node configuration -``` - -### 2. Set API token - -```sh -export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN_SECRET" -``` - -### 3. Deploy - -```sh -source .env -tofu init -tofu plan -var="cluster_name=dev1" -tofu apply -var="cluster_name=dev1" -``` - -### Example `.env` file - -```sh -# --- Proxmox API --- - -# API token - set manually via: export PROXMOX_VE_API_TOKEN="terraform@pve!provider=..." -# Format: @!= - -# Proxmox VE API endpoint (the URL you use to access the web UI) -export TF_VAR_proxmox_endpoint="https://pve1.lab.example.com:8006/" - -# --- SSH Access --- - -# SSH username on the Proxmox hosts (used by the provider for file uploads via SCP) -export TF_VAR_proxmox_ssh_username="terraform" - -# Path to SSH private key that authenticates as the above user on each Proxmox host -export TF_VAR_ssh_private_key_path="~/.ssh/proxmox-tf" - -# Path to SSH public key injected into VMs via cloud-init (the tf-user account) -export TF_VAR_ssh_public_key_path="~/.ssh/proxmox-tf-cloudinit.pub" - -# --- Proxmox Nodes --- -# -# Physical Proxmox hosts where VMs are placed (round-robin). -# Each node object: -# name - Proxmox node name exactly as shown in the Proxmox UI sidebar -# address - SSH-reachable hostname or IP of the Proxmox host -# bridge - Linux bridge for VM network interfaces (check Proxmox UI > Node > Network) -# datastore_id - Proxmox storage pool for VM disks (check Proxmox UI > Node > Storage) - -export TF_VAR_nodes='[ - { - "name": "pve1", - "address": "pve1.lab.example.com", - "bridge": "vmbr0", - "datastore_id": "local-lvm" - }, - { - "name": "pve2", - "address": "pve2.lab.example.com", - "bridge": "vmbr0", - "datastore_id": "local-lvm" - }, - { - "name": "pve3", - "address": "pve3.lab.example.com", - "bridge": "vmbr0", - "datastore_id": "local-lvm" - } -]' -``` - -### Environment variable reference - -| Variable | Required | Description | -|---|---|---| -| `PROXMOX_VE_API_TOKEN` | Yes | Proxmox API token (read by provider directly) | -| `TF_VAR_proxmox_endpoint` | Yes | Proxmox VE API URL (e.g. `https://host:8006/`) | -| `TF_VAR_proxmox_ssh_username` | No | SSH user on Proxmox hosts (default: `terraform`) | -| `TF_VAR_ssh_private_key_path` | Yes | SSH private key for Proxmox host access | -| `TF_VAR_ssh_public_key_path` | Yes | SSH public key injected into VMs via cloud-init | -| `TF_VAR_nodes` | Yes | JSON array of Proxmox node objects | - -Each node object in `TF_VAR_nodes`: - -| Field | Example | Description | -|---|---|---| -| `name` | `pve1` | Proxmox node name (as shown in UI sidebar) | -| `address` | `pve1.lab.example.com` | SSH-reachable hostname or IP | -| `bridge` | `vmbr0` | Network bridge for VM interfaces | -| `datastore_id` | `local-lvm` | Storage pool for VM disks | - -### Cluster variables (passed via `-var` or `TF_VAR_`) - -| Variable | Default | Description | -|---|---|---| -| `cluster_name` | (required) | Cluster name — used in VM names and tags | -| `node_count` | `3` | Number of VMs to create (1-10) | -| `cpu_cores` | `4` | CPU cores per VM | -| `memory_mb` | `16384` | Memory per VM in MB | -| `disk_size_gb` | `32` | Disk size per VM in GB | -| `os_image` | `local:iso/debian-12-...` | Proxmox image for boot disk | - -## Destroy - -```sh -source .env -tofu destroy -var="cluster_name=dev1" -``` - -## SSH access - -Connect to provisioned VMs using the cloud-init public key: - -```sh -ssh -i ~/.ssh/your-cloud-init-key tf-user@ -``` - -## Known issues - -- `~/spinifex/config/spinifex.toml` - Does not add node1, node2, node3 from config automatically -- `~/spinifex/config/predastore/predastore.toml` - Uses previous static local node config, needs to use the IPs for each node in the cluster -- On multi-node deployments, NATS on the primary can timeout waiting for other nodes to start, causing a race condition where NATS fails and all dependent services fail -- When adding 3 nodes, `nats.conf` is not updated with the 3rd node. Each node's cluster name is hardcoded to `C1` instead of per-node names diff --git a/scripts/iac/proxmox/cloud-config.tf b/scripts/iac/proxmox/cloud-config.tf deleted file mode 100644 index c8f30db82..000000000 --- a/scripts/iac/proxmox/cloud-config.tf +++ /dev/null @@ -1,48 +0,0 @@ -data "local_file" "ssh_public_key" { - filename = pathexpand(var.ssh_public_key_path) -} - -# When cloning from a template, packages are pre-installed — skip apt update/upgrade. -locals { - cloud_config_template = <<-EOF - #cloud-config - runcmd: - - timedatectl set-timezone America/Los_Angeles - - systemctl enable qemu-guest-agent - - systemctl start qemu-guest-agent - - echo "done" > /tmp/vendor-cloud-init-done - EOF - - cloud_config_image = <<-EOF - #cloud-config - package_update: true - package_upgrade: true - packages: - - qemu-guest-agent - - net-tools - - git - - make - runcmd: - - timedatectl set-timezone America/Los_Angeles - - systemctl enable qemu-guest-agent - - systemctl start qemu-guest-agent - - ufw disable - - sysctl -w net.core.rmem_max=4194304 - - sysctl -w net.core.wmem_max=4194304 - - usermod -aG kvm tf-user - - echo "done" > /tmp/vendor-cloud-init-done - EOF -} - -resource "proxmox_virtual_environment_file" "cloud_config" { - count = var.node_count - - content_type = "snippets" - datastore_id = "local" - node_name = var.nodes[count.index % length(var.nodes)].name - - source_raw { - data = var.nodes[count.index % length(var.nodes)].template_vmid != null ? local.cloud_config_template : local.cloud_config_image - file_name = "${var.cluster_name}-${count.index + 1}-cloud-config.yaml" - } -} diff --git a/scripts/iac/proxmox/files.tf b/scripts/iac/proxmox/files.tf deleted file mode 100644 index 63d34d178..000000000 --- a/scripts/iac/proxmox/files.tf +++ /dev/null @@ -1,17 +0,0 @@ -# Files already exist, skip - -#resource "proxmox_virtual_environment_download_file" "debian_cloud_image_neon" { -# content_type = "iso" -# datastore_id = "local" -# node_name = "neon" -# url = "https://cdimage.debian.org/images/cloud/bookworm/20240211-1654/debian-12-genericcloud-amd64-20240211-1654.qcow2" -# file_name = "debian-12-genericcloud-amd64-20240211-1654.img" -#} - -#resource "proxmox_virtual_environment_download_file" "debian_cloud_image_radon" { -# content_type = "iso" -# datastore_id = "local" -# node_name = "radon" -# url = "https://cdimage.debian.org/images/cloud/bookworm/20240211-1654/debian-12-genericcloud-amd64-20240211-1654.qcow2" -# file_name = "debian-12-genericcloud-amd64-20240211-1654.img" -#} \ No newline at end of file diff --git a/scripts/iac/proxmox/main.tf b/scripts/iac/proxmox/main.tf deleted file mode 100644 index 4a5f81440..000000000 --- a/scripts/iac/proxmox/main.tf +++ /dev/null @@ -1,28 +0,0 @@ -terraform { - required_providers { - proxmox = { - source = "bpg/proxmox" - version = "0.52.0" - } - } -} - -# API token is read from PROXMOX_VE_API_TOKEN env var automatically -provider "proxmox" { - endpoint = var.proxmox_endpoint - insecure = true - - ssh { - agent = false - username = var.proxmox_ssh_username - private_key = file(pathexpand(var.ssh_private_key_path)) - - dynamic "node" { - for_each = var.nodes - content { - name = node.value.name - address = node.value.address - } - } - } -} diff --git a/scripts/iac/proxmox/outputs.tf b/scripts/iac/proxmox/outputs.tf deleted file mode 100644 index 8eebac06f..000000000 --- a/scripts/iac/proxmox/outputs.tf +++ /dev/null @@ -1,14 +0,0 @@ -output "inventory" { - value = { - cluster_name = var.cluster_name - node_count = var.node_count - nodes = [for i, vm in proxmox_virtual_environment_vm.hive_node : { - name = "${var.cluster_name}-${i + 1}" - index = i + 1 - management = vm.ipv4_addresses[1][0] - data = vm.ipv4_addresses[2][0] - }] - ssh_user = "tf-user" - ssh_key_path = trimsuffix(var.ssh_public_key_path, ".pub") - } -} diff --git a/scripts/iac/proxmox/variables.tf b/scripts/iac/proxmox/variables.tf deleted file mode 100644 index 0286e9f9f..000000000 --- a/scripts/iac/proxmox/variables.tf +++ /dev/null @@ -1,106 +0,0 @@ -variable "proxmox_endpoint" { - type = string - description = "Proxmox VE API endpoint URL (e.g. https://pve.example.com:8006/)" - - validation { - condition = can(regex("^https?://", var.proxmox_endpoint)) - error_message = "proxmox_endpoint must be a valid URL starting with http:// or https://" - } -} - -variable "proxmox_ssh_username" { - type = string - description = "SSH username for Proxmox host access" - default = "terraform" -} - -variable "ssh_private_key_path" { - type = string - description = "Path to SSH private key for Proxmox host access" - - validation { - condition = length(var.ssh_private_key_path) > 0 - error_message = "ssh_private_key_path must not be empty." - } -} - -variable "ssh_public_key_path" { - type = string - description = "Path to SSH public key injected into VMs via cloud-init" - - validation { - condition = length(var.ssh_public_key_path) > 0 - error_message = "ssh_public_key_path must not be empty." - } -} - -variable "nodes" { - type = list(object({ - name = string - address = string - bridge = string - datastore_id = string - template_vmid = optional(number) - })) - description = "Proxmox hosts for VM placement (VMs are distributed round-robin). Set template_vmid to clone from a pre-built template instead of booting from os_image." - - validation { - condition = length(var.nodes) > 0 - error_message = "At least one Proxmox node must be defined." - } - - validation { - condition = alltrue([for n in var.nodes : length(n.name) > 0 && length(n.address) > 0]) - error_message = "Each node must have a non-empty name and address." - } - - validation { - condition = length(distinct([for n in var.nodes : n.name])) == length(var.nodes) - error_message = "Each node must have a unique name." - } -} - -variable "cluster_name" { - type = string - description = "Name for the Spinifex cluster (used in VM names and tags)" - - validation { - condition = can(regex("^[a-z0-9-]+$", var.cluster_name)) - error_message = "cluster_name must contain only lowercase letters, numbers, and hyphens." - } -} - -variable "node_count" { - type = number - description = "Number of Spinifex VMs to create" - default = 3 - - validation { - condition = var.node_count >= 1 && var.node_count <= 10 - error_message = "node_count must be between 1 and 10." - } -} - -variable "cpu_cores" { - type = number - description = "Number of CPU cores per VM" - default = 4 -} - -variable "memory_mb" { - type = number - description = "Memory in MB per VM" - default = 16384 -} - -variable "disk_size_gb" { - type = number - description = "Disk size in GB per VM" - default = 32 -} - -variable "os_image" { - type = string - description = "Proxmox image file ID for VM boot disk (used when node has no template_vmid)" - default = "local:iso/debian-12-genericcloud-amd64-20240211-1654.img" -} diff --git a/scripts/iac/proxmox/vms.tf b/scripts/iac/proxmox/vms.tf deleted file mode 100644 index 573ad12c4..000000000 --- a/scripts/iac/proxmox/vms.tf +++ /dev/null @@ -1,74 +0,0 @@ -resource "proxmox_virtual_environment_vm" "hive_node" { - count = var.node_count - - name = "${var.cluster_name}-${count.index + 1}" - description = "Managed by Terraform" - tags = ["terraform", "spinifex", var.cluster_name] - node_name = var.nodes[count.index % length(var.nodes)].name - - initialization { - datastore_id = "local" - vendor_data_file_id = proxmox_virtual_environment_file.cloud_config[count.index].id - - user_account { - username = "tf-user" - keys = [trimspace(data.local_file.ssh_public_key.content)] - } - - # Management - ip_config { - ipv4 { - address = "dhcp" - } - } - - # Data/VM - ip_config { - ipv4 { - address = "dhcp" - } - } - } - - cpu { - cores = var.cpu_cores - type = "host" - } - - memory { - dedicated = var.memory_mb - } - - agent { - enabled = true - } - - network_device { - bridge = var.nodes[count.index % length(var.nodes)].bridge - } - - network_device { - bridge = var.nodes[count.index % length(var.nodes)].bridge - } - - # Clone from template if the target node has template_vmid set, otherwise boot from cloud image. - dynamic "clone" { - for_each = var.nodes[count.index % length(var.nodes)].template_vmid != null ? [1] : [] - content { - vm_id = var.nodes[count.index % length(var.nodes)].template_vmid - full = true - } - } - - dynamic "disk" { - for_each = var.nodes[count.index % length(var.nodes)].template_vmid == null ? [1] : [] - content { - file_id = var.os_image - datastore_id = var.nodes[count.index % length(var.nodes)].datastore_id - interface = "virtio0" - iothread = true - discard = "on" - size = var.disk_size_gb - } - } -} diff --git a/scripts/iac/spinifex-test.sh b/scripts/iac/spinifex-test.sh deleted file mode 100755 index da39a28e3..000000000 --- a/scripts/iac/spinifex-test.sh +++ /dev/null @@ -1,223 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -TOFU_DIR="$SCRIPT_DIR/proxmox" -CLUSTERS_DIR="$TOFU_DIR/clusters" - -usage() { - cat <<'USAGE' -Usage: spinifex-test.sh [options] - -Commands: - up Provision VMs with OpenTofu - configure Clone repo, build, form cluster, start services - test Run smoke tests against the cluster - down Destroy VMs and clean up state - full up → configure → test → down (preserves test exit code) - status Show cluster health and node IPs - ssh SSH to node N (1-based) - -Options: - --node-count=N Number of VMs (default: 3) - --memory-mb=N Memory per VM in MB (default: 16384) - --cpu-cores=N CPU cores per VM (default: 4) - --disk-size-gb=N Disk size per VM in GB (default: 32) - -Examples: - ./scripts/iac/spinifex-test.sh up spinifex-test1 - ./scripts/iac/spinifex-test.sh configure spinifex-test1 - ./scripts/iac/spinifex-test.sh test spinifex-test1 - ./scripts/iac/spinifex-test.sh status spinifex-test1 - ./scripts/iac/spinifex-test.sh ssh spinifex-test1 2 - ./scripts/iac/spinifex-test.sh down spinifex-test1 - ./scripts/iac/spinifex-test.sh full spinifex-test1 --node-count=3 -USAGE - exit 1 -} - -[ $# -lt 2 ] && usage - -COMMAND="$1" -CLUSTER_NAME="$2" -shift 2 - -# Parse options -NODE_COUNT=3 -MEMORY_MB=16384 -CPU_CORES=4 -DISK_SIZE_GB=32 -SSH_NODE="" - -for arg in "$@"; do - case "$arg" in - --node-count=*) NODE_COUNT="${arg#*=}" ;; - --memory-mb=*) MEMORY_MB="${arg#*=}" ;; - --cpu-cores=*) CPU_CORES="${arg#*=}" ;; - --disk-size-gb=*) DISK_SIZE_GB="${arg#*=}" ;; - [0-9]*) SSH_NODE="$arg" ;; - *) echo "Unknown option: $arg"; usage ;; - esac -done - -STATE_DIR="$CLUSTERS_DIR/$CLUSTER_NAME" -STATE_FILE="$STATE_DIR/terraform.tfstate" - -tofu_cmd() { - tofu -chdir="$TOFU_DIR" "$@" -} - -get_inventory() { - tofu_cmd output -json -state="$STATE_FILE" inventory 2>/dev/null -} - -get_node_ip() { - local index=$1 - get_inventory | jq -r ".nodes[$((index - 1))].management" -} - -get_ssh_key() { - get_inventory | jq -r ".ssh_key_path" -} - -get_node_count() { - get_inventory | jq -r ".node_count" -} - -cmd_up() { - echo "==> Provisioning cluster '$CLUSTER_NAME' ($NODE_COUNT nodes, ${MEMORY_MB}MB RAM, ${CPU_CORES} cores)" - mkdir -p "$STATE_DIR" - - tofu_cmd init - - tofu_cmd apply \ - -state="$STATE_FILE" \ - -var="cluster_name=$CLUSTER_NAME" \ - -var="node_count=$NODE_COUNT" \ - -var="memory_mb=$MEMORY_MB" \ - -var="cpu_cores=$CPU_CORES" \ - -var="disk_size_gb=$DISK_SIZE_GB" \ - -auto-approve - - echo "" - echo "==> Cluster '$CLUSTER_NAME' provisioned" - echo "" - get_inventory | jq '.nodes[] | {name, management, data}' -} - -cmd_configure() { - echo "==> Configuring cluster '$CLUSTER_NAME'" - if [ ! -f "$STATE_FILE" ]; then - echo "Error: No state file found at $STATE_FILE. Run 'up' first." - exit 1 - fi - - "$SCRIPT_DIR/configure-cluster.sh" "$CLUSTER_NAME" "$STATE_FILE" -} - -cmd_test() { - echo "==> Testing cluster '$CLUSTER_NAME'" - if [ ! -f "$STATE_FILE" ]; then - echo "Error: No state file found at $STATE_FILE. Run 'up' first." - exit 1 - fi - - "$SCRIPT_DIR/test-cluster.sh" "$CLUSTER_NAME" "$STATE_FILE" -} - -cmd_down() { - echo "==> Destroying cluster '$CLUSTER_NAME'" - if [ ! -f "$STATE_FILE" ]; then - echo "Error: No state file found at $STATE_FILE." - exit 1 - fi - - tofu_cmd destroy \ - -state="$STATE_FILE" \ - -var="cluster_name=$CLUSTER_NAME" \ - -var="node_count=$NODE_COUNT" \ - -var="memory_mb=$MEMORY_MB" \ - -var="cpu_cores=$CPU_CORES" \ - -var="disk_size_gb=$DISK_SIZE_GB" \ - -auto-approve - - rm -rf "$STATE_DIR" - echo "==> Cluster '$CLUSTER_NAME' destroyed" -} - -cmd_full() { - cmd_up - - cmd_configure - - local test_exit=0 - cmd_test || test_exit=$? - - cmd_down - - exit $test_exit -} - -cmd_status() { - if [ ! -f "$STATE_FILE" ]; then - echo "Error: No state file found at $STATE_FILE." - exit 1 - fi - - local inventory - inventory=$(get_inventory) - local count - count=$(echo "$inventory" | jq -r '.node_count') - local ssh_key - ssh_key=$(echo "$inventory" | jq -r '.ssh_key_path') - - echo "==> Cluster '$CLUSTER_NAME' status ($count nodes)" - echo "" - - for i in $(seq 1 "$count"); do - local ip - ip=$(echo "$inventory" | jq -r ".nodes[$((i - 1))].management") - local data_ip - data_ip=$(echo "$inventory" | jq -r ".nodes[$((i - 1))].data") - local name - name=$(echo "$inventory" | jq -r ".nodes[$((i - 1))].name") - - printf " %-20s mgmt=%-16s data=%-16s " "$name" "$ip" "$data_ip" - - local health - if health=$(curl -sk --connect-timeout 3 "https://$ip:4432/health" 2>/dev/null); then - echo "health=$health" - else - echo "health=unreachable" - fi - done -} - -cmd_ssh() { - if [ -z "$SSH_NODE" ]; then - echo "Usage: spinifex-test.sh ssh " - exit 1 - fi - - local ip - ip=$(get_node_ip "$SSH_NODE") - local ssh_key - ssh_key=$(get_ssh_key) - - echo "==> SSH to $CLUSTER_NAME node $SSH_NODE ($ip)" - ssh -i "$(eval echo "$ssh_key")" \ - -o StrictHostKeyChecking=no \ - -o UserKnownHostsFile=/dev/null \ - "tf-user@$ip" -} - -case "$COMMAND" in - up) cmd_up ;; - configure) cmd_configure ;; - test) cmd_test ;; - down) cmd_down ;; - full) cmd_full ;; - status) cmd_status ;; - ssh) cmd_ssh ;; - *) echo "Unknown command: $COMMAND"; usage ;; -esac diff --git a/scripts/iac/test-cluster.sh b/scripts/iac/test-cluster.sh deleted file mode 100755 index a2e00caba..000000000 --- a/scripts/iac/test-cluster.sh +++ /dev/null @@ -1,71 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -# Run smoke tests against a configured Spinifex cluster. -# Usage: test-cluster.sh - -CLUSTER_NAME="${1:?Usage: test-cluster.sh }" -STATE_FILE="${2:?Usage: test-cluster.sh }" - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -TOFU_DIR="$SCRIPT_DIR/proxmox" - -inventory=$(tofu -chdir="$TOFU_DIR" output -json -state="$STATE_FILE" inventory) -NODE1_IP=$(echo "$inventory" | jq -r '.nodes[0].management') - -AWS_ENDPOINT="https://$NODE1_IP:9999" -PASS=0 -FAIL=0 - -run_test() { - local name="$1" - shift - printf " %-40s " "$name" - if output=$("$@" 2>&1); then - echo "PASS" - PASS=$((PASS + 1)) - else - echo "FAIL" - echo " $output" | head -5 - FAIL=$((FAIL + 1)) - fi -} - -echo "==> Smoke tests for cluster '$CLUSTER_NAME'" -echo " Endpoint: $AWS_ENDPOINT" -echo "" - -# Wait for gateway to be reachable -echo "==> Waiting for AWS gateway..." -attempts=0 -while ! curl -sk --connect-timeout 3 "$AWS_ENDPOINT" >/dev/null 2>&1; do - attempts=$((attempts + 1)) - if [ $attempts -ge 30 ]; then - echo "Error: Gateway not reachable at $AWS_ENDPOINT after 150 seconds" - exit 1 - fi - sleep 5 -done -echo " Gateway reachable" -echo "" - -echo "==> Running tests..." - -run_test "describe-regions" \ - aws --endpoint-url "$AWS_ENDPOINT" --no-verify-ssl --region ap-southeast-2 \ - ec2 describe-regions - -run_test "describe-instance-types" \ - aws --endpoint-url "$AWS_ENDPOINT" --no-verify-ssl --region ap-southeast-2 \ - ec2 describe-instance-types --query 'InstanceTypes[0].InstanceType' - -run_test "describe-instances" \ - aws --endpoint-url "$AWS_ENDPOINT" --no-verify-ssl --region ap-southeast-2 \ - ec2 describe-instances - -echo "" -echo "==> Results: $PASS passed, $FAIL failed" - -if [ $FAIL -gt 0 ]; then - exit 1 -fi