diff --git a/INSTALL.md b/INSTALL.md index ff0d4ad8c..627a8e714 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -22,7 +22,6 @@ cd ~/Development/mulga/ git clone https://github.com/mulgadc/hive.git ``` - ### Quick Install To bootstrap the dependencies of Hive in one simple step (QEMU, Go, AWS CLI) @@ -253,12 +252,58 @@ Once Hive is successfully installed and bootstrapped with a system AMI and SSH k export HIVE_AMI="ami-XXX" ``` -Launch a new instance, note `hive-key` is the SSH key specified in the previous stage. If using an x86_64 host specify the `--instance-type t3.micro` otherwise if running on ARM64 use `--instance-type t4g.micro` +### Query instance types available + +Depending on the host platform Hive is installed for the CPU capabilities and vendor (e.g AMD, Intel, ARM), different compute instances will be available. + +Query available instance types and choose an instance type available on your host with 2 vCPUs: + +```bash +aws ec2 describe-instance-types +``` + +```json +{ + "InstanceTypes": [ + { + "InstanceType": "m8a.small", + "CurrentGeneration": true, + "SupportedRootDeviceTypes": [ + "ebs" + ], + "SupportedVirtualizationTypes": [ + "hvm" + ], + "Hypervisor": "kvm", + "ProcessorInfo": { + "SupportedArchitectures": [ + "x86_64" + ] + }, + "VCpuInfo": { + "DefaultVCpus": 2 + }, + "MemoryInfo": { + "SizeInMiB": 1024 + }, + "BurstablePerformanceSupported": false + }, + ] +} +``` + +Export the instance-type: + +```sh +export HIVE_INSTANCE="m8a.small" +``` + +Next, launch a new instance, note `hive-key` is the SSH key specified in the previous stage. Be sure to specify the instance-type that is available from your host as above. ```bash aws ec2 run-instances \ --image-id $HIVE_AMI \ - --instance-type t3.micro \ + --instance-type $HIVE_INSTANCE \ --key-name hive-key \ --security-group-ids sg-0123456789abcdef0 \ --subnet-id subnet-6e7f829e \ diff --git a/TODO.md b/TODO.md index caf02d335..8575ec16b 100644 --- a/TODO.md +++ b/TODO.md @@ -4,8 +4,8 @@ Big rocks: -- Add multi-node support for predastore (S3) using reed solomon encoding - - Implement KV store for object lookup, WAL files, offset. Use hash-ring to determine which nodes an object belongs to. +- [DONE] Add multi-node support for predastore (S3) using reed solomon encoding + - [DONE] Implement KV store for object lookup, WAL files, offset. Use hash-ring to determine which nodes an object belongs to. - Implement VPC support using Open vSwitch across multiple nodes, core VPC functionality included - Add support for NVIDIA Bluefield DPU with Open vSwitch - Implement basic IAM using NATS Jetstream as KV store, vs IAM/access-keys in local config/TOML files for beta. @@ -20,11 +20,29 @@ Big rocks: Implementation gaps: - EC2 - Support extended features for `run-instance` - - Volume resize of AMI - - Confirm cloud-init will resize volume, configured correctly. + - Volume resize of AMI. Note `nbd` does not support live resize events, instance needs to be stopped, resized, and started. + - Confirm cloud-init will resize volume on boot and configured correctly. - Attach additional volumes - Attach to VPC / Security group (required Open vSwitch implementation) +### Issues to investigate + +- When the host disk volume is full and a new instance is launched, disk corruption will occur within the instance since the host volume is out of space. Consider a pre-flight check, check available storage on the S3 server, local viperblock for WAL cache availability and provide improved guard rails to prevent a new instance running, if available disk space is nearing to be exceeded. + +Host: + +```json +{"time":"2026-01-27T21:30:29.368164556+10:00","level":"ERROR","msg":"WAL sync failed","error":"write /home/hive/hive/predastore/distributed/nodes/node-2/0000000000000737.wal: no space left on device"} +``` + +Instance `dmesg`: + +``` +[ 1445.573495] EXT4-fs error (device vda1): ext4_journal_check_start:83: comm systemd-journal: Detected aborted journal +[ 1445.583565] EXT4-fs (vda1): Remounting filesystem read-only +``` + + ### EC2 - [DONE] - describe-instances