Fresh install tweaks

INSTALL.md

@@ -22,7 +22,6 @@ cd ~/Development/mulga/
 git clone https://github.com/mulgadc/hive.git
 ```
 
-
 ### Quick Install
 
 To bootstrap the dependencies of Hive in one simple step (QEMU, Go, AWS CLI)
@@ -253,12 +252,58 @@ Once Hive is successfully installed and bootstrapped with a system AMI and SSH k
 export HIVE_AMI="ami-XXX"
 ```
 
-Launch a new instance, note `hive-key` is the SSH key specified in the previous stage. If using an x86_64 host specify the `--instance-type t3.micro` otherwise if running on ARM64 use `--instance-type t4g.micro`
+### Query instance types available
+
+Depending on the host platform Hive is installed for the CPU capabilities and vendor (e.g AMD, Intel, ARM), different compute instances will be available.
+
+Query available instance types and choose an instance type available on your host with 2 vCPUs:
+
+```bash
+aws ec2 describe-instance-types
+```
+
+```json
+{
+  "InstanceTypes": [
+    {
+      "InstanceType": "m8a.small",
+      "CurrentGeneration": true,
+      "SupportedRootDeviceTypes": [
+        "ebs"
+      ],
+      "SupportedVirtualizationTypes": [
+        "hvm"
+      ],
+      "Hypervisor": "kvm",
+      "ProcessorInfo": {
+        "SupportedArchitectures": [
+          "x86_64"
+        ]
+      },
+      "VCpuInfo": {
+        "DefaultVCpus": 2
+      },
+      "MemoryInfo": {
+        "SizeInMiB": 1024
+      },
+      "BurstablePerformanceSupported": false
+    },
+  ]
+}
+```
+
+Export the instance-type:
+
+```sh
+export HIVE_INSTANCE="m8a.small"
+```
+
+Next, launch a new instance, note `hive-key` is the SSH key specified in the previous stage. Be sure to specify the instance-type that is available from your host as above.
 
 ```bash
 aws ec2 run-instances \
   --image-id $HIVE_AMI \
-  --instance-type t3.micro \
+  --instance-type $HIVE_INSTANCE \
   --key-name hive-key \
   --security-group-ids sg-0123456789abcdef0 \
   --subnet-id subnet-6e7f829e \

TODO.md

@@ -4,8 +4,8 @@
 
 Big rocks:
 
-- Add multi-node support for predastore (S3) using reed solomon encoding
-  - Implement KV store for object lookup, WAL files, offset. Use hash-ring to determine which nodes an object belongs to.
+- [DONE] Add multi-node support for predastore (S3) using reed solomon encoding
+  - [DONE] Implement KV store for object lookup, WAL files, offset. Use hash-ring to determine which nodes an object belongs to.
 - Implement VPC support using Open vSwitch across multiple nodes, core VPC functionality included
   - Add support for NVIDIA Bluefield DPU with Open vSwitch
 - Implement basic IAM using NATS Jetstream as KV store, vs IAM/access-keys in local config/TOML files for beta.
@@ -20,11 +20,29 @@ Big rocks:
 Implementation gaps:
 
 - EC2 - Support extended features for `run-instance`
-  - Volume resize of AMI
-  - Confirm cloud-init will resize volume, configured correctly.
+  - Volume resize of AMI. Note `nbd` does not support live resize events, instance needs to be stopped, resized, and started.
+  - Confirm cloud-init will resize volume on boot and configured correctly.
   - Attach additional volumes
   - Attach to VPC / Security group (required Open vSwitch implementation)
 
+### Issues to investigate
+
+- When the host disk volume is full and a new instance is launched, disk corruption will occur within the instance since the host volume is out of space. Consider a pre-flight check, check available storage on the S3 server, local viperblock for WAL cache availability and provide improved guard rails to prevent a new instance running, if available disk space is nearing to be exceeded.
+
+Host:
+
+```json
+{"time":"2026-01-27T21:30:29.368164556+10:00","level":"ERROR","msg":"WAL sync failed","error":"write /home/hive/hive/predastore/distributed/nodes/node-2/0000000000000737.wal: no space left on device"}
+```
+
+Instance `dmesg`:
+
+```
+[ 1445.573495] EXT4-fs error (device vda1): ext4_journal_check_start:83: comm systemd-journal: Detected aborted journal
+[ 1445.583565] EXT4-fs (vda1): Remounting filesystem read-only
+```
+
+
 ### EC2
 
 - [DONE] - describe-instances