spinifex/TODO.md

Hive Development Roadmap

Update Jan 2026

Big rocks:

• [DONE] Add multi-node support for predastore (S3) using reed solomon encoding
  • [DONE] Implement KV store for object lookup, WAL files, offset. Use hash-ring to determine which nodes an object belongs to.
• Implement VPC support using Open vSwitch across multiple nodes, core VPC functionality included
  • Add support for NVIDIA Bluefield DPU with Open vSwitch
• Implement basic IAM using NATS Jetstream as KV store, vs IAM/access-keys in local config/TOML files for beta.
  • [DONE] Move daemon.go instances.json state to Jetstream KV
• Add support using the hive CLI tool to provision a new user with AWS access-keys/IAM.
  • Support multi-tenant operations and isolation
• Add support to include capabilities when adding a new hardware node to MulgaOS (e.g EC2 target, S3, EBS, NATs, etc) - Features can be turned on/off depending on hardware scope.
• [DONE] Add simple Web UI console, using the AWS JS SDK, communicating to local AWS gateway.
  • [DONE] Implement ShadCNblocks for UI framework
  • [DONE] Simple Go webserver, static files, easy build process.

Implementation gaps:

• EC2 - Support extended features for run-instance
  • [DONE] Volume resize of AMI. Note nbd does not support live resize events, instance needs to be stopped, resized, and started.
  • Confirm cloud-init will resize volume on boot and configured correctly.
  • [DONE] Attach additional volumes
  • Attach to VPC / Security group (required Open vSwitch implementation)
• only allow hive to use x% of cpu and memory. dont allow allocating 100% of resources.
• placement group instances, allow spreading instance group between nodes.
• capabilities flag
• more instance types - per cpu gen
• [DONE] multi part uploads on frontend
• [DONE] delete volumes on termination
• fix describe instances
• [DONE] change all log into slog
• nbd does not support resizing disks live. Requires instance to be stopped, boot/root volume, or attached volume will need to be resized, and instance started again. Limitation for Hive v1 using NBD, aiming to resolve with vhost-user-blk to create a virtio-blk device for extended functionality and performance.

EC2 describe-volume-status

Required to track progress of new volumes created, deleted, detached or running.

aws ec2 describe-volume-status --volume-ids vol-1234567890abcdef0

{
  "VolumeStatuses": [
    {
      "VolumeStatus": {
        "Status": "ok",
        "Details": [
          {
            "Status": "passed",
            "Name": "io-enabled"
          },
          {
            "Status": "not-applicable",
            "Name": "io-performance"
          }
        ]
      },
      "AvailabilityZone": "us-east-1a",
      "VolumeId": "vol-1234567890abcdef0",
      "Actions": [],
      "Events": []
    }
  ]
}

Issues to investigate

• When the host disk volume is full and a new instance is launched, disk corruption will occur within the instance since the host volume is out of space. Consider a pre-flight check, check available storage on the S3 server, local viperblock for WAL cache availability and provide improved guard rails to prevent a new instance running, if available disk space is nearing to be exceeded.

Host:

{
  "time": "2026-01-27T21:30:29.368164556+10:00",
  "level": "ERROR",
  "msg": "WAL sync failed",
  "error": "write /home/hive/hive/predastore/distributed/nodes/node-2/0000000000000737.wal: no space left on device"
}

Instance dmesg:

[ 1445.573495] EXT4-fs error (device vda1): ext4_journal_check_start:83: comm systemd-journal: Detected aborted journal
[ 1445.583565] EXT4-fs (vda1): Remounting filesystem read-only

---

AWS Command Implementation Matrix

EC2 - Instance Management

Command	Implemented Flags	Missing Flags	Prerequisites	Basic Logic	Test Cases	Status
run-instances	--image-id, --instance-type, --count (Min/MaxCount), --key-name, --user-data, --block-device-mappings (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination)	--security-group-ids, --subnet-id, --placement, --tag-specifications, --dry-run, --client-token, --disable-api-termination, --ebs-optimized, --iam-instance-profile, --network-interfaces, --private-ip-address, --monitoring, --credit-specification, --cpu-options, --metadata-options, --launch-template, --hibernate-options	describe-images (AMI must exist), create-key-pair (optional), VPC/SG (optional)	Gateway parses AWS query → NATS ec2.runinstances → daemon creates QEMU/KVM VM with viperblock-backed root volume via NBD → cloud-init injects user-data/keys → returns reservation with instance ID	1. Launch with valid AMI and key pair 2. Launch with invalid AMI ID (error) 3. Launch with block device mappings (custom volume size) 4. Launch multiple instances (MinCount/MaxCount) 5. DryRun returns validation-only 6. Invalid instance type returns error	DONE
describe-instances	--instance-ids	--filters, --max-results, --next-token, --dry-run	None	Gateway fans out NATS ec2.DescribeInstances to all nodes (no queue group) → each daemon returns local instances → gateway aggregates and returns combined list	1. Describe all instances (no filter) 2. Describe by instance ID 3. Describe with filters (e.g. instance-state-name) 4. Instance not found returns empty set 5. Multi-node aggregation returns instances from all nodes	DONE
start-instances	--instance-ids	--dry-run, --force	run-instances (instance must exist in stopped state)	Gateway sends NATS ec2.cmd.{instance-id} → daemon restarts stopped QEMU process with same config → state transitions stopped→pending→running	1. Start a stopped instance 2. Start already-running instance (error: IncorrectInstanceState) 3. Start with invalid instance ID 4. Verify volumes re-mount on start	DONE
stop-instances	--instance-ids	--force, --hibernate, --dry-run	run-instances (instance must be running)	Gateway sends NATS to target node → daemon issues QMP system_powerdown for graceful shutdown → monitors heartbeat until QEMU exits → state transitions running→stopping→stopped	1. Graceful stop of running instance 2. Force stop (kills QEMU process) 3. Stop already-stopped instance (error) 4. Verify ~30s heartbeat detection	DONE
terminate-instances	--instance-ids, DeleteOnTermination (per-volume flag, default true)	--dry-run	run-instances (instance must exist)	Gateway sends NATS to target node → daemon kills QEMU process → cleans up NBD mounts → deletes volumes with DeleteOnTermination=true via volumeService.DeleteVolume() (S3 cleanup of vol/, vol-efi/, vol-cloudinit/) → internal volumes (EFI, cloud-init) always cleaned up via ebs.delete NATS → volumes with DeleteOnTermination=false left in available state → state→terminated	1. Terminate running instance 2. Terminate stopped instance 3. Terminate with DeleteOnTermination=true deletes volumes 4. Terminate with DeleteOnTermination=false preserves volumes 5. Terminate already-terminated (idempotent) 6. Internal volumes (EFI, cloud-init) always cleaned up 7. Invalid instance ID	DONE
reboot-instances	—	--instance-ids, --dry-run	run-instances (instance must be running)	Gateway sends NATS to target node → daemon issues QMP system_reset → instance reboots without stopping	1. Reboot running instance 2. Reboot stopped instance (error) 3. Verify instance stays in running state after reboot	NOT STARTED (parser test exists)
describe-instance-types	--filters (capacity filter only)	--instance-types, --max-results, --next-token, --dry-run, all other filters	None	Gateway fans out NATS ec2.DescribeInstanceTypes to all nodes → each daemon reports supported types (t3.micro/small/medium/large) with vCPU/memory specs → gateway deduplicates and returns	1. List all instance types 2. Filter by specific type 3. Filter with capacity=true shows available slots 4. Verify vCPU/memory specs match hardware	DONE
modify-instance-attribute	—	--instance-id, --instance-type, --user-data, --disable-api-termination, --ebs-optimized	Instance must be stopped	Validate instance stopped → update instance metadata in NATS KV store → changes applied on next start	1. Change instance type while stopped 2. Modify running instance (error) 3. Toggle disable-api-termination	NOT STARTED
get-console-output	—	--instance-id, --latest	Instance must exist	QMP command to read serial console ring buffer → return base64-encoded output with timestamp	1. Get output from running instance 2. Get output from stopped instance 3. Verify cloud-init boot logs visible	NOT STARTED
monitor-instances	—	--instance-ids	Instance must exist	Enable basic monitoring (CPU, disk, network) → store metrics in NATS KV → return monitoring state	1. Enable monitoring 2. Verify monitoring state in describe-instances	NOT STARTED

EC2 - Key Pair Management

Command	Implemented Flags	Missing Flags	Prerequisites	Basic Logic	Test Cases	Status
create-key-pair	--key-name, --key-type (rsa/ed25519)	--key-format (pem/ppk), --tag-specifications, --dry-run	None	NATS ec2.CreateKeyPair → daemon generates SSH keypair → stores public key in Predastore S3 (/bucket/ec2/{name}.pub) → returns private key material (one-time)	1. Create RSA key pair 2. Create ED25519 key pair 3. Duplicate key name (error: InvalidKeyPair.Duplicate) 4. Verify key material returned only on creation	DONE
describe-key-pairs	--key-names, --key-pair-ids	--filters, --max-results, --dry-run	None	NATS ec2.DescribeKeyPairs → daemon lists public keys from Predastore S3 → returns key names and fingerprints	1. List all key pairs 2. Filter by key name 3. Filter by key pair ID 4. Non-existent key returns empty	DONE
delete-key-pair	--key-name, --key-pair-id	--dry-run	Key must exist	NATS ec2.DeleteKeyPair → daemon deletes public key from Predastore S3 → returns success	1. Delete existing key pair 2. Delete non-existent key (idempotent, no error) 3. Verify key no longer in describe-key-pairs	DONE
import-key-pair	--key-name, --public-key-material	--tag-specifications, --dry-run	None	NATS ec2.ImportKeyPair → daemon stores provided public key in Predastore S3 → returns key name and fingerprint	1. Import valid SSH public key 2. Import invalid key material (error) 3. Import duplicate name (error) 4. Verify imported key usable with run-instances	DONE

EC2 - AMI Image Management

Command	Implemented Flags	Missing Flags	Prerequisites	Basic Logic	Test Cases	Status
describe-images	--image-ids (format validation only)	--owners, --executable-users, --filters, --include-deprecated, --include-disabled, --max-results, --next-token, --dry-run	None	NATS ec2.DescribeImages → daemon reads AMI metadata from Predastore S3 buckets (ami-*) → returns image list with state, architecture, block device mappings	1. List all images 2. Filter by image ID 3. Filter by owner (self/amazon) 4. Non-existent AMI returns empty 5. Verify metadata fields (architecture, state, rootDeviceName)	DONE
create-image	Parsed only (mock): --instance-id, --name, --description, --no-reboot, --block-device-mappings, --tag-specifications	All flags need real business logic (currently returns hardcoded ami-ID). Not wired to gateway.	Instance must exist	Stop instance (unless --no-reboot) → snapshot root volume via viperblock → create AMI metadata in S3 → register as new AMI → restart instance → return ami-ID	1. Create image from running instance 2. Create with --no-reboot 3. Invalid instance ID (error) 4. Verify new AMI appears in describe-images 5. Launch new instance from created AMI	PARTIAL (mock service)
register-image	Parsed only (mock): --name, --description, --architecture, --root-device-name, --virtualization-type, --block-device-mappings, --boot-mode	All flags need real business logic. Not wired to gateway. Also: --ena-support, --sriov-net-support, --tpm-support, --imds-support	Image data must exist in S3	Create AMI metadata entry in Predastore → assign ami-ID → set state to available → return ami-ID	1. Register with valid metadata 2. Missing required name (error) 3. Verify registered image in describe-images	PARTIAL (mock service)
deregister-image	Parsed only (mock): --image-id (validated)	Real business logic needed. Not wired to gateway. --dry-run	AMI must exist, no instances using it	Mark AMI metadata as deregistered in S3 → AMI no longer available for new launches → existing instances unaffected	1. Deregister existing AMI 2. Deregister non-existent AMI (error) 3. Verify deregistered AMI not in describe-images 4. Existing instances from AMI still run	PARTIAL (mock service)
copy-image	Parsed only (mock): --source-image-id, --source-region, --name, --description, --encrypted, --kms-key-id	All flags need real business logic. Not wired to gateway. --client-token, --copy-image-tags, --tag-specifications	Source AMI must exist	Copy AMI data between regions/nodes in Predastore → create new AMI metadata with new ami-ID → return new ami-ID	1. Copy image within same region 2. Copy non-existent image (error) 3. Verify copied image independent of source	PARTIAL (mock service)
import-image	—	--disk-containers (Format, Url/S3Bucket+S3Key), --description, --architecture, --platform	S3 bucket with disk image	Download disk image from S3 → convert format (VMDK/VHD/RAW→QCOW2) → create viperblock volume → register as AMI	1. Import QCOW2 image 2. Import RAW image 3. Invalid format (error) 4. Verify imported image launchable	NOT STARTED (gateway stub only)
describe-image-attribute	Parsed only (mock): --image-id (validated), --attribute (validated)	Real business logic needed. --dry-run	AMI must exist	Read specific attribute from AMI metadata in S3 → return attribute value	1. Get description attribute 2. Get launch permission 3. Invalid attribute name (error)	PARTIAL (mock service)
modify-image-attribute	Parsed only (mock): --image-id (validated), --launch-permission, --description, --operation-type	Real business logic needed. --user-ids, --user-groups, --organization-arns, --dry-run	AMI must exist	Update specific attribute in AMI metadata → persist to S3	1. Add launch permission 2. Modify description 3. Invalid AMI (error)	PARTIAL (mock service)
reset-image-attribute	Parsed only (mock): --image-id (validated), --attribute (validated)	Real business logic needed. --dry-run	AMI must exist	Reset attribute to default value in AMI metadata	1. Reset launch permission to owner-only 2. Invalid AMI (error)	PARTIAL (mock service)

EC2 - Volume (EBS) Management

Command	Implemented Flags	Missing Flags	Prerequisites	Basic Logic	Test Cases	Status
describe-volumes	--volume-ids (fast-path lookup), DeleteOnTermination (from persisted VolumeMetadata)	--filters, --max-results, --next-token, --dry-run	None	NATS ec2.DescribeVolumes → daemon queries viperblock for volume metadata → returns volume list with state, size, attachments, type, DeleteOnTermination flag	1. List all volumes 2. Filter by volume ID 3. Filter by attachment state 4. Non-existent volume returns empty 5. DeleteOnTermination reflects persisted value	DONE
modify-volume	--volume-id, --size, --volume-type, --iops	--throughput, --dry-run, --multi-attach-enabled	Volume must exist	NATS ec2.ModifyVolume → daemon sends resize request to viperblock → NBD does not support live resize, instance must be stopped → returns modification state	1. Increase volume size 2. Modify volume type 3. Decrease size (error - not supported) 4. Modify attached volume (requires stop/start)	DONE
create-volume	--size, --availability-zone, --volume-type (gp3 only)	--iops, --encrypted, --snapshot-id, --tag-specifications	Valid AZ configured via hive init	Gateway validates input → NATS ec2.CreateVolume → daemon generates vol-ID via viperblock → creates empty volume of specified size → persists config.json to Predastore S3 → returns vol-ID with state=available	1. Create 80GB gp3 volume 2. Boundary sizes (1 GiB min, 16384 GiB max) 3. Invalid AZ (error) 4. Verify volume in describe-volumes 5. Unsupported volume type (error - only gp3) 6. Size out of range (error)	DONE
delete-volume	--volume-id	--dry-run	Volume must exist and be detached (state=available)	Gateway validates vol- prefix → NATS ec2.DeleteVolume → daemon confirms state=available and no AttachedInstance → NATS ebs.delete to viperblockd (stops nbdkit/WAL) → deletes S3 objects under vol-id/, vol-id-efi/, vol-id-cloudinit/ → returns success	1. Delete detached volume 2. Delete attached volume (error: VolumeInUse) 3. Delete non-existent volume (error: InvalidVolume.NotFound) 4. Verify volume gone from describe-volumes 5. Malformed volume ID (error: InvalidVolumeID.Malformed) 6. Double delete (idempotent NotFound)	DONE
attach-volume	--volume-id, --instance-id, --device (optional, auto-assigns /dev/sd[f-p])	--dry-run	Volume must exist (available), instance must exist (running)	Gateway sends to ec2.cmd.{instanceId} → daemon validates volume (Predastore) → ebs.mount via NATS (viperblock starts NBD server) → QMP blockdev-add (nbd-{volId}) → QMP device_add (virtio-blk-pci, vdisk-{volId}) → three-phase rollback on failure → update EBSRequests + BlockDeviceMappings → persist to JetStream + Predastore → respond with VolumeAttachment	1. Attach volume to running instance 2. Auto-assign device name 3. Attach already-attached volume (VolumeInUse) 4. Attach to non-existent instance (InvalidInstanceID.NotFound) 5. Attach to stopped instance (IncorrectInstanceState) 6. Volume not found (InvalidVolume.NotFound) 7. All device slots full (AttachmentLimitExceeded) 8. Volume persists across stop/start	DONE
detach-volume	--volume-id, --instance-id (optional, resolved via DescribeVolumes), --device (optional cross-check), --force	--dry-run	Volume must be attached, instance must be running	Gateway resolves InstanceId if omitted (via DescribeVolumes) → sends to ec2.cmd.{instanceId} → daemon validates (running, attached, not boot/EFI/CloudInit, device match) → three-phase hot-unplug: QMP device_del (force continues on failure) → QMP blockdev-del (abort if fails, preserves state to prevent double-attach) → ebs.unmount via NATS (best-effort) → remove from EBSRequests + BlockDeviceMappings → update volume metadata to available → persist state → respond with VolumeAttachment (state=detaching)	1. Detach with explicit InstanceId 2. Detach without InstanceId (gateway resolution) 3. Detach with correct --device cross-check 4. Missing VolumeId (InvalidParameterValue) 5. Volume not attached (IncorrectState) 6. Nonexistent volume (InvalidVolume.NotFound) 7. Nonexistent instance (InvalidInstanceID.NotFound) 8. Instance not running (IncorrectInstanceState) 9. Device mismatch (InvalidParameterValue) 10. Boot volume protection (OperationNotPermitted) 11. Force flag (continues past device_del failure) 12. Volume reusability (re-attach after detach)	DONE
describe-volume-status	—	--volume-ids, --filters, --max-results	None	Query viperblock for volume health status → return io-enabled status, availability zone, events	1. Check healthy volume status 2. Check volume during modification 3. Filter by volume ID	NOT STARTED
describe-volumes-modifications	—	--volume-ids, --filters, --max-results	None	Query pending/completed volume modifications → return modification state, progress, original/target size	1. Check in-progress modification 2. Check completed modification 3. No modifications returns empty	NOT STARTED

EC2 - Snapshot Management

Command	Implemented Flags	Missing Flags	Prerequisites	Basic Logic	Test Cases	Status
create-snapshot	—	--volume-id, --description, --tag-specifications	Volume must exist	Trigger viperblock WAL checkpoint → copy volume data to snapshot in Predastore → assign snap-ID → return snapshot with state=pending→completed	1. Snapshot attached volume 2. Snapshot detached volume 3. Invalid volume ID (error) 4. Verify snapshot in describe-snapshots	NOT STARTED
create-snapshots	—	--instance-specification, --description, --tag-specifications	Instance must exist	Create snapshots of all volumes attached to instance → return list of snapshot IDs	1. Snapshot all volumes on instance 2. Instance with no volumes	NOT STARTED
delete-snapshot	—	--snapshot-id	Snapshot must exist, not in use by AMI	Delete snapshot data from Predastore → return success	1. Delete existing snapshot 2. Delete snapshot used by AMI (error) 3. Delete non-existent snapshot (error)	NOT STARTED
describe-snapshots	—	--snapshot-ids, --owner-ids, --filters, --max-results	None	Query Predastore for snapshot metadata → return snapshot list with state, volume ID, size, progress	1. List all snapshots 2. Filter by snapshot ID 3. Filter by volume ID	NOT STARTED
copy-snapshot	—	--source-snapshot-id, --source-region, --description, --encrypted	Source snapshot must exist	Copy snapshot data between regions/nodes in Predastore → assign new snap-ID	1. Copy within same region 2. Copy non-existent snapshot (error)	NOT STARTED

EC2 - Tags

Command	Implemented Flags	Missing Flags	Prerequisites	Basic Logic	Test Cases	Status
create-tags	—	--resources, --tags (Key=,Value=)	Resources must exist	Store tags in NATS KV or Predastore keyed by resource ID → return success	1. Tag an instance 2. Tag a volume 3. Tag non-existent resource (error) 4. Overwrite existing tag	NOT STARTED
delete-tags	—	--resources, --tags	Resources must exist	Remove specified tags from resource metadata → return success	1. Delete existing tag 2. Delete non-existent tag (idempotent)	NOT STARTED
describe-tags	—	--filters, --max-results	None	Query tag store with filters → return tag list with resource type and ID	1. List all tags 2. Filter by resource type 3. Filter by key/value	NOT STARTED

EC2 - Regions & Availability Zones

Command	Implemented Flags	Missing Flags	Prerequisites	Basic Logic	Test Cases	Status
describe-regions	(returns configured region from config — all input params ignored)	--region-names, --filters, --all-regions, --dry-run	None	Return configured region from hive init config (e.g. us-east-1) → local-only response, no NATS	1. List all regions 2. Filter by region name 3. Verify endpoint URL returned	DONE
describe-availability-zones	—	--zone-names, --filters, --all-availability-zones	None	Return configured AZ from hive init (e.g. ap-southeast-2a) → single AZ for Hive v1	1. List all AZs 2. Filter by zone name 3. Verify zone state is available	NOT STARTED

EC2 - VPC Networking (Core)

Requires Open vSwitch (apt install openvswitch-switch openvswitch-common). Single AZ for Hive v1.

Command	Implemented Flags	Missing Flags	Prerequisites	Basic Logic	Test Cases	Status
create-vpc	—	--cidr-block, --tag-specifications, --instance-tenancy	None	Create OVS bridge for VPC → assign vpc-ID → store VPC metadata in NATS KV → configure CIDR range → return VPC with state=available	1. Create VPC with /16 CIDR 2. Overlapping CIDR (error) 3. Invalid CIDR format (error) 4. Verify in describe-vpcs	NOT STARTED
delete-vpc	—	--vpc-id	VPC must be empty (no subnets, gateways, instances)	Verify no dependent resources → delete OVS bridge → remove metadata → return success	1. Delete empty VPC 2. Delete VPC with subnets (error: DependencyViolation) 3. Delete non-existent VPC (error)	NOT STARTED
describe-vpcs	—	--vpc-ids, --filters, --max-results	None	Query NATS KV for VPC metadata → return VPC list with CIDR, state, tags	1. List all VPCs 2. Filter by VPC ID 3. Filter by CIDR block	NOT STARTED
create-subnet	—	--vpc-id, --cidr-block, --availability-zone, --tag-specifications	VPC must exist	Validate CIDR within VPC range → create OVS port group with VLAN tag → assign subnet-ID → store metadata	1. Create subnet within VPC CIDR 2. Subnet CIDR outside VPC range (error) 3. Overlapping subnet CIDRs (error)	NOT STARTED
delete-subnet	—	--subnet-id	Subnet must be empty (no instances)	Verify no instances in subnet → remove OVS port group → delete metadata	1. Delete empty subnet 2. Delete subnet with instances (error)	NOT STARTED
describe-subnets	—	--subnet-ids, --filters, --max-results	None	Query NATS KV for subnet metadata → return subnet list	1. List all subnets 2. Filter by VPC ID 3. Filter by AZ	NOT STARTED
create-security-group	—	--group-name, --description, --vpc-id, --tag-specifications	VPC must exist	Create security group metadata → create default OVS flow rules (deny all inbound, allow all outbound) → assign sg-ID	1. Create SG in VPC 2. Duplicate name in same VPC (error) 3. Verify default rules	NOT STARTED
delete-security-group	—	--group-id	SG must not be in use by instances	Verify no instances reference SG → remove OVS flow rules → delete metadata	1. Delete unused SG 2. Delete SG in use (error)	NOT STARTED
describe-security-groups	—	--group-ids, --group-names, --filters, --max-results	None	Query NATS KV for SG metadata → return SG list with rules	1. List all SGs 2. Filter by VPC ID 3. Filter by group name	NOT STARTED
authorize-security-group-ingress	—	--group-id, --protocol, --port, --cidr, --source-group	SG must exist	Add inbound rule → create OVS OpenFlow rule → persist to metadata	1. Allow SSH (port 22) from 0.0.0.0/0 2. Allow from specific CIDR 3. Duplicate rule (idempotent)	NOT STARTED
authorize-security-group-egress	—	--group-id, --protocol, --port, --cidr	SG must exist	Add outbound rule → create OVS OpenFlow rule → persist to metadata	1. Allow HTTPS outbound 2. Restrict to specific CIDR	NOT STARTED
revoke-security-group-ingress	—	--group-id, --protocol, --port, --cidr	SG must exist, rule must exist	Remove inbound rule → delete OVS OpenFlow rule → update metadata	1. Revoke existing rule 2. Revoke non-existent rule (error)	NOT STARTED
revoke-security-group-egress	—	--group-id, --protocol, --port, --cidr	SG must exist, rule must exist	Remove outbound rule → delete OVS OpenFlow rule → update metadata	1. Revoke existing rule 2. Revoke non-existent rule (error)	NOT STARTED
create-internet-gateway	—	--tag-specifications	None	Create IGW metadata → assign igw-ID → configure OVS NAT bridge for external access	1. Create IGW 2. Verify in describe-internet-gateways	NOT STARTED
attach-internet-gateway	—	--internet-gateway-id, --vpc-id	IGW and VPC must exist	Link IGW to VPC → configure OVS flows for internet routing → update metadata	1. Attach IGW to VPC 2. Attach already-attached IGW (error) 3. Attach to non-existent VPC (error)	NOT STARTED
detach-internet-gateway	—	--internet-gateway-id, --vpc-id	IGW must be attached to VPC	Unlink IGW from VPC → remove OVS internet routing flows	1. Detach attached IGW 2. Detach unattached IGW (error)	NOT STARTED
delete-internet-gateway	—	--internet-gateway-id	IGW must be detached	Verify IGW detached → remove OVS NAT bridge → delete metadata	1. Delete detached IGW 2. Delete attached IGW (error)	NOT STARTED
describe-internet-gateways	—	--internet-gateway-ids, --filters, --max-results	None	Query NATS KV for IGW metadata → return list with attachment info	1. List all IGWs 2. Filter by attachment VPC	NOT STARTED
create-route-table	—	--vpc-id, --tag-specifications	VPC must exist	Create route table metadata → add default local route for VPC CIDR → assign rtb-ID	1. Create route table 2. Verify default local route	NOT STARTED
create-route	—	--route-table-id, --destination-cidr-block, --gateway-id or --nat-gateway-id	Route table and target must exist	Add route entry → configure OVS flow for destination CIDR → update metadata	1. Add internet route via IGW 2. Add route to NAT gateway 3. Conflicting route (error)	NOT STARTED
describe-route-tables	—	--route-table-ids, --filters, --max-results	None	Query NATS KV for route table metadata → return tables with routes	1. List all route tables 2. Filter by VPC	NOT STARTED
create-nat-gateway	—	--subnet-id, --allocation-id, --tag-specifications	Subnet must exist, EIP must be allocated	Create NAT gateway in subnet → configure OVS SNAT rules → assign nat-ID → state=pending→available	1. Create NAT gateway 2. Invalid subnet (error) 3. Verify NAT connectivity from private subnet	NOT STARTED
allocate-address	—	--domain (vpc), --tag-specifications	None	Allocate Elastic IP from pool → assign eipalloc-ID → store in metadata	1. Allocate EIP 2. Verify in describe-addresses	NOT STARTED
associate-address	—	--allocation-id, --instance-id or --network-interface-id	EIP and target must exist	Associate EIP with instance/ENI → configure OVS DNAT/SNAT rules	1. Associate with instance 2. Re-associate (moves EIP) 3. Associate non-existent EIP (error)	NOT STARTED
release-address	—	--allocation-id	EIP must exist and be disassociated	Release EIP back to pool → remove from metadata	1. Release disassociated EIP 2. Release associated EIP (error)	NOT STARTED
describe-addresses	—	--allocation-ids, --filters, --public-ips	None	Query metadata for EIP allocations → return list with association info	1. List all EIPs 2. Filter by allocation ID	NOT STARTED

EC2 - Network Interfaces

Command	Implemented Flags	Missing Flags	Prerequisites	Basic Logic	Test Cases	Status
attach-network-interface	—	--network-interface-id, --instance-id, --device-index	ENI and instance must exist	Attach OVS port to instance QEMU process → configure MAC/IP → update metadata	1. Attach ENI to instance 2. Already attached (error)	NOT STARTED
detach-network-interface	—	--attachment-id, --force	ENI must be attached	Detach OVS port from instance → update metadata	1. Detach ENI 2. Force detach	NOT STARTED

EC2 - Launch Templates

Command	Implemented Flags	Missing Flags	Prerequisites	Basic Logic	Test Cases	Status
create-launch-template	—	--launch-template-name, --launch-template-data (ImageId, InstanceType, etc.), --tag-specifications	None	Store template configuration in NATS KV → assign lt-ID → return template metadata	1. Create template with full config 2. Duplicate name (error) 3. Verify in describe output	NOT STARTED
describe-launch-templates	—	--launch-template-ids, --launch-template-names, --filters	None	Query NATS KV for template metadata → return list	1. List all templates 2. Filter by name	NOT STARTED

EBS Direct API

Command	Implemented Flags	Missing Flags	Prerequisites	Basic Logic	Test Cases	Status
start-snapshot	—	--volume-size, --parent-snapshot-id, --description, --encrypted	None	Initialize snapshot in viperblock → return snap-ID for block-level writes	1. Start new snapshot 2. Start incremental from parent	NOT STARTED
put-snapshot-block	—	--snapshot-id, --block-index, --block-data, --checksum	start-snapshot	Write block data to snapshot at specified index in viperblock → verify checksum	1. Write single block 2. Write with bad checksum (error)	NOT STARTED
get-snapshot-block	—	--snapshot-id, --block-index	Snapshot must exist	Read block data from snapshot at specified index → return data with checksum	1. Read existing block 2. Read sparse block (zeros)	NOT STARTED
complete-snapshot	—	--snapshot-id, --changed-blocks-count	start-snapshot, put-snapshot-block	Finalize snapshot → mark as completed → sync to Predastore	1. Complete valid snapshot 2. Complete with wrong block count (error)	NOT STARTED
list-snapshot-blocks	—	--snapshot-id, --max-results, --next-token	Snapshot must exist	List all non-empty blocks in snapshot with tokens and sizes	1. List blocks of populated snapshot 2. List blocks of empty snapshot	NOT STARTED
list-changed-blocks	—	--second-snapshot-id, --first-snapshot-id, --max-results	Snapshots must exist	Compare two snapshots → return list of differing block indexes	1. Compare parent and child snapshots 2. Compare unrelated snapshots	NOT STARTED

S3 (via Predastore)

Currently S3 operations are handled directly by Predastore. Consider moving control/data plane to Hive format for AWS gateway integration.

Command	Implemented Flags	Missing Flags	Prerequisites	Basic Logic	Test Cases	Status
s3 mb (CreateBucket)	—	s3://bucket-name	None	NATS s3.createbucket → Predastore creates bucket → return success	1. Create valid bucket 2. Duplicate bucket name (error) 3. Invalid bucket name (error)	NOT STARTED (Predastore handles directly)
s3 rb (DeleteBucket)	—	s3://bucket-name, --force	Bucket must exist	NATS s3.deletebucket → verify bucket empty (or force) → Predastore deletes bucket	1. Delete empty bucket 2. Delete non-empty bucket (error) 3. Force delete non-empty bucket	NOT STARTED
s3 ls (ListBuckets/Objects)	—	s3://bucket/prefix, --recursive	None	NATS s3.listobjects → Predastore lists objects with prefix → return list	1. List all buckets 2. List objects in bucket 3. List with prefix filter	NOT STARTED
s3 cp (PutObject/GetObject)	—	source dest, --recursive, --acl	Bucket must exist	NATS s3.putobject/s3.getobject → Predastore stores/retrieves with Reed-Solomon encoding	1. Upload single file 2. Download single file 3. Recursive upload/download 4. Large file (multipart)	NOT STARTED
s3 rm (DeleteObject)	—	s3://bucket/key, --recursive	Object must exist	NATS s3.deleteobject → Predastore deletes object shards	1. Delete single object 2. Recursive delete 3. Delete non-existent (idempotent)	NOT STARTED
s3 sync	—	source dest, --delete, --exclude, --include	Bucket must exist	Compare source and dest → upload changed/new files → optionally delete removed files	1. Sync local dir to S3 2. Sync with --delete 3. Sync with exclude filter	NOT STARTED

IAM (Planned)

Command	Implemented Flags	Missing Flags	Prerequisites	Basic Logic	Test Cases	Status
create-access-key	—	--user-name	User must exist	Generate access key ID and secret → store in NATS JetStream KV → return credentials	1. Create key for user 2. Max keys exceeded (error)	NOT STARTED (gateway stub exists)
delete-access-key	—	--access-key-id, --user-name	Key must exist	Remove access key from NATS KV → return success	1. Delete existing key 2. Delete non-existent key (error)	NOT STARTED
list-access-keys	—	--user-name, --max-items	None	Query NATS KV for user's access keys → return list	1. List keys for user 2. No keys returns empty	NOT STARTED

Update Nov 2025

[PARTIAL] _ Implement multi-tenant support [PARTIAL] _ Move config settings from ~/hive/*.toml, to using Nats Jetstream for core config which can be synced between nodes.

• Implement a lightweight IAM using Nats Jetstream, vs current config files for auth settings. [ONGOING] _ Implement Reed Solomon Encoding for Predastore (S3) objects and a WAL implementation, for storing multiple objects in a single shard (e.g 4MB), with the WAL referencing the location of each object (e.g 4kb min) _ Implement basic KY lookup, object key sha512 (bucket/key), location to shard on S3 (obj.0000124.bin), read WAL (e.g first 4096 bytes) to determine location of the object. e.g key-1234 => obj.0000124.bin, wal header (4096 bytes) > key-1234 == offset location (seek) 16384, len 32768. _ Read multiple nodes (e.g 5 predastore instances, k = 3 (data shards), n = 5 (total shards), n - k = 2 (parity shards) ) [ONGOING] _ Complete core scaffolding AWS SDK/API requirements (ec2 describe-instances, run-instances, etc)
• Implement UEFI support for image downloads and qemu exec in vm.go
• Confirm Alpine Linux, fails import image AMI > (run-instance) ""Failed to read block from AMI source" err="request out of range" - Block size correct?
• Improve shutdown gracefully, ./scripts/stop-dev.sh waits 60 seconds, while qemu/nbd could still be shutting down.
• [DONE] Add delete-volume support via EBS (s3 vol-*) for terminated instance (DeleteOnTermination flag)
• Add default LRU cache support for viperblock, depending on the instance type / volume size and system memory available.

Multi-node setup

[DONE] - Development release - v0.1.0 (single node) [PARTIAL] - Production release - v0.2.0 (multi node)

Design layout for multi-node configuration.

# node1
hive admin init --region ap-southeast-2 --az ap-southeast-2a --node node1 --nodes 3 --hive-dir ~/hive/

# node2
hive admin join --region ap-southeast-2 --az ap-southeast-2a --node node2 --host node1.local:8443 --data-dir ~/hive/

# node3, optionally toggle EBS/EC2/NATs support only
hive admin join --region ap-southeast-2 --az ap-southeast-2a --node node3 --host node1.local:8443 --data-dir ~/hive/ --cap ebs,ec2,nats

If --host is missing, the join command tries multicast broadcast to find parent (leader) node.

Set region with --region ap-southeast-2 which will create a new Hive cluster for the specified region.

For production, it is recommended to run Hive on at least three physical nodes. In this minimum setup, Hive S3 and EBS volumes use 2 data shards and 1 parity shard RS(2,1) to distribute each chunk across the cluster. This configuration tolerates a full node failure while adding only 1.5x storage overhead.

Overview:

• init creates a cluster-id, node-id, and a short join token, starts a tiny control server on :8443, and writes DNS hint files if available.

• init sets target size to 3 and waits until all 3 nodes join and ack.

• join nodes contact node1, present the token or cluster-id, and advertise capabilities.

• node1 appends them to the member set and immediately pushes the current settings bundle to them.

• When member count reaches 3, node1 broadcasts the final cluster settings to all nodes.

• Every node writes the same cluster.json and starts the services that match its capabilities.

Node exchange payload

{
  "cluster_id": "c-82d5",
  "node_id": "n-rpi2",
  "addr": "rpi2.local:8443",
  "version": "0.1.0",
  "caps": ["ec2", "s3"],
  "ts": 1731388800
}

Settings bundle (identical on all 3 once committed)

{
  "cluster_id": "c-82d5",
  "target_size": 3,
  "members": [
    {
      "node_id": "n-rpi1",
      "addr": "rpi1.local:8443",
      "caps": ["ec2", "s3", "nats", "ebs"]
    },
    { "node_id": "n-rpi2", "addr": "rpi2.local:8443", "caps": ["ec2", "s3"] },
    { "node_id": "n-rpi3", "addr": "rpi3.local:8443", "caps": ["nats", "ebs"] }
  ],
  "services": {
    "ec2": { "api_bind": ":9001" },
    "s3": { "api_bind": ":9002", "replicas": 2 },
    "nats": { "cluster": "c-82d5", "quorum": 2 },
    "ebs": { "replicas": 3, "block_size": 4096 }
  },
  "epoch": 1,
  "sig": "ed25519:..." // signed by rpi1 during init
}

Original TODO Items → Development Plan Integration

✅ Completed Integration

All original TODO items have been incorporated into the structured development phases:

Original TODO	Status	Integrated Into	Phase
#1: Binary compile and install.sh	✅ Planned	Phase 7: Task 7.1 (Binary Compilation)	Phase 7
#2: Move daemon.go to services/hived/	✅ Planned	Phase 3: Task 3.2 (Specialized Services)	Phase 3
#3: VPC with openvs-switch as vpcd	✅ Planned	Phase 5: Task 5.1 (VPC with Open vSwitch)	Phase 5
#4: AWS HTTP gateway (awsd)	✅ Planned	Phase 3: Task 3.1 (AWS Gateway Service)	Phase 3
#5: AWS SDK v2 input/output	✅ Planned	Phase 2: Task 2.1 & Phase 3: Task 3.1	Phase 2-3
#6: hive.service for systemd	✅ Planned	Phase 7: Task 7.2 (System Service Integration)	Phase 7
#7: Gossip and RAFT protocols	✅ Planned	Phase 0: Task 0.1 (Service Registry)	Phase 0
#8: etcd/KV for configuration sync	✅ Planned	Phase 0: Task 0.1 (Distributed Config)	Phase 0
#9: Smithy model code generation	✅ Planned	Phase 2: Task 2.1 (Smithy Code Generation)	Phase 2

Development Phase Overview

Phase 0: Distributed Systems Foundation (1-2 weeks)

• Gossip, RAFT, and distributed configuration (#7, #8)

Phase 1: Development Environment Automation (2-3 weeks)

• Multi-service orchestration and hot reloading

Phase 2: AWS API Model Implementation (3-4 weeks)

• Smithy-based code generation (#9)
• AWS SDK v2 integration (#5)

Phase 3: Scalable Gateway and Daemon Architecture (2-3 weeks)

• AWS gateway service awsd (#4)
• Service refactoring to hived (#2)
• VPC daemon vpcd foundation (#3)

Phase 4: Service Integration and Orchestration (3-4 weeks)

• NATS clustering and cross-service coordination

Phase 5: Infrastructure Services (4-5 weeks)

• VPC networking with Open vSwitch (#3)
• Advanced VM features

Phase 6: Testing and Validation Framework (2-3 weeks)

• AWS CLI compatibility testing

Phase 7: Production Deployment and Packaging (2-3 weeks)

• Binary compilation and distribution (#1)
• Systemd service integration (#6)
• Production configuration management

Service Architecture

The development plan implements these services:

• awsd - AWS API Gateway (TODO #4)
• hived - Main Hive compute daemon (TODO #2)
• vpcd - VPC networking daemon (TODO #3)
• Predastore - S3 service (existing)
• Viperblock - EBS service (existing)

Getting Started

For development setup:

./scripts/dev-setup.sh     # Setup complete development environment
./scripts/start-dev.sh     # Start all services

For detailed implementation guidance, see HIVE_DEVELOPMENT_PLAN.md.