spinifex/HIVE_DEVELOPMENT_PLAN.md

Hive Development Plan: Proof-of-Concept to Beta Release

This document provides a comprehensive roadmap for transforming Hive from a proof-of-concept into a production-ready AWS-compatible infrastructure platform. Use this as the primary guide for development work on the Hive codebase.

Project Overview

Goal: Transform Hive into a beta release that provides full AWS CLI compatibility for EC2, EBS, S3, and VPC services with seamless development environment automation.

Current State: Working proof-of-concept with:

• Basic EC2 instance provisioning via NATS
• Viperblock integration for EBS volumes
• Predastore S3-compatible storage
• TLS-enabled HTTP gateway with partial AWS API support
• QEMU/KVM-based virtualization

Target State: Production-ready platform with:

• Full AWS CLI compatibility (aws ec2 run-instances, aws s3 mb, etc.)
• Auto-scaling development environment with hot reloading
• Comprehensive AWS API coverage (EC2, EBS, S3, VPC)
• Multi-node deployment capability
• Robust testing framework with >90% AWS compatibility

Architecture Analysis

Scalable Message-Driven Architecture

Hive uses a message-driven microservices architecture that separates API handling from request processing:

AWS SDK (custom endpoint) → Hive API Gateway → NATS Topics → Specialized Daemons
                                    ↑                              ↓
                         AWS-compatible response ←  NATS Topics  ←  Processing Results

Core Components:

1. Hive API Gateway Service (hive/gateway/)

   • Handles AWS SDK requests on custom endpoints (https://localhost:9999)
   • Parses AWS API calls (EC2, S3, VPC, EBS)
   • Translates to NATS messages and broadcasts on topics
   • Receives NATS responses and formats AWS-compatible XML/JSON responses
   • Handles AWS Signature V4 authentication

2. Specialized Daemon Services (horizontally scalable)

   • EC2 Daemons: Listen on ec2.* topics, manage QEMU/KVM instances
   • EBS Daemons: Listen on ebs.* topics, coordinate with Viperblock
   • S3 Daemons: Listen on s3.* topics, proxy to Predastore
   • VPC Daemons: Listen on vpc.* topics, manage networking

3. NATS Message Broker (clustering capable)

   • Topic-based routing: ec2.runinstances, ebs.createvolume, s3.createbucket
   • Queue groups for load balancing: multiple daemons can handle same topic
   • Request-response pattern with timeouts
   • Service discovery and health monitoring

Current Strengths

• Message-Driven Foundation: Already implements NATS request-response pattern (daemon.go:351)
• Queue Groups: Uses "hive-workers" queue group for load balancing (daemon.go:749)
• TLS Gateway: Working AWS SDK integration with TLS (daemon.go:330)
• Service Integration: Viperblock (EBS) and Predastore (S3) via NATS messaging
• VM Management: QEMU/KVM with QMP integration for lifecycle management, support " QMP event event=BLOCK_IO_ERROR" to catch corrupt disks.

Scalability Benefits

• Horizontal Scaling: Add more daemon instances to handle increased load
• Service Isolation: Each AWS service type runs in dedicated daemons
• Fault Tolerance: NATS queue groups provide automatic failover
• Development Flexibility: Services can be developed and deployed independently

Areas for Enhancement

• API Gateway Separation: Extract gateway logic from daemon.go into dedicated service
• Service Specialization: Create dedicated daemon types for each AWS service
• Authentication: Implement full AWS Signature V4 validation
• Auto-scaling: Dynamic daemon scaling based on NATS queue depth

Development Phases

Phase 0: Distributed Systems Foundation (1-2 weeks)

Priority: CRITICAL - Multi-node coordination foundation

Task 0.1: Service Registry and Discovery

# Distributed coordination infrastructure:
hive/cluster/                    # Cluster coordination package
hive/cluster/gossip.go          # Gossip protocol for node discovery
hive/cluster/raft.go            # RAFT consensus for configuration
hive/cluster/registry.go        # Service registry and health monitoring
hive/cluster/config.go          # Distributed configuration management

Integration with Existing TODO Items:

• ✅ TODO #7: "Add gossip and RAFT protocol for Hive nodes to communicate and sync"
• ✅ TODO #8: "Add support for etcd or simple key/value for configuration sync"

Key Features:

• Node Discovery: Gossip protocol for automatic node discovery and failure detection
• Configuration Consensus: RAFT-based distributed configuration management
• Service Health: Cluster-wide service health monitoring and failover
• State Synchronization: Distributed state management for multi-node deployments

Task 0.2: Cluster-Aware NATS Configuration

# Enhanced NATS clustering:
hive/nats/cluster.go            # NATS cluster configuration with service discovery
hive/nats/election.go           # Leader election for gateway services
hive/nats/sharding.go           # Topic sharding for scalability

Phase 1: Development Environment Automation (2-3 weeks)

Priority: CRITICAL - Essential for productive development

Task 1.1: Multi-Service Orchestration Framework

# Create new files:
scripts/dev-setup.sh              # One-command environment setup
tools/hive-dev/main.go           # CLI tool for development services
tools/hive-dev/cmd/start.go      # Start all services with dependencies
tools/hive-dev/cmd/stop.go       # Graceful service shutdown
tools/hive-dev/cmd/status.go     # Service health checking
tools/hive-dev/cmd/logs.go       # Aggregated logging
tools/hive-dev/cmd/scale.go      # Scale daemon instances

Service Startup Order:

1. NATS Server (message broker)
2. Predastore (S3-compatible storage)
3. Viperblock (EBS block storage)
4. NBDkit (network block device)
5. Hive API Gateway (AWS API endpoint)
6. Hive Daemons (EC2, EBS, S3, VPC workers)

Key Features:

• Multi-daemon management: start multiple EC2/EBS/VPC daemon instances
• NATS queue monitoring and daemon auto-scaling
• Health checks via NATS service discovery
• TLS certificate generation for gateway service

Task 1.2: Auto-restart Development Environment

# Configuration files:
.air.toml                        # Root air configuration
hive/.air.toml                   # Hive service hot-reload config
predastore/.air.toml             # Predastore hot-reload config
viperblock/.air.toml             # Viperblock hot-reload config
tools/file-watcher/main.go       # Custom file watcher for cross-service dependencies

Integration Points:

• File watcher monitors Go files across all components
• Graceful service restart on code changes
• Automatic service dependency restart when interfaces change
• Development log aggregation and filtering

Task 1.3: Configuration Management

# Configuration system:
config/dev.yaml                 # Development environment config
config/test.yaml                # Testing environment config
config/prod.yaml                # Production environment config
tools/config-gen/main.go        # Configuration generator and validator

Phase 2: AWS API Model Implementation (3-4 weeks)

Priority: HIGH - Foundation for all AWS compatibility

Task 2.1: Smithy-Based AWS API Code Generation Framework

# Code generation tools:
tools/aws-codegen/main.go        # Main code generator
tools/aws-codegen/smithy/        # Smithy model parser
tools/aws-codegen/templates/     # Go code templates
tools/aws-codegen/models/        # Downloaded AWS Smithy models
hive/services/generated/         # Generated AWS types and interfaces

Integration with Existing TODO Items:

• ✅ TODO #9: "Use Smithy e.g sdk-codegen/aws-models/ec2.json to generate struct/XML input/output"

Smithy Model Integration:

• Download official AWS Smithy models from sdk-codegen/aws-models/
• Parse ec2.json, s3.json, vpc.json service definitions
• Generate Go structs with proper JSON/XML tags matching AWS exactly
• Create operation interfaces that mirror AWS SDK v2 patterns
• Auto-generate request/response validation logic

Generated Structure Example:

// hive/services/generated/ec2/types.go
type RunInstancesInput struct {
    ImageId          *string                 `json:"ImageId" xml:"ImageId"`
    InstanceType     InstanceType           `json:"InstanceType" xml:"InstanceType"`
    KeyName          *string                `json:"KeyName" xml:"KeyName"`
    MaxCount         *int32                 `json:"MaxCount" xml:"MaxCount"`
    MinCount         *int32                 `json:"MinCount" xml:"MinCount"`
    SecurityGroupIds []string               `json:"SecurityGroupIds" xml:"SecurityGroupId"`
    SubnetId         *string                `json:"SubnetId" xml:"SubnetId"`
    UserData         *string                `json:"UserData" xml:"UserData"`
    // ... all AWS EC2 fields
}

// hive/services/generated/ec2/service.go
type EC2Service interface {
    RunInstances(ctx context.Context, input *RunInstancesInput) (*RunInstancesOutput, error)
    DescribeInstances(ctx context.Context, input *DescribeInstancesInput) (*DescribeInstancesOutput, error)
    StartInstances(ctx context.Context, input *StartInstancesInput) (*StartInstancesOutput, error)
    StopInstances(ctx context.Context, input *StopInstancesInput) (*StopInstancesOutput, error)
    TerminateInstances(ctx context.Context, input *TerminateInstancesInput) (*TerminateInstancesOutput, error)
    CreateVolume(ctx context.Context, input *CreateVolumeInput) (*CreateVolumeOutput, error)
    AttachVolume(ctx context.Context, input *AttachVolumeInput) (*AttachVolumeOutput, error)
    DetachVolume(ctx context.Context, input *DetachVolumeInput) (*DetachVolumeOutput, error)
    // ... all EC2 operations
}

Task 2.2: Service Implementation Structure

# Service implementations:
hive/services/ec2/               # EC2 service implementation
hive/services/ec2/instances.go   # Instance management
hive/services/ec2/volumes.go     # EBS volume management
hive/services/ec2/security.go    # Security groups and key pairs
hive/services/s3/                # S3 service (proxy to Predastore)
hive/services/vpc/               # VPC networking service
hive/services/iam/               # IAM service for authentication

Phase 3: Scalable Gateway and Daemon Architecture (2-3 weeks)

Priority: CRITICAL - Core scalable architecture

Task 3.1: AWS Gateway Service (awsd)

# New dedicated AWS gateway service:
cmd/awsd/main.go                 # AWS gateway service entry point (TODO #4)
hive/services/awsd/              # AWS gateway service package
hive/services/awsd/server.go     # TLS HTTP server (extracted from daemon.go:252)
hive/services/awsd/router.go     # AWS service routing
hive/services/awsd/nats.go       # NATS client for broadcasting requests
hive/services/awsd/auth/         # AWS Signature V4 authentication
hive/services/awsd/handlers/ec2.go   # EC2 → NATS translation
hive/services/awsd/handlers/s3.go    # S3 → NATS translation
hive/services/awsd/handlers/ebs.go   # EBS → NATS translation
hive/services/awsd/handlers/vpc.go   # VPC → NATS translation
hive/services/awsd/middleware/   # Request/response middleware

Integration with Existing TODO Items:

• ✅ TODO #4: "Implement AWS Server HTTP gateway (awsd) to accept commands to the AWS control plane"
• ✅ TODO #5: "Implement command input/output from aws-sdk-go-v2"

Gateway Responsibilities:

• Listen on :9999 for AWS SDK requests
• Parse AWS API calls and validate signatures using aws-sdk-go-v2 structures
• Translate to NATS messages: aws.Request("ec2.runinstances", data, 30s)
• Format NATS responses back to AWS XML/JSON
• Handle timeouts and error responses

Task 3.2: Specialized Service Daemons

# Refactor current daemon.go into specialized services (TODO #2):
cmd/hived/main.go                # Main Hive coordination daemon
cmd/vpcd/main.go                 # VPC networking daemon

hive/services/hived/             # Main Hive service (TODO #2)
hive/services/hived/instances.go # Instance lifecycle (from daemon.go:1874)
hive/services/hived/qemu.go      # QEMU/KVM management
hive/services/hived/nats.go      # NATS subscriber: "ec2.*"
hive/services/hived/ebs.go       # EBS volume coordination

hive/services/vpcd/              # VPC daemon package (TODO #3)
hive/services/vpcd/networking.go # Open vSwitch integration
hive/services/vpcd/subnets.go    # Subnet management
hive/services/vpcd/nats.go       # NATS subscriber: "vpc.*"

Service Architecture Changes:

• hived: Main compute service (replaces current daemon.go functionality)
• awsd: AWS API gateway (new service for AWS SDK compatibility)
• vpcd: VPC networking service (new service for network management)
• Predastore: S3 service (existing, integrated via NATS)
• Viperblock: EBS service (existing, integrated via NATS)

NATS Topic Structure:

# Request topics (Gateway → Daemons):
ec2.runinstances                 # Launch new instances
ec2.describeinstances            # Query instance status
ec2.startinstances               # Start stopped instances
ec2.stopinstances                # Stop running instances
ec2.terminateinstances           # Terminate instances

ebs.createvolume                 # Create new EBS volume
ebs.attachvolume                 # Attach volume to instance
ebs.detachvolume                 # Detach volume
ebs.describevolumes              # List volumes

s3.createbucket                  # Create S3 bucket
s3.putobject                     # Store object
s3.getobject                     # Retrieve object

vpc.createvpc                    # Create VPC
vpc.createsubnet                 # Create subnet
# ... etc

Task 3.3: Horizontal Scaling Support

# Scaling infrastructure:
tools/hive-scaler/main.go        # Daemon scaling utility
hive/scaling/                    # Auto-scaling logic
hive/scaling/metrics.go          # NATS queue depth monitoring
hive/scaling/policies.go         # Scaling policies (CPU, queue depth)

Scaling Features:

• Multiple daemon instances subscribe to same NATS topics
• NATS queue groups ("hive-workers") provide load balancing
• Monitor queue depths: scale up daemons when queues grow
• Scale down during low load periods

Phase 4: Service Integration and Orchestration (3-4 weeks)

Priority: MEDIUM-HIGH - Service coordination

Task 4.1: NATS Clustering and Service Discovery

# NATS clustering and coordination:
hive/nats/                       # NATS infrastructure management
hive/nats/cluster.go             # NATS cluster configuration
hive/nats/discovery.go           # Service discovery via NATS
hive/nats/health.go              # Health check messaging patterns
hive/nats/auth.go                # NATS authentication and ACLs

Features:

• NATS clustering for high availability
• Service registration: daemons announce capabilities on startup
• Health monitoring via heartbeat topics: health.ec2.daemon.1, health.ebs.daemon.2
• Load balancing metrics: queue depths, response times per daemon
• Auto-discovery of available daemon services

Task 4.2: Predastore S3 Integration

# S3 service integration:
hive/services/s3/proxy.go        # Direct proxy to Predastore
hive/services/s3/buckets.go      # Bucket management
hive/services/s3/events.go       # S3 event notifications

Task 4.3: Enhanced Viperblock Integration

# EBS service integration:
hive/services/ebs/volumes.go     # Volume lifecycle management
hive/services/ebs/snapshots.go   # Snapshot management
hive/services/ebs/encryption.go  # Volume encryption
hive/services/ebs/monitoring.go  # Performance monitoring

Phase 5: Infrastructure Services (4-5 weeks)

Priority: MEDIUM - Advanced features

Task 5.1: VPC Network Management with Open vSwitch

# VPC implementation with openvs-switch:
hive/services/vpcd/              # VPC service implementation (TODO #3)
hive/services/vpcd/ovs.go        # Open vSwitch integration and management
hive/services/vpcd/subnets.go    # Subnet management with OVS bridges
hive/services/vpcd/routing.go    # Route table management via OVS flows
hive/services/vpcd/security.go   # Security groups via OVS rules
hive/services/vpcd/gateways.go   # Internet/NAT gateway management
hive/networking/ovs/             # Open vSwitch wrapper and utilities

Integration with Existing TODO Items:

• ✅ TODO #3: "Implement VPC support using openvs-switch (add VPC and subnets, single AZ to start)"

Open vSwitch Integration:

• OVS Bridge Management: Create and manage virtual switches for VPCs
• VLAN Tagging: Subnet isolation using VLAN tags
• Flow Rules: Security group implementation via OpenFlow rules
• Port Management: VM network interface attachment to OVS ports
• Single AZ: Initial implementation focuses on single availability zone

Task 5.2: Resource Management Enhancement

# Resource management:
hive/resources/                  # Resource management system
hive/resources/quotas.go         # Resource quotas and limits
hive/resources/allocation.go     # Cluster-wide resource allocation
hive/resources/scaling.go        # Auto-scaling capabilities
hive/resources/tagging.go        # Resource tagging and cost allocation

Task 5.3: Advanced VM Features

# VM enhancements:
hive/services/metadata/          # Instance metadata service (169.254.169.254)
hive/vm/userdata.go             # Enhanced user data and cloud-init
hive/vm/snapshots.go            # VM backup and snapshot
hive/vm/migration.go            # VM migration capabilities

Phase 6: Testing and Validation Framework (2-3 weeks)

Priority: HIGH - Quality assurance

Task 6.1: AWS CLI Integration Tests

# Comprehensive testing:
tests/integration/aws-cli/       # AWS CLI integration tests
tests/compatibility/             # AWS compatibility validation
tests/performance/               # Performance and load tests
tests/scenarios/                 # End-to-end scenario tests

Test Coverage:

• All AWS CLI commands supported by Hive
• Parameter validation and error handling
• Performance benchmarks vs. real AWS
• Multi-service workflow testing

Task 6.2: Service Testing Framework

# Testing framework:
tests/unit/                      # Unit tests for all services
tests/integration/               # Service integration tests
tests/chaos/                     # Chaos engineering tests
tools/test-runner/               # Custom test runner and reporter

Task 6.3: Development Testing Tools

# Development tools:
tools/mock-aws/                  # Mock AWS services for offline dev
tools/test-data/                 # Test data generators
tools/test-env/                  # Test environment management

Phase 7: Production Deployment and Packaging (2-3 weeks)

Priority: HIGH - Production readiness

Task 7.1: Binary Compilation and Distribution

# Build and packaging system:
scripts/build-release.sh         # Multi-platform binary compilation
scripts/install.sh               # Production installation script
packaging/                       # Package management files
packaging/debian/                # Debian/Ubuntu packages
packaging/rpm/                   # RedHat/CentOS packages
packaging/docker/                # Docker containers for production

Integration with Existing TODO Items:

• ✅ TODO #1: "Add support for binary compile and simple install.sh to setup a new node"

Production Build Features:

• Cross-Platform Compilation: Linux (amd64, arm64), Windows, macOS binaries
• Static Linking: Self-contained binaries with minimal dependencies
• Installation Scripts: One-command node setup for production deployments
• Package Management: Native packages for major Linux distributions

Task 7.2: System Service Integration

# System integration:
systemd/                         # Systemd service files
systemd/hived.service           # Main Hive compute daemon
systemd/awsd.service            # AWS gateway service
systemd/vpcd.service            # VPC networking daemon
init/                           # SysV init scripts for legacy systems
scripts/node-setup.sh          # Complete node configuration script

Integration with Existing TODO Items:

• ✅ TODO #6: "Implement hive.service for service management and boot"

System Integration Features:

• Systemd Services: Proper service definitions with dependencies and restart policies
• Boot Integration: Automatic service startup on system boot
• Health Monitoring: Service health checks and automatic restart on failure
• Log Management: Proper logging integration with journald/syslog
• User Management: Dedicated system users and proper permissions

Task 7.3: Production Configuration Management

# Production configuration:
config/production.yaml          # Production configuration template
config/cluster.yaml             # Multi-node cluster configuration
scripts/cluster-init.sh         # Cluster initialization script
tools/hive-admin/               # Cluster administration tool
tools/hive-admin/node.go        # Node management commands
tools/hive-admin/cluster.go     # Cluster operations
tools/hive-admin/health.go      # Health monitoring and diagnostics

Production Features:

• Configuration Templates: Production-ready configuration with security defaults
• Cluster Bootstrapping: Automated multi-node cluster setup
• Administrative Tools: Command-line tools for cluster management
• Security Hardening: Production security configurations and best practices

Implementation Guidelines

Development Workflow

1. Start with Phase 1 - Essential for productive development
2. Use TDD approach - Write tests before implementation
3. Maintain backward compatibility - Don't break existing functionality
4. Document as you go - Update CLAUDE.md with new patterns
5. Test AWS CLI compatibility - Validate every implemented operation

Code Organization Principles

• Generate don't hand-write AWS API types - Use code generation for consistency
• Separate concerns - Keep service logic separate from HTTP handling
• Use interfaces - Enable testing and future extensibility
• Follow existing patterns - Maintain consistency with current codebase
• Error handling - Implement proper AWS-compatible error responses

Key Integration Points and Refactoring Guide

Current Code Mapping to New Architecture:

From hive/daemon/daemon.go → Multiple Services:

1. Gateway Service (cmd/hive-gateway/main.go):

   • Extract: StartAWSGateway() (daemon.go:252-332)
   • Extract: HTTP handlers (daemon.go:337-478)
   • Extract: AWS XML response generation (daemon.go:586-627)
   • Keep: NATS client for request broadcasting

2. EC2 Daemon Service (cmd/hive-ec2-daemon/main.go):

   • Extract: handleEC2Launch() (daemon.go:1044-1107)
   • Extract: LaunchInstance() (daemon.go:1874-1960)
   • Extract: StartInstance() (daemon.go:1962-2168)
   • Extract: QMP client management (daemon.go:1815-1872)
   • Keep: NATS subscriber patterns

3. EBS Daemon Service (cmd/hive-ebs-daemon/main.go):

   • Extract: MountVolumes() (daemon.go:2170-2221)
   • Extract: Viperblock integration patterns (daemon.go:1265-1457)
   • New: Volume lifecycle management
   • New: EBS-specific NATS topics

NATS Message Patterns (Already Working):

• Request Pattern: d.natsConn.Request("ec2.runinstances", requestData, 30*time.Second) (daemon.go:351)
• Subscribe Pattern: d.natsConn.QueueSubscribe("ec2.launch", "hive-workers", d.handleEC2Launch) (daemon.go:749)
• Queue Groups: Already uses "hive-workers" for load balancing

Configuration Extensions (hive/config/config.go):

• Add daemon-specific configurations
• Service discovery endpoints
• NATS clustering configuration
• Auto-scaling policies

Existing Integration Patterns to Preserve:

• Viperblock Integration: EBS mounting patterns (daemon.go:2170-2221)
• Predastore Integration: S3 client patterns (daemon.go:1659-1676)
• QMP Integration: VM lifecycle management (daemon.go:922-989)
• NATS Messaging: Request-response patterns throughout daemon.go

Success Criteria

• AWS CLI Commands Work: aws ec2 run-instances, aws s3 mb, aws ec2 create-volume function identically to real AWS
• Development Experience: ./tools/hive-dev start brings up entire environment in <2 minutes
• Performance: <100ms response time for API operations, <2 second cold start
• Compatibility: >90% AWS API compatibility for supported operations
• Scalability: Support for 50+ concurrent VM instances

Implementation Priority and Dependencies

Integrated Development Phases (Total: 19-26 weeks)

Phase 0 (1-2 weeks): Distributed Systems Foundation

• ✅ Addresses TODO #7 (Gossip/RAFT) and #8 (etcd/KV configuration)

Phase 1 (2-3 weeks): Development Environment Automation

• Critical for productive cross-service development

Phase 2 (3-4 weeks): AWS API Model Implementation

• ✅ Addresses TODO #9 (Smithy code generation) and #5 (AWS SDK v2)

Phase 3 (2-3 weeks): Scalable Gateway and Daemon Architecture

• ✅ Addresses TODO #4 (awsd gateway) and #2 (hived refactoring)

Phase 4 (3-4 weeks): Service Integration and Orchestration

• NATS clustering and multi-service coordination

Phase 5 (4-5 weeks): Infrastructure Services

• ✅ Addresses TODO #3 (VPC with Open vSwitch as vpcd)

Phase 6 (2-3 weeks): Testing and Validation Framework

• AWS CLI compatibility and comprehensive testing

Phase 7 (2-3 weeks): Production Deployment and Packaging

• ✅ Addresses TODO #1 (binary compilation) and #6 (systemd services)

Key Achievements

✅ 100% TODO Integration: All original TODO items incorporated into structured phases ✅ Service Architecture: Adopts planned service names (awsd, hived, vpcd) ✅ Technology Choices: Integrates specific selections (Open vSwitch, Smithy models) ✅ Production Ready: Comprehensive deployment and packaging strategy ✅ Scalable Foundation: Distributed systems and message-driven architecture

Getting Started (for Claude Code)

When working on this codebase:

1. Read current CLAUDE.md for build commands and architecture understanding
2. Start with Phase 0 tasks if working on distributed systems/clustering
3. Start with Phase 1 tasks if improving development experience
4. Begin with Phase 2 tasks if implementing AWS API compatibility
5. Always run tests after making changes: make test in each component
6. Update documentation when adding new patterns or significantly changing architecture
7. Test with real AWS CLI to ensure compatibility

The existing code in hive/daemon/daemon.go provides an excellent foundation - it already implements TLS, AWS SDK integration, NATS coordination, and service integration patterns that should be extended rather than replaced. All original TODO items have been preserved and integrated into this comprehensive development plan.

AWS Compatibility Enhancement Recommendations

This section provides detailed recommendations for improving AWS functionality replication in Hive, organized by priority and implementation complexity.

High Priority Enhancements

These features provide immediate value and are relatively straightforward to implement:

1. Request ID Tracking

Purpose: AWS returns a unique RequestId with every response for traceability and debugging.

Implementation:

// hive/services/common/metadata.go
type ResponseMetadata struct {
    RequestId string `locationName:"requestId" xml:"RequestId"`
}

// Add to all response wrappers
type EC2Response struct {
    XMLName  xml.Name         `xml:"RunInstancesResponse"`
    Result   *ec2.Reservation `xml:"instancesSet"`
    Metadata ResponseMetadata `xml:"ResponseMetadata"`
}

// Generate request IDs
func GenerateRequestId() string {
    return fmt.Sprintf("req-%s", uuid.New().String())
}

Benefits:

• Enables request tracing across distributed systems
• Matches AWS CLI output format exactly
• Essential for debugging multi-service operations
• Required for AWS CloudTrail-like audit logging

2. DryRun Implementation

Purpose: Allow validation-only mode for all operations without making actual changes.

Implementation:

// Add to each operation handler
func ValidateCreateImageInput(input *ec2.CreateImageInput) (err error) {
    // ... existing validation ...
    return nil
}

func EC2_Process_CreateImage(jsonData []byte) (output []byte) {
    var input ec2.CreateImageInput
    err := json.Unmarshal(jsonData, &input)
    if err != nil {
        return utils.GenerateErrorPayload(awserrors.ErrorValidationError)
    }

    // Validate input
    err = ValidateCreateImageInput(&input)
    if err != nil {
        return utils.GenerateErrorPayload(err)
    }

    // If DryRun, return success after validation
    if input.DryRun != nil && *input.DryRun {
        return utils.GenerateSuccessPayload(map[string]bool{"Return": true})
    }

    // ... actual implementation ...
}

Benefits:

• Safety feature for testing scripts and automation
• Validates permissions and parameters without side effects
• Standard AWS behavior expected by users
• Critical for CI/CD pipeline validation

3. Comprehensive Error Code Mapping

Purpose: Expand error types to match AWS error codes precisely.

Implementation:

// hive/awserrors/ec2_errors.go
package awserrors

import "github.com/aws/aws-sdk-go/aws"

var (
    // AMI Errors
    ErrorInvalidAMIIDNotFound = ResponseError{
        Code:    aws.String("InvalidAMIID.NotFound"),
        Message: aws.String("The image id '[%s]' does not exist"),
    }
    ErrorInvalidAMIIDMalformed = ResponseError{
        Code:    aws.String("InvalidAMIID.Malformed"),
        Message: aws.String("Invalid id: \"%s\" (expecting \"ami-...\")"),
    }
    ErrorInvalidAMIIDUnavailable = ResponseError{
        Code:    aws.String("InvalidAMIID.Unavailable"),
        Message: aws.String("The image id '[%s]' is no longer available"),
    }

    // Instance Errors
    ErrorInvalidInstanceIDNotFound = ResponseError{
        Code:    aws.String("InvalidInstanceID.NotFound"),
        Message: aws.String("The instance ID '%s' does not exist"),
    }
    ErrorInvalidInstanceIDMalformed = ResponseError{
        Code:    aws.String("InvalidInstanceID.Malformed"),
        Message: aws.String("Invalid id: \"%s\" (expecting \"i-...\")"),
    }
    ErrorIncorrectInstanceState = ResponseError{
        Code:    aws.String("IncorrectInstanceState"),
        Message: aws.String("The instance '%s' is not in a state from which this command can be executed."),
    }

    // Volume Errors
    ErrorInvalidVolumeNotFound = ResponseError{
        Code:    aws.String("InvalidVolume.NotFound"),
        Message: aws.String("The volume '%s' does not exist."),
    }
    ErrorVolumeInUse = ResponseError{
        Code:    aws.String("VolumeInUse"),
        Message: aws.String("Volume %s is currently attached to %s"),
    }

    // Key Pair Errors
    ErrorInvalidKeyPairNotFound = ResponseError{
        Code:    aws.String("InvalidKeyPair.NotFound"),
        Message: aws.String("The key pair '%s' does not exist"),
    }
    ErrorInvalidKeyPairDuplicate = ResponseError{
        Code:    aws.String("InvalidKeyPair.Duplicate"),
        Message: aws.String("The keypair '%s' already exists."),
    }

    // Resource Limit Errors
    ErrorInstanceLimitExceeded = ResponseError{
        Code:    aws.String("InstanceLimitExceeded"),
        Message: aws.String("You have requested more instances than your current instance limit allows."),
    }
    ErrorVolumeLimitExceeded = ResponseError{
        Code:    aws.String("VolumeLimitExceeded"),
        Message: aws.String("You have exceeded the maximum number of volumes."),
    }
)

// Helper to format error messages
func FormatError(err ResponseError, args ...interface{}) ResponseError {
    if err.Message != nil {
        formatted := fmt.Sprintf(*err.Message, args...)
        err.Message = &formatted
    }
    return err
}

Usage:

// In validation functions
if !strings.HasPrefix(*input.ImageId, "ami-") {
    return awserrors.FormatError(awserrors.ErrorInvalidAMIIDMalformed, *input.ImageId)
}

// In business logic
image := findImage(imageId)
if image == nil {
    return utils.GenerateErrorPayload(
        awserrors.FormatError(awserrors.ErrorInvalidAMIIDNotFound, imageId))
}

4. Integration Testing with AWS SDK

Purpose: Validate Hive compatibility using real AWS SDK clients.

Implementation:

// tests/integration/aws_sdk_test.go
package integration

import (
    "testing"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/ec2"
    "github.com/stretchr/testify/assert"
)

func TestEC2WithRealAWSSDK(t *testing.T) {
    // Configure SDK to point to Hive endpoint
    sess := session.Must(session.NewSession(&aws.Config{
        Endpoint:         aws.String("https://localhost:9999"),
        Region:           aws.String("us-east-1"),
        DisableSSL:       aws.Bool(false),
        S3ForcePathStyle: aws.Bool(true),
    }))

    svc := ec2.New(sess)

    t.Run("CreateKeyPair", func(t *testing.T) {
        result, err := svc.CreateKeyPair(&ec2.CreateKeyPairInput{
            KeyName: aws.String("test-key-" + uuid.New().String()),
        })

        assert.NoError(t, err)
        assert.NotNil(t, result.KeyName)
        assert.NotNil(t, result.KeyFingerprint)
        assert.NotNil(t, result.KeyMaterial)
        assert.NotNil(t, result.KeyPairId)
    })

    t.Run("DescribeKeyPairs", func(t *testing.T) {
        result, err := svc.DescribeKeyPairs(&ec2.DescribeKeyPairsInput{})

        assert.NoError(t, err)
        assert.NotNil(t, result.KeyPairs)
    })

    t.Run("CreateImage", func(t *testing.T) {
        // First create an instance
        runResult, err := svc.RunInstances(&ec2.RunInstancesInput{
            ImageId:      aws.String("ami-test"),
            InstanceType: aws.String("t2.micro"),
            MinCount:     aws.Int64(1),
            MaxCount:     aws.Int64(1),
        })
        assert.NoError(t, err)
        instanceId := runResult.Instances[0].InstanceId

        // Create image from instance
        imgResult, err := svc.CreateImage(&ec2.CreateImageInput{
            InstanceId: instanceId,
            Name:       aws.String("test-image"),
        })

        assert.NoError(t, err)
        assert.NotNil(t, imgResult.ImageId)
        assert.True(t, strings.HasPrefix(*imgResult.ImageId, "ami-"))
    })
}

Medium Priority Enhancements

These features improve reliability and correctness:

5. Idempotency Token Support

Purpose: Allow safe retries of operations without creating duplicates.

What is Idempotency? Idempotency means performing the same operation multiple times produces the same result as performing it once. Critical for handling network failures, timeouts, and retry logic.

Example Problem:

Client: CreateKeyPair(name="prod-key")
[Network timeout - did it work?]
Client: CreateKeyPair(name="prod-key") [retry]
Server: ERROR - key already exists
[Client never got the private key material!]

Solution with Idempotency Token:

Client: CreateKeyPair(name="prod-key", token="abc-123")
[Creates key, stores result with token]
[Network timeout]
Client: CreateKeyPair(name="prod-key", token="abc-123") [same token!]
[Returns cached result with same private key - safe!]

Implementation:

// hive/idempotency/idempotency.go
package idempotency

import (
    "sync"
    "time"
)

type TokenCache struct {
    cache map[string]CachedResponse
    mu    sync.RWMutex
}

type CachedResponse struct {
    Response  []byte
    ExpiresAt time.Time
}

var globalCache = &TokenCache{
    cache: make(map[string]CachedResponse),
}

func (t *TokenCache) Get(token string) ([]byte, bool) {
    t.mu.RLock()
    defer t.mu.RUnlock()

    cached, exists := t.cache[token]
    if !exists || time.Now().After(cached.ExpiresAt) {
        return nil, false
    }
    return cached.Response, true
}

func (t *TokenCache) Set(token string, response []byte, ttl time.Duration) {
    t.mu.Lock()
    defer t.mu.Unlock()

    t.cache[token] = CachedResponse{
        Response:  response,
        ExpiresAt: time.Now().Add(ttl),
    }
}

func (t *TokenCache) Cleanup() {
    t.mu.Lock()
    defer t.mu.Unlock()

    now := time.Now()
    for token, cached := range t.cache {
        if now.After(cached.ExpiresAt) {
            delete(t.cache, token)
        }
    }
}

// Start cleanup goroutine
func init() {
    go func() {
        ticker := time.NewTicker(5 * time.Minute)
        defer ticker.Stop()
        for range ticker.C {
            globalCache.Cleanup()
        }
    }()
}

Usage in Operations:

func EC2_Process_CreateImage(jsonData []byte) (output []byte) {
    var input ec2.CreateImageInput
    json.Unmarshal(jsonData, &input)

    // Check idempotency token
    if input.ClientToken != nil {
        if cached, found := idempotency.globalCache.Get(*input.ClientToken); found {
            slog.Info("Returning cached result for idempotency token",
                "token", *input.ClientToken)
            return cached
        }
    }

    // Validate and process
    err := ValidateCreateImageInput(&input)
    if err != nil {
        return utils.GenerateErrorPayload(err)
    }

    // Create the image
    result := actuallyCreateImage(input)
    jsonResponse, _ := json.Marshal(result)

    // Cache result with token (24 hour TTL)
    if input.ClientToken != nil {
        idempotency.globalCache.Set(*input.ClientToken, jsonResponse, 24*time.Hour)
    }

    return jsonResponse
}

6. Resource State Machines

Purpose: Enforce valid state transitions for resources (instances, volumes, etc.).

Implementation:

// hive/statemachine/instance.go
package statemachine

type InstanceState int

const (
    StatePending      InstanceState = 0
    StateRunning      InstanceState = 16
    StateShuttingDown InstanceState = 32
    StateTerminated   InstanceState = 48
    StateStopping     InstanceState = 64
    StateStopped      InstanceState = 80
)

var stateNames = map[InstanceState]string{
    StatePending:      "pending",
    StateRunning:      "running",
    StateShuttingDown: "shutting-down",
    StateTerminated:   "terminated",
    StateStopping:     "stopping",
    StateStopped:      "stopped",
}

// Valid state transitions
var validTransitions = map[InstanceState][]InstanceState{
    StatePending: {
        StateRunning,
        StateTerminated, // Can terminate while pending
    },
    StateRunning: {
        StateStopping,
        StateShuttingDown,
    },
    StateStopping: {
        StateStopped,
        StateTerminated, // Force terminate
    },
    StateStopped: {
        StatePending, // Restart
        StateTerminated,
    },
    StateShuttingDown: {
        StateTerminated,
    },
    StateTerminated: {
        // Terminal state - no transitions
    },
}

func (s InstanceState) String() string {
    return stateNames[s]
}

func (s InstanceState) Code() int64 {
    return int64(s)
}

func ValidateTransition(from, to InstanceState) error {
    allowed, exists := validTransitions[from]
    if !exists {
        return fmt.Errorf("invalid current state: %s", from)
    }

    for _, valid := range allowed {
        if valid == to {
            return nil
        }
    }

    return fmt.Errorf("cannot transition from %s to %s", from, to)
}

// Usage in instance operations
func StopInstance(instanceId string) error {
    instance := findInstance(instanceId)
    currentState := InstanceState(instance.State.Code)

    if err := ValidateTransition(currentState, StateStopping); err != nil {
        return awserrors.FormatError(
            awserrors.ErrorIncorrectInstanceState,
            instanceId)
    }

    // Proceed with stop operation
    instance.State.Code = aws.Int64(StateStopping.Code())
    instance.State.Name = aws.String(StateStopping.String())
    // ... perform actual stop

    return nil
}

7. Pagination Framework

Purpose: Handle large result sets consistently across all operations.

Implementation:

// hive/pagination/pagination.go
package pagination

import (
    "encoding/base64"
    "encoding/json"
    "fmt"
)

type PageToken struct {
    Offset    int    `json:"offset"`
    Timestamp int64  `json:"timestamp"`
}

type Paginator struct {
    MaxResults int
    NextToken  string
}

func (p *Paginator) Paginate(items []interface{}) (page []interface{}, nextToken string, err error) {
    maxResults := p.MaxResults
    if maxResults == 0 {
        maxResults = 100 // Default page size
    }

    offset := 0
    if p.NextToken != "" {
        var token PageToken
        decoded, err := base64.StdEncoding.DecodeString(p.NextToken)
        if err != nil {
            return nil, "", fmt.Errorf("invalid next token")
        }
        if err := json.Unmarshal(decoded, &token); err != nil {
            return nil, "", fmt.Errorf("invalid next token")
        }
        offset = token.Offset
    }

    if offset >= len(items) {
        return []interface{}{}, "", nil
    }

    end := offset + maxResults
    if end > len(items) {
        end = len(items)
    }

    page = items[offset:end]

    // Generate next token if more results exist
    if end < len(items) {
        token := PageToken{
            Offset:    end,
            Timestamp: time.Now().Unix(),
        }
        tokenJSON, _ := json.Marshal(token)
        nextToken = base64.StdEncoding.EncodeToString(tokenJSON)
    }

    return page, nextToken, nil
}

Usage:

func DescribeImages(input *ec2.DescribeImagesInput) (output ec2.DescribeImagesOutput, err error) {
    // Get all matching images
    allImages := queryAllImages(input.Filters)

    // Paginate results
    paginator := pagination.Paginator{
        MaxResults: aws.IntValue(input.MaxResults),
        NextToken:  aws.StringValue(input.NextToken),
    }

    page, nextToken, err := paginator.Paginate(interfaceSlice(allImages))
    if err != nil {
        return output, err
    }

    output.Images = convertToImageList(page)
    if nextToken != "" {
        output.NextToken = aws.String(nextToken)
    }

    return output, nil
}

Lower Priority (Production Features)

These features are important for production but can be implemented later:

8. ARN Support

// hive/arn/arn.go
func GenerateARN(service, region, accountId, resourceType, resourceId string) string {
    return fmt.Sprintf("arn:aws:%s:%s:%s:%s/%s",
        service, region, accountId, resourceType, resourceId)
}

// Example: arn:aws:ec2:us-east-1:123456789012:instance/i-1234567890abcdef0

9. Tagging Framework

// hive/tags/tags.go
type TagService struct {
    store TagStore
}

func (t *TagService) CreateTags(resourceId string, tags []*ec2.Tag) error
func (t *TagService) DeleteTags(resourceId string, tags []*ec2.Tag) error
func (t *TagService) DescribeTags(filters []*ec2.Filter) ([]*ec2.TagDescription, error)

10. Audit Logging (CloudTrail-like)

// hive/audit/audit.go
type AuditLog struct {
    Timestamp    time.Time       `json:"timestamp"`
    RequestId    string          `json:"requestId"`
    UserIdentity string          `json:"userIdentity"`
    EventName    string          `json:"eventName"`
    SourceIP     string          `json:"sourceIP"`
    Request      json.RawMessage `json:"request"`
    Response     json.RawMessage `json:"response"`
}

func LogAPICall(log AuditLog) {
    // Write to audit log storage (file, database, S3)
}

11. Filter Engine

// hive/filters/filters.go
type FilterEngine struct{}

func (f *FilterEngine) Match(filters []*ec2.Filter, resource interface{}) bool {
    // Parse AWS filter format
    // Support wildcards
    // Handle tag filters
}

12. Resource Dependency Tracking

// hive/dependencies/dependencies.go
type DependencyGraph struct {
    edges map[string][]string
}

func (d *DependencyGraph) AddDependency(parent, child string)
func (d *DependencyGraph) CanDelete(resourceId string) error

Implementation Strategy

Phase 1: Core Compatibility (Weeks 1-2)

1. Request ID tracking
2. DryRun implementation
3. Comprehensive error codes
4. Integration tests with AWS SDK

Phase 2: Reliability (Weeks 3-4) 5. Idempotency tokens 6. Resource state machines 7. Pagination framework

Phase 3: Production Features (Weeks 5-8) 8. ARN support 9. Tagging framework 10. Audit logging 11. Filter engine 12. Dependency tracking

Success Metrics

• AWS CLI Compatibility: 100% command format compatibility
• Error Handling: All AWS error codes properly mapped
• Retry Safety: All write operations support idempotency
• Integration Tests: >95% pass rate with real AWS SDK
• State Correctness: Zero invalid state transitions in testing